Skip to content

Draft: Add initial evaluation dag for search-platform

Moves over one of the simpler dags from the search-platform airflow v1 installation to the shared airflow repository. Migrates a small amount of the test suite that is relevant to this dag as well. Overall intent is to build some familiarity with the relevant conventions and begin to verify the shared airflow installation will work for search platforms needs.

Adds one bit outside of the dedicated search_platform directories, to facilitate passing the REBUILD_FIXTURES env var through tox and past the fixture that overrides environment vars. This supports the existing search-platform test methodology of fully rendering a task and recording how it will be run in production to files. This method has helped in the past to recognize what all is being changed when shared functionality is changed, along with giving an easy place to lookup the complete invocation to understand what exactly airflow is going to invoke under the hood when running a task in spark. The general idea is that when the fixture tests fail the fixture will be deleted, pytest will be run with REBUILD_FIXTURES=yes, and the git diff output will be reviewed to ensure all changes are expected and desired.

Bug: T318414

Merge request reports