Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • Airflow DAGs Airflow DAGs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Custom issue tracker
    • Custom issue tracker
  • Merge requests 9
    • Merge requests 9
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • repos
  • data-engineering
  • Airflow DAGsAirflow DAGs
  • Merge requests
  • !218
The source project of this merge request has been removed.

Draft: Add initial evaluation dag for search-platform

Closed Ebernhardson requested to merge (removed):work/ebernhardson/search-platform-cirrus-namespace-map into main Jan 24, 2023
  • Overview 17
  • Pipelines 0
  • Changes 9

Moves over one of the simpler dags from the search-platform airflow v1 installation to the shared airflow repository. Migrates a small amount of the test suite that is relevant to this dag as well. Overall intent is to build some familiarity with the relevant conventions and begin to verify the shared airflow installation will work for search platforms needs.

Adds one bit outside of the dedicated search_platform directories, to facilitate passing the REBUILD_FIXTURES env var through tox and past the fixture that overrides environment vars. This supports the existing search-platform test methodology of fully rendering a task and recording how it will be run in production to files. This method has helped in the past to recognize what all is being changed when shared functionality is changed, along with giving an easy place to lookup the complete invocation to understand what exactly airflow is going to invoke under the hood when running a task in spark. The general idea is that when the fixture tests fail the fixture will be deleted, pytest will be run with REBUILD_FIXTURES=yes, and the git diff output will be reviewed to ensure all changes are expected and desired.

Bug: T318414

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: work/ebernhardson/search-platform-cirrus-namespace-map