Skip to content

T388634 Remove export to published datasets from all DAGs

Andrew McAllister (WMDE)requested to merge
T388634-remove-export-to-published-datasets into main

Contributor checklist

  • I have written tests for this DAG that will be merged into data-engineering/airflow-dags/tests/wmde
  • I have locally ran the above tests and code quality checks as outlined in the tests section of the Airflow DAGs project readme
  • I have tested the jobs for this DAG in my local database using the process defined in wmde/analytics/hql/airflow_jobs/PATH_TO_JOB_TESTS
    • No need to test as this is just removing DAG jobs to run the queries, which in turn will be removed
  • I have tested the included DAGs in my local database using the process outlined in TEST_AIRFLOW_DAGS.md and the test variable files provided for each DAG
    • No need to test as this is just removing DAG jobs
  • All Hive tables that are needed by the included DAG jobs have been created and are accessible by the analytics-wmde Airflow user
    • No changes in tables

Description

  • T388634: WMDE made the decision to stop to practice of exporting the data generated by our DAGs to analytics.wikimedia.org/published/datasets/wmde. The practice was causing our task delivery to slow down dramatically and we saw no indication that the data was actually being used.

Test outputs

  • No tests needed as this is just removing DAG jobs

Merge request reports