T389209 T389208 T389207 T389206 T389205 Graphite process DAGs

Contributor checklist

• I have written tests for this DAG that will be merged into data-engineering/airflow-dags/tests/wmde
• I have locally ran the above tests and code quality checks as outlined in the tests section of the Airflow DAGs project readme
• I have tested the jobs for this DAG in my local database using the process defined in the following directories within wmde/analytics/hql/airflow_jobs:
  • wd_article_placeholder_metrics/_test_daily
  • wd_entity_schema_namespace_metrics_daily/_test_daily
  • wd_reliability_metrics_daily/_test_daily
  • wd_special_entity_data_metrics_daily/_test_daily
  • wd_special_entity_schema_text_metrics_daily/_test_daily
• I have tested the included DAGs in my local database using the process outlined in TEST_AIRFLOW_DAGS.md and the test variable files provided for each DAG
  • Testing has changed since the last time I was releasing DAGs given the migration to K8s and I need to deploy given a March 28th deadline, so I will not be testing on a local Airflow instance
  • I've already made plans with a WMF Data SRE to set up the new testing process
  • All queries do work, and DAGs are copies of processes that we have in production
• All Hive tables that are needed by the included DAG jobs have been created and are accessible by the analytics-wmde Airflow user
  • wmde.wd_article_placeholder_metrics_daily
  • wmde.wd_entity_schema_namespace_metrics_daily
  • wmde.wd_reliability_metrics_daily
  • wmde.wd_special_entity_data_metrics_daily
  • wmde.wd_special_entity_schema_text_metrics_daily

Description

• T389209, T389208, T389207, T389206, T389205: These tasks have been made to migrate the wikidata_metrics_to_graphite_daily_dag sub-processes to their own respective DAGs
  • Each has their own directory with create table and job queries in GitLab:wmde/analytics/hql/airflow_jobs
    • wd_article_placeholder_metrics: Runs a new version of wikidata_articleplaceholder_metrics.hql
    • wd_entity_schema_namespace_metrics_daily: Runs a new version of wikidata_entity_schema_namespace_metrics.hql
    • wd_reliability_metrics_daily: Runs a new version of wikidata_reliability_metrics.hql
    • wd_special_entity_data_metrics_daily: Runs a new version of wikidata_specialentity_data_metrics.hql
    • wd_special_entity_schema_text_metrics_daily: Runs a new version of wikidata_special_entity_schema_text_metrics.hql
  • Basic tests that the DAGs are properly structured have been written.

Test outputs

I've tested the new queries against the old version that were generating metrics for Graphite and results are exact and or consistent given minor WHERE clause alterations that should be done.