User-Agent compliance + refine API requests DAGs

Bug: T419522

---

The MR has two main sets of DAGs

User Agent Compliance Metrics

• compute_user_agent_compliance_hourly_dag.py
  • The DAG uses a private artifact, published from the repo: product-analytics/infrastructure-metrics/user-agent-compliance-metrics (reason for keeping private: T422183)
    • Followed Adding an artifact from a private GitLab repository instructions for adding the artifact.
  • Classified User-Agents per compliance definitions, and writes the output to
    • wmf_traffic.user_agent_compliance_classified_hourly
  • Computes compliance of rate of User-Agents of Wikimedia traffic, per the WMF user-agent policy.
    • destination: wmf_traffic.user_agent_compliance_hourly
  • Writes the top non-compliant User-Agents by various dimensions of interest.
    • destination: wmf_traffic.user_agent_top_non_compliant_hourly
• purge_user_agent_classifications_daily_dag.py
  • Purges tables with granular data: wmf_traffic.user_agent_top_non_compliant_hourly and wmf_traffic.user_agent_compliance_classified_hourly
  • per the 90-day retention policy.

API requests DAG

• refine_api_requests_hourly_dag.py
  • Produces refined + enriched API logs data product, destination table: wmf_traffic.api_requests_refined
  • Also, uses are job artifact, published from the report: product-analytics/infrastructure-metrics/api-analytics

Additional info

• Both the DAGs are dependent on https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1288444 to be merged, which is scheduled to be merged on 2026-05-26.
• The DDLs can be seen on this MR: repos/product-analytics/data-pipelines!49 (diffs)

Testing

Have tested all the DAGs using airflow-dev instance

Verified outputs at

• kcvelaga_test.user_agent_compliance_classified_hourly
• kcvelaga_test.user_agent_compliance_hourly
• kcvelaga_test.user_agent_top_non_compliant_hourly
• kcvelaga_test.api_requests_refined