Skip to content

Add Automoderator monthly snapshot files

KCVelaga requested to merge kcvelaga/data-pipelines:automod-monthly-snapshot into main

Bug: T375480

  • Table creation file: create_automoderator_activity_snapshot_monthly.hql
  • Monthly generation and load file: generate_automoderator_activity_snapshot_monthly.hql
  • Publishing a CSV to a temporary directory: publish_automoderator_activity_snapshot_monthly.hql
  • Removing snapshots older than 3 months: purge_automoderator_activity_snapshot_monthly.hql

These are similar to the daily monitoring snapshots. The major difference being that the monthly snapshots are based on mediawiki_history, and are more reliable for analysis, and will be used for reporting metrics related to Automoderator, while the daily snapshots (based on MariaDB-replicas) are regular monitoring.

The step to remove snapshots can be part of the generation step, but I think it is better to separate this step in the Airflow DAG, so kept it as a separate file.

Merge request reports