Skip to content

test_k8s/dumpsv1: generate multiple DAGs for regular wiki dumps to keep UI loading time low

This patch splits the regular dumps DAG in multiple DAGs, each one managing a bucket of wikis, associated to an first letter interval. Example: the first bucket contains all wikis with first letter going from a to c.

This will ensure that the DAGs can be observed by a human while minimizing UI load time.

The changes are currently visible at https://airflow-test-k8s.wikimedia.org/home?tags=dumps

Signed-off-by: Balthazar Rouberol brouberol@wikimedia.org Bug: T391684

Edited by Brouberol

Merge request reports

Loading