Skip to content

Split dump task into a chain of tasks, one per dump job type

Brouberol requested to merge T390852 into main

This allows us to execute one airflow task per dump job type, which will in the end allow us to interweave synchronization steps between each dump job (once a dump job/task is done, synchronize the generated files to NFS).

This generates a chain of tasks (one per job type), each of them executing for each wiki returned by the NOC api call (or from an airflow variable override)

Screenshot_2025-04-03_at_11.36.00

Signed-off-by: Balthazar Rouberol brouberol@wikimedia.org Bug: T390852

Edited by Brouberol

Merge request reports

Loading