Skip to content

Add DAG to do monthly historic reconcile.

Xcollazo requested to merge T377852-optimize-reconcile-for-historic-runs into main

(Depends on repos/data-engineering/dumps/mediawiki-content-dump!52 (merged).)

In this MR we:

  • Introduce a new DAG to do historic reconciles.
  • Refactor existing daily reconcile job a bit to share code with new DAG.

We pondered whether to repeat less code by having a single python file that could instantiate both daily and monthly runs, but the code was quickly becoming quite hard to read and follow. Thus decided to stick with separate DAGs that repeat code a bit.

Bug: T377852

Edited by Xcollazo

Merge request reports