Skip to content

search: updater reconcile, do not create the dag dynamically

DCausse requested to merge search-build-updater-reconcile-statically into main

Creating multiple sensors looping over an array set by config makes the dag dynamic. This error prone and has many undesired side-effets in airflow:

  • the tasks might be marked as "removed" if it disappears for one run
  • if the tasks is created again it might remain inactive (unclear state)
  • might cause the whole dag to stall in some unknown conditions

Prefer to have a dag that is statically created and not dependent on a config var.

WIP: needs a new build of the rdf-tools jar with: https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/923380

Merge request reports