Skip to content

Use a `URLSensor` to wait for the whole wikitext snapshot

Marco Fossati requested to merge section-alignment-wikitext-sensor-fix into main

The default behavior of Spark tasks is to process all Wikipedias. Hence, a NamedHivePartitionSensor won't work if we wait for the whole wikitext monthly snapshot, while it's fine if specific Wikipedias are passed.

  • Separate the wikitext sensor building logic accordingly
  • set the poke interval of all sensors to 1 hour, as we do section topics and image suggestions DAGs

Merge request reports