Skip to content

Use URLSensor to wait on monthly partitions

Xcollazo requested to merge T311417-use-urlsensor into main

This work is part of https://phabricator.wikimedia.org/T311417.

There was a correctness issue with using HivePartitionSensor: We were waiting on the availability of the 'snapshot' partition, but that does not guarantee that the inner 'wiki_db' partitions were present. By waiting on special file '_PARTITIONED', we avoid this issue. We now run the monthly sensors in parallel as well.

Merge request reports