refine_to_hive_hourly: Increase dag_delay to 1h30m

Adjust dag_delay from 25 minutes to 1 hour 30 minutes to ensure DAG runs are enqueued to ensure Gobblin has begun creating raw partitions. This change accounts for the sensor's 3-hour data existence check window (current and next 2 hours).

Example:

  • refine_to_hive dag run of 8:00am It will sense for raw folders data of hours 8, 9 and 10.
  • currently refine_to_hive is queued at 9:25am
    • at this time Gobblin (running at 9:20 and 9:25 depending on events) is about to import data from hours 8 and 9 (if not already done)
    • canary events for hour 10 will be generated at 10:15
    • beginning 10:20 Gobblin will create new raw directory for hour 10
    • so refine_to_hive waits at least 1 hour between 9:25 and 10:20 before works actually begins.
    • and 2 refine_to_hive dagruns are active for part of the hour
  • With this patch refine_to_hive queued at 10:30am
    • as soon as sensors are launched, some will succeed.
    • In regular mode a single dagrun will be active at any given time

Fewer dagruns active, fewer task instances active, less work for Airflow to check their scheduling status.

Bug: T392668

Edited by Aqu

Merge request reports

Loading