refine_to_hive_hourly: Increase dag_delay to 1h30m
Adjust dag_delay from 25 minutes to 1 hour 30 minutes to ensure DAG runs are enqueued to ensure Gobblin has begun creating raw partitions. This change accounts for the sensor's 3-hour data existence check window (current and next 2 hours).
Example:
- refine_to_hive dag run of 8:00am It will sense for raw folders data of hours 8, 9 and 10.
- currently refine_to_hive is queued at 9:25am
- at this time Gobblin (running at 9:20 and 9:25 depending on events) is about to import data from hours 8 and 9 (if not already done)
- canary events for hour 10 will be generated at 10:15
- beginning 10:20 Gobblin will create new raw directory for hour 10
- so refine_to_hive waits at least 1 hour between 9:25 and 10:20 before works actually begins.
- and 2 refine_to_hive dagruns are active for part of the hour
- With this patch refine_to_hive queued at 10:30am
- as soon as sensors are launched, some will succeed.
- In regular mode a single dagrun will be active at any given time
Fewer dagruns active, fewer task instances active, less work for Airflow to check their scheduling status.
Bug: T392668