Skip to content

Customize DAG to use dedicated pool for canary tasks

Aqu requested to merge T369845_put_canary_event_generation_in_custom_pool into main

Canary events are crucial for empty streams, enabling Gobblin to mark _IMPORTED flags correctly.

This patch allows selecting the pool where canary tasks run. By using a dedicated pool, we aim to prevent delays caused by the default pool congestion at start of the hour.

Bug: T369845

Merge request reports