dbt: Generate one DAG per logical group from YAML config

The dbt_scheduled system generated exactly three static DAGs (dbt_daily, dbt_weekly, dbt_monthly), forcing unrelated projects to share a single start_date, one of three fixed schedules, a common DAGRun state, and a shared alerting surface. As a result a project could not have its own start date or an arbitrary cadence, one failed task failed the whole run for everyone, and run history became ragged as jobs were added to the long-lived DAGs over time.

Extend the YAML configuration with a "group" entry that generates its own DAG (dbt_) with an independent schedule, start_date and catchup, plus an optional pool. Job definitions are shared via a DbtJobConfig model and wired (sensors + DbtSkeinOperator) through a common helper. The legacy flat "scheduled: daily|weekly|monthly" format is left untouched and still produces the three DAGs, so existing team configs keep working and can migrate incrementally. The "dbt_" DAG-id prefix is now a single named constant.

Add a demo_group.yaml fixture and a test covering a pooled group, a poolless group, and an arbitrary cron cadence.

Add support for new weekly run schedules that occur on specific weekdays.

Airflow devenv testing:

  • Proof that the new code - using the current (old) version of YAML config file - produces the same dbt monthly DAG:

Screenshot_2026-07-14_at_11.17.28_AM

  • Proof that the new code - using an updated YAML that creates a group/DAG out of two existing jobs, scheduled to run on Wednesdays - produces the correct DAGs using the following new config:
- group: media_traffic_pa
  schedule: weekly_starting_wednesday
  start_date: 2026-07-01
  catchup: false
  pool: dbt_weekly_pool
  jobs:
  - job_name: media_traffic_ua_compliance_daily
    models:
    - +mrt_media_traffic_ua_compliance_daily

Screenshot_2026-07-14_at_11.49.30_AM Screenshot_2026-07-14_at_3.53.02_PM

Bug: T429439

Assisted-by: Claude Opus 4.8

Closes T429439

Edited by Aleksandar Mastilovic

Merge request reports

Loading