dbt: Generate one DAG per logical group from YAML config
The dbt_scheduled system generated exactly three static DAGs (dbt_daily, dbt_weekly, dbt_monthly), forcing unrelated projects to share a single start_date, one of three fixed schedules, a common DAGRun state, and a shared alerting surface. As a result a project could not have its own start date or an arbitrary cadence, one failed task failed the whole run for everyone, and run history became ragged as jobs were added to the long-lived DAGs over time.
Extend the YAML configuration with a "group" entry that generates its own DAG (dbt_) with an independent schedule, start_date and catchup, plus an optional pool. Job definitions are shared via a DbtJobConfig model and wired (sensors + DbtSkeinOperator) through a common helper. The legacy flat "scheduled: daily|weekly|monthly" format is left untouched and still produces the three DAGs, so existing team configs keep working and can migrate incrementally. The "dbt_" DAG-id prefix is now a single named constant.
Add a demo_group.yaml fixture and a test covering a pooled group, a poolless group, and an arbitrary cron cadence.
Add support for new weekly run schedules that occur on specific weekdays.
Airflow devenv testing:
- Proof that the new code - using the current (old) version of YAML config file - produces the same dbt monthly DAG:
- Proof that the new code - using an updated YAML that creates a group/DAG out of two existing jobs, scheduled to run on Wednesdays - produces the correct DAGs using the following new config:
- group: media_traffic_pa
schedule: weekly_starting_wednesday
start_date: 2026-07-01
catchup: false
pool: dbt_weekly_pool
jobs:
- job_name: media_traffic_ua_compliance_daily
models:
- +mrt_media_traffic_ua_compliance_dailyBug: T429439
Assisted-by: Claude Opus 4.8
Closes T429439


