Skip to content
  • Aleksandar Mastilovic's avatar
    Adding support for "produced_by" configuration of datasets · 1f183ef3
    Aleksandar Mastilovic authored
    * Linter fix
    
    * Addressing code review nitpick, fix CI/CD pipeline
    
    * Fixing ensure_multiple_reviewers.py - mr.discussions.list() needs an extra param to retrieve all of the comments
    
    * Making the linter happy...
    
    * Making the linter happy
    
    * Addressing review comments
    
    * Moved AIRFLOW_INSTANCE_MAP into wmf_airflow_common.config
    * Added tests for produced_by sensors
    * produced_by sensor task_id abbreviated
    * Depending on which Airflow instance is configured in produced_by,
    return either ExternalTaskSensor or RestExternalTaskSensor
    * Better class and method docs
    
    * Typing and linting fixes
    
    * Adding support for "produced_by" configuration of datasets
    
    Implementation of the proposal outlined in
    https://phabricator.wikimedia.org/T372647
    
    Airflow producer dataset annotation
    
    NOTE: Missing implementation of automatic configuration of
    `execution_delta` based on target DAG's schedule
    
    Example:
    
    ```
      produced_by:
        airflow:
          instance: search
         ...
    1f183ef3