Skip to content

Add Dataset module to easily generate Sensors

Mforns requested to merge add-datasets into main

Adds the new Dataset concept. Datasets are objects defined by YAML configuration that represent a source dataset. They have the ability to easily generate a sensor for the given DAG. This change also adds some tests and a syntactic sugar (in analytics dag_config.py) so it's easy to get a particular dataset from a DAG's code.

The datasets are defined in a datasets.yaml file within each project's config folder. In the future, we plan to populate dataset properties from DataHub.

DEPENDS ON: !207 (merged)

Edited by Mforns

Merge request reports