Move anomaly_detection jobs to spark3 (!84) · Merge requests · repos / data-engineering / Airflow DAGs · GitLab

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Joal requested to merge spark3_anomaly_detection into main Jun 10, 2022

Update factory to differenciate between dag_id and processed dataset
Move DAG factory to the DAG folder as it is not reused
Use spark3 default_args in dags
Use refinery-job v0.2.1 in dags
Add minimal test for each dag

Edited Jun 13, 2022 by Joal