main: ingest Superset data into datahub

Brouberolrequested to merge
T306903 into main

We add the superset source to the datahub ingestion DAG. We have to copy the config file from ${AIRFLOW_HOME/secrets, as the file contains a password.

We stumbled into an issue seemingly related to the old version (0.10.4) of the datahub CLI/library we use (we run datahub 0.13.3). However, as we keep the datahub library installed in the airflow image in sync with datahub itself, we have exactly what we need in the airflow worker image itself. We can thus resort to a simple BashOperator task to subprocess to the datahub CLI, instead of going through Skein/YARN.

The same logic should apply to the other tasks as well, which will be migrated in a subsequent MR.

Depends on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1178829

Signed-off-by: Balthazar Rouberol brouberol@wikimedia.org Bug: T306903

Closes T306903

Edited by Brouberol

Merge request reports