Skip to content

#T329876: Migrate ores_predictions DAG

Peter Fischer requested to merge migrate-search-dag-ores_predictions into main

@ebernhardson, the migrated DAG does not pass the tests yet. One of the reasons is, that the code relies on macros.hive. While initialising the underlying HiveOperator, rendering the task fails as no credentials are mapped to the connection_id=metastore_default. How can we initialise the macro properly?

Further I wondered, if we can replace SkeinOperator with a tuple of wmf_airflow_common.operators.skein.SimpleSkeinOperator (execute python) and wmf_airflow_common.operators.hdfs.HDFSArchiveOperator (upload file). Any thoughts?

Merge request reports