Skip to content

Make wmf_airflow_common installable.

Gmodena requested to merge gmodena-package-wmf_airflow_common into main

@otto @mforns I'm marking this MR as draft because it's just a conversation starter, that I opened to brainstorm.

This configuration change allows to build a binary distribution of wmf_airflow_common with bdist_wheel.

The artefact will be available under dist/.

This, I think, is the minimum amount of work we could to:

  1. keep you debian packaging build workflow sane.
  2. make wmf_airflow_common reusable / orthogonal to the monorepo.

There will be a few wrinkles to iron out, but hopefully we can start to evaluate this approach.

Example

$ pip install wheel 
$ python setup.py bdist_wheel
$ pip install dist/wmf_airflow_dags-0.1.0-py3-none-any.whl
$ cd /tmp # I'm out of the monorepo PYTHONPATH.
$ python
from  wmf_airflow_common.operators.spark import SparkSkeinSubmitHook
/private/tmp/venv/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/private/tmp/venv/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.

We can also install an sdist from the gitlab repo with:

$ pip install git+https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git@gmodena-package-wmf_airflow_common#wmf_airflow_common
Edited by Gmodena

Merge request reports