Make wmf_airflow_common installable.
@otto @mforns I'm marking this MR as draft because it's just a conversation starter, that I opened to brainstorm.
This configuration change allows to build
a binary distribution of wmf_airflow_common
with bdist_wheel
.
The artefact will be available under dist/
.
This, I think, is the minimum amount of work we could to:
- keep you debian packaging build workflow sane.
- make
wmf_airflow_common
reusable / orthogonal to the monorepo.
There will be a few wrinkles to iron out, but hopefully we can start to evaluate this approach.
Example
$ pip install wheel
$ python setup.py bdist_wheel
$ pip install dist/wmf_airflow_dags-0.1.0-py3-none-any.whl
$ cd /tmp # I'm out of the monorepo PYTHONPATH.
$ python
from wmf_airflow_common.operators.spark import SparkSkeinSubmitHook
/private/tmp/venv/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/private/tmp/venv/lib/python3.8/site-packages/airflow/configuration.py:345 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
We can also install an sdist
from the gitlab repo with:
$ pip install git+https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git@gmodena-package-wmf_airflow_common#wmf_airflow_common