Workaround for Spark 2 (!181) · Merge requests · repos / data-engineering / Airflow DAGs · GitLab

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Marco Fossati requested to merge image-suggestions-spark2-workaround into main Nov 07, 2022

This is a temporary fix that should be reverted as soon as https://phabricator.wikimedia.org/T322366 is resolved.

pass spark2-submit path and Spark 2 configuration directory to every Airflow task but the sensors
improve comments