Move clickstream job to spark3 (!79) · Merge requests · repos / data-engineering / Airflow DAGs · GitLab

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Joal requested to merge spark3_clickstream into main Jun 09, 2022

Update dag default_args to use spark3
Move sla to the ETL step for consistency
Update HDFSArchiveOperator classpath to be spark3 compatible (this operator is only used in this job for now)