Add monthly Airflow DB cleaner DAG to all instances (!1899) · Merge requests · repos / data-engineering / Airflow DAGs · GitLab

How to register an account on GitLab. Due to spam, new accounts are locked until approved by an admin or the approver bot. Your GitLab account gets automatically approved within one hour if you are a member of Trusted Contributors in Gerrit, or a member of the Trusted-Contributors group in Phabricator and linked your Developer account ("LDAP") to your Phabricator account. If none of these apply, file an unlock request to expedite access. Only if you are WMF staff: Create your GitLab account, have "Wmf" group membership in IDM, link your Developer account ("LDAP") to your Phabricator account, and get automatically approved.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Create db_cleaner_dag_factory and add maintenance DAGs that run monthly on day 3:

to clean old Airflow DB records (180 day retention).
to run vacuum and reindex on PG tables
to check global cache hit ratio of PG db

Deploy:

we should unpause those dags progressively on each instance.
First re-index vacuum operatins may be slow. Watch out.

Edited Jan 12, 2026 by Aqu