main: Add DAGs to do MW Content File Export

Xcollazorequested to merge
file-export-dag into main

This MR introduces 3 new DAGs:

mw_content_xml_export_history_monthly: exports wmf_content.mediawiki_content_history_v1 on the 1st.

mw_content_xml_export_current_monthly: exports wmf_content.mediawiki_content_current_v1 on the 1st.

mw_content_xml_export_current_mid_month: exports wmf_content.mediawiki_content_current_v1 on the 15th.

Some general notes:

  • The idea here is to honor the schedule of DumpsV1, thus 3 DAGs.
  • All these DAGs reuse the same code, but change the schedule and/or the source table to accomplish the file export.
  • We declare 3 separate tunings for huge, big, and regular wikis. Huge wikis will use ~20% of the cluster reasources, big wikis ~12%, and regular wikis ~3%.

A note on cluster usage:

A first implementation used a single dynamically mapped task group to do all the exports, controlled with a single Airflow pool. However, since Airflow does not support dynamically setting pool_slots, this resulted in overwhelming the cluster when multiple huge or big wikis were running in parallel. Thus, the current implementation segregates wikis into two dynamically mapped task groups: export_small_wikis, and export_big_and_huge_wikis. We declare a pool with 16 slots for the small wikis, and a pool with 1 slot for the rest.

This way, we cap reasource usage quite well, with shorter bursts to ~25% of cluster, but typically closer to ~20%.

Some statistics from a full run of mw_content_xml_export_history_monthly with the 2 pools approach:

export_small_wikis exported 855 wikis in ~11h30m. For this group, loginwiki was the fastest at 36s, while metawiki was the longest at 2h11m.

export_big_and_huge_wikis exported 26 wikis in ~5d18n35m. Here are the individual runtimes of these bigger wikis, in order:

shwiki	0:49:29
fawiki	0:50:57
kowiki	0:57:27
svwiki	0:58:00
enwiktionary	0:59:57
ukwiki	1:00:58
frwiktionary	1:03:12
hewiki	1:03:28
arwiki	1:03:42
cebwiki	1:05:42
viwiki	1:06:29
trwiki	1:07:28
plwiki	1:10:29
cawiki	1:12:58
ptwiki	1:18:31
nlwiki	1:19:02
zhwiki	1:57:18
jawiki	2:00:05
itwiki	2:17:04
commonswiki	2:32:50
eswiki	2:42:00
ruwiki	2:52:22
frwiki	3:07:39
dewiki	4:45:46
enwiki	1d02:20:55
wikidatawiki	3d05:36:40

Side note: On a recent test run, 2 wikis (mswikiquote and thwikimedia) failed because they were added to the open dblist quite recently but data for their SiteInfo was not available yet. This will generate a bit of work for an OpsWeek person to double check wheter failures are legit, or because of new wikis not ready yet.

Bug: T384381

Edited by Xcollazo

Merge request reports