Skip to content

search: Port incoming_links from airflow 1

Ebernhardson requested to merge work/ebernhardson/incoming-links into main

Dag performs weekly aggregation over cirrus indexes, imported to hadoop by another dag, to determine the number of incoming links for all wiki pages. Those are then shipped by a third dag, along with other weekly updates, back to the production search clusters.

Relatively direct copy of the dag from airflow 1. Typical adjustments to where variables are loaded from and replacing the render-time templating with import-time tempating for the most part.

Bug: T329875

Merge request reports