Create the first webrequest-based pipeline for SRE

Elukeyrequested to merge
elukey_T402512 into main

Due to an increased abuse of our infrastructure, the SRE team is looking for ways to tag traffic that we have been blocking during the past days via requestctl rules, in order to be able to observe patterns and to hopefully apply more effective countermeasures.

This is the first pipeline of its kind, but we'll have more down the line. The idea is to create a multi-stage DAG that does roughly the following:

  • Retrieve/Query/Crunch/etc.. data from Webrequest via Hive, and dump the result to HDFS (TXT file).
  • Copy the result from HDFS to a Ceph volume.
  • Aggregate and reformat the results to be more easily injected in Haproxy (CDN) via Python operators.
  • Rsync the data to the puppetservers (and from there, deliver them to the right hosts like the CDN).

I have been running it successfully with the Airflow dev environment.

Bug: T402512

Edited by Elukey

Merge request reports