Added pageview_wme_transfer

This MR adds the pageview_wme_transfer DAG, which transfers hourly WME pageview output from WMF HDFS to WME S3.

What data is being transferred Hourly output of the wme_pageviews_hourly DAG's format_output task, written to hdfs://analytics-hadoop/wmf/data/wme/pageview_formatted_hourly//part-00000.json.snappy. This is aggregated WME-formatted data. Output is written to s3://wme-analytics/pageviews/////part_0.json.snappy.

Design Schedule: @hourly Tasks: 1 URLSensor polling for HDFS output file + 1 transfer pod Credentials: IAM Roles Anywhere, no static AWS credentials Encryption: AES256 server-side encryption Egress: via url-downloader.eqiad.wikimedia.org:8080 proxy Idempotency: deterministic S3 keys, retries safely overwrite Validation: post-upload head_object check inside transfer image

Docker image docker-registry.wikimedia.org/repos/wme/pageviews-hdfs-transfer:v0.1.0 Source: https://gitlab.wikimedia.org/repos/wme/pageviews-hdfs-transfer

Bug: T417029 Bug: T426017

Edited by Sg912

Merge request reports

Loading