Add DAG for druid loading of webrequest_sampled_128 (!364) · Merge requests · repos / data-engineering / Airflow DAGs · GitLab

GitLab now enforces expiry dates on tokens that originally had no set expiration date. Those tokens were given an expiration date of one year later. Please review your personal access tokens, project access tokens and group access tokens to ensure you are aware of upcoming expirations. Administrators of GitLab can find more information on how to identify and mitigate interruption, please take a look at our documentation.

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Mforns requested to merge migrate-druid-load-webrequest-128 into main Apr 21, 2023

Adds the DAG for Druid loading of wmf.webrequest sampled at 1/128 rate.

Also fixes the partitions_builder to support multiple-value pre-partitions.

This change should be deployed together with (the queries): https://gerrit.wikimedia.org/r/c/analytics/refinery/+/911890

Edited Apr 25, 2023 by Mforns