Skip to content

search: drop_old_data: refactor to util.drop_older_than method.

Gmodena requested to merge search-refactor-drop_old_data into main

This MR is a proposal to refactor data retention tasks to use the wmf_airflow_common drop_older_than helper method.

Context

!940 (merged) introduced a new drop_older_than utility method that wraps refinery's drop-older-than script for data retention.

Data retention, and execution of python scripts, for the drop_old_data could now be expressed using wmf_airflow_common APIs instead of an ad-hoc implementation.

Note however that the API is tagged as experimental because of an RFC (pending review) that propose to generalize data retention config via the DatasetRegistry API.

cc / @dcausse @ebernhardson

References

Merge request reports