Skip to content
  • Gmodena's avatar
    wmf_airflow_common: add drop_older_than utility method. · 8fd0654a
    Gmodena authored
    - Adds a utility method to delete partitions older
    than a specified threshold. This method wraps an existing
    refinery script used to purge data.
    
    - Adds a decorator to execute Python scripts by wrapping
    the `BashOperator`. This approach is designed to facilitate
    swapping in the `SkeinOperator` once the refinery Python code
    is extracted into a standalone module.
    
    - Mocks `os.path.isfile` in the `dagbag` test fixture so
    that it always returns `True` when tests involving `dagbag`
    include tasks requiring access to 
    `/srv/deployment/analytics/refinery/bin/refinery-drop-older-than`.
    Without this mock, refinery must be deployed on the test infrastructure. As noted in the comments, this is a workaround necessitated by the fact that refinery Python modules are not yet packaged and bundled.
    
    
    
    
    8fd0654a