Skip to content

common: drop_older_then: check script only on bare metal.

Follow up to !1047 (merged). Fixes behaviour on k8s scheduler.

On bare metal, the refinery script is available on the scheduler host. On k8s it will only be available in a dedicated executor pod.

When importing the dag on k8s, the airflow scheduler will evaluate the path-checking logic and fail with:

Broken DAG: [/opt/airflow/dags/airflow_dags/search/dags/drop_old_data_daily.py]
Traceback (most recent call last):
  File "/opt/airflow/dags/airflow_dags/wmf_airflow_common/operators/python.py", line 61, in wrapper
    script_args = func(**script_kwargs)
  File "/opt/airflow/dags/airflow_dags/wmf_airflow_common/operators/refinery.py", line 49, in drop_older_than
    raise FileNotFoundError(error_msg)
FileNotFoundError: Script not found at /opt/refinery/bin/refinery-drop-older-than. Please ensure refinery is properly installed on your target airflow instance,and the path is correct.

We need to check if drop_older_than_script_path exists on bare metal, but delegate k8s-specific logic to python_script_executor. The decorator is responsible for providing a pod with the necessary dependencies.

cc / @dcausse @brouberol @joal

Bug: T384255

Edited by Gmodena

Merge request reports

Loading