Add a factory for temporary volumes scoped to a DAG run

Some pipelines need scratch space that is shared between the tasks of a
single DAG run, but has no value once the run is over: for example
downloading source data, transforming it, and uploading the result.

The new temp_volume() factory in wmf_airflow_common builds a pair of
tasks that create a PersistentVolumeClaim at the start of the run and
delete it at the end, together with the volume and mount objects that
the tasks in between pass to their KubernetesPodOperator. The teardown
task runs with trigger_rule ALL_DONE, so the volume is removed however
the run ends.

Storage is requested by its properties rather than by storage class
name, so DAG authors do not need to know the cluster's storage classes
and the mapping is maintained in one place, next to the team operating
the storage. The properties are the location of the storage (network,
the default, or node-local for I/O too intensive for network storage),
whether several pods need to mount the volume at the same time, and the
kind of disks backing it (hdd, ssd or nvme). Combinations that no
storage class offers, such as a node-local volume with concurrent
access, are refused at DAG parse time.

The create and delete tasks tolerate conflicts with themselves: a
retried create finds the PVC of its own run and reuses it (keeping any
data written so far), and a teardown whose create never ran succeeds on
finding nothing to delete.

The temp_volume DAG on the test-k8s instance is both the integration
test and the reference for copying. Triggering it with the run
configuration {"fail": true} makes its verify task fail on purpose, to
demonstrate that the volume is deleted on the failure path too. Its
task pods do not name an image, so they run whatever airflow image the
instance currently deploys.

Before DAGs can use this, the instance namespace needs an extra_rbac
rule in operations/deployment-charts allowing the airflow service
account to manage PVCs, and it must be listed in the tenantNamespaces
of the release providing the storage (topolvm for node-local volumes,
ceph-csi-rbd or ceph-csi-cephfs for network ones).

Bug: T428235

Edited by Btullis

Merge request reports

Loading