Two changes aimed to improve the development:
- when running the airflow dev instance as a service user (to be able to use
SparkSkeinSubmitHookduring development), add an flag to
wmf_airflow_commoninto the conda environment used for the airflow dev instance.
- add a
postambleargument to complement the
preambleargument to the
SparkSkeinSubmitHook. One use case is to use
command_postamble = "|| sleep 600", which keeps the skein application running in case the submit command failed for debugging.