Skip to content

Add Spark2 config, and apply it to platform_eng and research instances.

Xcollazo requested to merge T322366-revert-spark3-for-all into main

Solves T322366.

In !141 (merged), we inadvertently changed the default Spark to Spark3 for all Airflow instances.

This happened because we modified wmf_airflow_common/config/dag_default_args.py, which is used by all instances.

In this MR we revert that behavior to what it was intended:

analytics and analytics_test to Spark3, and research and platform_eng to Spark2.

Merge request reports