Skip to content

Use Spark 3.3 Shuffler for Dumps 2.0 pipelines.

Xcollazo requested to merge T352890-use-spark-33-shuffler into main

In this MR we modify the Dumps 2.o pipelines so that they leverage the Spark 3.3 shuffler that we setup on https://phabricator.wikimedia.org/T344910.

We also tidy up all the parameters needed to signal that we want to use Spark 3.3.2.

Bug: T352890

Edited by Xcollazo

Merge request reports