Skip to content

Update the spark3 yarn shuffler jar file

Btullis requested to merge add_spark3_yarn_shuffle into main

The previous attempt to include the spark3 yarn shuffler jar had an error, in that it used an incorrect file.

This version copies the jar file from a binary distribution of spark3 that we build in the production-images repository.

We also update the location of the jar file so that it can be found in /usr/lib/spark3/yarn which is a more standard location.

Bug: T332765

Merge request reports