Skip to content

Create a script for packaging multiple spark shufflers for yarn

Btullis requested to merge rewrite_script into main

We now want to deploy multiple version of the spark shuffler service for yarn. Currently we need to build packages for both buster and bullseye, although the jar file will be binary compatible.

We have already built the required spark versions from source and published the build image on docker-registry.wikimedia.org so this script uses docker to pull those images and extract the jar file. It then uses templated files to create the required debian control files and runs pdebuild to create the deb files on a Wikimedia build server.

Bug: T344910

Merge request reports