analytics: Allow SparkSqlOperator to run Spark 3.3.2. Do Iceberg table maintenance with Spark 3.3.2.

Xcollazorequested to merge
table-maintenance-via-iceberg-161 into main

(This MR pulls artifacts from https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver and https://gitlab.wikimedia.org/repos/data-engineering/patches/pyspark-sql-conda-env).

In this MR we:

  • Introduce a spark_version: Literal["3.1.2", "3.3.2"] parameter on SparkSqlOperator to be able to run Spark 3.3.2 via a conda environment on top of YARN. This mechanism tries to hide the complexities of conda environments away by configuring all 3.3.2 bits automatically. We default to the usual Spark 3.1.2 run directly over YARN otherwise.
  • Refactors Iceberg table maintenance tasks to use the above.

Why a SparkSql332Operator operator? Why not just add a parameter to SparkSqlOperator?

The configuration necessary for SparkSql332Operator is very specific to the use of a conda enviornment, the settings to pickup Spark 3.3.2, and the fact that this operator may be temporary till we upgrade the cluster. An inheritance relationship seems more appropriate, and keeps SparkSqlOperator clean.

Bug: T391280

Edited by Xcollazo

Merge request reports