Configure fileoutputcommitter.algorithm.version in the Spark Operator

We wish to set a default value of 2 for the spark mapreduce fileoutputter committer algorithm in order to reduce the likelihood of data quality issues.

The preferred way of achieving this is to pass this spark configuration option from the WMF Spark Operator, using a new optional parameter.

The default value of 2 will be passed, but users may override this to 1.

Bug: T351388

Merge request reports

Loading