Skip to content

Always set SPARK_HOME on Spark jobs

Xcollazo requested to merge T336800-set-spark-home into main

Phew! This one was nasty!... yet, simple. The answer was in front of me all the time....

In this MR we:

  • Partially revert commit 294f3a27. We only revert the code and not the .expected files since they have evolved.
  • Solve the issue of not being able to find the iceberg jar when running for_virtualenv() with deploy_mode='client' by always setting SPARK_HOME.
  • Update all .expected files accordingly.

Bug: T336800

Edited by Xcollazo

Merge request reports