Skip to content

Upgrade image-suggestions to Spark3. Take two.

( This is a redo of !186 (merged) )

In this MR we upgrade the platform_eng Airflow instance to default to Spark3, and we upgrade the image-suggestions job (which is currently the only job on this instance.). We also fix some correctness issues regarding the use of merge().

The airflow job runs all the Spark jobs successfully, including all the Cassandra loading steps.

Edited by Xcollazo

Merge request reports