Skip to content

Upgrade image-suggestions to Spark3.

Xcollazo requested to merge T323108-upgrade-image-suggestions-to-spark3 into main

(Marking as draft since it depends on repos/structured-data/image-suggestions!6 (merged).)

In this MR we upgrade the platform_eng Airflow instance to default to Spark3, and we upgrade the image-suggestions job (which is currently the only job on this instance.).

The airflow job runs all the Spark jobs successfully. I did not run the refinery jobs which move data fro Hive to Cassandra, as these point to the production cassandra instance. Once we merge this MR, I'll wait for and monitor the workflow for a full regular run.

Edited by Xcollazo

Merge request reports