(Marking as draft since it depends on repos/structured-data/image-suggestions!6 (merged).)
In this MR we upgrade the platform_eng Airflow instance to default to Spark3, and we upgrade the image-suggestions job (which is currently the only job on this instance.).
The airflow job runs all the Spark jobs successfully. I did not run the refinery jobs which move data fro Hive to Cassandra, as these point to the production cassandra instance. Once we merge this MR, I'll wait for and monitor the workflow for a full regular run.