( This is a redo of !186 (merged) )
In this MR we upgrade the platform_eng Airflow instance to default to Spark3, and we upgrade the image-suggestions job (which is currently the only job on this instance.). We also fix some correctness issues regarding the use of merge().
The airflow job runs all the Spark jobs successfully, including all the Cassandra loading steps.