Skip to content

Revert Canary events switch from Spark operator to Skein

Aqu requested to merge T381073_fix_canary_events into main

Following !996 (merged)

We are reverting to SimpleSkeinOperator instead of SparkSubmitOperator. Because:

  • ProduceCanaryEvents expects event_service_config_uri to be an absolute path uri with protocol
  • In the context of a Skein container we can rebuild the absolute path with $(pwd)
  • SparkSubmitOperator puts the spark arguments into simple quotes
  • Simple quotes does not allow the substitution of the $(pwd) variable
  • Whereas SimpleSkewinOperator does not quote the params and allows the substitution of the $(pwd) variable

Bug: T381073

Merge request reports

Loading