Skip to content

Disable Spark Lineage on questionable Spark operations

TChin requested to merge disable-spark-lineage into main

Proactively disable lineage on some operators to prevent mistakes.

Questionable operations include:

  • WIP Dags
  • Touches Iceberg tables
  • Touches externals DBs (Cassandra, MariaDB)
  • Creates external dynamically-named temporary hive tables
  • Dynamically generated operators

Probably missed a few, but we'll find out eventually.

Bug: T306896

Edited by TChin

Merge request reports

Loading