In order to be able to access the hdfs cluster the spark pods need to have access to the core/hdfs-site.xml files Those files are available in common hadoop- configmap in the spark NS Mounting them inside the pods and exposing them to the env var HADOOP_CONF_DIR
Bug: T332909
Need https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/902402 to be merged and applied first