Remove spark dag factories
Refactor current jobs so we are not using DAG templates (except for AnomalyDetectionDAG, which will be transformed to a TaskGroup factory in another patch).
This patch also adds DynamicConfig, to make it easier to test the DAGs in the development environment.