Skip to content

Differential Privacy Airflow MR

Htriedman requested to merge htriedman/airflow-dags:airflow-dags-merge into main

This MR is seeking to add functionality for automating a daily-running differential privacy script to the analytics airflow instance. It should operate completely independently from other airflow jobs.

So far, I've been able to get it to run successfully using ./run_dev_instance.sh, but have had to make various adjustments to analytics/dags/differential_privacy/country_project_page_daily.py script in order to get it to work locally on stat1006. I've tried to make the necessary changes within that python script to ensure compatibility with the production airflow cluster, but I don't know if it will work out of the box here.

The only task that needs to be done that I am unable to do at the moment is putting the conda env that this script runs on into archiva or the airflow-dags hdfs file, which I don't currently have access to.

Let me know if you can think of other tasks that need to be completed here, and I'll do my best to address them!

Merge request reports