Add a simple DAG developped by data platform SREs
This DAG will be deployed in the airflow-test-k8s
instance, and will
be the starting point of the DP SRE team developping more and more
DAGs to be able to perform end-to-end tests of our deployments, and
Kerberos, Hive, Superset, Datahub, S3, etc, integrations.
I've been able to test it locally:
~/wmf/airflow-dags/data_platform_sre main ?2 ❯ airflow dags test -S . addition ✘ INT 9s .env 09:49:45 AM
/Users/brouberol/wmf/airflow-dags/.env/lib/python3.12/site-packages/airflow/plugins_manager.py:30 DeprecationWarning: 'cgitb' is deprecated and slated for removal in Python 3.13
/Users/brouberol/wmf/airflow-dags/.env/lib/python3.12/site-packages/airflow/cli/commands/dag_command.py:48 UserWarning: Could not import graphviz. Rendering graph to the graphical format will not be possible.
[2024-09-03T09:49:46.888+0200] {dagbag.py:587} INFO - Filling up the DagBag from /Users/brouberol/wmf/airflow-dags/data_platform_sre
[2024-09-03T09:49:46.905+0200] {example_local_kubernetes_executor.py:40} WARNING - Could not import DAGs in example_local_kubernetes_executor.py
Traceback (most recent call last):
File "/Users/brouberol/wmf/airflow-dags/.env/lib/python3.12/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 38, in <module>
from kubernetes.client import models as k8s
ModuleNotFoundError: No module named 'kubernetes'
[2024-09-03T09:49:46.905+0200] {example_local_kubernetes_executor.py:41} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]
[2024-09-03T09:49:46.961+0200] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
[2024-09-03T09:49:46.962+0200] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes]
[2024-09-03T09:49:46.972+0200] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
[2024-09-03T09:49:46.977+0200] {dag.py:4393} INFO - dagrun id: addition
[2024-09-03T09:49:46.980+0200] {dag.py:4409} INFO - created dagrun <DagRun addition @ 2024-09-03 07:49:46.888908+00:00: manual__2024-09-03T07:49:46.888908+00:00, state:running, queued_at: None. externally triggered: False>
[2024-09-03T09:49:46.986+0200] {dag.py:4354} INFO - [DAG TEST] starting task_id=add_one_and_two map_index=-1
[2024-09-03T09:49:46.986+0200] {dag.py:4357} INFO - [DAG TEST] running task <TaskInstance: addition.add_one_and_two manual__2024-09-03T07:49:46.888908+00:00 [scheduled]>
[2024-09-03T09:49:46.992+0200] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
[2024-09-03 09:49:47,002] {taskinstance.py:3122} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='addition' AIRFLOW_CTX_TASK_ID='add_one_and_two' AIRFLOW_CTX_EXECUTION_DATE='2024-09-03T07:49:46.888908+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-09-03T07:49:46.888908+00:00'
[2024-09-03T09:49:47.002+0200] {taskinstance.py:3122} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='addition' AIRFLOW_CTX_TASK_ID='add_one_and_two' AIRFLOW_CTX_EXECUTION_DATE='2024-09-03T07:49:46.888908+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-09-03T07:49:46.888908+00:00'
Task instance is in running state
Previous state of the Task instance: queued
Current task name:add_one_and_two state:scheduled start_date:None
Dag name:addition and current dag run status:running
[2024-09-03T09:49:47.003+0200] {taskinstance.py:731} INFO - ::endgroup::
Result = 3
[2024-09-03 09:49:47,003] {python.py:240} INFO - Done. Returned value was: 3
[2024-09-03T09:49:47.003+0200] {python.py:240} INFO - Done. Returned value was: 3
[2024-09-03T09:49:47.006+0200] {taskinstance.py:340} INFO - ::group::Post task execution logs
[2024-09-03T09:49:47.006+0200] {taskinstance.py:352} INFO - Marking task as SUCCESS. dag_id=addition, task_id=add_one_and_two, run_id=manual__2024-09-03T07:49:46.888908+00:00, execution_date=20240903T074946, start_date=, end_date=20240903T074947
Task instance in success state
Previous state of the Task instance: running
Dag name:addition queued_at:None
Task hostname:wmf3487 operator:PythonOperator
[2024-09-03T09:49:47.008+0200] {dag.py:4368} INFO - [DAG TEST] end task task_id=add_one_and_two map_index=-1
[2024-09-03T09:49:47.009+0200] {dagrun.py:854} INFO - Marking run <DagRun addition @ 2024-09-03 07:49:46.888908+00:00: manual__2024-09-03T07:49:46.888908+00:00, state:running, queued_at: None. externally triggered: False> successful
Dag run in success state
Dag run start:2024-09-03 07:49:46.888908+00:00 end:2024-09-03 07:49:47.009638+00:00
[2024-09-03T09:49:47.009+0200] {dagrun.py:905} INFO - DagRun Finished: dag_id=addition, execution_date=2024-09-03 07:49:46.888908+00:00, run_id=manual__2024-09-03T07:49:46.888908+00:00, run_start_date=2024-09-03 07:49:46.888908+00:00, run_end_date=2024-09-03 07:49:47.009638+00:00, run_duration=0.12073, state=success, external_trigger=False, run_type=manual, data_interval_start=2024-09-03 07:49:46.888908+00:00, data_interval_end=2024-09-03 07:49:46.888908+00:00, dag_hash=None
~/wmf/airflow-dags/data_platform_sre main ?2 ❯ g nb T373837-airflow-data-platform-sre
Bug: T373837