Skip to content

Queries for incident reporting system metrics tables

Conniecc1 requested to merge conniecc/irs_query_T386672 into main

Bug: T386672

The updates include:

  1. File to create the destination table: create_incident_reporting_system_incident_type_daily
  2. File to create the destination table: create_incident_reporting_system_overall_metrics_daily
  3. File to pull and ingest data:  generate_incident_reporting_system_incident_type_daily
  4. File to pull and ingest data: generate_incident_reporting_system_overall_metrics_daily

The files have been tested locally using the command below

spark3-sql -f create_incident_reporting_system_incident_type_daily.hql -d table_name=cchen_airflow_test_dag.incident_reporting_system_incident_type_daily -d base_directory=/user/hive/warehouse/cchen_airflow_test_dag

spark3-sql -f create_incident_reporting_system_overall_metrics_daily.hql -d table_name=cchen_airflow_test_dag.incident_reporting_system_overall_metrics_daily -d base_directory=/user/hive/warehouse/cchen_airflow_test_dag

spark3-sql --master yarn --executor-memory 16G --executor-cores 8 --driver-memory 4G --conf spark.dynamicAllocation.maxExecutors=64 \-f generate_incident_reporting_system_incident_type_daily.hql -d source_table=event.mediawiki_product_metrics_incident_reporting_system_interaction -d destination_table=cchen_airflow_test_dag.incident_reporting_system_incident_type_daily -d target_year=2025 -d target_month=2 -d target_day=22 -d coalesce_partitions=1

spark3-sql --master yarn --executor-memory 16G --executor-cores 8 --driver-memory 4G --conf spark.dynamicAllocation.maxExecutors=64 \-f generate_incident_reporting_system_overall_metrics_daily.hql -d source_table=event.mediawiki_product_metrics_incident_reporting_system_interaction -d destination_table=cchen_airflow_test_dag.incident_reporting_system_overall_metrics_daily -d target_year=2025 -d target_month=2 -d target_day=22 -d coalesce_partitions=1

Edited by Conniecc1

Merge request reports