Skip to content

Queries for trust_safety_admin_request_monthly table

Jenniferwang requested to merge jiawang/data-pipelines:tsp_T380318_query into main

Bug: T380318

The updates include:

  1. One file to create the destination table trust_safety_admin_request_monthly
  2. One file to pull and ingest data into  trust_safety_admin_request_monthly

The files have been tested locally using the command below

spark3-sql  -f create_admin_request_monthly_table.hql  -d table_name=jiawang_airflow_test.trust_safety_admin_request_monthly -d base_directory=/user/hive/warehouse/jiawang_airflow_test 

spark3-sql --master yarn --executor-memory 16G --executor-cores 8 --driver-memory 4G --conf spark.dynamicAllocation.maxExecutors=64 \-f generate_admin_request_monthly_table.hql   -d source_logging_table=wmf_raw.mediawiki_private_cu_log   -d destination_table=jiawang_airflow_test.trust_safety_admin_request_monthly -d canonical_table=canonical_data.wikis   -d snapshot=2024-10  \  -d coalesce_partitions=1

Merge request reports