Skip to content

fix output formating of unique devices

Mforns requested to merge fix_unique_devices into main

After deploying unique devices DAGs, we discovered a couple issues with the formatting of the ouput: mostly file permissions and success files:

  • Modify the hdfs_temp_directory from '/tmp/analytics' to '/wmf/tmp/analytics'. Because we were using the former, the unique_devices archive was getting the wrong permissions. The latter is the one that has proper permissions and the one that should be used.
  • Add _SUCCESS files to the Hive table partition directories. They are still needed by Oozie for the unique_devices Druid loading jobs.
  • Change the ArchiveOperator parent_umask and perms, to match the existing output perms.
  • Modifies the tests to account for the extra task.
Edited by Mforns

Merge request reports