Skip to content

Improve handling of ores threshold fetching

Ebernhardson requested to merge work/ebernhardson/popularity-score-skein into main

  • While the script provided to skein worked in our old airflow instance, the new one doesn't seem to be interpreting newlines as the end of a command to bash. Add semi-colons to explicitly delimit shell commands

  • The path also seems to have changed, in our airflow 1 instance skein striped the suffix off the venv when decompressing it to the executor, testing shows that the directory created now retains the suffix.

  • Adds a -f flag when copying the result of execution into hdfs. This causes it to overwrite the existing results, much like all of our spark based tasks.

Merge request reports