Skip to content

Bump image_suggestions executors from 9GB to 16GB.

Xcollazo requested to merge T323614-bump-executor-mem into main

In this MR we bump executor memory from 9 to 16GB for the Spark jobs on image_suggestions.

The job now requires more memory for the executor nodes to be reliable. This is due to the coalesce introduced in https://phabricator.wikimedia.org/T323614 creating bigger parquet files, which require more memory for buffering before writing.

We also make sure this setting is now a varprop for easier tinkering in future.

Edited by Xcollazo

Merge request reports