Skip to content

T275162 enable spark metrics collection

Created by: gmodena

This PR improves spark session creation in notebooks and scripts, enables control for metric collection, and aligns with resource utilisation patterns defined by Analytics.


  • For notebook we init SparkSession using the wmfdata library.

  • For scripts, the recommended way is to specify a file.

  • A file is also provided, that shows an example of how to plug in custom configuration files.

Merge request reports