Skip to content
Snippets Groups Projects

T275162 enable spark metrics collection

Merged Gmodena requested to merge github/fork/gmodena/T275162-enable-spark-metrics-collection into main

Created by: gmodena

This PR improves spark session creation in notebooks and scripts, enables control for metric collection, and aligns with resource utilisation patterns defined by Analytics.

Changelog:

  • For notebook we init SparkSession using the wmfdata library.

  • For scripts, the recommended way is to specify a spark.properties file.

  • A metrics.properties.template file is also provided, that shows an example of how to plug in custom configuration files.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply