Skip to content

feat: allow to set number of threads in catboost models

Ilias Sarantopoulos requested to merge isaranto/readability-liftwing:add-threads-arg into main

Allow to set the number of threads for catboost model when calling the classify function.

Why: Catboost doesn't support cgroupsv2 yet which results in the framework to miscalculate the number of cpus available. It uses the number of cpus of the bare metal node which ends up in throttling and increased latencies.

What: Allow to manually set the number of threads when calling the classify function from the inference service where we can calculate the number of cpus properly.

Bug: T353461

Edited by Ilias Sarantopoulos

Merge request reports