Backport thread pool PRs

This MR backports two PRs from varnish, the first one is https://github.com/varnishcache/varnish-cache/pull/3827 (wrk: Give bgfetch tasks the lowest priority) and from the original PR:

The goal is to prevent grace mode from adding load to a saturated Varnish instance.

This change splits tasks in 3 categories:

dispensable tasks essential tasks, subject to limits vital tasks A background fetch entering the queue will enter the queue and block the client tasks that triggerred it until it starts its execution to the point where it no longer needs to hold onto its req.

On a saturated system this can result in significant client latency despite a grace hit.

A background fetch is at the "lowest" queue priority in the sense that a BG task is actually not queued, simply abandoned. This turns the Pool_Task() failure dead branch from VBF_Fetch() into a reachable one. This is the new TASK_QUEUE_LOWEST_PRIORITY macro.

The macro TASK_QUEUE_CLIENT was also renamed to TASK_QUEUE_LIMITED to better reflect its meaning, and keep the door open to non-client limited tasks.

A new TASK_QUEUE_RESERVE macro is added to better reflect operations that cater to the thread reserve. It also allows to allocate the right number of reserve heads in struct pool, instead of having one for each priority.

Second PR is https://github.com/varnishcache/varnish-cache/pull/3947 (pool: Safely update pool statistics)

Bug: T391334

Edited by Vgutierrez

Merge request reports

Loading