kubernetes: Add alert on web service 5xx rate
Add an alert when a too large percentage of $TOOL.toolforge.org web
requests is returning a 5xx response, as measured by the HAProxy layer
in front of the Kubernetes ingress.
Setting a good threshold here is difficult because some number of 5xx
errors is "expected" (from individual broken tools, from which we do not
want to get paged about), so I picked 25% as something that's clearly
higher than the couple of days of "normal" rates we have in Prometheus
history but still low enough to alert us on actual breakage.
This patch adds the alert as a severity: critical, it can be raised to a
page if there are no false positives or other issues found.
Bug: T399870