Skip to content

Check bare metal and mw-on-k8s canaries

Ahmon Dancy requested to merge master-I46672fb9e38a6ce6816af7f1f212b412ab600629 into master

  • Use the updated /usr/local/bin/logstash_checker.py which supports
    "--host canaries" to indicate that all bare metal and mw-on-k8s
    canaries should be checked in aggregate.

  • Make sure canary checks will continue to work properly when the last
    bare metal canary goes away.

  • Raise SystemExit instead of RuntimeError when canary checks fail.
    This will avoid dumping a stack trace on the user.

  • Mention that canaries have not been rolled back for both types of
    canary check failures.

  • Use ">=" when comparing the number of failed canaries against
    max_failed_canaries. Previously, for the case where there is only
    one canary (such as when I was testing in train-dev), an error would
    never be reported because 1 is not greater than 1.

  • Replace hard-coded 10x with the configured threshold in a canary
    check failure message.

Note, swagger checks only happen for bare metal canaries at this time.
A ticket will be filed to deal with that for mw-on-k8s.

Bug: T357402

Merge request reports