magnum: Use the Magnum Cluster API (CAPI)

Dduvallrequested to merge
review/use-magnum-capi into main

The Magnum Heat driver is now considered deprecated. Migrate to using CAPI which provides a number of benefits:

  1. A newer Kubernetes version
  2. Ability to add the haproxy hostname to the k8s certificate SANs

The latter means we no longer have to rely on tls-server-name in the client kubeconfig which in turns works around a limitation in Zuul (whereby zuul-launcher Launcher loses tls-server-name when serializing and storing kubeconfig for later use by zuul-executor and causes k8s jobs to fail).

Note however, that CAPI is still very new and there are a few edge cases to be aware of which are commented inline.

Bug: T424061

Merge request reports