Draft: wmcs: Support WMCS Magnum as a k8s provider

Dduvallrequested to merge
spike/wmcs-magnum into main

Note this change was developed during an experiment spike and is meant to illustrate the viability of running gitlab-cloud-runner resources on WMCS Magnum. It should be broken up into smaller changes and further refactored.

  1. buildkit: Allow configuration of storage class and whether autoscaling should be enabled (we won't need to autoscale on WMCS). Also remove the configuration of a s3 based cache which has never been utilized.
  2. gitlab: Remove node selector/tolerations and parameterize s3 cache server configuration.
  3. digitalocean: Define outputs for configuration that can vary between k8s providers such k8s auth info, s3 server, and ingress cluster IP. Remove unused variables.
  4. wmcs: Introduce new module for WMCS Magnum based k8s provisioning. Outputs are all consistent with the digitalocean module so either can be used in the main cluster config.
  5. externaldns: Refactor to use the newer version.
  6. externaldns-designate: New module for DNS management via OpenStack Designate. This gives us host names for externally facing services (e.g. registry.gitlab-runners-staging.wmcloud.org).
  7. cluster: Select k8s provider module based on new cluster_provider variable and set a cluster local for provider module outputs. Refactor old references to digitalocean properties and use local.cluster instead. Move provider specific resources into wmcs.tf and digitalocean.tf.

Bug: T403125

Merge request reports