digitalocean: Separate management of cluster and in-cluster resources
According to the kubernetes provider docs:
When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources.
Follow the recommended model and perform separate apply operations for
the cluster itself and the resources managed within the cluster.
Refactor all variables common to digitalocean and other modules as
outputs of the former and use terraform_remote_state in the main
module to fetch data about the cluster resources. This establishes a
clear separation of concerns between the main module (responsible for
in-cluster resource management) and the module responsible for managing
the cluster resources themselves. This also establishes an interface of
sorts for alternative cluster modules to implement.
Refactor buildkitd and gitlab modules to remove hardcoded DO
specific configuration. Remove use of S3 storage from buildkitd as it
is unused.
Refactor .gitlab-ci.yml to perform separate apply and destroy
operations for the cluster and in-cluster resources. Define mixins to
centralize the matrix of defined variables and apply arguments for
staging/production and cluster/resources dimensions.
Move production and staging variables specific to digitalocean to a
digitalocean.tfvars file within the respective directories.
Note deployment of this change will require splitting both the
production and staging combined state files into separate "cluster" and
"resources" state files. This will be performed ad-hoc with tofu state pull and some jq fiddling.
Bug: T416260