Deployments
Cloud resources and the components inside a cluster change at different rates and depend on different things, so they don't have to be deployed the same way. Four ways are common, each with upsides and downsides. Cloud resources, the clusters included, fit infrastructure as code from CI. Cluster components start out from CI too, as in the Simple blueprint. They move to a GitOps tool in each cluster when drift has to be corrected between pipeline runs, or when CI shouldn't hold credentials for every cluster. A tool in the cluster is also the only way that keeps correcting a cluster's components while CI or the git host is down. Cloud resources from inside a cluster fit the databases and buckets of one app. A component can move from one way to another without a reinstall.
Cloud resources are what the cloud provider runs: networks, clusters, node pools, databases, buckets, DNS zones and access policies. Cluster components are the platform components that run inside a cluster for every app on it, such as the ingress controller, cert-manager, the monitoring agents and the GitOps tool itself. A deployment tool is whatever installs either of them, such as OpenTofu, Helm run from CI, Flux, Argo CD or Crossplane. Flux and Argo CD are also GitOps tools: they run in a cluster and pull from git. Apps are deployed by the teams that build them, which the DevOps Handbook covers.
Infrastructure as code from CI
A pipeline runs OpenTofu or Terraform against a state in a remote backend, such as a bucket. It plans on every merge request and applies after the merge.
Upsides
- Reviewers read the plan of every change before it's applied, and applying a plan saved with
tofu plan -outchanges exactly what that plan showed. - A plan names every resource it would replace or destroy, so a change that recreates a database or a cluster shows up in review.
- OpenTofu orders the changes by its dependency graph across providers, so a cluster, its DNS records and its backup bucket land in one apply.
- The backend locks the state during an apply, so two pipelines can't change the same resources at once.
- Nothing runs between changes: the cost is pipeline minutes and a bucket for the state.
Downsides
- Drift stays until the next apply. A change made in the cloud console shows up in the next plan, and nothing reverts it before then.
- The state holds every attribute of every resource, generated passwords and private keys included, so read access to the state is read access to those secrets. OpenTofu can encrypt the state, and ephemeral values and write-only arguments keep a secret out of it.
- CI holds credentials that can change everything the pipeline manages. GitLab CI and GitHub Actions issue ID tokens that the cloud exchanges for short-lived credentials. That removes the stored secret, but a job that gets the credentials can still change all of it.
- When CI or the git host is down, nothing gets changed or fixed. What exists keeps running, and drift stays.
- Every plan reads every resource in its state, so a large state slows every pipeline down, and splitting it into several states adds dependencies between them.
Cluster components from CI
A pipeline installs and upgrades each component with Helm, or with the Helm provider of OpenTofu or Terraform in the same apply that creates the cluster. The Simple blueprint installs its components this way.
Upsides
- No controller runs in the cluster to install components, so there's nothing extra to run, upgrade or monitor.
- With the Helm provider, a component lands in the same apply as the cloud resources it needs, and gets their IDs as values.
helm diff upgradefrom the helm-diff plugin, or the provider's plan, shows a change before it's applied.- Helm keeps each release's history in the cluster, so
helm rollbackreturns to an earlier revision.
Downsides
- Drift stays until the next run.
helm upgradesets the fields the chart renders back to the chart's values, but nothing does that between runs. - The Helm provider reads only the Helm release record, so its plan misses changes to live objects, such as a
kubectl edit. Its experimentalmanifestoption makes it read the live objects as well. - Helm stores each release's values in Secrets in the cluster, and the Helm provider stores them in the state too. The provider's write-only
set_wokeeps a value out of the state. - CI holds credentials for every cluster it deploys to, usually cluster-admin, since components install CRDs and cluster roles.
- CI has to reach each cluster's API, so a cluster with a private API endpoint needs a runner inside its network.
- Helm 4 installs new releases with server-side apply, so an upgrade fails on a field that another tool changed until it runs with
--force-conflicts. - When CI or the git host is down, components keep running but get no upgrades or fixes.
GitOps tool in each cluster
Flux or Argo CD runs in every cluster, pulls a git repository on an interval, and applies the components it describes. The Advanced blueprint offers this for platform components. The DevOps Handbook covers application deployment with the same tools.
Upsides
- The tool corrects drift on every interval, without a pipeline run. Flux Kustomizations do so by default, Flux HelmReleases once drift detection is enabled, and Argo CD Applications once self-heal is on.
- The credentials that install components stay in the cluster. CI needs no access to the cluster API, and the tool pulls from git with a read-only key.
- When the git host is down, Flux keeps applying the last revision it fetched, so the cluster still corrects its own drift.
- A cluster with a private API endpoint needs no route from CI.
- A cluster rebuilt from scratch gets every component back from git once the tool runs in it.
Downsides
- Every cluster runs one more component to upgrade and monitor. Flux runs four controllers by default, each requesting 64 MiB of memory with a limit of 1 GiB. Argo CD 3.6 runs four workloads in its core install. The full install runs seven, adding an API server with the UI, Dex for login and a notifications controller.
- Reviewers see a values or manifest diff in the merge request, but not what will change in the cluster.
- Components often need IDs of cloud resources, such as the role a component runs as. Those come from infrastructure as code, through git or a ConfigMap that the apply writes.
- Self-heal reverts a fix made by hand during an incident, so an emergency change goes through git, or the tool is paused first.
- Flux keeps the last fetched revision in the source-controller pod's storage. When that pod moves to another node while the git host is down, Flux stops applying until git is back.
Bootstrap
The tool has to run before it can install anything, so something else installs it first. Either way, the tool then manages itself from git.
The infrastructure as code that creates a cluster can install the tool in the same apply, with the Helm provider or Flux's flux_bootstrap_git resource. A cluster then comes up with its GitOps tool and no command is run by hand, which fits clusters that a pipeline creates and recreates. The pipeline then holds the tool's git key. When OpenTofu generates that key with tls_private_key, as Flux's own examples do, the state holds it unencrypted.
A bootstrap command does the same once per cluster. flux bootstrap gitlab installs Flux, commits its manifests to the repository and stores a read-only deploy token in the cluster. Whoever runs it needs cluster admin and a git token, and the step happens outside any pipeline, which fits a few clusters that live for years. Argo CD has no bootstrap command: install its manifests or Helm chart once, then add an Application that points at Argo CD's own manifests.
Cloud resources from inside a cluster
A controller in a cluster creates and updates cloud resources from Kubernetes objects. Crossplane maps each cloud resource to a Kubernetes object, and the Tofu Controller for Flux runs OpenTofu plans and applies in pods in the cluster. The Advanced blueprint uses this as a Kubernetes-native application layer for application teams that create their own databases and buckets.
Upsides
- A database or bucket that one app needs sits in that app's manifests. It's created with the app and, under Crossplane's default management policy, deleted with it.
- The controller corrects drift on every poll. Crossplane reverts a change made in the cloud console to what the object says, and the Tofu Controller applies a plan that fixes the drift.
- Cloud credentials stay in the cluster, so CI holds no cloud credentials.
- An app and its cloud resources are reviewed in one merge request and deployed by the same deployment tool.
Downsides
- The controller keeps a second state store next to the infrastructure as code: Crossplane in the cluster's Kubernetes API, the Tofu Controller in a Secret per stack unless it's given another backend.
- When that cluster is lost, its cloud resources keep running without anything managing them, and each one has to be imported again by its ID.
- The cloud credentials are a Kubernetes Secret, readable by anyone who can read Secrets in that namespace.
- Crossplane providers install many CRDs: a large provider installs over 100, and
provider-aws-ec2about 200. Each takes about 3 MiB of API server memory. Activation policies, an alpha feature, limit them to the kinds in use. - Provider coverage differs per cloud. The AWS provider 2.9.0 defines 1,062 kinds of resource, while the DigitalOcean provider 1.0.2 defines 68, against 76 resources in DigitalOcean's provider for OpenTofu and Terraform.
- The cluster that runs the controller has to exist first, so that cluster, its network and the controller's own credentials still come from infrastructure as code.
- A Crossplane change gets no plan to review. The Tofu Controller can hold a plan until someone approves it with
approvePlan, but that approval happens in the cluster, not in the merge request.
Choosing
| Way | Drift correction | State | Credentials | When the central side is down | Running cost |
|---|---|---|---|---|---|
| Infrastructure as code from CI | At the next apply | A remote backend, secrets included | In CI, for everything it manages | Nothing changes, resources keep running | Pipeline minutes and a bucket |
| Cluster components from CI | At the next upgrade | Release Secrets, plus the state with the Helm provider | In CI, usually cluster-admin on each cluster | Nothing changes, components keep running | Pipeline minutes |
| GitOps tool in each cluster | Every interval, once enabled for Flux HelmReleases and Argo CD | Git, and the cluster's own objects | In the cluster, with a read-only git key | Flux keeps applying the last revision it fetched | One more tool per cluster |
| Cloud resources from inside a cluster | Every poll | The cluster's Kubernetes API, or a Secret | In the cluster, as a Secret | Resources keep running, nothing corrects them | A controller, and API server memory per CRD |
Lifecycle
Cloud resources change rarely, other resources depend on them, and replacing one by mistake can lose data, so a reviewed plan before every apply fits them. Cluster components change with each upstream release and depend on the cluster and on a few cloud resources, such as a DNS zone or a role. From CI they land in the same apply as those resources, which fits a few clusters whose components change only in planned upgrades. A GitOps tool upgrades them from git without a pipeline run, which fits many clusters. A database or bucket that belongs to one app changes with that app. A controller in the cluster fits it, since the controller creates and deletes it with the app's manifests.
Drift
The ways that run from CI leave a change made by hand in place until the next run. Some clusters get changes made by hand that have to be reverted, such as a cluster that several teams have admin access to. There, a GitOps tool or Crossplane reverts them within one interval. Flux HelmReleases need driftDetection.mode: enabled for that, and Argo CD Applications selfHeal: true.
Credentials
Every way that runs from CI puts credentials in CI, and the more clusters and accounts a pipeline reaches, the more a leaked pipeline token can change. A GitOps tool in each cluster keeps the credentials that install components inside that cluster, and CI only pushes to git. A controller that manages cloud resources moves the cloud credentials into the cluster too, so reading the cluster's Secrets gives access to the cloud account. The infrastructure as code state and Helm's release Secrets both hold the values they were given, secrets included.
Outages
A cluster that runs its own GitOps tool keeps its components corrected while CI, the git host or any other central system is down. Flux keeps applying the last revision it fetched, and nothing outside the cluster has to reach the cluster's API. One Argo CD instance can deploy to every cluster, with each cluster's credentials stored as a Secret. While that instance is down, no cluster gets corrected, and whoever gets into it can change every cluster. The ways that run from CI behave like that central instance: what's deployed keeps running, but nothing gets fixed or upgraded until CI and git are back.
Running cost
The ways that run from CI cost pipeline minutes and a state bucket, and nothing between changes. A GitOps tool costs a few controllers in every cluster, and upgrades of the tool itself. Crossplane costs its controllers plus about 3 MiB of API server memory for every CRD its providers install, so provider-aws-ec2 alone takes about 600 MiB. Running two deployment tools for cloud resources also means two places to look when a resource changes.
Moving a component
To move a component without a reinstall, the old deployment tool lets go of it without deleting it, and the new one adopts it under the same identity. A Helm release is identified by its name and namespace, a Kubernetes object by its kind, namespace and name, and a cloud resource by its ID. Let go first: while two deployment tools manage one component, each reverts the other's changes, and removing a component from the old tool's configuration deletes it there.
| From | To | Let go | Adopt |
|---|---|---|---|
| Helm from CI | Helm provider | Remove the Helm job from the pipeline | An import block for the helm_release, with the ID <namespace>/<release> |
| Helm provider | Helm from CI | A removed block with lifecycle { destroy = false } | helm upgrade with the same release name and namespace |
| Helm from CI or the provider | Flux | Remove the job, or add the removed block | A HelmRelease whose releaseName, targetNamespace and storageNamespace match the release, which Flux upgrades instead of installing |
| Helm from CI or the provider | Argo CD | As for Flux, then delete the release's Secrets (label owner=helm); helm uninstall would delete the objects too | An Application for the same chart and values, which Argo CD applies over the existing objects |
| Flux | Helm from CI or Argo CD | Set .spec.suspend: true on the HelmRelease, then delete it; deleting it unsuspended uninstalls the release | helm upgrade with the same release name and namespace, or an Application as above |
| Argo CD | Helm from CI or Flux | argocd app delete --cascade=false, which leaves the objects in place | helm upgrade --install --take-ownership, or a HelmRelease, which takes ownership of existing objects by default |
| Infrastructure as code | Crossplane | A removed block with lifecycle { destroy = false } | A managed resource with the crossplane.io/external-name annotation set to the resource's ID, first with managementPolicies: ["Observe"] |