Most production incidents we're called in to fix share a root cause: a change went to production without being tested in an environment that looked like production. This is the release workflow we use on Kubernetes projects, including government platforms where downtime isn't an option.
The principles
- Build once, deploy many. The exact image tested in staging is the one promoted to production. No rebuilds.
- Everything is versioned. Images have explicit tags (
2.8.1), never justlatest. - Git is the source of truth for what runs in each environment.
- Environments differ only in configuration, never in code.
The environments
- Staging: a separate cluster (or at least a separate, isolated namespace) that mirrors production: same Kubernetes version, same manifests, realistic but anonymised data, and the same integrations pointing at sandbox endpoints.
- Production: locked down, changed only through the pipeline.
Branches usually map to environments, for example a staging branch that deploys to staging and a main branch that deploys to production, with merges flowing in one direction.
Why a private registry like Harbor
Harbor is an open-source container registry that many organisations run themselves. It's popular in government and enterprise settings because it offers:
- images that stay inside your own infrastructure,
- projects with role-based access per team,
- built-in vulnerability scanning,
- replication between registries, for example from a dev registry to a production registry,
- retention policies to clean up old tags automatically.
The pipeline, step by step
- Merge to the staging branch triggers CI: tests, lint and a security scan.
- CI builds the images (multi-arch where needed) with the next version tag and pushes them to Harbor.
- CI (or a bot) updates the image tag in the deployment repository, a separate Git repo with Helm values or Kustomize overlays per environment.
- A GitOps controller such as Argo CD or Flux notices the change and syncs it to the staging cluster.
- QA tests staging, with automated end-to-end tests plus manual checks on the changed features.
- Promotion: once approved, the same tag is set in the production overlay through a reviewed pull request, and GitOps rolls it out.
Why GitOps
- Auditability: every change to production is a Git commit with an author and a review.
- Easy rollback: revert the commit and the cluster returns to the previous version.
- Drift detection: manual
kubectl editchanges are spotted and reverted. - Fewer credentials: CI doesn't need admin access to production clusters.
Keeping staging in sync with production
Hotfixes applied straight to production must flow back into staging, otherwise the next release can quietly undo them. Merge production back into the staging branch regularly, but never let production configuration overwrite staging configuration. Keep credentials and endpoints in per-environment secrets, not in shared code.
Safe rollouts
- Readiness probes and rolling updates so traffic only reaches healthy pods,
- database migrations designed to be backwards-compatible with the previous version,
- canary or blue-green releases for high-risk changes,
- a documented, rehearsed rollback.
Want a release process your team can trust? Our DevOps team sets up and documents staging, registries and GitOps pipelines.