CI/CD & GitOps for Kubernetes

CI/CD & GitOps for Kubernetes — Architect-Level Deep Dive

1. CI/CD Pipelines for Kubernetes

The fundamental architectural shift Kubernetes forces on CI/CD design is the split between CI (build/test/package) and CD (deploy/reconcile) — and how cleanly that split is drawn is one of the clearest signals of pipeline maturity in any review.

The standard CI stage sequence for a Kubernetes-targeted service:

  1. Build & unit test — compile, run unit tests, fail fast.
  2. Container image build — increasingly via Buildpacks or Kaniko/BuildKit (rootless, daemon-less builds) rather than mounting the Docker socket inside a CI runner pod — mounting /var/run/docker.sock into a CI pod is a well-known container-breakout/privilege-escalation vector (any pod with that mount effectively has root on the node), so rootless build tooling is the correct default to flag in any pipeline security review, not an optional hardening step.
  3. Image scanning (Trivy/Grype/ECR scanning, from the security module) — gate the pipeline on critical CVEs here, not after deployment.
  4. Push to registry — tagged immutably (commit SHA, never :latest — reinforcing the admission-controller policy from the security module; the registry-push step and the runtime-policy enforcement step should agree on this convention, not just the policy layer catching violations after the fact).
  5. Image signing (Cosign/Sigstore) — sign the image as part of CI, so the admission-controller-level signature verification mentioned in the security module actually has something trustworthy to check — this is the step that closes the loop between “we scan in CI” and “we verify at admission” into a genuine chain of custody.

Where CD diverges into two architecturally distinct philosophies, and this is the decision point worth being crisp about in any platform design conversation:

  • Push-based CD — the CI pipeline itself runs kubectl apply/helm upgrade directly against the cluster at the end of the pipeline, typically requiring the CI runner to hold cluster credentials.
  • Pull-based / GitOps CD — the CI pipeline’s job ends at “update a manifest/Helm-values file in a Git repo”; a separate in-cluster controller (ArgoCD/Flux) detects that Git change and reconciles the cluster to match — covered fully in the next section, but worth naming here as the architectural fork that determines everything downstream about credential management and audit trail.

The credential-exposure problem push-based CD creates, and why this is the most common real-world finding in pipeline security reviews: a CI runner with kubectl apply rights to production needs long-lived cluster-admin-adjacent credentials sitting in CI secret storage — and CI systems (Jenkins, GitHub Actions runners, GitLab runners) are a well-documented high-value attack target precisely because compromising the pipeline means compromising the deployment target directly. This is the single biggest architectural argument for GitOps over push-based CD: it removes the cluster-credential requirement from the CI system entirely.

Worth naming concrete tooling for completeness without dwelling: GitHub Actions, GitLab CI, Jenkins, Tekton (the K8s-native, Pipeline-as-CRD option — runs entirely as Kubernetes resources, no external CI server, increasingly relevant if the platform team wants the pipeline execution itself to be Kubernetes-native rather than calling out to an external CI product).

2. GitOps with ArgoCD / Flux

GitOps formalizes a specific operating principle: Git is the single source of truth for desired cluster state, and a reconciliation controller continuously ensures the live cluster matches it — not “Git is where we keep YAML,” but specifically “an automated control loop, not a human running kubectl apply, is what makes changes happen.”

The reconciliation loop, mechanically — this is the part worth being able to describe precisely:

  • The controller (ArgoCD’s application-controller, Flux’s source-controller+kustomize-controller) polls (or receives a webhook from) the Git repo on an interval (default a few minutes, often tightened via webhook-triggered immediate sync).
  • It computes the diff between desired state (the manifests in Git) and live cluster state (via the K8s API).
  • It applies whatever’s needed to converge live state to desired state — and critically, this reconciliation is continuous and self-healing: if someone manually kubectl edits a resource that’s GitOps-managed, the next reconciliation cycle reverts that manual change back to what Git declares — this is the property that genuinely eliminates configuration drift, and it’s also the thing that catches teams off guard the first time they forget and make a manual emergency kubectl patch that silently gets reverted minutes later.

ArgoCD vs Flux — the architectural distinction worth being precise about, since they’re not just two skins on the same idea:

  • ArgoCD ships a full UI and Application CRD model — a rich visual dependency graph of synced resources, manual “sync” buttons alongside auto-sync, and a strong multi-tenancy story via AppProjects (scoping which repos/clusters/namespaces a given team’s Applications are allowed to target — a real governance primitive in a multi-team platform). ArgoCD’s UI-centric, application-object-centric model tends to resonate with teams wanting visibility/control surfaces non-platform-engineers can use directly.
  • Flux is more CLI/API-first and deliberately composable — separate controllers for separate concerns (source-controller for fetching Git/Helm sources, kustomize-controller/helm-controller for applying them, notification-controller, image-automation-controller) wired together via CRDs, with no built-in UI (Weave GitOps or other third-party UIs layer on top if wanted). Flux’s image-automation-controller is worth naming specifically — it can detect a new image tag in a registry and automatically commit the updated tag back to the Git repo, closing the loop for fully automated continuous deployment of new builds without a human-authored commit triggering it — a capability ArgoCD doesn’t have natively (ArgoCD Image Updater is the comparable but separately-installed add-on).
  • Neither is “better” in the abstract — ArgoCD tends to win when visibility and self-service UI for app teams matters; Flux tends to win when minimal footprint, GitOps-native automation (like image-automation), and tight composability with existing Flux-ecosystem tooling matters more than a UI.

App-of-Apps / multi-tenancy pattern, worth naming as the standard way this scales past a handful of services: rather than one Application per microservice manually registered, a root “app-of-apps” Application (or Flux’s equivalent Kustomization referencing a directory of other Kustomizations) declares the set of Applications/Kustomizations themselves — onboarding a new microservice becomes “add a folder to the apps-of-apps repo,” not “someone manually runs argocd app create” — this is the pattern that actually makes GitOps scale to dozens/hundreds of services without becoming a manual-registration bottleneck.

Worth being explicit about the “no human runs kubectl apply against prod” cultural shift this requires — GitOps is as much an operating-model change as a tooling adoption, and the most common real-world implementation failure isn’t tooling misconfiguration, it’s teams adopting ArgoCD/Flux while quietly continuing to kubectl apply directly during incidents “just this once,” which both breaks the self-healing guarantee and erodes the audit-trail value GitOps is supposed to provide (every change should be a Git commit, with PR review and commit history as the change-audit log — that property only holds if the bypass discipline actually holds too).

3. Helm Charts

Helm is Kubernetes’ package manager — solving the problem that raw YAML manifests don’t parameterize, version, or template well on their own, and that distributing a “deployable unit” (not just a single Deployment, but the Deployment+Service+ConfigMap+HPA+PVC that together constitute one application) needs a packaging format.

Core architecture, worth being able to name precisely:

  • A Chart is a directory (or packaged .tgz) containing Chart.yaml (metadata/version), values.yaml (default configurable parameters), and a templates/ directory of Go-template-annotated Kubernetes YAML.
  • Helm renders templates client-side (or via Tiller in the long-deprecated Helm 2 — worth knowing Helm 3 removed the in-cluster Tiller component entirely for security reasons, since Tiller’s broad cluster-admin-equivalent permissions were a recurring security finding) — helm template/helm install substitutes values.yaml (or --set/-f custom-values.yaml overrides) into the templates and applies the resulting plain YAML.
  • Releases — each helm install/upgrade creates a tracked Release (state stored as a Secret in-cluster by default in Helm 3) recording exactly what was deployed, enabling helm rollback <release> <revision> to a specific prior rendered state — this is Helm’s own, separate rollback mechanism from kubectl rollout undo, operating at the whole-chart-release level rather than the single-Deployment level.
  • Hooks (pre-install, post-upgrade, etc., via annotations) — let a chart run one-off Jobs at specific lifecycle points (e.g., a DB migration Job that must complete before the new application version’s pods start) — a frequently necessary but easy-to-get-wrong capability, since hook ordering/failure-handling needs deliberate design (a failed migration hook should block the rest of the upgrade, and Helm’s hook-failure policies need to be configured to actually do that rather than silently continuing).

Where Helm’s templating model creates real friction, worth naming honestly rather than presenting Helm as purely additive:

  • Go-template-over-YAML is fundamentally string templating, not structural manipulation — this means whitespace/indentation errors in complex nested templates are a constant, genuinely annoying failure mode ({{- if }} whitespace-control syntax exists specifically to fight this), and debugging a broken render often means staring at helm template --debug output rather than getting a clean structural error.
  • Values schema sprawl — as a chart grows, values.yaml can become a deeply nested, under-documented contract that’s hard for chart consumers (as opposed to chart authors) to reason about without reading the templates themselves — values.schema.json (JSON Schema validation for values) is the mitigation worth naming, though adoption is inconsistent across the ecosystem.

Architect’s framing for when Helm is the right packaging tool: Helm earns its complexity when you’re distributing a reusable, parameterized application — to multiple environments, multiple tenants, or as a public/internal chart others install with their own values — which is precisely the scenario where templating logic (conditionals, loops over a list of ingress hosts, etc.) pays for itself. For a single internal app deployed to a fixed set of environments with only minor per-environment differences, Helm’s templating machinery is often more than needed — which leads directly into Kustomize.

4. Kustomize

Kustomize takes a deliberately different philosophy from Helm: instead of templating (placeholders filled in at render time), Kustomize works by patching plain, valid YAML — you write a complete, real base manifest, then declare structured overlays that patch specific fields for specific environments, with no templating language involved at all.

Core structure:

base/
  deployment.yaml
  service.yaml
  kustomization.yaml
overlays/
  dev/
    kustomization.yaml
    replica-patch.yaml
  prod/
    kustomization.yaml
    resource-patch.yaml
  • base/kustomization.yaml lists the base resources.
  • Each overlay’s kustomization.yaml references the base and applies patches — strategicMergePatch, JSON 6902 patches, or simpler patches: with inline target+spec — to override specific fields (replica count, image tag, resource limits) without duplicating the entire manifest per environment.
  • kustomize build overlays/prod (or kubectl apply -k overlays/prod, since Kustomize is natively built into kubectl, no separate binary required for basic usage) renders the final merged YAML.

The architectural case for Kustomize over Helm in specific scenarios, worth stating precisely rather than treating this as just personal preference:

  • No templating language to learn/debug — since it’s patches over real YAML, kustomize build output is always valid-by-construction YAML at every layer; you’re never debugging a Go-template whitespace issue.
  • Native kubectl integration means zero additional tooling dependency for basic use — a meaningful operational-simplicity win for platform teams trying to minimize the toolchain a new engineer needs to learn.
  • The “is this Deployment.yaml actually correct” question is directly answerable by reading the base file — there’s no values-substitution step obscuring what the rendered resource will actually contain, which some teams find genuinely easier to reason about and code-review than a templated chart.

Where it falls short of Helm, equally worth naming honestly:

  • No packaging/distribution/versioning model comparable to a Helm chart repository — there’s no equivalent of helm repo add + helm search + semantic chart versioning for sharing Kustomize bases across teams/organizations as a discrete, versioned artifact.
  • No conditional logic or loops — patches are structurally declarative, which is a strength for clarity but a real limitation if you genuinely need “generate N similar resources from a list” logic (a common Helm range use case) — Kustomize’s generators (ConfigMapGenerator, SecretGenerator) cover some of this gap but not the general case.

The pattern increasingly common in mature platforms, worth naming as the practical synthesis rather than forcing a binary choice: Helm for packaging genuinely reusable, parameterized applications (especially third-party charts — Prometheus, cert-manager, ingress-nginx — almost universally distributed as Helm charts, so you adopt Helm at minimum as a consumer even if you don’t author charts yourself) + Kustomize for environment-specific overlay/patching of your own internal manifests (including, notably, kustomize build-ing on top of a helm template-rendered base, via Kustomize’s helmCharts generator — a hybrid pattern that lets you consume an upstream Helm chart while still applying your own structural Kustomize overlays on top, rather than fighting with that chart’s values.yaml schema to express something it wasn’t designed to parameterize).

5. Image Registries & Security

The registry is the trust boundary between “code someone wrote” and “artifact your cluster will actually run” — and the architectural maturity of an organization’s pipeline shows clearly in how much enforcement actually happens at and around this boundary versus being purely advisory.

Registry options worth being fluent across, since real estates often run more than one:

  • Amazon ECR — the natural default for EKS-centric architectures, with IAM-based repository policies (fine-grained, per-repo push/pull permissions tied to the same IAM model as everything else AWS-side), native vulnerability scanning (basic on-push, or continuous “enhanced scanning” re-evaluating stored images against newly disclosed CVEs), and pull-through cache (proxying/caching public registries like Docker Hub through ECR, which also sidesteps Docker Hub’s rate limiting on anonymous pulls — a genuinely common production pain point worth naming, since unauthenticated Docker Hub pulls are rate-limited per-IP and a busy node pool pulling base images directly from Docker Hub can hit those limits during a large rollout).
  • Harbor — the CNCF-graduated, self-hosted option, worth naming for organizations needing on-prem/air-gapped registry control, with built-in vulnerability scanning (Trivy-integrated), image signing/notation support, replication between registries (useful for the multi-region/multi-cluster, possibly multi-cloud, image-distribution problem), and project-based RBAC.
  • GitHub Container Registry (GHCR), GitLab Container Registry — relevant when the registry is naturally co-located with the CI system already in use, reducing one integration hop.

Image signing and provenance — the piece that elevates registry security from “we scan images” to “we can cryptographically prove where this image came from,” and worth treating as the modern baseline rather than an advanced/optional step:

  • Cosign (Sigstore project) — signs images with either a stored keypair or, increasingly, keyless signing via OIDC identity (the signature is tied to “this was signed by the CI pipeline running as this specific GitHub Actions workflow identity,” verifiable via the public Rekor transparency log, rather than requiring you to manage and protect a long-lived private signing key yourself) — this keyless model has become the practical default recommendation specifically because long-lived signing keys are themselves a high-value credential to protect and rotate.
  • SBOM generation (Syft, or Docker’s buildx native SBOM support) — generating a Software Bill of Materials at build time, attached to the image, gives downstream consumers (and your own admission-control/policy layer) the ability to ask “does this running image contain library X version Y” without re-scanning from scratch — increasingly a compliance requirement in regulated industries, worth naming as the direction supply-chain security is heading even if not yet universally mandated.
  • In-toto/SLSA framework — worth naming at the conceptual level for a senior conversation: SLSA defines graduated levels of build-provenance guarantee (was this artifact built by a trusted, tamper-resistant CI system, from this exact source commit, with no unreviewed manual injection points) — the framework that signing+SBOM+admission-verification are collectively working toward satisfying, and a term that’s increasingly likely to surface in enterprise security/compliance conversations around container supply chains in 2026.

Closing the loop back to the admission-controller discussion from the security module: none of this signing/scanning infrastructure actually constrains anything unless the admission layer verifies it at deploy time — Kyverno/Gatekeeper policies checking Cosign signatures, or rejecting images without an attached SBOM, or blocking images with unresolved critical CVEs from the registry’s scan results — is the step that turns “we have a signing pipeline” into “unsigned/unscanned images cannot run in this cluster, structurally, not by convention.” This full chain — CI builds and scans → signs → pushes to a controlled registry → GitOps commits the new tag → admission controller verifies signature/scan status before allowing the pod → ArgoCD/Flux reports sync health — is the complete, modern, architecturally sound answer to “how does code go from a developer’s commit to safely running in production,” and is worth being able to draw end-to-end on a whiteboard as the synthesis of this entire module.