Kubernetes Autoscaling & Workload Health

Kubernetes Autoscaling & Workload Health — Architect-Level Deep Dive

1. Horizontal Pod Autoscaler (HPA)

HPA scales the number of pod replicas in a Deployment/StatefulSet/ReplicaSet in/out, based on observed metrics against a target — it’s the “more workers, not bigger workers” lever.

Mechanics:

  • The HPA controller (a control loop inside kube-controller-manager, polling every --horizontal-pod-autoscaler-sync-period, default 15s) queries the Metrics API.
  • For CPU/memory, this comes from the metrics-server add-on (implements metrics.k8s.io, aggregating kubelet’s cAdvisor stats) — metrics-server is a hard prerequisite; without it, basic CPU-based HPA simply won’t function.
  • For custom/external metrics (queue depth, request latency, business KPIs), HPA queries the custom.metrics.k8s.io or external.metrics.k8s.io API, typically backed by Prometheus Adapter translating PromQL queries into the metrics API shape.

The scaling formula architects should be able to derive on demand:

desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue))

e.g., 4 replicas, current avg CPU 80%, target 50% → ceil(4 × (80/50)) = 7 replicas.

v2 API (autoscaling/v2) — what a senior conversation should reference, not the legacy v1 CPU-only spec:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
    - type: Pods
      pods:
        metric:
          name: requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

Multiple metrics are evaluated independently, and HPA picks the highest resulting replica count among them — the conservative-toward-scale-up logic.

The behavior block is the part most teams skip and then get bitten by: without explicit stabilizationWindowSeconds on scale-down, you get flapping — replicas oscillate rapidly as load hovers near the threshold. Default scale-down stabilization is 300s (5 min), meaning HPA looks at the max recommended replica count over the trailing 5-minute window before actually scaling down — this dampens noise but is exactly the kind of default an architect should be able to explain when someone asks “why didn’t it scale down immediately after load dropped?”

A known limitation worth flagging: HPA reacts to current metrics — it’s fundamentally reactive, not predictive. For workloads with sharp, predictable traffic spikes (e.g., a 9am login surge), pure HPA means a lag window where pods are still spinning up while load has already arrived. This is the practical justification for either pairing HPA with scheduled scaling (via KEDA’s cron scaler, or simple CronJob-driven minReplicas patches) or for event-driven autoscaling via KEDA (which extends the custom metrics model to scale off queue depth, Kafka lag, SQS message count, etc., including scale-to-zero, which vanilla HPA cannot do — minReplicas: 0 isn’t supported in stock HPA).

2. Vertical Pod Autoscaler (VPA)

VPA takes the orthogonal axis — instead of more pods, it adjusts CPU/memory requests and limits on existing pods, based on observed historical usage. It’s the “right-sizing” lever, distinct from and (by default) incompatible with running simultaneously against the same metric as HPA — running CPU-based HPA and CPU-based VPA on the same workload causes the two controllers to fight each other (VPA changes the request, HPA’s utilization-% calculation shifts as a result, creating feedback instability). The supported combination is HPA on custom/external metrics + VPA on CPU/memory, not both on CPU.

Three components, each worth being able to name:

  • Recommender — watches historical resource usage (via metrics-server/Prometheus history) and computes recommended requests/limits.
  • Updater — checks running pods against recommendations; if a pod’s current request is far enough off the recommendation, it evicts the pod (this is the part that surprises people — VPA in Auto mode causes pod restarts to apply new resource values, since you can’t hot-patch a running container’s resource allocation in most modes).
  • Admission Controller — a mutating webhook that injects the recommended values into new pod specs at creation time (this is how the recommendation actually gets applied without the updater needing to evict-and-recreate every single time).

updateMode — the field that determines blast radius:

  • Off — recommendation-only, visible via kubectl describe vpa, nothing auto-applied. The safe default for first adopting VPA in production — observe before you let it act.
  • Initial — applies recommendation only at pod creation, never touches running pods.
  • Auto/Recreate — actively evicts and recreates pods to apply new sizing — disruptive, needs to be paired with PodDisruptionBudgets to avoid availability gaps during the resize churn.

In-place resize (1.27+ feature, increasingly relevant in 2026 clusters): Kubernetes added the ability to resize CPU/memory without pod restart via the resize subresource — this is gradually making VPA’s Auto mode less disruptive as it adopts in-place resize instead of evict-and-recreate, but as of most current stable VPA releases this integration is still maturing — worth a “check current VPA release notes” caveat if this comes up in an interview, since it’s an area actively evolving.

Architect’s practical framing: VPA is most valuable for batch/single-replica workloads where HPA’s “more copies” lever doesn’t apply at all (e.g., a single stateful primary, a batch job) — there, right-sizing via VPA recommendations (even in Off mode, just reading the recommendation manually) is often the more honest cost/performance optimization than guessing requests by hand.

3. Cluster Autoscaler

Where HPA/VPA operate at the pod layer, Cluster Autoscaler (CA) operates at the node layer — adding/removing actual EC2 instances (or equivalent) based on scheduling pressure, not raw utilization metrics.

Core trigger logic — this is the detail interviewers probe for:

  • CA scales up when it detects Pending pods that failed to schedule due to insufficient resources — it doesn’t watch CPU/memory utilization directly; it simulates whether adding a node from a given node group would let unschedulable pods fit, then provisions accordingly.
  • CA scales down when a node’s utilization stays below a threshold (default 50% combined CPU+memory request, --scale-down-utilization-threshold) for a sustained period (--scale-down-unneeded-time, default 10 min) and all pods on that node could be rescheduled elsewhere without violating constraints.

Scale-down is blocked by several conditions every architect should know cold, since these are the classic “why won’t my node count go down” debugging checklist:

  • Pods with no controller (bare pods not managed by a Deployment/Job/etc.) on the node.
  • Pods with local storage (emptyDir) unless explicitly annotated as safe to evict (cluster-autoscaler.kubernetes.io/safe-to-evict: "true").
  • PodDisruptionBudget violations — if evicting that node’s pods would breach a PDB’s minAvailable, CA won’t touch it.
  • kube-system pods without a PDB, by default, block scale-down of their node — this is precisely why most managed K8s distros (EKS included) ship default PDBs for core components like CoreDNS.
  • Pods with node affinity/anti-affinity or topology constraints that CA can’t satisfy on remaining nodes.

In AWS, CA’s unit of scale is the Auto Scaling Group / EKS Managed Node Group — each node group maps to a CA “node group” concept, and CA’s binpacking simulation has to reason in terms of instance type homogeneity per ASG. This is the architectural reason for running multiple node groups segmented by instance type/purpose (e.g., a general m5 group, a memory-optimized r5 group for specific workloads, a GPU g5 group) — CA picks which group to scale based on which would satisfy the pending pod’s requests/tolerations/nodeSelector, and a single monolithic mixed-instance group makes that simulation messier and less predictable.

Karpenter — worth raising proactively in any 2026 AWS conversation, since it’s increasingly displacing Cluster Autoscaler in EKS architectures:

  • Karpenter doesn’t manage pre-defined node groups at all — it provisions right-sized nodes on demand directly from EC2 (any instance type/AZ combination matching a NodePool‘s constraints), reacting to the same “pending pod” signal but without the ASG abstraction layer.
  • Faster provisioning (skips ASG launch-template/scaling-activity overhead), tighter binpacking (genuinely picks the cheapest instance type that fits, including effective Spot/On-Demand mixing via NodePool weighting), and consolidation logic that actively repacks underutilized nodes rather than just waiting for natural drain.
  • Worth framing as: “Cluster Autoscaler is the traditional, ASG-centric model; Karpenter is the AWS-native, instance-type-flexible model — most new EKS builds in 2026 default to Karpenter unless there’s a specific multi-cloud-portability reason to stick with CA.”

4. Readiness & Liveness Probes

These are the health-signal primitives every autoscaler, rollout mechanism, and Service endpoint list depends on — get probes wrong and everything downstream (rolling updates, HPA accuracy, Service routing) inherits that wrongness.

Liveness probe — answers “is this container alive enough to keep running, or should kubelet kill and restart it?”

  • Failure → kubelet kills the container, applying the pod’s restartPolicy.
  • The most common production mistake: pointing liveness at a deep dependency check (e.g., “can I reach the database?”). If the DB blips, every pod’s liveness probe fails simultaneously → kubelet restarts all of them → you’ve turned a transient DB issue into a self-inflicted full outage via cascading restarts. Liveness should check “is my own process deadlocked/hung,” not “are my dependencies healthy.”

Readiness probe — answers “is this pod ready to receive traffic right now?”

  • Failure → pod is removed from the Service’s Endpoints/EndpointSlice (traffic stops routing to it), but the pod is not killed or restarted — this is the core distinction interviewers check for.
  • This is exactly the right place to check dependency health — if the DB connection pool isn’t established yet, readiness should fail, traffic should route elsewhere, and the pod can recover/retry without being killed.

Startup probe — exists specifically to solve the conflict between slow-starting apps and aggressive liveness timing:

  • While a startup probe is defined and hasn’t yet succeeded, liveness and readiness probes are disabled entirely. Once startup succeeds, normal liveness/readiness checking begins.
  • Without this, a JVM app with a 90-second cold start under a liveness probe with initialDelaySeconds: 30, periodSeconds: 10, failureThreshold: 3 would get killed at ~60s, every single time, before it ever finished starting — a classic real-world bug pattern this probe type was added specifically to fix.

Probe mechanics worth knowing precisely:

  • Three probe handler types: httpGet (expects 2xx/3xx), tcpSocket (just checks the port accepts a connection), exec (runs a command inside the container, success = exit code 0).
  • gRPC probes (stable since 1.27) — native support for gRPC health-checking protocol, avoiding the old workaround of shelling out to grpc_health_probe via exec.
  • Tuning fields: initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold, failureThreshold — the architect-level point worth making explicitly: these four numbers directly determine your Mean-Time-To-Detect for a bad pod, and should be set deliberately based on the app’s actual characteristics, not left at defaults copy-pasted from a tutorial.

Architect’s summary framing for this trio: liveness = “kill it,” readiness = “route around it,” startup = “be patient before judging it” — and conflating liveness with a dependency-health check is the single most common production-grade mistake to flag in a review.

5. Rolling Updates & Rollbacks

The Deployment controller manages rollout via an underlying mechanism of creating a new ReplicaSet and scaling it up while scaling the old one down, governed by strategy.rollingUpdate:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 25%
    maxSurge: 25%
  • maxUnavailable — how many pods (of desired count) can be unavailable during the rollout. Lower = safer, slower.
  • maxSurge — how many extra pods beyond desired count can exist temporarily during rollout. Higher = faster rollout, more transient resource pressure.
  • These can be absolute numbers or percentages, and the combination determines rollout pace — e.g., maxUnavailable: 0, maxSurge: 1 gives a strictly additive, zero-downtime-guaranteed rollout (spin up one new pod, wait for ready, kill one old pod, repeat) at the cost of needing headroom for that extra pod’s resources during the transition.

Readiness probes are the gate that makes rolling updates safe — a new pod is only counted as “available” toward satisfying maxUnavailable/surge math once its readiness probe passes. This is precisely why a misconfigured or missing readiness probe is one of the most common causes of a rollout that looks successful (kubectl rollout status reports done) but is actually routing traffic to pods that aren’t really ready — the Deployment controller has no other signal to know otherwise.

Rollback mechanics:

  • Kubernetes retains old ReplicaSets (count controlled by spec.revisionHistoryLimit, default 10) specifically to enable kubectl rollout undo.
  • kubectl rollout undo deployment/api reverts to the immediately prior ReplicaSet; kubectl rollout undo deployment/api --to-revision=N targets a specific historical revision (viewable via kubectl rollout history deployment/api).
  • A rollback is itself just another rolling update in reverse — same maxUnavailable/maxSurge mechanics apply, it’s not a special-cased instant operation.

The detection-gap problem worth raising at architect level: native Deployment rollouts have no automated failure detection or auto-rollback built in — if a bad version passes its (possibly too-shallow) readiness probe but is actually broken (wrong business logic, silent error rate spike), the rollout completes “successfully” with no automatic intervention. This is the exact gap that canary/progressive-delivery tooling — Argo Rollouts or Flagger — fills: automated traffic-shifted canary stages, real-time metric analysis (error rate, latency via Prometheus) gating progression, and automatic rollback triggered by metric regression rather than relying on a human watching a dashboard. Worth naming explicitly in any senior conversation: vanilla Deployment rolling updates are “rollout mechanics,” Argo Rollouts/Flagger add the “is this rollout actually safe” intelligence layer on top — directly parallel to the StatefulSet-vs-Operator distinction from the storage module.