Kubernetes Operations & Troubleshooting

Kubernetes Operations & Troubleshooting — Architect-Level Deep Dive

1. Troubleshooting Pods, Nodes, Deployments

This is the practical synthesis of nearly every prior module — effective troubleshooting is really just knowing which layer to look at first given a symptom, and reading the signals each layer surfaces correctly.

Pod-level triage — the standard escalation sequence worth having automatic:

  • kubectl get pods — Pending means the scheduler hasn’t placed it yet (almost always insufficient resources, an unsatisfiable node selector/affinity, or a missing/unbound PVC blocking scheduling — kubectl describe pod surfaces the scheduler’s exact rejection reason in the Events section, which is the first place to look, not a guess).
  • CrashLoopBackOff — the container is starting and exiting repeatedly; kubectl logs <pod> --previous is the critical flag here (the current container instance may have produced no useful output yet, but the previous crashed instance’s logs almost always have the actual error) — and the exponential backoff itself (10s, 20s, 40s… capped at 5 minutes) is worth knowing precisely, since “why is my pod taking 5 minutes to restart” is a recurring confusion this backoff timing directly explains.
  • ImagePullBackOff — almost always one of: wrong image tag/typo, private registry without imagePullSecrets configured on the ServiceAccount/pod, or IRSA/Pod-Identity misconfiguration if pulling from ECR with IAM-based auth — kubectl describe pod again surfaces the exact pull error string, which usually names the root cause directly (401 vs 404 vs DNS failure each point to a different one of these three).
  • OOMKilled (visible in kubectl describe pod‘s “Last State” reason) — the container exceeded its memory limit specifically (not request) — this is the direct, concrete link back to the resource-limits discussion in section 2, and the fix is rarely “increase the limit blindly” — it’s “profile actual usage and right-size, since an OOMKill at a too-low limit and a genuine memory leak look identical from this signal alone without further investigation.”
  • Readiness vs Liveness failure (from the health-checks module, worth the explicit callback here) — kubectl describe pod‘s Events distinguish these clearly, and the diagnostic path differs: a readiness failure means “check the app/dependency,” a liveness failure (causing restarts) means “check if the process is genuinely hung, or if the probe itself is misconfigured against a slow-but-healthy app.”

Node-level triage — kubectl describe node surfaces Conditions (MemoryPressure, DiskPressure, PIDPressure, Ready) and Taints — a node under DiskPressure will actively evict pods to reclaim space, which is the frequent real-world explanation for “why did my pod get killed even though it looked healthy” when the actual root cause is node-level, not pod-level. kubectl top node (backed by metrics-server, from the observability module) gives the quick resource-pressure gut-check before digging into Conditions.

Deployment-level triage — kubectl rollout status deployment/<name> and kubectl describe deployment reveal whether a rollout is stuck (new ReplicaSet not progressing — almost always because new pods aren’t passing readiness, directly tying back to the rolling-updates module’s point that readiness probes gate rollout progress) versus a Deployment that’s “done” but serving broken traffic anyway (the detection-gap the canary/progressive-delivery discussion exists to close).

The single architectural point worth stating explicitly in any troubleshooting-methodology conversation: kubectl describe before kubectl logs, always — Events almost always narrow the search space (scheduling failure vs image failure vs crash vs OOM vs probe failure are all distinguishable from Events alone) before you need to read a single line of application log output, and skipping straight to logs without checking Events first is the most common avoidable inefficiency in real incident response.

2. Resource Limits & Requests

Requests are what the scheduler uses to decide pod placement — a pod is only scheduled onto a node that has at least the requested CPU/memory available (unreserved by other pods’ requests) — requests are a scheduling-time guarantee, not a runtime cap.

Limits are what the kubelet/container runtime enforces at runtime — and CPU and memory limits behave fundamentally differently, which is the single most important nuance to get right:

  • CPU limits are throttled, not killed — exceeding a CPU limit means the container gets CFS-throttled (its CPU time is capped via the kernel’s Completely Fair Scheduler quota mechanism) — the process keeps running, just slower, manifesting as latency degradation rather than a crash. This is frequently misdiagnosed: a service experiencing mysterious periodic latency spikes under load is very often hitting CPU throttling at its limit, not a code-level performance bug — container_cpu_cfs_throttled_periods_total (a cAdvisor/kube-state-metrics metric) is the specific signal that confirms this, worth naming as the first thing to check before chasing a phantom application-code investigation.
  • Memory limits result in OOMKill, not throttling — there’s no equivalent “slow down” mechanism for memory; exceeding the limit means the kernel’s OOM killer terminates the container outright (the OOMKilled status from section 1) — this asymmetry between CPU (soft, throttled) and memory (hard, fatal) is worth stating explicitly, since it directly informs how conservatively each should be set.

QoS Classes — the consequence of how requests/limits are set relative to each other, and a frequently underappreciated scheduling/eviction lever:

  • Guaranteed — requests == limits for every container in the pod, both CPU and memory — these pods are the last to be evicted under node resource pressure.
  • Burstable — requests < limits (or only some resources/containers specify limits) — evicted before Guaranteed, but after BestEffort.
  • BestEffort — no requests/limits specified at all — first to be evicted under any node pressure, and notably also gets the lowest CPU-shares priority during contention even absent eviction.
  • Architect’s takeaway worth stating directly: for genuinely critical workloads (a payment-processing service, a primary database), setting requests == limits (Guaranteed QoS) is a deliberate reliability decision, not just a resource-governance nicety — it’s the mechanism that actually protects that workload from being evicted first when a node comes under pressure from noisy neighbors.

Namespace-level governance, the layer above individual pod specs:

  • ResourceQuota — caps aggregate resource consumption (total CPU/memory requests+limits, total PVC storage, total object counts) per namespace — the primary defense against the dynamic-provisioning cost-sprawl concern raised in the storage module, and the standard multi-tenancy guardrail in any shared cluster.
  • LimitRange — sets default requests/limits for containers that don’t specify their own, plus min/max bounds — this is what prevents the “someone deployed a pod with zero resource specification and it became a noisy, unbounded BestEffort neighbor” scenario by default, rather than relying on every team remembering to set requests/limits themselves every time.

The right-sizing problem this all points back to: setting requests/limits accurately requires actual usage data, not guesswork — this is precisely where VPA’s Off-mode recommendations (from the autoscaling module) earn their keep even when you don’t want VPA actively resizing anything — using it purely as an observability tool to inform manually-set, deliberate requests/limits is a legitimate and common production pattern.

3. Node Maintenance & Draining

Node maintenance (OS patching, K8s version upgrades, instance replacement) requires safely evacuating workloads before touching the node — kubectl drain is the primitive, but the safety guarantees it provides depend entirely on whether the workloads being drained are configured to cooperate.

kubectl drain <node> mechanics:

  1. Cordon (kubectl cordon, which drain performs automatically first) — marks the node unschedulable (Schedulable: false), so no new pods land there, while existing pods keep running undisturbed — this is the first, non-disruptive step, and worth knowing it can be done independently of a full drain when you just want to stop new scheduling without yet evicting anything.
  2. Evict every pod on the node via the Eviction API (which respects PodDisruptionBudgets, distinct from a raw delete) — pods managed by a controller (Deployment/StatefulSet/etc.) get rescheduled elsewhere automatically; bare/unmanaged pods block drain by default unless --force is passed (and forcing means that pod is simply deleted with no replacement — worth flagging as a real, sometimes-overlooked data-loss risk if that bare pod was running something stateful).

PodDisruptionBudget (PDB) is the object that makes draining actually safe rather than just fast, and is the direct connective tissue between this section and the Cluster Autoscaler scale-down discussion from the autoscaling module (CA’s scale-down respects PDBs identically):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels: {app: api}
  • Drain (and CA scale-down, and any other voluntary disruption) will not evict a pod if doing so would violate the PDB’s minAvailable (or maxUnavailable) — this is specifically about voluntary disruptions (drains, scale-downs); it does not protect against involuntary disruptions (node hardware failure, kernel panic) — a distinction worth stating precisely, since PDBs are sometimes mistakenly assumed to be a general availability guarantee rather than specifically a “don’t let our own maintenance operations cause an outage” guard rail.
  • No PDB defined means drain can evict down to zero replicas if the controller would otherwise satisfy availability through rescheduling — for genuinely critical services, an explicit PDB is a deliberate, necessary safety control, not optional boilerplate, and its absence on a critical workload is a real finding worth flagging in an operational-readiness review.

Graceful shutdown — the piece on the workload’s own side that needs to cooperate with eviction for draining to be truly zero-impact, worth naming explicitly: preStop hooks (allowing in-flight requests to finish, or deregistering from a load balancer before the container actually stops) plus an adequate terminationGracePeriodSeconds (default 30s — often too short for connections with longer-lived requests/websockets) — without these, a pod can be “successfully evicted” per Kubernetes’ bookkeeping while still dropping in-flight requests, which is the same graceful-termination concern raised earlier in the blue-green discussion, just generalized to any voluntary pod termination, not only deployment cutovers.

Managed-node-group specific consideration worth naming for cloud contexts: EKS Managed Node Groups, AKS node pools, and GKE node pools all provide managed node upgrade/replacement workflows that internally perform this same cordon-drain-replace sequence automatically (respecting PDBs) when you trigger a node-group version upgrade — meaning for routine OS/K8s-version patching on managed node groups, you’re usually triggering the managed upgrade workflow rather than running kubectl drain by hand node-by-node — manual draining remains the right tool for one-off, ad-hoc node-level interventions (debugging a specific problematic node, an unplanned hardware-level concern) rather than routine fleet-wide patching.

4. Backup & Restore Strategies

Kubernetes backup needs to address two genuinely distinct layers, and conflating them is a common real-world gap in disaster-recovery planning:

Layer 1 — Cluster/application configuration state (the Kubernetes API objects themselves: Deployments, Services, ConfigMaps, RBAC, CRDs):

  • For managed control planes (EKS/AKS/GKE), the etcd backup itself is the cloud provider’s responsibility — you don’t manage etcd snapshots directly on managed offerings, which is one of the genuine, concrete benefits of “managed” from the cloud module’s earlier discussion.
  • For self-managed clusters, etcdctl snapshot save (or a CronJob automating it) against etcd directly is the backup mechanism, and restoring requires careful sequencing (stopping the API server, restoring the snapshot, restarting) — a meaningfully more operationally involved DR story, worth naming as another concrete cost of the self-managed path raised in the cloud module.
  • Velero is the standard tool regardless of managed/self-managed status, and worth distinguishing clearly from raw etcd snapshots: Velero backs up Kubernetes objects (via the API, namespace-by-namespace or cluster-wide, with label-selector filtering) and can coordinate PV snapshotting (via cloud-native snapshot APIs — EBS snapshots, Azure Disk snapshots, GCP PD snapshots — through Velero’s plugin model) in the same backup operation — this object-level, API-driven approach is generally more practical and portable than raw etcd snapshots for application-level backup/restore (e.g., “restore just this one namespace,” or “migrate this application to a different cluster entirely”) versus etcd snapshots, which are really a whole-cluster, same-cluster-topology disaster-recovery mechanism, not a flexible application-migration tool.

Layer 2 — Persistent application data (database contents, not Kubernetes objects):

  • This is fundamentally a storage-layer concern, not a Kubernetes-native one — EBS/Azure-Disk/PD snapshots (often Velero-orchestrated for consistency with the API-object backup, as above) for volume-level point-in-time recovery, plus application-native backup tooling for anything requiring transactional consistency a filesystem-level snapshot can’t guarantee on its own (a database’s own pg_dump/binary-log-based backup, or an Operator’s built-in backup CRD — Zalando/CrunchyData Postgres Operators and Strimzi for Kafka, referenced back in the storage module’s StatefulSet discussion, typically ship their own backup-scheduling CRDs specifically because they understand the application’s consistency requirements in a way a generic snapshot tool can’t).
  • The RWX/RWO distinction from the storage module resurfaces here directly: backing up an RWX (EFS) volume used by multiple pods concurrently needs to account for in-flight writes during the snapshot window differently than a single-writer RWO volume — worth naming as a real consideration when designing backup windows for shared-filesystem workloads specifically.

The DR architecture worth being able to state as a coherent whole: GitOps (from the CI/CD module) already gives you most of Layer 1’s recovery story for free — if your Deployments/Services/ConfigMaps/RBAC are all declared in Git, rebuilding a destroyed cluster’s configuration state is “point ArgoCD/Flux at the same Git repo,” not “restore from a Velero backup” — Velero’s real, irreplaceable value is specifically for stateful data Layer 2 can’t recover any other way, and for cluster objects that aren’t GitOps-managed (rare in a mature setup, but worth checking for, e.g., dynamically-created Secrets that were never meant to live in Git at all, like cert-manager-issued certificates, which have their own regeneration story rather than needing backup). A mature DR runbook should explicitly state, for each category of cluster state, which of “GitOps reconciliation,” “Velero restore,” or “application-native backup restore” is the actual recovery path — rather than treating “do we have backups” as a single undifferentiated yes/no.

5. Cost Optimization in Kubernetes

Kubernetes cost optimization is genuinely cross-cutting — touching nearly every module covered so far — worth synthesizing as the closing topic since it pulls together compute, storage, and autoscaling decisions into a single bottom-line lens.

Compute — typically the largest line item, with several concrete levers:

  • Right-sizing via VPA recommendations (autoscaling module) — over-provisioned requests are the single most common, most fixable source of waste; a fleet-wide audit comparing actual usage (via VPA Off-mode or simply historical Prometheus data) against declared requests routinely surfaces 30-50%+ over-provisioning in unaudited estates — worth naming as the first, lowest-effort cost lever to pull before reaching for anything more architecturally involved.
  • Karpenter/Cluster Autoscaler consolidation (autoscaling module) — Karpenter’s active binpacking/consolidation specifically targets the “we’re running more, larger nodes than current load actually needs” waste pattern that a static, never-revisited node group sizing tends to accumulate over time.
  • Spot instances — for fault-tolerant, interruption-tolerant workloads (stateless services with multiple replicas, batch jobs, CI runners), Spot pricing typically runs 60-90% cheaper than On-Demand — Karpenter’s native Spot/On-Demand weighted provisioning (or Cluster Autoscaler with mixed-instance ASGs) is the standard mechanism, paired with Pod Disruption Budgets and the AWS Node Termination Handler (or Karpenter’s native interruption handling) to gracefully drain pods on the 2-minute Spot-interruption warning rather than losing them abruptly — worth being explicit that Spot adoption without that graceful-handling layer is a reliability risk, not just a cost lever, and the two need to be designed together.
  • Fargate/Autopilot’s per-pod billing model (cloud module) — worth a quick callback: this shifts the cost-optimization unit from “are our nodes right-sized” to “are our individual pod requests right-sized,” since you’re billed directly for what each pod requests rather than amortized across a node — meaning the VPA right-sizing lever above becomes even more directly tied to your actual bill on these platforms, not just an indirect efficiency improvement.

Storage:

  • StorageClass tiering (storage module) — defaulting every workload to the highest-performance tier (io2, provisioned-IOPS) when most don’t need it is a common, easily-audited waste source — most workloads are well served by gp3 (notably cheaper than the older gp2 at equal or better baseline performance, worth naming as a frequently-missed “just switch the default StorageClass” quick win in older clusters still defaulting to gp2).
  • Reclaim policy discipline — Delete-policy PVs cleaning up automatically on PVC deletion (versus orphaned Retain volumes silently accumulating cost indefinitely) — an unglamorous but real recurring finding: orphaned EBS volumes from deleted PVCs/StatefulSets with no cleanup process is a genuinely common source of slow, invisible cost creep worth specifically auditing for (aws ec2 describe-volumes --filters Name=status,Values=available finds exactly this category of waste directly).

Networking:

  • Cross-AZ data transfer cost — referenced earlier via topology-aware load balancing (service mesh module) — this is a real, often surprisingly large line item at scale that’s invisible until someone actually breaks down a cloud bill by line item; locality-aware routing (mesh-native, or even just topology.kubernetes.io/zone-aware internalTrafficPolicy: Local on Services, a lighter-weight option than adopting a full mesh purely for this) is worth naming as a concrete, costed lever.
  • Consolidating LoadBalancer provisioning behind a shared Ingress/Gateway (cloud module) rather than one cloud LB per microservice — directly reduces the count of billed LB resources.

Observability tooling cost itself, worth naming as a meta-point: high-cardinality metrics (the Prometheus cardinality-explosion concern from the observability module) and verbose, unfiltered logging at scale are themselves genuine cost centers (CloudWatch/Loki/OpenSearch ingestion and storage costs scale directly with volume) — a mature cost-optimization review includes auditing the observability stack’s own footprint, not just compute/storage for application workloads — log-level discipline, metric-cardinality discipline, and trace-sampling rates (tracing module) are cost levers in their own right, not purely technical-design decisions.

The architect’s closing synthesis worth stating plainly: Kubernetes cost optimization isn’t a separate discipline bolted onto the architecture after the fact — it’s the natural byproduct of doing the right-sizing (resource requests/limits), autoscaling (HPA/VPA/Karpenter), and storage-tiering practices already covered throughout this entire curriculum, correctly and with actual usage data rather than defaults and guesswork. Teams that struggle with Kubernetes cost almost always have a gap in one of the technical practices covered in the earlier modules, not a missing “cost optimization” initiative as a distinct workstream — which is worth stating explicitly as the closing thread tying all ten modules together.