Kubernetes Storage

Kubernetes Storage — Architect-Level Deep Dive

1. Volumes

A Volume in Kubernetes is fundamentally different from a Docker volume — it’s a pod-scoped (not container-scoped) abstraction, mounted into one or more containers within that pod, with a lifecycle tied to the pod, not any individual container restart. If a container crashes and restarts, the volume’s data survives; if the pod itself is deleted, most volume types’ data does not (this distinction is the entire reason PV/PVC exists, covered below).

Ephemeral volume types (lifecycle = pod lifecycle, no decoupled lifecycle):

  • emptyDir — scratch space, created empty on pod start, deleted on pod removal. Backed by node disk or, with medium: Memory, tmpfs (RAM-backed — fast but counts against pod memory limits, and data is lost on node reboot). Classic use: scratch space for a sidecar pattern (e.g., a log-shipper sidecar reading from a shared emptyDir that the main container writes to).
  • configMap / secret — projects config/secret data as files into the pod’s filesystem, dynamically updated on change (with a propagation delay, typically tied to kubelet sync period — usually ~1 minute, not instant, which trips people up when they expect immediate config reload).
  • downwardAPI — exposes pod/container metadata (labels, annotations, resource requests) as files.
  • projected — combines multiple volume sources (secret + configMap + downwardAPI + serviceAccountToken) into a single mount point.
  • hostPath — mounts a path from the node’s filesystem directly into the pod. Powerful but dangerous — breaks the pod-portability model (data is tied to a specific node), and is a well-known privilege-escalation vector if misused (mounting / or /var/run/docker.sock is a classic container-breakout technique) — architecturally, treat hostPath as something you restrict via PodSecurityStandards/admission policy, not something app teams get by default.

The architectural takeaway: ephemeral volumes solve inter-container sharing within a pod and scratch space. They do not solve durability across pod rescheduling — that’s the PV/PVC system’s job.

2. Persistent Volumes (PV)

A PersistentVolume is a cluster-scoped resource representing an actual piece of provisioned storage — an AWS EBS volume, an EFS mount target, an NFS export, a Ceph RBD image — abstracted away from whichever pod eventually consumes it. The PV object is infrastructure; it exists independently of any namespace or pod, with its own lifecycle.

Key PV fields an architect should reason about:

  • capacity — size of the volume.
  • accessModes — defines concurrency semantics:
    • ReadWriteOnce (RWO) — mountable read-write by a single node (not single pod — multiple pods on the same node can mount it, a frequent point of confusion). EBS volumes are RWO-only.
    • ReadOnlyMany (ROX) — multiple nodes, read-only.
    • ReadWriteMany (RWX) — multiple nodes, read-write. Requires a storage backend that genuinely supports concurrent multi-node writers — EFS, NFS, CephFS, Azure Files. EBS cannot do RWX — this is one of the single most common AWS architecture interview traps: “can two pods on different nodes both write to this EBS-backed PVC?” — no, full stop, you need EFS for that.
    • ReadWriteOncePod (RWOP) — newer (1.22+), restricts to a single pod cluster-wide, not just single node — useful for strict StatefulSet-style exclusivity guarantees (e.g., a database that must never have two writers under any circumstance, even during a rolling update overlap window).
  • persistentVolumeReclaimPolicy — Retain (PV and underlying storage survive PVC deletion — data preserved, but PV needs manual cleanup/rebind before reuse), Delete (underlying cloud storage is destroyed when the PVC is released — the common default for dynamically provisioned volumes), Recycle (deprecated, basic scrub-and-reuse — don’t use this in 2026, dynamic provisioning has superseded it).
  • volumeMode — Filesystem (default, mounted as a directory) vs Block (raw block device passed directly into the pod — used for performance-sensitive workloads like databases that want to manage their own filesystem, or specific storage engines like some Ceph RBD raw-device patterns).

Static vs dynamic provisioning is the fork in the road here: an architect can hand-create PV objects pointing at pre-existing storage (static provisioning — rare in cloud-native setups, more common in on-prem/legacy NFS environments), or let a StorageClass + provisioner create both the underlying cloud resource and the PV object on demand (dynamic provisioning — the default modern pattern, covered in section 5).

3. Persistent Volume Claims (PVC)

A PersistentVolumeClaim is the namespaced, developer-facing request for storage — “I need 10Gi, RWO, at least gp3 performance” — without the requester needing to know or care about the underlying cloud storage mechanics. This is the deliberate separation of concerns: PV = infrastructure team’s domain (or dynamic provisioner’s), PVC = application team’s domain.

Binding mechanics:

  • The PVC controller watches for unbound PVCs and tries to find a PV satisfying the claim’s accessModes, storageClassName, and resources.requests.storage (must be ≥ requested, not necessarily exact match in static provisioning).
  • Binding is one-to-one and exclusive — once a PV binds to a PVC, no other PVC can claim it, even if there’s “room” conceptually (a 100Gi PV claimed by a 5Gi PVC just wastes the remaining 95Gi — PVs aren’t subdivided).
  • If no suitable static PV exists and the PVC references a StorageClass with a provisioner, the dynamic provisioning path triggers — a new PV (and the underlying cloud disk) is created on-demand to satisfy exactly that claim.

Pod consumption:

volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-app-pvc

A pod references a PVC by name, never a PV directly — this indirection is what makes pod specs portable across environments (dev cluster’s my-app-pvc might dynamically bind to a small gp3 volume, prod’s identical pod spec binds to a much larger io2 volume — same YAML, different StorageClass behind the scenes).

A subtlety worth flagging architecturally: PVCs are immutable in most fields post-creation except size (volume expansion, if allowVolumeExpansion: true on the StorageClass) — you cannot change access mode or storage class of an existing PVC; the standard pattern for “I need to migrate storage classes” is create-new-PVC + data migration job + cutover, not an in-place edit.

4. Storage Classes

A StorageClass is the template that tells the dynamic provisioning system how to create storage when a PVC asks for it — which provisioner to invoke, what parameters to pass it, what reclaim policy and binding behavior to apply.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

volumeBindingMode is the single most important field architects get wrong:

  • Immediate (legacy default) — PV is provisioned as soon as the PVC is created, before any pod scheduling decision. This causes a notorious failure mode in multi-AZ clusters: the EBS volume gets created in us-east-1a, but the scheduler later places the pod in us-east-1c — pod gets stuck Pending forever because EBS volumes are AZ-locked and can’t attach cross-AZ.
  • WaitForFirstConsumer (the modern correct default for AWS) — provisioning is delayed until a pod actually references the PVC and gets scheduled, so the CSI driver provisions the volume in the same AZ the scheduler already chose. This is a near-mandatory setting for any AWS EBS-backed StorageClass — if you’re ever reviewing a cluster and see Immediate with multi-AZ node groups and EBS, that’s a red flag worth raising.

CSI (Container Storage Interface) is the modern provisioner mechanism — replacing the old in-tree volume plugins (kubernetes.io/aws-ebs etc., now deprecated/removed in current K8s versions). Each storage backend ships a CSI driver (a set of pods — typically a controller Deployment + node DaemonSet) implementing the standardized CSI gRPC interface (CreateVolume, ControllerPublishVolume, NodeStageVolume, etc.). This decoupling means Kubernetes core doesn’t need backend-specific code anymore — AWS, Azure, GCP, Ceph, Portworx, Longhorn all ship their own out-of-tree CSI drivers following the same contract.

Multiple StorageClasses per cluster is the norm, not the exception — a typical production EKS setup has at minimum: a fast io2/high-IOPS class for databases, a general-purpose gp3 default class, and an efs-sc class (via the EFS CSI driver) for RWX workloads — letting application teams pick storage tier via a single field (storageClassName) without needing infra knowledge.

5. Dynamic Provisioning

Putting it together, the dynamic provisioning flow end-to-end:

  1. Developer creates a PVC referencing a StorageClass (or relies on the cluster’s default StorageClass, marked via the storageclass.kubernetes.io/is-default-class: "true" annotation — exactly one should be default per cluster, or you get ambiguous/unexpected binding behavior).
  2. If volumeBindingMode: WaitForFirstConsumer, the PVC sits unbound until a pod referencing it is scheduled.
  3. Scheduler picks a node (factoring in topology constraints, resource fit, any existing PV topology affinity).
  4. The external-provisioner sidecar (part of the CSI driver’s controller pod) sees the pending PVC, calls the CSI driver’s CreateVolume gRPC method, which talks to the actual cloud API (e.g., ec2:CreateVolume for EBS).
  5. A PV object is created in Kubernetes representing that newly created cloud volume, bound to the PVC.
  6. The external-attacher and node-driver-registrar components handle attaching the volume to the correct EC2 instance and mounting it into the pod’s filesystem on the target node.

Why this matters architecturally: dynamic provisioning is what makes Kubernetes-managed storage actually elastic — no infra ticket, no pre-allocated volume pool, storage capacity scales with declared intent. The tradeoff is cost governance — without ResourceQuota on requests.storage per namespace, or LimitRange, teams can dynamically provision unbounded EBS/EFS spend. This is a real FinOps conversation worth having in any architecture review — dynamic provisioning needs a quota guardrail layered on top in multi-tenant clusters.

Volume expansion (allowVolumeExpansion: true) is part of this same CSI-driven lifecycle — a developer edits the PVC’s requests.storage upward, the CSI driver’s ControllerExpandVolume resizes the underlying cloud disk, and (depending on filesystem) a NodeExpandVolume call grows the filesystem online — for EBS+ext4/xfs this is typically a no-downtime operation, which is a meaningful operational win over the old “detach, snapshot, resize, reattach” manual dance.

6. Stateful Workloads

StatefulSet is the workload controller purpose-built for anything that isn’t a stateless, interchangeable replica — databases, message brokers, anything with stable identity, ordered deployment, and stable per-instance storage.

The three guarantees StatefulSet provides that Deployment doesn’t:

  1. Stable, predictable pod names — <statefulset-name>-0, -1, -2… (not random hashes like Deployment’s ReplicaSet-generated names). This stability is what lets ordinal-based clustering logic work (e.g., a Kafka broker knowing it’s broker -0 and reading config accordingly).
  2. Stable network identity — paired with a headless Service, each pod gets a DNS name in the form <pod-name>.<service-name>.<namespace>.svc.cluster.local that persists across pod rescheduling. Even if kafka-0 gets evicted and rescheduled to a different node, it comes back as kafka-0 with the same DNS entry — critical for systems doing peer-discovery via DNS (etcd, Cassandra, Kafka/ZooKeeper ensembles).
  3. Stable, per-pod storage via volumeClaimTemplates — this is the storage-specific magic. Instead of one shared PVC, StatefulSet generates a dedicated PVC per ordinal, named <volumeClaimTemplate-name>-<statefulset-name>-<ordinal>. When kafka-0 is deleted and recreated, it reattaches to its own data-kafka-0 PVC — not a fresh volume, not another pod’s volume. This is the mechanism that makes a rescheduled stateful pod resume with its actual prior data, rather than starting empty.
volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi

Ordered, graceful rollout semantics:

  • Default podManagementPolicy: OrderedReady — pods scale up/down strictly sequentially (-0 must be Running+Ready before -1 starts), and scale-down happens in reverse order. This matters for systems with bootstrap dependencies (e.g., the first node in a cluster needs to fully initialize before subsequent nodes join).
  • podManagementPolicy: Parallel — opts out of ordering when the workload doesn’t need it (faster scaling, but the app must tolerate concurrent startup).
  • Rolling updates proceed in reverse ordinal order by default (-2, then -1, then -0) — and critically, StatefulSet rolling updates will not proceed past a pod that fails its readiness probe, which is the built-in safety valve against rolling out a broken config across an entire stateful cluster.

The deletion gotcha every architect needs to know: deleting a StatefulSet does not delete its PVCs by default — this is intentional (data safety), but means scale-down-then-scale-up reattaches old data, and a full StatefulSet deletion leaves orphaned PVCs/PVs behind that need explicit cleanup (or rely on persistentVolumeClaimRetentionPolicy, GA since 1.27, which lets you declare whenDeleted: Delete / whenScaled: Delete to control this behavior natively instead of relying on manual cleanup scripts).

Real-world architecture pattern worth naming explicitly: for genuinely production-grade stateful workloads (Postgres, Kafka, etc.), most architects in 2026 reach for a purpose-built Operator (Zalando/CrunchyData Postgres Operator, Strimzi for Kafka) layered on top of StatefulSet rather than hand-rolling raw StatefulSet YAML — the operator encodes domain-specific operational logic (failover, backup scheduling, version upgrades, leader election) that vanilla StatefulSet semantics alone don’t provide. Worth mentioning this distinction in any senior-level discussion: StatefulSet gives you the primitives (identity, ordering, storage stability); the Operator pattern gives you the operational intelligence on top.