Kubernetes Security

Kubernetes Security — Architect-Level Deep Dive

1. RBAC (Role-Based Access Control)

RBAC governs who can do what to which resources within the Kubernetes API itself — it’s an authorization layer sitting after authentication (who you are) and before admission control (is this request otherwise valid).

Four core objects, and the distinction that gets tested most:

  • Role — namespaced, grants permissions on resources within that namespace only.
  • ClusterRole — cluster-scoped, grants permissions either cluster-wide or can be bound to a single namespace via a RoleBinding (yes — a ClusterRole can be namespace-scoped in its binding even though the Role definition itself isn’t namespaced; this is the single most common point of confusion in RBAC discussions). ClusterRoles are also required for cluster-scoped resources (Nodes, PersistentVolumes, Namespaces themselves) since a namespaced Role simply cannot grant access to non-namespaced objects.
  • RoleBinding — binds a Role (or ClusterRole) to subjects (users/groups/ServiceAccounts), scoped to one namespace.
  • ClusterRoleBinding — binds a ClusterRole to subjects across the entire cluster.

The four-combination matrix every architect should have memorized:

Role ClusterRole
RoleBinding Permissions in this namespace only ClusterRole’s rules applied only within this namespace
ClusterRoleBinding (not valid — Role isn’t cluster-scoped) Permissions cluster-wide

Rule anatomy:

rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  • apiGroups: [""] is the core API group (pods, services, configmaps, secrets, etc.) — a frequent gotcha is forgetting that core resources use empty string, not "v1".
  • Verbs map to HTTP-ish semantics: get/list/watch (read), create/update/patch/delete (write), plus special ones like deletecollection, and the bind/escalate/impersonate verbs which are themselves security-sensitive (a subject with escalate rights can grant itself broader permissions than its current Role — i.e., RBAC has a self-amplification vector you must explicitly guard against, not assume away).

Architect-level principle worth stating explicitly: RBAC is additive-only. There is no “deny” rule in core Kubernetes RBAC — permissions are a union across every Role/ClusterRole bound to a subject, and you cannot subtract from that. If you need explicit deny semantics (e.g., “this team can do everything except delete production secrets”), you need either careful Role decomposition (smaller, more numerous Roles, granted precisely) or an admission-control layer (OPA Gatekeeper/Kyverno) enforcing deny policies on top — RBAC alone structurally cannot express “allow X except Y.”

kubectl auth can-i is the day-to-day debugging tool (kubectl auth can-i delete pods --as=system:serviceaccount:default:my-sa -n production) — worth naming as the first thing to reach for when diagnosing “why can’t this pod/user do X.”

2. Service Accounts

A ServiceAccount is the identity a pod (not a human) uses to authenticate to the Kubernetes API — every pod runs as some ServiceAccount, defaulting to the namespace’s default ServiceAccount if none is specified.

Mechanics of how identity actually reaches the pod:

  • Modern Kubernetes (1.22+) uses projected, time-bound, audience-scoped tokens via the TokenRequest API — these are short-lived JWTs (default 1-hour expiry, auto-rotated by kubelet), mounted at /var/run/secrets/kubernetes.io/serviceaccount/token, a meaningful security improvement over the old (pre-1.22) behavior of mounting a long-lived, non-expiring Secret-based token automatically into every pod — those legacy tokens were a standing credential-exposure risk if a pod was compromised, since they never expired and were valid cluster-wide for that ServiceAccount’s permissions indefinitely.
  • automountServiceAccountToken: false (settable at ServiceAccount or pod-spec level) is the standard hardening move for pods that genuinely never call the K8s API — most application workloads fall into this category, and leaving the token auto-mounted on a pod that never uses it is pure attack-surface with zero benefit.

The architectural anti-pattern worth flagging explicitly: granting the default ServiceAccount in a namespace broad permissions (or worse, cluster-admin) is a frequent real-world finding in security reviews — every pod in that namespace that doesn’t explicitly specify a different ServiceAccount inherits that access by default. The correct pattern is one dedicated, narrowly-scoped ServiceAccount per workload that actually needs API access, with the namespace’s default ServiceAccount left deliberately unprivileged (often explicitly bound to no RoleBinding at all).

IRSA (IAM Roles for Service Accounts) — the AWS-specific extension architects working in EKS need fluent: ServiceAccounts are annotated with an IAM role ARN, and via OIDC federation between the EKS cluster’s OIDC provider and AWS IAM, pods using that ServiceAccount can assume the AWS IAM role directly — no static AWS credentials anywhere in the cluster. This is the standard pattern replacing the older, much worse practice of mounting AWS access keys as Secrets, or relying on the node’s instance-profile role (which over-grants every pod on that node the same AWS permissions, regardless of which workload actually needs them). Note: Pod Identity (newer AWS feature, simpler setup than IRSA, no OIDC trust-policy juggling) is increasingly the recommended default for new EKS clusters in 2026 — worth mentioning IRSA is still extremely common in existing infrastructure, but Pod Identity is the direction AWS is pushing.

3. Network Policies (security-lens recap)

Already covered the mechanics (default-allow until selected, podSelector/namespaceSelector/ipBlock, L3/L4-only limitation, CNI-dependent enforcement) in the networking module — worth adding the security-specific framing here since this list re-surfaces it explicitly:

  • Network Policies are your in-cluster segmentation control, equivalent to security groups inside the cluster boundary. Without them, lateral movement post-compromise is trivially easy — a single compromised pod in a default-allow cluster can reach every other pod, every Service, often the metadata API (cloud credential theft vector if IMDS isn’t otherwise locked down).
  • A baseline default-deny-all + explicit allow posture (shown in the storage/networking module) is the minimum bar for any production namespace handling sensitive data — this is the kind of finding that shows up repeatedly in real security assessments, and “we have Network Policies defined” without verifying default-deny is in place is a common false sense of security.
  • Worth explicitly naming the metadata API egress risk in AWS specifically: without an egress NetworkPolicy restricting access to 169.254.169.254 (IMDS), a compromised pod can query instance metadata and, depending on IMDSv1 vs IMDSv2 and node IAM role scope, potentially extract node-level AWS credentials — this is precisely the kind of cross-cutting finding (storage + networking + IAM all intersecting) a senior architect should be able to articulate in a threat model, not just recite the policy YAML.

4. Secrets Management

Kubernetes Secrets solve the API ergonomics of injecting sensitive values into pods — they do not, out of the box, solve genuine secret security, and conflating the two is the most common architectural mistake in this space.

What native Secrets actually give you:

  • Base64-encoded (not encrypted by default) storage in etcd.
  • Injection as env vars or mounted files, RBAC-gated read access.

What native Secrets do not give you by default, and why each matters:

  • etcd-at-rest encryption is opt-in, not automatic — without an EncryptionConfiguration enabling aescbc/secretbox/KMS-provider encryption at the API server level, Secrets sit in etcd as base64 (not encryption — trivially reversible). Anyone with etcd access (or an etcd snapshot/backup) has the secrets in the clear. In EKS, etcd is fully managed by AWS and encrypted at rest by default — but if discussing self-managed clusters or kubeadm-based builds, this is a real configuration step, not a given.
  • Secrets are visible to anyone with get/list RBAC permission on Secrets in that namespace — and critically, anyone who can create a pod in a namespace can typically mount any Secret in that namespace into a pod they control and read it that way, even without direct get rights on the Secret object itself — a frequently underappreciated RBAC escalation path.
  • No native rotation, versioning, or audit trail of access beyond what the API server’s general audit logging captures.

Production-grade patterns architects should be steering toward:

  • External Secrets Operator (ESO) — syncs secrets from AWS Secrets Manager / SSM Parameter Store / HashiCorp Vault into native K8s Secret objects, keeping the actual secret material’s source of truth, rotation, and access auditing in the external system, while still letting application pods consume them the normal Kubernetes way.
  • CSI Secrets Store driver — mounts secrets from external stores directly into the pod’s filesystem without ever creating a Kubernetes Secret object at all — avoids the etcd-persistence and Secret-RBAC exposure entirely, at the cost of a slightly more complex driver/provider setup.
  • Sealed Secrets / SOPS — solves the GitOps half of the problem (how do you commit secret manifests to Git safely) by encrypting the Secret content such that only the cluster’s controller can decrypt it — orthogonal to, and often combined with, the runtime-secrets-store patterns above.
  • For AWS-native architectures specifically, the typical 2026 default is Secrets Manager + ESO, with automatic rotation configured at the Secrets Manager level (Lambda rotation function) so application teams never handle rotation logic themselves — worth naming as the “what does good look like” answer in an architecture review.

5. Pod Security Standards

Pod Security Standards (PSS) replaced the deprecated PodSecurityPolicy (removed entirely in 1.25) — PSS defines three levels of restriction, enforced via the built-in Pod Security Admission (PSA) controller (a built-in admission plugin, no separate component to install, unlike PSP’s old webhook model):

  • Privileged — unrestricted, effectively no policy. Appropriate only for genuinely infrastructure-level workloads (CNI plugins, CSI node drivers, monitoring agents needing host access) — should be the rare exception, never the default for app namespaces.
  • Baseline — blocks known privilege-escalation vectors while minimizing disruption to typical workloads: disallows privileged containers, host namespaces (hostNetwork, hostPID, hostIPC), most hostPath volume usage, and dangerous capabilities — but is still fairly permissive (e.g., doesn’t enforce running as non-root).
  • Restricted — the hardened tier, additionally requiring: non-root execution (runAsNonRoot: true), a read-only root filesystem encouraged (not strictly required by PSS itself, but the standard pairs naturally with this), dropped Linux capabilities (drop: [ALL], only adding back what’s explicitly needed like NET_BIND_SERVICE), seccompProfile set to RuntimeDefault, and disallowed privilege escalation (allowPrivilegeEscalation: false).

Enforcement is applied per-namespace via labels — this is the mechanic worth knowing precisely, since it’s a clean, no-extra-controller way to enforce baseline hygiene cluster-wide:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
  • enforce actually blocks non-compliant pod creation.
  • audit/warn flag violations (audit log entry / kubectl warning) without blocking — the standard rollout pattern is warn+audit first to assess blast radius, then flip to enforce once existing workloads are confirmed compliant, rather than enabling enforce cold and breaking things unexpectedly.

The honest limitation to name: PSS/PSA is intentionally simpler and less expressive than PSP ever was, and doesn’t cover everything (e.g., it doesn’t restrict specific image registries, doesn’t do fine-grained resource-based rules) — for anything beyond the three predefined tiers, you’re reaching for OPA Gatekeeper or Kyverno regardless, which leads directly into the next section.

6. Image Scanning & Admission Controllers

Two distinct concerns that get conflated and shouldn’t be:

Image scanning — finding known vulnerabilities (CVEs) in container image layers, typically at build/CI time (shift-left) and again at registry/runtime (catch newly disclosed CVEs in already-deployed images).

  • Trivy, Grype, Snyk Container, and cloud-native options (Amazon ECR’s native scanning, on-push or continuous) are the common tools — ECR scanning specifically integrates cleanly for AWS-centric architectures, scanning on push and (with “enhanced scanning”) continuously as new CVEs are published against already-stored images, not just at push-time.
  • The architectural decision point worth raising: scan-and-block in CI (fail the pipeline on critical CVEs before the image ever reaches a registry) vs scan-and-alert post-deploy — mature setups do both, with CI as the primary gate and registry/runtime scanning as a safety net for zero-day disclosures against images already running.

Admission controllers — the policy-enforcement layer that intercepts API requests after authentication+authorization, before persistence to etcd — this is where “deny” semantics that RBAC structurally can’t express actually get enforced.

  • Built-in examples: PodSecurity (PSA, above), ResourceQuota, LimitRanger, NamespaceLifecycle.
  • Policy-as-code admission controllers — the part most architect interviews are actually probing for: OPA Gatekeeper (Rego-based policy language, mature, steeper learning curve) and Kyverno (YAML-native policies, generally considered more approachable for teams without a Rego background) — both implemented as ValidatingAdmissionWebhooks (and increasingly MutatingAdmissionWebhook for auto-remediation, e.g., auto-injecting resource limits or labels rather than just rejecting).
  • Common real-world policies enforced at this layer (worth being able to rattle off concretely, since this is where security reviews get specific): block images not from an approved registry (prevents pulling arbitrary public Docker Hub images into production), require resource requests/limits on every container, disallow :latest tags (forces immutable, traceable deployments), require specific labels for cost-allocation/ownership tracking, enforce that images have passed scanning (often integrated via image signing — Sigstore/Cosign — where the admission controller verifies a cryptographic signature proving the image came from your CI pipeline and passed its gates, rather than trusting tag/registry alone, which is trivially spoofable).
  • ValidatingAdmissionPolicy (GA since 1.30) is worth naming as the newest evolution here — it lets you express a meaningful subset of these policies natively in-cluster using CEL (Common Expression Language) without deploying a separate webhook controller at all, reducing the operational overhead and the webhook-as-availability-risk concern (a misbehaving/unreachable admission webhook can block all pod creation cluster-wide if failurePolicy: Fail and the webhook pod itself is down — a real production incident pattern worth flagging when discussing webhook-based admission control’s own failure modes).

The architect’s synthesis worth stating in any security-design conversation: RBAC controls who can call the API, Network Policy controls what can talk to what over the wire, Pod Security Standards/admission controllers control what a workload is allowed to declare in its spec, and image scanning controls what’s actually inside the thing being run — these are four genuinely independent layers, and a mature security posture needs all four; strength in one doesn’t compensate for a gap in another (e.g., perfect RBAC means nothing if any pod can declare privileged: true and break out to the host).