Kubernetes Networking

Kubernetes Networking — Architect-Level Deep Dive

1. Cluster Networking Model

Kubernetes networking rests on four foundational rules that every implementation must satisfy:

  • Every Pod gets its own IP address — no NAT between pods, no port mapping gymnastics.
  • Pods on any node can communicate with pods on any other node without NAT.
  • Agents on a node (kubelet, system daemons) can communicate with all pods on that node.
  • Containers within a Pod share network namespace — they see each other on localhost, differentiated only by port.

This is the flat network model. Unlike Docker’s default bridge networking (which requires explicit port mapping and NAT), Kubernetes assumes a single flat L3 space where pod IPs are routable (or at least reachable via overlay) cluster-wide.

Pod networking mechanics:

  • Each pod gets a network namespace, attached to a virtual ethernet pair (veth), bridged to a node-local bridge (e.g., cbr0 or cni0).
  • The pause container (infra container) owns the network namespace; all other containers in the pod join it via --net=container:<pause_id>.
  • Pod IPs come from a per-node CIDR block carved out of the cluster’s pod CIDR (e.g., cluster 10.244.0.0/16 → node gets 10.244.1.0/24). The kube-controller-manager’s node-cidr-mask-size flag controls this allocation.

The architect’s mental model: think of three separate IP address spaces you must reconcile —

  1. Node network (the underlying VPC/subnet — your AWS VPC CIDR)
  2. Pod network (CNI-managed, e.g., 10.244.0.0/16)
  3. Service network (ClusterIP range, virtual, e.g., 10.96.0.0/12)

In AWS EKS specifically, the default CNI (amazon-vpc-cni-k8s) breaks the “overlay” pattern — pod IPs come directly from VPC subnet ENI secondary IPs, meaning pods are first-class VPC citizens, directly routable, but you’ll hit IP exhaustion fast on smaller subnets (a classic interview/real-world gotcha — m5.large supports only 3 ENIs × 10 IPs). This is why prefix delegation mode (ENABLE_PREFIX_DELEGATION=true) matters at scale — it assigns /28 prefixes instead of individual IPs per ENI slot, massively increasing pod density per node.

2. CNI Plugins (Calico, Flannel, Cilium)

The Container Network Interface (CNI) is a spec, not an implementation — kubelet calls a CNI plugin binary at pod create/delete time, passing it network namespace info via stdin/env vars. The plugin’s job: allocate an IP, wire up the veth pair, and (optionally) program routing/policy.

Flannel — simplicity-first overlay network.

  • Default backend: VXLAN (encapsulates pod traffic in UDP packets, decoupling pod network from underlying infra — works even when nodes aren’t on the same L2/L3 domain).
  • No native NetworkPolicy support — Flannel is pure connectivity, no policy enforcement layer. You’d need to pair it with Calico (Canal) for policy.
  • Lowest operational overhead, lowest feature ceiling. Good fit for dev/test clusters or when you genuinely don’t need micro-segmentation.

Calico — the “enterprise default” for years.

  • Uses BGP for route distribution in non-overlay mode — each node becomes a BGP peer, advertising its pod CIDR; no encapsulation overhead, near bare-metal performance.
  • Can fall back to IP-in-IP or VXLAN encapsulation when BGP isn’t viable (e.g., crossing L3 boundaries without BGP support, or in cloud VPCs that block arbitrary routing).
  • NetworkPolicy is first-class and predates the upstream K8s NetworkPolicy API — Calico’s own GlobalNetworkPolicy CRD gives you cluster-wide policies, not just namespace-scoped ones, plus layer 7 policy via Calico Enterprise/eBPF dataplane integration.
  • Enforcement happens via iptables (legacy) or eBPF dataplane (modern, faster, bypasses kube-proxy entirely if configured that way).

Cilium — the eBPF-native, architecturally distinct choice.

  • Doesn’t rely on iptables for forwarding or policy at all (in eBPF mode) — programs are attached at the XDP/TC hook points in the kernel, giving line-rate performance and avoiding the iptables rule-explosion problem that plagues large clusters (thousands of Services × thousands of pods = iptables chains in the tens of thousands, with O(n) lookup cost).
  • Identity-based security model: instead of IP-based policy matching, Cilium assigns each pod a security identity derived from labels, and policy decisions reference identities — this survives pod IP churn gracefully and scales better for policy evaluation.
  • Hubble gives you flow-level observability (who talked to whom, was it allowed/denied, L7 visibility for HTTP/gRPC/Kafka) — this is the feature that tips most architects toward Cilium for production observability requirements.
  • Can fully replace kube-proxy (kubeProxyReplacement=true), handling Service load-balancing directly in eBPF — meaningfully better performance at scale (avoids iptables NAT chain traversal per packet).
  • Higher operational complexity, kernel version sensitivity (older kernels lack eBPF features you’d want), steepest learning curve of the three.

Architect’s decision framework:

Need Reach for
Dead simple, dev/test, no policy needs Flannel
BGP-native, mature NetworkPolicy, hybrid on-prem/cloud Calico
eBPF performance, L7 visibility, kube-proxy replacement, deep observability Cilium
AWS EKS, VPC-native IP management aws-vpc-cni (often paired with Calico/Cilium for policy only, since vpc-cni itself doesn’t enforce NetworkPolicy by default)

3. Services: ClusterIP, NodePort, LoadBalancer

Services solve the problem that pod IPs are ephemeral — a Service gives you a stable virtual IP (VIP) and DNS name fronting a dynamic set of pod endpoints, tracked via Endpoints/EndpointSlice objects (label selector match).

ClusterIP (default):

  • Virtual IP from the Service CIDR, only routable within the cluster. There’s no actual “listener” on that IP anywhere — it’s a routing fiction implemented by kube-proxy.
  • kube-proxy historically implements this via iptables DNAT rules (each Service → chain of rules → random/weighted selection among endpoint pods) or, in newer setups, IPVS mode (better performance at high Service-count, uses kernel IPVS load balancer instead of linear iptables chain traversal — O(1) hash lookup vs O(n)).
  • This is the layer that headless Services (clusterIP: None) bypass — no VIP at all, DNS returns the actual pod IPs directly (critical for StatefulSets where clients need to address specific pod identities, e.g., Kafka brokers, Cassandra nodes).

NodePort:

  • Superset of ClusterIP — additionally opens a port (default range 30000-32767) on every node’s IP, regardless of whether that node runs a matching pod. Traffic hitting <any-node-ip>:<nodeport> gets forwarded (possibly to a pod on a different node, which means an extra network hop unless externalTrafficPolicy: Local is set).
  • externalTrafficPolicy: Local preserves source IP and avoids the extra hop, but creates uneven load distribution if pod replicas aren’t evenly spread across nodes — a frequent real-world gotcha worth knowing for architect interviews.

LoadBalancer:

  • Superset of NodePort — additionally provisions an external cloud load balancer (ELB/ALB/NLB in AWS) via the cloud-controller-manager, which then targets the NodePort across all nodes.
  • In AWS specifically: Classic ELB (legacy, L4 only) vs NLB (service.beta.kubernetes.io/aws-load-balancer-type: nlb, preserves source IP, handles millions of connections, L4) vs using ALB via Ingress instead (L7, path/host routing, but technically a different controller path — the AWS Load Balancer Controller intercepting Ingress objects, not the in-tree LoadBalancer Service type).
  • Architect note worth flagging in interviews: provisioning one LoadBalancer Service per microservice gets expensive and operationally messy at scale (one ELB/NLB per service = cost + IP sprawl). This is the core justification for Ingress as a Service-multiplexing layer.

4. Ingress & Ingress Controllers

Ingress is an API object describing L7 HTTP/HTTPS routing rules — host-based and path-based routing into one or more backend Services — but the Ingress object itself does nothing without an Ingress Controller actually watching and implementing it. This split (spec vs implementation) is a frequent point of confusion worth being crisp about.

Common controllers:

  • NGINX Ingress Controller — the most widely deployed, runs as a pod (often DaemonSet or Deployment behind a single LoadBalancer Service), reads Ingress objects, generates nginx.conf dynamically, reloads.
  • AWS Load Balancer Controller — translates Ingress objects into actual ALB resources (target groups, listener rules) — this is “Ingress” implemented as native AWS infrastructure rather than a pod-based proxy. Critical distinction for AWS-centric architecture: you’re provisioning real ALBs, not routing through an nginx pod.
  • Traefik, HAProxy Ingress, Contour (Envoy-based), Istio Gateway — increasingly, Gateway API is superseding Ingress as the modern, more expressive spec (separates infrastructure/platform concerns from application routing concerns via GatewayClass/Gateway/HTTPRoute — worth mentioning in any 2026-era architect conversation as the direction the ecosystem is moving).

TLS termination: Ingress supports a tls block referencing a Secret (cert + key) — termination typically happens at the controller/load balancer, with optional re-encryption to backend (TLS passthrough is controller-specific, e.g., NGINX ssl-passthrough annotation, or ALB’s TLS pass-through limitations that often push people toward NLB+cert-manager+app-level TLS for true end-to-end encryption requirements).

Annotations are the escape hatch — since the Ingress spec itself is minimal, controller-specific behavior (rewrite rules, rate limiting, canary weighting, sticky sessions, CORS) is configured almost entirely through annotations, which is also the spec’s biggest portability weakness — annotations are not portable across controllers. This is precisely the gap Gateway API is designed to close.

5. DNS in Kubernetes

CoreDNS (replaced kube-dns as default since 1.13) runs as a Deployment in kube-system, exposed via a ClusterIP Service (kube-dns by convention), referenced by every pod’s /etc/resolv.conf (injected by kubelet based on the cluster’s --cluster-dns flag).

Naming convention (the part everyone gets quizzed on):

<service-name>.<namespace>.svc.cluster.local
  • Same-namespace pods can just use <service-name>.
  • Cross-namespace requires at least <service-name>.<namespace>.
  • For headless Services, DNS returns A/AAAA records for each pod (not a single VIP) — and for StatefulSet pods specifically, each gets its own stable DNS name: <pod-name>.<service-name>.<namespace>.svc.cluster.local — this is the mechanism that makes ordered, addressable StatefulSet members possible (e.g., kafka-0.kafka-headless.default.svc.cluster.local).

Pod DNS policy (dnsPolicy field) controls resolution behavior:

  • ClusterFirst (default) — queries go to CoreDNS first, with non-cluster-domain queries forwarded upstream.
  • Default — inherits node’s resolv.conf (bypasses cluster DNS entirely — used for performance-sensitive workloads avoiding the CoreDNS hop).
  • None + custom dnsConfig — full manual control, used when integrating with external DNS systems (Route53 private zones, on-prem DNS for hybrid clusters).

Operational gotcha worth flagging architecturally: CoreDNS becomes a real bottleneck/SPOF at scale if under-provisioned — classic symptom is intermittent DNS resolution failures under load, usually traced to CoreDNS pod CPU throttling or ndots:5 in resolv.conf causing unnecessary extra DNS queries (every non-FQDN lookup tries 5 search-domain permutations before succeeding) — a known performance trap for chatty microservices making many short-lived DNS lookups. NodeLocal DNSCache (a DaemonSet caching resolver on each node) is the standard mitigation.

6. Network Policies

NetworkPolicy is Kubernetes’ native (if minimal) micro-segmentation primitive — a namespaced resource that, by label selector, defines allowed ingress/egress traffic for a set of pods. Critically: with no NetworkPolicy applied, all traffic is allowed (default-allow). The moment any policy selects a pod, that pod’s unspecified traffic direction becomes default-deny for that direction.

Core building blocks:

  • podSelector — which pods this policy applies to.
  • policyTypes — Ingress, Egress, or both.
  • ingress[].from / egress[].to — allowed peers, expressed via podSelector, namespaceSelector, or ipBlock (CIDR-based, for external traffic).
  • Ports — protocol + port restriction per rule.

A foundational architect pattern — default-deny-all, then allow explicitly:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Followed by targeted allow rules per service (e.g., “frontend can talk to backend on 8080,” “backend can talk to DB on 5432,” “only egress to DNS + specific external CIDR allowed”).

Critical limitation to call out: the upstream NetworkPolicy API is L3/L4 only — no HTTP method/path matching, no L7 awareness. If you need L7 policy (e.g., “allow GET but not DELETE on this path”), you need Calico’s L7 policy via Envoy sidecar, Cilium’s toFQDNs/toPorts.rules.http, or a service mesh (Istio AuthorizationPolicy). This is a very common “gap between what people assume K8s NetworkPolicy does and what it actually does” interview/design-review trap.

Another subtlety worth knowing: NetworkPolicy enforcement is entirely dependent on the CNI plugin implementing it — the API object can exist with zero effect if your CNI (e.g., plain Flannel, or aws-vpc-cni without an add-on) doesn’t enforce policy. This is precisely why production EKS clusters using vpc-cni for IP management often layer Calico for policy enforcement only (Calico for policy, vpc-cni for networking is a well-known combined deployment pattern), or move to Cilium for a single unified dataplane.