Kubernetes in Cloud

Kubernetes in Cloud — Architect-Level Deep Dive

1. EKS (AWS)

Amazon EKS runs the Kubernetes control plane (API server, etcd, scheduler, controller-manager) as a fully managed AWS service, spread across multiple AZs by AWS, with the control plane itself living in an AWS-managed VPC, not the customer’s — customers only ever see the API endpoint, never the underlying control-plane infrastructure.

Networking — the single most distinctive architectural fact about EKS versus other managed offerings:

  • The default CNI, amazon-vpc-cni-k8s, assigns pods real VPC IP addresses (secondary IPs on ENIs attached to worker nodes) rather than an overlay network — pods are first-class VPC citizens, directly addressable by security groups, VPC routing, and Transit Gateway/PrivateLink without any encapsulation. This is the same point raised in the networking module’s IP-exhaustion discussion — it’s a direct consequence of this architectural choice and worth repeating here since it’s specifically an EKS characteristic, not a general Kubernetes one.
  • Security Groups for Pods — EKS uniquely lets you attach actual AWS Security Groups directly to individual pods (via SecurityGroupPolicy CRD + the VPC CNI’s security-group-for-pods feature, requiring Nitro-based instances) — a capability that doesn’t exist in this form on AKS/GKE, since it leans on the VPC-native pod IP model specifically.

Compute options, worth being fluent across all three since real architectures mix them:

  • Managed Node Groups — AWS-managed ASGs running standard EC2 worker nodes, AWS handles AMI patching/node lifecycle, you control instance types and scaling policy. The traditional default.
  • Fargate for EKS — fully serverless pods, no node management at all, AWS provisions per-pod isolated compute (microVM-backed) — trades node-level control (no DaemonSets, limited to specific networking/storage configurations) for zero node-patching/capacity-planning overhead. A common pattern: Fargate for stateless/bursty workloads, Managed Node Groups (or self-managed) for stateful/DaemonSet-dependent or GPU workloads that Fargate can’t serve.
  • Karpenter, raised in the autoscaling module — increasingly the default node-provisioning layer for new EKS builds in 2026, displacing the older Managed-Node-Group-plus-Cluster-Autoscaler pattern for its faster, more flexible, more cost-efficient binpacking.

IAM integration is the other defining EKS characteristic — IRSA (OIDC federation between the cluster and IAM) and the newer EKS Pod Identity (simpler trust setup, increasingly the recommended default) let individual pods assume scoped IAM roles directly, which is the foundation almost every other AWS-integrated capability (ALB Controller provisioning real ALBs, EBS/EFS CSI drivers provisioning real storage, External Secrets Operator reading Secrets Manager) ultimately depends on for its own permissions.

Add-ons worth naming as the standard EKS bootstrap set: VPC CNI, CoreDNS, kube-proxy (all available as EKS-managed add-ons, meaning AWS handles version compatibility/upgrade orchestration for these specifically, rather than you tracking compatibility manually), plus the AWS Load Balancer Controller (ALB/NLB provisioning from Ingress/Service objects) and EBS/EFS CSI drivers — none of these ship by default; they’re explicit installs, a frequent source of “why doesn’t my Ingress create an ALB” confusion for teams new to EKS expecting more out-of-the-box behavior.

2. AKS (Azure)

Azure Kubernetes Service follows the same “AWS manages control plane, customer manages/owns nodes” model, with a few Azure-specific architectural distinctions worth knowing precisely rather than assuming AKS is just “EKS with different branding.”

Networking — two genuinely different modes, and this binary choice matters architecturally more than the EKS CNI discussion does:

  • Kubenet (legacy, simpler) — pods get IPs from a separate, non-VNet-routable CIDR, with NAT applied for pods to reach resources outside the cluster — closer to the “overlay” model other CNIs use, IP-address-conserving but with the usual overlay limitations (less direct integration with VNet-level security/routing).
  • Azure CNI — pods get real VNet IP addresses directly (conceptually identical to the AWS VPC CNI’s model) — full VNet integration, but the same IP-exhaustion planning concern applies (subnet sizing needs to account for max pods × max nodes upfront, since unlike EKS’s prefix-delegation evolution, Azure CNI traditionally required reserving the full IP range per node upfront — Azure has since added Azure CNI Overlay and Azure CNI with dynamic IP allocation specifically to relax this constraint, worth a “check current Azure docs” caveat since this area has evolved significantly).

Identity — Azure’s equivalent of IRSA: Azure AD Workload Identity (the modern replacement for the older, now-deprecated AAD Pod Identity) federates a Kubernetes ServiceAccount with an Azure AD application via OIDC, letting pods authenticate to Azure resources (Key Vault, Storage, SQL) without embedded credentials — directly analogous in purpose and mechanics to EKS’s IRSA/Pod Identity, just Azure AD-flavored.

Distinctive AKS-specific capabilities worth naming:

  • Virtual Nodes (ACI-backed, via the virtual-kubelet project) — AKS’s rough equivalent of Fargate, bursting pods into Azure Container Instances for serverless, node-free scaling, though with a narrower set of supported scenarios than EKS Fargate.
  • AKS is free for the control plane in its Standard tier’s base offering (you pay for nodes, plus an optional Uptime SLA tier for the formal control-plane availability guarantee) — a genuine cost-model difference worth knowing versus EKS, which charges a per-cluster-hour control-plane fee regardless of tier.
  • Tight first-party integration with Azure AD for cluster RBAC — Azure AD groups can be mapped directly to Kubernetes RBAC RoleBinding/ClusterRoleBinding subjects natively, which is a smoother out-of-the-box experience than AWS’s aws-auth ConfigMap-based IAM-to-RBAC mapping historically required (AWS has since added EKS Access Entries as a more native API-based alternative to the old ConfigMap approach, closing much of this gap).

3. GKE (Google Cloud)

Google Kubernetes Engine is worth framing as the most “Kubernetes-native” of the three, unsurprising given Google’s origin role in Kubernetes itself — and it offers a genuinely distinct operational model that the other two don’t quite match.

Autopilot vs Standard — the single most important GKE-specific decision, more consequential than EKS/AKS’s Fargate/Virtual-Nodes side options:

  • Standard mode — the familiar model: you manage node pools, choose instance types, configure autoscaling — comparable to EKS Managed Node Groups/AKS node pools.
  • Autopilot mode — Google manages nodes entirely — no node pools to configure, no node-level patching, billing is per-pod resource request, not per-node — Google handles bin-packing/node-provisioning transparently underneath. This is a meaningfully more “hands-off” operating model than anything EKS/AKS offers as their default (Fargate/Virtual-Nodes are opt-in side-modes for specific workloads in those platforms; Autopilot is GKE’s fully-supported primary operating mode for entire clusters). The tradeoff: Autopilot restricts certain capabilities (privileged pods, some hostPath usage, specific DaemonSet patterns) precisely because Google needs to maintain its own node-level security/operational guarantees — worth flagging as a real constraint, not just upside, in an architecture review choosing between Standard and Autopilot.

Networking — GKE uses the VPC-native (alias IP) mode by default in current versions — conceptually the same “pods get real VPC-routable IPs” model as EKS VPC CNI and Azure CNI, via GCP’s alias IP ranges feature, with the legacy “routes-based” (overlay-ish) mode now largely deprecated for new clusters.

Distinctive GKE capabilities worth being able to name:

  • Workload Identity — Google’s version of the IRSA/Workload-Identity pattern, federating Kubernetes ServiceAccounts to GCP IAM service accounts — same architectural role as AWS IRSA/Pod Identity and Azure AD Workload Identity; worth noting all three clouds converged on essentially the same OIDC-federation pattern independently, which is a good “see, the underlying problem and solution shape is consistent across clouds” point in any multi-cloud discussion.
  • Release channels (Rapid/Regular/Stable) — a genuinely distinctive, more formalized version-management UX than EKS/AKS’s simpler version-pinning model, letting teams explicitly opt into Google’s risk-tiered Kubernetes version rollout cadence rather than manually tracking individual version support windows.
  • GKE’s control plane has historically had the deepest/earliest feature parity with upstream Kubernetes releases (unsurprising given Google’s role on the Kubernetes steering committee/release process) — worth mentioning as a soft factor when “which cloud gets new K8s features fastest” comes up, though this gap has narrowed considerably as EKS/AKS release cadences have matured.

4. Managed vs Self-Managed Clusters

This is the framing question underneath all three providers above, and worth addressing directly as its own architectural decision rather than just implicit in “which cloud.”

What “managed” (EKS/AKS/GKE) actually takes off your plate:

  • Control-plane HA, etcd backup/management, control-plane patching/upgrades, and (in EKS/AKS/GKE Autopilot) increasingly node-level patching too.
  • SLA-backed control-plane availability (with the appropriate tier — EKS’s standard SLA, AKS’s Uptime SLA add-on, GKE’s regional cluster guarantee).

What you still own even on a fully managed cluster, which is the point most often underestimated in planning:

  • Node-level patching (unless using Fargate/Autopilot/Virtual Nodes specifically) — OS CVEs, container runtime updates, kubelet version alignment with control plane (the N-2 version skew policy — kubelet can lag the API server by up to 2 minor versions, but not arbitrarily, which becomes a real constraint during multi-cluster fleet upgrades).
  • All application-layer concerns — RBAC design, Network Policy, resource governance, the entire observability stack, CI/CD, everything covered in the prior six modules — “managed Kubernetes” only ever means “managed control plane (+ optionally managed nodes),” never “managed application platform.”
  • Add-on lifecycle — CNI, CSI drivers, Ingress controllers, cert-manager, the Prometheus stack — these are your responsibility to install, upgrade, and keep compatible with the underlying K8s version, regardless of cloud (EKS-managed-add-ons soften this slightly for a specific short list, as noted above, but it’s not comprehensive).

Self-managed (kubeadm on EC2/VMs, or fully on-prem/bare-metal) — when this is a legitimate architectural choice rather than just extra toil:

  • Air-gapped/regulated environments where no managed control plane (which inherently involves some cloud-provider-operated infrastructure touching your cluster) is acceptable under compliance requirements.
  • Specific control-plane customization managed offerings don’t expose (custom admission controllers compiled into the API server, non-standard etcd topology, specific kernel/CNI combinations managed offerings don’t support).
  • Genuine multi-cloud/portability requirements where standardizing on kubeadm (or a distribution like Rancher/RKE2, OpenShift) across AWS/Azure/GCP/on-prem uniformly is valued over each cloud’s native managed convenience — worth being honest in a review that this trades meaningfully more operational burden (you now own control-plane HA, etcd backup/restore, and version upgrade orchestration entirely yourself) for that portability/control, and the bar for choosing this path deliberately should be high — “we might go multi-cloud someday” is usually not sufficient justification given the ongoing operational tax.

The architect’s honest framing for almost any net-new cloud-hosted workload in 2026: default to managed (EKS/AKS/GKE) unless there’s a specific, named constraint (compliance, control-plane customization, genuine multi-cloud commitment already underway) that managed offerings can’t satisfy — self-managed control planes are a real ongoing operational cost, not a one-time setup cost, and that cost needs to be justified against a concrete requirement, not abstract flexibility.

5. Cloud Load Balancers & Storage Integration

This is where the cloud-controller-manager and CSI drivers translate Kubernetes-native objects (Service type LoadBalancer, PersistentVolumeClaim) into real, billable cloud infrastructure — and the implementation specifics differ enough across clouds to matter in any cross-cloud or migration conversation.

Load balancer provisioning, per cloud:

LoadBalancer Service → Ingress →
EKS Classic ELB (legacy default) or NLB (aws-load-balancer-type: nlb) ALB, via AWS Load Balancer Controller
AKS Azure Load Balancer (L4) Application Gateway (via AGIC — Application Gateway Ingress Controller) or NGINX/other controller
GKE Google Cloud Network Load Balancer (L4) Google Cloud Load Balancer (L7), natively via GKE’s built-in Ingress-for-GKE controller — notably more tightly built-in than EKS/AKS’s equivalents, which generally require installing a separate controller

The recurring architectural point worth naming explicitly across all three: a LoadBalancer Service or an Ingress object is a request, not the infrastructure itself — the actual cloud load balancer is provisioned, configured, and billed for separately by the respective cloud, and one Service/Ingress typically maps to one real cloud LB resource — which is precisely why, at scale, teams consolidate behind a single Ingress/Gateway rather than provisioning a LoadBalancer Service per microservice (raised back in the networking module) — this is a real, ongoing cost-control lever in any of the three clouds, not just an AWS-specific concern.

Storage integration, per cloud — same CSI-driven dynamic-provisioning model from the storage module, different concrete backends:

Block storage (RWO) Shared filesystem (RWX)
EKS EBS CSI driver (gp3/io2) EFS CSI driver
AKS Azure Disk CSI driver Azure Files CSI driver
GKE Persistent Disk CSI driver Filestore CSI driver

The WaitForFirstConsumer binding-mode AZ-mismatch gotcha flagged in the storage module is universal across all three clouds, not an AWS-specific quirk — Azure Disks and GCP Persistent Disks are equally zone-locked, so the same StorageClass best practice applies regardless of which cloud you’re standardizing on, which is worth stating explicitly as a portable principle rather than something to re-derive per cloud.

The cross-cloud synthesis worth closing on in any architecture review: the concepts (Service/Ingress as declarative requests, CSI as the storage-provisioning abstraction, cloud-controller-manager as the translation layer) are fully portable across EKS/AKS/GKE — this is precisely Kubernetes’ value proposition as a cloud-abstraction layer. What’s not portable is the underlying cloud-specific implementation detail (which LB SKU you get, exact IAM-federation mechanics, specific CSI driver quirks/limits) — a genuinely multi-cloud-aware architecture writes Kubernetes manifests that lean on the portable abstractions (Service, Ingress/Gateway API, PVC+StorageClass) and isolates the cloud-specific configuration (StorageClass parameters, LoadBalancer annotations, IAM federation setup) into clearly separated, per-cloud overlay layers (Kustomize overlays, separate Helm values files) — rather than hand-coding cloud-specific assumptions directly into application-team-owned manifests.