Kubernetes Advanced Topics

Kubernetes Architect Prep — Advanced Topics Module

1. Custom Resource Definitions (CRDs)

Core concept: A CRD extends the Kubernetes API by registering a new resource type (group/version/kind) with the API server, without modifying Kubernetes source code. Once registered, instances of that type (“Custom Resources” or CRs) are stored in etcd and managed through kubectl, RBAC, and the standard API machinery exactly like built-in resources (Pods, Deployments).

Why it matters architecturally: CRDs are the foundation of the “Kubernetes as a platform” pattern — they let you model domain-specific abstractions (a Database, a KafkaTopic, a CertificateRequest) declaratively, then drive reconciliation through controllers/Operators.

Anatomy of a CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.example.com
spec:
  group: platform.example.com
  scope: Namespaced          # or Cluster
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames: [db]
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine: { type: string, enum: ["postgres","mysql"] }
                replicas: { type: integer, minimum: 1 }
              required: ["engine"]
          required: ["spec"]
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine

Key architectural details:

Aspect Detail
Validation OpenAPI v3 schema (structural schemas mandatory since v1). Use XValidation (CEL expressions) for cross-field validation since 1.25+
Versioning Multiple versions[] with exactly one storage: true; conversion webhooks handle version translation when schemas diverge
Status subresource Separates spec (desired) from status (observed) updates — prevents controllers and users from racing on the same object via optimistic concurrency (resourceVersion)
Scale subresource /scale endpoint lets HPA target CRs directly without bespoke logic
Defaulting/Pruning Default values declared in schema; unknown fields pruned unless x-kubernetes-preserve-unknown-fields: true
Storage CRDs are stored as generic Unstructured objects in etcd unless you build a typed API (aggregated API server)

CRD vs Aggregated API Server:

CRD Aggregated API Server
Implementation effort Low — declarative YAML High — implement full API server
Storage etcd (via main API server) Pluggable, can use own datastore
Performance Good for moderate scale Better for high-throughput, custom storage needs
Use case 95% of platform extension needs Specialized needs (e.g., metrics API, custom auth)

Interview-grade Q&A:

  • Q: How does the API server validate a CR without compiled Go types? — It uses the structural OpenAPI schema embedded in the CRD itself, validated via reflection/dynamic typing, not compiled structs.
  • Q: What happens during a CRD version migration? — A conversion webhook (strategy: Webhook) is invoked by the API server to convert between storage version and served versions on read/write; None strategy assumes identical schemas.
  • Q: Why have a separate status subresource? — It gets its own RBAC verb and prevents clients updating spec from clobbering controller-written status (and vice versa) under concurrent writes.

2. Operators

Core concept: An Operator = CRD + Controller implementing the reconciliation control loop to encode operational knowledge (deployment, scaling, backup, failover) for a specific application, turning a human SRE runbook into code.

Control loop pattern (the core of all of Kubernetes):

   ┌─────────────┐     watch      ┌──────────────┐
   │  API Server │ ─────────────► │  Controller  │
   │   (etcd)     │ ◄───────────── │  Reconcile() │
   └─────────────┘   write status └──────────────┘
          ▲                              │
          │        desired state         │
          └────────── compare ───────────┘

Reconcile loop pseudocode (controller-runtime / Go, the de facto standard via Kubebuilder/Operator SDK):

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var db platformv1.Database
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    desired := buildStatefulSet(&db)
    var existing appsv1.StatefulSet
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), &existing)
    if errors.IsNotFound(err) {
        return ctrl.Result{}, r.Create(ctx, desired)
    }
    if !reflect.DeepEqual(existing.Spec, desired.Spec) {
        existing.Spec = desired.Spec
        return ctrl.Result{}, r.Update(ctx, &existing)
    }

    db.Status.Phase = "Ready"
    return ctrl.Result{RequeueAfter: 30 * time.Second}, r.Status().Update(ctx, &db)
}

Key design principles architects are expected to know:

Principle Explanation
Level-based, not edge-based Reconcile compares full desired vs actual state every time — never assumes it knows what “changed.” Makes it self-healing and idempotent.
Idempotency Reconcile() can be called any number of times for the same state with no side effects beyond convergence.
Eventual consistency RequeueAfter / exponential backoff on error, not synchronous guarantees.
Owner references Child objects (StatefulSet, Service) set ownerReferences to the CR for garbage collection cascade.
Finalizers Block deletion until external cleanup (e.g., cloud DB snapshot, deregistering from external system) completes.
Informer/cache pattern Controllers don’t poll the API server directly — they use a shared informer cache (list-watch + local indexed store) to avoid hammering etcd.

Operator Maturity Model (commonly tested):

  1. Basic Install – installs the application
  2. Seamless Upgrades – handles version upgrades
  3. Full Lifecycle – backup, restore, failover
  4. Deep Insights – metrics, alerts, log processing
  5. Auto Pilot – auto-scaling, auto-tuning, anomaly detection

Operators vs Helm:

Helm Operator
Model Templating + one-shot install/upgrade Continuous reconciliation
Day-2 ops Manual (helm upgrade) Automated, self-healing
State awareness None post-install Continuously watches actual state
Best for Stateless apps, simple config Stateful apps with complex operational logic (databases, message brokers)

Interview-grade Q&A:

  • Q: How does a controller avoid overwhelming the API server with requests? — Client-go informers maintain a local cache via list-watch; reconcile reads from cache, writes go through a rate-limited workqueue with exponential backoff.
  • Q: What’s the difference between a Controller and an Operator? — A Controller manages built-in or generic resources; an Operator is a Controller specifically encoding application-specific operational knowledge via a CRD.
  • Q: How do you prevent two controller replicas from reconciling simultaneously? — Leader election (coordination.k8s.io/Lease objects) ensures only one active reconciler; others stand by.

3. Multi-Cluster Management

Why multi-cluster: blast-radius isolation, regulatory data residency, latency-sensitive edge deployments, vendor/cloud diversification, and avoiding single control-plane scale limits (etcd practically caps around 5k nodes / 150k pods per cluster).

Architectural patterns:

Pattern Description Tooling
Cluster API (CAPI) Declarative cluster lifecycle management — clusters as CRs in a management cluster cluster-api, provider plugins (CAPA, CAPZ, CAPG)
Fleet / Hub-Spoke Central “hub” cluster pushes config/policy to “spoke” clusters Argo CD ApplicationSets, Flux, Rancher Fleet, Open Cluster Management (OCM)
Federation (KubeFed v2) Single API to propagate resources across member clusters Largely superseded by GitOps-based fleet patterns
Service Mesh Federation Cross-cluster service discovery and mTLS at L7 Istio multi-primary / primary-remote, Linkerd multicluster
Multi-cluster Ingress/Gateway Global load balancing across clusters GKE Multi-Cluster Gateway, AWS Global Accelerator + Route53 latency routing

GitOps-driven multi-cluster (most common architect-level answer today):

Git Repo (source of truth)
   │
   ▼
Argo CD (on hub cluster)
   │
   ├── ApplicationSet (cluster generator) ──► Cluster A
   ├── ApplicationSet                      ──► Cluster B
   └── ApplicationSet                      ──► Cluster C

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-service
spec:
  generators:
    - clusters:
        selector:
          matchLabels: { env: production }
  template:
    metadata:
      name: '{{name}}-payments'
    spec:
      destination:
        server: '{{server}}'
      source:
        repoURL: https://git.example.com/platform/payments
        targetRevision: main
        path: overlays/{{name}}

Cross-cluster networking decisions an architect must make:

  • Flat network (cloud VPC peering / Cilium ClusterMesh) vs gateway-mediated (Istio east-west gateway) connectivity
  • Consistent CIDR planning across clusters to avoid pod-IP overlap before any peering
  • DNS strategy: per-cluster CoreDNS + stub domains, or a global service registry
  • Failover: active-active (traffic split via global LB) vs active-passive (DR cluster, RTO/RPO driven)

Multi-cluster vs multi-tenant single cluster trade-off (frequently asked):

Driver Favors Multi-Cluster Favors Multi-Tenant Single Cluster
Blast radius / isolation ✅ ❌
Operational overhead ❌ (N control planes) ✅
Compliance/data residency ✅ ❌
Resource utilization efficiency ❌ ✅
Noisy-neighbor risk ✅ avoided ❌ requires strict quotas/NetworkPolicy

Interview-grade Q&A:

  • Q: How do you handle config drift across 50 clusters? — GitOps reconciliation (Argo CD/Flux) continuously pulls desired state from Git per cluster; drift is auto-corrected, not manually patched.
  • Q: How does Cluster API differ from a managed Kubernetes service? — CAPI manages cluster lifecycle as Kubernetes objects on a management cluster, giving a uniform declarative API across any infrastructure provider, vs each cloud’s bespoke control plane API.
  • Q: What’s your approach to cross-cluster service discovery? — Either a service mesh’s multicluster control plane (shared trust domain, ServiceEntry sync) or a global DNS/service registry pattern with per-cluster failover policies.

4. Kubernetes API Internals

The request lifecycle (the single most important diagram to know cold):

kubectl/client
     │
     ▼
┌─────────────────────────────────────────────────────────┐
│                    kube-apiserver                        │
│  1. Authentication  (certs, tokens, OIDC)                 │
│  2. Authorization   (RBAC, Node, Webhook)                 │
│  3. Admission Control                                      │
│     a. Mutating Admission Webhooks  (+ built-in mutators)  │
│     b. Object Schema Validation                            │
│     c. Validating Admission Webhooks                       │
│  4. etcd write (via watch-cache / registry layer)          │
└─────────────────────────────────────────────────────────┘
     │
     ▼
  Watch events fan out to all watchers (controllers, kubelets, schedulers)

Key internal mechanisms:

Mechanism Detail
etcd Distributed key-value store (Raft consensus); the only persistent state store in K8s. API server is the sole writer.
Watch/List pattern Clients don’t poll; they LIST once (get resourceVersion) then WATCH from that version — long-lived HTTP streams, chunked encoding
Resource Versioning Every object has a resourceVersion (maps to etcd’s mod-revision) used for optimistic concurrency — Update fails with 409 Conflict if stale
API Aggregation Layer Routes requests for registered APIService paths (e.g., metrics.k8s.io) to extension API servers behind the front-end proxy
OpenAPI/Discovery /openapi/v2, /apis endpoints let kubectl and clients discover available resources and schemas dynamically
Admission Webhooks External HTTP services called synchronously during the request; must respond fast (typically <1s, configurable timeoutSeconds); failure policy (Fail/Ignore) determines behavior on webhook unavailability
Server-Side Apply (SSA) Field-level ownership tracking (managedFields) lets multiple controllers co-own an object’s different fields without clobbering — the modern alternative to client-side 3-way merge patches
Watch bookmarks & informer relist resourceVersion too old (etcd compaction) triggers 410 Gone; client must re-LIST. Informers handle this transparently.

Why kube-scheduler and kubelet are “just clients”: Architecturally critical insight — the scheduler and kubelet have no special API access; they’re regular clients that watch/list/patch objects via the same API surface as kubectl. The scheduler watches unscheduled Pods (spec.nodeName == ""), runs filtering/scoring, then patches spec.nodeName — it doesn’t “launch” anything. The kubelet on that node then watches for Pods bound to it and starts containers via CRI.

Admission control ordering (commonly tested detail): Mutating webhooks run before validating webhooks, and built-in admission plugins are interleaved per the configured --enable-admission-plugins order — so a mutation can still pass through validation logic after modification, but never the reverse.

Interview-grade Q&A:

  • Q: What happens on a 409 Conflict? — Client sent a stale resourceVersion; it must re-GET, re-apply changes, and retry — this is the entire basis of optimistic concurrency control in K8s.
  • Q: Difference between client-side apply and server-side apply? — Client-side computes a 3-way merge patch locally and has no concept of field ownership across multiple writers; SSA delegates merge logic to the server and tracks ownership per field/manager, resolving multi-controller conflicts deterministically.
  • Q: How does the watch cache scale to thousands of watchers? — The API server maintains an in-memory ring buffer (watchCache) per resource type fed from etcd’s watch stream, multiplexing a single etcd watch to N client watches instead of N etcd connections.

5. Event-Driven Autoscaling (KEDA)

Core concept: KEDA (Kubernetes Event-Driven Autoscaling) extends HPA to scale workloads based on external event sources (queue depth, topic lag, cron, custom metrics) rather than just CPU/memory — and uniquely, can scale to and from zero, which the native HPA cannot do.

Architecture:

External Source (Kafka lag, SQS depth, Cron, Prometheus, Redis...)
        │
        ▼
┌───────────────────┐      ┌──────────────────────┐
│  KEDA Operator     │      │ Metrics Adapter        │
│  (ScaledObject      │◄────►│ (implements external.  │──► HPA (native K8s)
│   controller)        │      │  metrics.k8s.io API)   │       │
└───────────────────┘      └──────────────────────┘       ▼
        │                                              scales Deployment
        ▼ (scale 0→1)
   Activates Deployment directly (bypassing HPA, which can't scale from 0)

KEDA has two operating modes:

  1. 0 → 1 (Activation): KEDA’s own controller scales the Deployment from zero based on polling the scaler — HPA can’t do this since it requires running pods to read metrics from.
  2. 1 → N (Scaling): KEDA registers as a Kubernetes External Metrics Adapter, and the native HPA controller takes over, polling KEDA’s metrics API and scaling per standard HPA algorithm.

ScaledObject example (Kafka consumer lag scaling):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-consumer-scaler
spec:
  scaleTargetRef:
    name: order-consumer            # Deployment to scale
  minReplicaCount: 0
  maxReplicaCount: 50
  cooldownPeriod: 60                # seconds before scaling to 0 after activity stops
  pollingInterval: 15               # seconds between scaler checks
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: order-consumer-group
        topic: orders
        lagThreshold: "50"          # target avg lag per replica
        activationLagThreshold: "5" # min lag to trigger 0→1 activation
      authenticationRef:
        name: kafka-trigger-auth

ScaledJob (alternative to ScaledObject) — for batch/queue-drain workloads:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: image-processing-job
spec:
  jobTargetRef:
    template:
      spec:
        containers:
          - name: processor
            image: registry/image-processor:v1
        restartPolicy: Never
  maxReplicaCount: 100
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/xxxx/image-jobs
        queueLength: "5"

Use ScaledObject for long-running services (Deployments/StatefulSets) and ScaledJob when each unit of work should map to a discrete Job pod (better isolation, natural retry semantics, no idle pod cost between work items).

Key architectural decisions:

Decision Guidance
Scaler count per workload One ScaledObject can have multiple triggers; KEDA scales on the max computed replica count across all triggers
Activation vs scaling threshold Separate thresholds (activationLagThreshold vs lagThreshold) prevent flapping at 0↔1 boundary
Polling vs push Most scalers (Kafka, SQS, Cron) poll on pollingInterval; some (Prometheus, custom metrics) can be near-real-time depending on backend
Cooldown tuning Too short → thrashing under bursty load; too long → wasted spend holding idle replicas
Authentication TriggerAuthentication/ClusterTriggerAuthentication CRs centralize secrets (IRSA, pod identity, secret refs) — never inline credentials in ScaledObject

KEDA vs native HPA vs Cluster Autoscaler/Karpenter (frequently conflated in interviews — be precise):

HPA KEDA Cluster Autoscaler / Karpenter
Scales Pod replica count Pod replica count (incl. to/from 0) Node count
Metric source CPU/memory, or custom/external metrics API 60+ event sources (queues, streams, cron, DBs) Pending/unschedulable pods
Scale to zero ❌ ✅ N/A (scales nodes, not pods)
Relationship KEDA feeds HPA via the metrics adapter; Cluster Autoscaler reacts to the pod scaling decisions made above it by provisioning nodes

Interview-grade Q&A:

  • Q: Why can’t native HPA scale to zero but KEDA can? — HPA computes metrics by querying running pods (resource metrics) or an external metrics API — at zero replicas there’s nothing to query. KEDA’s operator runs independently of the workload, polling the event source directly, so it can detect “should be 1” before any pod exists.
  • Q: How does KEDA integrate with HPA rather than replace it? — KEDA implements the external.metrics.k8s.io API; it creates an HPA object behind the scenes once replicas > 0, and the standard Kubernetes HPA controller owns the 1→N scaling decision loop, with KEDA only supplying the metric value.
  • Q: How do you avoid scaling storms with bursty Kafka traffic? — Tune activationLagThreshold separately from lagThreshold, set a sane cooldownPeriod, and consider stabilizationWindowSeconds on the underlying HPA behavior to dampen scale-down flapping.