Kubernetes Observability

Kubernetes Observability — Architect-Level Deep Dive

1. Logging Architecture

Kubernetes itself provides no built-in log aggregation — it gives you the primitives (container stdout/stderr capture via the container runtime) and leaves aggregation, retention, and search entirely to you. Understanding the layered architecture here is foundational.

Where logs actually live before you do anything:

  • Container runtimes (containerd) write stdout/stderr to JSON log files on the node (typically /var/log/containers/*.log, symlinked from /var/log/pods/), rotated via kubelet‘s log rotation settings (containerLogMaxSize, containerLogMaxFiles).
  • kubectl logs works by the API server proxying a request to the kubelet on that node, which tails the file — this only works while the pod (or its very recent history, if using --previous) still exists on a live node. The moment a pod is evicted/deleted and its node-local log file rotates out, that log is gone forever unless something already shipped it elsewhere. This single fact is the entire justification for cluster logging architecture existing at all.

Two dominant collection patterns, and the tradeoff between them is a real architectural decision, not just a tool choice:

  • Node-level agent (DaemonSet) pattern — Fluent Bit, Fluentd, or Vector deployed as a DaemonSet, one per node, tailing every container’s log file on that node and forwarding to a central sink. This is the standard default: one agent per node regardless of pod count, low per-pod overhead, simplest to reason about. Fluent Bit specifically has become the prevailing choice over Fluentd in new builds — smaller footprint (C, not Ruby), lower memory overhead, fully capable for the standard tail-parse-enrich-forward pipeline.
  • Sidecar pattern — a logging agent container running alongside the app container in the same pod, used when an app can’t write cleanly to stdout (legacy apps writing to log files inside the container, multiple log streams needing different handling per-container). Higher resource cost (one extra container per pod, not per node) — reach for this only when the DaemonSet pattern genuinely can’t address the need, since it doesn’t scale as cleanly.

The pipeline architecture worth being able to whiteboard: Application stdout → containerd log file → Fluent Bit DaemonSet (tail + parse + enrich with K8s metadata via kubernetes filter + route) → aggregation/buffer layer (often Kafka for high-volume decoupling) → storage/index (OpenSearch, Loki, CloudWatch Logs) → query/visualization (Kibana, Grafana).

A specific AWS-centric architectural decision worth naming explicitly: CloudWatch Container Insights / Fluent Bit → CloudWatch Logs is the path-of-least-resistance for EKS (managed, minimal ops overhead, but cost scales with volume and query flexibility is weaker than purpose-built log stores), versus self-managed OpenSearch or Loki (more control, cheaper at high volume — especially Grafana Loki, which deliberately indexes only metadata/labels rather than full text, making it dramatically cheaper to run at scale than Elasticsearch-style full-text indexing, at the cost of less powerful ad-hoc full-text search). The architect’s framing in a review: pick CloudWatch for speed-to-value on smaller estates, pick Loki/OpenSearch when log volume and query sophistication needs grow past what CloudWatch’s cost model tolerates well.

Structured logging (JSON, not freeform text) is the non-negotiable foundation underneath all of this — without consistent structured fields (timestamp, level, trace_id, service name), the aggregation layer can enrich with K8s metadata (namespace, pod, node labels) but can’t meaningfully correlate across services, which is precisely the gap distributed tracing exists to close (section 4).

2. Metrics Server

metrics-server is a narrowly-scoped, lightweight cluster add-on with exactly one job: aggregate resource usage (CPU, memory) from every kubelet’s cAdvisor stats and expose it via the metrics.k8s.io Metrics API — nothing more.

What it is not, and this is the distinction architects need crisp: metrics-server is not a general-purpose monitoring system. It holds no historical data (in-memory only, most recent snapshot, no time-series storage), supports no alerting, and only covers CPU/memory — no custom application metrics, no request rates, no business KPIs.

What actually consumes it: kubectl top nodes / kubectl top pods, and critically, the HPA controller for CPU/memory-based scaling decisions (from the autoscaling module) — this is the load-bearing reason metrics-server is a near-universal cluster prerequisite even in clusters that also run full Prometheus: HPA’s resource-metric path specifically depends on the Metrics API that only metrics-server implements (Prometheus Adapter handles the custom-metrics path, not this one, unless explicitly configured to also backfill the resource metrics API, which is unusual).

Operational note worth flagging: metrics-server itself needs to scale with cluster size — at large node/pod counts, under-provisioned metrics-server (default resource requests are conservative) becomes a real bottleneck, manifesting as kubectl top timing out or HPA decisions lagging — a known “small add-on, real blast radius at scale” gotcha worth mentioning in a capacity-planning conversation.

3. Prometheus & Grafana

This pairing is the de facto standard for metrics-based monitoring and alerting in Kubernetes — distinct from metrics-server (point-in-time resource snapshot) in that Prometheus builds genuine time-series history across arbitrary metrics, application and infrastructure alike.

Prometheus’s core architectural model — pull-based, not push:

  • Prometheus scrapes HTTP /metrics endpoints on a configured interval (typically 15-30s), rather than applications pushing metrics to it. This is a deliberate design choice with real implications: Prometheus itself becomes the natural point of failure-detection (a target that stops responding to scrapes is itself a signal — “target down” — versus push-based systems where silence is ambiguous between “healthy and quiet” and “dead”).
  • Service discovery in Kubernetes is what makes this practical at scale — Prometheus’s kubernetes_sd_configs automatically discovers scrape targets via the API server (Pods, Services, Endpoints annotated appropriately), so you’re not hand-maintaining a static target list as pods churn.
  • The Prometheus Operator + ServiceMonitor/PodMonitor CRDs is the standard modern deployment pattern (via the kube-prometheus-stack Helm chart) — instead of editing Prometheus’s scrape config directly, you declare a ServiceMonitor CRD next to your Service, and the Operator reconciles it into Prometheus’s actual config. This is the K8s-native, GitOps-friendly way almost every production cluster manages Prometheus configuration now, rather than hand-editing YAML scrape configs.

Data model: time-series identified by metric name + label set (http_requests_total{method="GET", status="200"}), queried via PromQL — and the architect-relevant nuance is cardinality: every unique label-value combination creates a new time series, and high-cardinality labels (raw user IDs, full request paths with IDs embedded, unbounded label values) are the single most common cause of Prometheus falling over in production — a “cardinality explosion” that’s worth naming explicitly as a design-review red flag whenever someone proposes adding a label like user_id directly to a metric.

The four “RED”/”USE” metric categories worth referencing in any monitoring-design conversation: RED (Rate, Errors, Duration — request-centric, ideal for services) and USE (Utilization, Saturation, Errors — resource-centric, ideal for infrastructure) are the standard frameworks for deciding what to actually instrument, rather than metrics sprawl for its own sake.

Alertmanager is the companion piece handling deduplication, grouping, silencing, and routing of alerts fired by Prometheus’s alerting rules (PromQL expressions evaluated continuously, firing when thresholds are breached for a sustained for: duration) — routing to Slack/PagerDuty/Opsgenie based on label-matching rules, with inhibition rules to suppress lower-severity alerts when a related higher-severity one is already firing (avoiding alert-storm noise during a real incident).

Grafana’s role is purely visualization/dashboarding — it queries Prometheus (and can simultaneously query Loki for logs and Tempo/Jaeger for traces, making it the natural single pane of glass across all three observability pillars) and renders dashboards; it holds no data itself. Worth naming Grafana’s “Explore” correlation feature — jumping from a metric spike directly into correlated logs/traces for the same time window/service — as the practical payoff of having all three pillars in one tool, rather than each pillar living in a separate disconnected system.

Long-term storage caveat worth raising: vanilla Prometheus has local-only storage with no built-in clustering/HA or long-term retention beyond local disk capacity — production setups at scale reach for Thanos or Mimir (Grafana’s own) to get horizontally scalable, long-term, deduplicated storage across multiple Prometheus replicas — worth mentioning as the “what comes next” answer when someone asks about multi-year retention or multi-cluster federation.

4. Distributed Tracing (Jaeger, OpenTelemetry)

Tracing exists to answer the question neither logs nor metrics answer well on their own: “this specific request was slow/failed — exactly which of the dozen microservices it touched caused that, and where in the call chain?”

Core concepts to be fluent in:

  • A trace represents one end-to-end request’s journey, composed of spans — each span representing one unit of work (one service call, one DB query) with a start time, duration, and parent-child relationship to other spans, forming a tree/waterfall structure.
  • Context propagation is the mechanically critical piece: the trace ID and span ID must be passed across every network hop (typically via HTTP headers — the W3C Trace Context standard, traceparent header, is now the common interoperable format replacing older vendor-specific formats like B3) so that spans from completely different services/pods can be stitched back into one coherent trace downstream.

OpenTelemetry (OTel) is the instrumentation/collection standard, not a backend — this distinction matters architecturally:

  • OTel provides SDKs (auto-instrumentation for common frameworks/languages, reducing the need to hand-write tracing code everywhere) and the OpenTelemetry Collector (a vendor-neutral pipeline: receive → process/batch/filter → export), which can fan out the same trace/metric/log data to multiple backends simultaneously (Jaeger, Tempo, a commercial APM, etc.) without re-instrumenting application code if you ever switch backends.
  • This vendor-neutrality is precisely why OTel has become the de facto standard since roughly 2022-2023 — it decouples “how do I instrument my code” from “where does the data end up,” which earlier vendor-specific tracing SDKs didn’t.

Jaeger is a backend/storage+UI for traces (originally CNCF, originally its own client SDKs which are now deprecated in favor of just using OTel SDKs configured to export to Jaeger) — receives spans (commonly now via OTLP, OpenTelemetry’s native protocol, rather than Jaeger’s older proprietary protocol), stores them (Cassandra/Elasticsearch/or simpler storage for smaller deployments), and provides the trace-search/waterfall-visualization UI.

Grafana Tempo is worth naming as the increasingly common alternative to Jaeger in Grafana-centric stacks (same “cheap, label-indexed rather than full-text” philosophy as Loki, deliberately simple object-storage-backed design) — the practical 2026 pattern in many shops is OTel Collector → Tempo + Loki + Prometheus, all visualized in Grafana, giving genuinely correlated traces/logs/metrics in one tool with one collection pipeline, rather than three separate vendor stacks.

Sampling is the architectural decision point that always comes up at scale: tracing every single request at high request-volume is often cost/storage-prohibitive — head-based sampling (decide to sample at the start of a trace, simple but can miss rare slow/erroring requests if they’re randomly excluded) versus tail-based sampling (buffer the whole trace, decide after seeing the outcome — keep all errors and slow traces, sample normal ones lightly) — tail-based is more valuable (you never miss the interesting traces) but requires buffering full traces before the sampling decision, which the OTel Collector’s tail-sampling processor specifically exists to handle.

5. Health Checks & Probes (observability framing)

Already covered the mechanics in depth (liveness/readiness/startup, probe types, the dependency-check anti-pattern) in the autoscaling module — worth adding the observability-specific framing since this list resurfaces it here:

  • Probes are themselves a metrics/logging source, not just a control mechanism — kubelet emits events on probe failures (kubectl describe pod surfaces Liveness probe failed/Readiness probe failed events), and these should be flowing into your alerting pipeline, not just sitting invisible until someone happens to describe the pod manually. A common real-world gap: teams instrument application-level metrics carefully but never alert on probe-failure events themselves, missing an early signal that’s often available well before application metrics show degradation.
  • Prometheus’s kube-state-metrics (a companion to metrics-server, exposing cluster object state — not resource usage — as Prometheus metrics: pod phase, restart counts, readiness state, deployment replica mismatch) is the standard way probe/health state actually becomes a first-class, alertable, dashboarded metric rather than something you only discover via kubectl describe. Worth naming kube_pod_container_status_restarts_total and kube_pod_status_ready specifically as the metrics most dashboards/alerts key off for exactly this purpose.
  • The /healthz vs /readyz vs /livez convention (used by Kubernetes’ own control-plane components, and good practice for application teams to mirror) separates “is the process up” from “is it ready to serve” at the application’s own HTTP-endpoint level — designing your app to expose this distinction natively (rather than one undifferentiated /health endpoint) is what lets liveness and readiness probes actually check semantically different things instead of both hitting the same endpoint and collapsing the distinction the probe types exist to provide.

The synthesis worth stating in any architecture review: probes are the fastest-available health signal (sub-30-second detection typically) but are narrow (binary up/down, per-pod); metrics (Prometheus) give you trend and magnitude (how degraded, for how long, across how many pods); traces give you causal attribution (which exact call in the chain is the actual problem); logs give you the detailed why once you know where to look. A mature observability architecture deliberately uses each pillar for what it’s good at rather than trying to make one pillar (usually logs, since it’s the most familiar) answer questions the others are far better suited for.