Microservices Observability

Microservices Observability

1. Distributed Tracing

When a single user request triggers a cascade of calls across dozens of microservices, traditional logging falls short. Distributed tracing tracks the lifecycle of that request as it flows through your system.

  • How it works: A request enters the system and is assigned a unique Trace ID. As it moves to other services, each sub-segment of work is tracked as a Span (with its own Span ID, parent ID, and timestamps).

  • The Ecosystem:

    • OpenTelemetry (OTel): The industry standard for CNCF (Cloud Native Computing Foundation). It provides a vendor-neutral set of APIs, SDKs, and tools to generate and export telemetry data (traces, metrics, logs).

    • Jaeger / AWS X-Ray: These are the backends or storage and visualization tools. OTel collects the data, and you send it to Jaeger (open-source) or X-Ray (AWS managed) to view the beautiful Gantt-chart-style trace graphs.

2. Centralized Logging & Correlation IDs

Logs tell you what happened inside a specific service, but without context, they are just noise. Centralized logging aggregates logs from all containers and services into a single searchable pool (e.g., OpenSearch, ELK Stack, Grafana Loki).

  • The Magic Link (Correlation IDs): A Correlation ID is a unique string injected into the HTTP header or metadata of a request. Every single service that touches that request must include this ID in its log outputs.

  • The OTel Convergence: Modern architecture heavily leverages OpenTelemetry to automatically inject the Trace ID as the Correlation ID. This means when you find a failed trace in Jaeger, you can instantly query your centralized logs for that exact Trace ID and see every single log line printed by every service involved in that specific failure.

3. Metrics & SLO / SLA Monitoring

If traces and logs are for debugging after something breaks, metrics are your smoke detector. They tell you how the system is performing in real-time.

  • The GOLDEN Signals: Usually tracked via Prometheus and visualized in Grafana, you want to monitor Latency, Traffic, Errors, and Saturation (the RED or USE methods).

  • The Boundaries:

    • SLA (Service Level Agreement): The legal contract with your customers (e.g., “99.9% uptime or we owe you money”).

    • SLO (Service Level Objective): Your internal target, which should always be stricter than the SLA (e.g., “99.95% uptime”). If you breach your SLO, your team gets paged before you breach the SLA.

    • SLI (Service Level Indicator): The actual metric measurement (e.g., “Current uptime is 99.97%”).

4. Health Checks: Liveness vs. Readiness Probes

In containerized environments like Kubernetes, the orchestrator needs to know the physical state of your applications. If you mix these two probes up, you can accidentally cause a cascading failure.

Probe Type What it asks Action on Failure Typical Use Case
Liveness “Are you alive, or are you deadlocked/stuck?” Kubernetes restarts the container. Deadlocks, infinite loops, or out-of-memory hangs where the app cannot recover on its own.
Readiness “Are you ready to receive network traffic right now?” Kubernetes removes the container from the load balancer routing. App is still booting up, loading a massive cache, or losing connection to a downstream DB.

⚠️ Critical Gotcha: Never make your Liveness probe dependent on a downstream database. If your database goes down, your database will be fine, but Kubernetes will think all your app containers are dead and start violently restarting them in a loop, making the outage significantly worse. Keep liveness probes lightweight and internal to the app container.