Table of Contents

Observability & Monitoring

Spring Boot Actuator

Production-readiness toolkit bundled into every Spring Boot app — exposes operational endpoints (health, metrics, env, beans, mappings, thread dumps) over HTTP without writing any custom monitoring code yourself.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health, info, metrics, prometheus, env, mappings
        # NEVER use 'include: *' in production — see security note below
  endpoint:
    health:
      show-details: when-authorized
      probes:
        enabled: true   # enables /actuator/health/liveness and /actuator/health/readiness
  info:
    env:
      enabled: true

Key endpoints architects actually reach for:

Endpoint	Purpose
`/actuator/health`	Aggregate health status — composes individual `HealthIndicator`s (DB, Redis, disk space, custom checks)
`/actuator/health/liveness`	“Is the process alive” — K8s liveness probe target; failing this triggers a pod restart
`/actuator/health/readiness`	“Is the app ready to receive traffic” — K8s readiness probe target; failing this removes the pod from the Service’s endpoint list without restarting it
`/actuator/metrics`	Raw metric browsing (also feeds Prometheus, see below)
`/actuator/env`	Resolved property sources and active values — invaluable for “why did it pick up the wrong config” debugging
`/actuator/mappings`	Every registered `@RequestMapping` — confirms a route is actually wired as expected
`/actuator/threaddump` / `/actuator/heapdump`	Live diagnostics for stuck-thread or memory investigations
`/actuator/loggers`	View/change log levels at runtime, no redeploy — extremely useful for temporarily elevating a noisy module to DEBUG in production during an incident

Custom health indicators — composing your own checks into the aggregate /health response:

@Component
public class DownstreamServiceHealthIndicator implements HealthIndicator {

    private final InventoryClient inventoryClient;

    @Override
    public Health health() {
        try {
            inventoryClient.ping();
            return Health.up().withDetail("inventory-service", "reachable").build();
        } catch (Exception e) {
            return Health.down(e).withDetail("inventory-service", "unreachable").build();
        }
    }
}

Liveness vs Readiness — a frequent K8s-integration interview question, and one teams get wrong in practice:

	Failing means	K8s response
Liveness	The JVM/process is deadlocked or unrecoverable	Restart the pod
Readiness	The app is alive but temporarily unable to serve (DB connection pool exhausted, downstream dependency briefly down, still warming up cache)	Stop sending traffic, but don’t restart — let it recover on its own

Architect-level pitfall — wiring downstream dependency checks into liveness instead of readiness: if a HealthIndicator checking an external dependency (Redis, a downstream API) is included in the liveness group rather than readiness, a transient outage in that dependency causes Kubernetes to restart the pod repeatedly even though the application process itself is perfectly healthy — restarting doesn’t fix an external outage and just adds churn/instability on top of it. Spring Boot lets you explicitly assign indicators to groups for exactly this reason:

management:
  endpoint:
    health:
      group:
        readiness:
          include: readinessState, downstreamServiceHealthIndicator
        liveness:
          include: livenessState   # deliberately excludes external dependency checks

Security note — Actuator endpoints leak operational detail and must be locked down: env, heapdump, threaddump, and detailed health output can expose secrets, internal topology, and memory contents — exposing all endpoints unauthenticated (include: * with no security config) is a documented, real-world vulnerability class. Standard practice: restrict sensitive endpoints behind Spring Security (/actuator/** requiring ROLE_ADMIN or a separate management port entirely), and expose only health/info/prometheus publicly if anything at all.

Metrics (Micrometer)

Micrometer is the metrics-facade layer Spring Boot Actuator builds on — a vendor-neutral instrumentation API (similar philosophy to SLF4J for logging) that can export to Prometheus, Datadog, New Relic, CloudWatch, or others without changing instrumentation code, just the registry/exporter dependency.

Auto-instrumented out of the box once Actuator + Micrometer registry dependency are present: HTTP request latency/count per endpoint, JVM memory/GC/thread metrics, DataSource/HikariCP pool metrics, Tomcat thread pool utilization, Kafka consumer lag (with the right binder) — meaning a meaningful baseline of observability exists with zero custom code, before any application-specific metric is even added.

@RestController
@RequiredArgsConstructor
public class OrderController {

    private final OrderService orderService;
    private final MeterRegistry meterRegistry;

    @PostMapping("/api/v1/orders")
    public ResponseEntity<OrderResponse> create(@Valid @RequestBody OrderRequest request) {
        Timer.Sample sample = Timer.start(meterRegistry);
        try {
            OrderResponse response = orderService.create(request);
            meterRegistry.counter("orders.created", "channel", "web").increment();
            return ResponseEntity.status(HttpStatus.CREATED).body(response);
        } finally {
            sample.stop(meterRegistry.timer("orders.creation.duration"));
        }
    }
}

The four core metric types — precise definitions worth having crisp:

Type	Use for	Example
Counter	Monotonically increasing count of events	`orders.created`, `payment.failures`
Gauge	A current point-in-time value that can go up or down	active sessions, current queue depth
Timer	Duration + count of timed events, with automatic percentile/histogram support	request latency, DB query time
DistributionSummary	Distribution of a non-time value (size, batch count)	payload size in bytes, items-per-order

Tags — the dimension that makes Micrometer genuinely powerful in production, not just a count of events:

meterRegistry.counter("orders.created", "channel", "web", "region", "eu-west").increment();

Tags let you slice the same metric name across dimensions in Grafana later (orders by channel, by region, by status) without defining a separate metric per combination — but cardinality discipline matters architecturally: tagging with something high-cardinality and unbounded (a raw customerId, a orderId) explodes the number of unique time-series Prometheus has to store and query, a well-documented operational failure mode (“cardinality explosion”) that can degrade or crash a Prometheus instance. Tag dimensions should be bounded, enumerable categories (status, region, channel) — never raw IDs.

@Timed declarative alternative for simple cases:

@Timed(value = "orders.create.duration", description = "Time to create an order")
@PostMapping("/api/v1/orders")
public ResponseEntity<OrderResponse> create(@Valid @RequestBody OrderRequest request) { ... }

Architect-level point — metrics vs logs vs traces, knowing what each is for: metrics answer “how much/how often/how fast, aggregated over time” (is error rate trending up); logs answer “what exactly happened in this one specific case” (why did this request fail); traces answer “where in a multi-service call chain did the time go” (covered earlier under distributed tracing). They’re complementary, not interchangeable — a common architecture-review correction is over-relying on logs for things that should be metrics (counting error occurrences by grepping logs instead of incrementing a counter) or vice versa, which scales poorly and is the wrong tool for trend/alerting use cases.

Prometheus & Grafana Integration

Prometheus scrapes (pulls) metrics from your application on a schedule — the inverse of most logging/tracing systems, which push. Spring Boot exposes a Prometheus-formatted endpoint that Prometheus polls directly.

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health, prometheus
  metrics:
    tags:
      application: order-service  # global tag on every metric — essential once multiple services share one Prometheus instance
    distribution:
      percentiles-histogram:
        http.server.requests: true  # enables proper histogram buckets for percentile queries in Prometheus/Grafana

This exposes /actuator/prometheus in the scrape-friendly text format Prometheus expects.

# prometheus.yml — Prometheus server config, scraping the app
scrape_configs:
  - job_name: 'order-service'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    static_configs:
      - targets: ['order-service:8080']
    # in K8s, typically discovered dynamically via kubernetes_sd_configs rather than static targets

Grafana then queries Prometheus (PromQL) to build dashboards — Prometheus is the storage/query engine, Grafana is the visualization layer; they’re a standard pairing but Grafana can equally visualize other data sources (Loki for logs, Tempo for traces) in the same dashboard, which is part of why this combination became the de facto open-source observability stack.

A few PromQL patterns worth knowing cold for an architect interview, since they come up directly when discussing SLOs/alerting:

# request rate over the last 5 minutes, per endpoint
rate(http_server_requests_seconds_count[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_server_requests_seconds_bucket[5m]))

# error rate as a percentage of total requests
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m])) * 100

Architect-level point — rate() vs raw counter values, a frequent point of confusion for teams new to Prometheus: Prometheus counters are cumulative since process start (they only go up, and reset to 0 on restart) — querying the raw counter value directly is rarely useful for dashboards/alerts; rate() (or irate() for faster-changing series) computes the per-second rate of increase over a window, which is what actually represents “current request rate” or “current error rate” in a way that’s meaningful and survives counter resets gracefully.

Alerting — where this ties back to operational maturity, not just dashboards:

# Prometheus alerting rule
groups:
  - name: order-service-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{status=~"5..", application="order-service"}[5m]))
          / sum(rate(http_server_requests_seconds_count{application="order-service"}[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Order service error rate above 5% for 5 minutes"

Architect-level positioning, tying the whole observability stack together for a system-design answer: Actuator exposes the raw operational surface (health, metrics endpoints), Micrometer is the vendor-neutral instrumentation layer translating application code into metric data, Prometheus is the time-series storage and query engine that scrapes and retains that data, Grafana visualizes it, and Alertmanager (paired with Prometheus) handles routing alerts to PagerDuty/Slack/etc. once a PromQL-defined threshold is breached. Being able to walk through this full chain — not just name-drop the tools — is usually exactly what’s being assessed when “how would you monitor a production microservice” comes up at the architect level, since it demonstrates the pipeline rather than isolated tool familiarity.