Resilience & Reliability

Resilience & Reliability

1. Circuit Breaker Pattern

The Circuit Breaker prevents a cascading failure by stopping requests to a struggling downstream service, giving it time to recover.

  • How it works: It operates like an electrical circuit breaker with three states:

    • Closed: Everything is healthy. Requests flow normally.

    • Open: Failures cross a threshold. The breaker trips, and all requests fail fast (or return a fallback) without hitting the downstream service.

    • Half-Open: After a “sleep window,” a limited number of requests are let through. If they succeed, the circuit closes; if they fail, it opens again.

  • Hystrix (Legacy): Originally built by Netflix, it used a thread-per-command model to isolate calls. It is officially deprecated and no longer maintained.

  • Resilience4j: The modern standard for Java/Spring. It’s lightweight, modular, and built around Java 8 functional programming and Vavr, using an in-memory sliding window (count-based or time-based) rather than dedicated thread pools.

2. Retry with Exponential Backoff + Jitter

When a network glitch or temporary overload causes a failure, you don’t want to give up immediately—but you also don’t want to spam the server.

  • Exponential Backoff: Instead of retrying every 1 second, you multiply the wait time exponentially (e.g., 1s, 2s, 4s, 8s…).

  • Jitter (Randomness): If a downstream service goes down and comes back up, thousands of clients retrying at the exact same exponential intervals will hit it in waves, causing a “thundering herd” problem. Adding “jitter” randomizes the backoff time (e.g., instead of exactly 4s, it waits 3.7s or 4.2s), spreading the load evenly.

Formula concept: $Wait\ Time = \min(Max\ Backoff, \text{base} \times 2^{\text{attempt}}) + \text{random\_jitter}$

3. Bulkhead Pattern

Named after the watertight compartments of a ship’s hull. If one compartment punctures, the ship doesn’t sink because the water is isolated.

  • How it works: It isolates resources (like thread pools, memory, or CPU) so that a failure in one downstream dependency doesn’t consume all system resources and bring down unrelated features.

  • Implementation Types:

    • Thread Pool Isolation: Assigns a dedicated pool of threads to a specific service. If Service A hangs, only its thread pool fills up; Service B keeps running fine.

    • Semaphore Isolation: Limits the number of concurrent requests to a service without switching threads. Shines in low-latency, high-throughput scenarios.

4. Timeout Strategies

Never rely on default network timeouts (which can sometimes be up to 60 seconds or infinite depending on the client).

  • Connection Timeout: How long to wait to establish the initial TCP handshake. (Keep this short, e.g., 200ms–500ms).

  • Read/Socket Timeout: How long to wait for the data to actually start flowing once connected.

  • Deadline/End-to-End Timeouts: In a chain of microservices ($A \rightarrow B \rightarrow C$), if Service A has a total timeout of 2 seconds, but Service B takes 1.8 seconds to talk to C, Service B should dynamically shrink its remaining timeout so it doesn’t waste time on a request that A has already abandoned.

5. Idempotency (Designing Idempotent APIs/Consumers)

An operation is idempotent if executing it multiple times yields the same system state as executing it once. This is crucial when used alongside Retries.

  • API Design (REST): GET, PUT, and DELETE are inherently idempotent by spec. POST is not.

  • The Idempotency Key Pattern: 1. The client generates a unique UUID (X-Idempotency-Key) and sends it in the header.

    2. The server checks a distributed cache (like Redis) for the key.

    3. If it exists, the server returns the cached response without reprocessing.

    4. If not, it processes the request, stores the result, and returns it.

  • Message Consumers (Kafka/RabbitMQ): Ensure database operations use UPSERT (Insert or Update) or unique constraints rather than blind INSERT statements to handle duplicate messages safely.

6. Rate Limiting & Throttling

While Circuit Breakers protect you from downstream failures, Rate Limiting protects your service from upstream overloads (or malicious actors).

  • Rate Limiting: Hard limits on requests over a time window (e.g., 100 requests per minute). Excess requests are rejected with a 429 Too Many Requests status.

  • Throttling: Dynamically degrading performance or queuing requests when a threshold is met to smooth out sudden spikes in traffic.

  • Common Algorithms:

    • Token Bucket: Allows bursts of traffic; tokens refill at a steady rate.

    • Leaky Bucket: Smooths out traffic; requests exit the bucket at a constant rate regardless of how fast they come in.

    • Fixed/Sliding Window: Counts requests within a specific time block.