Data Management

Data Management

1. Distributed Transactions: Why 2PC Fails at Scale

In a monolithic system, ensuring data consistency is easy: you use a database transaction (ACID). If something goes wrong, the database rolls everything back. In a microservices architecture where each service has its own database, you cannot use local DB transactions.

Historically, Two-Phase Commit (2PC) was used to achieve atomic distributed transactions. It works in two phases:

  1. Prepare Phase: A coordinator asks all participating services if they are ready to commit.

  2. Commit Phase: If everyone says “yes,” the coordinator tells everyone to commit. If anyone says “no” (or times out), everyone aborts.

Why 2PC Fails at Scale:

  • Blocking Protocol: During both phases, resources (database rows/tables) are locked. If the coordinator crashes or a network partition occurs during the second phase, those locks are held indefinitely, paralyzing your system.

  • Performance Bottleneck: The system’s latency is tied to the slowest participating service. As you add more microservices, the probability of a delay or failure increases exponentially.

  • No CAP Theorem Synergy: 2PC prioritizes Consistency over Availability. In modern cloud-native systems, we usually prefer Eventual Consistency to ensure high availability.

2. The Saga Pattern

To replace blocking 2PC transactions, we use the Saga Pattern. A Saga is a sequence of local transactions. Each local transaction updates the database within a single service. If a step fails, the Saga executes compensating transactions (explicit rollbacks) to undo the changes made by the preceding steps.

There are two primary ways to implement a Saga:

A. Choreography-Based Saga

In choreography, there is no central orchestrator. Every service listens to events and decides when to execute its local transaction.

[Order Service] ──(Order Created)──> [Payment Service] ──(Payment Approved)──> [Inventory Service]

  • Pros: Highly decoupled; simple for short workflows with few services.

  • Cons: Hard to track the current state of a complex workflow; risk of cyclic dependencies; difficult to debug.

B. Orchestration-Based Saga

In orchestration, a centralized service (the Orchestrator) acts as the brain. It explicitly tells each participant service what local transaction to execute based on the current state.

                    ┌─── 1. Authorize Payment ───> [Payment Service]
[Order Orchestrator]├─── 2. Reserve Stock ─────> [Inventory Service]
                    └─── 3. Ship Items ─────────> [Shipping Service]

  • Pros: Centralized visibility; easier to reason about complex business logic; prevents cyclic dependencies.

  • Cons: Risk of over-centralizing logic into the orchestrator (turning it into a “smart orchestrator, dumb services” monolith); additional infrastructure to manage.

3. The Outbox Pattern (Reliable Event Publishing)

When using Sagas or event-driven architectures, services frequently need to do two things atomically:

  1. Update their local database.

  2. Publish an event to a message broker (e.g., Kafka, RabbitMQ).

If the database write succeeds but the network blips before publishing the event, your system becomes inconsistent. This is known as the Dual-Write Problem.

How the Outbox Pattern Fixes This:

Instead of publishing directly to the broker, your application does both database writes inside a single local transaction:

  1. It updates the business entity table (e.g., Orders).

  2. It inserts a record describing the event into an Outbox table.

A separate, independent process (a Message Relayer or a Change Data Capture (CDC) tool like Debezium) constantly polls the Outbox table, reads the new events, publishes them to the message broker, and marks them as processed. This guarantees at-least-once delivery.

4. CQRS (Command Query Responsibility Segregation)

In traditional CRUD applications, the same data model is used to read and write data. While simple, this breaks down under high load because read and write performance requirements are vastly different.

CQRS splits your application into two separate paths:

  • Commands (Writes): Focus on business logic, validation, and data modification. They do not return data (except perhaps a status or ID).

  • Queries (Reads): Fast, read-only data fetching. The read database can be optimized specifically for UI presentation (e.g., using a denormalized view in Elasticsearch or Redis).

  • Why use it? You can scale reads and writes independently. If your application has a 100:1 read-to-write ratio, you can spin up 100 read replicas without scaling your write database.

5. Event Sourcing

Traditional databases only store the current state of an object. If a user changes their email three times, you only see the last email. You lose the history of how they got there.

Event Sourcing changes this completely. Instead of storing the current state, you store a sequence of append-only immutable events that represent every change that ever occurred to that object.

  • To find the current state of an object, you start from blank and replay all its historical events in order.

  • To optimize performance, the system periodically takes snapshots of the state so you don’t have to replay millions of events from day one.

The Perfect Match: CQRS + Event Sourcing

Event Sourcing and CQRS are frequently used together.

  1. The Command side receives a request, validates it against the current state, and appends a new event to the Event Store (Write DB).

  2. An event handler project detects this new event, updates a denormalized Read DB (like MongoDB or Postgres).

  3. The Query side simply reads straight from that fast Read DB.