Interview/Architect-Level Topics

Interview/Architect-Level Topics

Understanding the Trade-offs: CAP vs. PACELC

When designing distributed systems, you can’t have it all. Network partitions (dropped or delayed messages) are an inevitability of hardware reality. How your system behaves during these disruptions defines its architecture.

The CAP Theorem: The Standard Baseline

The CAP Theorem states that a distributed system can guarantee at most two out of three properties at the same time:

  • Consistency (C): Every read receives the most recent write or an error.

  • Availability (A): Every non-failing node returns a non-error response (without guaranteeing it contains the most recent write).

  • Partition Tolerance (P): The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network.

Because network partitions ($P$) cannot be avoided in real-world networks, systems must choose between Consistency (CP) or Availability (AP) when a partition occurs.

The PACELC Theorem: The Real-World Extension

CAP only talks about what happens during a network partition. But partitions are rare. What happens during normal operation? PACELC fixes this blind spot:

$$\text{If there is a }\mathbf{P}\text{artition, choose }\mathbf{A}\text{vailability vs }\mathbf{C}\text{onsistency}; \mathbf{E}\text{lse, choose }\mathbf{L}\text{atency vs }\mathbf{C}\text{onsistency}.$$
System Type Partition Behavior Normal Behavior Real-World Examples
PC/EC Chooses Consistency Chooses Consistency Bigtable, HBase, MongoDB (Primary-only routing)
PA/EL Chooses Availability Chooses Latency DynamoDB, Cassandra, Couchbase
PC/EL Chooses Consistency Chooses Latency Spanner (uses TrueTime to mitigate latency costs)

Consensus Algorithms in the Wild (Raft & Paxos)

Consensus algorithms allow a cluster of machines to agree on a single data value or system state, even if some nodes fail. They are the backbone of CP (Consistent/Partition-tolerant) systems.

Where They Show Up

Instead of implementing Paxos or Raft directly in every microservice, modern infrastructure relies on dedicated coordination stores:

1. etcd (Raft)

  • Role: The single source of truth for Kubernetes. It stores cluster configuration, secrets, and live state.

  • Why Raft? Kubernetes requires strict consistency. If two nodes think they both have the right to schedule a pod, chaos ensues. Raft ensures a strict, linearized log of changes.

2. Apache Kafka (KRaft / Event Logs)

  • Historically (ZooKeeper/ZAB): Kafka used Apache ZooKeeper (which runs the ZAB consensus protocol) to manage cluster metadata, leader elections for partitions, and topic configurations.

  • Modern Kafka (KRaft): Kafka replaced ZooKeeper with KRaft (Kafka Raft metadata mode). Metadata is now treated as a internal, replicated Kafka topic managed via a native Raft implementation. This removed the bottleneck of synchronizing two separate distributed systems and allowed Kafka to scale to millions of partitions.

Real-World Case Study: Designing a High-Throughput Financial Ledger

Let’s put these trade-offs into practice by designing a core system where architectural choices have literal monetary consequences.

The Challenge

Design the core Ledger Microservice for a global digital wallet.

  • Scale: 50,000 write transactions per second (TPS) peak.

  • Strict Constraint: Zero double-spending allowed. Balances must never become negative due to race conditions.

Architectural Design

1. Data Store Strategy (The PC/EC Choice)

For a financial ledger, an AP database like Cassandra is a dangerous choice. Eventual consistency could allow a user to withdraw $100 twice from two different nodes before the nodes synchronize.

  • The Choice: A PC/EC approach using a relational database with strict ACID compliance (e.g., PostgreSQL with serializable isolation) or a distributed SQL database like Google Spanner or CockroachDB.

  • The Trade-off: We sacrifice write latency. Every transaction requires a consensus round (via Raft/Paxos under the hood in CockroachDB/Spanner) to ensure data is committed across a quorum of nodes before acknowledging the user.

2. High-Availability via Raft (etcd)

To ensure the system doesn’t grind to a halt if a database primary node dies, we use a distributed consensus store like etcd to manage distributed locks and leader election for our application workers.

  • If Worker A is processing a settlement for User 123, it acquires a TTL-backed lease/lock in etcd.

  • If a network partition occurs and splits the etcd cluster, the minority side will refuse to grant locks, preventing split-brain states where two workers process the same user’s money simultaneously.

3. Asynchronous Read Side (The PA/EL Compromise)

If we forced every single query (like “View Balance History”) to hit our strictly consistent ledger database, the system would collapse under the read load.

  • The Solution (CQRS Pattern): We use Kafka (KRaft) to stream ledger updates asynchronously to a read-optimized View Database (like Elasticsearch or an AP-configured DynamoDB).

  • The Trade-off: The user profile page is PA/EL. When a user sends money, the write is immediately consistent (PC/EC), but their balance homepage might take 200ms to update (eventual consistency). This is a trade-off businesses accept: strict consistency for money movement, low-latency/eventual consistency for data visualization.