Apache Kafka

Apache Kafka — Topics, Partitions, and Architecture

What a Topic Actually Is

A topic is a named, append-only log — a category to which records are published. Critically, Kafka topics are partitioned: a topic isn’t one log, it’s a set of ordered logs (partitions) spread across brokers. Order is only guaranteed within a partition, never across the whole topic. This single fact drives most Kafka design decisions architects need to get right.

Topic: order-events
 ├── Partition 0: [msg0][msg1][msg4][msg7]...
 ├── Partition 1: [msg2][msg5][msg8]...
 └── Partition 2: [msg3][msg6][msg9]...

Partitions — The Real Unit of Parallelism and Ordering

  • Partition count = max consumer parallelism. If a topic has 6 partitions, at most 6 consumers in one consumer group can actively read in parallel — extra consumers sit idle.
  • Partition key determines ordering guarantees. Producing with a key (e.g., orderId) routes all records for that key to the same partition via hash(key) % numPartitions, guaranteeing per-key ordering — essential for event-sourced aggregates (you don’t want OrderCreated processed after OrderShipped).
  • Choosing partition count is a one-way door (mostly). You can increase partitions later, but it breaks key-based ordering for existing keys (the hash mapping changes), and you can’t decrease partition count at all. Architects size partitions upfront based on target throughput ÷ per-partition throughput (roughly 10MB/s/partition is a common planning baseline, validate per workload) and expected consumer parallelism — not on a guess.

Replication — Durability and Availability

Each partition has:

  • One leader — handles all reads/writes for that partition.
  • N-1 followers — replicate the leader’s log.
Partition 0:  Leader (Broker 1) → Follower (Broker 2) → Follower (Broker 3)
  • replication.factor (topic-level): typically 3 in production — tolerates one broker failure with no data loss when paired with correct ack settings.
  • min.insync.replicas (min.isr): the minimum replicas that must acknowledge a write for it to be considered committed. With replication.factor=3 and min.insync.replicas=2, you can lose one broker and keep accepting writes safely.
  • Producer acks setting is what ties this together:
    • acks=0 — fire and forget, no durability guarantee.
    • acks=1 — leader acknowledges only (can lose data if leader fails before replication).
    • acks=all (-1) — waits for all in-sync replicas — the only setting that, combined with min.insync.replicas=2+, gives real durability guarantees.

Architect rule of thumb for critical data: replication.factor=3, min.insync.replicas=2, acks=all.

Retention and Cleanup Policy

Two cleanup strategies, chosen per topic via cleanup.policy:

Policy Behavior Use case
delete (default) Deletes records older than retention.ms or beyond retention.bytes Event streams, logs, telemetry
compact Keeps only the latest record per key, deletes older versions “Latest state” topics — current account balance, current order status, change-data-capture (CDC) feeds

Compacted topics are an underused architectural tool. They let Kafka double as a durable key-value store — useful for materializing current state without a separate database, or for feeding a “latest snapshot” into a downstream cache.

Consumer Groups — How Partitions Map to Consumption

Topic: order-events (3 partitions)
Consumer Group: order-processor
 ├── Consumer A → Partition 0
 ├── Consumer B → Partition 1
 └── Consumer C → Partition 2
  • Each partition is consumed by exactly one consumer within a given group at a time — this is what gives you parallel, ordered-per-key processing without explicit locking.
  • Multiple consumer groups can independently read the same topic at their own pace — this is the foundation of Kafka’s pub/sub fan-out (e.g., order-events feeding both a billing-service group and an analytics-service group, independently).
  • Rebalancing happens when group membership changes (consumer joins/leaves/crashes) — partitions get reassigned. Frequent rebalances (“rebalance storms”) are a common production pain point, usually from overly aggressive session.timeout.ms/max.poll.interval.ms settings or slow consumer processing — a topic an architect should know to watch for in incident postmortems.

Topic Design Patterns Architects Actually Use

  1. Entity-per-topic with key-based partitioning — order-events keyed by orderId. Standard event-sourcing pattern.
  2. Compacted “state” topics — customer-current-state keyed by customerId, cleanup.policy=compact, feeding a KTable in Kafka Streams or a read-side cache.
  3. Dead-letter topics — order-events-dlt for records that fail processing after retries, so the main topic isn’t blocked by poison messages.
  4. Topic-per-event-type vs topic-per-aggregate — a real architectural tradeoff:
    • Per-event-type (order-created, order-shipped): simpler schemas, but loses cross-event ordering for the same entity unless you partition all of them by the same key.
    • Per-aggregate (order-events containing all order lifecycle events as a tagged union): preserves full ordering per entity, common in event-sourced systems, requires schema evolution discipline (see below).
  5. Internal vs external topics — naming conventions like team.domain.event-name.v1 matter at scale; version in the topic name (or in the schema via a registry) so consumers can migrate without a hard cutover.

Schema Management — The Part Teams Get Wrong

Kafka itself stores bytes; it has no schema awareness. Production systems pair topics with a Schema Registry (Confluent Schema Registry, or Apicurio) using Avro, Protobuf, or JSON Schema, enforcing backward/forward compatibility rules on schema evolution. Without this, a producer deploy can silently break every downstream consumer — this is the Kafka analog of the contract-testing problem discussed earlier: the schema registry is effectively your compatibility contract for the streaming layer.

Quick Architect Checklist When Designing a New Topic

  • Partition count: based on target throughput and consumer parallelism, not a round number
  • Key strategy: does ordering matter? If yes, what’s the partition key?
  • Replication factor + min.insync.replicas + acks: matched to the durability requirement
  • Retention: delete vs compact, and how long
  • Schema: registered, with explicit compatibility mode (BACKWARD/FORWARD/FULL)
  • Dead-letter strategy: what happens to messages that fail processing
  • Naming convention: consistent across the org, versioned