How Kafka Works: Partitioned Logs, Consumer Groups, Replication, and Exactly-Once Delivery From Producer to Consumer

Most distributed messaging problems eventually expose a harder problem underneath: how do you persist a high-throughput stream of events reliably, replay it on demand, and let multiple independent consumers each track their own position without interfering? A traditional message queue deletes messages after acknowledgment. That works fine for task distribution but breaks the moment you need event replay, fan-out to heterogeneous consumers, or auditable event history. Kafka was designed to solve exactly that class of problem, and its design choices are non-obvious enough that using it without understanding the internals leads to predictable production incidents: data loss from misconfigured replication, duplicate processing from incorrect offset management, and runaway consumer lag that is impossible to reason about.

This article covers the full Kafka architecture from the log abstraction up through producers, consumers, replication, exactly-once semantics, and log retention. It ends with concrete production tuning parameters.

The Append-Only Partitioned Log

The core abstraction in Kafka is not a queue. It is a log: an append-only, ordered sequence of records stored on disk. A topic is a named log. A topic is divided into one or more partitions, each of which is an independent ordered log stored as a sequence of segment files on a single broker.

Topic: orders
  Partition 0: [offset 0][offset 1][offset 2]...[offset N]
  Partition 1: [offset 0][offset 1][offset 2]...[offset M]
  Partition 2: [offset 0][offset 1][offset 2]...[offset K]

Each record has an offset: a monotonically increasing integer within its partition. Offsets are not global across partitions. Ordering is guaranteed within a partition only. If you need strict global ordering, you must use a single partition, but that limits throughput to the capacity of one broker.

On disk, each partition is a directory of segment files. The active segment receives all writes. When a segment exceeds log.segment.bytes (default 1 GiB) or log.roll.hours, Kafka closes it and opens a new one. Closed segments are immutable. Reads can serve any segment concurrently because there are no locks on immutable data. This is why Kafka sustains high read throughput even under heavy write load.

The index file stored alongside each segment maps offsets to byte positions within the segment file. A consumer seeking to offset N does a binary search on the index, then reads from the corresponding byte position directly. Disk seeks are minimized because reads within a segment are sequential.

Kafka relies on the OS page cache rather than maintaining its own in-memory buffer pool. Writes go to the page cache and are flushed to disk asynchronously. Reads are served from the page cache when the data is still hot. This makes Kafka’s performance heavily dependent on having enough free memory for the OS to cache active partition data.

Producer Partition Selection

When a producer sends a record, it must decide which partition to write to. Kafka gives the producer full control over this decision.

Key-based partitioning: If the producer includes a message key, Kafka hashes the key (using murmur2 by default) and maps it to a partition via modulo:

const partition = Math.abs(murmur2(key)) % numPartitions;

All records with the same key always go to the same partition, which guarantees ordering within a key. This is the standard pattern for event streams keyed by entity ID (user ID, order ID, device ID).

Sticky partitioning: When a producer sends records without a key, the pre-2.4 round-robin behavior sent each record to a different partition, producing many small batches and degrading throughput. Kafka 2.4 introduced sticky partitioning: the producer accumulates records into a batch for one partition until the batch is full or the linger.ms timer fires, then switches partitions. This produces larger batches, reduces the number of requests, and improves both latency and throughput for keyless producers.

Custom partitioner: You can implement the Partitioner interface to route records based on any logic, for example, routing high-priority orders to a dedicated partition.

The producer batches records in memory before sending. Two settings control batch formation. batch.size (default 16 KiB) is the maximum batch size in bytes. linger.ms (default 0) is the maximum time to wait for additional records before sending an incomplete batch. Setting linger.ms to 5 or 10 ms dramatically improves throughput on high-volume producers by allowing more records to accumulate per batch.

Compression is applied at the batch level. snappy is a reasonable default: it compresses well and decompresses fast. lz4 is faster on both axes. zstd offers the best compression ratio at the cost of more CPU. For network-bound workloads, any compression reduces broker write and consumer read load.

Consumer Groups and Partition Assignment

A consumer group is a set of consumer instances that cooperatively read a topic. Each partition is assigned to exactly one consumer in the group at a time. This is Kafka’s unit of parallelism: the maximum useful number of consumers in a group equals the number of partitions.

Topic: orders (3 partitions)
Consumer Group: order-processor

  Consumer A: Partition 0
  Consumer B: Partition 1
  Consumer C: Partition 2

Each consumer independently tracks its position in its assigned partitions via offsets. The consumer commits an offset to Kafka’s internal __consumer_offsets topic after processing records up to that position. On restart, the consumer fetches its last committed offset and resumes from there.

The two commit strategies are:

• Auto-commit (enable.auto.commit=true): The client commits the last fetched offset on a timer (auto.commit.interval.ms, default 5 s). If the consumer crashes between the commit and processing those records, they are lost. If the consumer crashes after processing but before the commit fires, they are reprocessed. Auto-commit is at-most-once or at-least-once depending on when the crash occurs.
• Manual commit: The application calls commitSync() or commitAsync() after processing. Combined with idempotent consumers, this achieves reliable at-least-once delivery.

Rebalancing

When a consumer joins or leaves a group, or a partition count changes, the group goes through a rebalance. The group coordinator broker assigns partitions to consumers using a configured assignor. During a rebalance, all consumers in the group stop fetching. This is the classic “stop-the-world” rebalance problem.

The eager rebalance protocol (the original) revokes all partition assignments before making new ones. Every consumer drops its partitions, waits for the coordinator to compute the new assignment, and then takes its new partitions. For large groups with slow consumers, this causes a significant pause.

Kafka 2.4 introduced the cooperative rebalance protocol (also called incremental rebalance). Instead of revoking all partitions, the coordinator incrementally transfers only the partitions that need to move. Consumers that retain their partitions keep fetching during the rebalance. Only consumers gaining or losing specific partitions pause briefly. This dramatically reduces rebalance impact on processing throughput.

The session.timeout.ms and heartbeat.interval.ms settings control how quickly the coordinator detects a dead consumer. A dead consumer means its partitions are unassigned until the timeout expires and a rebalance is triggered. Setting session.timeout.ms too low causes spurious rebalances when consumers run slow GC or processing spikes. Setting it too high means dead consumers leave partitions unassigned for a long time. A common production default is session.timeout.ms=45000 with heartbeat.interval.ms=15000.

ISR-Based Replication and Leader Election

Kafka replicates each partition across multiple brokers for durability and availability. One broker is the leader for each partition. All reads and writes go to the leader. The other replicas are followers that asynchronously fetch from the leader.

The In-Sync Replica set (ISR) is the subset of replicas that are considered caught up. A replica is in the ISR if it has fetched all messages from the leader within replica.lag.time.max.ms (default 30 s). Replicas that fall behind are removed from the ISR.

The acks producer setting determines the durability guarantee:

• acks=0: Fire and forget. The producer does not wait for any acknowledgment. Maximum throughput, zero durability.
• acks=1: The leader writes the record to its local log and acknowledges. If the leader crashes before followers replicate, the record is lost.
• acks=all (or acks=-1): The leader waits until all ISR replicas have written the record before acknowledging. Combined with min.insync.replicas=2 (or higher), this ensures the record survives a broker failure.

const producer = kafka.producer({
  idempotent: true, // requires acks=all automatically
});

// Or explicit:
const producer = kafka.producer({
  allowAutoTopicCreation: false,
});

await producer.send({
  topic: "orders",
  acks: -1, // wait for all ISR replicas
  messages: [{ key: order.id, value: JSON.stringify(order) }],
});

When a leader fails, the Kafka controller (or KRaft metadata quorum in newer versions) elects a new leader from the ISR. If the ISR contains only the failed leader, Kafka must choose: wait for the leader to recover, or elect an out-of-sync replica and potentially lose committed data. The unclean.leader.election.enable flag controls this. The default is false, meaning Kafka waits for an in-sync replica to become available. Enabling unclean election improves availability at the cost of potential data loss.

The move from ZooKeeper to KRaft (Kafka Raft Metadata mode), complete in Kafka 3.3+, replaced the ZooKeeper dependency with an internal Raft-based metadata quorum. KRaft reduces operational complexity, speeds up controller failover from tens of seconds to milliseconds, and allows scaling to millions of partitions without ZooKeeper becoming a bottleneck.

Exactly-Once Semantics

Getting to exactly-once delivery in a distributed system requires solving two distinct problems: preventing the producer from writing duplicates, and ensuring that consuming and producing (in a stream processing pipeline) happen atomically.

Idempotent Producers

With enable.idempotence=true, Kafka assigns each producer a Producer ID (PID) and a sequence number per partition. The broker tracks the last written sequence number for each (PID, partition) pair. If the producer retries a failed send and the broker receives a duplicate (same PID, same sequence number), it silently discards it instead of writing it twice.

This eliminates duplicates caused by producer retries within a single session. It does not survive a producer restart because a new session gets a new PID.

Transactional Producers

Kafka’s transaction API extends idempotency to multi-partition atomic writes. A transactional producer can write records to multiple partitions and commit or abort them as a single atomic unit. Consumers configured with isolation.level=read_committed see only records from committed transactions.

const producer = kafka.producer({
  transactionalId: "order-service-producer-1", // stable across restarts
  idempotent: true,
});

await producer.connect();
await producer.transaction(async (tx) => {
  await tx.send({
    topic: "processed-orders",
    messages: [{ key: order.id, value: JSON.stringify(order) }],
  });
  await tx.sendOffsets({
    consumerGroupId: "order-processor",
    topics: [{ topic: "orders", partitions: [{ partition: 0, offset: "42" }] }],
  });
});

The sendOffsets call inside the transaction is the key. It writes the consumer offset commit as part of the same transaction as the output write. This means consuming from the input topic and producing to the output topic happen atomically. If the transaction aborts, the output record is invisible to read_committed consumers and the offset is not committed, so the input record is reprocessed. This is the mechanism that enables exactly-once stream processing.

The transactionalId must be stable and unique per producer instance. The broker uses it to fence zombie producers: if two instances share a transactionalId, the newer one invalidates the older one’s epoch. This prevents split-brain scenarios where a slow or restarted producer continues writing after its replacement takes over.

Log Compaction vs. Time-Based Retention

Kafka supports two retention strategies, configurable per topic:

Time-based retention (log.retention.hours, default 168 h / 7 days): Kafka deletes segment files that are older than the retention threshold. This is the right strategy for event streams where you want a rolling window of recent events. The total data volume is bounded by the write rate times the retention period.

Log compaction (cleanup.policy=compact): Kafka retains the latest record for each key indefinitely and removes earlier records with the same key. Compaction runs in the background by a pool of compaction threads. The result is a topic that always contains the most recent value per key, with no bound on key cardinality.

Before compaction:
  [key=user-1, value=A] [key=user-2, value=X] [key=user-1, value=B] [key=user-1, value=C]

After compaction:
  [key=user-2, value=X] [key=user-1, value=C]

Log compaction is the foundation for Kafka as a state store. A consumer that reads a compacted topic from the beginning rebuilds the full current state without needing a separate snapshot. This is the pattern used by Kafka Streams to initialize state stores on startup. It is also the pattern used by Change Data Capture (CDC) pipelines: a compacted topic keyed on the database primary key is a complete, replayable changelog of the table.

A null value tombstone marks a key for deletion. After the next compaction cycle, records with that key are removed.

You can combine both strategies with cleanup.policy=compact,delete to compact but also delete old data beyond a retention period, which prevents unbounded growth on topics with high key churn.

Tradeoffs Table

Dimension	Kafka	RabbitMQ	Amazon SQS	Redpanda	Pulsar
Throughput	Very high (millions msg/s per cluster)	Moderate (tens of thousands/s)	High (managed, scales automatically)	Very high (Kafka-compatible, no JVM)	High (comparable to Kafka)
Latency	Low (single-digit ms typical)	Very low (sub-ms possible)	Higher (10-200ms typical)	Low (often lower than Kafka)	Low (comparable to Kafka)
Ordering guarantees	Per-partition ordering	Per-queue ordering (limited)	Best-effort (FIFO queues available)	Per-partition ordering	Per-partition ordering
Consumer model	Pull-based, consumer controls offset	Push-based (AMQP)	Pull-based (short poll/long poll)	Pull-based, Kafka-compatible	Pull-based with cursor
Retention and replay	Yes, configurable retention and compaction	No (messages deleted after ack)	No (messages deleted after ack)	Yes, same as Kafka	Yes, topic-level retention and cursors
Exactly-once support	Yes (idempotent + transactions)	Not native, requires app-level dedup	No (at-least-once only)	Yes (Kafka-compatible EOS)	Yes (transaction API)
Operational complexity	High (brokers, ZooKeeper or KRaft, partition tuning)	Medium (broker, vhost, queue config)	Low (fully managed, no servers)	Medium (simpler than Kafka, no ZooKeeper)	High (brokers + BookKeeper + ZooKeeper)
Multi-tenancy	Limited (topic-level isolation)	vhosts provide isolation	Queue-level, account-level	Namespace-level (same as Kafka)	First-class: tenants, namespaces, topics
Ecosystem maturity	Very mature (13+ years, broad connector ecosystem)	Mature (20+ years)	Mature (AWS-native, limited ecosystem)	Growing fast (Kafka API compatibility)	Growing (strong in geo-replication use cases)

Kafka is the right choice when you need replay, fan-out to multiple independent consumer groups, or log compaction as a state store. RabbitMQ is better suited for task queues with complex routing via exchanges and bindings where replay is not required. SQS is the right choice when you want zero operational overhead and AWS-native integration. Redpanda is increasingly attractive for teams that want Kafka semantics without the JVM operational burden. Pulsar has better native multi-tenancy and built-in geo-replication but adds BookKeeper as a dependency, which increases operational complexity.

Production Considerations

Replication factor and min.insync.replicas: For any topic where data loss is unacceptable, set replication.factor=3 and min.insync.replicas=2. This tolerates one broker failure while still requiring two acknowledgments before a write is confirmed. Setting min.insync.replicas equal to replication.factor means any broker failure makes the topic unavailable for writes.

Partition count planning: Partitions are the unit of parallelism. More partitions enable higher throughput and more consumer concurrency, but each partition has overhead: open file handles, replication fetch threads, and controller metadata. A reasonable starting point is 3-6 partitions per topic for moderate workloads, scaling to 20-50 for high-throughput topics. Avoid creating hundreds of partitions per topic without measuring: partition count is hard to reduce without data loss.

Producer tuning for throughput:

batch.size=65536          # 64 KiB batches
linger.ms=10              # wait up to 10ms to fill batches
compression.type=snappy   # compress before network send
buffer.memory=67108864    # 64 MiB producer buffer

Consumer tuning for throughput:

fetch.min.bytes=65536      # wait for at least 64 KiB before returning
fetch.max.wait.ms=500      # or up to 500ms
max.poll.records=500       # process up to 500 records per poll

Offset commit strategy: Commit offsets after processing is complete and any downstream writes or side effects are durable. Committing before processing gives at-most-once. Committing after gives at-least-once. For exactly-once, use the transactional API with sendOffsets inside the transaction.

Monitoring consumer lag: Consumer lag (the difference between the latest offset in a partition and the consumer’s committed offset) is the primary health signal for a Kafka consumer. A growing lag means the consumer is falling behind the write rate. Track kafka.consumer.group.lag per (group, topic, partition). Alert when lag exceeds the time budget: if you can tolerate up to 60 seconds of delay, alert when lag divided by the current consume rate exceeds 60 s.

Disk throughput sizing: Kafka’s write path is sequential, which is fast on both spinning disks and SSDs. For high-throughput clusters, provision dedicated disks (not network-attached storage) per broker. The replication traffic doubles disk write load because each follower pulls from the leader. With replication factor 3, each record is written three times across the cluster.

Heap and OS page cache: Kafka brokers run on the JVM. The broker heap should be kept small (4-8 GiB) to avoid long GC pauses. The rest of available RAM should be left free for the OS page cache. A broker that serves mostly cached reads has dramatically lower latency than one that has to read from disk. As a rule: total broker RAM minus 8 GiB JVM heap should exceed the sum of your hottest partitions’ active data size.

Log retention and disk capacity: Retention is per-topic. Calculate disk capacity as: write_rate_MBps * retention_seconds * replication_factor. For a topic receiving 100 MB/s, with 7-day retention and replication factor 3, each broker in a 6-broker cluster stores approximately 100 * 604800 * 3 / 6 = 30 TiB. Plan for at least 20% headroom above this number.

Understanding Kafka at this level transforms it from a black box into a predictable, tunable system. The partitioned log model, once internalized, makes the consumer group semantics, replication behavior, and exactly-once guarantees all feel like logical consequences of the same foundational design rather than separate features to memorize.