How Apache Kafka Works Internally: Log-Structured Storage, Partition Replication, and Consumer Group Coordination

Most engineers know Kafka as “the durable message queue that scales.” Fewer understand why it scales the way it does, or why certain failure modes are not bugs but deliberate tradeoffs baked into its storage model. The partitioned, append-only log is not a detail — it is the entire design. Everything else in Kafka, from replication to consumer group coordination to exactly-once delivery, is a direct consequence of that one structural choice.

This article works through Kafka’s internals from the ground up: how data lands on disk, how it moves to consumers without copying bytes through user space, how replicas stay in sync, how consumer groups coordinate partition assignment, how KRaft replaced ZooKeeper, and how the producer path achieves exactly-once semantics.

The Partitioned Log: Segments, Indexes, and the Page Cache

A Kafka topic is divided into partitions. Each partition is a strictly ordered, append-only log stored as a sequence of segment files on disk. At any moment, one segment per partition is active (the “head”) and the rest are sealed (the “tail”).

Each segment is represented by three files:

• {base_offset}.log — the raw message data, sequentially written
• {base_offset}.index — a sparse index mapping relative message offsets to byte positions within the .log file
• {base_offset}.timeindex — a sparse index mapping timestamps to offsets, enabling time-based seeks

The .index file is sparse on purpose. Kafka does not write an entry for every message; it writes one every index.interval.bytes of data (default 4096 bytes). Lookups binary-search the index to find the closest lower-bound entry, then scan forward through the .log file to the exact offset. The scan is short because index entries are dense enough to keep the linear scan under a handful of reads.

Kafka relies heavily on the OS page cache. Producers write to the active segment via a buffered channel; the kernel flushes to disk asynchronously. Consumers read from segments that are almost always already in page cache because the consumer lag is typically small relative to total log size. Kafka does not maintain a separate in-process read cache. It lets the OS manage memory, which means the JVM heap can stay small (typically 6-8 GB even on high-throughput brokers), and consumer reads can be served entirely from RAM without a JVM GC pause in the hot path.

A new segment is rolled when the active segment exceeds segment.bytes (default 1 GB) or segment.ms (default 7 days), whichever comes first. Old segments are deleted or compacted according to the topic’s retention policy.

Zero-Copy Transfer with sendfile

When a consumer fetches messages, Kafka uses the sendfile syscall (exposed as FileChannel.transferTo in Java) to move data from the segment file directly from the kernel’s page cache to the network socket buffer. No data is copied into user space. The path is:

disk → page cache → socket buffer → NIC

Without zero-copy, the path would be:

disk → page cache → user-space buffer → kernel socket buffer → NIC

At 1 GB/s throughput, eliminating two memory copies is measurable. It is also why Kafka’s wire format is deliberately simple and unmodified during transfer: the broker cannot re-serialize messages on the way out if it wants zero-copy to work. Message format changes (like the introduction of record batches in Kafka 0.11) require careful versioning precisely because the format on disk is the format on the wire.

ISR-Based Replication and Leader Election

Each partition has one leader broker and zero or more follower replicas. Producers and consumers always talk to the leader. Followers pull from the leader using the same fetch protocol consumers use.

The in-sync replica set (ISR) is the set of replicas that are caught up with the leader within replica.lag.time.max.ms (default 30 seconds). A replica that falls behind is removed from the ISR. One that catches back up is re-added.

The acks configuration on the producer controls durability:

• acks=0 — fire and forget, no acknowledgment
• acks=1 — leader has written to its local log (ISR not consulted)
• acks=-1 (or all) — all current ISR members have written the message

With acks=-1, the leader does not acknowledge the produce request until every ISR member has written the message to its local log. The min.insync.replicas broker/topic setting (commonly set to 2) defines the minimum ISR size required to accept writes. If the ISR shrinks below this threshold, the partition goes offline for writes, which is the correct behavior: it prevents data loss at the cost of availability.

Unclean leader election (unclean.leader.election.enable, default false in modern Kafka) allows a replica outside the ISR to become the leader if all ISR members are unavailable. It restores availability but guarantees data loss for messages that were written to the old leader but not yet replicated. Enabling it is appropriate for use cases where availability beats durability (metrics, logs), and catastrophic for financial event streams.

Leader election itself uses a simple majority vote among the ISR. The controller (now a KRaft quorum) selects the first ISR member as the new leader and writes the new partition metadata.

KRaft: Replacing ZooKeeper for Metadata Management

Before Kafka 3.3, cluster metadata (broker registration, partition leadership, topic configuration) lived in ZooKeeper. This created operational complexity: two distributed systems to operate, version-compatibility constraints, and a metadata propagation lag that forced controllers to fetch state from ZooKeeper before acting.

KRaft (Kafka Raft) replaces ZooKeeper entirely. Metadata is stored in an internal topic called __cluster_metadata. A subset of brokers (or dedicated controller nodes) form a Raft quorum and maintain a replicated log of metadata events. The active controller is the Raft leader.

Key improvements over ZooKeeper:

• Metadata is a Raft log, so the controller always has a complete, consistent view without a fetch round-trip to an external system
• Partition leadership changes propagate as log entries, giving followers a clear linearized history of cluster state
• Controller failover is faster because the incoming leader replays the metadata log locally rather than rebuilding state from ZooKeeper watches
• Single system to operate for most deployments

KRaft became production-ready in Kafka 3.3 and is the default mode since Kafka 3.7. ZooKeeper mode is deprecated.

The Producer Path: Batching, Compression, and acks

A Kafka producer accumulates records into batches per partition before sending. Two settings control the batching behavior:

• batch.size — maximum batch size in bytes (default 16384)
• linger.ms — time to wait for the batch to fill before sending anyway (default 0)

Setting linger.ms to a small nonzero value (5-20 ms for high-throughput pipelines) dramatically improves throughput at the cost of a slight latency increase. Compression (compression.type: lz4, snappy, zstd, gzip) is applied at the batch level. Larger batches compress better, so linger.ms and batch.size compound compression gains.

A TypeScript producer using the kafkajs library illustrates the key tuning knobs:

import { Kafka, CompressionTypes } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"],
});

const producer = kafka.producer({
  // Wait up to 10ms to fill a batch before sending
  lingerMs: 10,
  // Send once all ISR members acknowledge
  acks: -1,
  // Retry on transient errors
  retry: { retries: 5 },
  // Enable idempotent delivery
  idempotent: true,
});

await producer.connect();

await producer.send({
  topic: "orders",
  compression: CompressionTypes.LZ4,
  messages: [
    {
      // Key determines partition assignment
      key: order.customerId,
      value: JSON.stringify(order),
      headers: { "content-type": "application/json" },
    },
  ],
});

Partition assignment for a keyed message uses a murmur2 hash of the key modulo the partition count. The same key always routes to the same partition, which gives you ordering guarantees per key. Unkeyed messages use sticky partitioning within a batch to reduce the number of partitions touched per request.

Idempotent Producers and Exactly-Once Semantics

By default, network retries can produce duplicate messages: the broker writes the message, sends the ack, the ack is lost in transit, and the producer retries, producing the message again. The idempotent producer (enable.idempotence=true) solves this.

When idempotency is enabled, the broker assigns each producer a 64-bit producer ID (PID). Each message carries a monotonically increasing sequence number per partition. The broker deduplicates within a window of 5 in-flight requests: if it receives a message with a sequence number it has already seen for that PID and partition, it discards the duplicate silently and returns success to the producer. This gives exactly-once delivery within a single producer session.

Transactional producers extend this across partitions and consumer offset commits. The producer is assigned a transactional ID, which allows the broker to recover an in-progress transaction after a producer restart:

const producer = kafka.producer({
  transactionalId: "order-processor-1",
  idempotent: true,
  maxInFlightRequests: 1,
});

await producer.connect();
await producer.transaction(async (tx) => {
  // Publish to output topic
  await tx.send({
    topic: "invoices",
    messages: [{ key: order.id, value: JSON.stringify(invoice) }],
  });

  // Commit the consumer offset atomically with the produce
  await tx.sendOffsets({
    consumerGroupId: "order-processor-group",
    topics: [{ topic: "orders", partitions: [{ partition: 0, offset: "42" }] }],
  });
});

The two-phase commit works as follows: the coordinator broker first writes a PREPARE_COMMIT marker to the transaction log, then writes COMMIT markers to all involved partitions. Consumers with isolation.level=read_committed skip over messages in an open or aborted transaction, making the whole pipeline exactly-once end-to-end.

Consumer Group Coordination

Consumer groups are the mechanism for parallel consumption. Each partition is consumed by exactly one consumer within a group at a time. Multiple groups consume the same partitions independently.

A designated broker acts as the group coordinator for each consumer group. The coordinator maintains the group’s state: current member list, generation ID, and committed offsets. The protocol works in two phases:

1. Join phase — all consumers send JoinGroup requests. The coordinator waits for a configurable timeout, then elects one consumer as the group leader (the first to join). The leader receives the full member list and is responsible for computing the partition assignment.

2. Sync phase — the leader sends SyncGroup with the computed assignment. The coordinator distributes the assignment to all members. Each member now knows which partitions it owns.

The cooperative sticky rebalancer (the default since Kafka 3.1) avoids the stop-the-world pause of the earlier eager rebalancer. In eager rebalancing, all consumers revoke all their partitions and rejoin from scratch. In cooperative rebalancing, only partitions that need to move are revoked. A consumer joining a group triggers one or two incremental rebalance rounds rather than a full pause across all members.

Offset commits go to the internal __consumer_offsets topic. Kafka retains only the latest committed offset per group/topic/partition. Consumers commit offsets either automatically on a configurable interval (auto.commit.interval.ms, default 5 seconds) or manually for tighter control over the at-least-once vs. exactly-once boundary.

Log Compaction

Time-based retention deletes entire segments after a configured period. Log compaction is a different retention model: instead of deleting by age, the broker retains the latest value per key. This is appropriate for use cases where the log represents a changelog or a materialized state (user profile updates, configuration changes, database CDC streams).

The compaction background thread scans the “dirty” (recently written) portion of the log and builds an in-memory offset map of the latest offset seen for each key. It then rewrites the “clean” portion, dropping any message whose key has a more recent version elsewhere in the log.

Deletion is signaled via tombstone records: a message with a non-null key and a null value. The compactor retains tombstones for a grace period (delete.retention.ms, default 24 hours) so consumers have time to observe the deletion before the tombstone is itself removed.

A topic can use cleanup.policy=compact,delete to combine both: compaction keeps the latest value per key, and time-based deletion removes segments that are entirely older than the retention period.

Production Tuning

Partition count sizing. More partitions allow higher parallelism, but each partition requires memory and file handles on the broker. A rule of thumb: target 1 MB/s throughput per partition, then divide total expected throughput. A topic expected to handle 500 MB/s with a replication factor of 3 needs roughly 500 partitions. You can increase partition count later, but you cannot decrease it without recreating the topic.

Replication factor. Use replication.factor=3 and min.insync.replicas=2 for any production topic. This tolerates one broker failure without data loss and without taking the partition offline for writes.

Consumer lag monitoring. Lag is the difference between the latest offset in a partition and the committed offset for a consumer group. Tools like kafka-consumer-groups.sh --describe, Burrow, or a metrics pipeline reading from the __consumer_offsets topic expose lag per partition. Alert on sustained lag growth, not on instantaneous lag spikes.

Segment and retention tuning. segment.bytes (default 1 GB) controls when a new segment starts. Smaller segments allow more frequent log compaction and faster cleanup but increase the number of open file handles. log.retention.bytes and log.retention.ms control how much data is kept. For high-volume topics, set both: log.retention.bytes prevents disk exhaustion regardless of time, and log.retention.ms prevents indefinite accumulation on low-volume topics.

Heap and page cache. Keep the broker JVM heap at 6-8 GB. Give the remainder of available RAM to the OS for page cache. Kafka’s read performance is almost entirely determined by page cache hit rate. A broker with 32 GB of RAM and a 6 GB heap has 26 GB available for page cache.

Producer tuning checklist for throughput:

# Larger batches, allow time to fill
batch.size=65536
linger.ms=20
compression.type=lz4

# Pipelining
max.in.flight.requests.per.connection=5

# Durability (adjust per use case)
acks=all
enable.idempotence=true

Tradeoffs: Kafka vs. Alternatives

Dimension	Kafka	NATS JetStream	RabbitMQ	Redpanda	Amazon Kinesis
Delivery model	Pull (consumer polls)	Push or pull	Push	Pull	Pull (shard iterator)
Ordering guarantees	Per partition	Per subject stream	Per queue (not topic)	Per partition	Per shard
Message replay	Yes, offset-based	Yes, sequence-based	No (queues are destructive)	Yes, offset-based	Yes, sequence number
Persistence model	Append-only log, disk	File or memory store	AMQP queue, optional disk	Append-only log, disk	AWS-managed, S3 offload
Clustering approach	KRaft Raft quorum	NRG Raft per stream	Quorum queues (Raft)	Internal Raft, no ZK	Managed, opaque
Latency profile	5-30 ms typical	Sub-millisecond	1-5 ms typical	5-20 ms, p99 lower	70-200 ms typical
Backpressure	Consumer-controlled fetch	Server-side max-pending	Consumer credits (AMQP)	Consumer-controlled fetch	GetRecords rate limits
Protocol complexity	Kafka binary TCP	NATS text or binary	AMQP 0-9-1 / STOMP	Kafka-compatible	AWS SDK / HTTP
Operational footprint	Medium (brokers + KRaft)	Low (single binary)	Medium (queues + plugins)	Low (single binary)	Zero (fully managed)
Sweet spot	High-throughput event streaming, audit log, CDC	Low-latency microservices, edge messaging	Task queues, routing logic	Kafka workloads with lower ops overhead	AWS-native streaming, elastic scale

Redpanda is worth a separate note: it is a Kafka-compatible broker written in C++ on the Seastar framework. It eliminates the JVM and ZooKeeper entirely and achieves lower tail latencies than Kafka on equivalent hardware, but it does not support every Kafka feature (tiered storage behavior differs, some admin APIs are missing). If you are starting fresh and want Kafka semantics without Kafka’s operational complexity, Redpanda is a serious option. If you already operate Kafka and need KRaft stability, upgrade rather than migrate.

Closing

The partitioned append-only log is an unusually good fit for a wide class of problems: event sourcing, CDC pipelines, audit trails, fan-out to heterogeneous consumers, and any workload where replay matters. Once that model is internalized, the rest of Kafka’s design feels inevitable: zero-copy follows from wanting to push log data to consumers efficiently without touching user space; ISR replication follows from needing durability without synchronous two-phase commit overhead; consumer group coordination follows from wanting stateless brokers that do not track per-consumer state. The complexity in Kafka is real, but it is load-bearing complexity. Every piece is there because the alternative is worse.