How Kafka Works: Partitions, Replication, Consumer Groups, and the Commit Log

Most engineers know Kafka as “the message queue that handles high throughput.” That description is accurate but it skips over the mechanics that make Kafka behave the way it does in production: why partition count matters for ordering, why ISR shrinkage causes latency spikes, why consumer rebalancing stalls processing, and why exactly-once is harder than the documentation suggests.

This article walks through Kafka’s core internals in the order they matter for building reliable systems.

The Append-Only Commit Log

The foundation of Kafka is not a queue. It is a distributed, partitioned, append-only log. Understanding this shapes every other decision.

Each topic partition is a directory on disk. Inside that directory, Kafka maintains a series of segment files. A segment is a pair of files: the log file (.log) containing the actual message bytes, and an offset index file (.index) mapping logical offsets to physical byte positions in the log.

When a producer writes a message, Kafka appends it to the active segment of the target partition. The write is sequential. Sequential I/O on spinning disks is within an order of magnitude of memory access speeds, and on SSDs it saturates the bus. This is why Kafka achieves high throughput without needing messages in memory.

// What a Kafka message looks like at the protocol level (simplified)
interface RecordBatch {
  baseOffset: number;        // first offset in this batch
  batchLength: number;
  magic: number;             // version byte, currently 2
  attributes: number;        // compression codec, timestamp type
  lastOffsetDelta: number;
  firstTimestamp: bigint;
  maxTimestamp: bigint;
  producerId: bigint;        // for idempotent producers
  producerEpoch: number;     // for idempotent producers
  baseSequence: number;      // for idempotent producers
  records: Record[];
}

interface Record {
  length: number;
  attributes: number;
  timestampDelta: number;
  offsetDelta: number;
  key: Buffer | null;
  value: Buffer | null;
  headers: Array<{ key: string; value: Buffer }>;
}

The offset index is sparse. Kafka does not write an index entry for every message. It writes one every log.index.interval.bytes bytes (default 4096). To find a given offset, Kafka binary-searches the index for the nearest entry below the target, then scans forward through the log file. This keeps index files small while maintaining acceptable read latency.

Segments roll when they hit log.segment.bytes (default 1 GB) or log.segment.ms (default 7 days). Old segments are eligible for deletion or compaction based on retention policy. Retention by time (log.retention.hours) deletes segments whose newest message is older than the threshold. Retention by size (log.retention.bytes) deletes oldest segments when total partition size exceeds the limit. Log compaction is a third mode: Kafka keeps only the latest message per key, which makes a partition behave like a changelog for a key-value store.

Partition-Level Ordering

Kafka guarantees ordering within a partition, not across partitions. This is the most commonly misunderstood constraint.

When you produce a message without a key, the default partitioner uses round-robin across partitions. Two messages for the same logical entity (say, two events for user_id: 42) may land in different partitions and be consumed in any relative order.

To get ordering for an entity, you must route all messages for that entity to the same partition. The standard approach is to set the message key to the entity identifier. Kafka hashes the key using murmur2 and takes hash % numPartitions. As long as partition count does not change, the same key always lands on the same partition.

import { Kafka, CompressionTypes } from "kafkajs";

const kafka = new Kafka({ brokers: ["localhost:9092"] });
const producer = kafka.producer();

await producer.connect();

// All events for user 42 go to the same partition via key routing
await producer.send({
  topic: "user-events",
  messages: [
    {
      key: "user-42",       // murmur2(key) % numPartitions = stable partition
      value: JSON.stringify({ type: "purchase", amount: 99.99 }),
    },
  ],
});

The implication: if you add partitions to a topic, the mapping changes and ordering breaks for in-flight messages. Teams that need key-based ordering and also need to scale out must either pre-provision enough partitions or accept a brief ordering window violation during the repartition.

Leader-Follower Replication and ISR

Each partition has one leader and zero or more followers, spread across brokers. Producers and consumers talk to the leader. Followers replicate from the leader asynchronously.

The in-sync replicas (ISR) set is the key concept for durability. A replica is in-sync if it has fetched from the leader within replica.lag.time.max.ms (default 30 seconds). If a follower falls behind, it is removed from the ISR. If it catches up, it is added back.

The acks producer configuration controls what “written” means:

• acks=0: fire and forget. No wait for any acknowledgment. Maximum throughput, zero durability guarantee.
• acks=1: wait for the leader to write to its local log. If the leader crashes before followers replicate, the message is lost.
• acks=all (or -1): wait for all ISR members to acknowledge. Combined with min.insync.replicas, this gives a durable write guarantee.

const producer = kafka.producer({
  // Wait for all ISR replicas to acknowledge
  // Combined with min.insync.replicas=2 on the broker,
  // this means at least 2 replicas confirm the write
});

await producer.send({
  topic: "payments",
  acks: -1,  // all ISR replicas
  messages: [{ key: "txn-001", value: JSON.stringify(payment) }],
});

The interaction between acks=all and ISR shrinkage is where production surprises happen. If your replication factor is 3 and min.insync.replicas=2, you tolerate one broker failure. But if two followers fall behind simultaneously (network partition, GC pause, slow disk), the ISR shrinks to just the leader. Now acks=all only waits for one replica, which may not match your intent.

Set min.insync.replicas and monitor ISR size. Alert when ISR drops below replication factor. The alert should wake someone up because your durability guarantee has silently degraded.

Unclean leader election (unclean.leader.election.enable) controls what happens when all ISR replicas are unavailable. If set to true, Kafka will elect an out-of-sync replica as leader to keep the partition available. You get availability but may lose committed messages. For most use cases, keep this false.

Consumer Groups and Cooperative Rebalancing

A consumer group is a set of consumers that collectively read a topic. Kafka assigns each partition to exactly one consumer in the group. If you have 6 partitions and 3 consumers, each consumer reads 2 partitions. If you have 6 partitions and 8 consumers, 2 consumers sit idle.

Partition assignment happens during a rebalance. A rebalance is triggered when:

• A consumer joins or leaves the group
• A consumer fails to send a heartbeat within session.timeout.ms
• The topic’s partition count changes
• A rebalance is explicitly triggered

The classic rebalance protocol (eager rebalancing) stops all consumers, revokes all partition assignments, then reassigns from scratch. During this stop-the-world phase, no messages are processed. For groups with many partitions and slow commit logic, this pause can run for tens of seconds.

Cooperative rebalancing, introduced in Kafka 2.4 and now the default in most clients, avoids the full stop. Consumers that are not affected by the rebalance continue processing. Only the partitions being reassigned are paused. This reduces rebalance impact significantly for large groups.

const consumer = kafka.consumer({
  groupId: "payment-processor",
  sessionTimeout: 30000,         // broker waits this long before declaring consumer dead
  heartbeatInterval: 3000,       // consumer sends heartbeats this often
  rebalanceTimeout: 60000,       // max time for a consumer to complete rebalance
});

await consumer.connect();
await consumer.subscribe({ topic: "payments", fromBeginning: false });

await consumer.run({
  // Partition-level concurrency. Each partition is processed sequentially.
  // eachBatch gives you explicit offset control.
  eachBatch: async ({ batch, resolveOffset, heartbeat, isRunning }) => {
    for (const message of batch.messages) {
      if (!isRunning()) break;

      await processPayment(message);
      resolveOffset(message.offset);

      // Call heartbeat inside long loops to prevent session timeout
      await heartbeat();
    }
  },
});

Offset commits determine what happens after a consumer crash. By default, KafkaJS commits offsets automatically every 5 seconds. If the consumer crashes after processing a message but before the auto-commit fires, the message is reprocessed after restart. For at-least-once semantics, this is acceptable. For exactly-once, it requires additional coordination.

Exactly-Once Semantics

Exactly-once in Kafka means a message is written exactly once to the output topic, even if the producer retries. This requires two features working together: idempotent producers and transactional writes.

An idempotent producer assigns a producer ID (PID) and a sequence number to each message. The broker deduplicates retries by tracking the last sequence number per (PID, partition). If a retry arrives with the same sequence number, the broker acknowledges it but does not write it again. This eliminates duplicates from network retries.

const producer = kafka.producer({
  idempotent: true,         // enables PID + sequence number tracking
  transactionalId: "payment-processor-1",  // enables transactions
  maxInFlightRequests: 5,   // must be <= 5 for idempotent producers
});

await producer.connect();

// Transactional write: atomically write to multiple partitions/topics
await producer.transaction(async (txn) => {
  await txn.send({
    topic: "processed-payments",
    messages: [{ key: payment.id, value: JSON.stringify(result) }],
  });

  // Also commit offsets within the transaction so read-process-write is atomic
  await txn.sendOffsets({
    consumerGroupId: "payment-processor",
    topics: [
      {
        topic: "payments",
        partitions: [{ partition: batch.partition, offset: nextOffset }],
      },
    ],
  });
});

The transaction coordinator is a special broker role. It logs transaction state to an internal topic (__transaction_state). When a transaction commits, the coordinator writes a commit marker to each partition involved. Consumers with isolation.level=read_committed skip messages that are part of open or aborted transactions. This is how exactly-once read-process-write works end to end.

The cost: transactional writes add latency. The two-phase commit between the coordinator and partition leaders is a synchronous round trip. For payment processing or inventory updates where correctness matters more than p99 latency, the cost is worth it. For metrics pipelines where duplicate counts are acceptable, idempotent producers without transactions are usually enough.

Tradeoffs: Kafka vs Redpanda vs Pulsar vs NATS JetStream

Dimension	Kafka	Redpanda	Pulsar	NATS JetStream
Storage model	Segment files on disk (JVM)	Segment files on disk (C++)	Bookkeeper ledgers	File-based streams
Protocol compatibility	Native	Kafka-compatible API	Pulsar + Kafka (KoP)	NATS protocol
Operational complexity	High (ZooKeeper or KRaft)	Lower (no ZooKeeper)	High (ZooKeeper + BookKeeper)	Low
Throughput ceiling	Very high	Very high (lower tail latency)	High	Moderate
Tail latency	p99 can spike under GC	Consistently low (no JVM)	Moderate	Low
Multi-tenancy	Via topic naming conventions	Via topic naming conventions	Native namespaces	Via accounts
Geo-replication	MirrorMaker 2	Built-in	Built-in	Built-in
Exactly-once	Yes (transactions)	Yes (Kafka-compatible)	Yes	Yes
Ecosystem maturity	Very mature	Growing fast	Mature	Growing
Hosted options	Confluent, AWS MSK, Aiven	Redpanda Cloud	StreamNative	Synadia

Redpanda is the most relevant Kafka alternative for teams that want Kafka semantics without JVM operational overhead. It speaks the Kafka protocol, so your existing producers and consumers work without changes. The absence of a JVM means no GC pauses and more predictable tail latency. The tradeoff is a smaller ecosystem and fewer third-party integrations.

Pulsar’s multi-tenancy and built-in geo-replication are genuine advantages for organizations running shared infrastructure across many teams or regions. The operational complexity is real: you are managing ZooKeeper, BookKeeper, and brokers as separate layers.

NATS JetStream is the right choice when you need a lightweight message store with at-least-once or exactly-once guarantees and operational simplicity matters more than raw throughput. It is not a Kafka replacement for high-volume event streaming, but it is a reasonable choice for microservice communication and command queues.

Production Considerations

Broker sizing. A common starting point is 3 brokers with 6-8 cores each, 32 GB RAM, and fast NVMe disks. Kafka’s page cache usage is the primary memory consumer: the OS caches recently written and read log segments. Giving the OS 60-70% of memory for page cache is normal. Do not over-tune the JVM heap beyond 6 GB; large heaps cause long GC pauses.

Network bandwidth is often the real ceiling. With replication factor 3, every byte written by producers flows over the network twice more (leader to each follower). A 10 Gbps NIC shared with replication traffic saturates before disk does. Size for peak producer throughput times (replication factor - 1) in outbound network bandwidth per broker.

Partition count. More partitions give you more parallelism but more overhead. Each partition is a directory, an open file handle, and memory for the replica. Kafka recommends no more than 4,000 partitions per broker and 200,000 partitions per cluster. Start with a number that supports your peak consumer parallelism (one partition per consumer thread at peak) and pre-provision for some headroom. Changing partition count later risks ordering violations.

Retention policy. Size-based retention (log.retention.bytes per partition) is more predictable for capacity planning than time-based retention. With time-based retention, a quiet topic retains data for the full window regardless of volume. Size-based retention reclaims space as soon as the limit is hit. Use both together: set a time bound to protect against stale data and a size bound to protect disk capacity.

Lag monitoring. Consumer lag is the number of messages a consumer group is behind the latest offset. Zero lag does not mean healthy: a consumer can be stuck with zero lag if it has stopped fetching entirely. Monitor both lag and consumer group state (Stable, PreparingRebalance, CompletingRebalance). Alert on lag growth rate, not just absolute lag. A group that is 100,000 messages behind but catching up is fine. A group that is 100 messages behind and growing is not.

// Programmatic lag check using kafkajs admin
const admin = kafka.admin();
await admin.connect();

const offsets = await admin.fetchTopicOffsets("payments");
const groupOffsets = await admin.fetchOffsets({
  groupId: "payment-processor",
  topics: ["payments"],
});

for (const partition of offsets) {
  const groupPartition = groupOffsets[0].partitions.find(
    (p) => p.partition === partition.partition
  );
  const lag =
    BigInt(partition.offset) - BigInt(groupPartition?.offset ?? "0");
  console.log(`Partition ${partition.partition}: lag = ${lag}`);
}

await admin.disconnect();

KRaft mode. Kafka 3.x shipped KRaft (Kafka Raft) as the replacement for ZooKeeper. With Kafka 4.0, ZooKeeper mode is removed. If you are deploying Kafka today, use KRaft. It simplifies operations (one fewer distributed system to manage), reduces metadata propagation latency, and removes the 200,000-partition soft limit that ZooKeeper imposed.

Closing

Kafka’s durability and ordering guarantees are precise: per-partition ordering, ISR-controlled write durability, and exactly-once semantics only when using idempotent producers with transactions and read_committed consumers. None of these are defaults. You have to opt in deliberately, understand the latency cost, and instrument everything.

The teams that run Kafka reliably are the ones who treat lag and ISR size as first-class signals, pre-provision partitions rather than retrofitting them under load, and size brokers around network throughput rather than IOPS.