Designing a Pub/Sub Messaging System: Topic Partitioning, Consumer Groups, and Exactly-Once Delivery at Scale

Request-response works until it does not. When a downstream service is slow, you block. When it is unavailable, you fail. When traffic spikes, everything degrades together. Pub/sub messaging decouples those concerns, but it introduces its own set of hard problems: ordering, delivery guarantees, consumer coordination, and schema drift. This article builds up the full architecture from first principles, with the specific decisions that separate a toy broker from a production system.

Why Request-Response Falls Apart

Consider an order placement flow. The HTTP handler must: charge the card, reserve inventory, trigger fulfillment, send a confirmation email, and emit an analytics event. If any downstream call fails, you have a partial state problem. If any call is slow, the user waits.

The synchronous coupling creates three failure modes:

1. Availability coupling: a slow email service makes checkout slow.
2. Throughput coupling: downstream systems must match the intake rate of the frontend.
3. Failure blast radius: one crashing consumer can cascade upstream.

Pub/sub addresses all three by inserting a durable log between producers and consumers. Producers append events. Consumers read at their own pace. The broker handles the mismatch.

Core Components

Topics and Partitions

A topic is a named feed of events. A partition is an ordered, immutable log segment within that topic. Partitions are the unit of parallelism: more partitions allow more consumers to read concurrently.

interface PartitionMetadata {
  topicName: string;
  partitionId: number;
  leaderId: number;   // broker ID holding the leader replica
  replicas: number[]; // broker IDs holding all replicas
  isr: number[];      // in-sync replica set
}

Each partition has a single leader replica that handles reads and writes. Follower replicas replicate from the leader. The in-sync replica set (ISR) determines durability: if you configure acks=all, a write is only acknowledged after all ISR members confirm it.

Producers

A producer assigns each message to a partition. The assignment strategy determines your ordering and load distribution properties.

Hash-based partitioning (the most common default):

function hashPartition(key: string, partitionCount: number): number {
  // murmur2 is what Kafka's default partitioner uses
  const hash = murmur2(Buffer.from(key));
  return Math.abs(hash) % partitionCount;
}

Messages with the same key always land in the same partition, which gives you per-key ordering. This is essential for event sourcing scenarios where you need all events for a given entity (user ID, order ID) processed in order.

Range-based partitioning maps key ranges to specific partitions. Useful for time-series data where you want recent data on specific partitions for tiered storage, but it tends to create hot partitions if recent keys are accessed more than historical ones.

Custom partitioning lets you route by any attribute. A common pattern is routing by geography to keep data co-located with consumers in the same region.

Consumer Groups

A consumer group is a set of consumers that cooperatively read a topic. Each partition is assigned to exactly one consumer within the group. This is the core contract that enables parallel consumption without double-processing.

interface ConsumerGroupState {
  groupId: string;
  members: ConsumerMember[];
  partitionAssignments: Map<string, number[]>; // memberId -> partitionIds
  generationId: number; // increments on every rebalance
}

interface ConsumerMember {
  memberId: string;
  clientId: string;
  subscribedTopics: string[];
  lastHeartbeat: Date;
}

When a consumer joins or leaves, the group coordinator triggers a rebalance. During a rebalance, all consumers stop processing and the coordinator redistributes partitions. This stop-the-world behavior is the main reason consumer group design matters.

Two rebalance protocols exist:

Eager rebalance (the original): all members revoke their partitions, then rejoin and get reassigned. Simple, but causes a full processing pause.

Cooperative incremental rebalance (newer): only the partitions that need to move are revoked. Other partitions continue processing during the rebalance. This is the protocol you want in production for large consumer groups.

Offset Management

An offset is the position of a consumer within a partition. Managing offsets correctly is what gives you control over delivery guarantees.

interface OffsetCommitRecord {
  groupId: string;
  topic: string;
  partition: number;
  offset: number;    // the NEXT offset to read (last processed + 1)
  metadata?: string; // arbitrary string, useful for storing processing context
  committedAt: Date;
}

The critical detail: committing offset N means you have processed message at offset N-1. Commit too early (before processing completes) and a crash will skip messages. Commit too late (after a long processing window) and a crash causes reprocessing.

Delivery Guarantees

At-Most-Once

Commit the offset before processing:

async function atMostOnceConsumer(
  consumer: Consumer,
  processFn: (msg: KafkaMessage) => Promise<void>
): Promise<void> {
  for await (const { message, partition } of consumer) {
    // commit first, then process
    await consumer.commitOffset({ partition, offset: message.offset + 1 });
    await processFn(message); // if this throws, message is lost
  }
}

Use this when losing a message is acceptable: analytics event counting, logging. Never use it for financial events or state mutations.

At-Least-Once

Commit after processing:

async function atLeastOnceConsumer(
  consumer: Consumer,
  processFn: (msg: KafkaMessage) => Promise<void>
): Promise<void> {
  for await (const { message, partition } of consumer) {
    await processFn(message);
    // if we crash here, we reprocess on restart
    await consumer.commitOffset({ partition, offset: message.offset + 1 });
  }
}

This is the practical default for most systems. The catch: your processing function must be idempotent, because messages will be redelivered after a crash between processFn and commitOffset.

Exactly-Once

True exactly-once requires coordination between the broker and the consumer’s downstream store. There are two real approaches:

Transactional producers + atomic offset commits (Kafka-native): wrap the produce and offset commit in a single transaction. The broker guarantees that either both happen or neither does. This works when consuming from Kafka and producing back to Kafka.

interface TransactionalProducerConfig {
  transactionalId: string;      // unique per producer instance
  transactionTimeoutMs: number; // default 60000
}

async function processWithTransaction(
  producer: TransactionalProducer,
  consumer: Consumer,
  messages: KafkaMessage[]
): Promise<void> {
  await producer.beginTransaction();
  try {
    // emit derived events
    await producer.send({
      topic: 'processed-events',
      messages: messages.map(transform),
    });
    // atomically commit consumer offsets within the transaction
    await producer.sendOffsets({
      consumerGroupId: 'my-group',
      topics: [{ topic: 'input-topic', partitions: [...] }],
    });
    await producer.commitTransaction();
  } catch (err) {
    await producer.abortTransaction();
    throw err;
  }
}

Idempotent writes with deduplication (works for any downstream store): store the offset or a message-derived deduplication key alongside the processed result in the same atomic write.

interface ProcessedRecord {
  orderId: string;
  status: string;
  // store the offset that produced this record
  sourceOffset: number;
  sourcePartition: number;
}

async function idempotentProcess(
  db: Database,
  message: KafkaMessage
): Promise<void> {
  const dedupeKey = `${message.partition}:${message.offset}`;

  await db.transaction(async (tx) => {
    // check if already processed
    const existing = await tx.query(
      'SELECT id FROM processed_records WHERE dedupe_key = $1',
      [dedupeKey]
    );
    if (existing.rows.length > 0) return; // already processed, skip

    await tx.query(
      'INSERT INTO processed_records (dedupe_key, payload) VALUES ($1, $2)',
      [dedupeKey, message.value]
    );
    // do the actual work inside the same transaction
    await tx.query('UPDATE orders SET status = $1 WHERE id = $2', [...]);
  });
}

The idempotent approach works regardless of broker, and is the right pattern when your downstream target is a relational database.

Production Problems

Poison Messages

A poison message is one your consumer cannot process successfully. Without handling, it blocks the entire partition indefinitely.

The solution: a dead letter topic (DLT). After N retries, route the message to a DLT for manual inspection and replay.

interface DeadLetterRecord {
  originalTopic: string;
  originalPartition: number;
  originalOffset: number;
  payload: Buffer;
  errorMessage: string;
  errorStack: string;
  failedAt: Date;
  retryCount: number;
}

async function withDeadLetterHandling(
  message: KafkaMessage,
  processFn: (msg: KafkaMessage) => Promise<void>,
  producer: Producer,
  maxRetries = 3
): Promise<void> {
  let lastError: Error | undefined;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      await processFn(message);
      return;
    } catch (err) {
      lastError = err as Error;
      if (attempt < maxRetries) {
        await delay(exponentialBackoff(attempt));
      }
    }
  }

  const dlt: DeadLetterRecord = {
    originalTopic: message.topic,
    originalPartition: message.partition,
    originalOffset: Number(message.offset),
    payload: message.value ?? Buffer.alloc(0),
    errorMessage: lastError!.message,
    errorStack: lastError!.stack ?? '',
    failedAt: new Date(),
    retryCount: maxRetries,
  };

  await producer.send({
    topic: `${message.topic}.DLT`,
    messages: [{ value: JSON.stringify(dlt) }],
  });
}

Consumer Lag Monitoring

Consumer lag is the difference between the latest offset in a partition and the consumer group’s committed offset. It is the primary signal for whether your consumers are keeping up.

interface PartitionLag {
  topic: string;
  partition: number;
  currentOffset: number;  // latest offset in the partition
  committedOffset: number; // what the consumer group has acknowledged
  lag: number;
}

function computeLag(
  endOffsets: Map<string, number>,
  committedOffsets: Map<string, number>
): PartitionLag[] {
  const result: PartitionLag[] = [];

  for (const [key, endOffset] of endOffsets) {
    const committed = committedOffsets.get(key) ?? 0;
    const [topic, partition] = key.split(':');
    result.push({
      topic,
      partition: Number(partition),
      currentOffset: endOffset,
      committedOffset: committed,
      lag: endOffset - committed,
    });
  }

  return result;
}

Alert on lag, not throughput. A consumer group processing 10,000 messages per second but falling 50,000 behind is in trouble. A group processing 100 messages per second with zero lag is healthy.

The practical thresholds depend on your SLOs. A typical setup: warn at 10,000 message lag, page at 100,000. Set the partition count so that at peak load, a single consumer can clear its partition in under 30 seconds.

Ordering Guarantees Across Partitions

Ordering is guaranteed within a partition only. If you need total ordering across a topic (rare, and expensive), you are limited to a single partition, which limits your throughput to what one consumer can process.

The more common requirement is per-entity ordering. For example: all events for a given user ID must be processed in order. Hash partitioning on user ID achieves this because all events for the same user land in the same partition.

The hard case is when you need cross-entity coordination. For example, a payment that debits account A and credits account B. Here, pub/sub alone is not sufficient. You need either a saga pattern (with compensating transactions) or an external coordination mechanism.

Schema Evolution

Schemas change. How you handle that change determines whether consumers break on upgrades.

Two compatible change types:

• Adding an optional field: consumers on old schema ignore it. New consumers use it.
• Removing an unused field: old consumers get undefined for the field. New consumers never see it.

Two breaking changes:

• Renaming a field: breaks all consumers that reference the old name.
• Changing a field type: breaks deserialization.

The standard mitigation is a schema registry with compatibility enforcement. On each produce, the producer registers the schema. The registry rejects schemas that are not backward compatible with the previous version. Consumers fetch the schema by ID embedded in the message envelope.

interface MessageEnvelope {
  schemaId: number;   // integer ID from the registry
  payload: Buffer;    // avro or protobuf encoded payload
}

async function deserialize<T>(
  envelope: MessageEnvelope,
  registry: SchemaRegistry
): Promise<T> {
  const schema = await registry.getSchema(envelope.schemaId);
  return schema.decode(envelope.payload) as T;
}

For JSON-only pipelines without a registry: use explicit version fields in your payloads and route by version in the consumer. This is less rigorous but works for small teams.

When to Use What

System	Throughput	Ordering	Delivery Guarantee	Replay	Best For
Kafka	Very high (millions/s)	Per-partition	At-least-once native, exactly-once with transactions	Yes, retention-based	High-throughput event streaming, event sourcing, audit logs
AWS SQS/SNS	High (thousands/s per queue)	FIFO queues only	At-least-once (SQS), at-most-once with deletion	No native replay	Decoupled microservices, task queues, fan-out notifications
Google Pub/Sub	High	No ordering guarantee by default, ordering keys optional	At-least-once	Snapshots (up to 7 days)	Google Cloud-native workloads, global fan-out
Redis Streams	Medium (depends on instance)	Per-stream, per-consumer-group	At-least-once with XACK	Yes, within retention	Low-latency workloads, real-time pipelines, small teams

The choice usually comes down to two questions. Do you need replay? If yes, Kafka or Redis Streams. Do you need managed simplicity over raw throughput? If yes, SQS/SNS for AWS-native or Pub/Sub for GCP-native. Kafka’s operational overhead (partition count decisions, consumer group rebalancing, schema registry, JVM tuning) is only justified when you genuinely need its retention, throughput, or exactly-once transaction semantics.

---

The core of a pub/sub system is not the broker. It is the contracts around it: how producers assign keys, how consumers coordinate, how offsets are committed relative to processing, and how schemas evolve without breaking downstream systems. Get those contracts right first. The broker selection follows from the constraints those contracts impose.