How Message Queues Work: Kafka, RabbitMQ, and SQS Compared

A message queue is not a cache, not a database, and not a pub/sub system (even though some brokers support all three). It is a mechanism for transferring work between a producer and a consumer without requiring them to be running at the same time, at the same speed, or even on the same infrastructure. That deceptively simple idea is the foundation of most durable, scalable distributed systems.

The problem is that Kafka, RabbitMQ, and SQS solve slightly different versions of this problem. They have different delivery semantics, different ordering guarantees, different operational footprints, and different failure modes. Picking the wrong one is not a disaster you discover on day one. It is a disaster you discover at 10x your current load, when switching costs are high.

This article covers the internals that matter for production decisions, with TypeScript examples for each broker. The decision framework at the end is opinionated on purpose.

What Message Queues Actually Solve

Before comparing brokers, it is worth being precise about which problems they solve, because teams often reach for a queue when a simpler tool would work fine.

Decoupling

A producer does not need to know which service processes its messages, how many instances of that service are running, or whether the service is currently healthy. It writes to the queue and returns. The consumer reads at its own pace. You can deploy, restart, or scale consumers without coordinating with producers.

This decoupling has a real cost: you lose the immediate acknowledgment of synchronous calls. A producer cannot know whether its message was processed successfully. If you need to know, you are either polling a shared store or building a response channel, which reintroduces coupling.

Backpressure

A slow consumer will accumulate messages in the queue rather than crashing or dropping work. The queue absorbs the burst. If your image processing service handles 10 images per second but your upload endpoint receives 100 per second during peak hours, a queue between them lets the upload endpoint stay responsive while the processor catches up.

Without a queue, your options are: block the producer (bad UX), drop work (data loss), or spawn unbounded worker threads (OOM crash under load). A queue gives you a fourth option: accept the work now, process it when capacity is available.

Reliability and Durability

A message written to a queue persists until it is explicitly acknowledged. If a consumer crashes mid-processing, the message is redelivered to another consumer. This at-least-once delivery guarantee is a property of the broker, not something you have to build yourself.

The flip side of at-least-once delivery is that your consumers must be idempotent. “At least once” means “sometimes twice.” Any consumer that sends an email, charges a credit card, or decrements inventory must handle duplicate delivery without producing duplicate effects.

Kafka

Kafka is a distributed commit log. That framing is important: Kafka is not a queue in the traditional sense. It is an append-only log of records, partitioned and replicated across a cluster. Consumers do not remove messages from Kafka; they maintain a pointer (an offset) into the log and advance it as they read. A message “expires” only when its retention period elapses.

Architecture Internals

A Kafka topic is divided into partitions. Each partition is an ordered, immutable sequence of records stored on disk. Producers write to a specific partition (determined by the message key, or round-robin if no key is set). Consumers within a consumer group each own one or more partitions exclusively. Two consumers in the same group will never read the same partition, which is how Kafka achieves parallel processing without duplicate delivery within a group.

Topic: order-events (4 partitions)

Partition 0: [msg1] [msg5] [msg9]  <-- Consumer A
Partition 1: [msg2] [msg6] [msg10] <-- Consumer B
Partition 2: [msg3] [msg7] [msg11] <-- Consumer C
Partition 3: [msg4] [msg8] [msg12] <-- Consumer D

Ordering is guaranteed within a partition, not across partitions. If you need all events for a given entity to be ordered, use the entity ID as the message key. Kafka will route all messages with the same key to the same partition.

Replication is handled at the partition level. Each partition has a leader and N-1 replicas. Producers write to the leader, which replicates to followers. If the leader fails, a follower is elected as the new leader. You configure the replication.factor per topic and the min.insync.replicas broker setting, which determines how many replicas must acknowledge a write before it is considered committed.

Producing and Consuming in TypeScript

import { Kafka, CompressionTypes, Partitioners } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["kafka1:9092", "kafka2:9092", "kafka3:9092"],
});

const producer = kafka.producer({
  // createPartitioner controls key-to-partition routing
  createPartitioner: Partitioners.DefaultPartitioner,
});

await producer.connect();

// Key-based routing: all events for the same orderId go to the same partition
// This preserves ordering across OrderPlaced -> OrderShipped -> OrderDelivered
await producer.send({
  topic: "order-events",
  compression: CompressionTypes.GZIP,
  messages: [
    {
      key: order.id,           // partition key — ensures ordered delivery per order
      value: JSON.stringify({
        type: "OrderPlaced",
        orderId: order.id,
        userId: order.userId,
        totalCents: order.totalCents,
        placedAt: new Date().toISOString(),
      }),
      headers: {
        "content-type": "application/json",
        "schema-version": "v2",
      },
    },
  ],
});

await producer.disconnect();

import { Kafka, EachMessagePayload } from "kafkajs";

const kafka = new Kafka({
  clientId: "inventory-service",
  brokers: ["kafka1:9092", "kafka2:9092", "kafka3:9092"],
});

const consumer = kafka.consumer({
  groupId: "inventory-service-group",
  // sessionTimeout must be > maxProcessingTime to avoid rebalances mid-processing
  sessionTimeout: 30_000,
  heartbeatInterval: 3_000,
});

await consumer.connect();
await consumer.subscribe({ topic: "order-events", fromBeginning: false });

await consumer.run({
  // autoCommit: false gives you control over when the offset is committed
  // This matters for exactly-once semantics (paired with idempotent consumers)
  autoCommit: false,
  eachMessage: async ({ topic, partition, message, heartbeat }: EachMessagePayload) => {
    const event = JSON.parse(message.value!.toString());

    if (event.type !== "OrderPlaced") {
      // Commit and skip events this consumer does not handle
      await consumer.commitOffsets([
        { topic, partition, offset: (BigInt(message.offset) + 1n).toString() },
      ]);
      return;
    }

    // Call heartbeat periodically during long-running processing
    // to prevent the broker from thinking this consumer is dead
    await heartbeat();

    await updateInventory(event);

    // Commit only after successful processing
    // On restart, we resume from this offset rather than reprocessing
    await consumer.commitOffsets([
      { topic, partition, offset: (BigInt(message.offset) + 1n).toString() },
    ]);
  },
});

The critical detail in the consumer: manual offset commits. When autoCommit is true, offsets are committed on an interval regardless of whether processing succeeded. A crash between the auto-commit and successful processing means the message is silently lost. Manual commits give you at-least-once delivery: a crash before commitOffsets causes the message to be redelivered.

Kafka’s Strengths and Limits

Kafka’s log retention is the feature that makes it qualitatively different from traditional queues. Consumers can replay the entire topic history, which enables backfilling a new service with historical events, reprocessing after a bug fix, or auditing what happened in a time range. No other mainstream broker offers this without significant additional infrastructure.

The operational cost is real. Kafka requires ZooKeeper (or KRaft in newer versions), careful partition count planning (you cannot reduce partitions after creation), JVM tuning, and disk capacity planning. Consumer group rebalances during deployments can pause message processing for seconds to minutes, depending on your partition count and max.poll.interval.ms configuration. Managed options (Confluent Cloud, MSK) reduce but do not eliminate this burden.

Throughput ceiling: millions of messages per second across a properly sized cluster.

RabbitMQ

RabbitMQ implements the AMQP protocol and uses a fundamentally different model: producers publish to exchanges, which route messages to queues based on routing rules. Consumers subscribe to queues and receive messages. When a message is acknowledged, it is removed from the queue.

Architecture Internals

The exchange-binding-queue model gives RabbitMQ routing flexibility that Kafka lacks. There are four exchange types:

Direct exchange: routes messages to queues where the binding key exactly matches the routing key.

Fanout exchange: broadcasts to all bound queues, ignoring routing keys. This is the pub/sub pattern.

Topic exchange: routes based on wildcard pattern matching. A routing key of order.placed.eu would match bindings with patterns order.# or order.*.eu.

Headers exchange: routes based on message headers rather than routing keys. Rarely used in practice.

Producer -> [order.placed] -> Topic Exchange
                                |
            +-----------+-------+---------+
            |           |                 |
    [inventory.queue] [email.queue] [analytics.queue]
    (binding: order.#) (binding: order.placed.*) (binding: #)

Messages removed from a queue are gone unless you have configured a dead letter exchange (DLX), which captures messages that were rejected, expired, or exceeded a delivery count limit. The DLX is your safety net for failed processing.

RabbitMQ does not have built-in log retention or replay. Once a message is acknowledged, it no longer exists. This is the correct behavior for a task queue, and it is a hard limitation if you need event sourcing or historical replay.

Producing and Consuming in TypeScript

import amqp, { Channel, Connection } from "amqplib";

let connection: Connection;
let channel: Channel;

async function getChannel(): Promise<Channel> {
  if (channel) return channel;
  connection = await amqp.connect(process.env.RABBITMQ_URL!);
  channel = await connection.createChannel();

  // Assert the exchange into existence — idempotent, safe to call on startup
  await channel.assertExchange("order-events", "topic", { durable: true });

  return channel;
}

async function publishOrderPlaced(order: Order): Promise<void> {
  const ch = await getChannel();

  const payload = Buffer.from(JSON.stringify({
    orderId: order.id,
    userId: order.userId,
    totalCents: order.totalCents,
    placedAt: new Date().toISOString(),
  }));

  // Routing key: consumers can subscribe to order.placed, order.*, or #
  const routingKey = "order.placed";

  ch.publish("order-events", routingKey, payload, {
    persistent: true,         // survive broker restart (requires durable queue)
    contentType: "application/json",
    messageId: order.id,      // useful for idempotency checks in consumers
    timestamp: Math.floor(Date.now() / 1000),
  });
}

import amqp, { ConsumeMessage } from "amqplib";

async function startInventoryConsumer(): Promise<void> {
  const connection = await amqp.connect(process.env.RABBITMQ_URL!);
  const channel = await connection.createChannel();

  await channel.assertExchange("order-events", "topic", { durable: true });

  // Assert the queue — durable queues survive broker restarts
  const { queue } = await channel.assertQueue("inventory-order-placed", {
    durable: true,
    arguments: {
      // Dead letter exchange: messages that fail or expire land here
      "x-dead-letter-exchange": "order-events-dlx",
      "x-message-ttl": 86_400_000, // 24 hours max age
    },
  });

  // Bind queue to exchange with routing pattern
  await channel.bindQueue(queue, "order-events", "order.placed");

  // prefetch(1) means this consumer handles one message at a time
  // before sending ack. Prevents one slow consumer from starving others
  // if you have multiple workers on the same queue.
  channel.prefetch(1);

  await channel.consume(queue, async (msg: ConsumeMessage | null) => {
    if (!msg) return; // consumer cancelled by broker

    try {
      const event = JSON.parse(msg.content.toString());

      // Idempotency check before processing
      const alreadyProcessed = await checkProcessed(msg.properties.messageId, "inventory");
      if (!alreadyProcessed) {
        await updateInventory(event);
        await markProcessed(msg.properties.messageId, "inventory");
      }

      // Acknowledge only after successful processing
      channel.ack(msg);
    } catch (err) {
      // requeue: false sends to dead letter exchange rather than requeuing indefinitely
      channel.nack(msg, false, false);
    }
  });
}

The prefetch(1) setting deserves attention. Without it, RabbitMQ pushes all available messages to the consumer’s in-memory buffer. If you have 10,000 messages and one consumer restarts, those buffered messages are lost (or redelivered only after the consumer’s heartbeat times out). Prefetch limits how many unacknowledged messages a consumer holds, which keeps in-flight message counts bounded.

RabbitMQ’s Strengths and Limits

The exchange-binding model is powerful for routing. Complex topologies where different consumers need different subsets of events, based on routing keys or headers, are straightforward in RabbitMQ. In Kafka you would filter inside the consumer code, which means every consumer reads everything and discards what it does not need.

RabbitMQ also has lower operational complexity than Kafka for moderate-scale workloads. No JVM tuning, no partition count planning. A two-node RabbitMQ cluster with quorum queues is resilient and easy to operate.

The limits: no log retention, no built-in replay, and throughput that tops out in the tens of thousands of messages per second per queue. RabbitMQ is not designed for Kafka-scale ingestion and will degrade under sustained high throughput before Kafka does.

AWS SQS

SQS is a managed queue service. You pay per API call, not per server. There is no cluster to operate, no partition count to choose, no JVM heap to tune. SQS scales automatically to effectively unlimited throughput.

Architecture Internals

SQS has two queue types with meaningfully different guarantees.

Standard queues deliver messages at least once in approximately-FIFO order. In practice, messages can arrive out of order and can be delivered more than once (beyond the usual at-least-once guarantee). Standard queues scale to nearly unlimited throughput and are the right default for workloads where order does not matter.

FIFO queues guarantee exactly-once processing (within a deduplication window) and strict ordering within a message group. The tradeoff is a throughput cap of 3,000 messages per second with batching (300 without). FIFO queues are priced higher per API call than standard queues.

The deduplication window on FIFO queues is worth understanding: if you send two messages with the same MessageDeduplicationId within a 5-minute window, the second message is discarded. This is the broker-level exactly-once guarantee. You still need idempotent consumers because the deduplication window does not help with messages that were received and partially processed before a consumer crash.

SQS does not have exchanges, routing, or fan-out natively. For fan-out, you combine SQS with SNS (Simple Notification Service): SNS topics receive published messages and fan out to multiple SQS queues. This SNS-to-SQS pattern is the AWS equivalent of RabbitMQ’s fanout exchange.

Producing and Consuming in TypeScript

import { SQSClient, SendMessageCommand, SendMessageBatchCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });

const QUEUE_URL = process.env.ORDER_QUEUE_URL!;

async function publishOrderPlaced(order: Order): Promise<void> {
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: QUEUE_URL,
      MessageBody: JSON.stringify({
        type: "OrderPlaced",
        orderId: order.id,
        userId: order.userId,
        totalCents: order.totalCents,
        placedAt: new Date().toISOString(),
      }),
      // For FIFO queues: MessageGroupId controls ordering, MessageDeduplicationId prevents duplicates
      // MessageGroupId: order.userId,         // all orders per user are ordered
      // MessageDeduplicationId: order.id,     // deduplicates within 5-minute window
      MessageAttributes: {
        EventType: {
          DataType: "String",
          StringValue: "OrderPlaced",
        },
        SchemaVersion: {
          DataType: "String",
          StringValue: "v2",
        },
      },
    })
  );
}

// Batching: up to 10 messages per SendMessageBatch call
// Reduces cost (fewer API calls) and latency under high throughput
async function publishOrdersBatch(orders: Order[]): Promise<void> {
  const entries = orders.map((order) => ({
    Id: order.id,
    MessageBody: JSON.stringify({
      type: "OrderPlaced",
      orderId: order.id,
      userId: order.userId,
      totalCents: order.totalCents,
      placedAt: new Date().toISOString(),
    }),
  }));

  await sqs.send(
    new SendMessageBatchCommand({
      QueueUrl: QUEUE_URL,
      Entries: entries,
    })
  );
}

import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageCommand,
  ChangeMessageVisibilityCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });

const QUEUE_URL = process.env.ORDER_QUEUE_URL!;
// VisibilityTimeout: how long SQS hides a message from other consumers
// while it is being processed. If processing takes longer, extend it.
const VISIBILITY_TIMEOUT_S = 30;

async function pollQueue(): Promise<void> {
  while (true) {
    const response = await sqs.send(
      new ReceiveMessageCommand({
        QueueUrl: QUEUE_URL,
        MaxNumberOfMessages: 10,    // process up to 10 messages per poll
        WaitTimeSeconds: 20,        // long polling: reduces empty-receive API calls
        VisibilityTimeout: VISIBILITY_TIMEOUT_S,
        MessageAttributeNames: ["All"],
      })
    );

    if (!response.Messages || response.Messages.length === 0) {
      continue;
    }

    await Promise.all(
      response.Messages.map(async (msg) => {
        const receiptHandle = msg.ReceiptHandle!;

        try {
          const event = JSON.parse(msg.Body!);

          // Extend visibility timeout for slow processing
          // Call this before VISIBILITY_TIMEOUT_S elapses
          const extendTimer = setInterval(async () => {
            await sqs.send(
              new ChangeMessageVisibilityCommand({
                QueueUrl: QUEUE_URL,
                ReceiptHandle: receiptHandle,
                VisibilityTimeout: VISIBILITY_TIMEOUT_S,
              })
            );
          }, (VISIBILITY_TIMEOUT_S - 5) * 1000);

          await processOrder(event);

          clearInterval(extendTimer);

          // Delete only after successful processing
          await sqs.send(
            new DeleteMessageCommand({
              QueueUrl: QUEUE_URL,
              ReceiptHandle: receiptHandle,
            })
          );
        } catch (err) {
          // Do not delete — let the message become visible again after VisibilityTimeout
          // After maxReceiveCount failures, SQS moves the message to the DLQ
          console.error({ err, messageId: msg.MessageId }, "Failed to process message");
        }
      })
    );
  }
}

Long polling (WaitTimeSeconds: 20) is almost always the right choice. Short polling returns immediately with an empty response when the queue is empty, and you pay for that API call. Long polling holds the connection open for up to 20 seconds, returning as soon as messages arrive. The cost reduction is significant at scale.

The visibility timeout extension pattern is essential for any processing that can take longer than your configured timeout. Without it, SQS assumes the consumer crashed and makes the message visible to other consumers while you are still processing it. The result is duplicate processing without any consumer failure.

Production Considerations

Idempotency: The Non-Optional Requirement

All three brokers guarantee at-least-once delivery. Your consumers will receive duplicate messages. This is not a theoretical edge case: consumer restarts, network timeouts, and broker-level retries all cause redelivery.

The standard pattern is a processed-message store: before processing, check whether you have seen this message ID. After processing, record it. Both checks must be in the same transaction as the business logic.

async function handleOrderPlaced(messageId: string, event: OrderPlacedEvent): Promise<void> {
  await db.transaction(async (tx) => {
    // Check and insert in the same transaction — prevents races under concurrent delivery
    const existing = await tx.processedMessages.findOne({
      messageId,
      consumer: "inventory-service",
    });

    if (existing) return; // Already handled this delivery

    for (const item of event.items) {
      await tx.inventory.decrement(item.sku, item.quantity);
    }

    await tx.processedMessages.insert({
      messageId,
      consumer: "inventory-service",
      processedAt: new Date(),
    });
  });
}

Dead Letter Queues

Every queue needs a dead letter destination. When a message fails N times (your configured maxReceiveCount in SQS, delivery count limit in RabbitMQ, or a consumer that explicitly sends to a DLQ in Kafka), it goes to the DLQ rather than retrying forever. Monitor DLQ depth and alert on it. An accumulating DLQ is a symptom of either a consumer bug or a malformed message that needs manual intervention.

Consumer Lag Monitoring

In Kafka, consumer group lag (the difference between the latest offset and the committed offset) is your primary health signal. Lag growing consistently means your consumers cannot keep up with production throughput. Track it per partition, not just in aggregate. A single stuck partition will hide behind a healthy average.

In RabbitMQ and SQS, monitor queue depth and message age. A queue with 100,000 messages and messages averaging 30 minutes old means your consumer throughput is insufficient or your consumers are failing.

Schema and Versioning

Message schemas are contracts. A field you remove or rename will break consumers that have not been updated. Version your schemas from the start: include a version field in every message and document which consumers handle which versions. Maintain backward compatibility for at least one version while consumers migrate.

Tradeoffs Table

Dimension	Kafka	RabbitMQ	SQS Standard	SQS FIFO
Throughput ceiling	Millions/sec	Tens of thousands/sec	Effectively unlimited	3,000/sec (batched)
Ordering guarantee	Per-partition	Per-queue (single consumer)	Best-effort	Per message group
Message retention	Configurable (days/indefinite)	Until acknowledged	Up to 14 days	Up to 14 days
Replay/backfill	Yes (native)	No	No	No
Delivery guarantee	At-least-once	At-least-once	At-least-once	Exactly-once (dedup window)
Routing flexibility	Topic + partition key	Exchanges, bindings, wildcards	None (use SNS)	None (use SNS)
Operational complexity	High (cluster, JVM, partitions)	Medium (cluster, exchanges)	None (fully managed)	None (fully managed)
Cost model	Infrastructure or cloud pricing	Infrastructure or cloud pricing	Per API call	Per API call (higher)
Self-hosted option	Yes	Yes	No	No
Best for	High-throughput streams, replay, event sourcing	Complex routing, moderate volume, task queues	Simple queues on AWS, unlimited scale, low ops	Ordered processing on AWS with volume constraints

Choosing Between Them

Start with the operational constraint, not the feature list.

If you are on AWS and do not want to operate message broker infrastructure, SQS is the default. The cost is low, the scale ceiling is high, and the operational burden is essentially zero. Use SNS-to-SQS for fan-out. Use FIFO only when you have a concrete ordering requirement and can live with the throughput cap.

If you need log retention or replay, Kafka is the only realistic option. The ability to replay the full event history for a new service, for a bug fix, or for an audit is genuinely not available in RabbitMQ or SQS. This feature alone often justifies Kafka’s operational overhead in data-intensive systems. Use Confluent Cloud or MSK if you want the capabilities without the self-hosted cluster management.

If you have complex routing requirements and moderate throughput, RabbitMQ is the most expressive. Wildcard topic exchanges, dead letter topologies, per-message TTLs, and fine-grained prefetch control make it the right tool for task queues with non-trivial routing logic. Throughput of tens of thousands of messages per second per queue is sufficient for most workloads.

If you need very high throughput and are already on a cloud provider, Kafka on a managed service is the better fit over RabbitMQ. RabbitMQ will require sharding and careful queue topology design to reach Kafka-comparable throughput. Kafka is designed for it.

The decision you will regret is choosing Kafka because it sounds serious and then spending three months tuning partition counts, JVM heap sizes, and consumer group rebalance configs for a system that sends 500 messages per minute. At that scale, SQS or RabbitMQ would have been running in production while you were still writing the Kafka cluster runbook.

Message queue selection is ultimately a question of which constraints matter most for your specific workload: operational simplicity, throughput, ordering guarantees, or replay. Each broker makes a different bet on which of those constraints matter most to its target user. The good news is that none of the three options is wrong in absolute terms, and the TypeScript clients for all three are mature, well-documented, and easy to replace if you discover you chose incorrectly early enough.