How Amazon SQS Works Internally: Distributed Message Storage, Visibility Timeouts, and the Architecture Behind the Most Widely Deployed Message Queue

SQS looks deceptively simple from the outside: put a message in, get a message out, delete it when you’re done. That simplicity is deliberate, and it comes with real tradeoffs. Under the surface, SQS runs on a distributed storage system that replicates messages across multiple availability zones, manages a visibility timeout state machine for at-least-once delivery semantics, and splits into two fundamentally different queue types with incompatible ordering guarantees. If you use SQS at scale without understanding these mechanics, you will hit visibility timeout races, duplicate processing bugs, and unexpected cost spikes from short polling. This article goes through how SQS actually works.

Distributed Message Storage

SQS does not expose its storage layer, but AWS documentation and infrastructure engineers who have traced network behavior describe a system built on redundant storage across multiple availability zones within a region. When you call SendMessage, SQS writes the message to multiple servers in at least two AZs before returning a successful response. This synchronous replication gives SQS its durability guarantee: a successful SendMessage means your message will survive individual host failures and AZ-level events.

The tradeoff is that standard queues are not strictly ordered. When messages land on different servers, ReceiveMessage polls from a distributed set of hosts. Because of propagation timing between replicas and the way the service load-balances read requests across servers, messages are returned in roughly FIFO order but not guaranteed FIFO. You can also receive the same message more than once if a replica lags and re-delivers a message that was already received but not yet deleted. SQS explicitly categorizes standard queues as at-least-once delivery.

This architecture gives SQS effectively unlimited throughput on standard queues. Because there is no single ordering coordinator, you can scale to tens of thousands of messages per second by adding producers and consumers. The cost is ordering and exactly-once semantics.

The Visibility Timeout Mechanism

The visibility timeout is the central mechanism behind SQS’s delivery model. When a consumer calls ReceiveMessage, SQS does not delete the message. Instead, it hides the message from other consumers for a configurable duration called the visibility timeout (default: 30 seconds, maximum: 12 hours). The consumer is expected to process the message and then call DeleteMessage before the timeout expires. If DeleteMessage never arrives, SQS makes the message visible again for another consumer to receive.

import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand, ChangeMessageVisibilityCommand } from "@aws-sdk/client-sqs";

const client = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/my-queue";

async function processMessages(): Promise<void> {
  const receive = await client.send(
    new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20, // long polling
      VisibilityTimeout: 60, // hide message for 60s
    })
  );

  for (const message of receive.Messages ?? []) {
    const processingStart = Date.now();

    try {
      // Extend visibility timeout if processing takes longer than expected
      const extendAt = setTimeout(async () => {
        await client.send(
          new ChangeMessageVisibilityCommand({
            QueueUrl: QUEUE_URL,
            ReceiptHandle: message.ReceiptHandle!,
            VisibilityTimeout: 120, // extend by another 120s
          })
        );
      }, 45_000); // extend after 45s

      await doWork(message.Body!);
      clearTimeout(extendAt);

      await client.send(
        new DeleteMessageCommand({
          QueueUrl: QUEUE_URL,
          ReceiptHandle: message.ReceiptHandle!,
        })
      );
    } catch (err) {
      // Do NOT delete the message. It will reappear after the visibility timeout.
      console.error("Processing failed, message will reappear:", err);
    }
  }
}

A few things to get right here. First, ReceiptHandle is not a stable identifier for the message. Each time a message is received, SQS returns a new ReceiptHandle. You must use the receipt handle from the most recent receive call to delete or extend visibility. Second, if your processing time is variable and can exceed the visibility timeout, use ChangeMessageVisibility to extend it rather than setting an excessively large default. Extending is cheap and avoids the consumer race condition where two workers process the same message concurrently because the first one timed out.

The ApproximateReceiveCount attribute tells you how many times a message has been received. This is the mechanism behind dead-letter queue routing.

Standard vs FIFO Queues

Standard and FIFO queues are fundamentally different in their architecture, not just in a configuration flag.

Standard queues use the distributed, multi-AZ architecture described above. Throughput is nearly unlimited (at least 3,000 messages per second with batching, effectively uncapped with enough throughput). Messages are delivered at least once and in roughly-FIFO order. Each ReceiveMessage call can return up to 10 messages from any replica.

FIFO queues add a coordination layer that enforces strict ordering and exactly-once delivery within message groups. Every message sent to a FIFO queue must include a MessageGroupId. SQS uses this group ID to route all messages with the same group ID through the same ordering sequence. Within a group, messages are delivered strictly in the order they were sent. Across different groups, ordering is independent.

import { SendMessageCommand } from "@aws-sdk/client-sqs";

// FIFO queues: queue name must end in .fifo
const FIFO_QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/orders.fifo";

await client.send(
  new SendMessageCommand({
    QueueUrl: FIFO_QUEUE_URL,
    MessageBody: JSON.stringify({ orderId: "ord_1234", event: "placed" }),
    MessageGroupId: "customer-42",           // ordering key
    MessageDeduplicationId: "ord-1234-placed", // dedup key (required without content-based dedup)
  })
);

FIFO queues support two deduplication modes:

1. MessageDeduplicationId: You provide an explicit deduplication key. SQS ignores duplicate sends with the same key within a 5-minute deduplication window.
2. Content-based deduplication: SQS computes a SHA-256 hash of the message body and uses that as the deduplication ID. Enable this at queue creation time. Useful when message content is naturally unique but you don’t want to manage explicit dedup IDs.

The throughput cost of FIFO is significant: up to 3,000 messages per second with batching, or 300 per second without. If you need higher throughput with FIFO semantics, you can increase concurrency by using more message group IDs, since SQS delivers messages from different groups in parallel. Think of MessageGroupId as a partition key: more groups means more parallelism, but messages within a group are still sequential.

Long Polling vs Short Polling

By default, ReceiveMessage uses short polling: it samples a random subset of SQS servers and returns immediately, even if no messages are available. This produces empty responses when the queue is sparse and runs up API request costs quickly.

Long polling sets WaitTimeSeconds (1 to 20 seconds). With long polling, SQS holds the connection open until a message arrives or the wait time expires. Long polling queries all SQS servers rather than a subset, which reduces false-empty responses and drops your cost per processed message significantly.

Short polling:
Client --> SQS (random subset of servers) --> returns immediately (possibly empty)
Client --> SQS (random subset of servers) --> returns immediately (possibly empty)
... repeated hundreds of times per minute

Long polling (WaitTimeSeconds=20):
Client --> SQS (all servers, hold connection) --> returns when message arrives or 20s elapsed
... one request per 20 seconds minimum

You can enable long polling at the queue level (set ReceiveMessageWaitTimeSeconds on the queue) or per-request. Per-request is more flexible for consumers with variable load. Queue-level is simpler for uniform workloads.

The practical impact: a consumer doing short polling on a low-traffic queue can generate millions of API requests per month with almost no messages processed. At $0.40 per million requests, this compounds. Long polling with WaitTimeSeconds=20 keeps costs proportional to actual message volume.

Dead-Letter Queues and Redrive Policies

A dead-letter queue (DLQ) is a separate queue where SQS routes messages that fail processing repeatedly. Configure it via a redrive policy on the source queue:

import { SetQueueAttributesCommand } from "@aws-sdk/client-sqs";

await client.send(
  new SetQueueAttributesCommand({
    QueueUrl: QUEUE_URL,
    Attributes: {
      RedrivePolicy: JSON.stringify({
        deadLetterTargetArn: "arn:aws:sqs:us-east-1:123456789:my-queue-dlq",
        maxReceiveCount: "5", // move to DLQ after 5 failed receives
      }),
    },
  })
);

maxReceiveCount uses ApproximateReceiveCount on the message. When a consumer receives a message and does not delete it (either because processing fails or because the visibility timeout expires), SQS increments this counter. After maxReceiveCount receives, the message moves to the DLQ instead of becoming visible again.

The DLQ should be the same type as the source queue: FIFO sources require FIFO DLQs. The DLQ is not a magic catch-all: if your consumer panics and crashes without calling DeleteMessage, the message accumulates receive counts and eventually lands in the DLQ as expected. But if your consumer successfully processes the message but throws an exception before calling DeleteMessage, the message also accumulates receive counts. Both scenarios look identical from SQS’s perspective. Your DLQ monitoring should alert on any non-zero depth, and your DLQ handler should log enough context to distinguish poison messages from transient processing failures.

SQS DLQ redrive (available in the console and via API) lets you replay DLQ messages back to the source queue after you’ve resolved the underlying issue, without requiring manual re-publishing.

The Full Message Lifecycle

SendMessage
    |
    v
Message stored, replicated across AZs
    |
    v
ReceiveMessage --> message becomes invisible (VisibilityTimeout starts)
    |
    +-- Processing succeeds --> DeleteMessage --> message permanently removed
    |
    +-- Processing fails / timeout expires --> message becomes visible again
    |       |
    |       v
    |   ApproximateReceiveCount incremented
    |       |
    |       +-- count < maxReceiveCount --> message available for retry
    |       |
    |       +-- count >= maxReceiveCount --> message moved to DLQ
    |
    +-- MessageRetentionPeriod expires --> message deleted (default: 4 days, max: 14 days)

One often-missed detail: the message retention period runs from the time the message was sent to SQS, not from the time it was last received. A message that has been in your DLQ for 13 days against a 14-day retention period will disappear one day after it was originally sent, even if you just routed it there.

Batching APIs

SQS supports batch operations for SendMessage, ReceiveMessage, and DeleteMessage. Batching is the single most impactful optimization for throughput and cost.

import {
  SendMessageBatchCommand,
  DeleteMessageBatchCommand,
} from "@aws-sdk/client-sqs";

// Send up to 10 messages in a single API call
await client.send(
  new SendMessageBatchCommand({
    QueueUrl: QUEUE_URL,
    Entries: [
      { Id: "1", MessageBody: JSON.stringify({ event: "user.created", userId: "u_001" }) },
      { Id: "2", MessageBody: JSON.stringify({ event: "user.created", userId: "u_002" }) },
      { Id: "3", MessageBody: JSON.stringify({ event: "user.created", userId: "u_003" }) },
    ],
  })
);

// Delete up to 10 messages after successful processing
await client.send(
  new DeleteMessageBatchCommand({
    QueueUrl: QUEUE_URL,
    Entries: messages.map((msg) => ({
      Id: msg.MessageId!,
      ReceiptHandle: msg.ReceiptHandle!,
    })),
  })
);

ReceiveMessage already returns up to 10 messages per call. The batch send and batch delete round out the picture. Batch operations reduce API call count by up to 10x at the same message volume, which translates directly to cost reduction. For high-throughput producers, batch sends also reduce the latency overhead of individual round trips.

One nuance: SendMessageBatch responses include both Successful and Failed arrays. Partial failures are possible. Your producer must check Failed and retry those entries. The Id field you provide in each batch entry is your correlation handle for matching failures back to the original messages.

Message Attributes and Metadata

SQS messages carry up to 10 user-defined attributes alongside the body. Attributes are typed (String, Number, or Binary) and available to consumers without parsing the body. This is useful for routing metadata, priority signals, or source identifiers:

await client.send(
  new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify(payload),
    MessageAttributes: {
      EventType: { DataType: "String", StringValue: "order.shipped" },
      Priority:  { DataType: "Number", StringValue: "1" },
      TenantId:  { DataType: "String", StringValue: "tenant-acme" },
    },
  })
);

To receive attributes, specify them in MessageAttributeNames on the receive call (["All"] to receive all). Attributes count toward the 256 KB message size limit, so keep them lean.

Delay Queues and Message Timers

Delay queues defer all messages by a fixed period (0 to 15 minutes) before they become visible. You set DelaySeconds at the queue level. Message timers override the queue-level delay for individual messages via the DelaySeconds parameter on SendMessage. FIFO queues do not support per-message timers, only queue-level delays.

Delay queues are useful for retry patterns where you want to backoff before reprocessing, or for scheduled job patterns where the producer knows the job should not run immediately. For longer delays or cron-style scheduling, delay queues fall short and you need a purpose-built scheduler.

Server-Side Encryption

SQS supports server-side encryption (SSE) using AWS KMS. You can use an AWS-managed key (alias/aws/sqs) at no extra cost, or a customer-managed key (CMK) for key rotation control and cross-account access policies. Encryption applies at rest; messages are encrypted when written to SQS’s storage layer and decrypted transparently when delivered.

The operational tradeoff with CMKs: every SendMessage, ReceiveMessage, and DeleteMessage call generates a KMS API call. KMS has its own throughput limits and costs. At high message volumes with a CMK, KMS can become a throughput bottleneck and a non-trivial cost line. Enable KMS key caching in your SDK clients where supported, and monitor KMSThrottling errors in CloudWatch if you’re seeing unexpected latency.

Production Considerations

Set visibility timeout generously, not optimistically. The right visibility timeout is the 99th percentile of your actual processing time plus a buffer. Setting it to the average processing time means that any slow outlier will cause double processing. Use ChangeMessageVisibility to extend dynamically for jobs with unpredictable duration.

Monitor ApproximateNumberOfMessagesNotVisible. This CloudWatch metric shows messages currently in flight (received but not deleted). A growing NotVisible count with stagnant Visible count indicates consumers are receiving messages but not processing or deleting them, which often signals a consumer-side bug or an undersized visibility timeout.

Use attribute filters on the DLQ. When replaying from the DLQ, add filtering logic to identify and skip true poison messages rather than blindly redriving everything. Otherwise a single broken message format can crash your consumer repeatedly.

Beware FIFO queue throughput limits at scale. If you’re routing all events through a single MessageGroupId, you’ve effectively created a single-threaded queue. Use a meaningful business key (customer ID, order ID, tenant ID) as the group ID so that unrelated entities can process in parallel.

Tune batch size and concurrency together. With Lambda triggers on SQS, the effective parallelism is (concurrency * batchSize). If your function takes 5 seconds per batch and your BatchSize is 10 with 50 concurrent executions, you’re processing 500 messages every 5 seconds. Tune Lambda reserved concurrency and MaximumConcurrency on the event source mapping to match your downstream capacity.

Tradeoffs

Dimension	SQS	Kafka	RabbitMQ	NATS JetStream	Redpanda
Ordering	Best-effort (standard); strict per group (FIFO)	Strict per partition	Exchange-dependent	Strict per stream subject	Strict per partition
Persistence	Up to 14 days, managed	Log-based, configurable retention	Per-queue durability flags	File-backed, configurable retention	Log-based, configurable retention
Consumer model	Competing consumers, at-least-once; FIFO adds exactly-once	Consumer groups, offset-based replay	Push or pull, ack-based	Push-based, ack with replay	Consumer groups, offset-based replay
Throughput	Nearly unlimited (standard); ~3K/s (FIFO)	Millions/s with correct partition count	Tens of thousands/s per queue	Hundreds of thousands/s	Millions/s
Operational complexity	None (fully managed)	High (ZooKeeper/KRaft, broker tuning, topic management)	Medium (broker, HA with quorum queues)	Low to medium (embedded or server)	Medium (managed or self-hosted)
Cost model	Per request + data transfer, no idle cost	Cluster compute + storage (MSK or self-host)	Cluster compute + storage	Cluster compute or NATS Cloud	Cluster compute or Redpanda Cloud
Replay	No native replay (DLQ redrive only)	Full log replay by consumer group	No native replay	Full replay within retention window	Full log replay
Sweet spot	Serverless workloads, AWS-native event processing, decoupled microservices with no replay requirements	High-throughput streaming, event sourcing, audit logs, exactly-once pipelines	Complex routing logic, RPC patterns, per-message routing decisions	Low-latency pub/sub and streaming with replay in a lightweight footprint	Kafka API compatibility without ZooKeeper operational burden

SQS fits when you want decoupled asynchronous processing with zero operational overhead and you don’t need replay or complex routing. Its durability and AWS integration (Lambda triggers, SNS fanout, EventBridge pipes) make it the lowest-friction choice for AWS-native architectures. The moment you need full message history replay, ordered event streams with multiple independent consumer offsets, or throughput beyond what FIFO queues support, Kafka or Redpanda serve better. RabbitMQ wins when routing logic is the primary concern. NATS JetStream sits between SQS and Kafka in complexity, offering replay with a significantly simpler operational model than Kafka.

The default SQS choice for most teams building on AWS is correct. Just understand what you’re giving up: replay is not coming, ordering requires accepting throughput limits, and exactly-once delivery requires FIFO queues plus idempotent consumers. Design for those constraints from the start rather than discovering them under production load.