Retry Patterns in Distributed Systems: Exponential Backoff, Jitter, and Dead Letter Queues

A payment service times out. Your code retries. The payment service was actually slow, not down, and the first request succeeded. Now you have charged the customer twice. That is the most expensive class of retry bug, and it is common enough that entire patterns exist around preventing it.

Retries are the most misunderstood resilience primitive in distributed systems. The concept is obvious. The implementation details are where systems fail. When to retry, how long to wait, how many times to try, which errors are retryable, and what to do when nothing works: these are the decisions that separate a system that recovers gracefully from one that amplifies failures.

The Four Retry Strategies

The strategies differ in how they calculate the delay between attempts. The choice matters more than it first appears.

Immediate Retry

Retry immediately after a failure, with no delay.

async function withImmediateRetry<T>(
  fn: () => Promise<T>,
  maxAttempts: number
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err as Error;
      if (attempt === maxAttempts) break;
      // no delay: next iteration fires immediately
    }
  }

  throw lastError!;
}

Immediate retry is appropriate for one narrow case: a transient error that has no external cause and will very likely be gone in microseconds (a lost packet in a local retry loop, for example). In practice, that case is rare. For network calls, database queries, or any external dependency, immediate retry creates a thundering herd: every client that received a failure hammers the struggling dependency at exactly the same time.

Use this only when the dependency is local and sub-millisecond retry is actually meaningful.

Fixed Delay

Wait a constant amount of time between each attempt.

async function withFixedDelay<T>(
  fn: () => Promise<T>,
  maxAttempts: number,
  delayMs: number
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err as Error;
      if (attempt === maxAttempts) break;
      await sleep(delayMs);
    }
  }

  throw lastError!;
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

Fixed delay is better than immediate retry. It gives the dependency time to recover. The problem is coordination: if 1,000 clients all failed at the same moment and all use a 1-second fixed delay, they retry in a synchronized wave. The dependency gets another spike at t+1s, then t+2s. You have converted one traffic spike into a series of them.

Fixed delay works well in scenarios with few clients or low contention. For anything at scale, you need jitter.

Exponential Backoff

Double the delay with each failed attempt (or multiply by some factor). The client backs off progressively, giving the dependency more time to recover.

interface BackoffConfig {
  baseDelayMs: number;
  maxDelayMs: number;
  factor?: number; // default 2 (doubling)
  maxAttempts: number;
}

async function withExponentialBackoff<T>(
  fn: () => Promise<T>,
  config: BackoffConfig
): Promise<T> {
  const { baseDelayMs, maxDelayMs, factor = 2, maxAttempts } = config;
  let lastError: Error;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err as Error;
      if (attempt === maxAttempts) break;

      // cap the delay so it does not grow unbounded
      const delay = Math.min(
        baseDelayMs * Math.pow(factor, attempt - 1),
        maxDelayMs
      );

      await sleep(delay);
    }
  }

  throw lastError!;
}

Attempt 1: 0ms wait, attempt 2: 100ms, attempt 3: 200ms, attempt 4: 400ms, attempt 5: 800ms (with base=100ms and factor=2). The maxDelayMs cap prevents the delay from growing to minutes, which would make the retry useless in practice.

Pure exponential backoff is much better than fixed delay. But it still has the coordination problem. All clients that failed at the same moment start with the same base delay and apply the same formula. They stay synchronized across every retry wave.

Exponential Backoff with Jitter

Add randomness to the delay. Clients that failed together now spread their retries across a time window instead of firing in unison.

interface JitterConfig extends BackoffConfig {
  jitter?: "full" | "equal"; // default "full"
}

function calculateDelay(attempt: number, config: JitterConfig): number {
  const { baseDelayMs, maxDelayMs, factor = 2, jitter = "full" } = config;

  const exponentialDelay = Math.min(
    baseDelayMs * Math.pow(factor, attempt - 1),
    maxDelayMs
  );

  if (jitter === "full") {
    // uniform random between 0 and the exponential delay
    // aggressive desynchronization, but average delay is half the exponential value
    return Math.random() * exponentialDelay;
  }

  if (jitter === "equal") {
    // random between half and full of the exponential delay
    // still spreads clients, but keeps a minimum delay floor
    return exponentialDelay / 2 + Math.random() * (exponentialDelay / 2);
  }

  return exponentialDelay;
}

async function withJitteredBackoff<T>(
  fn: () => Promise<T>,
  config: JitterConfig
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err as Error;
      if (attempt === config.maxAttempts) break;

      const delay = calculateDelay(attempt, config);
      await sleep(delay);
    }
  }

  throw lastError!;
}

Full jitter (uniform random from 0 to the exponential cap) is the most aggressive desynchronization. Equal jitter (random from half to full) gives you a minimum floor, which can matter if the dependency needs a minimum recovery window before it is worth probing again.

AWS published analysis on this in 2015, and the conclusion has held up: full jitter produces the best overall throughput under contention because it spreads load most evenly. Equal jitter is a reasonable choice when you want to guarantee a minimum wait.

Pattern Comparison

Pattern	Avg Load on Dependency	Client Wait Time	Desynchronization
Immediate retry	Very high	Near zero	None
Fixed delay	High (waves)	Predictable	None
Exponential backoff	Medium (waves)	Grows fast	None
Full jitter	Low	Moderate	High
Equal jitter	Low	Moderate + floor	High

For any production system with more than a handful of clients, exponential backoff with full jitter is the correct default. The others are useful in specific contexts but harmful at scale.

Idempotency: The Prerequisite for Safe Retries

Before you can safely retry anything, the operation must be idempotent: calling it multiple times must produce the same result as calling it once.

Read operations (GET requests, database selects) are naturally idempotent. Side-effecting operations are not, unless you make them so.

The standard approach is idempotency keys. The client generates a unique key per logical operation and sends it with the request. The server stores a record of processed keys and their results. If the same key arrives again (from a retry), the server returns the stored result instead of processing again.

The idempotency keys article covers this in depth, including the database schema and atomic check-and-insert patterns. The key point for retry design: you cannot safely retry non-idempotent operations. If you cannot make the operation idempotent, you must not retry it automatically.

The error types tell you most of what you need to know:

type RetryDecision = "retry" | "no-retry" | "retry-with-idempotency-check";

function classifyError(err: unknown): RetryDecision {
  if (!(err instanceof Error)) return "no-retry";

  // connection-level errors: network not available, DNS failure
  // safe to retry because the request never reached the server
  if (isNetworkError(err)) return "retry";

  if ("status" in err) {
    const status = (err as { status: number }).status;

    // 408 Request Timeout, 429 Too Many Requests, 503 Service Unavailable:
    // transient server-side conditions, safe to retry
    if ([408, 429, 503].includes(status)) return "retry";

    // 500 Internal Server Error: ambiguous — server received the request
    // but we do not know if it was processed. Requires idempotency.
    if (status === 500) return "retry-with-idempotency-check";

    // 400, 401, 403, 404, 422: client error, retrying will not help
    return "no-retry";
  }

  // timeout errors from your HTTP client:
  // the server may or may not have processed the request.
  // treat as ambiguous and require idempotency.
  if (isTimeoutError(err)) return "retry-with-idempotency-check";

  return "no-retry";
}

The ambiguous case (timeout, 500) is where most retry bugs happen. The request reached the server. Whether it was processed is unknown. Retrying without idempotency means potentially executing the operation twice. Retrying with an idempotency key means the server can detect the duplicate and return the cached result.

Circuit Breaker Integration

Retries and circuit breakers solve different problems. Retries handle transient failures in an otherwise healthy dependency. Circuit breakers handle systemic failures where a dependency is down or overloaded and retrying will make things worse.

The circuit breaker article covers the state machine in detail. The important interaction to understand here: retry logic should respect the circuit breaker state.

class RetryableCircuitBreaker {
  constructor(
    private breaker: CircuitBreaker,
    private retryConfig: JitterConfig
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    let lastError: Error;

    for (let attempt = 1; attempt <= this.retryConfig.maxAttempts; attempt++) {
      // check the breaker state before each attempt
      // if the circuit is open, fail fast without calling fn()
      if (!this.breaker.allowRequest()) {
        throw new Error("Circuit open: dependency unavailable");
      }

      try {
        const result = await fn();
        this.breaker.recordSuccess();
        return result;
      } catch (err) {
        lastError = err as Error;
        this.breaker.recordFailure();

        const decision = classifyError(err);

        if (decision === "no-retry") throw err;

        // if the breaker opened during this attempt, stop retrying
        // further attempts will only drive failure counts higher
        if (!this.breaker.allowRequest()) {
          throw new Error(`Circuit opened after failure: ${lastError.message}`);
        }

        if (attempt < this.retryConfig.maxAttempts) {
          const delay = calculateDelay(attempt, this.retryConfig);
          await sleep(delay);
        }
      }
    }

    throw lastError!;
  }
}

Without this integration, retry logic and circuit breakers fight each other. The circuit breaker opens because failure counts exceed the threshold. Your retry logic keeps calling the dependency, which keeps recording failures and keeping the circuit open. The check-before-attempt pattern short-circuits this: once the breaker opens, retries stop.

The breaker should be shared across all callers of a dependency, not instantiated per request. A per-request breaker has no memory of previous failures and cannot protect anyone.

Dead Letter Queues for Permanent Failures

Not every failure is transient. Some messages will never succeed: the payload is malformed, the referenced resource was deleted, the operation is logically impossible. Retrying these indefinitely wastes resources and blocks queue progress.

A dead letter queue (DLQ) captures these permanently failed messages so they can be inspected, corrected, or replayed later. The pattern applies to any message queue (Kafka, SQS, RabbitMQ), but the mechanics are similar.

interface MessageProcessor<T> {
  process(message: T): Promise<void>;
  deserialize(raw: string): T;
}

interface DLQConfig {
  maxAttempts: number;
  backoffConfig: JitterConfig;
  dlqName: string;
}

async function processWithDLQ<T>(
  rawMessage: string,
  currentAttempt: number,
  processor: MessageProcessor<T>,
  queue: QueueClient,
  config: DLQConfig
): Promise<void> {
  let message: T;

  try {
    message = processor.deserialize(rawMessage);
  } catch (err) {
    // deserialization failure is always permanent: send to DLQ immediately
    await queue.sendToDLQ(config.dlqName, {
      raw: rawMessage,
      failureReason: "deserialization_failure",
      error: (err as Error).message,
      timestamp: new Date().toISOString(),
    });
    return;
  }

  try {
    await processor.process(message);
  } catch (err) {
    const decision = classifyError(err);

    if (decision === "no-retry" || currentAttempt >= config.maxAttempts) {
      // permanent failure or exhausted retries: send to DLQ
      await queue.sendToDLQ(config.dlqName, {
        raw: rawMessage,
        failureReason:
          currentAttempt >= config.maxAttempts
            ? "max_attempts_exceeded"
            : "permanent_failure",
        error: (err as Error).message,
        attemptCount: currentAttempt,
        timestamp: new Date().toISOString(),
      });
      return;
    }

    // transient failure within attempt budget: requeue with delay metadata
    const delay = calculateDelay(currentAttempt, config.backoffConfig);
    await queue.requeue(rawMessage, {
      attempt: currentAttempt + 1,
      notBefore: Date.now() + delay,
    });
  }
}

The DLQ entry must carry enough context to debug the failure later: the original message, the error, the attempt count, and a timestamp. Without this metadata, a DLQ is a black hole.

DLQ Monitoring and Reprocessing

A DLQ without monitoring is useless. Messages accumulate silently and the first signal you get is a customer complaint.

interface DLQMonitor {
  queueDepth(): Promise<number>;
  oldestMessageAge(): Promise<number>; // milliseconds
  failureReasonBreakdown(): Promise<Record<string, number>>;
}

async function checkDLQHealth(monitor: DLQMonitor): Promise<void> {
  const depth = await monitor.queueDepth();
  const ageMs = await monitor.oldestMessageAge();
  const reasons = await monitor.failureReasonBreakdown();

  metrics.gauge("dlq.depth", depth);
  metrics.gauge("dlq.oldest_message_age_seconds", ageMs / 1000);

  for (const [reason, count] of Object.entries(reasons)) {
    metrics.gauge("dlq.failure_reason", count, { reason });
  }

  // alert on depth or age thresholds
  if (depth > 100) {
    alerts.fire("dlq_depth_high", { depth });
  }

  if (ageMs > 4 * 60 * 60 * 1000) {
    // 4 hours
    alerts.fire("dlq_message_stale", { ageMs });
  }
}

Alert on depth and age, not just depth. A DLQ with five messages that are three days old is a bigger problem than one with twenty messages that arrived in the last hour.

Reprocessing DLQ messages after fixing the underlying bug is an operational event, not a feature. Build the tooling before you need it. You want a way to replay individual messages or a batch with filtering by failure reason, not just “flush everything back to the main queue.”

Production Considerations

Retry Budgets

Retries amplify load. Under failure conditions, a system that retries three times per request is sending three times as many requests to an already struggling dependency. At scale, this converts a partial failure into a full outage.

A retry budget limits total retry volume. Instead of “each request may retry N times,” you track the total number of retries in flight and cap it.

class RetryBudget {
  private used = 0;

  constructor(
    private budget: number, // max concurrent retries
    private resetIntervalMs: number
  ) {
    // reset the budget periodically
    setInterval(() => {
      this.used = 0;
    }, resetIntervalMs);
  }

  canRetry(): boolean {
    return this.used < this.budget;
  }

  consume(): void {
    this.used++;
  }

  release(): void {
    this.used = Math.max(0, this.used - 1);
  }
}

async function withRetryBudget<T>(
  fn: () => Promise<T>,
  budget: RetryBudget,
  config: JitterConfig
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err as Error;

      const decision = classifyError(err);
      if (decision === "no-retry") throw err;

      if (attempt === config.maxAttempts) break;

      // only retry if the budget allows it
      if (!budget.canRetry()) {
        metrics.increment("retry.budget_exhausted");
        throw new Error(
          `Retry budget exhausted after attempt ${attempt}: ${lastError.message}`
        );
      }

      budget.consume();
      try {
        const delay = calculateDelay(attempt, config);
        await sleep(delay);
      } finally {
        budget.release();
      }
    }
  }

  throw lastError!;
}

The budget should be sized based on your normal retry rate plus a headroom buffer. If 1% of requests normally retry once, a budget of 5% of your request rate gives you room for elevated failure without cascading. Start conservative and adjust based on metrics.

Observability

Retries that you cannot see are retries you cannot tune.

interface RetryMetrics {
  attempt: number;
  maxAttempts: number;
  delayMs: number;
  errorCode: string;
  operation: string;
  success: boolean;
}

function recordRetryAttempt(m: RetryMetrics): void {
  metrics.increment("retry.attempt", {
    operation: m.operation,
    attempt: String(m.attempt),
    error_code: m.errorCode,
  });

  if (m.attempt > 1) {
    metrics.histogram("retry.delay_ms", m.delayMs, {
      operation: m.operation,
    });
  }

  if (m.success) {
    metrics.increment("retry.success", {
      operation: m.operation,
      recovered_on_attempt: String(m.attempt),
    });
  }

  if (m.attempt === m.maxAttempts && !m.success) {
    metrics.increment("retry.exhausted", {
      operation: m.operation,
    });
  }
}

Track: attempt number per operation, delay values, which attempt a recovery happened on, and exhaustion events. The “recovered on attempt N” metric tells you whether your retry config is calibrated correctly. If 90% of recoveries happen on attempt 2 and your max is 5, you can reduce the max. If recoveries are spread across all attempts, you have real intermittent failures.

Propagating Deadlines

Retries consume time. A caller with a 5-second timeout that spawns a callee doing three retries with 2-second delays will see every call time out, not just the ones that actually fail.

Use context propagation or explicit deadlines to ensure retries stay within the budget available from the caller:

async function withDeadlineAwareRetry<T>(
  fn: () => Promise<T>,
  deadlineMs: number, // absolute timestamp, not relative duration
  config: JitterConfig
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    const remaining = deadlineMs - Date.now();

    if (remaining <= 0) {
      throw new Error(`Deadline exceeded before attempt ${attempt}`);
    }

    try {
      // pass remaining time to the operation so it can set its own timeout
      return await fn();
    } catch (err) {
      lastError = err as Error;

      if (attempt === config.maxAttempts) break;

      const delay = calculateDelay(attempt, config);
      const remainingAfterDelay = deadlineMs - Date.now() - delay;

      if (remainingAfterDelay <= 0) {
        // no point in waiting: we would exceed the deadline before the next attempt
        throw new Error(
          `Insufficient time for retry after attempt ${attempt}: deadline in ${deadlineMs - Date.now()}ms`
        );
      }

      await sleep(delay);
    }
  }

  throw lastError!;
}

This pattern prevents the situation where retries outlive the request context and run as orphans after the caller has already returned a failure to the end user.

Closing Thoughts

The retry pattern looks simple. It is not. The common implementation errors are predictable: retrying non-idempotent operations, missing jitter, retrying client errors that will never succeed, and not capping total retry volume.

The patterns in this article compose. Exponential backoff with jitter desynchronizes clients. The circuit breaker stops retries when the dependency is systemically down. The retry budget limits retry amplification. Dead letter queues handle permanent failures without blocking progress. Idempotency keys make it safe to retry ambiguous failures.

None of these patterns work in isolation as well as they work together. Build the retry layer, then add the circuit breaker, then add the DLQ. Instrument all of it from the start. The metrics will tell you where to tune.