Designing a Webhook Delivery System: Reliable Outbound Events, Retry Logic, and Endpoint Management at Scale

A customer’s payment succeeds. Your platform fires a payment.completed event. The webhook hits the customer’s endpoint. The endpoint returns a 200 and their fulfillment flow kicks off. That is the happy path, and it takes about 200ms.

Now consider what happens when the endpoint is slow, returning 500s, or simply offline. Your code fires the request, waits, times out, and moves on. The customer’s fulfillment flow never runs. They raise a support ticket three hours later. You have no delivery log, no retry history, no way to tell them what happened. This is where most webhook implementations live: HTTP calls in the request path with no queuing, no retry, and no visibility.

Building a production webhook delivery system means building a durable outbound event pipeline. That is a different problem than making HTTP calls.

The Core Architecture

The system has three distinct layers:

1. Event ingestion: capturing what happened, durably, before any delivery attempt
2. Delivery worker: consuming events, attempting delivery, managing retries
3. Endpoint management: storing subscriber endpoints, secrets, and delivery state

These layers must be decoupled. If the delivery attempt fails, the event must not be lost. That means the event exists in a queue or database before you ever make an HTTP request.

Event Schema

Every event needs a stable schema. Consumers depend on field names and types across versions.

interface WebhookEvent {
  id: string;           // globally unique, idempotency key
  topic: string;        // e.g. "payment.completed"
  tenantId: string;
  payload: Record<string, unknown>;
  createdAt: string;    // ISO 8601
  attemptCount: number;
  nextAttemptAt: string | null;
  status: "pending" | "delivered" | "failed" | "dead";
  endpointId: string;
}

The id field is non-negotiable. Subscribers use it to deduplicate events on their side. If a retry delivers an event a second time, the subscriber needs a way to detect it and skip the duplicate processing. Without a stable event ID, at-least-once delivery becomes “you might process this twice and have no idea.”

Durably Enqueuing Events

When a domain event fires (payment completed, user invited, subscription changed), write the webhook event to a database table before returning. Do not fire the HTTP request in the same transaction.

async function enqueueWebhookEvent(
  db: Database,
  topic: string,
  tenantId: string,
  payload: Record<string, unknown>
): Promise<void> {
  const endpoints = await db.query<Endpoint>(
    `SELECT id FROM webhook_endpoints
     WHERE tenant_id = $1 AND enabled = true
     AND $2 = ANY(subscribed_topics)`,
    [tenantId, topic]
  );

  const events = endpoints.map((endpoint) => ({
    id: crypto.randomUUID(),
    topic,
    tenantId,
    payload,
    createdAt: new Date().toISOString(),
    attemptCount: 0,
    nextAttemptAt: new Date().toISOString(),
    status: "pending" as const,
    endpointId: endpoint.id,
  }));

  if (events.length > 0) {
    await db.insertMany("webhook_events", events);
  }
}

One domain event can fan out to multiple endpoints. Create one WebhookEvent row per endpoint. This gives you per-endpoint delivery tracking without joining across event and endpoint state.

Delivery Worker and Retry Logic

The delivery worker polls for pending events where nextAttemptAt <= now, attempts delivery, and updates the event record based on the result.

Exponential Backoff with Jitter

Retry delays must grow exponentially, and they must include jitter. Without jitter, all the endpoints that failed in the same minute retry at the same second after the backoff window, creating a synchronized spike of outbound traffic.

function calculateNextAttempt(attemptCount: number): Date {
  const BASE_DELAY_MS = 30_000;        // 30 seconds
  const MAX_DELAY_MS = 3_600_000;      // 1 hour
  const JITTER_FACTOR = 0.25;

  const exponential = BASE_DELAY_MS * Math.pow(2, attemptCount);
  const capped = Math.min(exponential, MAX_DELAY_MS);
  const jitter = capped * JITTER_FACTOR * Math.random();

  return new Date(Date.now() + capped + jitter);
}

The jitter is added on top of the capped delay, not subtracted from it. Adding jitter means retries spread out over the jitter window rather than collapsing to a single point. Subtracting would reduce the backoff for some events below the intended minimum.

Typical retry schedule with these parameters:

Attempt	Base delay	With jitter
1	30s	30-38s
2	1m	1m-1m15s
3	2m	2m-2m30s
4	4m	4m-5m
5	8m	8m-10m
10	1h (cap)	1h-1h15m

The Delivery Attempt

async function attemptDelivery(
  event: WebhookEvent,
  endpoint: Endpoint
): Promise<"delivered" | "retryable" | "dead"> {
  const MAX_ATTEMPTS = 10;
  const body = JSON.stringify({
    id: event.id,
    topic: event.topic,
    createdAt: event.createdAt,
    data: event.payload,
  });

  const signature = signPayload(body, endpoint.signingSecret);
  const attemptedAt = new Date().toISOString();

  let responseStatus: number | null = null;
  let responseBody: string | null = null;
  let durationMs: number;

  const start = Date.now();
  try {
    const response = await fetch(endpoint.url, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "X-Webhook-ID": event.id,
        "X-Webhook-Timestamp": attemptedAt,
        "X-Webhook-Signature": signature,
      },
      body,
      signal: AbortSignal.timeout(10_000), // 10 second timeout
    });

    durationMs = Date.now() - start;
    responseStatus = response.status;
    responseBody = await response.text().catch(() => null);

    if (response.ok) {
      return "delivered";
    }

    // 4xx errors (except 429) are not retryable: the endpoint rejected the payload
    if (response.status >= 400 && response.status < 500 && response.status !== 429) {
      return "dead";
    }

    // 5xx and 429 are retryable
    return event.attemptCount + 1 >= MAX_ATTEMPTS ? "dead" : "retryable";
  } catch (err) {
    durationMs = Date.now() - start;
    // Network errors and timeouts are retryable
    return event.attemptCount + 1 >= MAX_ATTEMPTS ? "dead" : "retryable";
  } finally {
    await logDeliveryAttempt({
      eventId: event.id,
      endpointId: event.endpointId,
      attemptedAt,
      responseStatus,
      responseBody,
      durationMs,
    });
  }
}

The distinction between retryable and non-retryable failures matters. A 400 means the payload was rejected: retrying will not help. A 500 means the endpoint had a problem: retrying might succeed. A 429 is the endpoint telling you to back off.

Updating Event State

async function processEvent(db: Database, event: WebhookEvent): Promise<void> {
  const endpoint = await db.findOne<Endpoint>("webhook_endpoints", { id: event.endpointId });
  if (!endpoint || !endpoint.enabled) {
    await db.update("webhook_events", { id: event.id }, { status: "dead" });
    return;
  }

  const result = await attemptDelivery(event, endpoint);
  const newAttemptCount = event.attemptCount + 1;

  if (result === "delivered") {
    await db.update("webhook_events", { id: event.id }, {
      status: "delivered",
      attemptCount: newAttemptCount,
      nextAttemptAt: null,
    });
  } else if (result === "retryable") {
    await db.update("webhook_events", { id: event.id }, {
      status: "pending",
      attemptCount: newAttemptCount,
      nextAttemptAt: calculateNextAttempt(newAttemptCount).toISOString(),
    });
  } else {
    await db.update("webhook_events", { id: event.id }, {
      status: "dead",
      attemptCount: newAttemptCount,
      nextAttemptAt: null,
    });
    await moveToDLQ(db, event);
  }
}

Security: HMAC Signing and Replay Protection

Every webhook payload must be signed so subscribers can verify it came from your platform and was not tampered with in transit.

Signing Payloads

import { createHmac } from "crypto";

function signPayload(body: string, secret: string): string {
  const timestamp = Date.now().toString();
  const signed = `${timestamp}.${body}`;
  const hmac = createHmac("sha256", secret)
    .update(signed)
    .digest("hex");
  return `t=${timestamp},v1=${hmac}`;
}

function verifySignature(
  body: string,
  signatureHeader: string,
  secret: string,
  toleranceMs = 300_000 // 5 minutes
): boolean {
  const parts = Object.fromEntries(
    signatureHeader.split(",").map((p) => p.split("=") as [string, string])
  );

  const timestamp = parts["t"];
  const received = parts["v1"];
  if (!timestamp || !received) return false;

  // Reject events older than tolerance window
  if (Math.abs(Date.now() - parseInt(timestamp, 10)) > toleranceMs) {
    return false;
  }

  const signed = `${timestamp}.${body}`;
  const expected = createHmac("sha256", secret)
    .update(signed)
    .digest("hex");

  // Constant-time comparison to prevent timing attacks
  return timingSafeEqual(Buffer.from(received), Buffer.from(expected));
}

The timestamp in the signature is what provides replay protection. An attacker who captures a valid request and replays it 10 minutes later will fail the timestamp check. The tolerance window (5 minutes by default) accounts for clock drift between your servers and the subscriber’s.

Use crypto.timingSafeEqual for the comparison. A standard string equality check is vulnerable to timing attacks: an attacker can measure how long the comparison takes to determine how many leading characters match.

Each endpoint gets its own signing secret, rotatable independently. If a subscriber’s secret is compromised, you can rotate it without affecting any other endpoint.

Per-Endpoint Rate Limiting

A subscriber’s endpoint might only be able to handle 10 requests per second. Without rate limiting, a burst of events on your platform would overwhelm their infrastructure. This is not a theoretical concern: any subscriber whose webhook handling is synchronous and slow will drop 5xx responses under burst load, causing cascading retries.

interface RateLimitConfig {
  requestsPerSecond: number;
  burstCapacity: number;
}

class TokenBucketLimiter {
  private tokens: number;
  private lastRefill: number;

  constructor(private config: RateLimitConfig) {
    this.tokens = config.burstCapacity;
    this.lastRefill = Date.now();
  }

  tryConsume(): boolean {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(
      this.config.burstCapacity,
      this.tokens + elapsed * this.config.requestsPerSecond
    );
    this.lastRefill = now;

    if (this.tokens >= 1) {
      this.tokens -= 1;
      return true;
    }
    return false;
  }
}

In practice, per-endpoint rate limit state should live in Redis rather than in worker memory, because multiple worker instances share the delivery load for any given endpoint. A Redis-backed sliding window or token bucket ensures the limit is enforced globally across workers.

The delivery worker checks the limiter before making the request. If the limiter rejects the attempt, reschedule the event slightly into the future rather than retrying immediately.

Dead Letter Queue

Events that exhaust all retry attempts move to a dead letter queue. The DLQ is not just a graveyard: it is where operators debug delivery failures and decide whether to replay events.

interface DLQEntry {
  eventId: string;
  endpointId: string;
  topic: string;
  payload: Record<string, unknown>;
  failedAt: string;
  attemptCount: number;
  lastResponseStatus: number | null;
  lastError: string | null;
}

async function moveToDLQ(db: Database, event: WebhookEvent): Promise<void> {
  const lastAttempt = await db.findOne<DeliveryAttemptLog>(
    "webhook_delivery_attempts",
    { eventId: event.id },
    { orderBy: "attemptedAt", direction: "desc" }
  );

  await db.insert("webhook_dlq", {
    eventId: event.id,
    endpointId: event.endpointId,
    topic: event.topic,
    payload: event.payload,
    failedAt: new Date().toISOString(),
    attemptCount: event.attemptCount,
    lastResponseStatus: lastAttempt?.responseStatus ?? null,
    lastError: lastAttempt?.error ?? null,
  });
}

The DLQ entry should include the full payload so operators can replay it without needing to reconstruct what the original event contained. Replay means re-inserting the event into the pending queue with attemptCount: 0 and nextAttemptAt: now.

Expose a replay endpoint in your management API. Do not make operators re-send test events from their side to recover from delivery failures.

Observability

Webhook delivery systems fail silently in ways that are easy to miss. An endpoint that consistently returns 500s will exhaust retries over several hours. By the time it hits the DLQ, the subscriber is already having a bad day. Good observability surfaces the problem early.

Metrics to Instrument

Metric	Description	Alert threshold
webhook.delivery.success_rate	Delivered / attempted per endpoint	Alert if < 95% over 15m
webhook.delivery.latency_p99	Time from event creation to first successful delivery	Alert if > 30s
webhook.delivery.attempt_count	Distribution of attempts per delivered event	Alert if p90 > 3
webhook.dlq.size	Events in DLQ per endpoint	Alert if > 0 for production endpoints
webhook.queue.depth	Pending events older than 5 minutes	Alert if > 100
webhook.endpoint.consecutive_failures	Count of consecutive failures per endpoint	Auto-disable at 50

Track success rate per endpoint, not globally. A global success rate of 99.9% hides a single endpoint with a 0% rate. Per-endpoint metrics let you surface unhealthy endpoints immediately and present them to the subscriber in your dashboard.

Auto-Disabling Unhealthy Endpoints

An endpoint that returns 500s for every request over many hours is still consuming retry capacity and DLQ space. Consider automatically disabling endpoints after a threshold of consecutive failures, and notifying the subscriber.

async function checkEndpointHealth(db: Database, endpointId: string): Promise<void> {
  const DISABLE_THRESHOLD = 50;

  const recentAttempts = await db.query<DeliveryAttemptLog>(
    `SELECT response_status FROM webhook_delivery_attempts
     WHERE endpoint_id = $1
     ORDER BY attempted_at DESC
     LIMIT $2`,
    [endpointId, DISABLE_THRESHOLD]
  );

  if (recentAttempts.length < DISABLE_THRESHOLD) return;

  const allFailed = recentAttempts.every(
    (a) => a.responseStatus === null || a.responseStatus >= 400
  );

  if (allFailed) {
    await db.update("webhook_endpoints", { id: endpointId }, { enabled: false });
    await notifySubscriber(endpointId, "endpoint_auto_disabled");
  }
}

Auto-disabling prevents the delivery system from wasting resources on a permanently broken endpoint. Send the subscriber an email or in-app notification with the reason and a link to re-enable. Most platforms give subscribers 72 hours to fix the endpoint before the DLQ is purged.

Retry Strategy Tradeoffs

Strategy	Delivery guarantee	Subscriber load	Storage cost	When to use
Synchronous HTTP call, no retry	At-most-once	Low	None	Never in production
Fixed delay retry (3 attempts)	Weak at-least-once	Medium burst	Low	Simple platforms, low event volume
Exponential backoff + jitter, 10 attempts	At-least-once	Spread over hours	Medium	Most SaaS webhook systems
Exponential backoff + per-endpoint queue	At-least-once, ordered per endpoint	Controlled	Higher	High-volume, order-sensitive events
Exponential backoff + circuit breaker	At-least-once, fast endpoint isolation	Low on failures	Medium	Large subscriber bases

Ordered delivery per endpoint is the hardest requirement to satisfy at scale. If subscribers need events in the order they occurred, you must serialize delivery per endpoint. A single worker consuming a per-endpoint queue achieves this but limits horizontal scaling. Most platforms document their webhooks as “unordered, use the event timestamp to reconcile” and avoid the problem.

Scaling the Delivery Pipeline

The polling worker approach works at low volume. At higher volume (millions of events per day), a few adjustments are needed.

Partitioned queues: Partition the event queue by endpoint ID. This ensures events for the same endpoint are consumed by the same worker instance, simplifying per-endpoint state (rate limit counters, consecutive failure tracking).

Claim-check pattern for large payloads: If payloads exceed a few KB, store them in object storage and include a presigned URL in the queue message. The worker fetches the payload only when delivering. This prevents large payloads from bloating the queue.

Separate queue for retries: Pending events and retry events have different scheduling semantics. A separate table or queue for events with nextAttemptAt > now + 60s prevents the retry scan from interfering with fresh event throughput.

Worker concurrency limits per endpoint: A single endpoint should never receive more than N concurrent requests from your delivery workers. Track in-flight count per endpoint in Redis and skip if at capacity.

Webhook endpoint verification: Before delivering any events, verify that the endpoint is reachable and controlled by the subscriber. Send a GET with a challenge token, or require the subscriber to echo a value in the response header. This prevents subscribers from inadvertently (or maliciously) pointing webhooks at third-party systems.

Endpoint Management UI Patterns

The subscriber-facing dashboard needs a few things to be genuinely useful:

• Delivery log: every attempt per event, with timestamp, HTTP status, response body snippet, and retry countdown
• Event detail: full payload, all attempt history, replay button for DLQ events
• Endpoint health summary: success rate over 24h, p50/p99 latency, consecutive failure count
• Secret rotation: rotate signing secret without downtime; deliver with both old and new secrets for a transition window
• Test event sender: fire a sample payload to the endpoint immediately, outside the delivery queue

The replay button matters more than most teams expect. When a subscriber’s system has downtime and recovers, they want to replay the missed events themselves, without filing a support ticket. Making replay self-serve removes an entire category of support escalation.

---

A webhook delivery system is an outbound event database with an HTTP delivery layer on top. The HTTP request is the last step, not the first. Get the durability, retry logic, and observability right before worrying about throughput. Most production failures in webhook systems trace back to events that were never durably stored, retries that fired without jitter, or delivery failures that were never surfaced to anyone with the ability to fix them.

The layer map for this system: domain event fires, event written to database, worker picks it up, signs it, respects per-endpoint rate limits, delivers, logs the attempt, schedules the retry if needed, and routes to DLQ at exhaustion. Each layer is independent and testable. That structure is what makes the system debuggable when something breaks at 2am.