Designing Webhooks That Scale: Delivery Guarantees, Retry Logic, and Security

Every SaaS product eventually needs webhooks. You add them because customers ask, the implementation looks trivial from the outside, and you ship in a week. Then six months later you have a Slack thread about a customer who missed 400 events during a deployment, another about duplicate charges because their system processed the same event twice, and a third about a security researcher who found you don’t validate signatures.

The fire-and-forget HTTP POST is not a webhook system. It is a starting point that becomes liability at scale. This article covers what a real webhook delivery system looks like: the data model, signing, retry logic, idempotency, fan-out, and the failure modes that bite you in production.

The Core Problem: HTTP Is Unreliable

Your customer’s endpoint will:

• Return 500 because their app is deploying
• Time out because their database is under load
• Return 200 but throw an exception before processing completes
• Go down for hours and come back expecting all missed events

A webhook system needs to handle all of these without you losing events, without hammering a down endpoint, and without creating duplicates when retrying.

The fundamental requirement is at-least-once delivery with recipient-side idempotency. You cannot guarantee exactly-once delivery across an HTTP boundary. What you can do is guarantee delivery with enough metadata that the recipient can safely deduplicate.

Data Model

Start with the right schema. Everything else builds on it.

interface WebhookEndpoint {
  id: string;
  organizationId: string;
  url: string;
  secret: string;           // HMAC signing secret, stored hashed
  events: string[];         // ["payment.succeeded", "subscription.cancelled"]
  enabled: boolean;
  createdAt: Date;
  metadata: Record<string, string>;
}

interface WebhookEvent {
  id: string;               // Stable, globally unique. Clients use this for dedup.
  organizationId: string;
  type: string;             // "payment.succeeded"
  payload: Record<string, unknown>;
  createdAt: Date;
}

interface WebhookDelivery {
  id: string;
  eventId: string;
  endpointId: string;
  status: "pending" | "success" | "failed" | "cancelled";
  attemptCount: number;
  nextAttemptAt: Date | null;
  lastAttemptAt: Date | null;
  lastResponseStatus: number | null;
  lastResponseBody: string | null;   // truncated, for debugging
  createdAt: Date;
}

The separation between WebhookEvent and WebhookDelivery is load-bearing. One event fans out to multiple endpoints, and each delivery tracks its own retry state independently. An endpoint being down should not block other endpoints from receiving the event.

Signing Requests

Unsigned webhook deliveries let anyone send fake events to your customers’ endpoints. HMAC-SHA256 is the standard approach: you sign the request body with a shared secret and include the signature in a header. The recipient verifies it before processing.

import { createHmac, timingSafeEqual } from "crypto";

function signPayload(secret: string, timestamp: number, body: string): string {
  const message = `${timestamp}.${body}`;
  return createHmac("sha256", secret).update(message).digest("hex");
}

function buildWebhookHeaders(
  secret: string,
  body: string
): Record<string, string> {
  const timestamp = Math.floor(Date.now() / 1000);
  const signature = signPayload(secret, timestamp, body);

  return {
    "Content-Type": "application/json",
    "X-Webhook-Timestamp": String(timestamp),
    "X-Webhook-Signature": `v1=${signature}`,
    "X-Webhook-ID": crypto.randomUUID(),
  };
}

// Recipient-side verification
function verifyWebhookSignature(
  secret: string,
  timestamp: string,
  body: string,
  signature: string,
  toleranceSeconds = 300
): boolean {
  const now = Math.floor(Date.now() / 1000);
  const ts = parseInt(timestamp, 10);

  // Reject stale requests to prevent replay attacks
  if (Math.abs(now - ts) > toleranceSeconds) {
    return false;
  }

  const expected = signPayload(secret, ts, body);
  const provided = signature.replace("v1=", "");

  // Timing-safe comparison to prevent timing attacks
  const expectedBuf = Buffer.from(expected, "hex");
  const providedBuf = Buffer.from(provided, "hex");

  if (expectedBuf.length !== providedBuf.length) {
    return false;
  }

  return timingSafeEqual(expectedBuf, providedBuf);
}

The timestamp in the signature message is critical. Without it, an attacker who intercepts a valid signed request can replay it indefinitely. The 5-minute tolerance window is conventional: tight enough to stop replays, loose enough to handle clock skew between servers.

Store the signing secret hashed in your database but deliver the raw secret to customers at endpoint creation. If a secret needs rotation, generate a new one and run both in parallel for a transition window.

Retry Logic with Exponential Backoff

Retrying immediately after a failure just hammers a down endpoint. Exponential backoff with jitter is the correct approach: each retry waits longer than the last, and randomized jitter prevents multiple deliveries from synchronizing into a thundering herd.

interface RetryConfig {
  maxAttempts: number;
  initialDelayMs: number;
  maxDelayMs: number;
  backoffMultiplier: number;
  jitterFactor: number;
}

const DEFAULT_RETRY_CONFIG: RetryConfig = {
  maxAttempts: 10,
  initialDelayMs: 1_000,       // 1 second
  maxDelayMs: 3_600_000,       // 1 hour
  backoffMultiplier: 2,
  jitterFactor: 0.25,          // ±25% randomization
};

function calculateNextAttemptDelay(
  attemptNumber: number,
  config: RetryConfig = DEFAULT_RETRY_CONFIG
): number {
  const exponential = Math.min(
    config.initialDelayMs * Math.pow(config.backoffMultiplier, attemptNumber - 1),
    config.maxDelayMs
  );

  // Full jitter: random value in [base - jitter, base + jitter]
  const jitter = exponential * config.jitterFactor;
  const delay = exponential + (Math.random() * 2 - 1) * jitter;

  return Math.round(Math.max(delay, 0));
}

// Retry schedule for 10 attempts (approximate, with jitter):
// Attempt 1: immediate
// Attempt 2: ~1s
// Attempt 3: ~2s
// Attempt 4: ~4s
// Attempt 5: ~8s
// Attempt 6: ~16s
// Attempt 7: ~32s
// Attempt 8: ~64s (~1 min)
// Attempt 9: ~128s (~2 min)
// Attempt 10: ~1 hour (capped)

function shouldRetry(responseStatus: number): boolean {
  // Retry on network errors (status 0), 5xx, and 429
  // Do not retry on 4xx (except 429) -- the request is bad, not the server
  if (responseStatus === 0) return true;       // network error
  if (responseStatus === 429) return true;     // rate limited
  if (responseStatus >= 500) return true;      // server error
  return false;
}

What counts as a failure matters. A 400 means the endpoint rejected the request because it was malformed or unauthorized. Retrying it will not help. A 500 means the endpoint had a transient error. A timeout (status 0 in the code above) means the endpoint was unreachable. Both should retry.

Set your delivery timeout low, around 10-30 seconds. If their endpoint takes 30 seconds to respond, you have a problem that retrying will not fix.

The Delivery Worker

The actual delivery loop runs as a background worker pulling from a queue:

interface DeliveryAttemptResult {
  status: "success" | "failed";
  responseStatus: number;
  responseBody: string;
  durationMs: number;
}

async function attemptDelivery(
  delivery: WebhookDelivery,
  event: WebhookEvent,
  endpoint: WebhookEndpoint
): Promise<DeliveryAttemptResult> {
  const body = JSON.stringify({
    id: event.id,
    type: event.type,
    createdAt: event.createdAt.toISOString(),
    data: event.payload,
  });

  const headers = buildWebhookHeaders(endpoint.secret, body);
  const start = Date.now();

  try {
    const response = await fetch(endpoint.url, {
      method: "POST",
      headers,
      body,
      signal: AbortSignal.timeout(30_000),
    });

    const responseBody = await response.text().catch(() => "");

    return {
      status: response.ok ? "success" : "failed",
      responseStatus: response.status,
      responseBody: responseBody.slice(0, 1024),
      durationMs: Date.now() - start,
    };
  } catch (error) {
    return {
      status: "failed",
      responseStatus: 0,
      responseBody: error instanceof Error ? error.message : "unknown error",
      durationMs: Date.now() - start,
    };
  }
}

async function processDelivery(deliveryId: string): Promise<void> {
  const delivery = await db.webhookDeliveries.findById(deliveryId);
  if (!delivery || delivery.status !== "pending") return;

  const [event, endpoint] = await Promise.all([
    db.webhookEvents.findById(delivery.eventId),
    db.webhookEndpoints.findById(delivery.endpointId),
  ]);

  if (!endpoint.enabled) {
    await db.webhookDeliveries.update(deliveryId, { status: "cancelled" });
    return;
  }

  const result = await attemptDelivery(delivery, event, endpoint);
  const attemptCount = delivery.attemptCount + 1;

  if (result.status === "success") {
    await db.webhookDeliveries.update(deliveryId, {
      status: "success",
      attemptCount,
      lastAttemptAt: new Date(),
      lastResponseStatus: result.responseStatus,
      lastResponseBody: result.responseBody,
      nextAttemptAt: null,
    });
    return;
  }

  const willRetry =
    shouldRetry(result.responseStatus) &&
    attemptCount < DEFAULT_RETRY_CONFIG.maxAttempts;

  const nextAttemptAt = willRetry
    ? new Date(Date.now() + calculateNextAttemptDelay(attemptCount))
    : null;

  await db.webhookDeliveries.update(deliveryId, {
    status: willRetry ? "pending" : "failed",
    attemptCount,
    lastAttemptAt: new Date(),
    lastResponseStatus: result.responseStatus,
    lastResponseBody: result.responseBody,
    nextAttemptAt,
  });

  if (willRetry && nextAttemptAt) {
    await queue.scheduleAt(nextAttemptAt, "deliver-webhook", { deliveryId });
  }
}

Fan-Out: One Event, Many Endpoints

When an event fires, you need to create a delivery record for every matching endpoint and enqueue each independently. Do this transactionally so you never create an event without deliveries, or deliveries without an event.

async function publishWebhookEvent(
  organizationId: string,
  type: string,
  payload: Record<string, unknown>
): Promise<void> {
  // Find all active endpoints that subscribe to this event type
  const endpoints = await db.webhookEndpoints.findAll({
    organizationId,
    enabled: true,
    events: { contains: type },
  });

  if (endpoints.length === 0) return;

  await db.transaction(async (tx) => {
    const event = await tx.webhookEvents.create({
      id: crypto.randomUUID(),
      organizationId,
      type,
      payload,
      createdAt: new Date(),
    });

    const deliveries = endpoints.map((endpoint) => ({
      id: crypto.randomUUID(),
      eventId: event.id,
      endpointId: endpoint.id,
      status: "pending" as const,
      attemptCount: 0,
      nextAttemptAt: new Date(),
      lastAttemptAt: null,
      lastResponseStatus: null,
      lastResponseBody: null,
      createdAt: new Date(),
    }));

    await tx.webhookDeliveries.createMany(deliveries);

    // Enqueue immediately
    await Promise.all(
      deliveries.map((d) =>
        queue.enqueue("deliver-webhook", { deliveryId: d.id })
      )
    );
  });
}

The transaction boundary matters. If the event is written but deliveries are not, you lose events silently. If deliveries are written but the queue enqueue fails, your cron job needs to pick up pending deliveries with nextAttemptAt <= now as a fallback. That cron is not optional.

Circuit Breaker for Consistently Failing Endpoints

Retrying against an endpoint that has been failing for 48 hours wastes resources and clutters logs. A circuit breaker pattern tracks failure rates per endpoint and disables it automatically after a threshold.

interface EndpointHealthState {
  consecutiveFailures: number;
  lastFailureAt: Date | null;
  circuitOpenUntil: Date | null;
}

function shouldCircuitBreak(state: EndpointHealthState): boolean {
  if (!state.circuitOpenUntil) return false;
  return state.circuitOpenUntil > new Date();
}

async function recordEndpointFailure(endpointId: string): Promise<void> {
  const health = await db.endpointHealth.upsert(endpointId);
  const failures = health.consecutiveFailures + 1;

  // Open circuit after 20 consecutive failures
  const circuitOpenUntil =
    failures >= 20
      ? new Date(Date.now() + 24 * 60 * 60 * 1000) // 24 hours
      : null;

  await db.endpointHealth.update(endpointId, {
    consecutiveFailures: failures,
    lastFailureAt: new Date(),
    circuitOpenUntil,
  });

  if (circuitOpenUntil) {
    // Notify the endpoint owner
    await notifications.send(endpointId, "webhook.endpoint.disabled", {
      reason: "20 consecutive failures",
      disabledUntil: circuitOpenUntil,
    });
  }
}

async function recordEndpointSuccess(endpointId: string): Promise<void> {
  await db.endpointHealth.update(endpointId, {
    consecutiveFailures: 0,
    circuitOpenUntil: null,
  });
}

In processDelivery, check the circuit state before attempting delivery and skip if open. This prevents pointless attempts while the endpoint is clearly down.

Tradeoffs: Delivery Strategies

Different queuing strategies have real consequences at scale:

Strategy	Throughput	Ordering	Complexity	Best for
Direct HTTP in-process	Low	N/A	Minimal	Prototypes only
Database-backed queue + cron	Medium	Per-endpoint FIFO	Low	< 10k events/day
Redis + BullMQ	High	Per-queue FIFO	Medium	10k-1M events/day
Dedicated queue (SQS, Pub/Sub)	Very high	Best-effort	High	> 1M events/day
Kafka per-organization partitions	Extreme	Strict per-org	Very high	Enterprise multi-tenant

Start with the database-backed queue. The schema you already have supports it. Add Redis when you hit the throughput limit. Add Kafka when you have isolated per-tenant ordering requirements and a team dedicated to operating it.

Production Concerns

Timeouts on your end, not theirs. Always set an explicit timeout on the outbound HTTP request. Without one, a slow endpoint can exhaust your worker pool.

Payload size limits. Cap event payloads at 64KB or 256KB. Large payloads slow delivery and make debugging harder. If an event needs more data, include a link to fetch it from your API.

Delivery logs visible to customers. Show them the attempt history: timestamp, response status, response body (truncated). This is the single biggest support cost reduction you can make. Engineers debugging integration issues need this data immediately.

Manual replay. Let customers trigger a manual re-delivery of any event from the last 30 days. This handles the case where they were down for maintenance and want to reprocess missed events in order.

Event ordering is not guaranteed across endpoints. If a customer needs strict ordering, they need to sequence events themselves using the createdAt timestamp or an explicit sequence number you include in every payload.

Thundering herd after outages. If an endpoint comes back up after 6 hours of failure, your retry queue may have thousands of pending deliveries all scheduled for the same moment. The jitter in the retry delay helps, but also consider a rate limit per endpoint: no more than N concurrent deliveries to the same URL.

async function acquireDeliverySlot(endpointId: string): Promise<boolean> {
  const concurrentLimit = 10;
  const key = `webhook:concurrency:${endpointId}`;
  const count = await redis.incr(key);
  await redis.expire(key, 60);

  if (count > concurrentLimit) {
    await redis.decr(key);
    return false;
  }

  return true;
}

async function releaseDeliverySlot(endpointId: string): Promise<void> {
  const key = `webhook:concurrency:${endpointId}`;
  await redis.decr(key);
}

Secret rotation without downtime. Support multiple active secrets per endpoint, check all of them during verification, and give customers a deprecation window. Forcing an immediate rotation breaks their integrations.

What Recipients Need to Know

This is worth documenting clearly for your customers:

Every event has a stable id. If their system receives the same id twice, it is a retry and they should process it at most once. Their endpoint should return 2xx after accepting the event for processing, not after completing all processing. If they return 2xx but then fail internally, that is their problem to handle with their own retry logic on the consuming side.

Their endpoint should respond within your timeout window (document this: 30 seconds is standard). Long-running processing should be queued internally, with the endpoint returning 200 immediately.

Closing

A webhook system done right is a small distributed system. It has a queue, retry state machines, circuit breakers, signature verification, and fan-out logic. None of that is complicated, but every piece that gets skipped becomes an incident. The implementation above is not exhaustive, but the patterns are stable: at-least-once delivery, idempotent consumption, HMAC signing with replay protection, exponential backoff with jitter, and per-endpoint circuit breaking.

Build the data model correctly from the start. Everything else can be bolted on incrementally.