Designing a Webhook Delivery System: Retry Policies, Ordering Guarantees, and Subscriber Management at Scale

Most webhook content focuses on the consumer side: how to receive, verify, and process inbound events. The provider side is a harder problem. You are now the one making HTTP requests to endpoints you do not control, against subscribers who go down, misconfigure their servers, or get slow under load. You need retry logic that does not amplify failures, ordering semantics that do not require a global lock, and a way to automatically disable subscribers that have been dead for three days without a human having to notice.

This article covers the full architecture for delivering webhooks at scale, from the moment an event is generated to the moment you know it was acknowledged.

Event Generation and the Outbox

The first design decision is how events enter the delivery pipeline. The naive approach is to fire HTTP requests directly in your application transaction. This couples delivery latency to the user-facing request and creates phantom events when the transaction rolls back.

The right model uses an outbox table. When your application commits a state change, it inserts an event record in the same transaction. A separate delivery process reads and dispatches those events asynchronously.

// Schema for the outbox table
interface WebhookEvent {
  id: string;           // UUIDv7 — embeds creation timestamp
  resourceType: string; // "order", "invoice", "subscription"
  resourceId: string;
  eventType: string;    // "order.completed", "invoice.paid"
  payload: Record<string, unknown>;
  createdAt: Date;
  dispatchedAt: Date | null;
}

UUIDv7 is a useful choice here: it encodes a millisecond timestamp in the high bits, so events sort lexicographically by creation time without a separate sequence column. This matters when you need to establish ordering across a resource’s event history.

Fan-Out to Subscribers

Once an event is ready, you need to determine which subscribers receive it. The fan-out step creates one delivery record per subscriber per event. This decouples subscriber-specific failures from the event itself.

interface WebhookDelivery {
  id: string;
  eventId: string;
  subscriberId: string;
  endpointUrl: string;
  signingSecret: string;       // per-subscriber key, never global
  status: DeliveryStatus;
  attemptCount: number;
  nextAttemptAt: Date | null;
  lastAttemptAt: Date | null;
  lastStatusCode: number | null;
  lastResponseBody: string | null;  // truncated to 1KB
  createdAt: Date;
}

type DeliveryStatus =
  | "pending"
  | "in_flight"
  | "delivered"
  | "retrying"
  | "failed"
  | "disabled";

Fan-out happens in a background job, not in the request path. A single event with 500 active subscribers should not block your API. The job queries active subscriber subscriptions, filters by eventType patterns the subscriber opted into, and bulk-inserts the delivery rows.

async function fanOutEvent(event: WebhookEvent, db: DB): Promise<void> {
  const subscriptions = await db.query<Subscription>(
    `SELECT s.id, s.endpoint_url, s.signing_secret, sub.event_patterns
     FROM subscriptions sub
     JOIN subscribers s ON s.id = sub.subscriber_id
     WHERE s.status = 'active'
       AND $1 = ANY(sub.event_patterns)`,
    [event.eventType]
  );

  if (subscriptions.length === 0) return;

  const deliveries = subscriptions.map((sub) => ({
    id: uuidv7(),
    eventId: event.id,
    subscriberId: sub.id,
    endpointUrl: sub.endpointUrl,
    signingSecret: sub.signingSecret,
    status: "pending" as DeliveryStatus,
    attemptCount: 0,
    nextAttemptAt: new Date(),
    createdAt: new Date(),
  }));

  await db.batchInsert("webhook_deliveries", deliveries);
}

The event_patterns column uses a simple array of strings like ["order.*", "invoice.paid"]. Matching a specific event type against patterns is a cheap in-database operation.

Payload Signing

Every delivery must be signed. Subscribers need a way to verify that the payload came from you and has not been tampered with in transit.

Use HMAC-SHA256 with a per-subscriber signing secret. Include a timestamp in the signed payload to allow replay attack prevention on the subscriber side.

import { createHmac, timingSafeEqual } from "crypto";

function signPayload(
  payload: string,
  secret: string,
  timestamp: number
): string {
  const signed = `${timestamp}.${payload}`;
  return createHmac("sha256", secret).update(signed).digest("hex");
}

function buildHeaders(
  payload: string,
  secret: string
): Record<string, string> {
  const timestamp = Math.floor(Date.now() / 1000);
  const signature = signPayload(payload, secret, timestamp);

  return {
    "Content-Type": "application/json",
    "X-Webhook-Timestamp": String(timestamp),
    "X-Webhook-Signature": `sha256=${signature}`,
    "X-Webhook-Delivery-Id": uuidv7(),
  };
}

Include X-Webhook-Delivery-Id as a stable identifier. Subscribers who store it can detect and skip duplicate deliveries from your retry logic. This is the subscriber-side deduplication hook that your retry policy depends on.

When rotating signing secrets, issue a new secret and keep the old one active during a grace period. Send both signatures in a comma-separated header. Subscribers can verify against either.

Retry Policy with Exponential Backoff

Your default retry schedule should be exponential with jitter. Deterministic retry intervals cause thundering-herd problems when many subscribers recover at the same time.

function computeNextAttemptAt(attemptCount: number): Date {
  // Base intervals in seconds: 10s, 30s, 2m, 10m, 30m, 2h, 6h, 12h, 24h
  const baseSeconds = [10, 30, 120, 600, 1800, 7200, 21600, 43200, 86400];
  const base = baseSeconds[Math.min(attemptCount, baseSeconds.length - 1)];

  // Add jitter: ±25% of the base interval
  const jitter = base * 0.25 * (Math.random() * 2 - 1);
  const delaySeconds = Math.round(base + jitter);

  return new Date(Date.now() + delaySeconds * 1000);
}

Set a maximum attempt count (16 is reasonable for a 72-hour window) and mark deliveries as failed when exhausted. Keep the rows: they are your audit log and your basis for manual replay.

The delivery worker picks up rows with status IN ('pending', 'retrying') AND next_attempt_at <= NOW() using SELECT ... FOR UPDATE SKIP LOCKED to allow concurrent workers without contention.

async function attemptDelivery(delivery: WebhookDelivery, db: DB): Promise<void> {
  const payload = JSON.stringify(await buildPayload(delivery, db));
  const headers = buildHeaders(payload, delivery.signingSecret);

  await db.update("webhook_deliveries", delivery.id, { status: "in_flight" });

  let statusCode: number;
  let responseBody: string;

  try {
    const response = await fetch(delivery.endpointUrl, {
      method: "POST",
      headers,
      body: payload,
      signal: AbortSignal.timeout(10_000), // 10-second hard timeout
    });

    statusCode = response.status;
    responseBody = (await response.text()).slice(0, 1024);
  } catch (err) {
    // Network error or timeout
    statusCode = 0;
    responseBody = String(err);
  }

  const succeeded = statusCode >= 200 && statusCode < 300;
  const newAttemptCount = delivery.attemptCount + 1;
  const exhausted = newAttemptCount >= 16;

  await db.update("webhook_deliveries", delivery.id, {
    status: succeeded ? "delivered" : exhausted ? "failed" : "retrying",
    attemptCount: newAttemptCount,
    lastAttemptAt: new Date(),
    nextAttemptAt: succeeded || exhausted ? null : computeNextAttemptAt(newAttemptCount),
    lastStatusCode: statusCode,
    lastResponseBody: responseBody,
  });

  if (!succeeded) {
    await updateSubscriberHealth(delivery.subscriberId, statusCode, db);
  }
}

Circuit Breaking Per Endpoint

Retrying a dead endpoint 16 times over 72 hours costs compute and pollutes your observability. A per-subscriber circuit breaker lets you stop attempting deliveries when the subscriber is clearly offline.

interface CircuitBreakerState {
  subscriberId: string;
  state: "closed" | "open" | "half_open";
  consecutiveFailures: number;
  openedAt: Date | null;
  nextProbeAt: Date | null;
}

const FAILURE_THRESHOLD = 5;         // open after 5 consecutive failures
const PROBE_INTERVAL_MS = 30 * 60 * 1000; // probe every 30 minutes

async function getCircuitState(
  subscriberId: string,
  db: DB
): Promise<CircuitBreakerState> {
  return db.queryOne(
    "SELECT * FROM subscriber_circuit_breakers WHERE subscriber_id = $1",
    [subscriberId]
  );
}

async function recordOutcome(
  subscriberId: string,
  succeeded: boolean,
  db: DB
): Promise<void> {
  if (succeeded) {
    await db.update("subscriber_circuit_breakers", { subscriberId }, {
      state: "closed",
      consecutiveFailures: 0,
      openedAt: null,
      nextProbeAt: null,
    });
    return;
  }

  const current = await getCircuitState(subscriberId, db);
  const newCount = (current?.consecutiveFailures ?? 0) + 1;

  if (newCount >= FAILURE_THRESHOLD) {
    const probeAt = new Date(Date.now() + PROBE_INTERVAL_MS);
    await db.upsert("subscriber_circuit_breakers", { subscriberId }, {
      state: "open",
      consecutiveFailures: newCount,
      openedAt: new Date(),
      nextProbeAt: probeAt,
    });
  } else {
    await db.upsert("subscriber_circuit_breakers", { subscriberId }, {
      consecutiveFailures: newCount,
    });
  }
}

When the circuit is open, skip delivery attempts for that subscriber. In half-open state, allow one probe delivery. A successful probe closes the circuit; another failure resets the probe timer.

This is distinct from automatic subscriber disabling. A circuit breaker is operational state that recovers automatically. Automatic disabling is a policy decision that requires explicit re-activation by the subscriber.

Delivery Ordering Guarantees

Global ordering across all events is expensive: it requires a single queue with a single consumer per subscriber, which does not scale. What most platforms actually need is per-resource ordering: events for a given order_id arrive in the order they were generated.

Achieve this by partitioning the delivery queue on (subscriber_id, resource_id). Events for the same resource, destined for the same subscriber, land in the same partition and are processed sequentially.

function computePartitionKey(subscriberId: string, resourceId: string): number {
  const raw = `${subscriberId}:${resourceId}`;
  // Simple djb2 hash, distribute across N partitions
  let hash = 5381;
  for (let i = 0; i < raw.length; i++) {
    hash = ((hash << 5) + hash) + raw.charCodeAt(i);
  }
  return Math.abs(hash) % PARTITION_COUNT;
}

Each partition has a dedicated worker. Within a partition, the worker processes deliveries in created_at order. This gives you per-resource ordering without a global lock.

The tradeoff: if a delivery for order:123 is stuck retrying, later events for that same order queue behind it. You need a policy for this, typically a maximum hold time (say, 4 hours) before you break the ordering guarantee and let later events proceed. Expose this via the delivery record so subscribers can detect the gap.

type DeliveryStatus =
  | "pending"
  | "in_flight"
  | "delivered"
  | "retrying"
  | "failed"
  | "disabled"
  | "ordering_skipped"; // ordering hold expired, sent out-of-sequence

Subscriber Lifecycle Management

Active subscriber management prevents your system from silently losing events to dead endpoints.

interface Subscriber {
  id: string;
  endpointUrl: string;
  status: "active" | "disabled" | "suspended";
  healthScore: number;        // 0-100
  lastDeliveredAt: Date | null;
  consecutiveFailures: number;
  disabledAt: Date | null;
  disabledReason: string | null;
  createdAt: Date;
}

Compute the health score from a rolling window of delivery outcomes. Weight recent attempts more heavily.

async function computeHealthScore(subscriberId: string, db: DB): Promise<number> {
  // Last 50 deliveries
  const recent = await db.query<{ succeeded: boolean; createdAt: Date }>(
    `SELECT (status = 'delivered') AS succeeded, created_at
     FROM webhook_deliveries
     WHERE subscriber_id = $1
     ORDER BY created_at DESC
     LIMIT 50`,
    [subscriberId]
  );

  if (recent.length === 0) return 100;

  // Exponential decay weighting: most recent = weight 1.0, oldest = ~0.1
  let weightedSuccess = 0;
  let totalWeight = 0;

  recent.forEach((row, index) => {
    const weight = Math.exp(-0.05 * index);
    weightedSuccess += row.succeeded ? weight : 0;
    totalWeight += weight;
  });

  return Math.round((weightedSuccess / totalWeight) * 100);
}

Run automatic disabling based on configurable thresholds:

• Health score below 10 for 24 consecutive hours
• No successful delivery in the past 72 hours with at least 10 attempts
• Circuit breaker has been open for more than 48 hours

When disabling, persist the reason and timestamp. Send a notification to the subscriber’s contact email. Require explicit re-activation, either through your dashboard or via API. This prevents re-activation loops where a misconfigured endpoint is re-enabled automatically and immediately starts failing again.

Observability

Four metrics matter most for webhook delivery:

Delivery latency by percentile. Track p50, p95, and p99 from event creation to first successful delivery attempt. Spikes in p99 indicate queue back-pressure from slow subscribers.

Delivery success rate by subscriber. Aggregate over 1-hour windows. Alert when a subscriber drops below 80% over a 4-hour window. This catches degraded endpoints before the circuit breaker fires.

Attempt distribution. What percentage of deliveries succeed on the first attempt vs. require retries? A rising retry rate without new subscribers signals a systemic problem in your delivery path or a widespread downstream outage.

Queue depth per partition. A partition with growing depth means one subscriber or resource group is a bottleneck. This is your primary scaling signal.

Emit a structured log entry for every delivery attempt:

interface DeliveryAttemptLog {
  deliveryId: string;
  eventId: string;
  eventType: string;
  subscriberId: string;
  attemptNumber: number;
  durationMs: number;
  statusCode: number;
  succeeded: boolean;
  circuitState: "closed" | "open" | "half_open";
}

This log feeds your failure rate dashboards and lets you answer “why did subscriber X not receive event Y” without querying the delivery table directly.

Scaling: Partitioned Queues and Back-Pressure

At low volume, a single delivery worker polling the database works fine. As you scale, you need to address two problems: throughput and slow-subscriber isolation.

Throughput scales with partitioned workers. Assign partitions to workers using consistent hashing. When you add workers, re-balance partitions; existing in-flight deliveries complete in the old partition before handoff.

Slow-subscriber isolation is harder. A subscriber with a 9-second response time consumes a worker thread for almost the full timeout window. If one subscriber is responsible for 20% of your delivery volume, a slow endpoint there can starve other subscribers in the same partition.

The practical solution is a separate slow-subscriber queue. When a subscriber’s moving average response time exceeds a threshold (say, 3 seconds), migrate their pending deliveries to the slow queue. The slow queue has more workers relative to its volume and uses longer timeouts. Healthy subscribers never share resources with degraded ones.

Concern	Approach	Tradeoff
Fan-out at volume	Async background job with bulk insert	Adds latency between event creation and delivery start
Per-resource ordering	Partition by (subscriber, resource)	Delivery for a resource stalls behind a failing predecessor
Subscriber isolation	Per-subscriber circuit breaker	Circuit state is eventually consistent across workers
Slow endpoint isolation	Dedicated slow-subscriber queue	Adds operational complexity; subscriber migration has a brief latency gap
Secret rotation	Dual-signature header with grace period	Subscribers must validate both signatures during rotation window
Delivery deduplication	Stable delivery ID in headers	Subscriber must store and check IDs; not all do

Production Checklist

A few things that are easy to miss:

Timeout your HTTP client. Default timeouts in most HTTP libraries are either infinite or far too long. Ten seconds is a reasonable ceiling. Slow subscribers should not hold a worker thread for minutes.

Cap response body storage. You want to log what the subscriber returned, especially for debugging 4xx errors. But a subscriber returning a 10MB HTML error page should not blow up your database row. Truncate at 1-2KB.

Back-pressure from your own database. If your delivery workers are inserting delivery attempt logs faster than your database can handle, you have a write amplification problem. Consider async log batching or a separate time-series store for high-frequency delivery metrics.

Test your replay path. Manual replay of a failed delivery batch should be a first-class operation, not a SQL query you run in an incident. Build the endpoint, document it, test it on a schedule.

Rotate signing secrets on a schedule. Even without a compromise, a 90-day rotation cadence limits blast radius. Automate it and make sure your dual-signature rotation window is long enough for subscribers to deploy their updated secret.

A well-built webhook delivery system is invisible to your subscribers when it works. They get events promptly, in order, with enough information to verify authenticity. The design decisions that make it invisible: outbox-driven fan-out, per-subscriber circuit breaking, partition-based ordering, and proactive health management. Each of those can be added incrementally. Start with the outbox and signing, and layer in the rest as your subscriber count grows.