Designing a Webhook Ingestion Pipeline: Signature Verification, Idempotent Processing, and Event Routing for Multi-Provider SaaS

Every SaaS product eventually ends up with a webhook receiver that looks like a switch statement. Stripe lands on /webhooks/stripe, GitHub on /webhooks/github, Twilio on /webhooks/twilio, and someone wired them all together with copy-pasted HMAC checks and inline business logic. It works until it doesn’t: a retry storm fills your database with duplicate charge events, a deployment redeploys your receiver mid-request and Stripe gives up retrying, or GitHub starts sending a new event type your router doesn’t handle and silently drops it.

This article covers the full inbound pipeline: unified signature verification, durable ingestion before any processing, idempotent event deduplication, dead letter queues, event routing with fan-out, replay strategies, and the observability layer that catches provider-side problems before your customers do.

The Core Architecture

The ingestion pipeline has four distinct stages that should be kept separate:

1. Receive and verify: Accept the HTTP request, verify the signature, return 200 immediately.
2. Persist: Write the raw payload to a durable store before any business logic runs.
3. Deduplicate and route: Pull from the durable store, check idempotency, dispatch to internal consumers.
4. Observe: Track per-provider event rates, processing lag, and error rates.

The critical insight is that stages 1 and 2 must complete within the HTTP request, but stages 3 and 4 run asynchronously. Providers have tight timeout windows (Stripe retries after 10 seconds, GitHub after a few minutes), and if you do any real work inside the HTTP handler you will eventually miss that window under load.

Unified Signature Verification

Each provider has a slightly different signature scheme. Stripe uses HMAC-SHA256 over a signed payload that includes a timestamp. GitHub uses HMAC-SHA256 over the raw body with a X-Hub-Signature-256 header. Twilio uses HMAC-SHA1 with the full request URL and sorted POST parameters. Svix (used by many SaaS platforms as their outbound webhook service) uses a similar timestamp-plus-body approach to Stripe.

Rather than scattering these checks across route handlers, define a common verifier interface:

interface WebhookVerifier {
  verify(request: RawRequest): Promise<VerifiedPayload>;
}

interface RawRequest {
  headers: Record<string, string>;
  rawBody: Buffer;
  url: string;
}

interface VerifiedPayload {
  provider: string;
  eventId: string;
  eventType: string;
  occurredAt: Date;
  payload: unknown;
}

Each provider gets its own implementation. Here is the Stripe verifier:

import crypto from "crypto";

const STRIPE_TOLERANCE_SECONDS = 300; // 5 minutes

class StripeVerifier implements WebhookVerifier {
  constructor(private readonly signingSecret: string) {}

  async verify(request: RawRequest): Promise<VerifiedPayload> {
    const signatureHeader = request.headers["stripe-signature"];
    if (!signatureHeader) throw new WebhookVerificationError("Missing stripe-signature header");

    const parts = Object.fromEntries(
      signatureHeader.split(",").map((p) => p.split("=") as [string, string])
    );

    const timestamp = parseInt(parts["t"] ?? "", 10);
    if (isNaN(timestamp)) throw new WebhookVerificationError("Invalid timestamp");

    const nowSeconds = Math.floor(Date.now() / 1000);
    if (Math.abs(nowSeconds - timestamp) > STRIPE_TOLERANCE_SECONDS) {
      throw new WebhookVerificationError("Timestamp outside tolerance window");
    }

    const signedPayload = `${timestamp}.${request.rawBody.toString("utf8")}`;
    const expected = crypto
      .createHmac("sha256", this.signingSecret)
      .update(signedPayload)
      .digest("hex");

    const provided = parts["v1"];
    if (!provided || !timingSafeEqual(expected, provided)) {
      throw new WebhookVerificationError("Signature mismatch");
    }

    const body = JSON.parse(request.rawBody.toString("utf8")) as Record<string, unknown>;

    return {
      provider: "stripe",
      eventId: body["id"] as string,
      eventType: body["type"] as string,
      occurredAt: new Date((body["created"] as number) * 1000),
      payload: body,
    };
  }
}

function timingSafeEqual(a: string, b: string): boolean {
  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(Buffer.from(a), Buffer.from(b));
}

The GitHub verifier is simpler because there is no timestamp, which means it has no replay protection built into the protocol. You have to handle that yourself at the deduplication stage:

class GitHubVerifier implements WebhookVerifier {
  constructor(private readonly secret: string) {}

  async verify(request: RawRequest): Promise<VerifiedPayload> {
    const signatureHeader = request.headers["x-hub-signature-256"];
    if (!signatureHeader) throw new WebhookVerificationError("Missing x-hub-signature-256");

    const expected =
      "sha256=" +
      crypto.createHmac("sha256", this.secret).update(request.rawBody).digest("hex");

    if (!timingSafeEqual(expected, signatureHeader)) {
      throw new WebhookVerificationError("Signature mismatch");
    }

    const body = JSON.parse(request.rawBody.toString("utf8")) as Record<string, unknown>;
    const eventType = request.headers["x-github-event"] ?? "unknown";
    const deliveryId = request.headers["x-github-delivery"] ?? crypto.randomUUID();

    return {
      provider: "github",
      eventId: deliveryId,
      eventType,
      occurredAt: new Date(),
      payload: body,
    };
  }
}

A registry ties this together at the HTTP layer:

class WebhookVerifierRegistry {
  private verifiers = new Map<string, WebhookVerifier>();

  register(provider: string, verifier: WebhookVerifier): void {
    this.verifiers.set(provider, verifier);
  }

  get(provider: string): WebhookVerifier {
    const verifier = this.verifiers.get(provider);
    if (!verifier) throw new Error(`No verifier registered for provider: ${provider}`);
    return verifier;
  }
}

Durable Ingestion

After signature verification, write the event to a durable store before returning 200. This is the checkpoint that lets you replay events, debug processing failures, and audit what exactly arrived from the provider.

interface RawWebhookEvent {
  id: string;
  provider: string;
  eventId: string;        // provider-assigned ID
  eventType: string;
  occurredAt: Date;
  receivedAt: Date;
  payload: unknown;
  processedAt: Date | null;
  failedAt: Date | null;
  retryCount: number;
}

The table schema (Postgres):

CREATE TABLE raw_webhook_events (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  provider        TEXT NOT NULL,
  event_id        TEXT NOT NULL,
  event_type      TEXT NOT NULL,
  occurred_at     TIMESTAMPTZ NOT NULL,
  received_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  payload         JSONB NOT NULL,
  processed_at    TIMESTAMPTZ,
  failed_at       TIMESTAMPTZ,
  retry_count     INTEGER NOT NULL DEFAULT 0
);

-- Unique constraint for idempotency check
CREATE UNIQUE INDEX raw_webhook_events_provider_event_id
  ON raw_webhook_events (provider, event_id);

-- Query index for the processor
CREATE INDEX raw_webhook_events_unprocessed
  ON raw_webhook_events (received_at)
  WHERE processed_at IS NULL AND failed_at IS NULL;

The HTTP handler is now minimal:

async function handleWebhook(
  provider: string,
  request: RawRequest,
  db: Database,
  registry: WebhookVerifierRegistry
): Promise<{ status: number }> {
  let verified: VerifiedPayload;

  try {
    verified = await registry.get(provider).verify(request);
  } catch (err) {
    if (err instanceof WebhookVerificationError) return { status: 401 };
    throw err;
  }

  try {
    await db.query(
      `INSERT INTO raw_webhook_events
         (provider, event_id, event_type, occurred_at, payload)
       VALUES ($1, $2, $3, $4, $5)
       ON CONFLICT (provider, event_id) DO NOTHING`,
      [verified.provider, verified.eventId, verified.eventType, verified.occurredAt, verified.payload]
    );
  } catch (err) {
    // Persist failure is retryable by the provider; return 500
    console.error("Failed to persist webhook event", err);
    return { status: 500 };
  }

  return { status: 200 };
}

ON CONFLICT DO NOTHING handles the case where the provider sends the same event twice before your processor has run. The insert is a no-op on the second attempt, so the HTTP handler returns 200 and the provider stops retrying. You process it exactly once.

The Processor: Deduplication and Routing

A background processor polls the raw_webhook_events table for unprocessed rows. The unique index on (provider, event_id) already prevents duplicate rows, but you still need to guard against concurrent processors picking up the same row.

Use SELECT ... FOR UPDATE SKIP LOCKED to claim a batch:

async function processBatch(db: Database, router: EventRouter): Promise<void> {
  await db.transaction(async (tx) => {
    const rows = await tx.query<RawWebhookEvent>(
      `SELECT * FROM raw_webhook_events
       WHERE processed_at IS NULL
         AND failed_at IS NULL
         AND retry_count < 5
       ORDER BY received_at
       LIMIT 50
       FOR UPDATE SKIP LOCKED`
    );

    for (const row of rows) {
      try {
        await router.route(row);
        await tx.query(
          `UPDATE raw_webhook_events SET processed_at = now() WHERE id = $1`,
          [row.id]
        );
      } catch (err) {
        await tx.query(
          `UPDATE raw_webhook_events
           SET retry_count = retry_count + 1,
               failed_at = CASE WHEN retry_count + 1 >= 5 THEN now() ELSE NULL END
           WHERE id = $1`,
          [row.id]
        );
      }
    }
  });
}

Events that exceed five retries move to the dead letter state (failed_at IS NOT NULL). They stay in the table for inspection and manual replay; nothing is deleted automatically.

Event Routing and Fan-out

The router maps (provider, eventType) to one or more handlers. Handlers should be independent: a payment handler failing should not prevent an audit log handler from recording the event.

type EventHandler = (event: RawWebhookEvent) => Promise<void>;

class EventRouter {
  private handlers = new Map<string, EventHandler[]>();

  on(provider: string, eventType: string | "*", handler: EventHandler): void {
    const key = `${provider}:${eventType}`;
    const existing = this.handlers.get(key) ?? [];
    this.handlers.set(key, [...existing, handler]);
  }

  async route(event: RawWebhookEvent): Promise<void> {
    const specificKey = `${event.provider}:${event.eventType}`;
    const wildcardKey = `${event.provider}:*`;

    const handlers = [
      ...(this.handlers.get(specificKey) ?? []),
      ...(this.handlers.get(wildcardKey) ?? []),
    ];

    if (handlers.length === 0) {
      // Unknown event type. Log it but do not fail.
      console.warn("No handlers for event", { provider: event.provider, eventType: event.eventType });
      return;
    }

    const results = await Promise.allSettled(handlers.map((h) => h(event)));

    const failures = results.filter((r): r is PromiseRejectedResult => r.status === "rejected");
    if (failures.length > 0) {
      // At least one handler failed. Surface the first error to trigger retry logic.
      throw failures[0].reason;
    }
  }
}

Register handlers during startup:

router.on("stripe", "payment_intent.succeeded", handlePaymentSucceeded);
router.on("stripe", "customer.subscription.deleted", handleSubscriptionCancelled);
router.on("stripe", "*", auditLogHandler); // catch-all for audit trail
router.on("github", "push", handleRepoPush);
router.on("github", "pull_request", handlePullRequest);

Dead Letter Queue Management

Events in the dead letter state need tooling for inspection and replay. At minimum, expose a query that returns DLQ contents by provider and time range:

async function getDLQEvents(
  db: Database,
  provider: string,
  since: Date,
  limit = 100
): Promise<RawWebhookEvent[]> {
  return db.query<RawWebhookEvent>(
    `SELECT * FROM raw_webhook_events
     WHERE provider = $1
       AND failed_at >= $2
     ORDER BY failed_at DESC
     LIMIT $3`,
    [provider, since, limit]
  );
}

async function replayEvent(db: Database, eventId: string): Promise<void> {
  await db.query(
    `UPDATE raw_webhook_events
     SET failed_at = NULL, retry_count = 0
     WHERE id = $1`,
    [eventId]
  );
}

Resetting failed_at and retry_count puts the event back in the processor’s unprocessed queue. The processor picks it up on the next poll cycle. This is your replay mechanism for individual events.

Replay and Backfill Strategies

Replay comes up in two scenarios. The first is a bug in your handler: you fix it, then need to reprocess events that failed during the broken window. The second is a provider-side gap: you missed events because your receiver was down, and the provider exposes an API to fetch historical events.

For handler bugs, the approach above (reset DLQ entries) works for events that landed in your table. For a full category replay:

async function replayEventsByType(
  db: Database,
  provider: string,
  eventType: string,
  since: Date,
  until: Date
): Promise<number> {
  const result = await db.query<{ count: string }>(
    `UPDATE raw_webhook_events
     SET processed_at = NULL, retry_count = 0, failed_at = NULL
     WHERE provider = $1
       AND event_type = $2
       AND received_at BETWEEN $3 AND $4
     RETURNING id`,
    [provider, eventType, since, until]
  );
  return result.length;
}

For provider-side gaps, most providers (Stripe, GitHub) have REST APIs to fetch historical events. Write a backfill job that pages through the provider API, inserts rows using the same ON CONFLICT DO NOTHING logic, and lets the processor handle them normally. This means your backfill code path is identical to your normal ingestion path, which simplifies testing.

Tradeoffs

Approach	Latency	Durability	Complexity
Inline processing (no queue)	Low	Low (lost if handler crashes)	Low
Raw table + background processor	Medium	High (survives restarts)	Medium
External queue (SQS, Pub/Sub)	Medium	High	High (more infrastructure)
In-memory queue (Redis)	Low	Medium (survives if persistent)	Medium

The raw Postgres table approach works well for most SaaS products. You avoid a new infrastructure dependency, you get queryable event history for free, and the FOR UPDATE SKIP LOCKED pattern handles concurrency without a separate queue. The downside is that at very high throughput (millions of events per day), Postgres table bloat from the unprocessed index becomes a concern. At that point, an external queue with a separate audit store is worth the complexity.

Production Considerations

Raw body preservation. Express and similar frameworks parse the body before your middleware runs. You must register a raw body capture before any JSON parser, or you will be computing HMAC over parsed-then-re-serialized JSON, which will not match the signature. Use express.raw({ type: 'application/json' }) on webhook routes, not express.json().

Clock skew in timestamp validation. The five-minute tolerance Stripe uses assumes your server clock is reasonably accurate. If your container’s clock drifts by more than a few seconds (not uncommon in VMs), legitimate events will fail verification. Add NTP to your container base image or use a cloud time sync service.

Provider event IDs are not always globally unique. Stripe event IDs (evt_...) are unique across your account. GitHub delivery IDs are GUIDs unique per delivery. Twilio message SIDs are unique per message. But some providers use event types as IDs (they send the same “resource.updated” ID for every update to the same resource). Check each provider’s documentation and add a synthetic ID if needed.

Processor polling interval. A five-second poll interval with a batch size of 50 means you can process 10 events per second per processor instance. For most SaaS products, this is more than enough. If you need lower latency, switch to LISTEN/NOTIFY from Postgres to wake the processor immediately when a row is inserted, rather than polling.

Secrets rotation. Webhook signing secrets need rotation without downtime. Keep two active secrets per provider during the rotation window and verify against both. Once all in-flight events use the new secret, remove the old one.

Observability: the silent provider problem. Providers sometimes stop sending events without any error on your end. Your error rate stays zero because nothing is arriving. The only way to catch this is to track event arrival rate per provider and alert on sudden drops. A simple approach is a per-provider heartbeat table:

// Run in the HTTP handler after successful persistence
await db.query(
  `INSERT INTO webhook_provider_heartbeats (provider, last_seen_at)
   VALUES ($1, now())
   ON CONFLICT (provider) DO UPDATE SET last_seen_at = now()`,
  [verified.provider]
);

Then alert if last_seen_at for any active provider exceeds two times its historical median inter-event interval. If Stripe normally sends an event every 30 minutes and you have not seen one in two hours, something is wrong, even if your pipeline shows no errors.

Idempotency in your handlers. The pipeline guarantees at-most-one delivery to the router (via the unique index), but your handlers still need to be idempotent if you allow replays. A handler that charges a customer should check whether the charge already exists before creating a new one. The deduplication at the ingestion layer protects against provider retries; idempotency in handlers protects against your own replay operations.

Closing

The webhook ingestion problem looks small until you are dealing with a dozen providers, each with their own quirks, retrying at different intervals, sending overlapping event types, and occasionally going silent. The pipeline described here separates concerns cleanly: verification is stateless and per-provider, persistence is universal and happens before any business logic, and routing is decoupled from both. That separation is what lets you add a new provider in a few lines of code, replay a buggy handler without re-implementing half your system, and detect problems before your customers file a support ticket.