Designing a Payment Processing System: Idempotency, Reconciliation, and Webhook Reliability at Scale

Payments look straightforward from the outside: user clicks “Pay”, money moves, confirmation appears. The reality is one of the most operationally demanding domains in backend engineering. Money is stateful, networks are unreliable, and the cost of a bug is measured in dollars rather than user experience points.

The failure modes are specific and expensive. A network timeout after a charge succeeds at the provider but before your server receives the response means you have no record of a payment that actually happened. A retry without idempotency protection charges the customer twice. A missing reconciliation step means your ledger diverges from your bank statement. Each of these is a production incident with real financial and legal consequences.

This article walks through the full architecture of a production payment system. Idempotency keys, webhook patterns, and saga-style distributed transactions are covered in depth elsewhere in this series. Here, the focus is on what makes payments specifically hard: state machine design, double-entry ledger accounting, reconciliation pipelines, PCI scope, and multi-provider routing.

---

Why Payments Are Harder Than They Look

A typical order flow touches multiple systems: your API server, a payment provider, your database, a fulfillment service, and an email service. Any of these can fail independently.

The core challenge is that payment operations are not atomic across system boundaries. When you call a provider’s charge API, you get a response back, but between the API call succeeding at the provider and your write landing in your database, your process can crash. At that point you have an orphaned charge with no internal record.

Three properties define a correct payment system:

1. Exactly-once execution: a payment intent must result in exactly one charge, no matter how many retries occur.
2. Consistent state: your internal ledger must always reflect the true state of money movement.
3. Auditability: every state transition must be traceable with timestamps, actors, and reasons.

Getting all three right requires deliberate design at the data model layer, not just the API layer.

---

Payment State Machines

The first design decision that matters is modeling payment as a state machine rather than a single status field.

A charge is not simply “pending” or “completed”. It moves through a sequence of states, some of which are terminal and some of which allow transitions in either direction depending on provider responses.

type PaymentStatus =
  | "created"
  | "authorized"
  | "capture_pending"
  | "captured"
  | "cancel_pending"
  | "canceled"
  | "refund_pending"
  | "refunded"
  | "failed"
  | "disputed";

interface PaymentTransition {
  from: PaymentStatus;
  to: PaymentStatus;
  trigger: string;
}

const VALID_TRANSITIONS: PaymentTransition[] = [
  { from: "created",         to: "authorized",      trigger: "authorization_succeeded" },
  { from: "created",         to: "failed",          trigger: "authorization_failed" },
  { from: "authorized",      to: "capture_pending", trigger: "capture_requested" },
  { from: "authorized",      to: "cancel_pending",  trigger: "cancel_requested" },
  { from: "capture_pending", to: "captured",        trigger: "capture_succeeded" },
  { from: "capture_pending", to: "failed",          trigger: "capture_failed" },
  { from: "cancel_pending",  to: "canceled",        trigger: "cancel_succeeded" },
  { from: "captured",        to: "refund_pending",  trigger: "refund_requested" },
  { from: "refund_pending",  to: "refunded",        trigger: "refund_succeeded" },
  { from: "captured",        to: "disputed",        trigger: "chargeback_opened" },
];

function canTransition(current: PaymentStatus, next: PaymentStatus): boolean {
  return VALID_TRANSITIONS.some(t => t.from === current && t.to === next);
}

async function applyTransition(
  db: DatabaseClient,
  paymentId: string,
  nextStatus: PaymentStatus,
  trigger: string,
  metadata: Record<string, unknown> = {}
): Promise<void> {
  await db.transaction(async (tx) => {
    const payment = await tx.queryOne<{ status: PaymentStatus }>(
      "SELECT status FROM payments WHERE id = $1 FOR UPDATE",
      [paymentId]
    );

    if (!canTransition(payment.status, nextStatus)) {
      throw new Error(
        `Invalid transition: ${payment.status} -> ${nextStatus} (trigger: ${trigger})`
      );
    }

    await tx.execute(
      "UPDATE payments SET status = $1, updated_at = NOW() WHERE id = $2",
      [nextStatus, paymentId]
    );

    await tx.execute(
      `INSERT INTO payment_events (payment_id, from_status, to_status, trigger, metadata, occurred_at)
       VALUES ($1, $2, $3, $4, $5, NOW())`,
      [paymentId, payment.status, nextStatus, trigger, JSON.stringify(metadata)]
    );
  });
}

The FOR UPDATE lock ensures that concurrent webhook deliveries for the same payment cannot both succeed in applying conflicting transitions. The events table is the audit log. Every state change is recorded as an immutable row, not an overwrite.

---

Idempotency at the Payment Layer

Safe retries require idempotency keys. The pattern itself is covered in the idempotency keys article in this series. In the payment context, the specific requirement is that idempotency keys must be scoped to a single payment intent and must survive across process restarts.

The critical implementation detail for payments is that you must store the idempotency key before making the provider API call, not after:

interface PaymentIntent {
  id: string;
  idempotencyKey: string;
  amount: number;
  currency: string;
  customerId: string;
  status: PaymentStatus;
  providerPaymentId: string | null;
  createdAt: Date;
}

async function createPaymentIntent(
  db: DatabaseClient,
  params: {
    amount: number;
    currency: string;
    customerId: string;
    idempotencyKey: string;
  }
): Promise<PaymentIntent> {
  // Upsert ensures concurrent requests with the same key return the same result
  const intent = await db.queryOne<PaymentIntent>(
    `INSERT INTO payment_intents (id, idempotency_key, amount, currency, customer_id, status)
     VALUES (gen_random_uuid(), $1, $2, $3, $4, 'created')
     ON CONFLICT (idempotency_key) DO UPDATE SET idempotency_key = EXCLUDED.idempotency_key
     RETURNING *`,
    [params.idempotencyKey, params.amount, params.currency, params.customerId]
  );

  return intent;
}

async function authorizePayment(
  db: DatabaseClient,
  provider: PaymentProvider,
  intentId: string,
  paymentMethodId: string
): Promise<void> {
  const intent = await db.queryOne<PaymentIntent>(
    "SELECT * FROM payment_intents WHERE id = $1",
    [intentId]
  );

  // If already authorized (idempotent replay), skip the provider call
  if (intent.status !== "created") {
    return;
  }

  // Pass our intent ID as the provider-level idempotency key
  const result = await provider.authorize({
    amount: intent.amount,
    currency: intent.currency,
    paymentMethodId,
    idempotencyKey: `auth-${intentId}`,
  });

  await applyTransition(db, intentId, "authorized", "authorization_succeeded", {
    providerPaymentId: result.id,
    providerResponse: result,
  });
}

Passing your internal intent ID as the provider-level idempotency key means that if your server crashes after the provider call succeeds but before your database write, the next retry sends the same key and the provider returns the same successful result rather than creating a second charge.

---

Double-Entry Ledger Design

A common mistake is using a single balance column on a user or account row. This creates multiple problems: concurrent updates require locking, historical balance is lost, and auditing requires reconstructing from transaction logs anyway.

The correct model is double-entry bookkeeping. Every financial movement is recorded as two entries: a debit from one account and a credit to another. The ledger is append-only. Balance is always computed from the sum of entries.

interface LedgerAccount {
  id: string;
  type: "asset" | "liability" | "revenue" | "expense";
  name: string;
  currency: string;
}

interface LedgerEntry {
  id: string;
  transactionId: string;
  accountId: string;
  amount: number;      // positive = credit, negative = debit
  currency: string;
  description: string;
  occurredAt: Date;
}

// A payment creates four ledger entries across three accounts:
// 1. Debit customer receivable (they owe us money, now collected)
// 2. Credit revenue account (we earned money)
// 3. Debit payment provider asset (money is now at the provider)
// 4. Credit clearing account (will zero out on settlement)

async function recordPaymentCapture(
  db: DatabaseClient,
  paymentId: string,
  amount: number,
  currency: string
): Promise<void> {
  const txId = crypto.randomUUID();

  const entries: Omit<LedgerEntry, "id">[] = [
    {
      transactionId: txId,
      accountId: "accounts.receivable.customer",
      amount: -amount,
      currency,
      description: `Payment captured: ${paymentId}`,
      occurredAt: new Date(),
    },
    {
      transactionId: txId,
      accountId: "accounts.revenue.payments",
      amount: amount,
      currency,
      description: `Payment captured: ${paymentId}`,
      occurredAt: new Date(),
    },
  ];

  await db.transaction(async (tx) => {
    for (const entry of entries) {
      await tx.execute(
        `INSERT INTO ledger_entries (id, transaction_id, account_id, amount, currency, description, occurred_at)
         VALUES (gen_random_uuid(), $1, $2, $3, $4, $5, $6)`,
        [entry.transactionId, entry.accountId, entry.amount, entry.currency, entry.description, entry.occurredAt]
      );
    }

    // Validate that the transaction balances (debits = credits)
    const balance = entries.reduce((sum, e) => sum + e.amount, 0);
    if (balance !== 0) {
      throw new Error(`Ledger transaction does not balance: ${balance}`);
    }
  });
}

async function getAccountBalance(
  db: DatabaseClient,
  accountId: string,
  asOf?: Date
): Promise<number> {
  const result = await db.queryOne<{ balance: number }>(
    `SELECT COALESCE(SUM(amount), 0) as balance
     FROM ledger_entries
     WHERE account_id = $1
     ${asOf ? "AND occurred_at <= $2" : ""}`,
    asOf ? [accountId, asOf] : [accountId]
  );

  return result.balance;
}

The balance invariant check inside the transaction is cheap insurance. If any code path produces an unbalanced entry, the entire transaction rolls back. Historical balance at any point in time is a simple query with a date filter, no reconstruction needed.

---

Reconciliation Pipelines

Reconciliation answers the question: does your internal ledger match what the payment provider reports? Without it, you accumulate silent discrepancies that only surface during audits or chargebacks.

A reconciliation pipeline runs on a schedule (typically nightly, or after each settlement batch) and performs three comparisons:

1. Charges reconciliation: every charge in your database appears in the provider’s charge list for that period, with matching amounts.
2. Settlement reconciliation: the total settled amount in your bank statement matches the sum of captured charges minus fees.
3. Refund reconciliation: every refund in your database appears in the provider’s refund list.

interface ReconciliationResult {
  periodStart: Date;
  periodEnd: Date;
  status: "matched" | "discrepancies_found";
  discrepancies: Discrepancy[];
}

interface Discrepancy {
  type: "missing_in_provider" | "missing_in_local" | "amount_mismatch";
  paymentId?: string;
  providerChargeId?: string;
  localAmount?: number;
  providerAmount?: number;
}

async function reconcileCharges(
  db: DatabaseClient,
  provider: PaymentProvider,
  periodStart: Date,
  periodEnd: Date
): Promise<ReconciliationResult> {
  // Fetch all captured payments from local database for the period
  const localCharges = await db.query<{
    id: string;
    providerPaymentId: string;
    amount: number;
    currency: string;
  }>(
    `SELECT id, provider_payment_id, amount, currency
     FROM payment_intents
     WHERE status = 'captured'
     AND created_at BETWEEN $1 AND $2`,
    [periodStart, periodEnd]
  );

  // Fetch all charges from the provider for the same period
  const providerCharges = await provider.listCharges({
    createdAfter: periodStart,
    createdBefore: periodEnd,
    status: "succeeded",
  });

  const providerChargeMap = new Map(
    providerCharges.map(c => [c.id, c])
  );

  const localChargeMap = new Map(
    localCharges.map(c => [c.providerPaymentId, c])
  );

  const discrepancies: Discrepancy[] = [];

  // Check every local charge exists at the provider with matching amount
  for (const local of localCharges) {
    const providerCharge = providerChargeMap.get(local.providerPaymentId);

    if (!providerCharge) {
      discrepancies.push({
        type: "missing_in_provider",
        paymentId: local.id,
        providerChargeId: local.providerPaymentId,
        localAmount: local.amount,
      });
      continue;
    }

    if (providerCharge.amount !== local.amount) {
      discrepancies.push({
        type: "amount_mismatch",
        paymentId: local.id,
        providerChargeId: providerCharge.id,
        localAmount: local.amount,
        providerAmount: providerCharge.amount,
      });
    }
  }

  // Check for provider charges with no local record (ghost charges)
  for (const [providerChargeId, providerCharge] of providerChargeMap) {
    if (!localChargeMap.has(providerChargeId)) {
      discrepancies.push({
        type: "missing_in_local",
        providerChargeId,
        providerAmount: providerCharge.amount,
      });
    }
  }

  return {
    periodStart,
    periodEnd,
    status: discrepancies.length === 0 ? "matched" : "discrepancies_found",
    discrepancies,
  };
}

Discrepancies of type missing_in_local are the most urgent. They represent charges that succeeded at the provider but have no local record. These require immediate investigation: did the user receive goods or services? If so, the ledger needs a corrective entry. If not, a refund may be required.

---

Webhook Ingestion and Verification

Payment providers send webhooks to notify you of asynchronous events: captures completing, disputes opening, refunds processing. The reliability requirements here are covered in depth in the webhooks article in this series. The payment-specific concerns are worth calling out.

Provider webhooks can arrive out of order. A payment.captured event can arrive before payment.authorized if there is a delay in your webhook processor. Your state machine must handle this gracefully:

async function handleWebhookEvent(
  db: DatabaseClient,
  event: ProviderWebhookEvent
): Promise<void> {
  // Verify the webhook signature before processing
  const isValid = verifyWebhookSignature(
    event.rawBody,
    event.signature,
    process.env.WEBHOOK_SECRET!
  );

  if (!isValid) {
    throw new Error("Invalid webhook signature");
  }

  // Deduplicate using provider event ID
  const existing = await db.queryOne(
    "SELECT id FROM processed_webhook_events WHERE provider_event_id = $1",
    [event.id]
  );

  if (existing) {
    return; // Already processed, idempotent return
  }

  await db.transaction(async (tx) => {
    // Mark as processed first to prevent concurrent processing
    await tx.execute(
      "INSERT INTO processed_webhook_events (provider_event_id, processed_at) VALUES ($1, NOW())",
      [event.id]
    );

    await processEventByType(tx, event);
  });
}

async function processEventByType(
  tx: DatabaseTransaction,
  event: ProviderWebhookEvent
): Promise<void> {
  switch (event.type) {
    case "payment_intent.succeeded": {
      const paymentId = event.data.metadata.internalPaymentId as string;
      // applyTransition handles invalid transitions gracefully
      try {
        await applyTransition(tx, paymentId, "captured", "capture_succeeded", {
          providerEventId: event.id,
        });
      } catch (err) {
        // Log but don't rethrow for out-of-order events that are already in terminal state
        if (!isTerminalState(await getCurrentStatus(tx, paymentId))) {
          throw err;
        }
      }
      break;
    }

    case "charge.dispute.created": {
      const paymentId = await lookupByProviderChargeId(tx, event.data.chargeId);
      await applyTransition(tx, paymentId, "disputed", "chargeback_opened", {
        disputeId: event.data.disputeId,
        reason: event.data.reason,
      });
      break;
    }
  }
}

---

Architecture Tradeoffs

Pattern	Benefit	Cost	When to use
Append-only ledger	Full audit history, no lock contention on balance	Storage grows indefinitely, balance needs aggregation	Always in financial systems
State machine with valid transitions	Prevents impossible states, self-documenting	More code, transition table maintenance	Always for payment status
Idempotency keys stored before provider call	Safe retries even after crash	Requires UPSERT logic, key expiration policy	Always for charge operations
Nightly reconciliation	Catches silent discrepancies	Delay in detection, needs alerting pipeline	Always in production
Separate capture from authorization	Allows late cancellation, reduces fraud	Two-step flow, auth expiry to manage	When order fulfillment has delay
Multi-provider routing	Resilience, cost optimization	Complexity in transaction tracking, reconciliation per-provider	When volume justifies the overhead

---

Production Considerations

PCI Scope Reduction

Handling raw card numbers puts your entire infrastructure in PCI DSS scope, which means annual audits, penetration tests, and significant compliance overhead. The practical approach for most systems is to never touch raw card data at all.

Use the provider’s JavaScript SDK to tokenize card details directly in the browser. The token (a provider-specific reference) is what your server receives and stores. Your servers never see the PAN, CVV, or expiry date. This moves you from SAQ D (full scope, hardest) to SAQ A (minimal scope, much easier).

The implication for your data model is that you store provider_payment_method_id on customer records, not anything resembling card data. Your reconciliation and ledger systems operate entirely on tokens and provider charge IDs.

Monitoring Payment Health

Standard application metrics miss payment-specific failure modes. The metrics that matter:

• Authorization rate: the percentage of authorization attempts that succeed. A drop here usually signals a card network issue or a fraud rule triggering incorrectly.
• Capture lag: time between authorization and capture. If this grows, your capture job may be stuck.
• Reconciliation gap: count of discrepancies found in the last reconciliation run. This should be zero in steady state.
• Webhook processing lag: age of the oldest unprocessed webhook event. If this climbs, your webhook consumer is falling behind.
• Dispute rate: chargebacks as a percentage of successful charges. Sustained rates above 1% risk losing your payment provider account.

Alert on all of these. The authorization rate and dispute rate in particular should have daily trend alerts, not just threshold alerts.

Multi-Provider Failover

A single payment provider creates a single point of failure. Provider outages are rare but do happen, and when they do, your checkout converts at zero.

The architecture for multi-provider support requires abstracting the provider behind an interface:

interface PaymentProvider {
  authorize(params: AuthorizeParams): Promise<AuthorizeResult>;
  capture(params: CaptureParams): Promise<CaptureResult>;
  refund(params: RefundParams): Promise<RefundResult>;
  listCharges(params: ListChargesParams): Promise<Charge[]>;
}

class ProviderRouter implements PaymentProvider {
  constructor(
    private primary: PaymentProvider,
    private fallback: PaymentProvider,
    private healthCheck: ProviderHealthCheck
  ) {}

  async authorize(params: AuthorizeParams): Promise<AuthorizeResult> {
    const primaryHealthy = await this.healthCheck.isHealthy("primary");

    if (!primaryHealthy) {
      console.warn("Primary provider unhealthy, routing to fallback");
      return this.fallback.authorize(params);
    }

    try {
      return await this.primary.authorize(params);
    } catch (err) {
      if (isRetryableProviderError(err)) {
        console.warn("Primary provider error, falling back", { err });
        return this.fallback.authorize(params);
      }
      throw err;
    }
  }

  // capture and refund must use the same provider that created the charge
  async capture(params: CaptureParams): Promise<CaptureResult> {
    const provider = await this.resolveProviderForCharge(params.chargeId);
    return provider.capture(params);
  }
}

The critical constraint is that capture and refund operations must go to the same provider that created the original authorization. Store which provider handled each payment intent and use that as the routing key for subsequent operations.

Reconciliation becomes more complex with multiple providers because you now run separate reconciliation jobs per provider and aggregate the results. Keep reconciliation per-provider rather than trying to unify at the query layer.

---

Closing

Payment systems are unforgiving because the invariants are financial and the failures are visible to customers in the worst possible way. The patterns covered here, state machines, double-entry ledgers, idempotent operations, and reconciliation pipelines, are not over-engineering. They are the minimum viable design for a system that handles real money.

The most important investment is the audit trail. An append-only ledger and a payment events table cost almost nothing in storage but make every future debugging session, compliance inquiry, and reconciliation run straightforward rather than painful.

Start with the state machine and the ledger. Everything else can be added incrementally once the foundation is correct.