Idempotency Keys in Distributed APIs: Preventing Double Charges and Phantom Orders

If your API handles money, inventory, provisioning, or anything else that cannot happen twice, retries are dangerous without idempotency.

Clients retry because networks fail. Mobile apps retry when users tap twice. Proxies retry on timeout. Background jobs retry after crashes. If your POST endpoint is not idempotent, one user action can create two charges, two orders, or two resources.

Idempotency keys are the production-grade fix: the client sends a unique key, and your API guarantees that repeated requests with the same key produce one logical operation.

What Idempotency Actually Guarantees

For a given (tenant, endpoint, idempotency_key) tuple:

• The first valid request executes the business operation.
• Retries with the same key return the exact same logical result.
• Concurrent duplicates do not execute the operation twice.

It does not mean different keys with the same payload are deduplicated. It also does not replace database constraints or transaction safety.

Data Model That Holds Up Under Load

A durable idempotency record needs enough state to survive crashes and race conditions.

interface IdempotencyRecord {
  tenantId: string;
  endpoint: string;                // e.g. POST /v1/payments
  key: string;                     // client-provided UUID
  requestHash: string;             // hash(method + path + canonical body)
  status: "in_progress" | "succeeded" | "failed";
  responseCode: number | null;
  responseBody: string | null;     // stored canonical response
  resourceType: string | null;     // e.g. "payment"
  resourceId: string | null;       // e.g. pay_123
  lockedUntil: Date | null;        // safety lease for crashed workers
  createdAt: Date;
  expiresAt: Date;
}

Use a unique index on (tenant_id, endpoint, key). That is the hard stop preventing duplicate inserts during concurrent retries.

Also store requestHash. If a client accidentally reuses a key with a different payload, return 409 Conflict. Silent acceptance here creates impossible debugging.

Race-Free Request Flow

Most failures come from getting this order wrong.

1. Begin transaction.
2. Attempt insert idempotency record as in_progress.
3. If insert succeeds, current request owns execution.
4. If duplicate key exists:
   • hash mismatch -> 409 Conflict
   • existing succeeded -> replay saved response
   • existing in_progress -> return 409 or 202 with retry hint
5. Owner executes business logic.
6. Persist canonical response into idempotency record.
7. Commit.

Postgres pattern

CREATE UNIQUE INDEX ux_idempotency
ON idempotency_records (tenant_id, endpoint, key);

async function handleCreatePayment(req: Request): Promise<Response> {
  const key = req.headers.get("Idempotency-Key");
  if (!key) return json({ error: "Missing Idempotency-Key" }, 400);

  const tenantId = req.auth.tenantId;
  const endpoint = "POST /v1/payments";
  const hash = canonicalRequestHash(req.method, req.url, await req.text());

  return db.tx(async (tx) => {
    const inserted = await tx.idempotency.tryInsert({
      tenantId,
      endpoint,
      key,
      requestHash: hash,
      status: "in_progress",
      lockedUntil: new Date(Date.now() + 30_000),
      expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000),
    });

    if (!inserted) {
      const existing = await tx.idempotency.get(tenantId, endpoint, key);
      if (!existing) return json({ error: "Retry" }, 409);

      if (existing.requestHash !== hash) {
        return json({ error: "Key reused with different payload" }, 409);
      }

      if (existing.status === "succeeded") {
        return new Response(existing.responseBody!, {
          status: existing.responseCode!,
          headers: { "Content-Type": "application/json", "Idempotent-Replay": "true" },
        });
      }

      return json({ error: "Request already in progress" }, 409);
    }

    const payment = await tx.payments.create({
      tenantId,
      amount: 4999,
      currency: "USD",
    });

    const body = JSON.stringify({ id: payment.id, status: "confirmed" });

    await tx.idempotency.markSucceeded({
      tenantId,
      endpoint,
      key,
      responseCode: 201,
      responseBody: body,
      resourceType: "payment",
      resourceId: payment.id,
    });

    return new Response(body, { status: 201, headers: { "Content-Type": "application/json" } });
  });
}

The Two Common Bugs

1) Writing idempotency record after business logic

If payment creation happens before the key is stored, a crash between those steps allows duplicate charges on retry.

2) Not storing the original response

If replayed requests recompute state, clients can observe inconsistent responses. Save and replay the original response payload and code.

Handling Crashes and Stuck in_progress Records

A request can die mid-flight after acquiring ownership. Without recovery, key stays blocked forever.

Use a lease (lockedUntil) and takeover rule:

• If in_progress and lockedUntil is in the past, a new retry may reclaim ownership.
• Reclaim only with compare-and-swap in SQL (WHERE locked_until < now()).

This prevents zombie locks while avoiding two workers owning the same key.

TTL Strategy: How Long Should Keys Live?

For payment/order APIs: 24-72 hours is practical.

• Too short: late retries create duplicates.
• Too long: storage grows and key collisions become support issues.

Use partitioned cleanup jobs and metrics on expired key volume.

Multi-Region Considerations

If writes happen in multiple regions with eventual consistency, duplicate execution can still occur before replica convergence.

Mitigations:

• Route same tenant+key to a single home region.
• Or use globally consistent datastore for idempotency table.
• Keep hard uniqueness at payment provider side too (defense in depth).

Never rely on cache-only idempotency for money movement.

Observability You Need From Day 1

Track these metrics:

• idempotency.duplicate_hit_rate
• idempotency.hash_mismatch_count
• idempotency.in_progress_timeout_count
• idempotency.replay_latency_ms

Log fields:

• tenant_id, endpoint, idempotency_key, request_hash, status_transition

Without this, duplicate incidents turn into guesswork.

Contract for API Consumers

Document this clearly:

• Send a UUIDv4 in Idempotency-Key for every non-idempotent POST.
• Reuse the same key only when retrying the exact same request.
• Treat Idempotent-Replay: true as normal success.
• Rotate to a new key for new operations.

Good client guidance reduces bad retries more than backend heroics.

Closing

Idempotency keys are not optional polish. They are core reliability infrastructure for distributed APIs. The winning pattern is simple but strict: unique key constraint, request hash validation, in-progress ownership, canonical response replay, and lease-based recovery.

Get those right and retries become safe instead of expensive incidents.