System Design ·

The Saga Pattern: Coordinating Distributed Transactions Without Two-Phase Commit

Two-phase commit is a distributed systems trap. The saga pattern is how real production systems coordinate multi-service transactions. This guide covers choreography vs orchestration, compensation logic, failure handling, idempotency, and a full e-commerce order flow in TypeScript.

The Saga Pattern: Coordinating Distributed Transactions Without Two-Phase Commit

Two-phase commit (2PC) sounds reasonable on paper. Lock all participants, get a consensus vote, commit or rollback. Clean, atomic, correct. In practice, it is a distributed systems trap: the coordinator becomes a single point of failure, locks are held across network boundaries, and a coordinator crash during the commit phase can leave participants in an indefinitely blocked state.

The saga pattern solves this by abandoning the idea of a global lock entirely. Instead of atomicity, you get eventual consistency with explicit compensation. Instead of a coordinator that holds locks, you get a sequence of local transactions, each of which can be independently undone.

This article walks through the saga pattern from first principles, using a real e-commerce order flow as the running example throughout.

The Problem with Two-Phase Commit

Consider a simple order placement: reserve inventory, charge the customer, create a fulfillment record. In a monolith with a single database, this is a three-line transaction. In a microservices architecture where inventory, payments, and fulfillment each have their own database, you need to coordinate writes across three separate systems.

2PC attempts this with a prepare phase (all participants vote yes or no) followed by a commit phase (the coordinator sends commit or rollback to all). The failure modes are severe:

  • If the coordinator crashes after sending some commit messages but not others, participants are split: some committed, some waiting indefinitely.
  • Participants hold locks across the entire prepare-to-commit window, which spans at least two network round trips.
  • A single slow participant stalls the entire transaction.

These are not hypothetical. In any system under load with even modest latency variance, 2PC creates contention that compounds quickly.

Sagas: Local Transactions with Compensation

A saga breaks a distributed transaction into a sequence of local transactions. Each step updates a single service’s data and either triggers the next step (on success) or triggers a compensating transaction to undo previously completed steps (on failure).

The critical property: each local transaction commits immediately. There is no global lock. Intermediate states are visible. Compensation is the mechanism for handling partial failures, not rollback in the traditional sense.

For the order flow, the saga looks like this:

  1. Create order (orders service) -> on fail: mark order failed
  2. Reserve inventory (inventory service) -> on fail: cancel order, release reservation
  3. Charge payment (payments service) -> on fail: cancel order, release reservation, refund charge
  4. Create fulfillment record (fulfillment service) -> on fail: cancel order, release reservation, refund charge, cancel fulfillment

The compensation chain runs in reverse. Each compensating action is itself a local transaction that must also be idempotent and reliable.

Two Coordination Approaches

There are two ways to coordinate a saga: choreography and orchestration. Both have legitimate uses. The choice affects coupling, observability, and complexity in different ways.

Choreography

In choreography, services communicate through events. Each service subscribes to events, performs its work, and emits new events. There is no central coordinator.

// Event types
type OrderCreatedEvent = {
  type: "ORDER_CREATED";
  orderId: string;
  customerId: string;
  items: Array<{ productId: string; quantity: number; price: number }>;
  totalAmount: number;
};

type InventoryReservedEvent = {
  type: "INVENTORY_RESERVED";
  orderId: string;
  reservationId: string;
};

type InventoryReservationFailedEvent = {
  type: "INVENTORY_RESERVATION_FAILED";
  orderId: string;
  reason: string;
};

type PaymentChargedEvent = {
  type: "PAYMENT_CHARGED";
  orderId: string;
  chargeId: string;
};

type PaymentFailedEvent = {
  type: "PAYMENT_FAILED";
  orderId: string;
  reason: string;
};

// Inventory service subscriber
async function handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
  const reservation = await db.transaction(async (tx) => {
    // Check all items are available
    for (const item of event.items) {
      const stock = await tx.inventory.findUnique({
        where: { productId: item.productId },
      });
      if (!stock || stock.available < item.quantity) {
        await eventBus.publish({
          type: "INVENTORY_RESERVATION_FAILED",
          orderId: event.orderId,
          reason: `Insufficient stock for product ${item.productId}`,
        });
        return null;
      }
    }

    // Decrement available, create reservation record
    const reservationId = generateId();
    for (const item of event.items) {
      await tx.inventory.update({
        where: { productId: item.productId },
        data: { available: { decrement: item.quantity } },
      });
    }
    await tx.reservations.create({
      data: { reservationId, orderId: event.orderId, items: event.items },
    });
    return reservationId;
  });

  if (reservation) {
    await eventBus.publish({
      type: "INVENTORY_RESERVED",
      orderId: event.orderId,
      reservationId: reservation,
    });
  }
}

// Orders service compensation subscriber
async function handleInventoryReservationFailed(
  event: InventoryReservationFailedEvent
): Promise<void> {
  await db.orders.update({
    where: { orderId: event.orderId },
    data: { status: "CANCELLED", cancelReason: event.reason },
  });
}

Choreography is low-coupling: services only know about events, not each other. The downside is that the saga flow is implicit. To understand what happens when a payment fails three steps into a six-step process, you need to trace event subscriptions across multiple codebases. Debugging a stuck saga means correlating events across multiple queues.

Orchestration

In orchestration, a central orchestrator drives the saga. It calls each service directly (often via API or command message), tracks state, and decides what to do next based on the result.

type SagaState =
  | "PENDING"
  | "INVENTORY_RESERVED"
  | "PAYMENT_CHARGED"
  | "FULFILLMENT_CREATED"
  | "COMPLETED"
  | "COMPENSATING"
  | "CANCELLED";

type OrderSagaContext = {
  sagaId: string;
  orderId: string;
  customerId: string;
  items: Array<{ productId: string; quantity: number; price: number }>;
  totalAmount: number;
  reservationId?: string;
  chargeId?: string;
  fulfillmentId?: string;
  state: SagaState;
  failureReason?: string;
};

class OrderSagaOrchestrator {
  async execute(context: OrderSagaContext): Promise<void> {
    await this.persistState(context);

    try {
      // Step 1: Reserve inventory
      const reservationResult = await inventoryClient.reserve({
        orderId: context.orderId,
        items: context.items,
        idempotencyKey: `${context.sagaId}:reserve`,
      });

      context.reservationId = reservationResult.reservationId;
      context.state = "INVENTORY_RESERVED";
      await this.persistState(context);

      // Step 2: Charge payment
      const paymentResult = await paymentsClient.charge({
        orderId: context.orderId,
        customerId: context.customerId,
        amount: context.totalAmount,
        idempotencyKey: `${context.sagaId}:charge`,
      });

      context.chargeId = paymentResult.chargeId;
      context.state = "PAYMENT_CHARGED";
      await this.persistState(context);

      // Step 3: Create fulfillment
      const fulfillmentResult = await fulfillmentClient.create({
        orderId: context.orderId,
        items: context.items,
        idempotencyKey: `${context.sagaId}:fulfillment`,
      });

      context.fulfillmentId = fulfillmentResult.fulfillmentId;
      context.state = "FULFILLMENT_CREATED";
      await this.persistState(context);

      context.state = "COMPLETED";
      await this.persistState(context);
    } catch (error) {
      context.state = "COMPENSATING";
      context.failureReason = error instanceof Error ? error.message : "unknown";
      await this.persistState(context);
      await this.compensate(context);
    }
  }

  private async compensate(context: OrderSagaContext): Promise<void> {
    // Compensate in reverse order, skipping steps that never ran

    if (context.fulfillmentId) {
      await fulfillmentClient.cancel({
        fulfillmentId: context.fulfillmentId,
        idempotencyKey: `${context.sagaId}:cancel-fulfillment`,
      });
    }

    if (context.chargeId) {
      await paymentsClient.refund({
        chargeId: context.chargeId,
        idempotencyKey: `${context.sagaId}:refund`,
      });
    }

    if (context.reservationId) {
      await inventoryClient.release({
        reservationId: context.reservationId,
        idempotencyKey: `${context.sagaId}:release-reservation`,
      });
    }

    await ordersClient.cancel({
      orderId: context.orderId,
      reason: context.failureReason,
      idempotencyKey: `${context.sagaId}:cancel-order`,
    });

    context.state = "CANCELLED";
    await this.persistState(context);
  }

  private async persistState(context: OrderSagaContext): Promise<void> {
    await db.sagas.upsert({
      where: { sagaId: context.sagaId },
      create: { sagaId: context.sagaId, data: context },
      update: { data: context, updatedAt: new Date() },
    });
  }
}

Orchestration makes the saga flow explicit. You can look at the orchestrator and understand the entire process. Observability is straightforward: query the sagas table to see what state any saga is in. The cost is centralization: the orchestrator is a participant in every transaction and can become a bottleneck or single point of failure if not deployed carefully.

Idempotency Is Not Optional

Every step and every compensation in a saga must be idempotent. The orchestrator will retry failed calls. The event bus will redeliver messages. Network timeouts mean you cannot tell whether a call succeeded before the connection dropped.

The pattern is to include an idempotency key with every request, and for each service to store completed operations keyed by that value.

// Inventory service: idempotent reservation
async function reserveInventory(request: {
  orderId: string;
  items: Array<{ productId: string; quantity: number }>;
  idempotencyKey: string;
}): Promise<{ reservationId: string }> {
  // Check if we already processed this request
  const existing = await db.reservations.findUnique({
    where: { idempotencyKey: request.idempotencyKey },
  });
  if (existing) {
    return { reservationId: existing.reservationId };
  }

  const result = await db.transaction(async (tx) => {
    for (const item of request.items) {
      const stock = await tx.inventory.findUniqueOrThrow({
        where: { productId: item.productId },
      });
      if (stock.available < item.quantity) {
        throw new Error(`Insufficient stock: ${item.productId}`);
      }
      await tx.inventory.update({
        where: { productId: item.productId },
        data: { available: { decrement: item.quantity } },
      });
    }

    const reservation = await tx.reservations.create({
      data: {
        reservationId: generateId(),
        orderId: request.orderId,
        items: request.items,
        idempotencyKey: request.idempotencyKey,
      },
    });

    return reservation;
  });

  return { reservationId: result.reservationId };
}

The idempotency key should encode the saga ID and the step name, so each step gets a unique key that is deterministic and repeatable across retries.

Compensation Is Not Rollback

This distinction matters in production. A compensating transaction undoes the business effect of a step, but it does not erase that the step happened. Other systems may have seen the intermediate state and acted on it.

For example, if inventory was reserved and the customer received an “items reserved” email before the payment failed, the compensation releases the inventory but cannot un-send the email. The compensation should also trigger a “reservation cancelled” email. Compensation is a forward-moving corrective action, not a time machine.

This also means compensating transactions can fail. If the inventory service is down when the orchestrator tries to release a reservation, the compensation itself fails. You need a retry loop with exponential backoff, and a dead-letter queue or alert for compensations that exhaust retries.

async function compensateWithRetry(
  sagaId: string,
  step: string,
  fn: () => Promise<void>,
  maxAttempts = 5
): Promise<void> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      await fn();
      return;
    } catch (error) {
      if (attempt === maxAttempts) {
        // Emit alert for manual intervention
        await alerting.critical({
          title: "Saga compensation failed after max retries",
          sagaId,
          step,
          error: error instanceof Error ? error.message : "unknown",
        });
        throw error;
      }
      const backoffMs = Math.min(1000 * Math.pow(2, attempt), 30_000);
      await sleep(backoffMs);
    }
  }
}

Tradeoffs

Property2PCChoreography SagaOrchestration Saga
AtomicityStrong (or blocked)None (eventual)None (eventual)
Intermediate state visibilityNo (locked)YesYes
CouplingTight (via coordinator)Low (event-driven)Medium (via orchestrator)
ObservabilityHard (distributed locks)Hard (trace events)Easy (central state)
Failure recoveryComplex (coordinator crash)Automatic (retry events)Explicit (retry logic)
ComplexityProtocol complexityImplicit flowExplicit flow
Cyclic dependency riskNoneHighNone

Choreography works well for simpler flows with two or three steps. As the number of steps grows, the implicit flow becomes hard to reason about, especially when compensations trigger further events. Orchestration adds an explicit state machine that is easier to debug but requires careful deployment to avoid the orchestrator becoming a bottleneck.

Production Considerations

Saga timeouts. A saga that is waiting on a service that never responds will sit in a pending state indefinitely. Set a maximum age on in-progress sagas and trigger compensation if they exceed it. For order flows, a saga that has not completed within 10 minutes likely indicates a failure that will never self-resolve.

// Background job: timeout stuck sagas
async function timeoutStaleSagas(): Promise<void> {
  const staleThreshold = new Date(Date.now() - 10 * 60 * 1000); // 10 minutes

  const staleSagas = await db.sagas.findMany({
    where: {
      state: { in: ["PENDING", "INVENTORY_RESERVED", "PAYMENT_CHARGED"] },
      updatedAt: { lt: staleThreshold },
    },
  });

  for (const saga of staleSagas) {
    await orchestrator.compensate({
      ...saga.data,
      failureReason: "saga_timeout",
    });
  }
}

Ordering and duplicate events. In choreography, events can arrive out of order. A PAYMENT_FAILED event might arrive before the INVENTORY_RESERVED event it is responding to, depending on queue behavior. Design handlers to be safe when called out of order: check current state before acting, and discard events that do not apply to the current state.

Testing compensation paths. The compensation path is the path that rarely runs in production and therefore rarely gets tested. Write explicit integration tests that inject failures at each step and verify the full compensation sequence runs correctly. This is the most valuable testing investment for saga implementations.

Saga log as audit trail. The persisted saga state doubles as an audit log. Every state transition records what happened and when. For financial operations, this is essential for reconciliation and dispute resolution.

Choreography vs Orchestration: When to Use Each

Choose choreography when:

  • The flow has two or three steps with a clear linear sequence.
  • Services are owned by separate teams who should not depend on a shared orchestrator.
  • You already have a reliable event bus and event-driven infrastructure.

Choose orchestration when:

  • The flow has four or more steps, or has conditional branches.
  • You need clear observability into saga state from a single place.
  • Compensation logic is complex and conditional.
  • Multiple teams need to reason about the flow as a unit.

Many systems end up with orchestration for the core flows (order placement, account setup) and choreography for peripheral reactions (notifications, analytics, cache invalidation).

Closing

The saga pattern is not simpler than 2PC. It trades one kind of complexity (distributed locking) for another (explicit compensation logic and eventual consistency). The difference is that saga failures are recoverable and observable. A stuck 2PC participant requires manual DBA intervention. A failed saga is a row in a database with a clear state and a retry queue.

The implementation work is in the compensating transactions: they must be idempotent, they must handle partial failures, and they must be tested as thoroughly as the happy path. If you treat compensation as an afterthought, you will have inconsistencies in production that are painful to diagnose and fix. Build the compensation paths first, test them explicitly, and the rest of the saga implementation falls into place.

More in System Design

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL
System Design ·

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL

A deep dive into Aurora's storage-compute separation, the log-is-the-database design that pushes redo log processing to storage nodes, quorum-based replication across 6 copies in 3 AZs, protection group architecture, fast cloning, Serverless v2 scaling mechanics, and honest tradeoffs versus RDS, self-managed PostgreSQL, CockroachDB, and Cloud Spanner.

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale
System Design ·

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale

A deep dive into CockroachDB's internals: the 512MB range-based data model with automatic splitting, per-range Raft replication, MVCC timestamp ordering with hybrid logical clocks, DistSQL physical planning, serializable transactions with timestamp refreshes, closed timestamps for follower reads, online schema changes using multi-version schema descriptors, and production considerations for hotspots and write amplification.

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation
System Design ·

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation

Containers are not virtual machines and they are not magic. They are a thin composition of Linux kernel primitives: namespaces, cgroups, and a layered filesystem. This article traces exactly what happens between docker run and a running process, covering the OCI spec, containerd, runc, overlay filesystems, and container networking at the kernel level.

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally
System Design ·

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally

A deep dive into YugabyteDB internals covering the DocDB storage engine built on RocksDB LSM trees, per-tablet Raft consensus, YSQL and YCQL dual query layers, distributed MVCC with hybrid logical clocks, automatic tablet splitting and rebalancing, xCluster multi-region replication, and production considerations for hotspots, connection pooling, and schema design.