Designing a Durable Execution Engine: Workflow Replay, Checkpointing, and Failure Recovery for Long-Running Distributed Processes

A payment workflow starts, charges a card, then needs to provision a subscription, send a confirmation email, and update a CRM. The process takes three seconds under normal conditions. Then your deployment rolls out mid-execution. Or a database goes briefly unavailable. Or the worker pod gets evicted.

Without infrastructure to handle this, you are left with partially applied state: the card was charged but the subscription was never created. The naive fix is retry logic around each step, but that creates duplicate side effects unless you build idempotency into every operation. The somewhat less naive fix is a state machine persisted to a database, but now you are managing schema evolution, transition serialization, and distributed locking on your own.

Durable execution engines take a different approach. They make the workflow code itself the source of truth, and replay it forward from a persistent history on every restart. This article covers how that mechanism works, the constraints it imposes, how checkpointing and compensation fit in, and what production operation looks like.

The Core Idea: History as State

In a durable execution system, a workflow is a regular function, but its execution is mediated by a runtime that intercepts every non-deterministic operation: activity invocations, timers, signals, and side effects. Each intercepted operation is recorded as an event in an append-only history before its result is returned to the workflow code.

When a worker crashes and restarts, the runtime replays the workflow function from the beginning, feeding it the previously recorded results for each operation instead of re-executing them. The function runs the same path it ran before, but without touching any external systems for steps that already completed. Once the replay catches up to the point of failure, execution resumes normally.

This is event sourcing applied at the execution layer rather than the data layer. The history is the checkpoint. You do not need to explicitly save state: completing a step is the act of persisting it.

// Conceptual workflow definition (Temporal SDK style)
import { proxyActivities, sleep, defineSignal, setHandler } from "@temporalio/workflow";
import type * as activities from "./activities";

const { chargeCard, createSubscription, sendConfirmationEmail, updateCRM, refundCard } =
  proxyActivities<typeof activities>({
    startToCloseTimeout: "30s",
    retry: {
      maximumAttempts: 3,
      initialInterval: "1s",
      backoffCoefficient: 2,
    },
  });

export const cancelSignal = defineSignal<[reason: string]>("cancel");

export async function provisionWorkflow(params: {
  userId: string;
  planId: string;
  paymentMethodId: string;
}): Promise<void> {
  const { userId, planId, paymentMethodId } = params;

  let cancelRequested = false;
  let cancelReason = "";

  setHandler(cancelSignal, (reason) => {
    cancelRequested = true;
    cancelReason = reason;
  });

  // Each activity call is recorded in history.
  // On replay, the runtime returns the stored result without calling the activity again.
  const chargeResult = await chargeCard({ userId, planId, paymentMethodId });

  if (cancelRequested) {
    // Compensation: undo the charge before proceeding further
    await refundCard({ chargeId: chargeResult.chargeId, reason: cancelReason });
    return;
  }

  // Timer is also recorded. On replay, the workflow skips forward if the timer
  // has already fired, or blocks until it fires if not yet elapsed.
  await sleep("5s");

  await createSubscription({ userId, planId, chargeId: chargeResult.chargeId });
  await sendConfirmationEmail({ userId, planId });
  await updateCRM({ userId, planId, chargeId: chargeResult.chargeId });
}

The activities live in separate workers. They are plain async functions with no replay constraints:

// activities.ts
export async function chargeCard(params: {
  userId: string;
  planId: string;
  paymentMethodId: string;
}): Promise<{ chargeId: string }> {
  // Idempotency key derived from workflow and activity ID prevents duplicate charges
  // even if the activity retries after a partial success.
  const idempotencyKey = `charge-${params.userId}-${params.planId}`;
  const result = await paymentClient.charge({
    ...params,
    idempotencyKey,
  });
  return { chargeId: result.id };
}

export async function refundCard(params: {
  chargeId: string;
  reason: string;
}): Promise<void> {
  await paymentClient.refund(params);
}

export async function createSubscription(params: {
  userId: string;
  planId: string;
  chargeId: string;
}): Promise<void> {
  await db.subscriptions.upsert({
    where: { userId: params.userId, planId: params.planId },
    create: { ...params, status: "active" },
    update: { status: "active", chargeId: params.chargeId },
  });
}

// ... sendConfirmationEmail, updateCRM omitted for brevity

Deterministic Constraints

Replay only works if the workflow code is deterministic. Given the same history, it must make the same sequence of runtime calls. This rules out several things you would normally do in application code:

Random values. Calling Math.random() inside a workflow produces different values on replay. Use the runtime-provided random source, or generate the value inside an activity and pass it back.

Current time. new Date() returns the real clock time on replay, which is different from the original execution time. Use the runtime’s now() equivalent, which returns the timestamp from history.

Direct I/O. Any network call, database read, or file access inside the workflow function itself (not wrapped in an activity) will re-execute on replay, producing different results or side effects. All I/O belongs in activities.

Async races. Promise.race() or Promise.all() over external operations can resolve in different orders on different runs. If you need concurrency, use the SDK’s primitives for parallel activity execution.

Branching on external state. If your workflow branches based on a value fetched directly (not via an activity or signal), replay may take a different branch if that value changed. Every external input must go through a recorded mechanism.

// Wrong: reads real clock on replay
if (new Date() > new Date("2026-06-01")) {
  // takes different branch on replay after June 1
}

// Correct: workflow time is fixed to the value in history
import { workflowInfo } from "@temporalio/workflow";
const { startTime } = workflowInfo();
if (startTime > new Date("2026-06-01")) {
  // deterministic
}

// Wrong: direct database read
const user = await db.users.findById(params.userId);

// Correct: wrapped in an activity so the result is recorded
const user = await fetchUser({ userId: params.userId });

Checkpointing Strategies

In a pure replay system, the checkpoint is the complete event history. The workflow function replays from event zero on every restart. This works for short histories, but a workflow running for weeks or months accumulates thousands of events, and replaying them from scratch takes measurable time.

Full-state snapshots. The runtime persists the entire workflow state at a given point and can resume from the snapshot rather than replaying from the beginning. Temporal calls these “continue-as-new” continuations: the workflow starts a new execution with a clean history but carries forward whatever state it needs as input parameters. You trigger this explicitly when the history grows large.

import { continueAsNew, workflowInfo } from "@temporalio/workflow";

export async function longRunningPoller(params: {
  resourceId: string;
  iteration: number;
}): Promise<void> {
  const info = workflowInfo();

  // Compact history every 500 events to prevent unbounded replay cost
  if (info.historyLength > 500) {
    await continueAsNew<typeof longRunningPoller>({
      resourceId: params.resourceId,
      iteration: params.iteration,
    });
    return;
  }

  await pollResource({ resourceId: params.resourceId });
  await sleep("1m");

  await continueAsNew<typeof longRunningPoller>({
    resourceId: params.resourceId,
    iteration: params.iteration + 1,
  });
}

Incremental checkpointing. Some frameworks persist an intermediate state object alongside the history. Rather than replaying all events, the runtime loads the snapshot and replays only the delta. This reduces replay latency proportional to snapshot frequency. The tradeoff is that snapshot serialization adds overhead to every checkpoint interval, and you need to handle schema migration for snapshots that persist across code deployments.

Activity-level checkpointing via heartbeating. For long-running activities (file processing, batch imports, ML inference), the runtime does not know how far along the activity is if the worker dies. Activity heartbeating solves this: the activity periodically records a progress token. On retry, it can read the token and resume rather than restart.

export async function importLargeDataset(params: {
  datasetId: string;
  batchSize: number;
}): Promise<void> {
  // Retrieve progress from the previous attempt's last heartbeat (if any)
  const { heartbeatDetails } = Context.current();
  const lastProcessedOffset: number = heartbeatDetails?.[0] ?? 0;

  let offset = lastProcessedOffset;

  while (true) {
    const batch = await fetchBatch({ datasetId: params.datasetId, offset, limit: params.batchSize });
    if (batch.length === 0) break;

    await processBatch(batch);
    offset += batch.length;

    // Record progress. If the activity times out or the worker dies,
    // the retry will read this offset and resume from here.
    Context.current().heartbeat(offset);
  }
}

Saga Compensation

Long-running workflows often span multiple external systems. If a later step fails permanently, earlier steps may need to be undone. This is the saga pattern: a sequence of steps, each with a corresponding compensation action, executed in reverse order if the saga fails.

In a durable execution system, compensation is just more workflow code. You build a compensation stack as you execute forward, then drain it on failure.

export async function orderFulfillmentWorkflow(params: {
  orderId: string;
  customerId: string;
  items: Array<{ sku: string; quantity: number }>;
}): Promise<void> {
  const { orderId, customerId, items } = params;

  // Compensation stack: populated as we commit side effects
  const compensations: Array<() => Promise<void>> = [];

  try {
    const reservationId = await reserveInventory({ orderId, items });
    compensations.push(() => releaseInventory({ reservationId }));

    const paymentId = await capturePayment({ orderId, customerId });
    compensations.push(() => refundPayment({ paymentId }));

    const shipmentId = await scheduleShipment({ orderId, items });
    compensations.push(() => cancelShipment({ shipmentId }));

    await notifyCustomer({ customerId, orderId, shipmentId });
    // No compensation needed for notification: it is a best-effort side effect
  } catch (err) {
    // Run compensations in reverse order
    for (const compensate of compensations.reverse()) {
      try {
        await compensate();
      } catch (compensationErr) {
        // Log and continue: compensation failures need manual intervention
        // The workflow history records which compensations were attempted
      }
    }
    throw err;
  }
}

The key difference from application-layer sagas is that the compensation stack itself is recorded in history. If the worker crashes mid-compensation, the replay picks up exactly where the compensation left off. You do not need external saga orchestration infrastructure.

Exactly-Once Side Effects

Activity retries create a semantic problem: the activity may have partially succeeded before the worker died. The network call reached the external service, but the response never arrived. On retry, the external call happens again.

The solution is idempotency at the activity boundary. Every activity that causes a side effect needs an idempotency key derived from stable, deterministic inputs: the workflow ID, the activity name, and the attempt number if the operation is intrinsically non-idempotent.

For operations where the external service does not support idempotency keys, you can use the workflow and activity ID to persist a record of completion before returning, then check that record at the start of each attempt:

export async function sendTransactionalEmail(params: {
  workflowId: string;
  activityId: string;
  recipientId: string;
  templateId: string;
}): Promise<void> {
  const dedupKey = `email-${params.workflowId}-${params.activityId}`;

  const alreadySent = await db.sentEmails.findFirst({ where: { dedupKey } });
  if (alreadySent) return;

  await emailProvider.send({
    to: params.recipientId,
    template: params.templateId,
  });

  await db.sentEmails.create({ data: { dedupKey, sentAt: new Date() } });
}

This achieves exactly-once delivery at the application level, even though the transport layer is at-least-once.

Tradeoffs

Approach	Failure recovery	Development complexity	Operational complexity	Long-running support	Versioning
Manual state machine (DB-persisted)	Full, explicit transitions	High: you write all state logic	Medium: your schema, your ops	Yes, but you manage everything	Hard: schema and code migrations
Polling retry loop	Partial: retries from last poll, not step	Low	Low	Poor: no step granularity	Trivial
Durable execution framework (Temporal, Restate)	Full, replay-based	Medium: learn SDK constraints	High: requires dedicated service	Excellent, built-in	Requires determinism care
Serverless step functions (AWS Step Functions, Inngest)	Full, service-managed	Low to medium	Low: managed service	Good for bounded workflows	Configuration-level versioning

Manual state machines give you full control but you pay for every feature: locking, step granularity, timer management, compensation ordering. Polling retry loops are easy to start with but break down as workflows grow in step count or duration. Durable execution frameworks require operational investment (running a Temporal cluster is non-trivial) and impose the determinism constraints covered above. Managed step function services (Step Functions, Inngest) abstract the runtime but constrain workflow logic to JSON-serializable state and limit what you can express compared to full code-based workflows.

Production Considerations

Workflow versioning. When you deploy code that changes a workflow definition, in-flight executions are still replaying against the old history. If the new code produces a different sequence of activity calls, the runtime detects nondeterminism and fails the workflow. The fix is explicit versioning using SDK primitives:

import { patched } from "@temporalio/workflow";

export async function myWorkflow(params: { userId: string }): Promise<void> {
  // Executions that started before this patch use the old path.
  // New executions use the new path.
  if (patched("add-crm-update-2026-05")) {
    await updateCRM({ userId: params.userId });
  }
  await sendConfirmationEmail({ userId: params.userId });
}

Patches accumulate over time. Clean them up by deprecating patches once all executions started before the patch are complete.

Nondeterminism detection. Frameworks compare the sequence of commands emitted during replay against the recorded history. A mismatch is a nondeterminism error. Common causes beyond the obvious ones: using Array.sort() on objects (V8’s sort is not guaranteed stable across versions), depending on Map iteration order, and importing modules with side effects that run on load.

Replay performance. Replay speed matters when workers restart frequently or history is long. Keep activity inputs and outputs small: the entire history is deserialized on replay. Pass IDs rather than full objects across the workflow/activity boundary.

Activity heartbeating SLAs. Set heartbeat timeouts shorter than your activity’s typical duration. If an activity should complete in ten minutes, heartbeat every thirty seconds and set the heartbeat timeout to ninety seconds. This bounds your recovery window after a worker crash.

Observability. Durable execution systems produce rich execution history that is itself the primary observability artifact. Instrument your workflows with structured search attributes so you can query across executions: customer ID, order ID, workflow type, and failure reason. Export workflow metrics (open executions, activity failure rates, schedule-to-start latency) to your existing monitoring stack.

Rate limiting and backpressure. Activities can be rate-limited by annotating them and configuring the worker accordingly. Without rate limits, a burst of workflow starts can overwhelm downstream services even though each individual activity has retry logic.

Closing

Durable execution shifts the contract: instead of building resilience into each step of your code, you express your business logic directly and the runtime guarantees that the code will eventually complete. The cost is operational (running or paying for the runtime) and cognitive (understanding the determinism constraints). For workflows that span multiple services over minutes or hours, that cost is almost always worth paying compared to the alternative of hand-rolling the same guarantees.

The replay model is not magic. It is event sourcing applied to function execution, with a runtime that mediates every non-deterministic operation. Once that clicks, the constraints follow naturally, and the failure recovery properties become straightforward to reason about.