Designing a Workflow Engine: State Machines, DAGs, and Durable Execution for Complex Business Logic

Every non-trivial product eventually accumulates what amounts to a workflow engine. It starts innocuously: a user signs up, you send a welcome email, provision a resource, and maybe trigger a third-party webhook. Straightforward. Then requirements grow. You need to wait for user confirmation before proceeding. Some steps depend on others. Failed steps need to retry with backoff. Certain paths should only run if a condition is met. Some steps are long-running and span minutes or hours.

The typical response is to wire this together with a job queue, a few cron expressions, and a boolean column called email_sent. Six months later, nobody knows what state any given process is in, retries are creating duplicate charges, and debugging requires reading a dozen tables in parallel.

The root cause is almost always the same: multi-step business logic with state, branching, and side effects was never modeled as a first-class workflow. This article covers the three main architectural patterns for doing that correctly: state machines, DAG-based engines, and durable execution frameworks. Each is appropriate for a different class of problem.

---

The Problem With Ad-Hoc Orchestration

Before examining the patterns, it helps to name what goes wrong when you skip them.

Lost progress. A job worker picks up a task, completes three of five steps, then crashes. On restart, the task runs from the beginning. If any of those steps have side effects (charge a card, send an email, provision infrastructure), they now execute twice.

Invisible state. A status enum with five values doesn’t tell you whether a retry is in flight, which step failed, or what the input to the next step should be. When something goes wrong, you read logs rather than querying state.

Entangled branching. Conditional logic scattered across job handlers and cron expressions creates control flow that is impossible to visualize. Adding a new condition requires reading five files to understand the implications.

No cancellation path. An ad-hoc pipeline started at step one has no clean way to abort partway through. You either let it finish (potentially causing more damage) or you manually patch the database.

These aren’t theoretical concerns. They are the operational reality of any sufficiently complex business process managed without an explicit workflow model.

---

Pattern 1: State Machines

A state machine is the right tool when your workflow has a well-defined, finite set of states and explicit allowed transitions between them. Order processing, payment flows, document approval chains, and subscription lifecycle management all fit this shape.

The core model is simple: a set of valid states, a set of valid events, and a transition table mapping (state, event) -> nextState. The value of this model is that invalid transitions are rejected by definition. You cannot accidentally move an order from shipped to created.

type OrderState =
  | "pending"
  | "payment_authorized"
  | "fulfillment_queued"
  | "shipped"
  | "delivered"
  | "canceled"
  | "refunded";

type OrderEvent =
  | "PAYMENT_AUTHORIZED"
  | "FULFILLMENT_ACCEPTED"
  | "SHIPMENT_CREATED"
  | "DELIVERY_CONFIRMED"
  | "CANCEL_REQUESTED"
  | "REFUND_ISSUED";

type Transition = {
  from: OrderState;
  event: OrderEvent;
  to: OrderState;
  guard?: (context: OrderContext) => boolean;
  action?: (context: OrderContext) => Promise<void>;
};

const transitions: Transition[] = [
  { from: "pending", event: "PAYMENT_AUTHORIZED", to: "payment_authorized" },
  { from: "payment_authorized", event: "FULFILLMENT_ACCEPTED", to: "fulfillment_queued" },
  { from: "fulfillment_queued", event: "SHIPMENT_CREATED", to: "shipped" },
  { from: "shipped", event: "DELIVERY_CONFIRMED", to: "delivered" },
  { from: "payment_authorized", event: "CANCEL_REQUESTED", to: "canceled" },
  { from: "fulfillment_queued", event: "CANCEL_REQUESTED", to: "canceled",
    guard: (ctx) => !ctx.shipmentCreated },
  { from: "delivered", event: "REFUND_ISSUED", to: "refunded" },
];

class OrderStateMachine {
  constructor(
    private state: OrderState,
    private context: OrderContext
  ) {}

  async send(event: OrderEvent): Promise<OrderState> {
    const transition = transitions.find(
      (t) => t.from === this.state && t.event === event
    );

    if (!transition) {
      throw new Error(`Invalid transition: ${this.state} + ${event}`);
    }

    if (transition.guard && !transition.guard(this.context)) {
      throw new Error(`Guard rejected transition: ${this.state} + ${event}`);
    }

    if (transition.action) {
      await transition.action(this.context);
    }

    this.state = transition.to;
    return this.state;
  }
}

The guard function handles conditional transitions cleanly. The action callback runs side effects atomically with the transition. To make this durable, persist the current state after every successful transition, and emit an event to an audit log.

async function applyOrderEvent(
  orderId: string,
  event: OrderEvent,
  db: Database
): Promise<OrderState> {
  return await db.transaction(async (tx) => {
    const order = await tx.orders.findById(orderId, { lock: "FOR UPDATE" });
    const machine = new OrderStateMachine(order.state, order.context);
    const nextState = await machine.send(event);

    await tx.orders.update(orderId, { state: nextState });
    await tx.orderEvents.insert({ orderId, event, from: order.state, to: nextState, ts: new Date() });

    return nextState;
  });
}

The row lock prevents concurrent events from corrupting state. The event log gives you a complete audit trail.

Where state machines break down: when your workflow is not a finite sequence of named states but instead a graph of tasks with arbitrary dependencies. Order of 50 steps where any subset can run in parallel, and the final step only runs when all predecessors complete, is a bad fit for a state machine. You need a DAG.

---

Pattern 2: DAG-Based Execution Engines

A directed acyclic graph models tasks as nodes and dependencies as edges. A task runs when all its upstream dependencies have completed successfully. This is the natural shape for data pipelines, build systems, batch processing, and any workflow where many things can happen in parallel but ordering constraints must be respected.

type TaskStatus = "pending" | "running" | "succeeded" | "failed" | "skipped";

type TaskNode = {
  id: string;
  dependencies: string[];
  run: (inputs: Record<string, unknown>) => Promise<unknown>;
};

type TaskState = {
  status: TaskStatus;
  output?: unknown;
  error?: string;
  startedAt?: Date;
  finishedAt?: Date;
};

class DagRunner {
  private state = new Map<string, TaskState>();

  constructor(private tasks: TaskNode[]) {
    for (const task of tasks) {
      this.state.set(task.id, { status: "pending" });
    }
  }

  private isReady(task: TaskNode): boolean {
    return task.dependencies.every((depId) => {
      const dep = this.state.get(depId);
      return dep?.status === "succeeded";
    });
  }

  private hasFailed(task: TaskNode): boolean {
    return task.dependencies.some((depId) => {
      const dep = this.state.get(depId);
      return dep?.status === "failed" || dep?.status === "skipped";
    });
  }

  async run(): Promise<Map<string, TaskState>> {
    const taskMap = new Map(this.tasks.map((t) => [t.id, t]));

    while (true) {
      const pending = this.tasks.filter((t) => this.state.get(t.id)!.status === "pending");
      if (pending.length === 0) break;

      const ready = pending.filter((t) => this.isReady(t));
      const toSkip = pending.filter((t) => this.hasFailed(t));

      for (const task of toSkip) {
        this.state.set(task.id, { status: "skipped" });
      }

      if (ready.length === 0 && toSkip.length === 0) {
        throw new Error("DAG is deadlocked: possible cycle or unresolvable dependency");
      }

      await Promise.all(
        ready.map(async (task) => {
          this.state.set(task.id, { status: "running", startedAt: new Date() });
          const inputs = Object.fromEntries(
            task.dependencies.map((depId) => [depId, this.state.get(depId)!.output])
          );

          try {
            const output = await task.run(inputs);
            this.state.set(task.id, {
              status: "succeeded",
              output,
              startedAt: this.state.get(task.id)!.startedAt,
              finishedAt: new Date(),
            });
          } catch (err) {
            this.state.set(task.id, {
              status: "failed",
              error: String(err),
              startedAt: this.state.get(task.id)!.startedAt,
              finishedAt: new Date(),
            });
          }
        })
      );
    }

    return this.state;
  }
}

This implementation runs all ready tasks in parallel, collects their outputs, and feeds them as inputs to downstream tasks. Failed tasks propagate skip status to their dependents.

For production use, you need to persist TaskState after each step completes, not hold it in memory. The DAG definition itself (task graph structure) should be serializable and stored with the workflow run, so it can be resumed after a crash or redeployment.

// Checkpoint after each task completes
async function runWithCheckpoints(
  runId: string,
  tasks: TaskNode[],
  store: WorkflowStore
): Promise<void> {
  const checkpointedState = await store.loadState(runId);
  // Restore previously completed tasks; skip re-running them
  const runner = new DagRunner(tasks);
  runner.restoreFrom(checkpointedState);

  runner.onTaskComplete(async (taskId, state) => {
    await store.saveTaskState(runId, taskId, state);
  });

  await runner.run();
}

Where DAGs break down: when your workflow has loops, human approval gates, or needs to wait for an external event that may arrive hours later. A DAG is a static graph; it cannot represent “wait up to 72 hours for a user to approve, then branch on their decision.” For that, you need durable execution.

---

Pattern 3: Durable Execution Frameworks

Durable execution is the most powerful of the three patterns and also the most operationally significant to adopt. The core idea: your workflow code runs as ordinary async functions, but the execution runtime persists the call stack and its inputs/outputs after every step. If the process crashes, execution resumes from the last persisted point, not from the beginning.

The programming model feels like writing normal code. Loops, conditionals, await calls, and exception handling work as expected. The framework handles durability transparently.

// Pseudo-code matching the Temporal/Restate programming model
// The framework replays event history to reconstruct state on resumption

async function userOnboardingWorkflow(userId: string): Promise<void> {
  // Each activity call is persisted. If the process crashes here,
  // on resume the framework replays history and skips already-completed steps
  const user = await activity.createUserAccount(userId);
  await activity.sendWelcomeEmail(user.email);

  const verified = await workflow.waitForSignal("email_verified", {
    timeout: "72h",
  });

  if (!verified) {
    await activity.sendReminderEmail(user.email);
    const verifiedAfterReminder = await workflow.waitForSignal("email_verified", {
      timeout: "48h",
    });
    if (!verifiedAfterReminder) {
      await activity.deactivateAccount(userId);
      return;
    }
  }

  await activity.provisionUserResources(userId);
  await activity.notifySlack(`User ${userId} fully onboarded`);
}

This function can take days to complete. The waitForSignal call suspends execution without blocking a thread or holding a database connection. When the signal arrives (from a webhook, a user action, or a scheduled check), execution resumes from that line.

The key implementation detail in these frameworks is event sourcing at the execution layer. Every activity result is stored as an immutable event. On restart, the framework replays these events to reconstruct the workflow’s state without re-executing the activities. Only pending activities run against live systems.

// Activity implementations must be idempotent
// The framework will call them exactly once, but your infrastructure
// should tolerate at-least-once delivery at the activity level

async function sendWelcomeEmail(email: string): Promise<void> {
  // Use idempotency keys so re-execution has no effect
  await emailProvider.send({
    to: email,
    template: "welcome",
    idempotencyKey: `welcome-${email}`,
  });
}

The tradeoff is operational complexity. You’re running a workflow service (or a hosted equivalent) that maintains execution history, handles scheduling, manages task queues, and provides visibility into running workflows. For ten simple workflows, this is overkill. For 200 workflows that span hours, touch external systems, and require auditability, it pays for itself quickly.

---

Tradeoffs Comparison

Dimension	State Machine	DAG Engine	Durable Execution
Programming model	Transition table	Declarative graph	Async/await code
Parallelism	Sequential (by default)	Native parallel execution	Explicit with activity dispatching
Waiting for external events	Complex	Not supported natively	First-class (waitForSignal)
Loops / dynamic branching	Supported	Not supported	Supported
Resume after crash	Manual (persist state to DB)	Manual (checkpoint per step)	Automatic (event replay)
Operational overhead	Low	Medium	High
Debuggability	State log	Task graph visualization	Full execution history
Best fit	Linear lifecycle flows	Batch processing, pipelines	Long-running, human-in-loop, complex business logic

---

Production Considerations

Versioning. All three patterns share a versioning problem: what happens to in-flight workflow instances when you deploy new code? For state machines, new states or transitions must be backward-compatible with existing rows. For DAG engines, the task graph is serialized with each run, so schema changes need migration scripts. For durable execution frameworks, changing workflow logic while instances are running requires explicit versioning APIs or side-by-side deployment.

Observability. A workflow without observability is no better than an ad-hoc pipeline. Every state transition, task completion, and signal should emit a structured event. For durable execution frameworks, most provide a built-in UI that shows in-flight workflows, their current step, and their history. For state machines and DAG engines, you build this yourself.

Idempotency in activities. Regardless of the pattern, any action with a side effect (HTTP call, database write, email send) must be idempotent. Durable execution frameworks guarantee at-most-once delivery of activity results by replaying history, but the activity itself may be scheduled multiple times before it succeeds. Always design activities to be safe to call twice with the same inputs.

Timeouts and retries. Each activity should declare its own timeout and retry policy. A step that calls a flaky third-party API needs exponential backoff with a max retry count. A step that provisions infrastructure might need a longer timeout with fewer retries. Do not apply a single global retry policy to all steps.

Deadletter paths. Every workflow needs an explicit failed terminal state. When a workflow exhausts retries or hits an unrecoverable error, it should land in a state that is visible, queryable, and actionable. “Stuck in retry loop” is not a state.

---

Choosing a Pattern

The decision is mostly about your workflow’s shape:

Use a state machine when the process has a named, auditable lifecycle with well-understood transitions. Order processing, subscription management, user account status, document approval. The state is your primary query dimension (“show me all orders in payment_authorized”).

Use a DAG engine when the problem is executing a set of tasks with dependency constraints, many of which can run in parallel. Data pipelines, build systems, report generation, batch import jobs. The inputs and outputs flow between tasks; the graph structure is the primary artifact.

Use durable execution when the workflow is long-running, involves waiting for external signals, contains loops or complex branching, or spans multiple external systems where at-most-once execution matters. Onboarding flows, billing cycle management, multi-party approval processes, anything that can reasonably take hours or days.

These patterns also compose. A durable execution workflow can invoke a DAG as a single activity. A state machine’s transition actions can enqueue DAG runs. The right answer for a complex product is often all three, used for the shapes they fit.

---

Workflow orchestration is one of those architectural decisions where the cost of getting it wrong compounds over time. An ad-hoc approach works until the business logic grows past what one engineer can hold in their head. At that point, retrofitting a real orchestration model is significantly harder than starting with one. Picking the right abstraction early, even an intentionally simple state machine, gives you a foundation that scales with complexity rather than against it.