How Temporal Works Internally: Event History, Deterministic Replay, and the Architecture Behind Durable Execution

Most engineers who adopt Temporal understand the surface contract: write a workflow function, write activity functions, and Temporal ensures the workflow runs to completion even if processes crash, networks partition, or deployments roll forward. What they often do not understand is the specific sequence of mechanisms that makes that contract hold: how a workflow execution is stored, how the SDK recovers state after a crash, what happens inside the cluster when a worker goes away, and where the system’s actual failure modes live.

These internals determine real production outcomes. Engineers who treat Temporal as a black box hit history size limits, non-determinism panics, and task queue saturation they cannot explain. This article gives you the vocabulary and mental models to reason about Temporal’s behavior from first principles.

The Four Cluster Services

A Temporal cluster is not a monolithic process. It is composed of four internal services, each with distinct ownership boundaries.

Frontend service is the entry point for all external traffic. Every SDK call (StartWorkflow, SignalWorkflow, QueryWorkflow, PollForActivityTask) arrives at the frontend over gRPC. The frontend performs request routing, enforces namespace-level rate limits and per-namespace request quotas, and fans out to the appropriate internal service. It is stateless and horizontally scalable. Running more frontend replicas increases throughput without any coordination overhead.

History service owns the durable state of every workflow execution. Internally, workflow executions are distributed across shards (default: 512 shards in the open-source configuration). Each shard owns a range of workflow IDs via consistent hashing, backed by a persistence layer (Cassandra, MySQL, or PostgreSQL). The history service for a given shard holds all events for every execution assigned to that shard, and it schedules timers, writes to the persistence store, and drives the workflow state machine. Scaling the history service means adding more nodes; the shard count is fixed at cluster initialization time, so capacity planning requires choosing an initial shard count high enough for expected peak load.

Matching service maintains task queues. When the history service determines that a workflow task or activity task needs to be executed, it writes the task to the matching service. The matching service pairs tasks with available workers through a long-poll mechanism. It does not persist tasks to the durable store by default (tasks are in-memory within the matching service, backed by a short-lived transient persistence path). If the matching service restarts before a task is picked up, the history service will re-create the task after a timeout. Task queues are logical channels; the matching service partitions them across its nodes by task queue name.

Worker service (sometimes called the internal worker service) runs Temporal’s own system workflows: namespace migration, archival, and cross-cluster replication workflows. It is operationally important but not directly involved in executing user-defined workflow and activity code. Your application workers are separate processes, outside the cluster, that communicate with the frontend service.

// Conceptual representation of how a workflow start fans through cluster services

interface WorkflowStartRequest {
  namespace: string;
  workflowId: string;
  workflowType: string;
  taskQueue: string;
  input: Uint8Array;
}

// 1. SDK calls Frontend.StartWorkflowExecution
// 2. Frontend routes to History service shard owner for workflowId
// 3. History service writes WorkflowExecutionStarted event, creates first workflow task
// 4. History service writes the workflow task to Matching service
// 5. Worker long-polling the task queue receives the workflow task
// 6. Worker executes the workflow function, returns commands
// 7. History service processes commands, appends events, schedules next tasks

Workflow Execution as Append-Only Event History

A workflow execution in Temporal is not stored as a snapshot of mutable state. It is stored as an ordered, append-only log of events. Every meaningful state transition produces a new event. The history for a workflow that schedules and completes one activity looks like this:

1. WorkflowExecutionStarted — contains the workflow type, input, task queue, and execution timeout
2. WorkflowTaskScheduled — the history service is ready to hand off a task to a worker
3. WorkflowTaskStarted — a worker picked up the workflow task
4. WorkflowTaskCompleted — the worker returned commands (in this case, ScheduleActivity)
5. ActivityTaskScheduled — the history service recorded the intent to run the activity
6. ActivityTaskStarted — an activity worker picked up the task
7. ActivityTaskCompleted — the activity worker returned a result
8. WorkflowTaskScheduled / WorkflowTaskStarted / WorkflowTaskCompleted — workflow resumes to process the activity result
9. WorkflowExecutionCompleted — the workflow function returned

This log is the ground truth for the execution. Nothing else is stored about the workflow’s logical state. Recovering the workflow’s in-memory state after any failure means replaying this log from the beginning.

The history service persists each event atomically before handing control back. A crash between steps 6 and 7 means the activity is re-scheduled; the worker receives the task again. Idempotency at the activity level is therefore a real requirement, not an optional best practice.

The Deterministic Replay Model

When a worker receives a workflow task, it does not receive the current “state” of the workflow. It receives the full event history (or a delta since the last completed workflow task). The SDK replays that history through the workflow function to reconstruct the execution’s current position.

During replay, every call that would produce a side effect (scheduling an activity, starting a timer, generating a random number, calling Date.now()) is intercepted by the SDK. Instead of executing the side effect, the SDK checks whether a corresponding event already exists in the history. If it does, the SDK returns the recorded result from the history event rather than re-executing the operation. Replay is complete when the history is exhausted, at which point the workflow function continues executing forward in “live” mode, generating new commands that the worker sends back to the history service.

This model has one hard constraint: the workflow function must be deterministic. Given the same event history, it must always produce the same sequence of commands. That means:

• No direct calls to Math.random(), Date.now(), or any external I/O
• No setTimeout or setInterval (use Temporal’s sleep API instead)
• No iteration over maps or sets where insertion order is not guaranteed in the language runtime
• No conditional branches that depend on environment variables or configuration read at runtime

When the SDK detects that the command the workflow function is issuing does not match the next event in the history (wrong event type, wrong activity name, wrong sequence), it throws a non-determinism error. This is not a bug; it is a safety mechanism. The history can never be replayed correctly if the code path changed in an incompatible way.

import { proxyActivities, sleep, workflow } from "@temporalio/workflow";
import type * as activities from "./activities";

const { chargePayment, sendConfirmationEmail } = proxyActivities<typeof activities>({
  startToCloseTimeout: "30 seconds",
  retry: {
    maximumAttempts: 3,
    initialInterval: "1 second",
    backoffCoefficient: 2,
  },
});

export async function orderFulfillmentWorkflow(orderId: string): Promise<void> {
  // During replay: SDK intercepts this, checks history for ActivityTaskCompleted
  // If found: returns the recorded result without calling the activity again
  // If not found: schedules a new ActivityTaskScheduled event
  await chargePayment(orderId);

  // During replay: SDK checks for TimerFired event at this position
  // If found: continues immediately without waiting
  // If not found: schedules a timer for 5 minutes
  await sleep("5 minutes");

  await sendConfirmationEmail(orderId);
}

Handling non-determinism safely when evolving workflows requires versioning. The patched API (Go SDK) and deprecatePatch / getVersion equivalents in other SDKs let you branch the code path based on whether the execution started before or after a code change. Executions that began before the patch follow the old path during replay; new executions follow the new path.

Activity Task Lifecycle: Timeouts, Heartbeating, and Retry Policies

Activities are where side effects live. They run outside the deterministic execution context and can do anything: call databases, write files, call external APIs. The history service tracks their lifecycle through a distinct set of timeout semantics.

Schedule-to-start timeout bounds how long a task can sit in the task queue waiting for a worker to pick it up. If no worker polls the queue within this window, the activity is failed with a ScheduleToStartTimeoutError. This timeout catches task queue saturation: workers are down, or the queue has grown beyond worker capacity.

Start-to-close timeout bounds the total time from when a worker picks up the task to when it must return a result. This is the primary execution timeout for the activity body.

Schedule-to-close timeout is an outer bound covering both queue time and execution time. It fires if the total time from scheduling exceeds the limit, regardless of which phase the activity is in.

Heartbeating is the mechanism for long-running activities. The worker calls Context.heartbeat(details) periodically during execution. Each heartbeat resets the heartbeat timeout clock. If the history service does not receive a heartbeat within the heartbeatTimeout, the activity is considered failed and will be retried. Critically, the details passed to heartbeat are persisted by the history service and returned to the worker in the ActivityTaskStarted event on retry. This lets a long-running activity resume from a checkpoint rather than restarting from scratch.

import { Context, activityInfo } from "@temporalio/activity";

interface ChunkProgress {
  lastProcessedOffset: number;
}

export async function processLargeDataset(datasetId: string): Promise<void> {
  const info = activityInfo();

  // Retrieve progress from heartbeat details on retry
  const lastProgress = info.heartbeatDetails as ChunkProgress | undefined;
  let offset = lastProgress?.lastProcessedOffset ?? 0;

  while (offset < DATASET_SIZE) {
    await processChunk(datasetId, offset);
    offset += CHUNK_SIZE;

    // Persist progress and prove to Temporal we are still alive
    Context.current().heartbeat({ lastProcessedOffset: offset });
  }
}

Retry policies compose with timeouts. A policy with maximumAttempts: 5, initialInterval: "1s", and backoffCoefficient: 2 will retry at 1s, 2s, 4s, 8s before the fifth and final attempt. Each retry produces a new ActivityTaskScheduled event in the history. The history service enforces the schedule-to-close timeout across all retry attempts combined, so a tight outer timeout can cause retry exhaustion before the maximum attempt count is reached.

Task Queue Mechanics and Sticky Execution

Task queues are the delivery mechanism between the history service and application workers. Workers register by long-polling: they send a PollWorkflowTaskQueue or PollActivityTaskQueue request to the frontend, which holds the connection open (up to 60 seconds by default) until a task is available or the poll times out. The matching service manages the in-flight polls and dispatches tasks to waiting connections.

Sticky execution is an optimization that avoids full history replay on every workflow task. When a worker completes a workflow task, it registers a “sticky task queue” specific to that worker instance (typically <hostname>:<pid>:sticky). Subsequent workflow tasks for the same execution are dispatched to this sticky queue first, where the worker’s in-process cache still holds the fully-replayed workflow state. The worker can continue forward from the cached state rather than replaying the entire history from scratch.

If the sticky worker is unavailable (worker crashed, sticky queue timeout elapsed, which defaults to 5 seconds), the history service falls back to the normal task queue. Any available worker picks up the task, replays the full history, and continues. The sticky cache hit rate is therefore a meaningful operational metric: consistently low cache hit rates indicate workflow tasks are frequently falling back to full replay, which adds latency and CPU load.

// SDK worker configuration that controls sticky execution and polling behavior

import { Worker } from "@temporalio/worker";

const worker = await Worker.create({
  workflowsPath: require.resolve("./workflows"),
  activities,
  taskQueue: "order-fulfillment",
  // Number of concurrent workflow task polls (sticky + normal)
  maxConcurrentWorkflowTaskExecutors: 40,
  // Number of concurrent activity task polls
  maxConcurrentActivityTaskExecutors: 100,
  // Max in-memory cached workflow states
  maxCachedWorkflows: 200,
  // Sticky schedule-to-start timeout before fallback to normal queue
  stickyScheduleToStartTimeout: "5 seconds",
});

Timer Implementation for Durable Sleep

workflow.sleep("30 days") is not a timer running in a process. It is a durable event scheduled by the history service. The history service writes a TimerStarted event and persists a timer record keyed to the workflow execution. When the timer fires, the history service writes a TimerFired event and schedules a new workflow task. The workflow’s sticky worker (or any available worker on replay) processes the TimerFired event during the next execution and the workflow function resumes past the await sleep(...) call.

This is why Temporal can implement sleep("1 year") reliably: the timer lives in the persistence store, not in process memory. There is no thread, no in-memory scheduler, and no heartbeat loop involved. The timer survives cluster restarts, node failures, and worker deployments.

Cron scheduling is implemented on top of the same mechanism. A cron workflow is a workflow that, when it completes, schedules a ContinueAsNew action with a timer for the next cron interval. ContinueAsNew is also the mechanism for managing history size: a workflow that would otherwise accumulate thousands of events can call continueAsNew with its current state as input, which closes the current history with a WorkflowExecutionContinuedAsNew event and starts a new execution from a clean history.

Signals and Queries

Signals and queries are the two mechanisms for external interaction with a running workflow.

A signal is an asynchronous write. The caller sends a SignalWorkflowExecution request; the frontend delivers it to the history service, which appends a WorkflowExecutionSignaled event. A new workflow task is scheduled. When the worker processes it, the workflow function’s signal handler receives the signal payload. Signals are durable: if the signal arrives while no workflow task is running, it waits in the event history until the next workflow task.

A query is a synchronous read. The caller sends a QueryWorkflow request; the frontend delivers it to the matching service, which routes it to a worker with the workflow in its sticky cache (or falls back to any worker for full replay). The worker executes the query handler against the current in-memory state and returns the result directly. Queries do not modify history. If no worker is available to process the query, it fails. Queries are not durable.

The asymmetry matters: signals accumulate reliably even if workers are down; queries fail fast if no worker is alive to answer. Design accordingly when choosing which mechanism to use for external interaction.

The Visibility System

The visibility system provides a queryable index of workflow executions. Temporal exposes a SQL-like query syntax: WorkflowType = 'OrderFulfillment' AND ExecutionStatus = 'Running' AND StartTime > '2026-01-01'. Internally, this query runs against a search attribute index backed by Elasticsearch (for advanced visibility) or a simpler SQL table (for standard visibility with limited operators).

Search attributes are key-value pairs you attach to a workflow execution via the UpsertSearchAttributes command. They are indexed asynchronously after the event is processed. Lag between the event and indexing means visibility queries may return stale results for recently-started or recently-completed workflows.

Standard visibility (the default without Elasticsearch) supports exact matches on built-in attributes: WorkflowId, RunId, WorkflowType, ExecutionStatus, StartTime, CloseTime. Advanced visibility adds range queries, ordering, custom search attributes, and full-text matching. For production systems where operational tooling needs to find, filter, or count workflow executions by business-level attributes (customerId, orderId, region), Elasticsearch-backed advanced visibility is effectively required.

Production Considerations

History size limits. The history service enforces a default maximum of 50,000 events per workflow execution. Approaching this limit is a signal that the workflow is doing too much in a single execution. Use continueAsNew before hitting the limit: check workflowInfo().historyLength and call continueAsNew proactively when it exceeds a safe threshold (typically 10,000 events for buffer).

Worker tuning. maxConcurrentWorkflowTaskExecutors controls how many workflow tasks a single worker process handles in parallel. Workflow task execution is CPU-bound during replay; setting this too high on a small instance causes context-switch pressure. Activity task executors are I/O-bound; they can be set much higher. A useful starting point: workflow task executors equal to 2x the number of CPU cores, activity task executors equal to 4-8x.

Namespace isolation. Namespaces are the primary isolation boundary. They have independent task queues, rate limits, and retention periods. Workflows in one namespace cannot signal or query workflows in another without explicit cross-namespace calls (which go through the frontend service). Production environments should use separate namespaces for different services or criticality tiers to prevent a noisy workflow from consuming the entire cluster’s task queue capacity.

Codec server for data encryption. All workflow input, activity results, signal payloads, and search attribute values are stored in the persistence layer in the format the worker sends them. The codec server pattern inserts a remotely-called codec between the SDK and the cluster: the SDK encrypts payloads before sending and decrypts after receiving. The cluster stores only encrypted bytes. The codec server (which you operate) holds the keys. Temporal Cloud documents this as the recommended pattern for PCI or HIPAA-scope data since Temporal’s servers never see plaintext.

Shard hotspots. If a large fraction of your workflow IDs are derived from the same prefix or hash to the same shard, that shard’s history service node becomes a bottleneck. Distribute workflow IDs with high entropy (UUIDs or ULIDs are natural fits). Avoid sequential integer IDs.

Tradeoffs Comparison

Dimension	Temporal	Inngest	AWS Step Functions	Cloudflare Workflows	Restate
Execution model	Durable workflow functions with event history replay	Event-driven step functions with retry and concurrency	State machine JSON definitions (ASL)	Durable execution on V8 isolates at edge	Durable function calls with journal-based replay
State management	Append-only event history in Cassandra / SQL	Event log in managed store	State machine transitions in DynamoDB	Durable object storage at edge	Journal persisted in RocksDB or managed backend
Language support	Go, Java, TypeScript, Python, .NET, PHP	TypeScript, JavaScript	Any (Lambda invocation)	TypeScript, JavaScript	TypeScript, Java, Kotlin, Go
Self-hosted option	Yes (full open-source server)	No (cloud only)	No (AWS-managed)	No (Cloudflare platform)	Yes (self-hosted or managed)
Latency profile	Low ms for task dispatch; replay adds overhead for large histories	Low ms for step invocation	Seconds for Standard type; ms for Express type	Sub-ms to ms (edge colocation)	Low ms; local journal read is fast
Pricing model	Infrastructure cost (self-hosted) or action-based (Temporal Cloud)	Action-based per step execution	State transition count (Standard) or duration (Express)	Cloudflare Workers pricing per request and CPU time	Self-hosted: infrastructure; managed: per invocation
Sweet spot	Long-running, complex workflows requiring full code expressiveness and strong consistency guarantees	Serverless background jobs with simple retry and fan-out needs	AWS-native workflows where ASL’s visual tooling and IAM integration outweigh flexibility	Short-duration workflows at the edge collocated with Cloudflare Workers	Low-latency durable calls where journal read speed matters and SDK ergonomics (RPC-style) are preferred

Temporal’s replay model gives it capabilities the others lack: arbitrary code logic, loops, recursion, local state that persists across failures, and signals that arrive at any point. The cost is operational complexity (cluster to run, shards to size, history to manage) and the determinism constraint that requires disciplined code review for every workflow function change.

Closing

The event history is the core abstraction in Temporal. Once you understand that a workflow execution is nothing more than an append-only log replayed through a deterministic function, every other behavior falls into place: why non-determinism is fatal, why timers survive cluster restarts, why signals are durable but queries are not, and why continueAsNew exists. The cluster services (frontend, history, matching) are the machinery that makes the log durable, the tasks deliverable, and the workers findable. Most production issues with Temporal trace back to one of three sources: history growth, task queue saturation, or worker non-determinism. Understanding the internal model gives you the diagnostic vocabulary to find and fix them.