Designing a Circuit Breaker: Failure Detection, State Machines, and Cascading Failure Prevention in Distributed Systems

The circuit breaker is probably the most cited resilience pattern in distributed systems. It is also among the most poorly implemented. Teams add a library dependency, configure a threshold somewhere between 50% and 80% without much thought, and consider the job done. Then the first real outage happens and the circuit either trips on noise at 2 a.m. or stays closed while a dependency melts down.

The gap between understanding the pattern and implementing it correctly is large. This article covers what that gap contains: how the state machine actually works, how to choose and calibrate a failure signal, how circuit breakers interact with retry policies and bulkheads, and where to put circuit breaking logic in modern infrastructure.

Why Cascading Failures Happen

Distributed systems fail in chains. A payment service starts returning 503s. Your checkout service keeps calling it, each request blocking for the full 30-second HTTP timeout. The checkout service’s thread pool fills. Incoming requests to checkout queue behind blocked calls. Within two minutes, checkout is effectively down. The order service, which calls checkout, starts timing out. The cascade continues until a single struggling service has disabled a significant fraction of your system.

The core mechanism is resource exhaustion driven by slow failures. Fast failures (immediate errors, immediate timeouts) release resources quickly. Slow failures accumulate. The checkout service does not need the payment service to be completely unreachable to fail. It only needs the payment service to be slow enough that requests pile up faster than they drain.

The circuit breaker interrupts this by converting slow failures into fast ones. When the circuit opens, calls to the failing dependency fail immediately rather than waiting for a timeout. Thread pools drain, queues clear, and the rest of the system recovers.

The Three-State Machine

The pattern has three states with specific behaviors and defined transitions between them.

Closed is normal operation. Requests pass through to the dependency. The breaker tracks a failure signal using a sliding window. As long as the signal stays below the configured threshold, the circuit remains closed.

Open is the tripped state. The dependency is presumed unavailable. Requests are rejected immediately, without contacting the dependency. This is the protection mechanism: a 30-second timeout becomes a sub-millisecond rejection.

Half-open is a probe state. After the configured recovery period expires, the circuit allows a small number of probe requests through to the dependency. If probes succeed, the circuit transitions to closed. If they fail, it transitions back to open and resets the timer. This is what makes circuit breakers self-healing rather than requiring manual reset.

The transitions are as important as the states:

• Closed to Open: failure signal exceeds threshold, given a minimum number of observed requests
• Open to Half-open: recovery timer expires
• Half-open to Closed: probe request succeeds
• Half-open to Open: probe request fails

The minimum request guard on the closed-to-open transition is easy to overlook and critical to get right. A 50% failure threshold with no minimum means the circuit opens after the first request in a new window fails. You need enough data points before the rate is meaningful.

Failure Detection Strategies

Choosing the right failure signal is more consequential than tuning the threshold. Different signals detect different failure modes and have different false positive characteristics.

Failure Rate

Track the percentage of requests that fail within a rolling window. The circuit opens when that rate exceeds a threshold, provided a minimum request count has been observed.

Failure rate works correctly regardless of traffic volume. At 10,000 requests per minute, a 2% failure rate is 200 errors per minute and might be acceptable noise. At 20 requests per minute, 2% is a fraction of one error per minute that will never accumulate enough to matter. Rate-based thresholds handle this correctly; count-based thresholds do not.

The tradeoff: at low throughput, you need a substantial minimum request count before the rate is statistically meaningful. A minimum of 10 requests with a 50% threshold means the circuit will not trip for a long time if the service only handles 5 requests per minute.

Failure Count (Consecutive)

Count consecutive failures without intervening successes. The circuit opens after N consecutive failures.

This is simpler to reason about but fragile. A dependency that fails, succeeds, fails, succeeds in alternation will never open the circuit no matter how degraded it is. It also opens immediately when the count is hit, regardless of total traffic context.

Consecutive failure count is appropriate for low-throughput, high-criticality operations where any consecutive failures are meaningful. It is the wrong choice for high-throughput services with background noise.

Slow Call Rate

Track the percentage of calls that exceed a response time threshold. Open when slow calls exceed a percentage of total calls.

This detects degradation that does not produce errors. A dependency that hangs for 25 seconds on every request looks healthy from an HTTP status code perspective but is actively draining your thread pool. Slow call tracking catches this.

The tradeoff: you need to define what “slow” means per dependency, which requires baseline data. Setting the slow threshold too low will trip the circuit on normal traffic spikes; too high will miss genuine degradation until it becomes severe.

Combined: Failure Rate with Slow Call Fallback

The production default is usually failure rate as the primary signal with slow call rate as a secondary signal. Open the circuit if either exceeds its threshold. This handles both error-producing failures and silent degradation.

Resilience4j uses this model. It is the right starting point for most services.

Sliding Window Design

The failure signal is computed over a window. Two approaches have meaningfully different characteristics.

Count-based window: track the last N requests, regardless of when they occurred. Simple to implement. The window turns over at a rate proportional to traffic. Sparse traffic means the window contains stale data from minutes or hours ago. Burst traffic means the window turns over in seconds. Neither is ideal.

Time-based sliding window: track all requests within the last N seconds. More accurate for variable traffic. An implementation using a pruned array is straightforward:

interface RequestRecord {
  timestamp: number;
  success: boolean;
  durationMs: number;
}

class SlidingWindow {
  private records: RequestRecord[] = [];

  constructor(
    private readonly windowMs: number,
    private readonly slowCallThresholdMs: number
  ) {}

  record(success: boolean, durationMs: number): void {
    const now = Date.now();
    this.prune(now);
    this.records.push({ timestamp: now, success, durationMs });
  }

  private prune(now: number): void {
    const cutoff = now - this.windowMs;
    // find the first record still within the window
    let i = 0;
    while (i < this.records.length && this.records[i].timestamp < cutoff) {
      i++;
    }
    if (i > 0) this.records = this.records.slice(i);
  }

  metrics(): { total: number; failureRate: number; slowCallRate: number } {
    this.prune(Date.now());
    const total = this.records.length;
    if (total === 0) return { total: 0, failureRate: 0, slowCallRate: 0 };

    const failures = this.records.filter((r) => !r.success).length;
    const slow = this.records.filter(
      (r) => r.durationMs >= this.slowCallThresholdMs
    ).length;

    return {
      total,
      failureRate: failures / total,
      slowCallRate: slow / total,
    };
  }
}

For most production uses, a time-based window with 60-second duration is the correct default. Count-based windows are simpler and appropriate when traffic is steady and high-volume.

Timeout Calibration

A circuit breaker without accurate timeouts is incomplete. The circuit can only detect a failure when the call throws. A call that hangs indefinitely is not seen as a failure until a timeout fires or the process crashes.

Timeout calibration requires three data points: the dependency’s normal response time distribution (p50, p95, p99), the upstream caller’s own timeout, and the acceptable latency budget.

The process:

1. Measure your dependency’s p99 response time under normal load. This is your baseline.
2. Set the timeout at 2x to 3x p99. This gives legitimate slow requests room to complete while catching genuine hangs before they exhaust resources.
3. Ensure the timeout is shorter than the upstream caller’s timeout by a margin that leaves room for retry or fallback logic.

If your API gateway cancels requests after 10 seconds, your outbound call timeout should be 4 to 5 seconds. That leaves room to execute a fallback and return a meaningful response before the gateway cancels the connection.

function withTimeout<T>(
  fn: () => Promise<T>,
  ms: number,
  label: string
): () => Promise<T> {
  return () =>
    new Promise<T>((resolve, reject) => {
      const timer = setTimeout(() => {
        reject(new Error(`Timeout: ${label} exceeded ${ms}ms`));
      }, ms);

      fn().then(
        (result) => {
          clearTimeout(timer);
          resolve(result);
        },
        (err) => {
          clearTimeout(timer);
          reject(err);
        }
      );
    });
}

The timeout label matters for debugging. When a timeout fires at 3 a.m., you want the error to say which dependency timed out, not just “operation timed out.”

The Retry Relationship

Circuit breakers and retry policies need to be composed carefully or they work against each other.

The wrong approach: wrap the circuit breaker call with retry logic that retries on any error. When the circuit opens and throws a fast-fail error, the retry logic catches it and tries again. This drives more calls into the circuit’s failure tracking, potentially extending the time it stays open.

The correct approach: the circuit breaker wraps a single attempt. Retry logic wraps the circuit breaker. When the circuit throws a CircuitOpenError, the retry logic stops immediately rather than retrying.

class CircuitOpenError extends Error {
  constructor(public readonly circuitName: string) {
    super(`Circuit open: ${circuitName}`);
    this.name = "CircuitOpenError";
  }
}

async function callWithResilience<T>(
  circuitBreaker: CircuitBreaker,
  fn: () => Promise<T>,
  maxRetries: number
): Promise<T> {
  let attempt = 0;

  while (true) {
    try {
      return await circuitBreaker.execute(fn);
    } catch (err) {
      // circuit is open: fail immediately, do not retry
      if (err instanceof CircuitOpenError) throw err;

      attempt++;
      if (attempt > maxRetries) throw err;

      // transient error: wait with exponential backoff and try again
      const delay = Math.min(100 * Math.pow(2, attempt - 1), 2_000);
      const jitter = Math.random() * delay;
      await new Promise((resolve) => setTimeout(resolve, delay + jitter));
    }
  }
}

A retry that fires on a CircuitOpenError converts your resilience layer into an amplification loop. The check for CircuitOpenError before any delay or requeue is the load-bearing line.

The second interaction: retries count as separate requests in the circuit’s failure window. If your retry fires three times for one logical user operation and all three fail, the circuit records three failures. This is correct behavior, not a bug. Three failed attempts against a struggling dependency are three real failures. If your failure threshold is too sensitive, retries will trip the circuit faster than you expect. Factor this into threshold calibration.

Composing with the Bulkhead Pattern

Circuit breakers react to failure rate. Bulkheads constrain concurrency. They address different dimensions of the same problem and compose naturally.

The problem bulkheads solve: a slow dependency accumulates in-flight requests. Even after the circuit opens and stops new requests, the already-in-flight requests hold their connections for the duration of their timeouts. For a service with a 30-second timeout and 100 concurrent requests in flight before the circuit trips, those 100 connections are unavailable for 30 seconds after the circuit opens.

With a bulkhead, the slow dependency is capped at a maximum concurrency. When the cap is hit, new requests fail fast at the bulkhead rather than queuing behind slow in-flight calls. The circuit handles the failure rate signal; the bulkhead limits the concurrency damage.

class Bulkhead {
  private inFlight = 0;

  constructor(private readonly maxConcurrent: number) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.inFlight >= this.maxConcurrent) {
      throw new BulkheadRejectionError(
        `Bulkhead full (max: ${this.maxConcurrent})`
      );
    }
    this.inFlight++;
    try {
      return await fn();
    } finally {
      this.inFlight--;
    }
  }
}

class BulkheadRejectionError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "BulkheadRejectionError";
  }
}

// Composition order:
// bulkhead (caps concurrency)
//   -> circuit breaker (tracks failure rate)
//     -> timeout (bounds per-call latency)
//       -> actual call

const bulkhead = new Bulkhead(20);
const circuit = new CircuitBreaker({ name: "inventory-service" });

async function getInventoryLevel(sku: string): Promise<number> {
  return bulkhead.execute(() =>
    circuit.execute(
      withTimeout(() => inventoryClient.getLevel(sku), 3_000, "inventory.getLevel")
    )
  );
}

Sizing the bulkhead: start with the maximum concurrency you observed for this dependency under peak load plus 20%. A dependency that peaks at 50 concurrent requests in production gets a bulkhead of 60. This gives headroom for normal spikes without being so large that it provides no protection.

Tradeoffs Table

Concern	Library-Level	Infrastructure-Level (Service Mesh)
Granularity	Per-operation	Per-service or per-route
Configuration	Code changes	Config/control plane
Language coverage	Language-specific	Language-agnostic
Request classification	Arbitrary (isFailure callback)	HTTP status codes only
Observability	Built into app metrics	Separate mesh telemetry
Deployment overhead	None	Sidecar per pod
Protocol support	HTTP, gRPC, queues, DB calls	HTTP, gRPC (mesh-native)
Fallback logic	In application code	Not available
Operational cost	Low	High

The tradeoffs are not trivial. Infrastructure-level circuit breaking (Istio, Envoy, Linkerd) requires a service mesh, adds sidecar latency, and provides no application-level fallback. When a circuit opens at the mesh layer, the caller receives a 503 with no way to execute cached responses or degraded logic. Library-level breaking lets you catch CircuitOpenError and return a cached or default response.

For non-HTTP protocols (database calls, queue consumption, gRPC streams, in-process calls) there is no infrastructure option. Library-level is the only path.

The right answer for most teams: library-level circuit breaking for critical dependencies with application-specific fallback requirements, mesh-level for perimeter traffic management where HTTP semantics are sufficient.

From Netflix Hystrix to Modern Alternatives

Netflix Hystrix popularized the pattern at scale. Introduced around 2012 and open-sourced in 2013, it was built to protect Netflix’s microservice architecture from the kind of cascading failures described above. Hystrix used a thread pool isolation model: each dependency got its own thread pool, and a bulkhead was implicit in the pool size. It also introduced the notion of a command pattern, where each remote call was wrapped in a HystrixCommand that handled circuit breaking, timeout, fallback, and metrics in a single abstraction.

Hystrix entered maintenance mode in 2018. Netflix’s own teams moved toward resilience handled at the infrastructure level, particularly as service meshes matured. The library’s thread pool model had real overhead at high request rates, and the command wrapper pattern was verbose.

The most common modern library-level replacement is Resilience4j (JVM ecosystem). Its design is function-decorator-based rather than command-based, which is lighter and more composable. The failure detection model is more flexible: it supports rate-based and count-based windows, slow call detection, and composite failure criteria natively.

In the TypeScript/Node.js ecosystem, there is no dominant equivalent. The implementation in this article, and the one in the companion circuit breaker article on this site, covers the same ground. cockatiel is a well-maintained library that provides circuit breakers and other resilience primitives with a clean API.

At the infrastructure level:

• Istio: outlier detection (passive circuit breaking based on 5xx rate) and retry policies. No fallback, no non-HTTP support.
• Envoy: the proxy underlying Istio; circuit breaking is configured via CircuitBreakers proto config, controlling pending requests, max connections, max retries.
• AWS App Mesh: similar to Istio, wraps Envoy.
• AWS SDK clients: built-in retry and timeout, no circuit breaking by default.

For most Node.js services, the correct approach is to implement a well-tested, typed circuit breaker class with a sliding window, rate-based threshold, and event emission. The core is under 200 lines. The complexity is in tuning and observability, not the state machine.

Production Considerations

Per-process vs. shared state. A circuit breaker that lives in application memory tracks failures per process instance. In a fleet of 50 application servers, each instance has its own circuit. One instance may open while 49 others still route traffic to the failing dependency. This is usually acceptable: each instance will eventually trip on its own data. For dependencies where coordinated tripping is critical (authentication services, payment processors), shared state in Redis adds coordination at the cost of a network call on every execute. Measure the latency impact before adopting.

Circuit per operation, not per service. A single circuit for “payments-service” trips when anything in that service fails. If the charge endpoint is failing but refund is healthy, the single circuit blocks refunds unnecessarily. Create per-operation circuits for services where endpoint health varies independently.

Configuration per environment. Thresholds calibrated against production traffic patterns will over-trip in staging, where traffic is sparse and dependencies are less reliable. Externalize minimumRequests, failureThreshold, and openDurationMs in environment-specific configuration.

Test the fallback, not just the circuit. Most teams verify that the circuit trips correctly under simulated load. Fewer teams verify that the fallback path holds under the peak traffic that will actually hit it when the circuit opens. A fallback that calls a cache service that gets overwhelmed has moved the cascading failure one layer up. Chaos testing the fallback path under load is not optional for critical services.

Half-open probe design. The default of one probe request in the half-open state is conservative. For high-traffic services, a single probe success might not represent the dependency’s full recovery. Consider halfOpenMaxRequests: 5 with a success threshold (e.g., 4 of 5 probes must succeed) before fully closing. This adds a transition-to-closed check that the basic state machine does not include.

// Extended half-open: require N successes before closing
private halfOpenSuccesses = 0;
private readonly halfOpenSuccessThreshold: number;

private onSuccess(previousState: CircuitState): void {
  this.window.push({ timestamp: Date.now(), success: true, durationMs: 0 });
  if (previousState === "half-open") {
    this.halfOpenSuccesses++;
    if (this.halfOpenSuccesses >= this.halfOpenSuccessThreshold) {
      this.halfOpenSuccesses = 0;
      this.transition("closed");
    }
  }
}

Alert on flapping. A circuit that repeatedly opens and closes within a short window indicates a dependency oscillating around your failure threshold. This is distinct from a clean open-recover cycle. Flapping usually means one of: the threshold is too tight for the dependency’s normal variability, the dependency has an intermittent problem, or the recovery window is too short for the dependency to fully stabilize before probing begins. Track consecutive transitions and alert when the count exceeds a threshold within a time window.

Where This Breaks Down

Low throughput, high-value operations. If a circuit covers an endpoint receiving 5 requests per minute, the minimum request guard means you accumulate enough data for a decision only every 2 minutes of sustained failures. The circuit provides meaningful protection only for services with enough traffic to produce a statistically valid signal in a reasonable time frame.

Write paths with partial state. A circuit that trips midway through a multi-step write sequence leaves the system in a partially-applied state. Step 1 wrote to service A. Step 2 calls service B, which is circuit-protected. The circuit opens. You now have an incomplete write. Saga-style compensating transactions handle this, but they add complexity. Evaluate whether the circuit is the right protection mechanism for write-heavy, multi-step operations, or whether you need a different approach entirely.

Self-inflicted failures. A circuit breaker protects against downstream failure. It provides no protection against failures you cause yourself: a bad deploy, a query that performs well under 100 rows but degrades at 10,000, a memory leak that builds slowly. These show up as rising failure rates and slow calls, and the circuit will eventually trip on them, but that is a symptom response. The underlying problem needs a different fix.

The state machine is simple. Everything around it is not. The investment is in calibrating the right failure signal, setting thresholds against real traffic data, integrating correctly with retry and bulkhead policies, and building fallback logic that actually handles the failure case. That work is the difference between a circuit breaker that protects you and one that surprises you.