The Circuit Breaker Pattern: Preventing Cascading Failures in Distributed Systems

Picture this: your inventory service starts returning HTTP 503 responses. Maybe a database migration went sideways, maybe a memory leak finally caught up. Your order service keeps calling inventory anyway, each request blocking for the full 30-second timeout before giving up. Thread pools fill. Incoming order requests queue behind those blocked calls. Within two minutes, the order service is effectively down. Then the checkout service, which depends on orders, starts timing out. The cascade continues until a single struggling microservice has taken out a significant portion of your system.

This failure mode has a name: cascading failure. And the circuit breaker pattern exists specifically to interrupt it.

The analogy is accurate: a physical circuit breaker in your electrical panel trips when current exceeds safe levels, cutting the circuit before wires overheat. In software, when a downstream dependency exceeds a failure threshold, the circuit breaker trips, stopping requests to that dependency before they exhaust connection pools, thread budgets, or async callback queues.

Most explanations cover the three-state machine and stop there. This article goes deeper: how to implement it correctly, what failure metric to actually track, how to tune it per environment, where it composes with retries and bulkheads, and where it actively makes things worse.

The State Machine

Three states, each with specific behaviors and transition conditions.

Closed is the normal operating state. Requests flow through to the dependency. The breaker tracks failures using a sliding window. As long as the failure rate stays below the configured threshold, the circuit stays closed.

Open is the tripped state. The dependency is presumed unavailable. Requests are rejected immediately, without touching the dependency. This is the core value proposition: a 30-second timeout becomes a sub-millisecond rejection. An open circuit caps the blast radius of a downstream failure.

Half-open is a probe state. After the configured recovery window expires, the circuit allows a small number of test requests through. If they succeed, the circuit closes and resumes normal operation. If they fail, the circuit re-opens and resets the timer. Half-open is what makes circuit breakers self-healing rather than requiring manual intervention.

The transitions matter as much as the states:

• Closed to Open: triggered when the failure rate within the current window exceeds the threshold, provided a minimum number of requests have been observed.
• Open to Half-open: triggered automatically after openDurationMs has elapsed.
• Half-open to Closed: triggered when a probe request succeeds.
• Half-open to Open: triggered when a probe request fails.

That minimum request guard is easy to overlook and critical to get right. If your threshold is 50% and the first request in a new window fails, you have a 100% failure rate from a single data point. Without a minimum, the circuit trips on noise.

Failure Rate vs. Failure Count

The threshold should be a rate, not a count. “5 consecutive failures” sounds reasonable until you consider throughput context. For a service handling 10,000 requests per minute, 5 failures in a 60-second window is statistical noise. For a service handling 20 requests per minute, 5 failures in 60 seconds is a 25% failure rate that probably warrants attention.

Rate-based thresholds work correctly at any traffic level. The tradeoff is that at very low throughput, the minimum request guard must be set appropriately so the circuit does not trip on a tiny denominator.

Sliding Window vs. Count Window

Two common approaches to tracking failure data:

Count-based window: track the last N requests regardless of time. Simple to implement, but N requests could span 1 second or 10 minutes depending on traffic. Sparse traffic means the window contains stale data; burst traffic means the window turns over rapidly.

Time-based sliding window: track all requests within the last N milliseconds. More accurate for variable traffic. The implementation requires pruning stale records, which the implementation below handles inline.

For most production uses, a time-based window is the right choice.

A Production Implementation

The following implementation uses a time-based sliding window, rate-based failure threshold, bounded half-open concurrency, and event emission for observability. It is intentionally self-contained with no external dependencies beyond Node’s EventEmitter.

import { EventEmitter } from "events";

type CircuitState = "closed" | "open" | "half-open";

interface CircuitBreakerOptions {
  name: string;
  failureThreshold?: number;     // failure rate to trip (0.0–1.0), default 0.5
  minimumRequests?: number;      // minimum requests before rate is evaluated, default 10
  windowMs?: number;             // rolling window duration in ms, default 60_000
  openDurationMs?: number;       // how long to stay open before probing, default 30_000
  halfOpenMaxRequests?: number;  // concurrent probes allowed in half-open, default 1
  isFailure?: (err: unknown) => boolean; // optional: filter which errors count as failures
}

interface RequestRecord {
  timestamp: number;
  success: boolean;
}

export class CircuitOpenError extends Error {
  readonly circuitName: string;
  constructor(circuitName: string) {
    super(`Circuit breaker is open: ${circuitName}`);
    this.name = "CircuitOpenError";
    this.circuitName = circuitName;
  }
}

export class CircuitBreaker extends EventEmitter {
  private readonly name: string;
  private readonly failureThreshold: number;
  private readonly minimumRequests: number;
  private readonly windowMs: number;
  private readonly openDurationMs: number;
  private readonly halfOpenMaxRequests: number;
  private readonly isFailure: (err: unknown) => boolean;

  private state: CircuitState = "closed";
  private window: RequestRecord[] = [];
  private openedAt: number | null = null;
  private halfOpenInFlight = 0;

  constructor(options: CircuitBreakerOptions) {
    super();
    this.name = options.name;
    this.failureThreshold = options.failureThreshold ?? 0.5;
    this.minimumRequests = options.minimumRequests ?? 10;
    this.windowMs = options.windowMs ?? 60_000;
    this.openDurationMs = options.openDurationMs ?? 30_000;
    this.halfOpenMaxRequests = options.halfOpenMaxRequests ?? 1;
    this.isFailure = options.isFailure ?? (() => true);
  }

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    this.pruneWindow();
    const state = this.currentState();

    if (state === "open") {
      this.emit("rejected", { name: this.name, state: "open" });
      throw new CircuitOpenError(this.name);
    }

    if (state === "half-open") {
      if (this.halfOpenInFlight >= this.halfOpenMaxRequests) {
        // probes already in flight, shed load rather than pile on
        this.emit("rejected", { name: this.name, state: "half-open" });
        throw new CircuitOpenError(this.name);
      }
      this.halfOpenInFlight++;
    }

    const startedAt = Date.now();
    try {
      const result = await fn();
      this.onSuccess(state);
      return result;
    } catch (err) {
      this.onFailure(state, err);
      throw err;
    } finally {
      if (state === "half-open") {
        this.halfOpenInFlight = Math.max(0, this.halfOpenInFlight - 1);
      }
      this.emit("request", {
        name: this.name,
        state,
        durationMs: Date.now() - startedAt,
      });
    }
  }

  private currentState(): CircuitState {
    if (this.state === "open" && this.openedAt !== null) {
      if (Date.now() - this.openedAt >= this.openDurationMs) {
        this.transition("half-open");
      }
    }
    return this.state;
  }

  private onSuccess(previousState: CircuitState): void {
    this.window.push({ timestamp: Date.now(), success: true });
    if (previousState === "half-open") {
      this.transition("closed");
    }
  }

  private onFailure(previousState: CircuitState, err: unknown): void {
    const countAsFailure = this.isFailure(err);
    this.window.push({ timestamp: Date.now(), success: !countAsFailure });

    if (previousState === "half-open") {
      this.transition("open");
      return;
    }

    if (previousState === "closed" && countAsFailure) {
      this.evaluateFailureRate();
    }
  }

  private evaluateFailureRate(): void {
    if (this.window.length < this.minimumRequests) return;
    const failures = this.window.filter((r) => !r.success).length;
    const rate = failures / this.window.length;
    if (rate >= this.failureThreshold) {
      this.transition("open");
    }
  }

  private transition(next: CircuitState): void {
    const prev = this.state;
    this.state = next;

    if (next === "open") {
      this.openedAt = Date.now();
      this.window = []; // clear window so post-recovery probe starts fresh
    } else if (next === "closed") {
      this.openedAt = null;
      this.window = [];
    }

    this.emit("stateChange", {
      name: this.name,
      from: prev,
      to: next,
      timestamp: Date.now(),
    });
  }

  private pruneWindow(): void {
    const cutoff = Date.now() - this.windowMs;
    this.window = this.window.filter((r) => r.timestamp >= cutoff);
  }

  getMetrics(): {
    state: CircuitState;
    total: number;
    failures: number;
    failureRate: number;
  } {
    this.pruneWindow();
    const total = this.window.length;
    const failures = this.window.filter((r) => !r.success).length;
    return {
      state: this.state,
      total,
      failures,
      failureRate: total > 0 ? failures / total : 0,
    };
  }
}

A few design decisions worth unpacking.

The isFailure callback lets callers decide which errors count as failures. HTTP 404 is not a dependency failure; it is an application error that you should not use to trip the circuit. HTTP 503 or a TCP connection refused absolutely is. Separating “error thrown” from “dependency failure” prevents well-behaved errors from tripping the circuit.

const searchCircuit = new CircuitBreaker({
  name: "search-service",
  isFailure: (err) => {
    // 404 Not Found is not a dependency failure
    if (err instanceof HttpError && err.status === 404) return false;
    // timeouts and 5xx errors count as failures
    return true;
  },
});

Window clearing on transition prevents the burst of failures recorded just before opening from immediately re-opening the circuit after a successful probe. The probe starts fresh.

Inline pruning keeps the window accurate without a background timer, at the cost of a filter on every call. For typical circuit breaker traffic volumes, this is not a bottleneck.

Integrating with Retries and Timeouts

The Layering Order

The circuit breaker and retry logic must be ordered correctly, or they undermine each other.

The wrong order: retry logic wraps the circuit breaker call, and the retry itself wraps multiple attempts. Each attempt gets its own failure record. A retry that fires three times generates three failure records for what the application considers a single logical operation. Under load, retries amplify the failure signal and trip the circuit prematurely.

The correct order: circuit breaker wraps a single attempt. Retry logic wraps the circuit breaker. When the circuit is open, the retry receives a CircuitOpenError and should stop immediately rather than retrying.

async function callInventoryService(sku: string): Promise<InventoryLevel> {
  let attempt = 0;
  const maxRetries = 3;

  while (attempt <= maxRetries) {
    try {
      return await inventoryCircuit.execute(() => inventoryClient.getLevel(sku));
    } catch (err) {
      if (err instanceof CircuitOpenError) {
        // the circuit is open, do not retry, fail immediately
        throw err;
      }

      attempt++;
      if (attempt > maxRetries) throw err;

      // exponential backoff for transient errors
      const delay = Math.min(200 * Math.pow(2, attempt - 1), 2_000);
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }

  throw new Error("unreachable");
}

Timeouts Are Mandatory

A circuit breaker without timeouts is incomplete. Failure tracking only triggers when a call throws. A call that hangs for 30 seconds is not counted as a failure until the timeout fires. In the meantime, every blocked call holds a resource.

Wrap the function passed to execute with an explicit timeout:

function withTimeout<T>(fn: () => Promise<T>, ms: number): () => Promise<T> {
  return () =>
    new Promise<T>((resolve, reject) => {
      const timer = setTimeout(
        () => reject(new Error(`Timeout after ${ms}ms`)),
        ms
      );
      fn().then(
        (result) => { clearTimeout(timer); resolve(result); },
        (err)    => { clearTimeout(timer); reject(err); }
      );
    });
}

// The circuit sees a failure after 5 seconds, not after 30
await inventoryCircuit.execute(
  withTimeout(() => inventoryClient.getLevel(sku), 5_000)
);

Set the timeout below your upstream caller’s own timeout. If your API gateway cancels requests after 10 seconds, use a 5-second circuit timeout so you can fail fast and return a meaningful error before the gateway kills the connection.

Monitoring and Observability

A circuit breaker that trips silently is harder to debug than not having one at all. You need to know when circuits open, how long they stay open, whether half-open probes succeed, and how many user requests are being rejected.

The event emitter in the implementation above is the hook. Wire it to your metrics system:

function instrumentCircuitBreaker(
  breaker: CircuitBreaker,
  metrics: MetricsClient,
  alerting: AlertingClient
): void {
  const CRITICAL = new Set(["payments-service", "auth-service"]);

  breaker.on("stateChange", ({ name, from, to, timestamp }) => {
    metrics.increment("circuit_breaker.transition", { name, from, to });

    if (to === "open") {
      metrics.gauge("circuit_breaker.open_at", timestamp, { name });
      if (CRITICAL.has(name)) {
        alerting.page(`Circuit ${name} opened`, "P2");
      }
    }

    if (from === "open" && to === "closed") {
      metrics.increment("circuit_breaker.recovered", { name });
    }
  });

  breaker.on("rejected", ({ name, state }) => {
    metrics.increment("circuit_breaker.rejected", { name, state });
  });

  breaker.on("request", ({ name, state, durationMs }) => {
    metrics.histogram("circuit_breaker.duration_ms", durationMs, { name, state });
  });
}

Three dashboards that are worth building:

Circuit state timeline. A state diagram per named circuit showing closed/open/half-open over time. Circuits that repeatedly open and close (“flapping”) indicate a dependency operating right at your failure threshold. This could mean the threshold needs adjustment, or the dependency has an intermittent problem worth investigating independently.

Rejection rate. The ratio of circuit-rejected requests to total attempted. A sustained high rejection rate means real user operations are failing fast rather than slowly. This is the intended protection behavior, but it also means you need a fallback or the dependency needs to be fixed.

Half-open probe success rate. What fraction of probes succeed on first attempt vs. causing the circuit to re-open. A low success rate indicates the recovery window (openDurationMs) is too short for the dependency to fully recover. Tune it up.

Tradeoffs

Concern	With Circuit Breaker	Without Circuit Breaker
Failure detection	Aggregate rate over time window	Per-request, no aggregation
Latency under failure	Near-zero (fast fail on open)	Full timeout per request
Resource exhaustion	Prevented when circuit is open	Thread/connection pools fill
False positives	Possible on bursty traffic	Not applicable
Recovery behavior	Automatic via half-open probing	Automatic, but full traffic immediately
Observability	Explicit state with events	Inferred from error rates
Fallback requirement	Yes: open state needs a response	No
Implementation cost	Moderate	None

The false positive risk deserves attention. A 50% failure rate threshold over 60 seconds will trip on transient spikes: a GC pause that causes a 30-second wave of timeouts, a database connection pool that exhausts briefly under load, a bad deploy that gets rolled back in under two minutes. Whether tripping is the right call depends on context. For idempotent reads, fast-failing for 30 seconds while the probe waits is acceptable. For payment processing, a false positive that blocks transactions during a brief self-recovering blip has real business cost.

Tuning Thresholds in Production

Default values rarely survive first contact with production traffic. Here is a framework for tuning:

Start conservative. A 50% failure threshold with a 60-second window and 30-second open duration is a reasonable starting point. These values are conservative enough that the circuit will not trip on noise but will trip on genuine degradation.

Calibrate to your error baseline. Some dependencies have a small but steady background error rate: 0.5% of calls might fail under perfectly healthy conditions due to network noise, transient timeouts, and edge cases. Set your threshold well above that baseline, or the circuit will trip permanently. Use historical error rate data from your metrics system, not intuition.

Tune the open duration to recovery time. Check your incidents: how long does it typically take for this dependency to recover after an outage? If the answer is “usually 2 to 5 minutes,” set openDurationMs to 60,000 (one minute) so half-open probing begins before full recovery. If the answer is “sometimes 10 minutes,” set it higher. Probing too early extends the outage duration for your users because each failed probe resets the timer.

Separate circuits per operation type. A single circuit for “payments-service” trips when anything in that service fails. If the charge endpoint is failing but refund is healthy, the single circuit blocks refunds unnecessarily. Create per-operation circuits for high-value endpoints where granularity matters.

const paymentsChargeCircuit = new CircuitBreaker({
  name: "payments.charge",
  failureThreshold: 0.3,  // more sensitive: payment failures are high-value
  minimumRequests: 5,
  openDurationMs: 60_000,
});

const paymentsRefundCircuit = new CircuitBreaker({
  name: "payments.refund",
  failureThreshold: 0.5,
  minimumRequests: 10,
  openDurationMs: 30_000,
});

Composing with the Bulkhead Pattern

The bulkhead pattern allocates a bounded pool of concurrent slots to each dependency. Where circuit breakers react to failure rate, bulkheads constrain concurrency. They solve different problems and compose naturally.

Without a bulkhead: a slow dependency accumulates in-flight requests. Even after the circuit opens and stops new requests, the already-in-flight requests hold their connections for the duration of the timeout. Those connections are gone from your pool until the timeouts fire.

With a bulkhead: the slow dependency is capped at a maximum number of concurrent requests. When that cap is hit, new requests fail fast at the bulkhead rather than piling up behind slow in-flight calls. The circuit breaker handles the failure rate signal; the bulkhead limits the concurrency damage.

class Bulkhead {
  private inFlight = 0;

  constructor(private readonly maxConcurrent: number) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.inFlight >= this.maxConcurrent) {
      throw new Error(`Bulkhead capacity exceeded (max: ${this.maxConcurrent})`);
    }
    this.inFlight++;
    try {
      return await fn();
    } finally {
      this.inFlight--;
    }
  }
}

const inventoryBulkhead = new Bulkhead(25);
const inventoryCircuit = new CircuitBreaker({
  name: "inventory-service",
  failureThreshold: 0.4,
  minimumRequests: 8,
});

async function getInventory(sku: string): Promise<InventoryLevel> {
  return inventoryBulkhead.execute(() =>
    inventoryCircuit.execute(
      withTimeout(() => inventoryClient.getLevel(sku), 4_000)
    )
  );
}

The composition order: bulkhead (caps concurrency) wraps circuit breaker (tracks failure rate) wraps timeout (bounds latency) wraps the actual call. Each layer addresses a different failure dimension.

Fallback Design

An open circuit needs to do something. Without a fallback, the circuit breaker converts slow failures into fast failures, which is useful for resource protection but does not improve user experience.

Fallback options, roughly in order of preference:

Cached response. If the last known good response is acceptable for the current request, return it. Works well for data that changes infrequently relative to cache TTL: product catalogs, user preferences, configuration.

Degraded response. Return a reduced version of the response that can be served without the failing dependency. An order service with a down inventory check might allow the order to proceed and verify inventory asynchronously.

Default/static response. Return a predetermined safe response. Works for non-personalized data: a list of featured products rather than personalized recommendations.

Explicit error with clear messaging. If no fallback is viable, fail with a clear error code that the caller can surface meaningfully. HTTP 503 with a Retry-After header is better than a timeout.

async function getProductRecommendations(userId: string): Promise<Product[]> {
  try {
    return await recommendationsCircuit.execute(
      withTimeout(() => recommendationsService.fetch(userId), 3_000)
    );
  } catch (err) {
    if (err instanceof CircuitOpenError || isTimeoutError(err)) {
      const cached = await cache.get<Product[]>(`recommendations:${userId}`);
      return cached ?? getStaticFeaturedProducts();
    }
    throw err;
  }
}

The fallback path should itself be tested under load before it is needed. Most teams validate that the circuit trips correctly but do not load-test the fallback at peak traffic levels.

When Circuit Breakers Are the Wrong Tool

Circuit breakers add value in specific scenarios. Several scenarios where they add complexity without proportional benefit:

Low throughput, high-value operations. If a circuit covers an endpoint that receives 5 requests per minute, you may never accumulate enough requests to reach minimumRequests before conditions change. The circuit breaker provides no protection because it never has enough data to make a decision.

Operations with no viable fallback. If the only response to an open circuit is an error, you have converted a slow error into a fast error. That reduces resource pressure, which is good, but does not change user experience. Sometimes that is worth it; sometimes simpler timeout tuning achieves the same result with less complexity.

Dependencies your team fully controls and can redeploy in minutes. For internal services where fast deploys are your resilience strategy, circuit breakers add operational overhead (tuning, observability, fallback logic) that may not pay off compared to just fixing the service quickly.

Write paths with partial failure risk. A circuit that trips midway through a multi-step write sequence leaves data partially applied. If step 1 writes to service A and step 2 writes to service B via a circuit-protected call, and the circuit opens on step 2, you need compensating logic to undo step 1. This is solvable with saga-style rollback, but the complexity budget must justify the investment.

Production Considerations

Distributed vs. in-process state. The implementation above is per-process. In a fleet of 100 application servers, each instance tracks its own failure window. One server may open its circuit while 99 others still send traffic to the failing dependency. For most scenarios this is acceptable: each server will eventually trip on its own data. For dependencies where coordinated tripping is critical (payments, auth), storing circuit state in a shared store adds coordination but also adds latency on every execute call. Measure before adopting.

Configuration per environment. Thresholds that work in production fail in staging, where traffic is sparse and dependency reliability is lower. Keep minimumRequests, failureThreshold, and openDurationMs in environment-specific configuration rather than hardcoded in the constructor call.

Test your fallbacks under realistic load. The scenario that matters is: circuit opens during peak traffic, thousands of concurrent requests hit the fallback path simultaneously. Test that the fallback itself does not become a bottleneck. A fallback that calls a cache service that then gets overwhelmed has just moved the cascading failure one level up.

Distinguish infrastructure errors from application errors. Not every exception deserves to trip the circuit. HTTP 400 (bad request) is a caller error. HTTP 429 (rate limited) is a sign to back off, not to cut the circuit. TCP connection refused or HTTP 503 are the signals the circuit breaker is designed for. The isFailure callback is the right place to implement this filtering.

The circuit breaker pattern solves a targeted problem: a slow or failing dependency that exhausts your resources before you can respond. The implementation is not complex. The investment is in tuning, instrumentation, and fallback design. A circuit breaker without observability is guesswork. A circuit breaker without a fallback is just a faster error. Get those two right, and the state machine handles the rest.