Multi-Provider LLM Reliability: Failover Strategies, Provider Abstraction, and Inference SLAs for Production AI

A single LLM provider is a single point of failure. OpenAI goes down for 40 minutes and your AI feature returns 503s the whole time. Anthropic throttles your org-level rate limit during a traffic spike and inference latency climbs from 800ms to 12 seconds. A model gets deprecated with 30 days notice and you scramble to swap every call site.

These are not edge cases. They have all happened in production systems. The fix is treating LLM providers the same way you treat any external dependency: with abstraction, health tracking, circuit breakers, and defined SLAs.

This article covers the full reliability stack for multi-provider LLM systems. Not the happy path: the failure modes, the routing decisions, and the operational contracts that let you answer “is our AI feature healthy right now?” with data instead of guesswork.

The Provider Abstraction Layer

Every provider has its own SDK, request shape, error codes, and streaming protocol. Before you can route intelligently, you need a uniform interface that hides those differences.

type Role = "system" | "user" | "assistant";

interface LLMMessage {
  role: Role;
  content: string;
}

interface InferenceRequest {
  messages: LLMMessage[];
  maxTokens: number;
  temperature?: number;
  stream?: boolean;
  metadata: {
    requestId: string;
    feature: string;
    budgetCents?: number; // max spend for this call
  };
}

interface InferenceResponse {
  content: string;
  model: string;
  provider: string;
  usage: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
    estimatedCostCents: number;
  };
  latencyMs: number;
}

interface LLMProvider {
  id: string;
  complete(req: InferenceRequest): Promise<InferenceResponse>;
  stream(req: InferenceRequest): AsyncIterable<string>;
  isHealthy(): boolean;
}

Each provider implements LLMProvider. The OpenAI adapter translates to the OpenAI SDK. The Anthropic adapter translates to the Anthropic SDK. An Ollama adapter wraps a self-hosted endpoint. Your application code never imports a provider SDK directly: it imports the abstraction.

class OpenAIProvider implements LLMProvider {
  id = "openai-gpt4o";
  private client: OpenAI;
  private healthy = true;

  constructor(private readonly model: string) {
    this.client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
  }

  async complete(req: InferenceRequest): Promise<InferenceResponse> {
    const start = Date.now();
    const response = await this.client.chat.completions.create({
      model: this.model,
      messages: req.messages,
      max_tokens: req.maxTokens,
      temperature: req.temperature ?? 0.7,
    });

    const choice = response.choices[0];
    const usage = response.usage!;

    return {
      content: choice.message.content ?? "",
      model: this.model,
      provider: "openai",
      usage: {
        promptTokens: usage.prompt_tokens,
        completionTokens: usage.completion_tokens,
        totalTokens: usage.total_tokens,
        estimatedCostCents: this.estimateCost(usage.prompt_tokens, usage.completion_tokens),
      },
      latencyMs: Date.now() - start,
    };
  }

  isHealthy(): boolean {
    return this.healthy;
  }

  private estimateCost(prompt: number, completion: number): number {
    // GPT-4o pricing at time of writing: $2.50/1M input, $10/1M output
    return (prompt / 1_000_000) * 250 + (completion / 1_000_000) * 1000;
  }

  async *stream(req: InferenceRequest): AsyncIterable<string> {
    const stream = await this.client.chat.completions.create({
      model: this.model,
      messages: req.messages,
      max_tokens: req.maxTokens,
      stream: true,
    });
    for await (const chunk of stream) {
      const delta = chunk.choices[0]?.delta?.content;
      if (delta) yield delta;
    }
  }
}

The Anthropic adapter follows the same pattern, translating messages into Anthropic’s message format, mapping max_tokens to max_tokens_to_sample, and converting error shapes. The implementation detail stays inside the adapter. The router never sees it.

Circuit Breakers for Inference Endpoints

A circuit breaker tracks failure rates on a sliding window and opens when the rate crosses a threshold. While open, calls route to a fallback instead of hammering a degraded endpoint. After a configurable timeout, the breaker enters half-open state and probes the endpoint with a single request. Success closes it; failure resets the timer.

type CircuitState = "closed" | "open" | "half-open";

interface CircuitBreakerConfig {
  failureThreshold: number;    // failures before opening (e.g., 5)
  successThreshold: number;    // successes to close from half-open (e.g., 2)
  windowMs: number;            // rolling window duration (e.g., 60_000)
  resetTimeoutMs: number;      // time before half-open probe (e.g., 30_000)
}

class CircuitBreaker {
  private state: CircuitState = "closed";
  private failures: number[] = []; // timestamps
  private successes = 0;
  private openedAt?: number;

  constructor(
    private readonly providerId: string,
    private readonly config: CircuitBreakerConfig,
  ) {}

  isOpen(): boolean {
    if (this.state === "closed") return false;

    if (this.state === "open") {
      const elapsed = Date.now() - (this.openedAt ?? 0);
      if (elapsed >= this.config.resetTimeoutMs) {
        this.state = "half-open";
        this.successes = 0;
        return false; // allow one probe
      }
      return true;
    }

    // half-open: let one through
    return false;
  }

  recordSuccess(): void {
    if (this.state === "half-open") {
      this.successes++;
      if (this.successes >= this.config.successThreshold) {
        this.state = "closed";
        this.failures = [];
      }
    } else {
      this.failures = this.failures.filter(
        (t) => Date.now() - t < this.config.windowMs,
      );
    }
  }

  recordFailure(): void {
    const now = Date.now();
    this.failures = this.failures
      .filter((t) => now - t < this.config.windowMs)
      .concat(now);

    if (
      this.state !== "open" &&
      this.failures.length >= this.config.failureThreshold
    ) {
      this.state = "open";
      this.openedAt = now;
      console.warn(`[circuit-breaker] ${this.providerId} opened`);
    }

    if (this.state === "half-open") {
      this.state = "open";
      this.openedAt = now;
    }
  }

  getState(): CircuitState {
    return this.state;
  }
}

Each provider gets its own breaker instance. The router checks isOpen() before routing to a candidate. This prevents the latency cascades that happen when a degraded provider responds slowly instead of failing fast: a breaker forces the fast failure that lets the router move on.

One subtlety: differentiate between failures that should trip the breaker and those that should not. A 400 from OpenAI (invalid request) is your bug, not a provider outage. A 429 (rate limit) is different from a 500 (server error). Only 5xx errors and timeout failures should increment the failure counter.

function isRetryableError(error: unknown): boolean {
  if (error instanceof APIError) {
    return error.status >= 500 || error.status === 429;
  }
  if (error instanceof Error && error.message.includes("timeout")) {
    return true;
  }
  return false;
}

Latency-Based Routing

Not all requests need the same provider. Route latency-sensitive paths (streaming chat, autocomplete) to the fastest available provider. Route quality-sensitive paths (document analysis, complex reasoning) to the best available provider, accepting higher latency.

Measure real p50 and p95 latency per provider on a rolling window rather than relying on provider marketing figures. A provider that advertises 500ms TTFT may deliver 2.5 seconds at your actual concurrency and token volumes.

interface LatencyStats {
  p50Ms: number;
  p95Ms: number;
  sampleCount: number;
  updatedAt: number;
}

class LatencyTracker {
  private windows = new Map<string, number[]>();

  record(providerId: string, latencyMs: number): void {
    if (!this.windows.has(providerId)) {
      this.windows.set(providerId, []);
    }
    const window = this.windows.get(providerId)!;
    window.push(latencyMs);
    // keep last 200 samples
    if (window.length > 200) window.shift();
  }

  getStats(providerId: string): LatencyStats | null {
    const window = this.windows.get(providerId);
    if (!window || window.length < 5) return null;

    const sorted = [...window].sort((a, b) => a - b);
    return {
      p50Ms: sorted[Math.floor(sorted.length * 0.5)],
      p95Ms: sorted[Math.floor(sorted.length * 0.95)],
      sampleCount: sorted.length,
      updatedAt: Date.now(),
    };
  }
}

With per-provider latency data, routing decisions become data-driven. For a streaming feature with a 1500ms TTFT SLA, skip any provider whose p95 exceeds that threshold. For a batch job with a 30-second budget, almost any provider qualifies.

type RoutingStrategy = "fastest" | "cheapest" | "most-capable" | "balanced";

function selectProvider(
  candidates: LLMProvider[],
  breakers: Map<string, CircuitBreaker>,
  tracker: LatencyTracker,
  strategy: RoutingStrategy,
  latencySlaMs?: number,
): LLMProvider | null {
  const available = candidates.filter(
    (p) => !breakers.get(p.id)?.isOpen(),
  );

  if (available.length === 0) return null;

  if (strategy === "fastest") {
    const ranked = available
      .map((p) => ({ provider: p, stats: tracker.getStats(p.id) }))
      .filter(({ stats }) => {
        if (!stats) return true; // no data: include optimistically
        if (latencySlaMs && stats.p95Ms > latencySlaMs) return false;
        return true;
      })
      .sort((a, b) => {
        const aP50 = a.stats?.p50Ms ?? Infinity;
        const bP50 = b.stats?.p50Ms ?? Infinity;
        return aP50 - bP50;
      });
    return ranked[0]?.provider ?? null;
  }

  // "cheapest" ranks by cost-per-token, "balanced" weights latency + cost
  // implementation follows same pattern
  return available[0];
}

Cost-Aware Model Selection

Token pricing varies by factor of 10-100x across models. GPT-4o Mini costs roughly 40x less per token than GPT-4o. Haiku costs roughly 20x less than Claude Opus. For workloads where response quality is measured (classification, structured extraction, simple Q&A), you can run cheaper models by default and escalate only when confidence is low.

Enforce per-request cost budgets at the router level to prevent runaway spend from prompt injection attacks or prompt design mistakes that cause verbose completions.

interface CostConfig {
  defaultBudgetCents: number;     // per-request ceiling
  monthlyBudgetCents: number;     // org-level monthly cap
  modelTiers: ModelTier[];
}

interface ModelTier {
  name: string;
  provider: LLMProvider;
  costPer1KInputCents: number;
  costPer1KOutputCents: number;
  maxContextTokens: number;
  capability: "fast" | "balanced" | "high-quality";
}

function estimateRequestCost(
  tier: ModelTier,
  promptTokens: number,
  maxOutputTokens: number,
): number {
  return (
    (promptTokens / 1000) * tier.costPer1KInputCents +
    (maxOutputTokens / 1000) * tier.costPer1KOutputCents
  );
}

function selectCheapestViableTier(
  tiers: ModelTier[],
  promptTokens: number,
  maxOutputTokens: number,
  budgetCents: number,
  minimumCapability: ModelTier["capability"],
): ModelTier | null {
  const capabilityRank: Record<ModelTier["capability"], number> = {
    fast: 0,
    balanced: 1,
    "high-quality": 2,
  };
  const minRank = capabilityRank[minimumCapability];

  const viable = tiers
    .filter((t) => capabilityRank[t.capability] >= minRank)
    .filter((t) => estimateRequestCost(t, promptTokens, maxOutputTokens) <= budgetCents)
    .sort(
      (a, b) =>
        estimateRequestCost(a, promptTokens, maxOutputTokens) -
        estimateRequestCost(b, promptTokens, maxOutputTokens),
    );

  return viable[0] ?? null;
}

The capability field represents your own classification of each model: not marketing copy. You determine it by running your eval suite against each model and recording pass rates on your tasks. A “fast” tier that scores 94% on your classification benchmark may be preferable to a “high-quality” tier at 40x the price for that specific workload.

Token Budget Enforcement Across Providers

Counting tokens before sending a request requires a tokenizer that matches the target model. OpenAI’s tiktoken counts accurately for GPT models. Anthropic provides its own tokenizer. For cost estimation and budget checks, a close approximation (character count divided by 3.5) works for pre-flight checks; run exact counts for billing.

import Anthropic from "@anthropic-ai/sdk";

async function estimateAnthropicTokens(
  client: Anthropic,
  messages: LLMMessage[],
  model: string,
): Promise<number> {
  const result = await client.messages.countTokens({
    model,
    messages: messages.map((m) => ({ role: m.role, content: m.content })),
  });
  return result.input_tokens;
}

function enforceTokenBudget(
  promptTokens: number,
  requestedMaxOutput: number,
  contextLimit: number,
  budgetCents: number,
  tier: ModelTier,
): number {
  // hard cap: can't exceed model context window
  const hardMaxOutput = contextLimit - promptTokens - 50; // 50 token buffer

  // soft cap: limit output to what fits in cost budget after input cost
  const inputCost = (promptTokens / 1000) * tier.costPer1KInputCents;
  const remainingBudget = budgetCents - inputCost;
  const affordableOutput = Math.floor(
    (remainingBudget / tier.costPer1KOutputCents) * 1000,
  );

  return Math.min(requestedMaxOutput, hardMaxOutput, affordableOutput);
}

Pass the adjusted maxTokens to the provider. If the budget math yields a negative number (prompt alone exceeds budget), reject the request before it leaves your system. This surfaces misconfigured prompts early rather than generating surprise invoices.

Defining and Monitoring Inference SLAs

An inference SLA has at minimum three dimensions: availability (what percentage of requests succeed), latency (at what percentile does response time fall within what threshold), and correctness (what percentage of responses pass your quality checks). Most teams only track the first two.

Define SLAs per feature, not globally. A chat autocomplete feature may have a 1200ms p95 TTFT SLA and a 99.5% availability target. A background document summarization job may have a 30-second end-to-end SLA and a 99% availability target. Global SLAs hide the features that are silently missing their targets.

interface InferenceSLA {
  featureId: string;
  availabilityTargetPct: number;   // e.g., 99.5
  latencyP95TargetMs: number;      // e.g., 1200
  latencyP99TargetMs: number;      // e.g., 3000
  windowHours: number;             // rolling evaluation window
}

interface SLAStatus {
  featureId: string;
  currentAvailabilityPct: number;
  currentLatencyP95Ms: number;
  currentLatencyP99Ms: number;
  breaching: boolean;
  evaluatedAt: number;
}

class SLAMonitor {
  private requests = new Map<
    string,
    Array<{ success: boolean; latencyMs: number; timestamp: number }>
  >();

  record(
    featureId: string,
    success: boolean,
    latencyMs: number,
  ): void {
    if (!this.requests.has(featureId)) {
      this.requests.set(featureId, []);
    }
    this.requests.get(featureId)!.push({
      success,
      latencyMs,
      timestamp: Date.now(),
    });
  }

  evaluate(sla: InferenceSLA): SLAStatus {
    const windowMs = sla.windowHours * 60 * 60 * 1000;
    const cutoff = Date.now() - windowMs;
    const samples = (this.requests.get(sla.featureId) ?? []).filter(
      (r) => r.timestamp >= cutoff,
    );

    if (samples.length === 0) {
      return {
        featureId: sla.featureId,
        currentAvailabilityPct: 100,
        currentLatencyP95Ms: 0,
        currentLatencyP99Ms: 0,
        breaching: false,
        evaluatedAt: Date.now(),
      };
    }

    const successCount = samples.filter((r) => r.success).length;
    const availabilityPct = (successCount / samples.length) * 100;

    const sorted = samples.map((r) => r.latencyMs).sort((a, b) => a - b);
    const p95Ms = sorted[Math.floor(sorted.length * 0.95)];
    const p99Ms = sorted[Math.floor(sorted.length * 0.99)];

    const breaching =
      availabilityPct < sla.availabilityTargetPct ||
      p95Ms > sla.latencyP95TargetMs ||
      p99Ms > sla.latencyP99TargetMs;

    return {
      featureId: sla.featureId,
      currentAvailabilityPct: availabilityPct,
      currentLatencyP95Ms: p95Ms,
      currentLatencyP99Ms: p99Ms,
      breaching,
      evaluatedAt: Date.now(),
    };
  }
}

Expose SLA status as a health endpoint your alerting system polls. When breaching flips to true, page on-call. Include the specific dimension that is failing (availability vs latency) so the responder knows where to look.

For availability calculations, be careful about what counts as a failure. A 429 that you handle by routing to a fallback provider and completing successfully is not an availability miss. A request that exhausts all fallbacks and returns an error to the caller is. Track “application-level success rate” separately from “provider-level success rate.” The gap between the two tells you how much your failover machinery is doing.

Tradeoffs

Decision	Tradeoff
Failover between providers	Different models produce different outputs. If feature A expects GPT-4o-style JSON and falls back to Claude Haiku, output format may silently drift. Test your prompts against every provider in your failover chain.
Latency-based routing	Historical p95 lags real-time conditions. During a provider brownout, latency climbs before errors accumulate. Add a latency spike detector that opens the circuit faster when p50 crosses 3x its rolling baseline.
Cost-aware selection	Cheap models fail in unpredictable ways on novel inputs. Running a cheaper model by default works until a real-world input falls outside its capability envelope. Invest in evals before downgrading.
Circuit breaker thresholds	Too sensitive and you route away from healthy providers on transient errors. Too loose and you hammer degraded endpoints. Start with 5 failures in 60 seconds; tune from your actual incident data.
Open-source model fallback	Self-hosted models remove provider dependency but add infrastructure dependency. A crashed GPU node is operationally different from an OpenAI outage but equally impactful. Apply the same circuit breaker and health tracking.
Token counting overhead	Calling a tokenizer API before every request adds 20-50ms. Use character-based approximations for pre-flight budget checks; save exact counts for billing reconciliation.

Production Considerations

Keep provider credentials in a secrets manager (AWS Secrets Manager, GCP Secret Manager, or Vault) and rotate them without redeployment. The abstraction layer is the natural place to reload credentials on a timer without requiring a restart.

Log every inference call with provider, model, feature, latency, token counts, and estimated cost. A structured log line costs almost nothing and makes debugging SLA breaches and billing anomalies straightforward. Without this data you will not know which feature caused a cost spike or which provider is degrading your p95.

For fallback chains, define explicit priority order per feature, not a global default. A financial document analysis feature might prefer Claude over GPT-4o because your evals showed better instruction following for that task. Encoding this at the feature level makes the routing logic auditable.

Rate limit tracking deserves its own mention. Most providers report remaining rate limit headers in every response. Capture these and use them to pre-emptively route away from a provider approaching its limit rather than waiting for 429 errors to trip the circuit breaker. A provider at 5% of its rate limit capacity is a degraded provider, not a healthy one.

Closing Insight

Provider abstraction is not over-engineering. It is the foundation that lets you swap models when a provider deprecates one, shift traffic when pricing changes, and keep your feature available when a provider has an outage. The circuit breakers, latency tracking, and SLA monitoring built on top of that foundation are what turn “we have multiple providers” into “we have a reliable AI feature.” Building this layer is straightforward. The cost of not building it shows up at 3am when one provider has an incident and you have no fallback.