Graceful Degradation for AI Features: Fallback Strategies, Timeout Budgets, and Keeping Your App Alive When LLMs Fail

Your AI-powered search feature went live three months ago and users love it. Relevance is measurably better than keyword search. Engagement is up. The product team is building the roadmap around it.

Then OpenAI returns 503s for 45 minutes on a Tuesday afternoon. Your search page breaks. The whole product looks broken, even though 90% of its functionality is unrelated to the LLM call. Your incident channel lights up.

This is not bad luck. It is a predictable consequence of embedding an external probabilistic service into a synchronous user-facing flow without designing for failure.

The question is not whether your LLM provider will have an outage. It is whether your application will survive it gracefully or fail visibly.

---

The Mental Model: AI Features Are Optional Enhancements

The framing shift that makes graceful degradation tractable: AI features are enhancement layers, not core application functionality. The product existed before the model. It can deliver reduced value without it.

This means every AI feature needs two things:

1. A non-AI baseline that works without the model (keyword search, static recommendations, manual classification)
2. A defined quality gap between the AI path and the baseline, so you know what you are giving up when you fall back

Without the baseline, you have built an AI-first feature that cannot degrade. You are one provider outage away from a P1 incident.

With the baseline, degradation becomes a product decision: users get less accurate results for a few minutes, not a broken page.

---

The Fallback Chain

A fallback chain is an ordered sequence of increasingly lower-cost, lower-quality alternatives that the application tries when a higher-quality option fails.

A typical chain for an AI-enhanced search feature looks like this:

1. Primary: Full LLM semantic search with query expansion and relevance reranking
2. Cached response: Serve a cached result if the same query was run recently
3. Simpler model: Route to a smaller, faster, self-hosted model (less accurate, still semantic)
4. Rule-based fallback: Keyword BM25 search with no AI involvement
5. Static fallback: Return popular or trending results

Each level trades quality for reliability. The chain should be ordered so that each level is strictly more available than the one above it.

Here is the core abstraction:

interface FallbackResult<T> {
  data: T;
  level: "primary" | "cached" | "simple-model" | "rule-based" | "static";
  degraded: boolean;
  latencyMs: number;
}

interface FallbackChain<TInput, TOutput> {
  execute(input: TInput): Promise<FallbackResult<TOutput>>;
}

class SearchFallbackChain implements FallbackChain<SearchQuery, SearchResult[]> {
  constructor(
    private readonly semanticSearch: SemanticSearchService,
    private readonly responseCache: ResponseCache,
    private readonly simpleModel: SimpleModelService,
    private readonly keywordSearch: KeywordSearchService,
    private readonly staticResults: StaticResultsService,
  ) {}

  async execute(query: SearchQuery): Promise<FallbackResult<SearchResult[]>> {
    const start = Date.now();

    // Level 1: full semantic search
    const primaryResult = await this.tryWithTimeout(
      () => this.semanticSearch.search(query),
      800, // tight budget for user-facing search
    );
    if (primaryResult.ok) {
      return {
        data: primaryResult.value,
        level: "primary",
        degraded: false,
        latencyMs: Date.now() - start,
      };
    }

    // Level 2: cached response
    const cached = await this.responseCache.get(query.text);
    if (cached) {
      return {
        data: cached,
        level: "cached",
        degraded: true,
        latencyMs: Date.now() - start,
      };
    }

    // Level 3: simpler self-hosted model
    const simpleResult = await this.tryWithTimeout(
      () => this.simpleModel.search(query),
      600,
    );
    if (simpleResult.ok) {
      return {
        data: simpleResult.value,
        level: "simple-model",
        degraded: true,
        latencyMs: Date.now() - start,
      };
    }

    // Level 4: keyword fallback
    const keywordResult = await this.keywordSearch.search(query.text);
    return {
      data: keywordResult,
      level: "rule-based",
      degraded: true,
      latencyMs: Date.now() - start,
    };
  }

  private async tryWithTimeout<T>(
    fn: () => Promise<T>,
    timeoutMs: number,
  ): Promise<{ ok: true; value: T } | { ok: false; error: Error }> {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), timeoutMs);
    try {
      const value = await fn();
      clearTimeout(timer);
      return { ok: true, value };
    } catch (error) {
      clearTimeout(timer);
      return { ok: false, error: error as Error };
    }
  }
}

The degraded flag matters. You will use it to emit metrics, display a subtle UI indicator (“showing approximate results”), and track how often each fallback level is hit.

---

Timeout Budgets for Multi-Step Pipelines

Single-call timeout budgets are straightforward. Multi-step agent pipelines are not.

When an agent has five steps, each with its own LLM call and tool executions, a naive approach assigns a timeout to each step independently. The problem: a user-facing request has a total latency budget, not per-step budgets. Three steps that each take 800ms will exceed a 2-second total budget even if each one “succeeded.”

The correct model is a deadline propagated through the pipeline: a single timestamp representing when the total operation must complete, decremented by each step as it runs.

interface PipelineContext {
  requestId: string;
  deadlineMs: number; // absolute timestamp
  remainingBudgetMs: () => number;
}

function createContext(totalBudgetMs: number): PipelineContext {
  const deadline = Date.now() + totalBudgetMs;
  return {
    requestId: crypto.randomUUID(),
    deadlineMs: deadline,
    remainingBudgetMs: () => Math.max(0, deadline - Date.now()),
  };
}

interface PipelineStep<TInput, TOutput> {
  name: string;
  execute(input: TInput, ctx: PipelineContext): Promise<TOutput>;
  fallback(input: TInput, ctx: PipelineContext): Promise<TOutput>;
  required: boolean; // if false, skip on budget exhaustion
}

async function runPipeline<T>(
  steps: PipelineStep<T, T>[],
  input: T,
  ctx: PipelineContext,
): Promise<{ result: T; completedSteps: string[]; skippedSteps: string[] }> {
  let current = input;
  const completedSteps: string[] = [];
  const skippedSteps: string[] = [];

  for (const step of steps) {
    const remaining = ctx.remainingBudgetMs();

    if (remaining <= 0) {
      if (step.required) {
        throw new Error(`Budget exhausted before required step: ${step.name}`);
      }
      skippedSteps.push(step.name);
      continue;
    }

    // Reserve a minimum slice for this step; skip optional steps if budget is low
    const MIN_STEP_BUDGET_MS = 200;
    if (remaining < MIN_STEP_BUDGET_MS && !step.required) {
      skippedSteps.push(step.name);
      continue;
    }

    try {
      current = await withDeadline(
        () => step.execute(current, ctx),
        remaining - 50, // leave 50ms for overhead
      );
      completedSteps.push(step.name);
    } catch {
      // Step failed or timed out — try fallback
      try {
        current = await step.fallback(current, ctx);
        completedSteps.push(`${step.name}:fallback`);
      } catch {
        if (step.required) throw new Error(`Required step failed: ${step.name}`);
        skippedSteps.push(step.name);
      }
    }
  }

  return { result: current, completedSteps, skippedSteps };
}

async function withDeadline<T>(fn: () => Promise<T>, timeoutMs: number): Promise<T> {
  return Promise.race([
    fn(),
    new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error("Step deadline exceeded")), timeoutMs),
    ),
  ]);
}

The key design decision is the required flag. Mark a step required only if the output is meaningless without it. An AI-powered reranking step is not required; the items are still useful without reranking. A personalization step that filters illegal content might be required.

When budget runs out, optional steps are skipped and the result returns with a partial annotation. The pipeline completes; it just completes with less enrichment.

---

Feature-Level Circuit Breakers

Circuit breakers for LLM providers protect against provider-level outages. But a provider being up does not mean your feature is healthy. A model might be returning malformed output. Hallucination rates might have spiked. Latency might be within the provider’s SLA but outside your feature’s budget.

Feature-level circuit breakers trip on signals specific to what you are measuring, not just on HTTP error codes.

type CircuitState = "closed" | "open" | "half-open";

interface FeatureCircuitBreaker {
  state: CircuitState;
  record(outcome: "success" | "failure" | "timeout" | "quality-failure"): void;
  isOpen(): boolean;
}

class SlidingWindowCircuitBreaker implements FeatureCircuitBreaker {
  private window: Array<{ ts: number; outcome: string }> = [];
  private _state: CircuitState = "closed";
  private openedAt: number | null = null;

  constructor(
    private readonly windowMs: number = 60_000,
    private readonly failureThreshold: number = 0.4, // 40% failure rate opens circuit
    private readonly minSamples: number = 10,
    private readonly halfOpenAfterMs: number = 30_000,
  ) {}

  get state(): CircuitState {
    // Transition from open to half-open after cooldown
    if (this._state === "open" && this.openedAt) {
      if (Date.now() - this.openedAt > this.halfOpenAfterMs) {
        this._state = "half-open";
      }
    }
    return this._state;
  }

  record(outcome: "success" | "failure" | "timeout" | "quality-failure"): void {
    const now = Date.now();
    this.window.push({ ts: now, outcome });
    // Evict entries outside the window
    this.window = this.window.filter((e) => now - e.ts < this.windowMs);

    if (this._state === "half-open") {
      if (outcome === "success") {
        this._state = "closed";
        this.openedAt = null;
      } else {
        this._state = "open";
        this.openedAt = now;
      }
      return;
    }

    if (this.window.length >= this.minSamples) {
      const failures = this.window.filter((e) => e.outcome !== "success").length;
      const rate = failures / this.window.length;
      if (rate >= this.failureThreshold && this._state === "closed") {
        this._state = "open";
        this.openedAt = now;
      }
    }
  }

  isOpen(): boolean {
    return this.state === "open";
  }
}

The important addition here is "quality-failure" as a first-class outcome. An HTTP 200 with a response that fails your quality checks is not a success. Recording it as a failure causes the circuit breaker to open when quality degrades, not just when the provider goes down.

The typical usage in a feature:

class AISearchFeature {
  private breaker = new SlidingWindowCircuitBreaker();

  async search(query: string): Promise<SearchResult[]> {
    if (this.breaker.isOpen()) {
      // Circuit is open: skip the AI call, go straight to fallback
      return this.fallbackSearch(query);
    }

    try {
      const result = await this.llmSearch(query);
      const quality = await this.qualityCheck(result, query);

      if (!quality.acceptable) {
        this.breaker.record("quality-failure");
        return this.fallbackSearch(query);
      }

      this.breaker.record("success");
      return result;
    } catch (err) {
      const outcome = isTimeout(err) ? "timeout" : "failure";
      this.breaker.record(outcome);
      return this.fallbackSearch(query);
    }
  }
}

---

Quality Degradation Detection

A model returning 200 responses with subtly worse quality is harder to detect than a 503. The latency looks fine. Error rates look fine. But users are getting worse answers.

Quality degradation detection requires defining what “good” looks like for your specific use case and measuring it continuously.

Three practical checks that run fast enough to be in-path:

1. Schema and format validation

The cheapest check. If your prompt asks for JSON with specific fields, validate the schema before returning the response. A surprising number of quality regressions show up as format drift.

import { z } from "zod";

const SearchResponseSchema = z.object({
  results: z.array(
    z.object({
      id: z.string(),
      relevanceScore: z.number().min(0).max(1),
      explanation: z.string().min(10),
    }),
  ),
  queryIntent: z.enum(["informational", "navigational", "transactional"]),
});

async function validateLLMResponse(raw: string): Promise<boolean> {
  try {
    const parsed = JSON.parse(raw);
    SearchResponseSchema.parse(parsed);
    return true;
  } catch {
    return false;
  }
}

2. Semantic coherence check

For responses that include natural language, a lightweight coherence check catches obvious regressions: empty explanations, repeated phrases, responses that ignore the query.

interface CoherenceCheck {
  minResponseLength: number;
  maxRepetitionRatio: number; // ratio of repeated n-grams
  queryTermCoverage: number; // fraction of query terms that appear in response
}

function checkCoherence(response: string, query: string, config: CoherenceCheck): boolean {
  if (response.length < config.minResponseLength) return false;

  // Basic repetition detection: split into trigrams, count duplicates
  const words = response.toLowerCase().split(/\s+/);
  if (words.length < 6) return false;

  const trigrams = words.slice(0, -2).map((w, i) => `${w} ${words[i + 1]} ${words[i + 2]}`);
  const unique = new Set(trigrams);
  const repetitionRatio = 1 - unique.size / trigrams.length;
  if (repetitionRatio > config.maxRepetitionRatio) return false;

  // Query term coverage
  const queryTerms = query.toLowerCase().split(/\s+/).filter((t) => t.length > 3);
  if (queryTerms.length === 0) return true;
  const covered = queryTerms.filter((term) => response.toLowerCase().includes(term)).length;
  return covered / queryTerms.length >= config.queryTermCoverage;
}

3. Latency-as-quality-signal

For some use cases, extreme latency is itself a quality failure. A 30-second response is not useful even if it is technically correct. Track p95 latency per model and feature combination, and emit a quality-failure signal when latency exceeds your feature’s acceptable bound.

class LatencyQualityMonitor {
  private samples: number[] = [];

  record(latencyMs: number): void {
    this.samples.push(latencyMs);
    // Keep a rolling window of 100 samples
    if (this.samples.length > 100) this.samples.shift();
  }

  p95(): number {
    if (this.samples.length === 0) return 0;
    const sorted = [...this.samples].sort((a, b) => a - b);
    const idx = Math.floor(sorted.length * 0.95);
    return sorted[idx];
  }

  isWithinBudget(budgetMs: number): boolean {
    return this.p95() <= budgetMs;
  }
}

---

Tradeoffs

Approach	Latency overhead	Complexity	Recovery speed	Risk
Static fallback only	Near zero	Low	Instant	Stays degraded until manual fix
Fallback chain	Low (cached hit) to moderate (model hop)	Medium	Automatic per request	Chain depth can hide systematic failures
Feature circuit breaker	Negligible when closed; zero when open	Medium	30-60s half-open probe	False positives during transient spikes
Quality detection in-path	5-15ms per check	Medium	Per-response	Adds latency; check must be fast
Deadline propagation	Negligible	Low-Medium	Automatic	Requires careful budget allocation

---

Production Considerations

Instrument every fallback transition. Each fallback level should emit a metric with the feature name, fallback level, and reason. Without this, you will not know whether your primary path is failing 1% or 40% of the time.

function recordFallback(feature: string, level: string, reason: string): void {
  metrics.increment("ai_feature.fallback", {
    feature,
    level,
    reason,
  });
}

Cache aggressively for the degraded path. A cached LLM response from five minutes ago is almost always better than a keyword result. Use semantic hashing to cache on query intent, not exact query string, so “show me laptops under $500” and “laptops cheaper than 500 dollars” share a cache entry.

Expose degradation state to the UI selectively. Some features warrant a visible “showing approximate results” badge. Others should silently degrade: a slightly less personalized recommendation feed does not need a user-facing banner. The product team should own this decision, but the engineering contract is clear: every fallback level has a degraded: boolean in its response type.

Test your fallback chain regularly. Chaos-test each fallback level in staging by injecting failures at the primary path. If your keyword fallback returns stale data or your cached fallback has a broken eviction policy, you want to find that in a drill, not during an actual outage.

Set a minimum viable response time for each level. The fallback chain should not be allowed to compound latency. If primary takes 800ms and fails, and the cached miss takes 50ms, and the keyword fallback takes 200ms, your total degraded path should still be under 1.5 seconds. If it is not, re-examine what the fallback chain is actually doing.

---

When LLM providers have outages, the apps that survive them are the ones built with the assumption that the AI call will fail at some point. The fallback chain is not a contingency plan; it is a first-class design requirement. The circuit breaker is not defensive engineering; it is what prevents a provider degradation from turning into a user-facing incident.

Design for failure first. The AI path is the happy path, not the only path.