AI Agent Reliability Engineering: Retry Semantics, Fallback Chains, and Graceful Degradation for Production Agents

Your agent worked perfectly in development. Then you deployed it and discovered that OpenAI returned a 429 at the worst possible moment, a tool call timed out mid-workflow after three successful steps, and the context window filled up on step seven of a ten-step task. The whole run failed. You had no partial result. You had to start over.

Traditional reliability patterns, exponential backoff, circuit breakers, retries, all apply here. But AI agents introduce failure modes that those patterns were not designed for. Non-determinism means a retry does not guarantee the same output. Tool calls may have side effects. Context state is ephemeral and expensive to rebuild. Provider outages are not uniform: one endpoint 503s while another returns 200 with degraded quality. A naive retry loop will waste tokens, trigger duplicate side effects, and still fail.

This article covers how to reason about AI agent reliability specifically: which failures are safe to retry, how to build a provider fallback chain without duplicating side effects, how to apply circuit breakers at the tool level, and how to degrade gracefully rather than fail entirely.

---

Why AI Agents Fail Differently

A stateless HTTP handler that fails can be retried unconditionally. The inputs are the same, the outputs will be equivalent, and no external state changes between attempts.

Agents do not work this way. An agent mid-workflow has accumulated state: prior tool call results, intermediate reasoning, memory writes, external API calls that already executed. A naive retry means:

• Token waste: You re-send the full conversation history and re-run reasoning that already completed.
• Duplicate side effects: If step three was “send a Slack message,” retrying from step one sends that message again.
• Non-deterministic divergence: The LLM response on retry may differ, invalidating assumptions made in steps that already ran.
• Context window pressure: Each retry attempt that re-includes prior messages pushes you closer to the limit.

The core mental model shift: retry at the step level, not the agent level. Identify the smallest retriable unit, enforce idempotency for any action with side effects, and build your fallback logic around that granularity.

---

Retry Semantics for LLM Calls

Not all LLM calls are equal from a retry standpoint.

Safe to retry unconditionally:

• Pure completion calls where the output is only used for reasoning (no external side effects)
• Embedding generation
• Classification or extraction over a fixed input

Retry with idempotency key enforcement:

• Tool calls that write data (database inserts, API mutations)
• Actions that trigger external workflows (webhooks, job queue submissions)
• Any call where the downstream system is not itself idempotent

Do not retry, compensate instead:

• Calls that already partially succeeded and have observable state (e.g., a payment initiated but not confirmed)
• Tool calls where the side effect is irreversible (email sent, physical action triggered)

The practical implementation is a step-level executor that tracks completion state and skips already-finished steps on retry:

type StepStatus = "pending" | "running" | "completed" | "failed";

interface AgentStep {
  id: string;
  name: string;
  idempotencyKey: string;
  status: StepStatus;
  result?: unknown;
  error?: string;
  completedAt?: Date;
}

interface AgentWorkflow {
  workflowId: string;
  steps: AgentStep[];
  startedAt: Date;
  timeoutMs: number;
}

When a step completes, persist that state before moving to the next step. On retry, skip any step with status === "completed" and use the previously stored result. This is checkpoint-based execution, and it is the foundation of reliable multi-step agents.

---

Provider Fallback Chains

Provider outages are not rare. OpenAI, Anthropic, and Google all have incident histories. The practical response is a fallback chain: try your primary provider, and if it fails (or is degraded), route to a secondary.

The key constraint is that the fallback must be triggered before a side effect occurs. Once a tool call fires, you cannot simply retry with a different provider and pretend the first attempt did not happen.

Here is a resilient agent executor with a three-level fallback chain:

import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";

interface ProviderConfig {
  name: string;
  maxTokens: number;
  timeoutMs: number;
  priority: number;
}

interface LLMRequest {
  messages: Array<{ role: string; content: string }>;
  systemPrompt: string;
  maxTokens?: number;
}

interface LLMResponse {
  content: string;
  provider: string;
  tokensUsed: number;
}

const PROVIDER_CONFIGS: ProviderConfig[] = [
  { name: "openai-gpt4o", maxTokens: 4096, timeoutMs: 30_000, priority: 1 },
  { name: "anthropic-claude", maxTokens: 4096, timeoutMs: 30_000, priority: 2 },
  { name: "openai-gpt4o-mini", maxTokens: 4096, timeoutMs: 20_000, priority: 3 },
];

class ProviderFallbackChain {
  private openai: OpenAI;
  private anthropic: Anthropic;
  private circuitBreakers: Map<string, CircuitBreaker>;

  constructor() {
    this.openai = new OpenAI();
    this.anthropic = new Anthropic();
    this.circuitBreakers = new Map(
      PROVIDER_CONFIGS.map((p) => [p.name, new CircuitBreaker(p.name)])
    );
  }

  async complete(request: LLMRequest): Promise<LLMResponse> {
    const sortedProviders = PROVIDER_CONFIGS.sort((a, b) => a.priority - b.priority);

    for (const provider of sortedProviders) {
      const breaker = this.circuitBreakers.get(provider.name)!;

      if (breaker.isOpen()) {
        console.warn(`[fallback] Circuit open for ${provider.name}, skipping`);
        continue;
      }

      try {
        const result = await this.callWithTimeout(provider, request);
        breaker.recordSuccess();
        return result;
      } catch (err) {
        const error = err as Error;
        breaker.recordFailure();

        // Rate limit: respect retry-after before trying next provider
        if (isRateLimitError(error)) {
          console.warn(`[fallback] Rate limited on ${provider.name}, trying next`);
          continue;
        }

        // Timeout or 5xx: try next provider immediately
        if (isRetriableError(error)) {
          console.warn(`[fallback] Retriable error on ${provider.name}: ${error.message}`);
          continue;
        }

        // Auth or invalid request: do not retry on any provider
        throw error;
      }
    }

    throw new Error("All providers exhausted in fallback chain");
  }

  private async callWithTimeout(
    provider: ProviderConfig,
    request: LLMRequest
  ): Promise<LLMResponse> {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), provider.timeoutMs);

    try {
      if (provider.name.startsWith("openai")) {
        return await this.callOpenAI(provider, request, controller.signal);
      } else {
        return await this.callAnthropic(provider, request, controller.signal);
      }
    } finally {
      clearTimeout(timer);
    }
  }

  private async callOpenAI(
    provider: ProviderConfig,
    request: LLMRequest,
    signal: AbortSignal
  ): Promise<LLMResponse> {
    const model = provider.name === "openai-gpt4o" ? "gpt-4o" : "gpt-4o-mini";

    const response = await this.openai.chat.completions.create(
      {
        model,
        messages: [
          { role: "system", content: request.systemPrompt },
          ...request.messages.map((m) => ({
            role: m.role as "user" | "assistant",
            content: m.content,
          })),
        ],
        max_tokens: request.maxTokens ?? provider.maxTokens,
      },
      { signal }
    );

    return {
      content: response.choices[0].message.content ?? "",
      provider: provider.name,
      tokensUsed: response.usage?.total_tokens ?? 0,
    };
  }

  private async callAnthropic(
    provider: ProviderConfig,
    request: LLMRequest,
    signal: AbortSignal
  ): Promise<LLMResponse> {
    const response = await this.anthropic.messages.create(
      {
        model: "claude-3-5-sonnet-20241022",
        system: request.systemPrompt,
        messages: request.messages.map((m) => ({
          role: m.role as "user" | "assistant",
          content: m.content,
        })),
        max_tokens: request.maxTokens ?? provider.maxTokens,
      },
      { signal }
    );

    const content = response.content[0];
    return {
      content: content.type === "text" ? content.text : "",
      provider: provider.name,
      tokensUsed: response.usage.input_tokens + response.usage.output_tokens,
    };
  }
}

function isRateLimitError(err: Error): boolean {
  return err.message.includes("429") || err.message.toLowerCase().includes("rate limit");
}

function isRetriableError(err: Error): boolean {
  return (
    err.name === "AbortError" ||
    err.message.includes("502") ||
    err.message.includes("503") ||
    err.message.includes("504")
  );
}

The fallback chain iterates providers in priority order, skips open circuit breakers, and propagates only non-retriable errors (auth failures, malformed requests). Rate limit errors skip to the next provider immediately rather than sleeping.

---

Circuit Breakers for Agent Tool Calls

Circuit breakers are standard for service-to-service calls, but they matter more for agent tools because tool failures are not just latency: they consume tokens and context space. An agent that keeps calling a broken tool will fill its context window with error messages and eventually produce nonsense.

type CircuitState = "closed" | "open" | "half-open";

interface CircuitBreakerConfig {
  failureThreshold: number;   // failures before opening
  successThreshold: number;   // successes in half-open before closing
  timeoutMs: number;          // how long to stay open
}

class CircuitBreaker {
  private state: CircuitState = "closed";
  private failureCount = 0;
  private successCount = 0;
  private lastFailureTime?: number;
  private readonly config: CircuitBreakerConfig;
  private readonly name: string;

  constructor(name: string, config: Partial<CircuitBreakerConfig> = {}) {
    this.name = name;
    this.config = {
      failureThreshold: config.failureThreshold ?? 3,
      successThreshold: config.successThreshold ?? 2,
      timeoutMs: config.timeoutMs ?? 60_000,
    };
  }

  isOpen(): boolean {
    if (this.state === "open") {
      const elapsed = Date.now() - (this.lastFailureTime ?? 0);
      if (elapsed > this.config.timeoutMs) {
        this.state = "half-open";
        this.successCount = 0;
        return false;
      }
      return true;
    }
    return false;
  }

  recordSuccess(): void {
    if (this.state === "half-open") {
      this.successCount++;
      if (this.successCount >= this.config.successThreshold) {
        this.state = "closed";
        this.failureCount = 0;
      }
    } else {
      this.failureCount = 0;
    }
  }

  recordFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();

    if (this.failureCount >= this.config.failureThreshold) {
      this.state = "open";
      console.error(`[circuit-breaker] ${this.name} opened after ${this.failureCount} failures`);
    }
  }

  getState(): CircuitState {
    return this.state;
  }
}

Apply a circuit breaker per tool, not per agent. If your searchWeb tool is failing, open that circuit and let the agent degrade to working without search results. The agent keeps running. Only the capability is removed.

---

Graceful Degradation Patterns

“Fail gracefully” usually means “return an error response.” For agents, it should mean “reduce capability and continue.” The distinction matters because agents are often doing useful work even when some tools are unavailable.

Three degradation patterns worth naming:

Capability reduction: Remove unavailable tools from the agent’s context. If the web search circuit is open, omit the search tool definition from the next LLM call. The agent will reason with the tools it has. The output will be less informed but still valid.

Result substitution: When a tool fails, inject a synthetic result into the conversation that accurately describes the failure. “The database query timed out. Reasoning from what I know without current data.” The agent can then produce a lower-confidence answer rather than aborting.

Scope narrowing: When context pressure or timeouts threaten the full workflow, reduce the scope of what the agent is trying to accomplish. Instead of “analyze all 50 documents,” shift to “analyze the 10 most recent documents and flag that the full analysis could not complete.”

interface AgentCapabilities {
  tools: ToolDefinition[];
  contextBudgetTokens: number;
  timeRemainingMs: number;
}

interface ToolDefinition {
  name: string;
  description: string;
  schema: Record<string, unknown>;
}

function computeDegradedCapabilities(
  allTools: ToolDefinition[],
  circuitBreakers: Map<string, CircuitBreaker>,
  workflowStartedAt: number,
  totalBudgetMs: number
): AgentCapabilities {
  const elapsed = Date.now() - workflowStartedAt;
  const timeRemainingMs = totalBudgetMs - elapsed;

  // Remove tools with open circuit breakers
  const availableTools = allTools.filter((tool) => {
    const breaker = circuitBreakers.get(tool.name);
    return !breaker || !breaker.isOpen();
  });

  // Reduce context budget as time pressure increases
  const timePressureRatio = elapsed / totalBudgetMs;
  const contextBudgetTokens = timePressureRatio > 0.8
    ? 2000   // late in the workflow: save context for final answer
    : 4000;

  return { tools: availableTools, contextBudgetTokens, timeRemainingMs };
}

The key insight here: degrade before the agent breaks, not after. Compute available capabilities at each step and pass reduced context to the LLM call. Do not wait for a timeout to discover you should have narrowed scope five steps ago.

---

Timeout Budgets for Multi-Step Workflows

A single LLM call has a timeout. A multi-step workflow needs a total budget with per-step allocation.

The failure mode without a budget: step one takes 25 seconds, step two takes 20 seconds, and you have no time left for step seven, which is the only step that produces the output the user actually needs. The whole workflow times out with nothing useful to show.

With a budget:

interface TimeoutBudget {
  totalMs: number;
  perStepMs: number;
  bufferMs: number;  // reserved for final output generation
  startedAt: number;
}

function createBudget(totalMs: number, stepCount: number): TimeoutBudget {
  const bufferMs = Math.min(totalMs * 0.2, 10_000);
  const workingMs = totalMs - bufferMs;
  const perStepMs = Math.floor(workingMs / stepCount);

  return { totalMs, perStepMs, bufferMs, startedAt: Date.now() };
}

function getRemainingBudget(budget: TimeoutBudget): number {
  return budget.totalMs - (Date.now() - budget.startedAt);
}

function shouldAbortEarly(budget: TimeoutBudget): boolean {
  const remaining = getRemainingBudget(budget);
  return remaining <= budget.bufferMs;
}

async function executeWorkflow(
  steps: Array<() => Promise<unknown>>,
  totalBudgetMs: number
): Promise<{ results: unknown[]; skippedSteps: number }> {
  const budget = createBudget(totalBudgetMs, steps.length);
  const results: unknown[] = [];
  let skippedSteps = 0;

  for (let i = 0; i < steps.length; i++) {
    if (shouldAbortEarly(budget)) {
      console.warn(`[budget] Aborting after step ${i}, insufficient time for remaining ${steps.length - i} steps`);
      skippedSteps = steps.length - i;
      break;
    }

    const stepTimeout = Math.min(budget.perStepMs, getRemainingBudget(budget) - budget.bufferMs);

    try {
      const result = await Promise.race([
        steps[i](),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error(`Step ${i} timed out after ${stepTimeout}ms`)), stepTimeout)
        ),
      ]);
      results.push(result);
    } catch (err) {
      results.push({ error: (err as Error).message });
    }
  }

  return { results, skippedSteps };
}

Reserve a buffer at the end. If step N is the step that writes the final answer, you need enough time to actually generate it. Without the buffer, you discover at the last moment that you have 200ms left for a call that needs 5 seconds.

---

Tradeoffs

Dimension	Aggressive Retry	Conservative Retry	Graceful Degrade
Token cost	High (re-runs full context)	Medium (step-level checkpoints)	Low (skips failed steps)
Output quality	Potentially highest (full retry)	Same as aggressive on success	Lower (partial results)
Side effect risk	High (no idempotency)	Low (keyed idempotency)	Low (degraded capability)
Implementation complexity	Low	Medium	High
Best for	Pure read-only workflows	Write workflows with idempotent tools	Long workflows with partial value

The choice is not binary. Most production agents use all three: retry at the step level with idempotency keys, run the provider fallback chain on LLM failures, and degrade capability when tools are unavailable.

---

Production Considerations

Observability. Instrument every step with: step name, provider used, tokens consumed, duration, and whether the result came from a fallback or retry. Without this data, you cannot tell whether your fallback chain is helping or whether you should adjust circuit breaker thresholds.

Idempotency key design. The key should encode the workflow run ID and the step index, not just a random UUID. This way you can safely retry a step even after a crash, because the same key is generated deterministically. Use ${workflowId}:${stepIndex}:${stepName} as the key format.

Context window accounting. Track token consumption per step. If consumption is growing faster than expected (because error messages are accumulating), truncate the conversation history before the next LLM call rather than letting the context fill and receiving a truncation error from the provider.

Provider capability differences. Not all fallback providers are equivalent. GPT-4o and Claude have different context window limits, tool calling formats, and response characteristics. Your fallback chain may need to adapt the request format, not just swap the API client.

Circuit breaker state persistence. In-memory circuit breakers reset on restart. For agent workflows that run over hours, use a shared store (Redis or a simple database table) to persist breaker state. Otherwise a restart resets all open circuits and immediately floods recovered providers.

Dead letter handling. When a workflow exhausts all retries and all fallbacks, save the partial state to a dead letter store with enough context to resume or replay. Do not discard the work. A human or a scheduled retry process can pick it up.

---

Reliable AI agents are not about making LLMs more deterministic. They are about building the execution layer to tolerate the non-determinism. The patterns here, checkpoint-based step execution, provider fallback chains, per-tool circuit breakers, and budget-aware degradation, all serve the same goal: produce partial, honest results rather than complete failures. An agent that delivers 80% of the answer reliably is far more useful than one that delivers 100% of the answer occasionally.