Building AI Agents That Actually Work: Orchestration Patterns for Production

The demo looks simple: call an LLM, parse the output, call a tool, repeat. You can ship that in a weekend. Then you take it to production and the first real user triggers a cascade of failures you never thought to test for. The tool call times out. The LLM returns malformed JSON. The context window fills up after three turns. The cost per session is 40x what you estimated.

AI agent orchestration is not a solved problem, and most of the writing about it is written by people who haven’t run agents in production. This article documents the patterns that actually hold up.

The Naive Loop and Why It Breaks

Every agent tutorial starts with this:

async function runAgent(userMessage: string): Promise<string> {
  const messages: Message[] = [{ role: "user", content: userMessage }];

  while (true) {
    const response = await llm.complete({ messages, tools });

    if (response.finishReason === "stop") {
      return response.content;
    }

    if (response.finishReason === "tool_calls") {
      for (const call of response.toolCalls) {
        const result = await executeTool(call.name, call.arguments);
        messages.push({ role: "tool", toolCallId: call.id, content: result });
      }
    }
  }
}

This breaks in production for several reasons:

• No loop bound. A hallucinating model can spin forever.
• Sequential tool execution. If the model calls three tools, you wait for each one to finish before starting the next.
• No timeout handling. A slow tool call blocks the entire agent.
• No cost tracking. Nothing stops a long context from being passed repeatedly.
• No error recovery. A tool failure ends the session, with no context about why.

Every production agent system eventually rebuilds the same set of guardrails around this loop. The question is whether you design them in upfront or bolt them on after the first incident.

Single-Agent Loops vs Multi-Agent Orchestration

Before adding complexity, decide whether you actually need multiple agents.

A single-agent loop with proper guardrails handles most tasks that look like multi-agent problems. Multi-agent systems are appropriate when:

• Tasks have genuinely independent subtasks that can be parallelized.
• You need specialized behavior that requires different system prompts or models.
• A single context window cannot hold all the relevant state.
• You want fault isolation: a subtask failure should not abort the entire workflow.

The two main topologies are supervisor and peer.

Supervisor Topology

One orchestrator agent decomposes the task and delegates to specialized worker agents. Workers report back; the supervisor synthesizes and decides what to do next.

interface AgentResult {
  agentId: string;
  output: string;
  tokensUsed: number;
  durationMs: number;
  error?: string;
}

class SupervisorAgent {
  private workers: Map<string, WorkerAgent>;

  async run(task: string): Promise<string> {
    const plan = await this.plan(task);

    const results = await Promise.allSettled(
      plan.steps.map((step) => this.delegate(step))
    );

    return this.synthesize(task, results);
  }

  private async delegate(step: PlanStep): Promise<AgentResult> {
    const worker = this.workers.get(step.agentType);
    if (!worker) throw new Error(`Unknown agent type: ${step.agentType}`);
    return worker.execute(step.input, { timeoutMs: step.timeoutMs ?? 30_000 });
  }

  private async synthesize(
    task: string,
    results: PromiseSettledResult<AgentResult>[]
  ): Promise<string> {
    const context = results
      .map((r, i) =>
        r.status === "fulfilled"
          ? `Step ${i + 1}: ${r.value.output}`
          : `Step ${i + 1}: FAILED - ${r.reason}`
      )
      .join("\n\n");

    const response = await this.llm.complete({
      messages: [
        { role: "system", content: this.systemPrompt },
        {
          role: "user",
          content: `Original task: ${task}\n\nResults:\n${context}\n\nSynthesize a final response.`,
        },
      ],
    });

    return response.content;
  }
}

Supervisor topology gives you centralized control and clear failure attribution. The tradeoff is a single point of coordination: if the supervisor stalls, everything stalls.

Peer Topology

Agents communicate directly, passing messages through a shared queue or event bus. No single agent is in charge. Each agent picks up work, processes it, and emits events that other agents may consume.

interface AgentMessage {
  id: string;
  fromAgent: string;
  toAgent: string | "broadcast";
  type: string;
  payload: unknown;
  timestamp: number;
}

class MessageBus {
  private handlers: Map<string, ((msg: AgentMessage) => Promise<void>)[]> =
    new Map();

  subscribe(agentId: string, handler: (msg: AgentMessage) => Promise<void>) {
    const existing = this.handlers.get(agentId) ?? [];
    this.handlers.set(agentId, [...existing, handler]);
  }

  async publish(message: AgentMessage) {
    const targets =
      message.toAgent === "broadcast"
        ? Array.from(this.handlers.keys())
        : [message.toAgent];

    await Promise.all(
      targets.flatMap((t) =>
        (this.handlers.get(t) ?? []).map((h) => h(message))
      )
    );
  }
}

Peer topology is more resilient but harder to reason about. Debugging requires distributed tracing because there is no central place where the full execution is visible.

Tool Use with Structured Outputs

Tools are where agents interact with the world, and they are the primary source of production failures.

Define tools with Zod schemas so you get validation at the boundary, not mid-execution:

import { z } from "zod";

const SearchTool = {
  name: "search_documents",
  description: "Search the document index for relevant content.",
  parameters: z.object({
    query: z.string().min(1).max(500),
    limit: z.number().int().min(1).max(20).default(5),
    filters: z
      .object({
        dateAfter: z.string().datetime().optional(),
        category: z.enum(["contract", "policy", "report"]).optional(),
      })
      .optional(),
  }),
};

type SearchParams = z.infer<typeof SearchTool.parameters>;

async function executeTool(
  name: string,
  rawArgs: unknown
): Promise<ToolResult> {
  const tool = toolRegistry.get(name);
  if (!tool) {
    return { error: `Unknown tool: ${name}`, output: null };
  }

  const parsed = tool.parameters.safeParse(rawArgs);
  if (!parsed.success) {
    return {
      error: `Invalid arguments: ${parsed.error.message}`,
      output: null,
    };
  }

  try {
    const output = await tool.handler(parsed.data);
    return { error: null, output };
  } catch (err) {
    return {
      error: err instanceof Error ? err.message : "Tool execution failed",
      output: null,
    };
  }
}

Returning structured errors back to the model instead of throwing exceptions is important. The model can recover from a bad tool call if it receives a clear error message. If you throw, the entire agent run ends.

Parallel Tool Execution

When the model issues multiple tool calls in a single turn, execute them concurrently:

async function executeToolCalls(
  calls: ToolCall[],
  options: { timeoutMs: number }
): Promise<ToolResult[]> {
  const withTimeout = (call: ToolCall) =>
    Promise.race([
      executeTool(call.name, call.arguments),
      new Promise<ToolResult>((resolve) =>
        setTimeout(
          () =>
            resolve({ error: `Tool timed out after ${options.timeoutMs}ms`, output: null }),
          options.timeoutMs
        )
      ),
    ]);

  return Promise.all(calls.map(withTimeout));
}

This turns sequential tool latency into parallel latency. For three independent tools each taking 500ms, you go from 1500ms to ~500ms.

Memory and Context Management

Context windows are finite and expensive. A naive agent that appends every message to a growing array will eventually hit the limit or make costs untenable.

Context Budgeting

Track token usage explicitly and decide what to keep:

interface ContextEntry {
  role: "user" | "assistant" | "tool";
  content: string;
  tokens: number;
  importance: "high" | "normal" | "low";
  timestamp: number;
}

class ContextManager {
  private entries: ContextEntry[] = [];
  private readonly maxTokens: number;

  constructor(maxTokens: number) {
    this.maxTokens = maxTokens;
  }

  add(entry: ContextEntry) {
    this.entries.push(entry);
    this.compact();
  }

  private compact() {
    const totalTokens = this.entries.reduce((sum, e) => sum + e.tokens, 0);
    if (totalTokens <= this.maxTokens) return;

    // Drop low-importance entries first, then normal, keeping high always
    const priorities: ContextEntry["importance"][] = ["low", "normal"];

    for (const priority of priorities) {
      const candidates = this.entries.filter((e) => e.importance === priority);
      for (const candidate of candidates) {
        const remaining = this.entries.reduce((sum, e) => sum + e.tokens, 0);
        if (remaining <= this.maxTokens) break;
        this.entries = this.entries.filter((e) => e !== candidate);
      }
    }
  }

  getMessages(): Message[] {
    return this.entries.map((e) => ({ role: e.role, content: e.content }));
  }
}

High-importance entries include the system prompt, the original user request, and any confirmed facts or decisions. Tool results from earlier in a long session are often low-importance once their information has been incorporated into the assistant’s response.

External Memory

For sessions that genuinely need long-term recall, store facts outside the context window and retrieve them at the start of each turn:

interface MemoryEntry {
  key: string;
  value: string;
  createdAt: number;
  lastAccessedAt: number;
}

class ExternalMemory {
  async recall(query: string, topK: number = 5): Promise<MemoryEntry[]> {
    // Semantic search over stored memories
    const embedding = await embedText(query);
    return this.vectorStore.search(embedding, topK);
  }

  async store(key: string, value: string): Promise<void> {
    const embedding = await embedText(value);
    await this.vectorStore.upsert({ key, value, embedding });
  }
}

This keeps the context window lean while preserving useful state across multi-turn sessions.

Error Handling and Retry Strategies

Agents fail in predictable ways. Design recovery paths for each failure mode.

type RetryConfig = {
  maxAttempts: number;
  backoffMs: number;
  backoffMultiplier: number;
  retryableErrors: string[];
};

async function withRetry<T>(
  fn: () => Promise<T>,
  config: RetryConfig
): Promise<T> {
  let lastError: Error | null = null;
  let delayMs = config.backoffMs;

  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err instanceof Error ? err : new Error(String(err));
      const isRetryable = config.retryableErrors.some((msg) =>
        lastError!.message.includes(msg)
      );

      if (!isRetryable || attempt === config.maxAttempts) {
        throw lastError;
      }

      await new Promise((resolve) => setTimeout(resolve, delayMs));
      delayMs *= config.backoffMultiplier;
    }
  }

  throw lastError!;
}

// Usage
const result = await withRetry(
  () => llm.complete({ messages, tools }),
  {
    maxAttempts: 3,
    backoffMs: 1000,
    backoffMultiplier: 2,
    retryableErrors: ["rate_limit", "overloaded", "timeout"],
  }
);

For model-level failures (malformed JSON output, tool call with invalid args), retry with a corrective message rather than starting over:

async function recoverFromParseError(
  messages: Message[],
  badOutput: string,
  expectedSchema: string
): Promise<Message[]> {
  return [
    ...messages,
    { role: "assistant", content: badOutput },
    {
      role: "user",
      content: `Your previous response could not be parsed. Expected schema: ${expectedSchema}. Please respond again in valid JSON.`,
    },
  ];
}

Tradeoffs Table

Pattern	Latency	Cost	Reliability	Complexity
Single agent, sequential tools	Highest	Lowest	Fragile on tool failure	Low
Single agent, parallel tools	Lower	Low	Better isolation	Medium
Supervisor + workers	Medium	Higher	High (fault isolation)	High
Peer topology (event bus)	Variable	Medium	Resilient, hard to debug	Very high
External memory + recall	Adds recall latency	Lower context cost	Context-independent	Medium

Supervisor topology is the right default for most production agent systems. Peer topology makes sense when you have truly autonomous agents that need to collaborate without a predefined workflow.

Observability

You cannot debug an agent system by reading logs. You need structured traces that capture the full turn-by-turn execution.

interface AgentSpan {
  traceId: string;
  spanId: string;
  parentSpanId?: string;
  agentId: string;
  type: "llm_call" | "tool_call" | "agent_call";
  startedAt: number;
  endedAt?: number;
  input: unknown;
  output?: unknown;
  error?: string;
  metadata: {
    model?: string;
    tokensIn?: number;
    tokensOut?: number;
    costUsd?: number;
    toolName?: string;
  };
}

class AgentTracer {
  private spans: AgentSpan[] = [];

  startSpan(
    params: Omit<AgentSpan, "spanId" | "startedAt">
  ): AgentSpan {
    const span: AgentSpan = {
      ...params,
      spanId: crypto.randomUUID(),
      startedAt: Date.now(),
    };
    this.spans.push(span);
    return span;
  }

  endSpan(span: AgentSpan, result: { output?: unknown; error?: string }) {
    span.endedAt = Date.now();
    span.output = result.output;
    span.error = result.error;
  }

  flush(): AgentSpan[] {
    const completed = [...this.spans];
    this.spans = [];
    return completed;
  }
}

Every LLM call, every tool call, and every agent delegation should produce a span. This gives you a flame graph of any agent run and lets you identify where latency comes from, which tool calls fail most often, and how token usage distributes across steps.

Cost Guardrails

Cost control is not optional once agents run in production. Two mechanisms are essential: per-session budgets and model routing.

class CostGuard {
  private sessionSpendUsd = 0;
  private readonly budgetUsd: number;

  constructor(budgetUsd: number) {
    this.budgetUsd = budgetUsd;
  }

  checkBudget() {
    if (this.sessionSpendUsd >= this.budgetUsd) {
      throw new Error(
        `Session budget exceeded: $${this.sessionSpendUsd.toFixed(4)} of $${this.budgetUsd}`
      );
    }
  }

  recordSpend(tokensIn: number, tokensOut: number, model: string) {
    const cost = calculateCost(tokensIn, tokensOut, model);
    this.sessionSpendUsd += cost;
  }
}

// Route short, simple tasks to cheaper models
function selectModel(
  task: AgentTask,
  remainingBudgetUsd: number
): string {
  if (task.complexity === "low" || remainingBudgetUsd < 0.01) {
    return "small-model";
  }
  if (task.requiresReasoning) {
    return "reasoning-model";
  }
  return "standard-model";
}

Set hard budget limits per session and per day. Log model selection decisions as spans so you can audit whether the routing logic is actually saving money or just routing tasks to models that fail and retry more.

Production Considerations

A few things that only become apparent once you have real traffic:

Loop bounds are non-negotiable. Set a maximum iteration count on every agent loop. A model that gets confused can run indefinitely without one. Ten iterations is a reasonable starting limit; increase it only for tasks you have profiled.

Tool failures should degrade gracefully. If a tool is unavailable, the agent should tell the user what it cannot do rather than silently looping or returning a hallucinated answer. Build explicit fallback paths.

Session state needs to be externalized. In-memory agent state does not survive server restarts. Store the message history and context in a database, keyed by session ID, so you can resume interrupted sessions.

Rate limit handling matters. LLM providers rate-limit by tokens per minute, not just requests per minute. If you are running concurrent agents, a single burst can exhaust your quota. Track token usage across concurrent sessions and add backpressure when approaching limits.

Test with adversarial inputs. Prompt injection through tool outputs is a real attack surface. A tool that returns user-controlled content can be used to hijack the agent’s next action. Sanitize tool outputs or use a separate parsing step before feeding them back to the model.

The Right Level of Complexity

The strongest signal that a system is over-engineered: you added multi-agent coordination and latency went up without reliability improving.

Start with a single-agent loop with parallel tool execution, proper retry logic, a context budget, and observability. That handles the majority of real use cases. Add supervisor coordination when you have a clear need for fault isolation across independent subtasks. Add peer topology only when your workflow genuinely does not have a fixed structure.

Complexity is a liability. The agent that runs reliably in production is the one that does less, not more.