Designing a Multi-Agent System: Task Routing, Shared Memory, and Handoff Protocols for Production AI Applications

Single-agent systems work well until they do not. The first version looks clean: one agent, one system prompt, a handful of tools. Then the requirements grow. You add retrieval, a code execution tool, a CRM lookup, a calendar integration, and a document summarizer. The context window fills with tool definitions before the conversation even starts. Response latency climbs because every call carries the full prompt. The agent starts hallucinating tool selection because the decision space is too wide. The architecture that shipped your demo cannot survive your production load. Multi-agent systems are the answer to this ceiling, but they introduce a different class of problems: routing correctness, memory consistency, clean handoffs between agents, and making failures visible across a distributed chain. This article covers each of those problems with concrete TypeScript implementations.

Why Single-Agent Systems Hit a Ceiling

Three forces push you toward multi-agent architecture:

Context window pressure. GPT-4o has a 128K token context. That sounds large until you factor in a system prompt, retrieval chunks, conversation history, tool definitions, and structured output constraints. In practice, 15-20 tool definitions consume 2,000-4,000 tokens just in schema declarations. An agent with 30+ tools is already spending a significant fraction of its context budget on capability declarations before processing a single user message.

Tool sprawl and decision quality. When an agent must choose from 30 tools, the selection accuracy degrades. The model is doing implicit classification over a large action space at every step. Decomposing that into specialized agents, each with a focused tool set (5-8 tools), shifts the hard classification problem to a dedicated routing layer where it can be solved more precisely.

Latency and parallelism. A single agent processes tasks sequentially. A multi-agent system can fan out parallel sub-tasks. A research request that requires pulling three different data sources, summarizing each, then synthesizing a final answer can run the three retrievals concurrently rather than waiting on each in turn.

Agent Registry and Capability Declaration

The foundation of a multi-agent system is knowing what agents exist and what they can do. An agent registry provides that index. Each registered agent declares its capabilities as a structured schema, not free-form prose, so the router can perform deterministic matching.

// Agent capability types
type CapabilityDomain =
  | "code-generation"
  | "data-retrieval"
  | "document-analysis"
  | "calendar-management"
  | "crm-operations"
  | "web-search"
  | "summarization";

interface AgentCapability {
  domain: CapabilityDomain;
  // Structured examples help the router classify ambiguous intents
  exampleIntents: string[];
  // Estimated latency bucket for scheduler-aware routing
  latencyProfile: "fast" | "medium" | "slow";
}

interface AgentRegistration {
  id: string;
  name: string;
  description: string;
  capabilities: AgentCapability[];
  // Agent endpoint for inter-agent communication
  endpoint: string;
  // Whether this agent can invoke other agents (prevents cycles)
  canOrchestrate: boolean;
  maxConcurrency: number;
}

class AgentRegistry {
  private agents = new Map<string, AgentRegistration>();

  register(agent: AgentRegistration): void {
    if (this.agents.has(agent.id)) {
      throw new Error(`Agent ${agent.id} is already registered`);
    }
    this.agents.set(agent.id, agent);
  }

  findByDomain(domain: CapabilityDomain): AgentRegistration[] {
    return Array.from(this.agents.values()).filter((agent) =>
      agent.capabilities.some((cap) => cap.domain === domain)
    );
  }

  getAll(): AgentRegistration[] {
    return Array.from(this.agents.values());
  }

  get(id: string): AgentRegistration | undefined {
    return this.agents.get(id);
  }
}

The canOrchestrate flag matters. Agents that can spawn sub-tasks must be tracked separately from leaf agents that only execute. This prevents unbounded delegation chains where agent A calls agent B which calls agent A again.

Task Routing: From Intent to Agent Selection

The router is the most critical component. It receives an incoming task, classifies the intent, and selects the best-fit agent. There are two viable approaches: embedding-based similarity matching for fuzzy intent classification, and LLM-based classification for nuanced disambiguation. In production you often want both: fast embedding lookup as a primary path, LLM fallback for low-confidence matches.

import { openai } from "@ai-sdk/openai";
import { generateObject } from "ai";
import { z } from "zod";

interface RoutingDecision {
  agentId: string;
  confidence: number;
  reasoning: string;
}

interface TaskContext {
  taskId: string;
  userMessage: string;
  conversationHistory: { role: "user" | "assistant"; content: string }[];
  // Metadata passed forward for observability
  traceId: string;
}

class TaskRouter {
  constructor(
    private registry: AgentRegistry,
    private embeddingClient: EmbeddingClient
  ) {}

  async route(context: TaskContext): Promise<RoutingDecision> {
    const agents = this.registry.getAll();

    // Build a concise capability map for the classifier
    const capabilityMap = agents.map((agent) => ({
      id: agent.id,
      name: agent.name,
      description: agent.description,
      exampleIntents: agent.capabilities.flatMap((c) => c.exampleIntents),
    }));

    const { object } = await generateObject({
      model: openai("gpt-4o-mini"), // Use a fast, cheap model for routing
      schema: z.object({
        agentId: z.string(),
        confidence: z.number().min(0).max(1),
        reasoning: z.string(),
      }),
      system: `You are a task router. Select the most appropriate agent for the given task.
Available agents: ${JSON.stringify(capabilityMap, null, 2)}

Rules:
- Select exactly one agent ID from the list above
- confidence reflects how clearly the task matches the agent
- If confidence < 0.5, route to the fallback orchestrator`,
      prompt: `Task: ${context.userMessage}
Recent context: ${context.conversationHistory
        .slice(-3)
        .map((m) => `${m.role}: ${m.content}`)
        .join("\n")}`,
    });

    return object;
  }
}

The key design decision here is using a lightweight model (gpt-4o-mini) for routing rather than the full model. Routing is a classification problem, not a reasoning problem. Keeping it cheap and fast means you can afford to re-route mid-conversation without adding noticeable latency.

Shared Memory Architecture

Shared memory in a multi-agent system splits into two distinct tiers with different access patterns, consistency requirements, and storage backends.

Short-term memory holds the active conversation state: the current task context, recent tool outputs, and intermediate results being assembled. It needs to be fast (sub-millisecond reads), mutable, and scoped to a single session. Redis is the natural fit.

Long-term memory holds durable knowledge: user preferences, summarized past interactions, domain-specific facts extracted from previous sessions. It needs vector search for semantic retrieval. PostgreSQL with pgvector or Cloudflare Vectorize handles this tier.

interface ShortTermMemory {
  sessionId: string;
  currentTaskId: string | null;
  activeAgentId: string | null;
  // Accumulated context from the current multi-agent chain
  contextChunks: {
    agentId: string;
    output: string;
    timestamp: number;
  }[];
  // Tool outputs cached to avoid redundant calls
  toolCache: Record<string, { result: unknown; cachedAt: number }>;
}

interface LongTermMemoryEntry {
  id: string;
  userId: string;
  content: string;
  embedding: number[];
  metadata: {
    sourceAgentId: string;
    createdAt: string;
    topic: string;
  };
}

class SharedMemoryStore {
  constructor(
    private redis: RedisClient,
    private vectorStore: VectorStoreClient
  ) {}

  // Short-term: session state
  async getSession(sessionId: string): Promise<ShortTermMemory | null> {
    const raw = await this.redis.get(`session:${sessionId}`);
    return raw ? JSON.parse(raw) : null;
  }

  async updateSession(
    sessionId: string,
    patch: Partial<ShortTermMemory>
  ): Promise<void> {
    const current = (await this.getSession(sessionId)) ?? {
      sessionId,
      currentTaskId: null,
      activeAgentId: null,
      contextChunks: [],
      toolCache: {},
    };
    const updated = { ...current, ...patch };
    // 30-minute session TTL
    await this.redis.set(`session:${sessionId}`, JSON.stringify(updated), {
      ex: 1800,
    });
  }

  async appendContextChunk(
    sessionId: string,
    chunk: ShortTermMemory["contextChunks"][number]
  ): Promise<void> {
    const session = await this.getSession(sessionId);
    if (!session) return;
    // Keep the last 20 chunks to bound memory size
    const chunks = [...session.contextChunks, chunk].slice(-20);
    await this.updateSession(sessionId, { contextChunks: chunks });
  }

  // Long-term: semantic retrieval
  async storeMemory(entry: Omit<LongTermMemoryEntry, "embedding">): Promise<void> {
    const embedding = await this.vectorStore.embed(entry.content);
    await this.vectorStore.insert({ ...entry, embedding });
  }

  async recallRelevant(
    userId: string,
    query: string,
    topK = 5
  ): Promise<LongTermMemoryEntry[]> {
    const queryEmbedding = await this.vectorStore.embed(query);
    return this.vectorStore.search({
      filter: { userId },
      vector: queryEmbedding,
      topK,
    });
  }
}

One important constraint: agents write to their own context chunk namespace and read from the shared session. They do not overwrite other agents’ chunks. The assembler that builds the final prompt from shared memory respects this ownership model. Without it, you get agents silently clobbering intermediate results that other agents in the chain are depending on.

Handoff Protocols

A handoff is the act of one agent transferring control, context, and responsibility to another. Done badly, it means copying a blob of text and hoping the next agent figures out where the first left off. Done well, it is a structured envelope with a clear contract.

interface HandoffEnvelope {
  // Immutable fields
  traceId: string;
  originAgentId: string;
  targetAgentId: string;
  taskId: string;
  issuedAt: string;

  // Task definition
  instruction: string;
  // Structured context, not a raw text dump
  context: {
    summary: string;
    completedSteps: string[];
    pendingSteps: string[];
    relevantData: Record<string, unknown>;
  };

  // What should come back
  expectedOutputSchema: Record<string, unknown>; // JSON Schema
  // Hard deadline for the receiving agent
  deadlineMs: number;
}

interface HandoffResult {
  traceId: string;
  taskId: string;
  agentId: string;
  status: "success" | "partial" | "failed";
  output: unknown;
  durationMs: number;
  // If partial, what remains
  remainingWork?: string;
}

class HandoffCoordinator {
  constructor(
    private registry: AgentRegistry,
    private memory: SharedMemoryStore
  ) {}

  async dispatch(envelope: HandoffEnvelope): Promise<HandoffResult> {
    const target = this.registry.get(envelope.targetAgentId);
    if (!target) {
      return {
        traceId: envelope.traceId,
        taskId: envelope.taskId,
        agentId: envelope.targetAgentId,
        status: "failed",
        output: null,
        durationMs: 0,
      };
    }

    // Record the handoff in shared memory so the trace is complete
    await this.memory.appendContextChunk(envelope.taskId, {
      agentId: "coordinator",
      output: `Handoff: ${envelope.originAgentId} -> ${envelope.targetAgentId}`,
      timestamp: Date.now(),
    });

    const startMs = Date.now();

    try {
      const response = await fetch(target.endpoint, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(envelope),
        signal: AbortSignal.timeout(envelope.deadlineMs),
      });

      if (!response.ok) {
        throw new Error(`Agent returned HTTP ${response.status}`);
      }

      const result: HandoffResult = await response.json();
      result.durationMs = Date.now() - startMs;

      // Record the result for downstream agents in the chain
      await this.memory.appendContextChunk(envelope.taskId, {
        agentId: envelope.targetAgentId,
        output: typeof result.output === "string"
          ? result.output
          : JSON.stringify(result.output),
        timestamp: Date.now(),
      });

      return result;
    } catch (error) {
      return {
        traceId: envelope.traceId,
        taskId: envelope.taskId,
        agentId: envelope.targetAgentId,
        status: "failed",
        output: error instanceof Error ? error.message : "unknown error",
        durationMs: Date.now() - startMs,
      };
    }
  }
}

The expectedOutputSchema field is not just documentation. The receiving agent validates its own output against it before responding. The calling agent validates the received output before using it. This double-validation catches schema drift between independently deployed agents before it causes silent data corruption downstream.

Error Boundaries

In a sequential agent chain, a failure in step three does not have to abort the entire task. Error boundaries define how failures propagate and what recovery paths exist.

type ErrorSeverity = "recoverable" | "degraded" | "fatal";

interface AgentError {
  agentId: string;
  taskId: string;
  traceId: string;
  severity: ErrorSeverity;
  code: string;
  message: string;
  // Whether the task can continue without this agent's output
  canSkip: boolean;
  // Fallback strategy if canSkip is true
  fallback?: "use-cached" | "use-partial" | "skip-step";
}

class ErrorBoundary {
  async handle(
    error: AgentError,
    memory: SharedMemoryStore
  ): Promise<"continue" | "abort"> {
    // Log the error into the trace
    await memory.appendContextChunk(error.taskId, {
      agentId: "error-boundary",
      output: `ERROR [${error.severity}] agent=${error.agentId} code=${error.code}: ${error.message}`,
      timestamp: Date.now(),
    });

    if (error.severity === "fatal") {
      return "abort";
    }

    if (error.severity === "recoverable" && error.canSkip) {
      // Apply fallback strategy and continue
      if (error.fallback === "use-cached") {
        const session = await memory.getSession(error.taskId);
        const cached = session?.toolCache[error.agentId];
        if (cached && Date.now() - cached.cachedAt < 300_000) {
          // 5-minute cache window
          return "continue";
        }
      }
      if (error.fallback === "skip-step") {
        return "continue";
      }
    }

    // degraded: surface a partial result rather than a hard failure
    if (error.severity === "degraded") {
      return "continue";
    }

    return "abort";
  }
}

The canSkip model forces explicit decisions at registration time about which agents are on the critical path. A summarization agent producing a “nice to have” summary can be skipped. A billing agent computing a charge cannot. Making this explicit in the registry prevents silent failures from producing subtly wrong outputs that go unnoticed.

Observability: Tracing Across Agent Boundaries

A request that crosses four agents is effectively a distributed system call chain. You need distributed tracing, not just per-agent logging.

import { trace, SpanStatusCode, context, propagation } from "@opentelemetry/api";

const tracer = trace.getTracer("multi-agent-system");

interface AgentSpanAttributes {
  "agent.id": string;
  "agent.name": string;
  "task.id": string;
  "handoff.from": string;
  "handoff.to"?: string;
  "token.input": number;
  "token.output": number;
  "model.name": string;
}

async function withAgentSpan<T>(
  name: string,
  attributes: AgentSpanAttributes,
  fn: () => Promise<T>
): Promise<T> {
  return tracer.startActiveSpan(name, async (span) => {
    span.setAttributes(attributes);
    try {
      const result = await fn();
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "unknown",
      });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

// Each agent wraps its execution in a span and propagates the trace context
// through the HandoffEnvelope so downstream agents can attach to the same trace
async function executeAgent(
  envelope: HandoffEnvelope,
  traceContext: Record<string, string>
): Promise<HandoffResult> {
  // Restore the trace context from the calling agent
  const activeContext = propagation.extract(context.active(), traceContext);

  return context.with(activeContext, () =>
    withAgentSpan(
      `agent.execute:${envelope.targetAgentId}`,
      {
        "agent.id": envelope.targetAgentId,
        "agent.name": envelope.targetAgentId,
        "task.id": envelope.taskId,
        "handoff.from": envelope.originAgentId,
        "token.input": 0, // Filled after model call
        "token.output": 0,
        "model.name": "gpt-4o",
      },
      async () => {
        // ... agent execution logic
        return { traceId: envelope.traceId, taskId: envelope.taskId, agentId: envelope.targetAgentId, status: "success", output: null, durationMs: 0 };
      }
    )
  );
}

The critical discipline here is propagating the trace context through the HandoffEnvelope as a carrier map. Without this, each agent creates an isolated trace that cannot be joined. With it, your APM dashboard (Datadog, Honeycomb, Grafana Tempo) shows the entire request as a single waterfall spanning all agents, with token counts, latency, and errors visible per hop.

Tradeoffs: Orchestrator vs. Mesh Topology

Multi-agent systems fall into two broad topologies, and the choice matters for how you handle routing, failure, and observability.

Dimension	Orchestrator-Based	Mesh (Peer-to-Peer)
Routing authority	Central orchestrator classifies and assigns every task	Each agent decides which peer to call next
Failure blast radius	Orchestrator failure halts all routing; single point of failure	No single point of failure; partial failures degrade gracefully
Observability	Trace originates from one place; easy to reconstruct call chain	Traces must be propagated across peer calls; requires strict context propagation
Cycle prevention	Orchestrator enforces directed graph; no cycles possible	Requires explicit cycle detection or max-hop counters per agent
Adding new agents	Register with orchestrator; no other agents change	Each agent that might delegate must be updated to know about the new peer
Latency	One extra hop through orchestrator for every task	Direct peer calls can be faster for known delegation paths
Complexity surface	Orchestrator logic is the complexity bottleneck	Complexity is distributed; harder to reason about emergent routing behavior
Best fit	Systems with dynamic or unpredictable task types	Systems with well-defined, stable delegation paths between specific agent pairs

For most production systems, start with an orchestrator topology. The observability and cycle-prevention properties are worth the single-point-of-failure risk, which you mitigate with redundant orchestrator replicas behind a load balancer. Move toward mesh only when you have stable, high-throughput delegation paths where the orchestrator becomes a measurable latency bottleneck.

Production Considerations

Rate limiting per agent. Each agent should enforce its own concurrency limit and expose a backpressure signal. The orchestrator reads this before dispatching: if an agent is at capacity, it either queues the task or routes to a fallback.

Agent versioning. Agents deploy independently. Version identifiers in the registry allow the router to pin a task chain to a specific agent version for consistency across a long-running session. A user mid-conversation should not get agent v1 responses for the first three turns and v2 responses for the last two.

Memory eviction. Short-term session memory has a TTL. Long-term memory accumulates indefinitely without a pruning strategy. Define a relevance decay function: entries that have not been retrieved in 90 days or whose embedding similarity to recent queries falls below a threshold become candidates for archival or deletion.

Cost accounting. In a four-agent chain, token costs compound. Instrument each agent with input and output token counts attached to the trace span. Aggregate per-session and per-task-type. You will find that the routing agent, used for every request, consumes a disproportionate share of cost. Moving it to a smaller, fine-tuned model for your specific domain is often the highest-ROI optimization available.

The architecture patterns here are not novel in isolation. What makes them production-ready is the combination: a typed registry that makes capabilities machine-readable, a structured handoff envelope with schema validation on both sides, a memory store that separates session state from durable knowledge, and trace context propagation that makes the entire chain visible as a unit. Get those four things right and the rest is tunable.