Designing an Agent Gateway: Centralized Routing, Lifecycle Management, and Fleet Observability for Production AI Agent Systems

Ten agents running in production at 95% individual reliability gives you roughly 60% system reliability. That math is just compound probability: 0.95 to the power of 10. If each agent in a task chain has a one-in-twenty chance of failing, a ten-step pipeline succeeds less than two-thirds of the time. Add framework upgrades that silently shift performance baselines by 30-40% with no visibility, and no single agent owner who can explain what changed, and you have the production reality most teams are shipping into right now.

Gartner named this problem “Agent Sprawl” in April 2026. Ninety-four percent of organizations surveyed said they are concerned about it. Only 12% have any centralized management approach. By end of 2026, Gartner expects 40% of enterprise applications to embed task-specific agents, meaning the median organization will be running 50 or more agents by Q4.

The fix is an agent gateway: a centralized infrastructure layer that sits in front of every agent in your fleet and provides unified routing, lifecycle management, permission enforcement, and observability. This is not a framework. It is not a multi-agent orchestrator. It is the infrastructure equivalent of an API gateway, but for agents instead of HTTP endpoints.

This article covers how to design one.

The Problem with Ad Hoc Agent Management

The pattern most teams follow is straightforward and wrong. A team needs an agent. They build it, wire it to a queue or an HTTP endpoint, and deploy it. Then another team needs an agent for a different task. Same pattern, different code, no shared infrastructure. Within a quarter, you have a dozen agents with different auth schemes, no standardized health checks, no token budget enforcement, and no way to answer “which agent made this LLM call?” when your billing alert fires at 2 AM.

Three named failure patterns emerge reliably from this setup.

The first is Framework-Invisible Complexity. When you upgrade LangChain, LlamaIndex, or any other orchestration framework, the prompts, tool-call formats, and context assembly can change in ways that shift output distributions without throwing errors. The agent still runs. Success rate drops 30-40%. Nobody notices until a downstream metric shows regression, and by then the framework version is already baked into four other agents.

The second is Multi-Model SLO Orphaning. The primary agent uses a well-monitored model with an error budget. The secondary and tertiary models in the pipeline have no named owner, no baseline, and no error budget. When one degrades, there is no alert to fire.

The third is LLM Tech Debt: deprecated model chains that remain in production because no inventory exists to find them. The model gets soft-deprecated, the provider starts returning slightly worse outputs, and the failure is invisible until it is not.

A gateway does not eliminate these problems, but it makes them visible, attributable, and enforceable.

Core Architecture

The agent gateway has four primary responsibilities: agent registration and discovery, request routing and dispatch, permission boundary enforcement, and observability. A fifth responsibility, lifecycle management, cuts across all four.

interface AgentRegistration {
  agentId: string;
  version: string;
  capabilities: AgentCapability[];
  requiredScopes: AgentScope[];
  tokenBudget: TokenBudget;
  healthEndpoint: string;
  owner: string;
  status: "active" | "deprecated" | "draining" | "offline";
  registeredAt: Date;
  deprecatedAt?: Date;
  sunsetAt?: Date;
}

interface AgentCapability {
  name: string;
  inputSchema: Record<string, unknown>;
  outputSchema: Record<string, unknown>;
  estimatedTokensPerCall: number;
  maxLatencyMs: number;
}

interface TokenBudget {
  dailyLimit: number;
  perCallLimit: number;
  burstLimit: number;
  alertThreshold: number; // 0.0 to 1.0, e.g. 0.8 for 80% warning
}

type AgentScope =
  | "read:user-data"
  | "write:user-data"
  | "read:documents"
  | "write:documents"
  | "call:external-api"
  | "call:database"
  | "send:email"
  | "read:billing";

The registry is the source of truth for every agent in the fleet. No agent runs without a registration. This is the first invariant the gateway enforces.

Agent Registry and Discovery

The registry stores agent metadata and exposes a lookup interface the gateway uses at dispatch time. Use a relational store for the registry: agent records have foreign keys to team ownership, and you want transactional guarantees on status transitions.

class AgentRegistry {
  private db: Database;
  private cache: Cache;

  async register(registration: AgentRegistration): Promise<void> {
    await this.db.transaction(async (tx) => {
      const existing = await tx.agents.findUnique({
        where: { agentId: registration.agentId },
      });

      if (existing && existing.status === "active") {
        // New version registration triggers drain on old version
        await tx.agents.update({
          where: { agentId: existing.agentId, version: existing.version },
          data: { status: "draining", drainStartedAt: new Date() },
        });
      }

      await tx.agents.upsert({
        where: { agentId: registration.agentId, version: registration.version },
        create: registration,
        update: { ...registration, registeredAt: new Date() },
      });
    });

    await this.cache.del(`agent:${registration.agentId}`);
  }

  async resolve(agentId: string): Promise<AgentRegistration | null> {
    const cached = await this.cache.get<AgentRegistration>(`agent:${agentId}`);
    if (cached) return cached;

    const agent = await this.db.agents.findFirst({
      where: { agentId, status: "active" },
      orderBy: { registeredAt: "desc" },
    });

    if (agent) {
      await this.cache.set(`agent:${agentId}`, agent, { ttl: 60 });
    }

    return agent;
  }

  async listByCapability(capabilityName: string): Promise<AgentRegistration[]> {
    return this.db.agents.findMany({
      where: {
        status: "active",
        capabilities: { some: { name: capabilityName } },
      },
    });
  }
}

The cache TTL is intentionally short. Sixty seconds means a status change (deprecation, drain signal) propagates to all gateway instances within a minute. Longer TTLs extend the window during which the gateway might route to a deprecated or draining agent.

Request Routing and Dispatch

The gateway receives routing requests and resolves them to agent instances. Routing logic should be a pure function of the request context and the registry state.

interface AgentRequest {
  requestId: string;
  capability: string;
  input: unknown;
  callerContext: CallerContext;
  priority: "low" | "normal" | "high";
  timeoutMs: number;
}

interface CallerContext {
  tenantId: string;
  userId?: string;
  serviceId?: string;
  traceId: string;
  spanId: string;
}

interface DispatchResult {
  requestId: string;
  agentId: string;
  agentVersion: string;
  output: unknown;
  tokensUsed: number;
  latencyMs: number;
  status: "success" | "agent-error" | "timeout" | "budget-exceeded" | "permission-denied";
}

class AgentGateway {
  constructor(
    private registry: AgentRegistry,
    private permissionEnforcer: PermissionEnforcer,
    private budgetEnforcer: BudgetEnforcer,
    private dispatcher: AgentDispatcher,
    private telemetry: GatewayTelemetry,
  ) {}

  async dispatch(request: AgentRequest): Promise<DispatchResult> {
    const startMs = Date.now();

    // Step 1: resolve agent from registry
    const candidates = await this.registry.listByCapability(request.capability);
    if (candidates.length === 0) {
      throw new Error(`No active agent found for capability: ${request.capability}`);
    }

    const agent = this.selectAgent(candidates, request);

    // Step 2: enforce permissions before any execution
    const permissionResult = await this.permissionEnforcer.check({
      agentId: agent.agentId,
      requiredScopes: agent.requiredScopes,
      callerContext: request.callerContext,
    });

    if (!permissionResult.allowed) {
      const result: DispatchResult = {
        requestId: request.requestId,
        agentId: agent.agentId,
        agentVersion: agent.version,
        output: null,
        tokensUsed: 0,
        latencyMs: Date.now() - startMs,
        status: "permission-denied",
      };
      await this.telemetry.record(result, request);
      return result;
    }

    // Step 3: enforce token budget before execution
    const budgetCheck = await this.budgetEnforcer.precheck({
      agentId: agent.agentId,
      tenantId: request.callerContext.tenantId,
      estimatedTokens: agent.capabilities.find((c) => c.name === request.capability)
        ?.estimatedTokensPerCall ?? 0,
      budget: agent.tokenBudget,
    });

    if (!budgetCheck.allowed) {
      const result: DispatchResult = {
        requestId: request.requestId,
        agentId: agent.agentId,
        agentVersion: agent.version,
        output: null,
        tokensUsed: 0,
        latencyMs: Date.now() - startMs,
        status: "budget-exceeded",
      };
      await this.telemetry.record(result, request);
      return result;
    }

    // Step 4: dispatch
    const dispatchResult = await this.dispatcher.call(agent, request);
    const result: DispatchResult = {
      ...dispatchResult,
      latencyMs: Date.now() - startMs,
    };

    // Step 5: record actual token usage
    await this.budgetEnforcer.record({
      agentId: agent.agentId,
      tenantId: request.callerContext.tenantId,
      tokensUsed: result.tokensUsed,
    });

    await this.telemetry.record(result, request);
    return result;
  }

  private selectAgent(
    candidates: AgentRegistration[],
    request: AgentRequest,
  ): AgentRegistration {
    // Simple: pick the most recently registered active version.
    // Production: add load, health weight, and latency p95 here.
    return candidates.sort(
      (a, b) => b.registeredAt.getTime() - a.registeredAt.getTime(),
    )[0];
  }
}

The ordering matters. Permission check happens before budget check, and budget check happens before dispatch. If you check budget after dispatch, you have already spent the tokens. If you check permissions after dispatch, you have already executed the agent.

Permission Boundary Enforcement

Each agent declares its required scopes at registration time. The permission enforcer validates that the caller context grants those scopes before dispatch proceeds.

interface PermissionCheck {
  agentId: string;
  requiredScopes: AgentScope[];
  callerContext: CallerContext;
}

interface PermissionResult {
  allowed: boolean;
  grantedScopes: AgentScope[];
  deniedScopes: AgentScope[];
  reason?: string;
}

class PermissionEnforcer {
  private scopeGrants: ScopeGrantStore;

  async check(params: PermissionCheck): Promise<PermissionResult> {
    const grantedScopes = await this.scopeGrants.getForCaller(
      params.callerContext.tenantId,
      params.callerContext.serviceId ?? params.callerContext.userId,
    );

    const grantedSet = new Set(grantedScopes);
    const deniedScopes = params.requiredScopes.filter((s) => !grantedSet.has(s));

    if (deniedScopes.length > 0) {
      return {
        allowed: false,
        grantedScopes,
        deniedScopes,
        reason: `Missing required scopes: ${deniedScopes.join(", ")}`,
      };
    }

    return { allowed: true, grantedScopes, deniedScopes: [] };
  }
}

The key design choice here is that scopes are declared on the agent, not on the route. When a new agent is registered with call:external-api in its required scopes, any caller without that grant is immediately blocked, without changes to routing configuration. The agent’s registration document is the policy.

This approach also makes the permission surface auditable. You can query the registry: “which agents require write:user-data?” and get a complete answer. With ad hoc per-agent auth, that query is a grep across multiple codebases.

Token Budget Enforcement

Budget enforcement at the gateway layer is categorically different from budget monitoring. Monitoring tells you after the fact that you spent too much. Enforcement prevents the spend from happening.

interface BudgetPrecheck {
  agentId: string;
  tenantId: string;
  estimatedTokens: number;
  budget: TokenBudget;
}

interface BudgetRecord {
  agentId: string;
  tenantId: string;
  tokensUsed: number;
}

interface BudgetCheckResult {
  allowed: boolean;
  currentDailyUsage: number;
  dailyLimit: number;
  reason?: string;
}

class BudgetEnforcer {
  private redis: Redis;

  async precheck(params: BudgetPrecheck): Promise<BudgetCheckResult> {
    const key = `budget:${params.agentId}:${params.tenantId}:daily`;
    const currentUsage = parseInt((await this.redis.get(key)) ?? "0", 10);
    const projectedUsage = currentUsage + params.estimatedTokens;

    if (projectedUsage > params.budget.dailyLimit) {
      return {
        allowed: false,
        currentDailyUsage: currentUsage,
        dailyLimit: params.budget.dailyLimit,
        reason: `Daily token budget would be exceeded: ${projectedUsage} > ${params.budget.dailyLimit}`,
      };
    }

    if (params.estimatedTokens > params.budget.perCallLimit) {
      return {
        allowed: false,
        currentDailyUsage: currentUsage,
        dailyLimit: params.budget.dailyLimit,
        reason: `Per-call token estimate exceeds limit: ${params.estimatedTokens} > ${params.budget.perCallLimit}`,
      };
    }

    // Emit warning if approaching threshold
    const usageRatio = projectedUsage / params.budget.dailyLimit;
    if (usageRatio >= params.budget.alertThreshold) {
      await this.emitBudgetWarning(params, projectedUsage);
    }

    return {
      allowed: true,
      currentDailyUsage: currentUsage,
      dailyLimit: params.budget.dailyLimit,
    };
  }

  async record(params: BudgetRecord): Promise<void> {
    const key = `budget:${params.agentId}:${params.tenantId}:daily`;
    const ttl = this.secondsUntilMidnightUTC();
    await this.redis.incrby(key, params.tokensUsed);
    await this.redis.expire(key, ttl);
  }

  private secondsUntilMidnightUTC(): number {
    const now = new Date();
    const midnight = new Date(now);
    midnight.setUTCHours(24, 0, 0, 0);
    return Math.floor((midnight.getTime() - now.getTime()) / 1000);
  }

  private async emitBudgetWarning(
    params: BudgetPrecheck,
    projectedUsage: number,
  ): Promise<void> {
    // Publish to internal event bus; budget alert subscribers handle notification
    await this.redis.publish(
      "budget:warnings",
      JSON.stringify({
        agentId: params.agentId,
        tenantId: params.tenantId,
        projectedUsage,
        dailyLimit: params.budget.dailyLimit,
        ratio: projectedUsage / params.budget.dailyLimit,
        timestamp: new Date().toISOString(),
      }),
    );
  }
}

The precheck uses the per-call estimate from the capability declaration, not a real token count, because the real count is not available until after the model call completes. This means you need reasonably accurate estimates in your capability registration. Budget enforcement is inherently approximate. The alternative, blocking nothing until actual usage is measured, eliminates the protection.

Fleet Observability

This is where the “Framework-Invisible Complexity” problem is solved. Every agent call flows through the gateway, so the gateway can emit unified telemetry regardless of what framework each agent uses internally.

interface AgentCallRecord {
  requestId: string;
  traceId: string;
  spanId: string;
  agentId: string;
  agentVersion: string;
  capability: string;
  tenantId: string;
  status: DispatchResult["status"];
  tokensUsed: number;
  latencyMs: number;
  timestamp: string;
}

class GatewayTelemetry {
  private metrics: MetricsClient;
  private events: EventStore;

  async record(result: DispatchResult, request: AgentRequest): Promise<void> {
    const record: AgentCallRecord = {
      requestId: result.requestId,
      traceId: request.callerContext.traceId,
      spanId: request.callerContext.spanId,
      agentId: result.agentId,
      agentVersion: result.agentVersion,
      capability: request.capability,
      tenantId: request.callerContext.tenantId,
      status: result.status,
      tokensUsed: result.tokensUsed,
      latencyMs: result.latencyMs,
      timestamp: new Date().toISOString(),
    };

    // Write to append-only event store for audit and replay
    await this.events.append("agent_calls", record);

    // Emit structured metrics for dashboards and alerts
    this.metrics.histogram("agent.latency_ms", result.latencyMs, {
      agent_id: result.agentId,
      agent_version: result.agentVersion,
      capability: request.capability,
      status: result.status,
    });

    this.metrics.counter("agent.tokens_used", result.tokensUsed, {
      agent_id: result.agentId,
      tenant_id: request.callerContext.tenantId,
    });

    this.metrics.counter("agent.calls_total", 1, {
      agent_id: result.agentId,
      status: result.status,
    });
  }
}

Because all calls are recorded with agentVersion, you can plot latency and success rate over time and see exactly when a framework upgrade shifted the baseline. The version field makes the Framework-Invisible Complexity problem visible: before version 3.2.1, p95 latency was 1200ms and success rate was 94%. After 3.2.1, p95 latency is 1800ms and success rate is 88%. The gateway catches this; per-agent monitoring does not, because each agent team only sees their own data.

The Multi-Model SLO Orphaning problem is addressed by the same mechanism. The gateway does not care which underlying model an agent uses. It records agent-level SLOs. If an agent’s success rate drops because it was silently migrated to a deprecated model, the gateway’s error rate chart for that agent shows the regression immediately.

Agent Lifecycle Management

Lifecycle management covers four transitions: registration, deprecation, draining, and sunset.

Registration is covered by the registry. Deprecation is the signal that a new version is preferred and the old version should stop receiving new traffic. Draining allows in-flight requests to complete before the agent is fully removed. Sunset is the hard cutoff.

class LifecycleManager {
  private registry: AgentRegistry;
  private healthChecker: HealthChecker;

  async deprecate(agentId: string, version: string, sunsetAt: Date): Promise<void> {
    await this.registry.updateStatus(agentId, version, "deprecated", { sunsetAt });
  }

  async drain(agentId: string, version: string): Promise<void> {
    await this.registry.updateStatus(agentId, version, "draining");
    // New requests stop routing here; existing requests complete.
    // The dispatcher checks status before routing.
  }

  async runHealthChecks(): Promise<void> {
    const agents = await this.registry.listAll({ status: "active" });

    await Promise.allSettled(
      agents.map(async (agent) => {
        const healthy = await this.healthChecker.ping(agent.healthEndpoint);
        if (!healthy) {
          await this.registry.updateStatus(agent.agentId, agent.version, "offline");
          await this.emitHealthAlert(agent);
        }
      }),
    );
  }

  async enforceSunsets(): Promise<void> {
    const now = new Date();
    const pastSunset = await this.registry.findBySunsetBefore(now);

    for (const agent of pastSunset) {
      await this.registry.updateStatus(agent.agentId, agent.version, "offline");
    }
  }

  private async emitHealthAlert(agent: AgentRegistration): Promise<void> {
    // Notify agent owner via internal alerting channel
    console.error(`[gateway] Agent ${agent.agentId}@${agent.version} failed health check. Owner: ${agent.owner}`);
  }
}

The health check runs on a schedule (every 30 seconds is reasonable for most fleets). The enforceSunsets job runs at least hourly. These are not optional: without them, the registry drifts from reality, and the gateway starts routing to dead or deprecated agents.

Tradeoffs

Decision	Option A	Option B	When to choose A
Registry store	Relational DB with cache	In-memory + periodic sync	Most cases; you want transactional status transitions
Permission model	Scopes on agent registration	ACLs per caller-agent pair	Scope model scales better; ACL model is more granular
Budget enforcement	Pre-call estimate check	Post-call hard cap	Pre-call is approximate but prevents spend; post-call is exact but allows overage
Health check frequency	30s active ping	Event-driven status updates	Active ping catches silent failures; event-driven requires agents to self-report
Telemetry store	Append-only event table	Time-series DB only	Event table enables audit and replay; TSDB enables fast dashboards. Use both.
Routing strategy	Capability-based	Agent-ID-specific	Capability-based allows version transparent upgrades; ID-specific required for pinned integrations

Production Considerations

A few things that will hurt you if you skip them.

The registry cache TTL should be short but not zero. Zero means every request hits the database. Sixty seconds means status changes propagate in under a minute. If you need faster propagation, use cache invalidation on write rather than reducing the TTL.

The agentVersion field in telemetry is load-bearing. Without it, you cannot distinguish a framework upgrade regression from an organic degradation. Tag every metric with version.

The sunsetAt field on deprecation notices is a contract with agent owners, not just metadata. Build a job that sends automated reminders to the owner email at 30 days, 7 days, and 24 hours before sunset. Owners will ignore deprecations that have no deadline. They will not ignore a reminder that their agent goes offline tomorrow.

Budget estimates in capability registration need to be reviewed quarterly. Token usage per call drifts as prompts change, context windows grow, and model behavior shifts across versions. An estimate that was accurate six months ago may be off by a factor of two.

Do not run the gateway as a single process. It is a critical path component. Deploy at least three instances behind a load balancer, and make the registry and budget stores external (Postgres, Redis) rather than in-process. If the gateway goes down, every agent in your fleet goes down with it.

The Math Compels It

The compound reliability argument is not hypothetical. Ten agents at 95% each give you 60% system reliability. Twenty agents give you 36%. The only way to drive that number back up is to either increase per-agent reliability or reduce the number of agents in a task’s critical path. Both require knowing what each agent’s actual reliability is, which requires centralized telemetry, which requires the gateway.

The Gartner numbers confirm that most engineering organizations have not built this yet. Twelve percent have centralized management in place. The other 88% are operating agent fleets the same way teams operated microservices before they built service meshes: one at a time, by hand, with a growing pile of incidents they cannot explain.

The gateway is the service mesh for your agent fleet. The math for building it is the same.