LLM Gateway Architecture: Rate Limiting, Caching, and Routing Across Multiple Providers

Most AI applications start the same way: one API call to OpenAI, one .env file with a key, and it works. Then production happens. You hit rate limits at 3am. A model gets deprecated. Costs spike because the same prompt gets re-run hundreds of times. You need GPT-4o for complex reasoning but a cheaper model for classification. Your team adds Anthropic as a backup and now you have two incompatible SDK integrations in the same codebase.

The fix is a gateway layer sitting between your application and every LLM provider. Not a SaaS product wrapping your calls: an internal service your team owns and controls. This article covers how to design that gateway for production: semantic caching, provider routing, rate limiting, response normalization, and observability. Every concept comes with TypeScript code you can actually use.

What the Gateway Owns

Before writing any code, be precise about responsibility boundaries. The gateway handles:

• Authentication and credential management for all providers
• Request normalization: translate one canonical format into provider-specific API shapes
• Routing: decide which provider and model handles each request
• Caching: return cached responses when semantically equivalent requests have been seen before
• Rate limiting: enforce per-model, per-tenant, and per-key limits before requests leave your infrastructure
• Observability: track token usage, latency, cost, and error rates per provider and model
• Fallback chains: retry with an alternate provider when the primary fails

Your application code stops caring about provider specifics. It sends requests to the gateway using one interface and receives normalized responses.

The Canonical Request Interface

Start with a provider-agnostic request type. This is the contract between your application and the gateway.

type Role = "system" | "user" | "assistant";

interface Message {
  role: Role;
  content: string;
}

interface GatewayRequest {
  messages: Message[];
  model: string;          // logical name, e.g. "fast", "smart", "vision"
  maxTokens?: number;
  temperature?: number;
  stream?: boolean;
  metadata?: {
    tenantId: string;
    requestId: string;
    feature: string;      // "summarization", "classification", etc.
  };
}

interface GatewayResponse {
  content: string;
  model: string;          // actual model used
  provider: string;       // actual provider used
  usage: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };
  latencyMs: number;
  cached: boolean;
}

The model field in the request is a logical name your team controls. The gateway resolves it to a concrete provider and model. If you want to swap GPT-4o for Claude 3.5 Sonnet behind “smart”, you change the routing table, not application code.

Provider Normalization

Each provider has a different API shape. OpenAI and Anthropic differ in message structure, authentication headers, and response formats. Write an adapter per provider.

interface ProviderAdapter {
  name: string;
  complete(request: NormalizedRequest): Promise<ProviderResponse>;
}

interface NormalizedRequest {
  messages: Message[];
  model: string;
  maxTokens: number;
  temperature: number;
}

interface ProviderResponse {
  content: string;
  usage: { promptTokens: number; completionTokens: number };
  rawModel: string;
}

// OpenAI adapter
class OpenAIAdapter implements ProviderAdapter {
  name = "openai";

  constructor(private apiKey: string, private baseUrl = "https://api.openai.com/v1") {}

  async complete(request: NormalizedRequest): Promise<ProviderResponse> {
    const res = await fetch(`${this.baseUrl}/chat/completions`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${this.apiKey}`,
      },
      body: JSON.stringify({
        model: request.model,
        messages: request.messages,
        max_tokens: request.maxTokens,
        temperature: request.temperature,
      }),
    });

    if (!res.ok) {
      throw new ProviderError("openai", res.status, await res.text());
    }

    const data = await res.json();
    return {
      content: data.choices[0].message.content,
      usage: {
        promptTokens: data.usage.prompt_tokens,
        completionTokens: data.usage.completion_tokens,
      },
      rawModel: data.model,
    };
  }
}

// Anthropic adapter
class AnthropicAdapter implements ProviderAdapter {
  name = "anthropic";

  constructor(private apiKey: string) {}

  async complete(request: NormalizedRequest): Promise<ProviderResponse> {
    // Anthropic separates system from user messages
    const systemMessage = request.messages.find((m) => m.role === "system");
    const conversationMessages = request.messages.filter((m) => m.role !== "system");

    const body: Record<string, unknown> = {
      model: request.model,
      max_tokens: request.maxTokens,
      messages: conversationMessages,
    };

    if (systemMessage) {
      body.system = systemMessage.content;
    }

    const res = await fetch("https://api.anthropic.com/v1/messages", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "x-api-key": this.apiKey,
        "anthropic-version": "2023-06-01",
      },
      body: JSON.stringify(body),
    });

    if (!res.ok) {
      throw new ProviderError("anthropic", res.status, await res.text());
    }

    const data = await res.json();
    return {
      content: data.content[0].text,
      usage: {
        promptTokens: data.usage.input_tokens,
        completionTokens: data.usage.output_tokens,
      },
      rawModel: data.model,
    };
  }
}

class ProviderError extends Error {
  constructor(
    public provider: string,
    public statusCode: number,
    public body: string
  ) {
    super(`${provider} returned ${statusCode}: ${body}`);
  }
}

With adapters in place, the gateway core never touches provider-specific APIs again.

Routing and Fallback Chains

The routing table maps logical model names to an ordered list of provider/model pairs. The gateway tries them in order, falling back when a provider fails.

interface RouteTarget {
  provider: string;
  model: string;
}

type RoutingTable = Record<string, RouteTarget[]>;

const defaultRoutes: RoutingTable = {
  fast: [
    { provider: "openai", model: "gpt-4o-mini" },
    { provider: "anthropic", model: "claude-haiku-3-5" },
  ],
  smart: [
    { provider: "anthropic", model: "claude-sonnet-4-5" },
    { provider: "openai", model: "gpt-4o" },
  ],
  vision: [
    { provider: "openai", model: "gpt-4o" },
  ],
};

class Router {
  constructor(
    private routes: RoutingTable,
    private adapters: Map<string, ProviderAdapter>
  ) {}

  async route(
    logicalModel: string,
    request: NormalizedRequest
  ): Promise<{ response: ProviderResponse; provider: string; model: string }> {
    const targets = this.routes[logicalModel];

    if (!targets || targets.length === 0) {
      throw new Error(`No route configured for model: ${logicalModel}`);
    }

    let lastError: Error | undefined;

    for (const target of targets) {
      const adapter = this.adapters.get(target.provider);

      if (!adapter) {
        continue;
      }

      try {
        const response = await adapter.complete({
          ...request,
          model: target.model,
        });

        return { response, provider: target.provider, model: target.model };
      } catch (err) {
        lastError = err as Error;

        // Only fall back on provider errors, not on bad requests
        if (err instanceof ProviderError && err.statusCode === 400) {
          throw err;
        }

        console.warn(`Provider ${target.provider} failed, trying next`, {
          error: (err as Error).message,
          target,
        });
      }
    }

    throw lastError ?? new Error(`All providers failed for model: ${logicalModel}`);
  }
}

The fallback logic is intentional: a 400 (bad request) means your prompt is malformed, so retrying on a different provider will fail the same way. Rate limit errors (429) and server errors (500, 503) are worth retrying elsewhere.

Semantic Caching

Exact-match caching on LLM prompts rarely helps. Two prompts asking the same thing with slightly different wording won’t share a cache key. Semantic caching uses embedding similarity to find “close enough” previous responses.

import { createHash } from "crypto";

interface CacheEntry {
  embedding: number[];
  response: GatewayResponse;
  createdAt: number;
}

class SemanticCache {
  private entries: CacheEntry[] = [];

  constructor(
    private embedder: (text: string) => Promise<number[]>,
    private similarityThreshold = 0.95,
    private ttlMs = 1000 * 60 * 60 // 1 hour
  ) {}

  async get(prompt: string): Promise<GatewayResponse | null> {
    const queryEmbedding = await this.embedder(prompt);
    const now = Date.now();

    let bestMatch: { entry: CacheEntry; similarity: number } | null = null;

    for (const entry of this.entries) {
      if (now - entry.createdAt > this.ttlMs) {
        continue;
      }

      const similarity = cosineSimilarity(queryEmbedding, entry.embedding);

      if (
        similarity >= this.similarityThreshold &&
        (!bestMatch || similarity > bestMatch.similarity)
      ) {
        bestMatch = { entry, similarity };
      }
    }

    return bestMatch ? { ...bestMatch.entry.response, cached: true } : null;
  }

  async set(prompt: string, response: GatewayResponse): Promise<void> {
    const embedding = await this.embedder(prompt);
    this.entries.push({ embedding, response, createdAt: Date.now() });
  }
}

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] ** 2;
    magB += b[i] ** 2;
  }

  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

In production, replace the in-memory entries array with Redis or a vector store. The embedder call is itself an API call, so cache embeddings separately with a fast exact-match key on the raw prompt text.

One important tradeoff: semantic cache hits return stale responses. This works well for factual Q&A, summarization, and classification. It is a bad fit for requests where freshness matters (news summaries, real-time data synthesis) or where user-specific context is baked into the prompt.

Rate Limiting Per Model and Tenant

LLM providers enforce rate limits on tokens per minute, requests per minute, and sometimes daily token budgets. Your gateway needs to track usage and enforce limits before requests leave your network, not after hitting a 429.

interface RateLimitConfig {
  requestsPerMinute: number;
  tokensPerMinute: number;
}

interface UsageWindow {
  requests: number;
  tokens: number;
  windowStart: number;
}

class RateLimiter {
  private windows = new Map<string, UsageWindow>();

  constructor(private limits: Map<string, RateLimitConfig>) {}

  private getWindow(key: string): UsageWindow {
    const now = Date.now();
    const existing = this.windows.get(key);

    if (!existing || now - existing.windowStart > 60_000) {
      const fresh: UsageWindow = { requests: 0, tokens: 0, windowStart: now };
      this.windows.set(key, fresh);
      return fresh;
    }

    return existing;
  }

  checkAndConsume(key: string, estimatedTokens: number): { allowed: boolean; retryAfterMs?: number } {
    const config = this.limits.get(key);

    if (!config) {
      // No limit configured: allow
      return { allowed: true };
    }

    const window = this.getWindow(key);

    if (
      window.requests >= config.requestsPerMinute ||
      window.tokens + estimatedTokens > config.tokensPerMinute
    ) {
      const retryAfterMs = 60_000 - (Date.now() - window.windowStart);
      return { allowed: false, retryAfterMs };
    }

    window.requests++;
    window.tokens += estimatedTokens;
    return { allowed: true };
  }

  recordActualUsage(key: string, actualTokens: number, estimatedTokens: number): void {
    const window = this.getWindow(key);
    // Correct the estimate with actual usage
    window.tokens += actualTokens - estimatedTokens;
  }
}

The key insight here: you need to estimate tokens before the request to enforce limits pre-flight. Use a simple heuristic (4 characters per token for English text) or a tokenizer library. After the response, correct the window with actual usage from the provider’s response.

Rate limit keys can be composite: ${tenantId}:${provider}:${model} lets you enforce different limits per tenant, per provider, and per model independently.

Putting It Together: The Gateway Core

class LLMGateway {
  constructor(
    private router: Router,
    private cache: SemanticCache,
    private rateLimiter: RateLimiter,
    private metrics: MetricsCollector
  ) {}

  async complete(request: GatewayRequest): Promise<GatewayResponse> {
    const startTime = Date.now();
    const { metadata } = request;

    // Build a cache key from the prompt text (system + user messages)
    const promptText = request.messages.map((m) => m.content).join("\n");

    // Check semantic cache first
    const cached = await this.cache.get(promptText);
    if (cached) {
      this.metrics.record({
        event: "cache_hit",
        model: request.model,
        tenantId: metadata?.tenantId,
        latencyMs: Date.now() - startTime,
      });
      return cached;
    }

    // Estimate tokens for rate limit pre-flight
    const estimatedTokens = Math.ceil(promptText.length / 4);
    const rateLimitKey = `${metadata?.tenantId ?? "global"}:${request.model}`;

    const { allowed, retryAfterMs } = this.rateLimiter.checkAndConsume(
      rateLimitKey,
      estimatedTokens
    );

    if (!allowed) {
      throw new RateLimitError(retryAfterMs ?? 60_000);
    }

    // Route and execute
    const normalized: NormalizedRequest = {
      messages: request.messages,
      model: request.model,
      maxTokens: request.maxTokens ?? 2048,
      temperature: request.temperature ?? 0.7,
    };

    const { response, provider, model } = await this.router.route(
      request.model,
      normalized
    );

    const latencyMs = Date.now() - startTime;
    const actualTokens = response.usage.promptTokens + response.usage.completionTokens;

    this.rateLimiter.recordActualUsage(rateLimitKey, actualTokens, estimatedTokens);

    const gatewayResponse: GatewayResponse = {
      content: response.content,
      model,
      provider,
      usage: {
        promptTokens: response.usage.promptTokens,
        completionTokens: response.usage.completionTokens,
        totalTokens: actualTokens,
      },
      latencyMs,
      cached: false,
    };

    // Store in cache for future similar requests
    await this.cache.set(promptText, gatewayResponse);

    this.metrics.record({
      event: "completion",
      model,
      provider,
      tenantId: metadata?.tenantId,
      feature: metadata?.feature,
      tokens: actualTokens,
      latencyMs,
    });

    return gatewayResponse;
  }
}

class RateLimitError extends Error {
  constructor(public retryAfterMs: number) {
    super(`Rate limit exceeded. Retry after ${retryAfterMs}ms`);
  }
}

Observability: What to Measure

Token usage and latency percentiles are the two signals that matter most for LLM infrastructure. Everything else is downstream of them.

interface MetricsEvent {
  event: "completion" | "cache_hit" | "provider_error" | "rate_limit";
  model?: string;
  provider?: string;
  tenantId?: string;
  feature?: string;
  tokens?: number;
  latencyMs?: number;
  errorCode?: number;
}

class MetricsCollector {
  record(event: MetricsEvent): void {
    // Emit to your metrics backend (Datadog, Prometheus, etc.)
    // Structure the metric name to enable slicing by dimension
    const tags = [
      `model:${event.model ?? "unknown"}`,
      `provider:${event.provider ?? "unknown"}`,
      `tenant:${event.tenantId ?? "global"}`,
      `feature:${event.feature ?? "unknown"}`,
      `event:${event.event}`,
    ];

    if (event.tokens !== undefined) {
      // track cost: tokens * per-model rate
      console.log(`metric llm.tokens ${event.tokens} ${tags.join(",")}`);
    }

    if (event.latencyMs !== undefined) {
      console.log(`metric llm.latency_ms ${event.latencyMs} ${tags.join(",")}`);
    }

    console.log(`metric llm.requests 1 ${tags.join(",")}`);
  }
}

The dimensions that matter in production:

• Token usage by tenant and feature (cost attribution)
• p50/p95/p99 latency by provider and model (SLA monitoring)
• Cache hit rate by feature (cache effectiveness)
• Provider error rate by status code (reliability)
• Fallback rate: how often you hit the secondary provider (primary health signal)

A high fallback rate on your primary provider is an early warning sign before you start seeing errors on the application side.

Tradeoffs Table

Decision	Option A	Option B	When to pick B
Cache granularity	Exact match on full prompt	Semantic similarity	Your users rephrase the same questions; high cache miss rate on exact match
Rate limit storage	In-memory (per instance)	Redis (distributed)	Running multiple gateway instances; limits must be enforced globally
Fallback trigger	Any 5xx from provider	5xx + latency threshold	Slow providers hurt UX as much as errors; p99 > 10s should trigger fallback
Model routing	Static table	Dynamic (cost or latency optimized)	You have real-time pricing data or want to shift load dynamically
Embedding model for cache	Same provider you’re caching for	Separate small model	Avoid circular dependency; a small local embedder is fine
Gateway deployment	In-process library	Separate HTTP service	Multiple services in different languages need to share the gateway

Production Considerations

Avoid caching sensitive prompts. If user data appears in the prompt, semantic cache hits could leak one user’s data to another. Scope cache keys by tenant or disable caching for sensitive features entirely.

Rate limit keys need tenant isolation by default. A single noisy tenant can exhaust global limits and affect everyone else. Default to per-tenant limits, with a global cap as a backstop.

Set a cache similarity threshold conservatively at first. Start at 0.97 and lower it only when you have data showing that 0.95-similar prompts produce acceptable responses for your use case. Wrong cache hits are worse than cache misses.

Budget for embedding API latency. Every cache lookup requires an embedding call. If your embedding provider has high latency, the cache check can add more latency than the cache hit saves. Pre-compute and store embeddings for known recurring prompts.

Track the real cost per feature, not just total spend. The feature tag in your metrics lets you see that your summarization feature costs $80/day while classification costs $5/day. That’s where optimization decisions come from.

Circuit break the cache, not just the providers. If your embedding service starts failing, the gateway should degrade gracefully (skip cache, go direct) rather than failing every request. The cache is an optimization layer, not a hard dependency.

The Shape of a Production Gateway

After building this system a few times, the pattern that holds up: keep the gateway as a thin HTTP service, not an in-process library. A separate service lets every team share it regardless of language, lets you deploy it with its own scaling policy, and makes it easier to update routing logic without touching application deployments.

The gateway should be boring infrastructure. Its job is to make every LLM call predictable: known cost, known latency bounds, graceful degradation when providers misbehave. Once it is in place, adding a new provider is an adapter class. Switching which model handles a feature is a config change. That leverage compounds as your AI surface area grows.