Designing a Tenant-Aware AI Gateway: Per-Customer Model Routing, Token Quotas, and Cost Attribution in Multi-Tenant SaaS

Most teams building SaaS products on top of LLMs start with a single shared API key and a flat request pipeline. It works until it doesn’t. One tenant spikes, and every other tenant’s latency degrades. Your monthly AI spend is $40,000 but you can’t tell which customers are responsible for $35,000 of it. A customer on your enterprise plan wants to pin to a specific model version for compliance reasons, and your current architecture has no way to support that without code changes.

These are not edge cases. They are the predictable failure modes of a naive shared-inference architecture, and they show up once you have more than a handful of paying customers.

The fix is an AI gateway layer that sits between your application code and the upstream LLM providers. This gateway handles tenant identity, routing policy, quota enforcement, and cost attribution as first-class concerns. This article covers how to design that layer.

The Core Data Model

Everything starts with tenant configuration. The gateway needs to know, per tenant, which model to use, what quota applies, and what priority they should receive in the queue.

interface TenantModelConfig {
  tenantId: string;
  provider: "openai" | "anthropic" | "google" | "azure-openai";
  model: string;
  modelVersion?: string;         // e.g., "gpt-4o-2024-11-20" for pinned deployments
  priority: "low" | "standard" | "high" | "critical";
  quota: TokenQuota;
  fallback?: {
    provider: "openai" | "anthropic" | "google" | "azure-openai";
    model: string;
    triggerOn: ("rate_limit" | "timeout" | "error")[];
  };
}

interface TokenQuota {
  windowSeconds: number;         // rolling window, e.g., 3600 for hourly
  inputTokenLimit: number;
  outputTokenLimit: number;
  totalTokenLimit: number;
  hardStop: boolean;             // if false, log and alert but allow through
}

interface TenantQuotaState {
  tenantId: string;
  windowStart: number;           // unix timestamp
  inputTokensUsed: number;
  outputTokensUsed: number;
  totalTokensUsed: number;
}

Store TenantModelConfig in your primary database. It is configuration, not hot-path state. Cache it aggressively with a short TTL (30-60 seconds) since it changes infrequently.

Store TenantQuotaState in Redis. It is hot-path state that needs atomic increment operations and must survive gateway restarts without expensive database lookups on every request.

Quota Enforcement at the Gateway

The naive approach reads the current usage from Redis, checks it against the limit, and either allows or rejects. This has a race condition under concurrent requests from the same tenant. Two requests can both read usage = 9,900 tokens, both see it below a 10,000 token limit, and both proceed, ending up at 19,900.

The fix is atomic increment with a post-check rollback. Use Redis INCRBY, check the result after the increment, and decrement if the limit was exceeded.

async function enforceQuota(
  redis: Redis,
  tenantId: string,
  estimatedTokens: number,
  quota: TokenQuota
): Promise<{ allowed: boolean; remaining: number }> {
  const key = `quota:${tenantId}:total`;
  const windowKey = `quota:${tenantId}:window`;

  // Initialize window if needed
  const windowStart = await redis.get(windowKey);
  const now = Math.floor(Date.now() / 1000);
  if (!windowStart || now - parseInt(windowStart) > quota.windowSeconds) {
    const pipe = redis.pipeline();
    pipe.set(windowKey, now.toString(), "EX", quota.windowSeconds);
    pipe.set(key, "0", "EX", quota.windowSeconds);
    await pipe.exec();
  }

  // Atomic increment first, check after
  const newTotal = await redis.incrby(key, estimatedTokens);
  const remaining = quota.totalTokenLimit - newTotal;

  if (newTotal > quota.totalTokenLimit) {
    // Roll back the increment
    await redis.decrby(key, estimatedTokens);

    if (quota.hardStop) {
      return { allowed: false, remaining: Math.max(0, remaining + estimatedTokens) };
    }

    // Soft quota: log and allow through
    console.warn("quota_exceeded_soft", {
      tenantId,
      newTotal,
      limit: quota.totalTokenLimit,
    });
  }

  return { allowed: true, remaining: Math.max(0, remaining) };
}

Pre-request token estimation is imprecise. For input tokens you can count with a tokenizer library like tiktoken, but output tokens are unknown until the response completes. A practical approach: reserve a conservative estimate at request time (input tokens + a configurable output estimate based on max_tokens), then reconcile against actual usage from the response’s usage field. Update the Redis counter with the delta after each completed request.

async function reconcileQuota(
  redis: Redis,
  tenantId: string,
  estimatedTokens: number,
  actualTokens: number
): Promise<void> {
  const delta = actualTokens - estimatedTokens;
  if (delta !== 0) {
    const key = `quota:${tenantId}:total`;
    if (delta > 0) {
      await redis.incrby(key, delta);
    } else {
      await redis.decrby(key, Math.abs(delta));
    }
  }
}

Tenant-Aware Routing

The routing layer resolves which provider and model to call based on the tenant’s config, applies any fallback logic, and selects the appropriate credentials.

interface RouteResolution {
  provider: string;
  model: string;
  apiKey: string;
  baseUrl?: string;   // for Azure OpenAI or self-hosted endpoints
  requestId: string;
}

async function resolveRoute(
  tenantConfig: TenantModelConfig,
  configStore: TenantConfigStore,
  credentialStore: CredentialStore
): Promise<RouteResolution> {
  const requestId = crypto.randomUUID();

  const apiKey = await credentialStore.getApiKey(
    tenantConfig.provider,
    tenantConfig.tenantId   // per-tenant keys, or shared pool key
  );

  return {
    provider: tenantConfig.provider,
    model: tenantConfig.modelVersion ?? tenantConfig.model,
    apiKey,
    baseUrl: tenantConfig.provider === "azure-openai"
      ? await configStore.getAzureEndpoint(tenantConfig.tenantId)
      : undefined,
    requestId,
  };
}

Model version pinning deserves explicit handling. When modelVersion is set, send that exact string to the provider. Do not let the provider silently upgrade. Compliance-sensitive tenants on enterprise plans need this guarantee: the model that passed their evaluation last quarter should be the model running today.

For Azure OpenAI, the “model” is actually the deployment name. The mapping from deployment name to model version is on the Azure side. Your config needs to store the deployment name as modelVersion, not the canonical model identifier.

Priority Queuing

Without a priority queue, a high-volume low-tier tenant can saturate your upstream rate limits and starve higher-priority tenants of capacity. Rate limits on most LLM providers are per-API-key, which means shared-key deployments share the rate limit budget.

The simplest effective approach: use separate API keys per priority tier. A critical tier tenant uses a dedicated API key with its own rate limit. standard and low tenants share keys per pool.

For more granular control, implement a token bucket queue per priority tier. Requests enter a tier-specific queue. A dispatcher pulls from queues in priority order with a weighted fair queuing algorithm that prevents starvation of lower-priority tiers.

type PriorityTier = "critical" | "high" | "standard" | "low";

// Weights for weighted fair queuing
const TIER_WEIGHTS: Record<PriorityTier, number> = {
  critical: 8,
  high: 4,
  standard: 2,
  low: 1,
};

interface QueuedRequest {
  requestId: string;
  tenantId: string;
  priority: PriorityTier;
  enqueuedAt: number;
  resolve: (route: RouteResolution) => void;
  reject: (error: Error) => void;
}

class PriorityDispatcher {
  private queues: Map<PriorityTier, QueuedRequest[]> = new Map([
    ["critical", []],
    ["high", []],
    ["standard", []],
    ["low", []],
  ]);

  // Round-robin counter across tiers, weighted
  private serviceCounters: Map<PriorityTier, number> = new Map();

  enqueue(request: QueuedRequest): void {
    const queue = this.queues.get(request.priority)!;
    queue.push(request);
  }

  dequeue(): QueuedRequest | null {
    // Select tier by weighted round-robin
    for (const [tier, weight] of Object.entries(TIER_WEIGHTS) as [PriorityTier, number][]) {
      const counter = this.serviceCounters.get(tier) ?? 0;
      const queue = this.queues.get(tier)!;

      if (queue.length > 0 && counter < weight) {
        this.serviceCounters.set(tier, counter + 1);
        return queue.shift()!;
      }
    }

    // Reset all counters and try again
    this.serviceCounters.clear();
    for (const [tier] of this.queues) {
      const queue = this.queues.get(tier)!;
      if (queue.length > 0) {
        return queue.shift()!;
      }
    }

    return null;
  }
}

A real implementation would distribute this queue across gateway instances using Redis sorted sets rather than in-process queues. The pattern is the same: score by priority tier and enqueue timestamp, pop from the lowest score.

Cost Attribution

The gateway is the right place to record cost data because it sits on every LLM request and has access to both tenant identity and token usage from the response.

interface InferenceEvent {
  eventId: string;              // idempotency key
  tenantId: string;
  requestId: string;
  provider: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  inputCostMicrodollars: number;  // integer microdollars, never floats
  outputCostMicrodollars: number;
  totalCostMicrodollars: number;
  latencyMs: number;
  status: "success" | "error" | "timeout";
  errorCode?: string;
  createdAt: Date;
}

// Pricing table in microdollars per 1000 tokens
const MODEL_PRICING: Record<string, { input: number; output: number }> = {
  "gpt-4o": { input: 2500, output: 10000 },
  "gpt-4o-mini": { input: 150, output: 600 },
  "claude-3-5-sonnet-20241022": { input: 3000, output: 15000 },
  "claude-3-haiku-20240307": { input: 250, output: 1250 },
};

function calculateCost(
  model: string,
  inputTokens: number,
  outputTokens: number
): { inputCostMicrodollars: number; outputCostMicrodollars: number; totalCostMicrodollars: number } {
  const pricing = MODEL_PRICING[model];
  if (!pricing) {
    throw new Error(`No pricing data for model: ${model}`);
  }

  // Integer arithmetic throughout. Round once at the end per component.
  const inputCost = Math.round((inputTokens * pricing.input) / 1000);
  const outputCost = Math.round((outputTokens * pricing.output) / 1000);

  return {
    inputCostMicrodollars: inputCost,
    outputCostMicrodollars: outputCost,
    totalCostMicrodollars: inputCost + outputCost,
  };
}

Write InferenceEvent rows asynchronously after the response completes. Use a background queue (not await) so cost recording does not add to request latency. Include an idempotency key derived from requestId to handle duplicate writes from retry logic.

For aggregating costs per tenant over a billing period, a simple SQL query over the inference_events table is sufficient for most volumes. At very high volumes (millions of rows per day), a pre-aggregated daily roll-up table keeps query times under a second.

Shared vs. Dedicated Model Deployments

The question of whether to run shared or dedicated model deployments comes up at every scale point.

Dimension	Shared Deployment	Dedicated Deployment	Notes
Cost per tenant	Lower (shared capacity)	Higher (fixed baseline)	Dedicated is only cost-effective above ~$5k/mo per tenant
Isolation	None at inference layer	Full isolation	Rate limits, model versions, and capacity are independent
Model version pinning	Provider-dependent	Full control	Azure dedicated allows exact version locks
Compliance/data residency	Hard to guarantee	Achievable	Dedicated deployments can be region-locked
Ops complexity	Low	High	Each dedicated deployment is infrastructure to maintain
Failure blast radius	Wide (all tenants share rate limit)	Narrow (per-tenant)	One tenant’s spike cannot affect others

The practical decision rule: shared for everything until a tenant crosses into enterprise pricing, then evaluate whether their SLA requirements justify dedicated infrastructure. The gateway design above supports both: per-tenant API key configuration means you can point a specific tenant at a dedicated deployment without changing application code.

Production Considerations

Quota window alignment. Rolling windows in Redis have a subtle issue: if you use a fixed window (reset at the top of the hour), a tenant can send half their quota at 11:59 and the other half at 12:01, effectively doubling their rate for that two-minute period. Rolling windows avoid this but are more expensive to implement precisely. A practical middle ground: use a 1-hour sliding window with 5-minute granularity buckets.

Streaming and token counting. For streaming responses, you do not get a usage object until the stream completes. Options: estimate during streaming based on character count, wait for the [DONE] event and then reconcile, or disable quota enforcement for streaming and rely on post-hoc alerting. The third option is acceptable for soft quotas; hard quotas require the second option.

Model version staleness. LLM providers retire models. If a tenant has a pinned model version that gets deprecated, your gateway will start returning errors for that tenant without warning. Implement a model health check job that validates all pinned versions against a provider’s available models list daily. Alert on any tenant configuration pointing to a deprecated or soon-to-be-deprecated model.

Fallback activation. When a fallback provider activates (due to rate limit or timeout on the primary), the response comes from a different model than the tenant’s configuration specifies. Log this event explicitly. Some tenants have contractual commitments about which model processes their data, and silent fallbacks violate those commitments. At minimum, return a response header indicating which provider actually handled the request.

Observability. The five metrics to instrument on the gateway:

• Per-tenant token usage rate (input/output/total, plotted against quota)
• Per-tenant routing distribution (primary vs. fallback)
• Queue depth and wait time per priority tier
• Cost per tenant per day (catch billing anomalies early)
• Provider error rates by provider and model (degradation detection)

interface GatewayMetrics {
  tenantId: string;
  requestId: string;
  provider: string;
  model: string;
  routedToFallback: boolean;
  queueWaitMs: number;
  inferenceLatencyMs: number;
  inputTokens: number;
  outputTokens: number;
  quotaRemainingAfter: number;
  status: "success" | "quota_exceeded" | "provider_error" | "timeout";
}

Emit this as a structured log on every request. Feed it into your log aggregation pipeline for dashboards and alerting. The quotaRemainingAfter field is especially useful: you can alert when any tenant’s quota drops below 20% remaining with more than half the billing window left.

Tenant config cache invalidation. When a customer support team or self-serve UI updates a tenant’s model config, the gateway cache needs to invalidate. A pub/sub channel in Redis works well: publish a tenant:config:updated event with the tenantId, and all gateway instances subscribe and evict that tenant’s cached config.

Closing Insight

The AI gateway is not a proxy. It is the enforcement plane for your product’s economics. Cost attribution only works if you capture every token from every request. Quota enforcement only works if you close the race condition on concurrent increments. Model version guarantees only hold if you actively monitor for upstream deprecations.

Build the gateway as a first-class service, not as middleware bolted onto your existing API layer. The multi-tenant complexity does not fit cleanly into request-scoped middleware, and the operational requirements (config sync, quota state, cost aggregation) belong to a system with its own lifecycle.