LLM Cost Engineering in Production: Token Budgets, Model Routing, and Unit Economics for AI-Powered Applications

A four-agent LangChain system running for 264 hours and burning $47,000 in a single incident is not a horror story about runaway AI. It is a predictable outcome of shipping agent loops without cost governance. Context accumulated, the agents kept calling the model, and nobody had set a hard stop anywhere in the stack.

LLM costs are not like compute costs. Compute scales linearly with load. LLM costs can spike exponentially inside a single request if an agent loop goes wide or a prompt template grows unbounded. You need a different mental model and a different set of controls.

This article covers the engineering side of LLM cost management: how to enforce token budgets at the request level, how to route intelligently across model tiers, how to cache semantically rather than exactly, how to contain agent loops before they become incidents, and how to measure success in unit economics terms that actually matter to the business.

Token Budget Enforcement at the Request Level

The first line of defense is bounding every LLM call before it happens. Not at the application level, not at the billing alert level: at the point where you construct the prompt.

interface TokenBudget {
  systemPrompt: number;    // max tokens for system instructions
  history: number;         // max tokens for conversation history
  retrieved: number;       // max tokens for RAG context
  completion: number;      // max tokens for model output
  total: number;           // hard ceiling across all slots
}

interface BudgetAllocation {
  systemPromptTokens: number;
  historyTokens: number;
  retrievedTokens: number;
  completionTokens: number;
  totalEstimated: number;
  withinBudget: boolean;
}

function allocateBudget(
  systemPrompt: string,
  history: Message[],
  retrievedChunks: string[],
  budget: TokenBudget,
  estimateTokens: (text: string) => number
): BudgetAllocation {
  const systemTokens = Math.min(
    estimateTokens(systemPrompt),
    budget.systemPrompt
  );

  // Trim history from oldest first, keep within slot
  let historyTokens = 0;
  const trimmedHistory: Message[] = [];
  for (let i = history.length - 1; i >= 0; i--) {
    const t = estimateTokens(history[i].content);
    if (historyTokens + t > budget.history) break;
    trimmedHistory.unshift(history[i]);
    historyTokens += t;
  }

  // Trim retrieved context greedily
  let retrievedTokens = 0;
  const usedChunks: string[] = [];
  for (const chunk of retrievedChunks) {
    const t = estimateTokens(chunk);
    if (retrievedTokens + t > budget.retrieved) break;
    usedChunks.push(chunk);
    retrievedTokens += t;
  }

  const totalEstimated =
    systemTokens + historyTokens + retrievedTokens + budget.completion;

  return {
    systemPromptTokens: systemTokens,
    historyTokens,
    retrievedTokens,
    completionTokens: budget.completion,
    totalEstimated,
    withinBudget: totalEstimated <= budget.total,
  };
}

The key design choice here: withinBudget: false should be a hard circuit break, not a warning. If the allocation exceeds the ceiling, you do not call the model. You return an error to the caller or invoke a degraded fallback path. Calling the model and hoping for the best is how incidents happen.

Tie the completion token cap directly to the max_tokens parameter you send to the API. The model will not exceed it, but you also need to budget for it on the input side: total = input_tokens + max_tokens. Many engineers only think about input tokens and then wonder why their cost estimates are off by 30-40%.

Intelligent Model Routing

Not every LLM call needs GPT-4o or Claude Opus. A large fraction of production requests, classification tasks, simple extraction, short Q&A against structured context, are well-handled by smaller, cheaper models. The ratio is almost always higher than engineers expect until they measure it.

A cascading router works on task complexity signals: input token count, presence of structured reasoning markers, confidence from a lightweight classifier, or explicit task-type tags from the application layer.

type ModelTier = "nano" | "small" | "large" | "frontier";

interface ModelConfig {
  tier: ModelTier;
  modelId: string;
  inputCostPerMToken: number;   // $ per million input tokens
  outputCostPerMToken: number;  // $ per million output tokens
  contextWindow: number;        // max context tokens
  maxOutputTokens: number;
}

const MODEL_REGISTRY: Record<ModelTier, ModelConfig> = {
  nano: {
    tier: "nano",
    modelId: "gpt-4o-mini",
    inputCostPerMToken: 0.15,
    outputCostPerMToken: 0.60,
    contextWindow: 128_000,
    maxOutputTokens: 16_384,
  },
  small: {
    tier: "small",
    modelId: "claude-3-haiku-20240307",
    inputCostPerMToken: 0.25,
    outputCostPerMToken: 1.25,
    contextWindow: 200_000,
    maxOutputTokens: 4_096,
  },
  large: {
    tier: "large",
    modelId: "claude-3-5-sonnet-20241022",
    inputCostPerMToken: 3.00,
    outputCostPerMToken: 15.00,
    contextWindow: 200_000,
    maxOutputTokens: 8_192,
  },
  frontier: {
    tier: "frontier",
    modelId: "claude-opus-4-5",
    inputCostPerMToken: 15.00,
    outputCostPerMToken: 75.00,
    contextWindow: 200_000,
    maxOutputTokens: 32_768,
  },
};

interface RoutingSignals {
  estimatedInputTokens: number;
  taskType: "classification" | "extraction" | "summarization" | "reasoning" | "generation";
  requiresToolUse: boolean;
  requiresStructuredOutput: boolean;
  userTier: "free" | "pro" | "enterprise";
}

function selectModelTier(signals: RoutingSignals): ModelTier {
  // Free tier users always get nano
  if (signals.userTier === "free") return "nano";

  // Simple classification or extraction with small context: nano
  if (
    (signals.taskType === "classification" || signals.taskType === "extraction") &&
    signals.estimatedInputTokens < 2_000 &&
    !signals.requiresToolUse
  ) {
    return "nano";
  }

  // Summarization and structured output without reasoning: small
  if (
    (signals.taskType === "summarization" || signals.requiresStructuredOutput) &&
    signals.estimatedInputTokens < 8_000 &&
    !signals.requiresToolUse
  ) {
    return "small";
  }

  // Multi-step reasoning, tool use, or large context: large
  if (
    signals.taskType === "reasoning" ||
    signals.requiresToolUse ||
    signals.estimatedInputTokens > 20_000
  ) {
    return signals.userTier === "enterprise" ? "large" : "large";
  }

  // General generation: small by default, large for enterprise
  return signals.userTier === "enterprise" ? "large" : "small";
}

The cascading pattern takes this further: try the cheaper model, evaluate the output for confidence or schema validity, and escalate to the next tier only on failure. This adds latency on the escalation path but dramatically reduces cost on the common path. For synchronous user-facing requests, only cascade when latency budgets allow. For async pipelines, cascade freely.

Track escalation rates per feature. A feature with a 90% escalation rate means your routing signals are wrong for that task type, not that the cheap model is bad.

Semantic Caching

Exact-match caching of LLM responses works only for literal prompt repetition. Semantic caching matches on embedding similarity, allowing you to return cached responses for prompts that are functionally equivalent even if the wording differs.

import { createClient } from "redis";
import OpenAI from "openai";

interface CacheEntry {
  promptEmbedding: number[];
  response: string;
  modelUsed: string;
  inputTokens: number;
  outputTokens: number;
  createdAt: number;
  ttlSeconds: number;
}

class SemanticCache {
  private redis = createClient({ url: process.env.REDIS_URL });
  private openai = new OpenAI();
  private readonly similarityThreshold: number;
  private readonly namespace: string;

  constructor(namespace: string, similarityThreshold = 0.94) {
    this.namespace = namespace;
    this.similarityThreshold = similarityThreshold;
  }

  private cosineSimilarity(a: number[], b: number[]): number {
    let dot = 0, normA = 0, normB = 0;
    for (let i = 0; i < a.length; i++) {
      dot += a[i] * b[i];
      normA += a[i] * a[i];
      normB += b[i] * b[i];
    }
    return dot / (Math.sqrt(normA) * Math.sqrt(normB));
  }

  async embed(text: string): Promise<number[]> {
    const res = await this.openai.embeddings.create({
      model: "text-embedding-3-small",
      input: text,
    });
    return res.data[0].embedding;
  }

  async lookup(promptEmbedding: number[]): Promise<CacheEntry | null> {
    const keys = await this.redis.keys(`${this.namespace}:*`);
    let bestMatch: CacheEntry | null = null;
    let bestScore = 0;

    for (const key of keys) {
      const raw = await this.redis.get(key);
      if (!raw) continue;
      const entry: CacheEntry = JSON.parse(raw);
      const score = this.cosineSimilarity(promptEmbedding, entry.promptEmbedding);
      if (score > bestScore && score >= this.similarityThreshold) {
        bestScore = score;
        bestMatch = entry;
      }
    }

    return bestMatch;
  }

  async store(
    promptEmbedding: number[],
    response: string,
    meta: Pick<CacheEntry, "modelUsed" | "inputTokens" | "outputTokens" | "ttlSeconds">
  ): Promise<void> {
    const key = `${this.namespace}:${Date.now()}`;
    const entry: CacheEntry = {
      promptEmbedding,
      response,
      createdAt: Date.now(),
      ...meta,
    };
    await this.redis.setEx(key, meta.ttlSeconds, JSON.stringify(entry));
  }
}

In production, replace the naive linear scan with a vector store (Pinecone, pgvector, or Redis Stack with vector similarity). Linear scan is fine up to a few thousand entries but degrades badly at scale.

Two practical notes. First, set your similarity threshold conservatively at first (0.94-0.96) and lower it only if you confirm that lower-similarity matches are returning acceptable responses. A false cache hit that returns a wrong answer is worse than a cache miss. Second, cache TTLs should reflect how stale the underlying data can be. A cached response about current pricing becomes a liability after a price change.

Agent Loop Detection and Containment

The $47K incident comes from a specific failure mode: agents accumulating context across a loop, calling the model repeatedly, each call consuming more tokens than the last because the history keeps growing. The cost curve is super-linear.

interface AgentLoopMetrics {
  taskId: string;
  agentId: string;
  iterationCount: number;
  totalInputTokens: number;
  totalOutputTokens: number;
  totalCostUsd: number;
  startedAt: number;
  lastIterationAt: number;
  toolCallCounts: Record<string, number>;
}

interface LoopGovernanceConfig {
  maxIterations: number;
  maxTotalInputTokens: number;
  maxTotalCostUsd: number;
  maxDurationMs: number;
  maxRepeatedToolCalls: number;  // same tool called N times without progress
}

class AgentLoopGovernor {
  private metrics = new Map<string, AgentLoopMetrics>();

  private readonly config: LoopGovernanceConfig = {
    maxIterations: 25,
    maxTotalInputTokens: 500_000,
    maxTotalCostUsd: 2.00,
    maxDurationMs: 5 * 60 * 1000,    // 5 minutes
    maxRepeatedToolCalls: 5,
  };

  initialize(taskId: string, agentId: string): void {
    this.metrics.set(taskId, {
      taskId,
      agentId,
      iterationCount: 0,
      totalInputTokens: 0,
      totalOutputTokens: 0,
      totalCostUsd: 0,
      startedAt: Date.now(),
      lastIterationAt: Date.now(),
      toolCallCounts: {},
    });
  }

  recordIteration(
    taskId: string,
    inputTokens: number,
    outputTokens: number,
    costUsd: number,
    toolsCalled: string[]
  ): { shouldContinue: boolean; reason?: string } {
    const m = this.metrics.get(taskId);
    if (!m) return { shouldContinue: false, reason: "Unknown task" };

    m.iterationCount += 1;
    m.totalInputTokens += inputTokens;
    m.totalOutputTokens += outputTokens;
    m.totalCostUsd += costUsd;
    m.lastIterationAt = Date.now();

    for (const tool of toolsCalled) {
      m.toolCallCounts[tool] = (m.toolCallCounts[tool] ?? 0) + 1;
    }

    if (m.iterationCount >= this.config.maxIterations) {
      return { shouldContinue: false, reason: `Iteration limit: ${m.iterationCount}` };
    }
    if (m.totalInputTokens >= this.config.maxTotalInputTokens) {
      return { shouldContinue: false, reason: `Token limit: ${m.totalInputTokens}` };
    }
    if (m.totalCostUsd >= this.config.maxTotalCostUsd) {
      return { shouldContinue: false, reason: `Cost limit: $${m.totalCostUsd.toFixed(4)}` };
    }
    if (Date.now() - m.startedAt >= this.config.maxDurationMs) {
      return { shouldContinue: false, reason: "Duration limit exceeded" };
    }

    const repeatedTool = Object.entries(m.toolCallCounts).find(
      ([, count]) => count >= this.config.maxRepeatedToolCalls
    );
    if (repeatedTool) {
      return {
        shouldContinue: false,
        reason: `Repeated tool: ${repeatedTool[0]} called ${repeatedTool[1]} times`,
      };
    }

    return { shouldContinue: true };
  }

  getMetrics(taskId: string): AgentLoopMetrics | undefined {
    return this.metrics.get(taskId);
  }
}

The repeated-tool check catches a common sub-pattern: an agent calling the same search tool or API endpoint in a loop because it is not making progress toward a goal. This usually indicates a poorly specified stopping condition or a tool that is returning ambiguous results. The governor surfaces it as a hard stop rather than letting the loop run to the cost ceiling.

Persist these metrics. Loop termination events are incidents worth reviewing. Aggregate over time to identify which task types or which agent configurations produce the most runaway loops, then fix the root cause.

Cost per Successful Task: The Right Unit Economic

Per-seat pricing for AI features is a business convenience, not an engineering metric. The number that tells you whether your AI feature is economically viable is Cost per Successful Task (CPST).

interface TaskOutcome {
  taskId: string;
  feature: string;
  teamId: string;
  modelTier: ModelTier;
  inputTokens: number;
  outputTokens: number;
  cacheHit: boolean;
  escalations: number;       // times the router escalated to a higher tier
  successful: boolean;        // did the task complete without error or rejection?
  durationMs: number;
  computedCostUsd: number;
}

interface FeatureCostSummary {
  feature: string;
  period: string;
  totalTasks: number;
  successfulTasks: number;
  successRate: number;
  totalCostUsd: number;
  cpst: number;              // Cost per Successful Task
  cacheHitRate: number;
  avgEscalations: number;
  p95DurationMs: number;
}

function computeFeatureCostSummary(
  outcomes: TaskOutcome[],
  period: string
): FeatureCostSummary {
  if (outcomes.length === 0) {
    throw new Error("No outcomes to summarize");
  }

  const feature = outcomes[0].feature;
  const totalTasks = outcomes.length;
  const successfulTasks = outcomes.filter((o) => o.successful).length;
  const successRate = successfulTasks / totalTasks;
  const totalCostUsd = outcomes.reduce((sum, o) => sum + o.computedCostUsd, 0);
  const cpst = successfulTasks > 0 ? totalCostUsd / successfulTasks : Infinity;
  const cacheHits = outcomes.filter((o) => o.cacheHit).length;
  const cacheHitRate = cacheHits / totalTasks;
  const avgEscalations = outcomes.reduce((sum, o) => sum + o.escalations, 0) / totalTasks;

  const durations = outcomes.map((o) => o.durationMs).sort((a, b) => a - b);
  const p95DurationMs = durations[Math.floor(durations.length * 0.95)];

  return {
    feature,
    period,
    totalTasks,
    successfulTasks,
    successRate,
    totalCostUsd,
    cpst,
    cacheHitRate,
    avgEscalations,
    p95DurationMs,
  };
}

CPST is the denominator you compare against the value the task delivers. If a document summarization feature costs $0.03 per successful summary and the user would pay $0.10 per summary in a usage-based model, you have margin. If CPST is $0.15, your routing or caching needs work before you can ship this feature profitably.

Track CPST trending over time. A rising CPST without a rising success rate means cost is growing faster than value. A falling CPST is the evidence that your cost engineering investments are working.

Cost Observability: Per-Feature and Per-Team Attribution

Cost observability requires that every LLM call is tagged before it is made. Tagging after the fact from logs is fragile and always incomplete.

interface LLMCallContext {
  featureId: string;
  teamId: string;
  userId?: string;
  taskId: string;
  sessionId?: string;
  environment: "production" | "staging" | "development";
}

interface LLMCallRecord {
  callId: string;
  context: LLMCallContext;
  modelId: string;
  inputTokens: number;
  outputTokens: number;
  computedCostUsd: number;
  cacheHit: boolean;
  durationMs: number;
  successful: boolean;
  errorCode?: string;
  timestamp: string;
}

async function trackedLLMCall(
  prompt: string,
  context: LLMCallContext,
  modelConfig: ModelConfig,
  callFn: (prompt: string, maxTokens: number) => Promise<{ content: string; inputTokens: number; outputTokens: number }>
): Promise<{ content: string; record: LLMCallRecord }> {
  const callId = crypto.randomUUID();
  const start = Date.now();

  let successful = true;
  let errorCode: string | undefined;
  let inputTokens = 0;
  let outputTokens = 0;
  let content = "";

  try {
    const result = await callFn(prompt, modelConfig.maxOutputTokens);
    content = result.content;
    inputTokens = result.inputTokens;
    outputTokens = result.outputTokens;
  } catch (err: unknown) {
    successful = false;
    errorCode = err instanceof Error ? err.constructor.name : "UnknownError";
    throw err;
  } finally {
    const durationMs = Date.now() - start;
    const computedCostUsd =
      (inputTokens / 1_000_000) * modelConfig.inputCostPerMToken +
      (outputTokens / 1_000_000) * modelConfig.outputCostPerMToken;

    const record: LLMCallRecord = {
      callId,
      context,
      modelId: modelConfig.modelId,
      inputTokens,
      outputTokens,
      computedCostUsd,
      cacheHit: false,
      durationMs,
      successful,
      errorCode,
      timestamp: new Date().toISOString(),
    };

    // Emit to your observability pipeline (Kafka, Postgres, ClickHouse, etc.)
    await emitCallRecord(record);
  }

  return { content, record: { callId, context, modelId: modelConfig.modelId, inputTokens, outputTokens, computedCostUsd: 0, cacheHit: false, durationMs: 0, successful, timestamp: new Date().toISOString() } };
}

Store call records in a columnar store (ClickHouse, BigQuery, or even Postgres with BRIN indexes on the timestamp column) for efficient aggregation queries. The queries you need most often: cost by feature by day, cost by team by week, CPST by feature, cache hit rate by feature.

Wire these into a dashboard accessible to team leads, not just the platform team. Cost accountability works when the teams spending the money can see the numbers.

Organizational Governance: Shadow AI Spend

Shadow AI spend is when engineers or teams call LLM APIs directly using personal or uncentralized API keys, bypassing any routing, caching, budget enforcement, or attribution the platform provides. It is common at every org that ships AI features without a clear internal API.

The structural fix is a single internal LLM gateway that all teams call. The gateway enforces budgets, records attribution, and applies routing and caching centrally. Individual API keys become unnecessary.

interface TeamBudgetConfig {
  teamId: string;
  monthlyBudgetUsd: number;
  alertThresholds: number[];   // fractions: [0.5, 0.75, 0.9]
  hardStop: boolean;           // reject calls when budget exhausted
}

interface BudgetState {
  teamId: string;
  month: string;               // "2026-05"
  spentUsd: number;
  budgetUsd: number;
  utilizationFraction: number;
  alertsSent: number[];
}

async function checkTeamBudget(
  teamId: string,
  estimatedCallCostUsd: number,
  config: TeamBudgetConfig,
  getState: (teamId: string, month: string) => Promise<BudgetState>
): Promise<{ approved: boolean; reason?: string }> {
  const month = new Date().toISOString().slice(0, 7);
  const state = await getState(teamId, month);

  const projectedSpend = state.spentUsd + estimatedCallCostUsd;
  const projectedUtilization = projectedSpend / config.monthlyBudgetUsd;

  if (config.hardStop && projectedUtilization > 1.0) {
    return {
      approved: false,
      reason: `Team ${teamId} monthly budget exhausted: $${state.spentUsd.toFixed(2)} / $${config.monthlyBudgetUsd}`,
    };
  }

  // Fire alerts asynchronously for threshold crossings
  for (const threshold of config.alertThresholds) {
    const currentUtilization = state.spentUsd / config.monthlyBudgetUsd;
    if (currentUtilization < threshold && projectedUtilization >= threshold) {
      void sendBudgetAlert(teamId, threshold, projectedSpend, config.monthlyBudgetUsd);
    }
  }

  return { approved: true };
}

Budget alerts at 50%, 75%, and 90% give teams time to respond before hitting the ceiling. Hard stops should be opt-in by team initially; forcing hard stops organization-wide before teams have visibility into their own spend creates conflict rather than accountability.

Tradeoffs Table

Strategy	Cost Reduction	Implementation Complexity	Latency Impact	Failure Mode
Token budget enforcement	10-40%	Low	None	Truncated context degrades output quality
Tier-based routing	30-70%	Medium	None on fast path, +500-1500ms on escalation	Routing signal accuracy determines savings
Semantic caching	20-60%	Medium	-50 to -200ms on hit, +20-50ms embed overhead on miss	Stale cache returns wrong answers after data changes
Agent loop limits	Prevents unbounded spend	Low	None in normal operation	Hard stops interrupt legitimate long-running tasks
Cascading (try-cheap, escalate)	40-65%	High	+1-3s on escalation path	Escalation latency unacceptable for synchronous UX
Centralized gateway	Overhead reduction via sharing	High	Negligible	Single point of failure if not designed for HA

Production Considerations

Token estimation accuracy. Tiktoken is the reference for OpenAI models. For Anthropic, count tokens via the API’s usage response and calibrate your pre-call estimator against real usage. Most character-based estimators (4 chars per token) are off by 15-25% for code and structured data.

Prefix caching. Both Anthropic and OpenAI offer prompt caching for repeated prefixes. If your system prompt is static and long, cache it. The input cost discount (typically 90%) on the cached portion compounds significantly at scale. Order your prompt: system instructions first (stable), retrieved context second (semi-stable), conversation history last (variable).

Cost attribution lag. Provider invoices come with a lag of up to 48 hours. Do not rely on invoices for real-time budget enforcement. Compute your own cost estimates at call time using the token counts from the usage response and your per-token rate. Accept that your estimates will differ from the invoice by a small amount (usually under 2%).

Model deprecations. Providers deprecate models on 3-6 month cycles. Hardcode model IDs in a central registry rather than scattering them across the codebase. When a deprecation notice arrives, you update one file.

Evaluation on the cheap path. Routing to cheaper models only saves money if the output quality is acceptable for the task. Set up an offline evaluation pipeline that samples cheap-path outputs and scores them against a reference. Run it weekly. If quality degrades, your routing thresholds need adjustment.

The $47K incident is a failure of governance, not a failure of the technology. The models did exactly what they were asked to do. The engineering work is building the layer that asks them sensibly: bounded prompts, tiered routing, semantic caching, loop containment, and measurement at the level of tasks completed rather than tokens consumed. That is what separates AI features that scale economically from ones that become budget incidents.