LLM Cost Optimization in Production: Token Budgets, Semantic Caching, and Model Routing

A team ships an LLM feature. The first month the bill is manageable. The second month, usage grows and the bill triples. By month three someone is in a Slack thread asking “why is this so expensive?” and nobody has a good answer. The feature is making the same API calls it always did. That is exactly the problem.

Most production LLM costs are not from model complexity or rare edge cases. They come from re-running identical prompts, sending 8,000-token context windows when 400 tokens would have been sufficient, using GPT-4-class models to answer questions that a 7B model handles correctly, and never measuring any of it. The optimization surface is large and most of it is untouched.

This article walks through the full cost reduction stack. Not theory: working TypeScript implementations you can drop into a production codebase, plus the mental model for deciding which techniques to apply where.

Understanding Where the Money Actually Goes

Before optimizing, instrument. The first step is breaking down costs by feature, user tier, and request type. Most teams are flying blind.

interface CostRecord {
  requestId: string;
  feature: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  costUsd: number;
  cachedHit: boolean;
  durationMs: number;
  timestamp: Date;
}

const MODEL_COSTS: Record<string, { input: number; output: number }> = {
  // Cost per 1M tokens in USD (update as pricing changes)
  "gpt-4o": { input: 2.50, output: 10.00 },
  "gpt-4o-mini": { input: 0.15, output: 0.60 },
  "claude-3-5-sonnet": { input: 3.00, output: 15.00 },
  "claude-3-haiku": { input: 0.25, output: 1.25 },
};

function computeCost(model: string, inputTokens: number, outputTokens: number): number {
  const pricing = MODEL_COSTS[model];
  if (!pricing) return 0;
  return (inputTokens / 1_000_000) * pricing.input
    + (outputTokens / 1_000_000) * pricing.output;
}

async function recordUsage(record: CostRecord, db: Database): Promise<void> {
  await db.insert("llm_usage", record);
}

Run this for two weeks and query the results before touching anything else. You will almost certainly find that 20% of your features account for 80% of the bill, and that a large fraction of requests are either identical or near-identical. That distribution shapes every subsequent decision.

Token Budget Enforcement

The single highest-leverage change most teams can make is capping output tokens aggressively and trimming context before it ever reaches the model.

Output Token Caps

Every LLM API exposes max_tokens. Most teams set it to something conservative like 4096 and leave it there forever. This is wrong. Different features have wildly different output requirements.

A classification request needs 5-20 tokens. A summarization request might need 200-400. A code generation request might legitimately need 1000-2000. Applying the same cap to all three and not checking which is which means your classification calls are billed with room for 4000 tokens of output they will never produce.

interface TokenBudget {
  maxOutputTokens: number;
  maxInputTokens: number;
}

const FEATURE_BUDGETS: Record<string, TokenBudget> = {
  classification: { maxOutputTokens: 20, maxInputTokens: 512 },
  sentiment: { maxOutputTokens: 10, maxInputTokens: 512 },
  summarization: { maxOutputTokens: 400, maxInputTokens: 4000 },
  extraction: { maxOutputTokens: 600, maxInputTokens: 6000 },
  code_generation: { maxOutputTokens: 2000, maxInputTokens: 8000 },
  chat_response: { maxOutputTokens: 800, maxInputTokens: 4000 },
};

function applyBudget(
  feature: string,
  request: ChatRequest
): ChatRequest {
  const budget = FEATURE_BUDGETS[feature] ?? { maxOutputTokens: 1000, maxInputTokens: 4000 };

  return {
    ...request,
    max_tokens: Math.min(request.max_tokens ?? budget.maxOutputTokens, budget.maxOutputTokens),
  };
}

For classification and structured extraction tasks, go further: use JSON mode or function calling, which constrains output to a schema. A JSON object with three string fields will never consume 4000 tokens.

Context Window Trimming

Context is often the bigger cost driver. A chat session that accumulates 20 turns quickly fills 8,000-16,000 input tokens per request. Not all of that context is contributing to quality.

The blunt approach is a sliding window: keep the system prompt and the last N turns. This works surprisingly well for most conversational tasks.

interface Message {
  role: "system" | "user" | "assistant";
  content: string;
  tokenCount?: number;
}

function trimContext(
  messages: Message[],
  maxInputTokens: number,
  estimateTokens: (text: string) => number
): Message[] {
  const systemMessages = messages.filter((m) => m.role === "system");
  const conversationMessages = messages.filter((m) => m.role !== "system");

  const systemTokens = systemMessages.reduce(
    (sum, m) => sum + (m.tokenCount ?? estimateTokens(m.content)),
    0
  );

  let budget = maxInputTokens - systemTokens - 200; // Reserve 200 for message overhead
  const trimmed: Message[] = [];

  // Walk backwards: keep most recent messages first
  for (let i = conversationMessages.length - 1; i >= 0; i--) {
    const msg = conversationMessages[i];
    const tokens = msg.tokenCount ?? estimateTokens(msg.content);
    if (tokens > budget) break;
    trimmed.unshift(msg);
    budget -= tokens;
  }

  return [...systemMessages, ...trimmed];
}

// Fast token estimation without a full tokenizer
function estimateTokens(text: string): number {
  // ~4 chars per token for English, conservative estimate
  return Math.ceil(text.length / 3.5);
}

A more sophisticated approach: summarize old turns into a compressed context block. This preserves more information at the cost of an extra (cheap) model call to generate the summary.

Semantic Caching

Exact string matching catches very little in practice. “What is the capital of France?” and “What’s France’s capital?” are semantically identical but byte-for-byte different. Semantic caching matches requests by meaning using embeddings, not string equality.

The architecture is straightforward: when a request arrives, embed the query, search a vector store for similar past queries, and return the cached response if similarity is above a threshold. Otherwise run the LLM call and store the result.

import { OpenAI } from "openai";

interface CacheEntry {
  id: string;
  embedding: number[];
  query: string;
  response: string;
  model: string;
  feature: string;
  createdAt: Date;
  hitCount: number;
}

class SemanticCache {
  private openai: OpenAI;
  private vectorStore: VectorStore; // Your pgvector, Pinecone, etc.
  private similarityThreshold: number;

  constructor(options: {
    openai: OpenAI;
    vectorStore: VectorStore;
    similarityThreshold?: number;
  }) {
    this.openai = options.openai;
    this.vectorStore = options.vectorStore;
    this.similarityThreshold = options.similarityThreshold ?? 0.95;
  }

  async get(query: string, feature: string): Promise<string | null> {
    const embedding = await this.embed(query);

    const results = await this.vectorStore.similaritySearch({
      embedding,
      filter: { feature },
      limit: 1,
    });

    if (results.length === 0) return null;

    const [best] = results;
    if (best.similarity < this.similarityThreshold) return null;

    // Update hit count for analytics
    await this.vectorStore.increment(best.id, "hitCount");
    return best.response;
  }

  async set(query: string, response: string, feature: string): Promise<void> {
    const embedding = await this.embed(query);
    await this.vectorStore.upsert({
      embedding,
      query,
      response,
      feature,
      createdAt: new Date(),
      hitCount: 0,
    });
  }

  private async embed(text: string): Promise<number[]> {
    const result = await this.openai.embeddings.create({
      model: "text-embedding-3-small",
      input: text,
    });
    return result.data[0].embedding;
  }
}

The threshold is the critical parameter. At 0.95, you cache aggressively but risk returning slightly wrong answers for queries that are close but not equivalent. At 0.99, you miss most cache opportunities. The right number depends on your feature: classification prompts can tolerate 0.93; factual Q&A needs 0.97 or higher.

Cache invalidation matters too. LLM outputs can become stale. Set a TTL appropriate to the domain: general knowledge questions might have a 30-day TTL, while questions about current events need much shorter windows or no caching at all.

One practical consideration: embedding calls are not free, but they are cheap. text-embedding-3-small costs $0.02 per million tokens. Compared to a GPT-4o call at $2.50 per million input tokens, the embedding cost is negligible even if your cache hit rate is modest.

For many production workloads, semantic caching alone reduces LLM costs by 30-60%. Repeated questions from multiple users, similar support tickets, recurring analysis tasks: these patterns generate enormous cache hit rates once the cache warms up.

Intelligent Model Routing

Not all requests need the same model. Using GPT-4o for a sentiment classification that GPT-4o-mini handles with equal accuracy costs 15x more per input token and 16x more per output token. Over millions of requests, that difference is the entire infrastructure budget.

Model routing works by classifying each request and dispatching it to the appropriate model.

type ModelTier = "nano" | "fast" | "smart" | "frontier";

interface RoutingRule {
  tier: ModelTier;
  model: string;
  conditions: RoutingCondition[];
}

interface RoutingCondition {
  type: "feature" | "input_length" | "complexity_score";
  operator: "eq" | "lt" | "gt" | "in";
  value: string | number | string[];
}

const ROUTING_TABLE: RoutingRule[] = [
  {
    tier: "nano",
    model: "gpt-4o-mini",
    conditions: [{ type: "feature", operator: "in", value: ["classification", "sentiment", "yes_no"] }],
  },
  {
    tier: "fast",
    model: "gpt-4o-mini",
    conditions: [{ type: "input_length", operator: "lt", value: 1000 }],
  },
  {
    tier: "smart",
    model: "gpt-4o",
    conditions: [{ type: "feature", operator: "in", value: ["code_generation", "complex_reasoning"] }],
  },
  {
    tier: "frontier",
    model: "claude-3-5-sonnet",
    conditions: [{ type: "complexity_score", operator: "gt", value: 0.8 }],
  },
];

function routeRequest(
  feature: string,
  inputText: string,
  complexityScore?: number
): string {
  const inputLength = inputText.length;

  for (const rule of ROUTING_TABLE) {
    const allMatch = rule.conditions.every((condition) => {
      switch (condition.type) {
        case "feature":
          return condition.operator === "in"
            ? (condition.value as string[]).includes(feature)
            : feature === condition.value;
        case "input_length":
          return condition.operator === "lt"
            ? inputLength < (condition.value as number)
            : inputLength > (condition.value as number);
        case "complexity_score":
          if (complexityScore === undefined) return false;
          return condition.operator === "gt"
            ? complexityScore > (condition.value as number)
            : complexityScore < (condition.value as number);
        default:
          return false;
      }
    });

    if (allMatch) return rule.model;
  }

  // Default to fast model
  return "gpt-4o-mini";
}

For complexity scoring, you have two options. Simple heuristics (question length, presence of code, technical vocabulary) work well enough for many routing decisions without adding an extra model call. For higher stakes routing, a dedicated classifier: a small model that predicts whether the request needs frontier-tier capability.

Routing signal	Cost	Accuracy	Best for
Feature name	Free	High (if features are well-defined)	Most production cases
Input length	Free	Medium	General fallback
Keyword heuristics	Free	Medium-low	Coarse classification
Small classifier model	Low ($0.001 per call)	High	Mixed workloads
LLM self-report	Medium	Variable	Experimental

Measure routing accuracy after deploying. The right metric is not “did GPT-4o-mini answer the question” but “did the answer quality differ from what GPT-4o would have produced?” A/B test a random 1-5% of nano-routed requests through the frontier model and compare outputs. This gives you a continuous quality signal.

Prompt Compression

System prompts and few-shot examples often contain more text than necessary. A system prompt that started as “let me add some context” and grew through six iterations can easily reach 2,000 tokens of instructions that could be expressed in 400.

Manual compression is the most effective: read every sentence in your system prompt and ask whether removing it changes the model’s behavior. In most cases, 30-40% can be cut without quality impact.

For dynamic context, consider a compression pre-pass using a cheap model:

async function compressContext(
  rawContext: string,
  targetTokens: number,
  openai: OpenAI
): Promise<string> {
  const currentEstimate = Math.ceil(rawContext.length / 3.5);
  if (currentEstimate <= targetTokens) return rawContext;

  const ratio = targetTokens / currentEstimate;

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini", // Use cheap model for compression
    messages: [
      {
        role: "system",
        content: `Compress the following text to approximately ${Math.round(ratio * 100)}% of its length. Preserve all factual content and key details. Remove redundancy and verbose phrasing. Output only the compressed text.`,
      },
      { role: "user", content: rawContext },
    ],
    max_tokens: targetTokens + 100,
  });

  return response.choices[0].message.content ?? rawContext;
}

This is most useful for document-grounded tasks: when you retrieve chunks from a vector store for a RAG pipeline and the combined context exceeds your token budget. Compressing the retrieved context with a cheap model before passing it to an expensive model can reduce costs significantly.

There is also a class of approaches called prompt compression or LLMLingua that use a small language model to identify and remove tokens unlikely to affect output quality. These are more aggressive and require evaluation on your specific task, but can achieve 3-5x compression with limited quality loss on many tasks.

Batch Processing vs Real-Time

Not all LLM work needs to happen synchronously. Report generation, content moderation at scale, bulk extraction, nightly analysis: these tasks can run asynchronously, which opens up batch processing APIs.

The OpenAI Batch API processes requests asynchronously with a 24-hour turnaround and charges 50% of the standard rate. For workloads where latency tolerance is measured in hours rather than seconds, this is one of the simplest cost reductions available.

import fs from "fs/promises";
import { OpenAI } from "openai";

interface BatchRequest {
  custom_id: string;
  method: "POST";
  url: "/v1/chat/completions";
  body: {
    model: string;
    messages: Array<{ role: string; content: string }>;
    max_tokens: number;
  };
}

async function submitBatch(
  requests: BatchRequest[],
  openai: OpenAI
): Promise<string> {
  // Write requests as JSONL
  const jsonl = requests.map((r) => JSON.stringify(r)).join("\n");
  await fs.writeFile("/tmp/batch_input.jsonl", jsonl);

  const file = await openai.files.create({
    file: await fs.readFile("/tmp/batch_input.jsonl"),
    purpose: "batch",
  });

  const batch = await openai.batches.create({
    input_file_id: file.id,
    endpoint: "/v1/chat/completions",
    completion_window: "24h",
  });

  return batch.id;
}

async function pollBatch(
  batchId: string,
  openai: OpenAI
): Promise<BatchResult[]> {
  const batch = await openai.batches.retrieve(batchId);

  if (batch.status !== "completed") {
    throw new Error(`Batch not ready: ${batch.status}`);
  }

  const outputFile = await openai.files.content(batch.output_file_id!);
  const lines = (await outputFile.text()).trim().split("\n");

  return lines.map((line) => JSON.parse(line));
}

The decision between real-time and batch comes down to latency tolerance:

Workload type	Latency tolerance	Use batch?	Savings
Chat response	< 2 seconds	No	0%
Document analysis	Minutes	Depends	50%
Nightly report generation	Hours	Yes	50%
Bulk content moderation	Hours	Yes	50%
A/B test evaluation	Days	Yes	50%
Training data generation	Days	Yes	50%

Queue batch requests behind a job system. Each job records the batch ID, polls on a schedule (or uses webhooks if the provider supports them), stores results, and notifies downstream consumers.

Building a Cost Model

Cost optimization without measurement is guesswork. Build a spreadsheet model (or equivalent database view) before committing to any optimization work.

The variables that matter:

Monthly cost = Sum over all features of:
  requests_per_month
  * (1 - cache_hit_rate)
  * average_input_tokens
  * model_input_cost_per_token
  + requests_per_month
  * (1 - cache_hit_rate)
  * average_output_tokens
  * model_output_cost_per_token

For each feature, plug in:

• Current numbers (no optimization)
• Post-routing numbers (if you route to a cheaper model)
• Post-caching numbers (if you add semantic cache)
• Post-trimming numbers (if you cap context)
• Combined scenario

This model gives you a priority ordering. Features with high request volume and low cache hit potential (every request is unique) benefit most from model routing. Features with repetitive queries benefit most from caching. Features with inflated contexts benefit most from trimming.

A simple projection: if your monthly LLM bill is $5,000, and 40% of requests are near-duplicate (semantic cache candidates at 50% hit rate), and 60% of your requests could use a 10x cheaper model with no quality loss, your theoretical optimized cost is roughly:

Original: $5,000
After caching (40% * 50% reduction): $5,000 * 0.80 = $4,000
After routing (60% of remaining at 1/10 cost): further 54% reduction
Combined floor: ~$1,800/month

Real-world results land somewhere between the floor and current spend, depending on implementation quality. A 50-70% reduction is common for teams starting from zero optimization.

Production Considerations

Measure quality at every routing decision. A cost reduction that introduces quality regressions is not a win. Log which model handled each request, sample outputs, and run evals on the sample. Set up alerts if quality metrics drop after a routing change.

Cache warming matters. A cold semantic cache has zero hit rate. For high-value features, pre-populate the cache with the most common queries you can identify from historical logs.

Budget alerts before cost alerts. Set up spend alerts at 50% and 80% of your monthly budget, not just at the limit. By the time you hit the limit, you have already spent too much.

Token counting is not free. The OpenAI tokenizer (tiktoken) runs locally and is fast, but adding tokenization to every request path adds latency. Use the estimateTokens approximation for routing decisions and reserve exact counting for billing records.

Prompt caching at the provider level. Anthropic and OpenAI both offer prompt caching for long system prompts: if the same prefix appears in many requests, the provider caches the KV state and charges less for the cached portion. Structure your prompts so the static system content comes first. This is free and requires only prompt restructuring.

LLM cost reduction is mostly an engineering problem, not a product problem. The token budgets, routing rules, and cache thresholds are parameters you control. Most teams set them once and forget them. The teams with efficient LLM economics treat these parameters as first-class metrics and tune them on a regular cadence.