LLM Prompt Caching in Production: Prefix Caching, KV Cache Reuse, and Cost-Latency Optimization at Scale

There are two very different things that engineers call “caching” when talking about LLMs, and conflating them leads to wrong architecture choices. Semantic caching stores previous responses and retrieves them when a new query is similar enough to an old one. It operates at the application layer, above the model. Prompt caching, also called prefix caching or KV cache reuse, operates at the inference layer, inside the model itself. It skips redundant attention computation over tokens the model has already processed.

The distinction matters because the optimization target is different. Semantic caching reduces API calls entirely. Prompt caching reduces the cost and latency of a single API call. You need to understand which problem you are solving before you pick a technique.

If you want the semantic caching deep dive, that is covered separately. This article is about the inference-layer mechanism: what it is, how each vendor exposes it, how to structure your prompts to benefit from it, and where it breaks down in production.

What the KV Cache Actually Is

Transformers process tokens by computing attention over all previous tokens at each layer. For each token, the model produces key and value tensors that it stores in what is called the KV cache. When generating the next token, the model does not recompute those tensors for tokens it has already seen. It reads them from the KV cache instead.

In a standard inference call, the KV cache builds up during the prefill phase (processing your input) and is then discarded when the response is complete. Every new call starts from scratch. If your prompt has a 10,000-token system prompt that never changes, you pay the full prefill cost every single time.

Prefix caching changes this. If the model inference server has already computed the KV tensors for a prefix of your input, it can reuse them for a new request that starts with the same prefix. The computation cost for those cached tokens drops to near zero. The latency for the prefill phase drops proportionally.

This is not a product feature bolted on top of a model. It is a natural consequence of how attention works. The vendors have built mechanisms to make this explicit and persistent rather than ephemeral.

Vendor Implementations

Anthropic: Explicit Cache Control

Anthropic’s prompt caching requires you to mark which parts of your prompt you want cached using a cache_control marker. This is explicit opt-in. You decide what gets cached and what does not.

Cache points can be placed in system prompts, user messages, and tool definitions. The minimum cacheable block is 1024 tokens for Claude 3 Haiku and 2048 tokens for Claude 3.5 Sonnet and Claude 3 Opus. Blocks smaller than the minimum are not cached even if you mark them. The default TTL is 5 minutes. Extended TTLs of 1 hour are available. Cached tokens on a write (the first call that populates the cache) cost 1.25x the normal input price. Cache hits cost 0.1x the normal input price.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Large, stable document loaded once
const documentContent = await loadDocumentContent(); // assume 8000+ tokens

const response = await client.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: "You are a technical assistant. Answer questions based only on the provided documentation.",
    },
    {
      type: "text",
      text: documentContent,
      // Mark the large document for caching.
      // The model will reuse KV tensors for this block on subsequent calls.
      cache_control: { type: "ephemeral" },
    },
  ],
  messages: [
    {
      role: "user",
      content: userQuestion, // dynamic per request
    },
  ],
});

// The response usage object tells you what was cached
const usage = response.usage;
console.log({
  inputTokens: usage.input_tokens,
  cacheReadTokens: usage.cache_read_input_tokens,   // tokens served from cache
  cacheWriteTokens: usage.cache_creation_input_tokens, // tokens written to cache
});

The usage fields are how you measure cache effectiveness. If cache_read_input_tokens is zero on what should be a cache hit, the prefix changed or the TTL expired.

One subtlety: Anthropic allows up to four cache breakpoints in a single request. If you have a long system prompt followed by a large document followed by conversation history, you can mark each boundary independently. The model caches each prefix up to each marker.

OpenAI: Automatic Prefix Caching

OpenAI’s implementation is automatic. There is no cache_control marker. When a request shares a prefix with a recent request, OpenAI’s infrastructure detects the match and reuses cached KV states. The prefix must be at least 1024 tokens and must align to a 128-token boundary. Cache hits are billed at 50% of the normal input price. There is no write surcharge.

import OpenAI from "openai";

const client = new OpenAI();

const STABLE_SYSTEM_PROMPT = `
You are a senior code reviewer specialized in TypeScript and distributed systems.
Your reviews focus on correctness, performance implications, error handling,
and operational concerns. You cite specific line numbers. You distinguish between
blocking issues and suggestions. You do not comment on formatting.

Review guidelines:
[... several thousand tokens of detailed guidelines ...]
`.trim();

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [
    {
      role: "system",
      content: STABLE_SYSTEM_PROMPT,
    },
    {
      role: "user",
      content: `Please review the following pull request:\n\n${pullRequestDiff}`,
    },
  ],
});

// OpenAI reports cache usage in the same usage object
const usage = response.usage;
if (usage?.prompt_tokens_details) {
  console.log({
    totalPromptTokens: usage.prompt_tokens,
    cachedTokens: usage.prompt_tokens_details.cached_tokens,
    newTokens: usage.prompt_tokens - (usage.prompt_tokens_details.cached_tokens ?? 0),
  });
}

Because OpenAI’s caching is automatic, the main lever you have is keeping your prefixes stable and long enough to cross the 1024-token minimum. Varying content must come after the stable prefix, not before it. A system prompt that includes today’s date, the user’s name, or any per-request content breaks prefix sharing for everyone sharing that deployment.

Gemini: Explicit Context Caching

Google’s Gemini API exposes caching through a dedicated resource called a cached content object. You create the cache explicitly, receive a cache name, and then reference that name in subsequent generation requests. Cached content has a configurable TTL and a minimum size of 32,768 tokens.

The cost model differs from Anthropic and OpenAI. You pay a per-token storage fee per hour for cached content, plus a reduced per-token input fee when you use the cache. For very long documents (hundreds of thousands of tokens) that are accessed infrequently, the storage cost can exceed the inference savings. Calculate before committing.

import { GoogleGenAI } from "@google/genai";

const client = new GoogleGenAI({ apiKey: process.env.GOOGLE_API_KEY });

// Step 1: create the cached content
const cache = await client.caches.create({
  model: "gemini-1.5-pro",
  contents: [
    {
      role: "user",
      parts: [{ text: largeDocumentText }], // must be >= 32768 tokens
    },
  ],
  ttl: "3600s", // 1 hour TTL
  systemInstruction: {
    parts: [{ text: "You are an expert document analyst." }],
  },
});

const cacheName = cache.name; // persist this

// Step 2: use the cached content in generation calls
const result = await client.models.generateContent({
  model: "gemini-1.5-pro",
  cachedContent: cacheName,
  contents: [
    {
      role: "user",
      parts: [{ text: userQuery }],
    },
  ],
});

console.log(result.usageMetadata);
// { promptTokenCount, cachedContentTokenCount, candidatesTokenCount }

Gemini’s explicit model is useful when you want fine-grained control over cache lifecycle and when you need to cache across sessions deterministically. The tradeoff is operational overhead: you manage cache objects as first-class resources, which means TTL tracking, renewal logic, and error handling when a cache name has expired.

Architecture Patterns

Stable Prefix Design

The single most impactful decision is where you draw the boundary between stable and dynamic content. Stable content must come first. Dynamic content must come last. This seems obvious but breaks in practice when teams add personalization to system prompts.

Common mistakes:

• Including the current timestamp in the system prompt
• Embedding the user’s name or account tier into the static instructions
• Prepending per-request metadata before the large document
• Rotating few-shot examples based on request type

The fix is to separate identity from instruction. System prompts describe the model’s behavior and knowledge. Per-request context goes in the user turn.

// Bad: personalization in the system prompt breaks prefix caching for all users
function buildSystemPrompt(user: User): string {
  return `You are an assistant for ${user.name} (tier: ${user.tier}).
Today is ${new Date().toISOString()}.
${LARGE_STABLE_INSTRUCTIONS}`;
}

// Good: stable system prompt, personalization in the user turn
const SYSTEM_PROMPT = `You are a technical assistant.
${LARGE_STABLE_INSTRUCTIONS}`;

function buildUserMessage(user: User, query: string): string {
  return `[Context: user tier is ${user.tier}]\n\n${query}`;
}

Cache Key Composition for Multi-Tenant Apps

In a multi-tenant system, you often have multiple sources of stable content: the base system prompt, a tenant-specific configuration, and sometimes a user-specific document. Each layer should map to its own cache breakpoint.

For Anthropic, this means placing cache markers at each stable boundary. For OpenAI, it means ensuring all stable content is concatenated at the front and the per-request content follows.

interface TenantConfig {
  tenantId: string;
  customInstructions: string;
  knowledgeBase: string; // large, stable per tenant
}

function buildCachedMessages(
  tenant: TenantConfig,
  userQuery: string
): Anthropic.MessageParam[] {
  return [
    {
      role: "user",
      content: [
        {
          type: "text",
          // Tenant-specific knowledge base: stable across all users in the tenant.
          // Cache breakpoint here. Each tenant gets its own cache entry.
          text: tenant.knowledgeBase,
          cache_control: { type: "ephemeral" },
        },
        {
          type: "text",
          // Tenant-specific instructions: stable within the tenant.
          text: tenant.customInstructions,
          cache_control: { type: "ephemeral" },
        },
        {
          type: "text",
          // Dynamic user query: not cached.
          text: userQuery,
        },
      ],
    },
  ];
}

The key insight is that cache reuse is per prefix, not per session. Two different users in the same tenant with the same stable prefix will both benefit from the cache. This is different from session-level context management where you store conversation history per user.

Cache Hit Rate Monitoring

Prompt caching only saves money if the cache is actually being hit. You need to instrument this explicitly. The usage fields in the response tell you everything you need.

interface PromptCacheMetrics {
  requestId: string;
  tenantId: string;
  feature: string;
  model: string;
  inputTokens: number;
  cacheWriteTokens: number;
  cacheReadTokens: number;
  outputTokens: number;
  costUsd: number;
  latencyMs: number;
  timestamp: Date;
}

const ANTHROPIC_PRICING: Record<string, {
  input: number;
  cacheWrite: number;
  cacheRead: number;
  output: number;
}> = {
  "claude-3-5-sonnet-20241022": {
    input: 3.00,       // per 1M tokens
    cacheWrite: 3.75,  // 1.25x input
    cacheRead: 0.30,   // 0.1x input
    output: 15.00,
  },
};

function computeAnthropicCost(
  model: string,
  usage: { input_tokens: number; cache_creation_input_tokens?: number; cache_read_input_tokens?: number; output_tokens: number }
): number {
  const pricing = ANTHROPIC_PRICING[model];
  if (!pricing) return 0;

  return (
    ((usage.input_tokens ?? 0) / 1_000_000) * pricing.input +
    ((usage.cache_creation_input_tokens ?? 0) / 1_000_000) * pricing.cacheWrite +
    ((usage.cache_read_input_tokens ?? 0) / 1_000_000) * pricing.cacheRead +
    ((usage.output_tokens ?? 0) / 1_000_000) * pricing.output
  );
}

function logCacheMetrics(
  response: Anthropic.Message,
  context: { requestId: string; tenantId: string; feature: string; latencyMs: number }
): PromptCacheMetrics {
  const usage = response.usage;
  const metrics: PromptCacheMetrics = {
    requestId: context.requestId,
    tenantId: context.tenantId,
    feature: context.feature,
    model: response.model,
    inputTokens: usage.input_tokens,
    cacheWriteTokens: usage.cache_creation_input_tokens ?? 0,
    cacheReadTokens: usage.cache_read_input_tokens ?? 0,
    outputTokens: usage.output_tokens,
    costUsd: computeAnthropicCost(response.model, usage),
    latencyMs: context.latencyMs,
    timestamp: new Date(),
  };

  // Derive cache hit rate over a rolling window in your metrics pipeline
  const hitRate = metrics.cacheReadTokens / (metrics.inputTokens + metrics.cacheReadTokens + metrics.cacheWriteTokens);
  console.log({ ...metrics, cacheHitRate: hitRate });

  return metrics;
}

Aggregate cacheReadTokens / (inputTokens + cacheReadTokens + cacheWriteTokens) as your cache hit rate per feature. A hit rate below 60% on a feature with a large stable system prompt suggests the prefix is changing on most requests.

Alert when hit rate drops sharply. A sudden drop from 80% to 10% almost always means a code change introduced dynamic content into the stable prefix.

Cost Economics

Prompt caching has a write cost. On Anthropic, the first request that populates the cache pays 1.25x the normal input price for the cached tokens. Every subsequent request within the TTL pays 0.1x. The breakeven is roughly 1.4 requests: if the cache is hit at least twice for every write, you come out ahead on the cached tokens.

For OpenAI, there is no write surcharge. Every request pays normal input pricing for uncached tokens and 50% for cached tokens. Breakeven is immediate from the second call forward.

These economics shift depending on your traffic pattern. A feature with a 20,000-token stable prompt, 200 requests per day, and a 5-minute TTL may see the cache expire between requests during quiet hours. The write cost is paid each time the TTL expires, not just once.

The right question is: what is the expected cache population cost divided by the expected number of hits before expiry? For a 5-minute TTL on a feature processing one request per minute, expect roughly 5 hits per write cycle, well into profitable territory. For a low-traffic internal tool making one request every 30 minutes, the cache may expire before it is ever hit again.

Tradeoffs

Dimension	Prompt Caching	Semantic Caching	Response Caching
What is reused	KV attention tensors	Previous responses	Previous responses
Where it operates	Inference layer (inside model)	Application layer	Application layer
Works with different queries	Yes, prefix must match	No, queries must be similar	No, exact or near-exact match
Reduces API calls	No	Yes	Yes
Reduces latency per call	Yes (prefill speedup)	Yes (skips the call)	Yes (skips the call)
Stale content risk	Low (TTL-bounded)	High (responses can go stale)	High (responses can go stale)
Infrastructure required	None (vendor-managed)	Vector DB + embedding model	Cache store (Redis etc.)
Cost impact	Saves on large stable prefixes	Saves on repeated similar queries	Saves on repeated identical queries

Choose prompt caching when: your prompts contain large stable sections (system prompts, documents, tool definitions) and queries are unique or personalized. Choose semantic caching when: many users ask semantically equivalent questions and response freshness is not critical. Use both when you have large stable prompts AND high query repetition.

Production Pitfalls

Dynamic content in the stable prefix. The most common failure mode. A developer adds Today is ${new Date().toDateString()} to the system prompt for context, and cache hit rates collapse. Audit every string interpolation in your prompt construction path.

TTL expiry during long request chains. In agentic workflows with many tool calls, a 5-minute TTL can expire mid-session. The next LLM call in the chain pays a cache write cost instead of a cache read cost, with no warning. For long-running agents, use the 1-hour TTL (where supported) or structure the workflow to complete within the TTL window.

Minimum token thresholds. Caching is silently skipped for blocks below the minimum. If your system prompt is 1800 tokens and the minimum is 2048, you get no caching and no error. Always verify cache write tokens in the response usage when first deploying a cached feature.

Billing implications of high-volume cache writes. In a fleet of workers all starting cold simultaneously (after a deployment or a TTL expiry), every worker’s first request pays the cache write cost. For a 20,000-token prompt at Anthropic’s pricing, this is non-trivial. Stagger cold start requests or implement a warm-up call from a single worker after deployment.

Assuming cross-region cache sharing. Cache entries are typically not shared across geographic regions. A deployment in us-east-1 and eu-west-1 will each maintain separate caches. Your hit rate calculations should account for traffic split by region.

Tool definition instability. Tool schemas count toward the cached prefix. If your tool registry is dynamic and definitions change between requests (for example, because available tools vary by user permissions), the prefix changes. Consider caching the full tool set and filtering available tools in the system prompt or user message instead.

When Prompt Caching Pays Off

The technique earns its complexity when these conditions hold simultaneously: the stable prefix is large enough to cross the minimum threshold (aim for at least 2x the minimum to leave headroom), request volume is high enough to generate multiple cache hits per TTL window, and the same stable prefix is shared across many requests rather than being unique per user.

The highest-value deployments are document Q&A over large reference corpora, code review assistants with extensive style guides and rules, multi-turn agents where the system prompt and tool definitions are stable across turns, and customer support bots with large knowledge bases baked into the prompt.

For features with small system prompts, highly dynamic prompts, or very low traffic, the operational overhead of managing cache breakpoints and monitoring hit rates rarely justifies itself. Optimize where the tokens are.

Prompt caching is not a magic cost button. It is a specific optimization for a specific access pattern: large, stable, frequently reused prefix content. When that pattern is present, the savings are real and the latency reduction is significant. When it is not, you are paying attention costs for a mechanism that barely fires.