Building a Semantic Cache for LLM Applications: Embedding Similarity, Eviction Policies, and Cost Reduction at Scale

Exact-match string caching for LLM calls is nearly useless in practice. “List three ways to improve API performance” and “What are 3 techniques for making APIs faster?” generate different bytes but identical semantic intent. A Redis GET keyed on the raw prompt misses this entirely. You spend $0.25 per thousand tokens re-running prompts your system already answered.

The failure mode is predictable. Teams add a cache, see a 2-3% hit rate, conclude that “LLM calls are too varied to cache,” and move on. The correct conclusion is that exact-match caching is the wrong tool. Semantic caching, backed by embedding similarity, routinely achieves 30-60% hit rates on real workloads: customer support queries, FAQ lookups, repeated analytical questions, document summaries. The architecture is straightforward, but the implementation details matter a lot.

This article builds a complete semantic cache: the lookup architecture, threshold calibration, key design decisions, eviction policies, cache warming, and the production metrics that tell you whether it is working.

Why Exact-Match Caching Fails for Natural Language

Consider a customer support chatbot handling “How do I reset my password?” across thousands of users. The phrasing varies: “forgot password,” “can’t log in, need to reset password,” “password reset instructions,” “steps to change my password.” Each is a cache miss under exact-match lookup. Each triggers a full LLM call.

The problem is that LLM inputs are natural language and natural language has high paraphrase density. Two queries can share zero tokens but have cosine similarity above 0.95 in embedding space. Exact-match caching treats them as unrelated. Semantic caching treats them as the same question.

The second reason exact-match fails: prompts include dynamic context. A support prompt might include a username, session ID, or timestamp. Exact-match caching requires you to strip this dynamic context before keying, which is fragile. Semantic caching operates on the semantically meaningful portion of the prompt by design.

Architecture Overview

The cache sits in front of your LLM provider. Every request passes through it. On lookup: embed the query, search for similar entries, return a cached response if similarity exceeds your threshold. On miss: call the LLM, store the result with its embedding, return the response. This is a read-through cache.

interface SemanticCacheEntry {
  id: string;
  queryText: string;
  queryEmbedding: number[];
  responseText: string;
  model: string;
  feature: string;
  createdAt: Date;
  lastAccessedAt: Date;
  hitCount: number;
  ttlSeconds: number;
}

interface CacheLookupResult {
  hit: boolean;
  entry: SemanticCacheEntry | null;
  similarity: number | null;
  latencyMs: number;
}

The key design choice here is feature. You do not want to mix cache entries across different LLM features. A “password reset” response cached from your support chatbot should not surface in your code documentation lookup, even if the queries are superficially similar. The feature field scopes every cache operation.

Embedding-Based Similarity Lookup

The lookup pipeline has three steps: embed the incoming query, run a nearest-neighbor search over cached embeddings filtered by feature, and evaluate whether the closest match clears the similarity threshold.

import { OpenAI } from "openai";
import { createClient, RedisClientType } from "redis";

interface VectorSearchResult {
  entry: SemanticCacheEntry;
  similarity: number;
}

class SemanticCache {
  private openai: OpenAI;
  private redis: RedisClientType;
  private vectorStore: VectorStore;
  private thresholds: Record<string, number>;
  private defaultThreshold: number;

  constructor(config: {
    openai: OpenAI;
    redis: RedisClientType;
    vectorStore: VectorStore;
    thresholds?: Record<string, number>;
    defaultThreshold?: number;
  }) {
    this.openai = config.openai;
    this.redis = config.redis;
    this.vectorStore = config.vectorStore;
    this.thresholds = config.thresholds ?? {};
    this.defaultThreshold = config.defaultThreshold ?? 0.95;
  }

  async lookup(query: string, feature: string): Promise<CacheLookupResult> {
    const start = Date.now();

    // Check Redis for exact-match first (zero embedding cost)
    const exactKey = `exact:${feature}:${hashQuery(query)}`;
    const exactHit = await this.redis.get(exactKey);
    if (exactHit) {
      const entry: SemanticCacheEntry = JSON.parse(exactHit);
      await this.recordAccess(entry.id);
      return { hit: true, entry, similarity: 1.0, latencyMs: Date.now() - start };
    }

    // Semantic lookup
    const embedding = await this.embed(query);
    const threshold = this.thresholds[feature] ?? this.defaultThreshold;

    const results = await this.vectorStore.search({
      embedding,
      filter: { feature },
      limit: 3, // Fetch top-3, evaluate in order
    });

    const best = results.find((r) => r.similarity >= threshold);
    if (!best) {
      return { hit: false, entry: null, similarity: results[0]?.similarity ?? null, latencyMs: Date.now() - start };
    }

    await this.recordAccess(best.entry.id);
    return { hit: true, entry: best.entry, similarity: best.similarity, latencyMs: Date.now() - start };
  }

  async store(
    query: string,
    response: string,
    feature: string,
    model: string,
    ttlSeconds: number
  ): Promise<void> {
    const embedding = await this.embed(query);
    const id = crypto.randomUUID();
    const now = new Date();

    const entry: SemanticCacheEntry = {
      id,
      queryText: query,
      queryEmbedding: embedding,
      responseText: response,
      model,
      feature,
      createdAt: now,
      lastAccessedAt: now,
      hitCount: 0,
      ttlSeconds,
    };

    // Store in vector store for similarity search
    await this.vectorStore.upsert(entry);

    // Also store in Redis for exact-match fast path and TTL enforcement
    const exactKey = `exact:${feature}:${hashQuery(query)}`;
    await this.redis.set(exactKey, JSON.stringify(entry), { EX: ttlSeconds });
  }

  private async embed(text: string): Promise<number[]> {
    const result = await this.openai.embeddings.create({
      model: "text-embedding-3-small",
      input: text,
    });
    return result.data[0].embedding;
  }

  private async recordAccess(entryId: string): Promise<void> {
    await this.vectorStore.update(entryId, {
      lastAccessedAt: new Date(),
      hitCount: { increment: 1 },
    });
  }
}

function hashQuery(query: string): string {
  // Normalize before hashing: lowercase, collapse whitespace
  const normalized = query.toLowerCase().trim().replace(/\s+/g, " ");
  return Buffer.from(normalized).toString("base64url").slice(0, 32);
}

The exact-match fast path (Redis first, before embedding) is worth the extra complexity. Embedding a query costs time and money. For repeated identical queries (which happen more than you expect in production), skipping the embedding call entirely is the right move. You get the cache hit at essentially zero cost.

Cache Key Design and Similarity Thresholds

The query you embed should not be the raw prompt text. Raw prompts often contain dynamic context: user IDs, session tokens, timestamps. Caching on these values eliminates hits. You need a canonical form: the parts of the query that determine what the ideal response is.

interface CacheKeyConfig {
  includeSystemPrompt: boolean;
  dynamicFieldPatterns: RegExp[];
  maxQueryLength: number;
}

function buildCacheKey(
  userQuery: string,
  systemPrompt: string | null,
  config: CacheKeyConfig
): string {
  let normalized = userQuery;

  // Strip dynamic fields: UUIDs, timestamps, session tokens
  for (const pattern of config.dynamicFieldPatterns) {
    normalized = normalized.replace(pattern, "[REDACTED]");
  }

  // Truncate to keep embedding focused on semantics, not boilerplate
  normalized = normalized.slice(0, config.maxQueryLength).trim();

  if (config.includeSystemPrompt && systemPrompt) {
    // Include a short hash of the system prompt to scope cache by prompt version
    const promptHash = simpleHash(systemPrompt).slice(0, 8);
    return `${promptHash}:${normalized}`;
  }

  return normalized;
}

const DEFAULT_KEY_CONFIG: CacheKeyConfig = {
  includeSystemPrompt: true,
  dynamicFieldPatterns: [
    /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,
    /\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/g,
    /Bearer\s+[A-Za-z0-9._-]+/g,
  ],
  maxQueryLength: 512,
};

Including a hash of the system prompt in the cache key is important. If you update your prompt and change how the model interprets queries, the old cache entries will return stale responses under the new prompt. The hash ensures old entries become unreachable when the prompt changes.

Calibrating Similarity Thresholds

The threshold controls the tradeoff between hit rate and response accuracy. Too low: cache hits return wrong answers. Too high: hit rate collapses and you are back to exact-match behavior.

How to calibrate:

1. Collect 500-1000 real queries for each feature from production logs.
2. For each pair of queries with similarity above 0.90, manually label whether they are semantically equivalent (would the same response satisfy both?).
3. Find the threshold where precision (fraction of hits that are correct) exceeds your quality target.

In practice, thresholds differ by feature:

Feature	Recommended threshold	Rationale
FAQ / support lookup	0.92-0.94	Questions about the same topic vary in phrasing but rarely change the correct answer
Factual Q&A	0.96-0.98	Small query differences can change the correct answer substantially
Code generation	0.98+	Nearly identical prompts can require different code; risk of wrong cache hit is high
Classification / tagging	0.90-0.93	The query content matters less than the category; paraphrases yield identical labels
Document summarization	0.99 or disable	The document content IS the query; cache is unlikely to help here

Do not set a single global threshold. The right threshold is feature-specific.

TTL and Eviction Strategies

TTL determines how long a cache entry remains valid regardless of access patterns. Eviction determines which entries get removed when the cache reaches capacity. Both need deliberate design.

TTL by Content Type

Not all cached responses have the same shelf life. A cached answer to “what are HTTP status codes?” is valid for months. A cached answer to “what are the current rates for plan X?” might be wrong by tomorrow.

const FEATURE_TTL: Record<string, number> = {
  // Long-lived: factual, slow-changing
  general_knowledge: 30 * 24 * 3600,   // 30 days
  documentation_lookup: 7 * 24 * 3600, // 7 days
  code_explanation: 14 * 24 * 3600,    // 14 days

  // Medium-lived: business logic that changes with releases
  feature_support: 24 * 3600,          // 24 hours
  pricing_lookup: 6 * 3600,            // 6 hours

  // Short-lived: depends on frequently-changing state
  account_status: 300,                  // 5 minutes
  live_inventory: 60,                   // 1 minute

  // Never cache: personalized, stateful, or real-time
  // recommendation: 0,
  // financial_projection: 0,
};

The TTL enforcement split between Redis and the vector store matters here. Redis handles TTL natively with EX. The vector store does not. You need a background job to purge expired vector store entries:

async function evictExpiredEntries(
  vectorStore: VectorStore,
  batchSize = 100
): Promise<number> {
  const now = new Date();
  let evicted = 0;
  let cursor: string | null = null;

  do {
    const { entries, nextCursor } = await vectorStore.scan({
      cursor,
      limit: batchSize,
    });

    const expired = entries.filter((entry) => {
      const expiresAt = new Date(
        entry.createdAt.getTime() + entry.ttlSeconds * 1000
      );
      return expiresAt < now;
    });

    if (expired.length > 0) {
      await vectorStore.deleteMany(expired.map((e) => e.id));
      evicted += expired.length;
    }

    cursor = nextCursor;
  } while (cursor);

  return evicted;
}

Run this on a schedule (every hour is fine for most use cases). For pgvector, this can be a simple SQL delete with a generated expires_at column indexed for performance.

Eviction When at Capacity: LRU vs Semantic Clustering

When the cache fills up beyond your memory or storage budget, you need an eviction policy. Two approaches are worth comparing: LRU (Least Recently Used) and semantic clustering with score-based eviction.

LRU is simple: evict whichever entry was accessed least recently. This is the standard for general caches and works well when access patterns are unpredictable.

Semantic clustering is more nuanced. Group entries by similarity: entries that are near-duplicates cluster together. When evicting, remove entire clusters that have collectively low hit rates rather than individual entries with low recency. This preserves coverage across the semantic space while pruning redundancy.

interface EvictionCandidate {
  clusterId: string;
  entryIds: string[];
  totalHits: number;
  avgLastAccessedAge: number; // seconds since last access
  clusterSize: number;
}

async function computeEvictionCandidates(
  vectorStore: VectorStore,
  clusterSimilarityThreshold = 0.97
): Promise<EvictionCandidate[]> {
  const all = await vectorStore.getAll({ feature: undefined });
  const now = Date.now();

  // Group near-duplicates into clusters
  const visited = new Set<string>();
  const clusters: EvictionCandidate[] = [];

  for (const entry of all) {
    if (visited.has(entry.id)) continue;

    const neighbors = await vectorStore.search({
      embedding: entry.queryEmbedding,
      filter: { feature: entry.feature },
      limit: 20,
    });

    const clusterEntries = neighbors
      .filter((n) => n.similarity >= clusterSimilarityThreshold)
      .map((n) => n.entry);

    const clusterIds = clusterEntries.map((e) => e.id);
    clusterIds.forEach((id) => visited.add(id));

    clusters.push({
      clusterId: entry.id, // Use first entry as cluster representative
      entryIds: clusterIds,
      totalHits: clusterEntries.reduce((sum, e) => sum + e.hitCount, 0),
      avgLastAccessedAge:
        clusterEntries.reduce(
          (sum, e) => sum + (now - e.lastAccessedAt.getTime()) / 1000,
          0
        ) / clusterEntries.length,
      clusterSize: clusterEntries.length,
    });
  }

  // Sort by eviction priority: low hits, high age, large redundant clusters
  return clusters.sort((a, b) => {
    const scoreA = a.totalHits / (a.avgLastAccessedAge + 1) * (1 / a.clusterSize);
    const scoreB = b.totalHits / (b.avgLastAccessedAge + 1) * (1 / b.clusterSize);
    return scoreA - scoreB; // Lower score = evict first
  });
}

In practice, LRU is the right starting point. Implement it in Redis natively (maxmemory-policy allkeys-lru) and replicate the eviction signals to your vector store. Semantic clustering is worth adding if you find your cache filling with many near-duplicate entries from bulk query patterns, where LRU would evict useful diverse entries while keeping redundant ones.

Caching Approaches Compared

Approach	Hit rate	Complexity	Risk of wrong hits	Best for
Exact-match (raw prompt)	1-5%	Low	Zero	Deterministic, templated prompts
Exact-match (normalized)	5-15%	Low	Zero	High-volume repetitive queries
Semantic cache (per-feature, high threshold 0.97+)	20-40%	Medium	Low	Mixed workloads with quality requirements
Semantic cache (per-feature, moderate threshold 0.93-0.96)	40-60%	Medium	Medium	Support, FAQ, classification
Semantic cache (global, no feature scoping)	50-70%	Medium	High	Never: cross-feature pollution
Cluster-aware eviction + semantic cache	40-65%	High	Low-medium	High cache memory pressure
Provider-side prompt caching (Anthropic/OpenAI)	Variable	Low	Zero	Long static system prompts

Provider-side prompt caching deserves a mention here. Both Anthropic and OpenAI cache the KV state of long, repeated system prompt prefixes. This is free, requires only prompt restructuring (static content first), and has zero risk of wrong responses since it only caches prompt processing, not outputs. It does not replace semantic caching but stacks with it.

Cache Warming

A cold semantic cache has zero hit rate. Deployments that bypass cache warming often observe poor hit rates for days until the cache organically fills. There is a better approach.

interface WarmingEntry {
  query: string;
  feature: string;
  expectedResponse?: string; // Pre-written if available
}

async function warmCache(
  entries: WarmingEntry[],
  cache: SemanticCache,
  llmClient: LLMClient,
  concurrency = 5
): Promise<{ warmed: number; skipped: number; errors: number }> {
  const results = { warmed: 0, skipped: 0, errors: 0 };
  const queue = [...entries];

  async function processOne(entry: WarmingEntry): Promise<void> {
    // Skip if already cached
    const existing = await cache.lookup(entry.query, entry.feature);
    if (existing.hit) {
      results.skipped++;
      return;
    }

    try {
      const response =
        entry.expectedResponse ??
        (await llmClient.complete(entry.query, entry.feature));

      const ttl = FEATURE_TTL[entry.feature] ?? 24 * 3600;
      await cache.store(entry.query, response, entry.feature, "gpt-4o-mini", ttl);
      results.warmed++;
    } catch {
      results.errors++;
    }
  }

  // Process in controlled concurrency batches
  while (queue.length > 0) {
    const batch = queue.splice(0, concurrency);
    await Promise.all(batch.map(processOne));
  }

  return results;
}

Where do the warming queries come from? Three sources:

1. Historical logs: the top 200-500 queries by frequency from the past 30 days, extracted from your LLM usage table. These are the highest-probability future hits.
2. Curated FAQ lists: for customer support or documentation use cases, you often have a canonical list of questions. Pre-generate answers for all of them.
3. Synthetic paraphrases: for each high-priority query, generate 5-10 paraphrases using a cheap model and store them as separate cache entries pointing to the same canonical response. This increases surface area for similarity matches.

The synthetic paraphrase approach requires care. You are storing multiple embeddings that all return the same response. This increases vector store size but meaningfully improves hit rate for queries that arrive with phrasing variations you did not anticipate.

Measuring Hit Rates and Cost Savings in Production

A semantic cache without measurement is a black box. Track the following:

interface CacheMetrics {
  feature: string;
  windowStart: Date;
  windowEnd: Date;
  totalRequests: number;
  exactHits: number;       // Redis exact-match hits
  semanticHits: number;    // Vector similarity hits
  misses: number;
  avgSimilarityOnHit: number;
  avgSimilarityOnMiss: number; // How close misses were to threshold
  embeddingCallsAvoided: number;
  estimatedCostSavedUsd: number;
}

function computeCostSavings(
  hits: number,
  avgInputTokens: number,
  avgOutputTokens: number,
  model: string,
  modelCosts: Record<string, { input: number; output: number }>
): number {
  const pricing = modelCosts[model];
  if (!pricing) return 0;

  const costPerRequest =
    (avgInputTokens / 1_000_000) * pricing.input +
    (avgOutputTokens / 1_000_000) * pricing.output;

  return hits * costPerRequest;
}

The avgSimilarityOnMiss metric is underappreciated. If your misses consistently cluster at 0.91-0.94 similarity and your threshold is 0.95, you are leaving significant cache potential on the table. Lower your threshold (and run quality evals to validate it is safe) to capture those near-hits.

Conversely, if you are seeing wrong-answer complaints correlating with cache hits, look at the distribution of similarity scores on hits. If you have many hits at 0.95-0.96, your threshold is too low for that feature.

Expected production outcomes on workloads with natural query repetition (support, FAQ, documentation):

• After cache warm (first week): 15-25% hit rate
• After one month: 35-55% hit rate
• Steady state: 40-65% hit rate
• Cost reduction: proportional to hit rate, minus embedding call cost (typically negligible)

Production Considerations

Embedding model consistency. The embedding model is baked into every stored vector. Changing embedding models (say, from text-embedding-ada-002 to text-embedding-3-small) invalidates your entire cache. You cannot compare vectors from different models. When you upgrade the embedding model, treat it as a cache invalidation event: flush all entries and re-warm.

Stale responses for time-sensitive features. The cache does not know that a policy changed, a price updated, or a feature shipped. For any feature where correctness depends on current state, set conservative TTLs and add a cache bypass mechanism for forced refreshes. A query parameter or header that bypasses the cache is useful for testing and for support teams who need authoritative current answers.

Multi-tenant data isolation. If your application serves multiple customers, semantic cache entries from one customer must not surface for another. Scope your feature key to include tenant ID, or use separate vector store namespaces per tenant. Cross-tenant cache hits are a data leakage risk.

Latency budget. Embedding a query takes 50-150ms on text-embedding-3-small. Vector similarity search over a modest collection (under 100K entries) with pgvector takes 5-20ms. Total cache lookup adds roughly 60-200ms to request latency. For interactive features with strict latency SLAs, measure this overhead. For most use cases it is acceptable because a cache hit avoids a 500-2000ms LLM call. For latency-critical paths, run the cache lookup in parallel with the LLM call and cancel the LLM request if the cache responds first.

Monitoring cache health in production. Three alerts worth setting:

• Hit rate drops below baseline by more than 15 percentage points (suggests embedding model issue or cache corruption)
• Average similarity on hits drops below threshold (suggests vector store drift or index corruption)
• Eviction rate spikes (suggests cache is undersized for traffic, leading to thrashing)

Building a semantic cache is one of the highest-leverage LLM cost reductions available because the return scales with traffic. The first thousand requests of the day prime the cache. The next ten thousand benefit from it. The engineering investment is bounded; the savings are not.