RAG Pipelines in Production: Chunking Strategies, Retrieval Tuning, and the Failure Modes Nobody Talks About

Most RAG tutorials get you to a working prototype in an afternoon. You pick a chunking size, embed your documents, store the vectors, query with cosine similarity, and the demo looks convincing. Then you put it in front of real users with a real corpus, and things break in ways the tutorial never mentioned.

This article is about what happens after the demo works. The chunking decisions that hurt retrieval quality at scale, the HNSW configuration that nobody explains, the failure modes that only appear when your index has a million documents and your embedding model gets upgraded.

Chunking Strategies and Their Actual Tradeoffs

Chunking is treated as a preprocessing detail, but it is the single biggest lever on retrieval quality. The choice compounds: a bad chunk boundary means a bad embedding, which means a bad retrieval, which means a bad generation result. You cannot fix it downstream.

Fixed-Size Chunking

The default everywhere because it is trivial to implement:

interface Chunk {
  id: string;
  text: string;
  metadata: Record<string, unknown>;
  sourceId: string;
  chunkIndex: number;
}

function fixedSizeChunk(
  text: string,
  chunkSize: number,
  overlap: number,
  sourceId: string
): Chunk[] {
  const chunks: Chunk[] = [];
  let start = 0;
  let index = 0;

  while (start < text.length) {
    const end = Math.min(start + chunkSize, text.length);
    chunks.push({
      id: `${sourceId}-${index}`,
      text: text.slice(start, end),
      metadata: {},
      sourceId,
      chunkIndex: index,
    });
    start += chunkSize - overlap;
    index++;
  }

  return chunks;
}

The overlap parameter exists because sentence-level semantics often straddle chunk boundaries. Without overlap, “The API returns 429 when rate limited. Retry after the Retry-After header value.” can be split so neither chunk contains both concepts. With overlap, you waste embedding budget on duplicate content.

Fixed-size chunking fails when document structure matters. An API reference where each method has its own header gets chunked mid-method. A contract where clause 3.2 references definitions in clause 1 gets split without carrying the context.

Recursive Chunking

Split on paragraph breaks first, then sentences, then words, only falling back to character-level splitting when the higher-level boundaries do not produce chunks small enough:

const SEPARATORS = ["\n\n", "\n", ". ", " ", ""];

function recursiveChunk(
  text: string,
  maxSize: number,
  overlap: number,
  sourceId: string,
  separators: string[] = SEPARATORS,
  index = 0
): Chunk[] {
  if (text.length <= maxSize) {
    return [{ id: `${sourceId}-${index}`, text, metadata: {}, sourceId, chunkIndex: index }];
  }

  const separator = separators.find((sep) => text.includes(sep)) ?? "";
  const parts = separator ? text.split(separator) : [text];
  const chunks: Chunk[] = [];
  let buffer = "";
  let chunkIdx = index;

  for (const part of parts) {
    const candidate = buffer ? buffer + separator + part : part;

    if (candidate.length <= maxSize) {
      buffer = candidate;
    } else {
      if (buffer) {
        chunks.push({
          id: `${sourceId}-${chunkIdx}`,
          text: buffer,
          metadata: {},
          sourceId,
          chunkIndex: chunkIdx++,
        });
        // carry overlap
        const words = buffer.split(" ");
        buffer = words.slice(-Math.floor(overlap / 6)).join(" ") + separator + part;
      } else {
        // part itself exceeds maxSize, recurse with next separator
        const sub = recursiveChunk(part, maxSize, overlap, sourceId, separators.slice(1), chunkIdx);
        chunks.push(...sub);
        chunkIdx += sub.length;
        buffer = "";
      }
    }
  }

  if (buffer) {
    chunks.push({ id: `${sourceId}-${chunkIdx}`, text: buffer, metadata: {}, sourceId, chunkIndex: chunkIdx });
  }

  return chunks;
}

Recursive chunking preserves natural language boundaries. It is almost always better than fixed-size for prose documents.

Document-Aware Chunking

For structured documents (Markdown, HTML, PDFs with heading hierarchies), parse structure and chunk at semantic boundaries. Each section becomes its own retrieval unit, and the section heading goes into the chunk text:

interface MarkdownSection {
  heading: string;
  level: number;
  content: string;
  path: string[]; // ancestor headings
}

function chunkMarkdown(markdown: string, sourceId: string): Chunk[] {
  const lines = markdown.split("\n");
  const sections: MarkdownSection[] = [];
  const headingStack: { text: string; level: number }[] = [];
  let current: MarkdownSection | null = null;

  for (const line of lines) {
    const headingMatch = line.match(/^(#{1,6})\s+(.+)/);
    if (headingMatch) {
      if (current) sections.push(current);
      const level = headingMatch[1].length;
      const text = headingMatch[2];
      // maintain ancestor stack
      while (headingStack.length > 0 && headingStack[headingStack.length - 1].level >= level) {
        headingStack.pop();
      }
      headingStack.push({ text, level });
      current = {
        heading: text,
        level,
        content: "",
        path: headingStack.map((h) => h.text),
      };
    } else if (current) {
      current.content += line + "\n";
    }
  }
  if (current) sections.push(current);

  return sections.map((sec, idx) => ({
    id: `${sourceId}-${idx}`,
    // include the full heading path so the embedding captures context
    text: `${sec.path.join(" > ")}\n\n${sec.content.trim()}`,
    metadata: { heading: sec.heading, level: sec.level, path: sec.path },
    sourceId,
    chunkIndex: idx,
  }));
}

Including the heading path in the chunk text is not cosmetic. It means the embedding for “Authentication” under “API Reference > Endpoints > Users” carries different semantics than “Authentication” under “Security Concepts”. Without it, both chunks embed to nearly the same vector and your retrieval conflates them.

Chunking Strategy Tradeoffs

Strategy	Best for	Failure mode	Typical chunk size
Fixed-size	Homogeneous prose, fast iteration	Splits semantic units mid-sentence	512-1024 tokens
Recursive	Mixed prose and lists	Merges unrelated short paragraphs	256-800 tokens
Document-aware	Structured docs (Markdown, HTML)	Misses implicit structure (tables, code)	Section-driven
Semantic (embedding-based)	Dense technical writing	Slow at index time, hard to tune	200-600 tokens

Semantic chunking (split when adjacent sentence embeddings diverge beyond a threshold) is worth mentioning, but it is computationally expensive at index time and introduces a feedback loop: your chunking quality depends on your embedding model quality. Fix the simpler strategies first.

Embedding Model Selection

The embedding model choice matters more than the vector store choice. A mediocre model with a good chunk structure will outperform a good model with bad chunks, but a bad model with any chunk structure is hard to recover from.

Key dimensions when selecting:

• Dimensionality vs. quality: text-embedding-3-small (1536d) vs. text-embedding-3-large (3072d). Larger is not always better when your query distribution is narrow. Test on your actual queries.
• Token limit: Many models max at 512 or 8192 tokens. Chunks that exceed the model’s token limit get silently truncated. Build a check.
• Domain fit: General models trained on web text will underperform on domain-specific corpora (legal, medical, code). Models fine-tuned on code (e.g., code-search-* variants) will outperform general models for code retrieval.
• Multilingual: If your corpus has multiple languages, multilingual models (e.g., multilingual-e5-large) outperform English-only models even for English queries against multilingual documents.

interface EmbeddingConfig {
  model: string;
  dimensions: number;
  maxTokens: number;
  batchSize: number;
}

async function embedChunks(
  chunks: Chunk[],
  config: EmbeddingConfig,
  embedFn: (texts: string[]) => Promise<number[][]>
): Promise<Array<Chunk & { embedding: number[] }>> {
  const results: Array<Chunk & { embedding: number[] }> = [];

  for (let i = 0; i < chunks.length; i += config.batchSize) {
    const batch = chunks.slice(i, i + config.batchSize);
    // Guard against token limit violations before sending to API
    const texts = batch.map((c) => {
      const tokenEstimate = c.text.length / 4; // rough approximation
      if (tokenEstimate > config.maxTokens) {
        console.warn(`Chunk ${c.id} may exceed token limit (estimated ${tokenEstimate})`);
      }
      return c.text;
    });

    const embeddings = await embedFn(texts);
    batch.forEach((chunk, j) => {
      results.push({ ...chunk, embedding: embeddings[j] });
    });
  }

  return results;
}

Vector Store Indexing: HNSW Tuning

Most teams accept the defaults for HNSW (Hierarchical Navigable Small World) and wonder why their recall drops at scale. Two parameters control the tradeoff:

• M (connections per node): Higher M means better recall but more memory. Default is often 16. For corpora over 500K documents, 32-64 improves recall measurably. Memory cost is roughly M * 8 bytes * numVectors.
• ef_construction (search width during index build): Controls how thoroughly the graph is built. Higher values improve index quality at the cost of longer build time. Default of 64-128 is acceptable; 200-400 is worth the build time overhead for large corpora.
• ef (or ef_search): Controls search quality at query time. This is a runtime parameter you can tune without rebuilding the index. Start at 50, increase until recall plateaus, watch latency.

// Qdrant example with explicit HNSW config
interface QdrantCollectionConfig {
  name: string;
  vectorSize: number;
  distance: "Cosine" | "Dot" | "Euclid";
  hnsw: {
    m: number;
    ef_construct: number;
    full_scan_threshold: number; // below this count, use brute force
  };
  optimizers: {
    indexing_threshold: number; // segment size before HNSW kicks in
  };
}

const collectionConfig: QdrantCollectionConfig = {
  name: "documents",
  vectorSize: 1536,
  distance: "Cosine",
  hnsw: {
    m: 32,
    ef_construct: 200,
    full_scan_threshold: 10_000,
  },
  optimizers: {
    indexing_threshold: 20_000,
  },
};

Metadata filtering interacts with HNSW in a non-obvious way. When you filter by tenantId or documentType before the vector search, the effective search space shrinks. If you have 1M vectors but 90% of queries filter to 10K vectors, your HNSW graph is doing extra work navigating paths that lead to filtered-out nodes. Some vector stores (Qdrant, Weaviate) support payload indexing to short-circuit this. Check whether your store supports filtered HNSW rather than post-filter (HNSW search followed by discard).

Measuring Retrieval Quality

You cannot tune what you do not measure. Two metrics matter most:

Recall@k: Of the N documents that are actually relevant to a query, how many appear in the top-k results? Requires a labeled evaluation set.

MRR (Mean Reciprocal Rank): Averages the reciprocal of the rank of the first relevant document across queries. A relevant result at rank 1 scores 1.0; at rank 3, 0.33.

interface RetrievalEvalCase {
  query: string;
  relevantDocIds: string[]; // ground truth
}

interface RetrievalResult {
  docId: string;
  score: number;
}

function computeRecallAtK(
  results: RetrievalResult[],
  relevant: string[],
  k: number
): number {
  const topK = results.slice(0, k).map((r) => r.docId);
  const hits = topK.filter((id) => relevant.includes(id)).length;
  return relevant.length === 0 ? 0 : hits / relevant.length;
}

function computeMRR(evalCases: Array<{ query: string; results: RetrievalResult[]; relevant: string[] }>): number {
  const reciprocalRanks = evalCases.map(({ results, relevant }) => {
    const firstHitIndex = results.findIndex((r) => relevant.includes(r.docId));
    return firstHitIndex === -1 ? 0 : 1 / (firstHitIndex + 1);
  });
  return reciprocalRanks.reduce((sum, rr) => sum + rr, 0) / evalCases.length;
}

Build an evaluation set from real user queries mapped to relevant documents. Synthetic queries from an LLM work as a starting point but tend to be cleaner than real user queries, which means your benchmarks will be optimistic. Seed the eval set with adversarial cases: short queries, queries with typos, queries that use different terminology than the documents.

Reranking with Cross-Encoders

Bi-encoder retrieval (embedding similarity) is fast but imprecise. The model encodes query and document independently, so it cannot capture fine-grained interactions between them. A cross-encoder takes query and document together as a pair and produces a relevance score, which is more accurate but much slower.

The standard pattern: retrieve top-50 via bi-encoder, rerank with cross-encoder, return top-5 to the LLM.

interface RankedResult {
  chunk: Chunk;
  biEncoderScore: number;
  crossEncoderScore?: number;
  finalRank: number;
}

async function retrieveAndRerank(
  query: string,
  k: number,
  retrievalK: number, // larger initial pool
  biEncoderSearch: (q: string, k: number) => Promise<Array<{ chunk: Chunk; score: number }>>,
  crossEncoderScore: (query: string, texts: string[]) => Promise<number[]>
): Promise<RankedResult[]> {
  const candidates = await biEncoderSearch(query, retrievalK);

  const texts = candidates.map((c) => c.chunk.text);
  const crossScores = await crossEncoderScore(query, texts);

  const ranked = candidates
    .map((c, i) => ({
      chunk: c.chunk,
      biEncoderScore: c.score,
      crossEncoderScore: crossScores[i],
      finalRank: 0,
    }))
    .sort((a, b) => (b.crossEncoderScore ?? 0) - (a.crossEncoderScore ?? 0))
    .slice(0, k)
    .map((r, i) => ({ ...r, finalRank: i + 1 }));

  return ranked;
}

Reranking adds 100-400ms latency depending on model and batch size. Run it in parallel with other operations when possible, and cache cross-encoder scores for repeated query-document pairs if your query distribution has repetition.

Failure Modes at Scale

Context Window Overflow

The chunks retrieved may total more tokens than your LLM’s context window. This surfaces silently: the API truncates input, the LLM generates a response based on incomplete context, and the user sees a plausible but wrong answer.

Fix: measure token count of the full prompt (system + retrieved context + query) before sending. Use a budget:

function buildPrompt(
  query: string,
  chunks: Chunk[],
  systemPrompt: string,
  maxContextTokens: number,
  tokenCounter: (text: string) => number
): { prompt: string; droppedChunks: number } {
  const systemTokens = tokenCounter(systemPrompt);
  const queryTokens = tokenCounter(query);
  const overhead = 200; // format tokens, separators
  let budget = maxContextTokens - systemTokens - queryTokens - overhead;

  const included: Chunk[] = [];
  for (const chunk of chunks) {
    const tokens = tokenCounter(chunk.text);
    if (tokens <= budget) {
      included.push(chunk);
      budget -= tokens;
    }
  }

  const contextBlock = included.map((c) => c.text).join("\n\n---\n\n");
  const prompt = `${systemPrompt}\n\nContext:\n${contextBlock}\n\nQuestion: ${query}`;

  return { prompt, droppedChunks: chunks.length - included.length };
}

Log droppedChunks. A consistent non-zero value means your retrieval is pulling too many chunks, or your chunks are too large.

Embedding Drift

When you upgrade your embedding model (or switch providers), existing vectors are no longer comparable to new query embeddings. The semantic space has shifted. Similarity scores become meaningless across the model boundary.

This is not hypothetical: OpenAI deprecated text-embedding-ada-002 and teams that did not reindex found their retrieval quality silently degrading as they continued embedding new documents with the new model while querying against the old index.

The mitigation is a versioned index:

interface IndexMetadata {
  indexId: string;
  embeddingModel: string;
  embeddingModelVersion: string;
  createdAt: Date;
  documentCount: number;
  status: "building" | "active" | "deprecated";
}

Keep the old index live during migration. Route new queries to the new index only after reindexing is complete and retrieval quality has been verified against your eval set. Never mix embeddings from different models in the same collection.

Stale Indexes

Documents get updated. The vector index does not, unless you build the update pipeline. Three weeks after launch you have chunks referencing outdated API versions, deprecated features, or incorrect prices.

The gap between document update time and index update time is a correctness boundary. Track it:

interface ChunkRecord {
  chunkId: string;
  sourceId: string;
  sourceUpdatedAt: Date;
  indexedAt: Date;
  staleSince?: Date; // set when source updated, cleared on reindex
}

function findStaleChunks(
  records: ChunkRecord[],
  maxStalenessMs: number
): ChunkRecord[] {
  const now = Date.now();
  return records.filter(
    (r) => r.staleSince && now - r.staleSince.getTime() > maxStalenessMs
  );
}

For high-update-frequency documents, consider a TTL-based approach: force reindexing of any chunk older than N days regardless of whether the source document changed.

Hallucination from Irrelevant Chunks

The LLM generates a confident answer based on retrieved chunks that are not actually relevant to the query. This is different from the LLM hallucinating from nothing: it is grounded in something, but the wrong something. Often harder for users to detect.

Two signals to catch this in production:

1. Minimum relevance threshold: If the top retrieval score is below your threshold, return no results or a “I don’t have information on this” response rather than passing low-confidence chunks to the LLM.
2. Chunk attribution: Structure your prompt to require the LLM to cite which chunk(s) it used. Log cases where it cites no chunks or cites a chunk with low relevance score.

function filterByMinScore(
  results: Array<{ chunk: Chunk; score: number }>,
  minScore: number
): Array<{ chunk: Chunk; score: number }> {
  const filtered = results.filter((r) => r.score >= minScore);

  if (filtered.length === 0) {
    // Caller should handle this explicitly, not pass empty context to LLM
    return [];
  }

  return filtered;
}

The right minimum score depends on your embedding model and distance metric. Calibrate it on your eval set: find the score below which precision drops below your acceptable threshold.

Latency Spikes During Reindexing

Reindexing large corpora puts write pressure on the vector store at the same time you are serving live queries. HNSW graph construction is CPU-intensive, and many vector stores share the same thread pool for reads and writes by default.

Mitigations:

• Use a blue-green index strategy: build the new index in a separate collection, swap the query target atomically when ready.
• Throttle the indexing pipeline to N upserts per second, leaving headroom for query traffic.
• Monitor query p95 latency during reindex runs. If it spikes, the reindex rate is too high for the current query load.

async function throttledUpsert<T>(
  items: T[],
  upsertFn: (batch: T[]) => Promise<void>,
  batchSize: number,
  ratePerSecond: number
): Promise<void> {
  const delayMs = (batchSize / ratePerSecond) * 1000;

  for (let i = 0; i < items.length; i += batchSize) {
    const batch = items.slice(i, i + batchSize);
    await upsertFn(batch);

    if (i + batchSize < items.length) {
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }
}

Production Retrieval Quality Checklist

Layer	What to verify	Tooling
Chunking	No semantic units split across boundaries	Manual inspection of 50 random chunks
Embeddings	Token limit violations caught before API call	Pre-embed token count check
Index	HNSW M and ef_construction tuned for corpus size	Recall@10 benchmark
Retrieval	Minimum score threshold set	Eval set with precision measurement
Reranking	Cross-encoder in place for precision-sensitive paths	MRR before/after comparison
Freshness	Stale chunk count monitored	Index metadata dashboard
Context budget	Token count checked before LLM call	Dropped chunk logging

Closing

RAG quality problems are usually indexing problems disguised as model problems. Before adjusting prompts or switching LLM providers, measure recall@k on a labeled eval set, check your chunk boundaries manually, and verify your HNSW configuration matches your corpus size. The retrieval layer is the one you fully control.