RAG Pipelines in Production: Lessons from Real Deployments

RAG gets oversold as an easy fix for LLM knowledge gaps. Drop your documents into a vector store, embed the query, fetch the top-k chunks, stuff them into the prompt. Done. Except in production, that naive path breaks in at least five distinct ways before you even get to scale.

This post covers what actually matters when you ship a RAG system to real users: chunking strategies that preserve semantics, hybrid search that beats pure vector similarity, reranking that rescues bad retrieval, evaluation that catches regressions, and the operational concerns that tutorials skip entirely.

Why Chunking Is the Hardest Part Nobody Talks About

Most tutorials chunk by fixed token count with a small overlap. That works for toy demos and almost nothing else.

The fundamental problem is that chunking is a semantic decision, not a formatting one. A 512-token window that splits in the middle of a numbered list, a code block, or a multi-sentence argument produces a fragment that makes no sense in isolation. When that fragment gets retrieved and dropped into a prompt, the model either ignores it or hallucinates a bridge from it to the answer.

Three strategies hold up in production:

Recursive character splitting is still a reasonable default, but only when paired with content-aware delimiters. Split on paragraphs first, then sentences, then fall back to token count. The LangChain RecursiveCharacterTextSplitter does this, but the defaults are too aggressive. Start with chunk_size=1000 and chunk_overlap=200 rather than the 256-token defaults you see in most examples.

Semantic chunking groups sentences by embedding similarity, cutting at points where cosine distance spikes. This is computationally more expensive at ingest time but produces far cleaner retrieval units. It is worth the cost for high-value knowledge bases where precision matters.

Document-structure-aware chunking is the most effort and the most reliable. Parse markdown headers, HTML structure, or PDF section markers and respect them as natural boundaries. A policy document, a technical spec, and a support transcript all have different natural units.

from langchain.text_splitter import RecursiveCharacterTextSplitter
import re

def chunk_with_structure(text: str, max_chunk_size: int = 1000) -> list[str]:
    # First try to split on markdown headers
    header_pattern = re.compile(r'^#{1,3}\s+.+$', re.MULTILINE)
    sections = header_pattern.split(text)
    headers = header_pattern.findall(text)

    chunks = []
    for i, section in enumerate(sections):
        header = headers[i - 1] if i > 0 else ""
        content = f"{header}\n{section}".strip() if header else section.strip()

        if len(content) <= max_chunk_size:
            if content:
                chunks.append(content)
        else:
            # Fall back to recursive splitting for long sections
            splitter = RecursiveCharacterTextSplitter(
                chunk_size=max_chunk_size,
                chunk_overlap=200,
                separators=["\n\n", "\n", ". ", " "],
            )
            chunks.extend(splitter.split_text(content))

    return chunks

One pattern that pays off consistently: store chunk metadata alongside the vector. Page number, section title, source document, and a full-document summary (generated once at ingest). The summary does not go into the chunk embedding, but it is available for downstream filtering and for enriching retrieved context before the prompt.

Embedding Model Selection: The Tradeoff Table

Picking the wrong embedding model is a latency, cost, and accuracy problem bundled together. The common mistake is using the cheapest model available and wondering why retrieval quality is poor.

The main axes are: embedding dimensionality, context window (how many tokens the model can meaningfully encode), domain specificity, and cost per token.

General-purpose models (OpenAI text-embedding-3-small, Cohere embed-english-v3.0, open-source bge-m3) perform well for general knowledge bases. For code retrieval, voyage-code-2 or a fine-tuned model on your codebase will outperform anything general-purpose. For multilingual content, multilingual-e5-large or Cohere’s multilingual embeddings handle cross-language queries correctly where other models fail silently.

A few things that are not obvious until you run the numbers:

Higher-dimensional embeddings are not always better. text-embedding-3-large (3072 dimensions) versus text-embedding-3-small (1536 dimensions) rarely produces meaningful accuracy gains on well-structured knowledge bases, but it doubles your storage and increases similarity search latency in proportion to dimensionality.

The model you use at ingest must match the model you use at query time. This is obvious, but the failure mode is subtle: if you migrate embedding models without re-indexing your corpus, you will get high cosine similarity scores between semantically unrelated documents because the vector spaces are incompatible.

Context window matters more than most people realize. A model with a 512-token context window will silently truncate your 800-token chunks. The embedding captures only the first 512 tokens, and the trailing content is ignored at retrieval time but still present in the chunk that gets sent to the LLM. This produces retrieved chunks that look relevant but contain the actual answer in their tail, which the embedding never saw.

Vector DB Architecture: What You Actually Need

The vector database question gets over-engineered early and under-engineered late. The practical decision tree is simpler than vendor marketing suggests.

If you have under 1 million vectors and are fine with approximate results, pgvector with hnsw indexing handles this without introducing a new infrastructure dependency. Millions of embeddings at reasonable dimensionality fit comfortably in managed Postgres. The operational burden of running Pinecone, Weaviate, or Qdrant for a small corpus is not worth it.

Above 1 million vectors, or when you need sub-10ms p99 at high QPS, a dedicated vector store is justified. Qdrant and Weaviate both have solid production stories. Pinecone is operationally simpler but gives you less control over index configuration.

The schema decision that matters most is how you model your metadata filters. If your users search across documents by tenant, product version, date range, or category, your vector store needs to support efficient pre-filtering before the similarity search, not post-filtering after. Post-filtering on large collections is expensive and produces inconsistent top-k counts.

// Qdrant example: filtered vector search
import { QdrantClient } from "@qdrant/js-client-rest";

const client = new QdrantClient({ url: process.env.QDRANT_URL });

async function retrieveChunks(
  queryEmbedding: number[],
  tenantId: string,
  limit: number = 10,
): Promise<RetrievedChunk[]> {
  const results = await client.search("knowledge_base", {
    vector: queryEmbedding,
    limit,
    filter: {
      must: [
        { key: "tenant_id", match: { value: tenantId } },
        { key: "is_active", match: { value: true } },
      ],
    },
    with_payload: true,
  });

  return results.map((r) => ({
    content: r.payload?.content as string,
    source: r.payload?.source as string,
    score: r.score,
    chunkId: r.id as string,
  }));
}

Hybrid Search: Why Pure Vector Similarity Is Not Enough

Vector similarity search finds semantically similar content well. It handles paraphrases, synonyms, and conceptual matches. It is poor at exact string matching: product names, model numbers, error codes, proper nouns, and anything where the spelling matters more than the meaning.

Hybrid search combines dense vector retrieval with sparse keyword retrieval (BM25 or similar) and merges the results. This is not a nice-to-have. In any real knowledge base with product documentation, technical specs, or support content, pure vector search will miss queries like “error code 0x8007045D” or “model XR-4200” with a surprisingly high frequency.

The merge step matters. Reciprocal Rank Fusion (RRF) is the standard approach: for each result, compute 1 / (k + rank) for some constant k (typically 60), then sum across both result sets. This produces a single ranked list that captures both signals.

from collections import defaultdict

def reciprocal_rank_fusion(
    dense_results: list[str],
    sparse_results: list[str],
    k: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = defaultdict(float)

    for rank, doc_id in enumerate(dense_results, start=1):
        scores[doc_id] += 1.0 / (k + rank)

    for rank, doc_id in enumerate(sparse_results, start=1):
        scores[doc_id] += 1.0 / (k + rank)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Some vector stores handle hybrid search natively (Weaviate, Elasticsearch, Azure AI Search). Others require you to run BM25 separately and merge in your application layer. Either approach works. The operational complexity of the native path is worth it if you are already using one of those stores.

Reranking: The Cheapest Accuracy Win

Retrieval returns candidates. Reranking determines which candidates actually make it into the prompt.

A cross-encoder reranker takes each (query, chunk) pair and scores them jointly, rather than independently embedding query and chunk. This is significantly more accurate than cosine similarity but too slow to run over your entire corpus. The correct pattern is to retrieve 20-50 candidates with fast approximate search, then rerank to your actual top-k (typically 3-8) with a cross-encoder.

Cohere’s rerank API is the easiest path if you do not want to self-host. For self-hosted, cross-encoder/ms-marco-MiniLM-L-6-v2 from Hugging Face is a solid general-purpose option. For domain-specific content, fine-tuning a cross-encoder on your own (query, relevant chunk, negative chunk) triples will outperform any off-the-shelf model.

The latency hit is real: adding a reranking step typically adds 100-300ms to your pipeline. Budget for this in your latency targets before adding it, not after users start complaining.

Prompt Construction from Retrieved Context

Most teams underinvest in this step and then blame retrieval quality for hallucinations that are actually prompt construction failures.

The retrieved chunks are not inherently ordered by relevance to the specific question even after reranking. A reranker scores global relevance, not positional importance within the prompt. The model is also sensitive to where the answer appears in the context window: content at the start and end of a long context tends to be recalled better than content in the middle (“lost in the middle” is a well-documented phenomenon in long-context models).

Put your highest-scored chunk first. If you have metadata available (section title, source document, date), include it as a preamble to each chunk. This gives the model a signal for how to weight conflicting information.

function buildContextBlock(chunks: RetrievedChunk[]): string {
  return chunks
    .map(
      (chunk, i) =>
        `[Source ${i + 1}: ${chunk.source}]\n${chunk.content}`,
    )
    .join("\n\n---\n\n");
}

function buildPrompt(query: string, chunks: RetrievedChunk[]): string {
  const context = buildContextBlock(chunks);

  return `You are answering questions based on the following documents.
Use only the information provided. If the answer is not in the documents, say so.

${context}

Question: ${query}

Answer:`;
}

One instruction that consistently reduces hallucination: tell the model explicitly to say “I don’t have enough information” rather than guess. Models are trained to be helpful and will fill gaps with plausible-sounding fabrications unless you give them explicit permission not to.

Evaluation: Measuring What Actually Matters

The failure mode for RAG evaluation is measuring the wrong thing. BLEU scores and exact match are not useful here. The three metrics that matter in practice are faithfulness, answer relevance, and context recall.

Faithfulness measures whether every claim in the generated answer is supported by the retrieved context. A model can retrieve the right documents and still hallucinate details not present in them.

Answer relevance measures whether the answer actually addresses the question. Retrieved context can be topically related but not answer the specific question asked.

Context recall measures whether the retrieved chunks contained the information needed to answer the question correctly. This diagnoses retrieval failures separately from generation failures.

RAGAS is the practical tool for this. It automates scoring using an LLM as judge, which has its own biases, but it is dramatically more scalable than human evaluation and catches regressions reliably.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

test_cases = [
    {
        "question": "What is the refund policy for annual subscriptions?",
        "answer": generated_answer,
        "contexts": [chunk.content for chunk in retrieved_chunks],
        "ground_truth": "Annual subscriptions are refundable within 30 days.",
    },
    # ... more test cases
]

dataset = Dataset.from_list(test_cases)
results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_recall],
)
print(results)

Build a regression suite of 50-100 questions before you make any change to the pipeline: chunking strategy, embedding model, retrieval parameters, reranker. Run it after every change. You will be surprised how often “improvements” to one metric degrade another.

Operational Challenges: Keeping the Knowledge Base Fresh

A RAG pipeline that answers questions correctly on day one will degrade over time as the underlying documents change. This is the operational challenge most architecture docs skip.

The naive approach is full re-indexing on a schedule. This works at small scale and is easy to reason about. It becomes expensive and slow as your corpus grows.

Incremental indexing is harder than it sounds. You need to track which source documents have changed, delete their existing chunks from the vector store, re-chunk and re-embed them, and insert the new chunks. The delete step is the tricky part: you need a reliable mapping from source document to chunk IDs, and that mapping needs to survive schema migrations, embedding model changes, and vector store migrations.

Store this mapping durably in a relational database, not in the vector store. The vector store is an index, not a source of truth.

For latency at scale, be aggressive about caching at two layers: the embedding layer (cache query embeddings for repeated or similar queries using semantic similarity) and the retrieval layer (cache top-k results for high-frequency queries with a short TTL). At the embedding layer, exact-match caching on the raw query string is cheap and effective for FAQ-style systems where users ask the same questions repeatedly.

Cost at scale is a function of three variables: embedding cost (proportional to corpus size and update frequency), vector search cost (proportional to QPS and corpus size), and LLM completion cost (proportional to context length and QPS). The LLM cost typically dominates and is the correct place to optimize first. Reduce context window size before reducing retrieval quality.

Failure Modes Worth Naming

Retrieval misses are the most common failure and the hardest to diagnose. The user asks a question that the corpus should answer. Nothing useful comes back. The model either says it does not know or hallucinates. The cause is usually one of: poor chunking that split the answer across multiple chunks, embedding model mismatch with the query style, or missing metadata filters that excluded the relevant documents.

Hallucination despite good retrieval is subtler. The right documents are retrieved. The model generates an answer that sounds consistent with them but adds details not present. This is a prompt construction problem, not a retrieval problem. The fix is stronger faithfulness instructions in the prompt and RAGAS faithfulness scores in your eval suite.

Prompt stuffing is when you retrieve more context than the model can reliably attend to. With 8k-token context windows, 10 chunks of 800 tokens each fills the window and leaves little room for the system prompt, question, and answer. The model starts ignoring middle chunks. Retrieve fewer, rerank harder, and keep context tight.

Closing Thoughts

The teams that ship reliable RAG systems treat retrieval quality as a first-class engineering problem, not a configuration detail. They run evaluation suites before merging changes, instrument retrieval quality separately from generation quality, and invest in chunking that matches the structure of their actual documents.

At Let’s Build Solutions, we have built RAG pipelines for legal document search, technical support systems, and internal knowledge bases. The patterns above hold across all of them. The interesting problems are always in the retrieval layer, not the generation layer, and they only surface when you instrument the pipeline end-to-end and look at where answers actually go wrong.