RAG Evaluation in Production: Retrieval Metrics, Answer Quality Scoring, and Automated Regression Testing

Most RAG pipelines get built twice. The first build is the happy-path demo: a retriever that pulls chunks from a vector store, a prompt that feeds them to a language model, and an answer that looks reasonable on a handful of hand-picked questions. The second build happens three months later, when someone notices that half the answers are subtly wrong, the retriever is returning irrelevant chunks, and there is no way to know when it started breaking.

The gap between those two builds is the absence of measurement. Retrieval quality degrades silently when the document corpus changes. Answer faithfulness drifts when the underlying model is updated. Neither shows up in standard application metrics. You need a separate evaluation layer, and it needs to run continuously.

This article covers what to measure, how to measure it, and how to wire it into CI/CD so quality regressions get caught before they reach production.

The Two Problems You Are Measuring

RAG has two distinct failure modes that require separate measurement strategies.

The first is retrieval failure: the correct document exists in your corpus but the retriever does not return it, or returns it too far down the ranked list to be included in the context window. The model then generates an answer without the information it needs. The answer may still sound confident and coherent, which makes this failure mode particularly dangerous.

The second is generation failure: the retriever returns the right documents but the model generates an answer that contradicts, ignores, or halluccinates beyond them. This happens even when retrieval is working correctly, and it compounds when retrieval is marginal.

You need metrics for both. Treating them as a single “RAG quality” score makes it impossible to diagnose which component is failing.

Retrieval Metrics

Retrieval quality is an information retrieval problem. The standard IR metrics apply: Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), and Recall@k.

Recall@k answers the simplest question: given a query, is the relevant document present in the top-k retrieved chunks? It is binary per query and averages across a test set.

interface RetrievalResult {
  queryId: string;
  retrievedChunkIds: string[];
  relevantChunkIds: string[];
}

function recallAtK(result: RetrievalResult, k: number): number {
  const topK = new Set(result.retrievedChunkIds.slice(0, k));
  const relevant = new Set(result.relevantChunkIds);
  let hits = 0;
  for (const id of relevant) {
    if (topK.has(id)) hits++;
  }
  return relevant.size === 0 ? 0 : hits / relevant.size;
}

function meanRecallAtK(results: RetrievalResult[], k: number): number {
  const scores = results.map((r) => recallAtK(r, k));
  return scores.reduce((a, b) => a + b, 0) / scores.length;
}

Recall@k tells you whether the retriever is finding the right content at all. It does not tell you how high the relevant document ranks within those k results.

MRR (Mean Reciprocal Rank) measures where the first relevant document appears. If the correct chunk is at position 1, the reciprocal rank is 1.0. At position 2 it is 0.5. At position 5 it is 0.2. MRR is most useful when you only include one context window’s worth of chunks in the prompt, because position matters: chunk rank 1 versus rank 5 in a 2048-token window is the difference between the model seeing the answer or not.

function reciprocalRank(result: RetrievalResult): number {
  const relevant = new Set(result.relevantChunkIds);
  for (let i = 0; i < result.retrievedChunkIds.length; i++) {
    if (relevant.has(result.retrievedChunkIds[i])) {
      return 1 / (i + 1);
    }
  }
  return 0;
}

function meanReciprocalRank(results: RetrievalResult[]): number {
  const scores = results.map(reciprocalRank);
  return scores.reduce((a, b) => a + b, 0) / scores.length;
}

NDCG (Normalized Discounted Cumulative Gain) handles the case where you have graded relevance rather than binary. A chunk that partially answers the question is worth less than one that directly answers it, but more than an irrelevant chunk. NDCG accounts for this and normalizes against the ideal ranking.

function dcgAtK(
  retrievedIds: string[],
  relevanceScores: Map<string, number>,
  k: number
): number {
  return retrievedIds.slice(0, k).reduce((sum, id, i) => {
    const rel = relevanceScores.get(id) ?? 0;
    return sum + rel / Math.log2(i + 2); // i+2 because i is 0-indexed
  }, 0);
}

function ndcgAtK(
  result: RetrievalResult & { relevanceScores: Map<string, number> },
  k: number
): number {
  const actualDcg = dcgAtK(result.retrievedChunkIds, result.relevanceScores, k);

  // ideal: sort all relevant chunks by their score descending
  const idealOrder = [...result.relevanceScores.entries()]
    .sort(([, a], [, b]) => b - a)
    .map(([id]) => id);
  const idealDcg = dcgAtK(idealOrder, result.relevanceScores, k);

  return idealDcg === 0 ? 0 : actualDcg / idealDcg;
}

In practice, most teams start with Recall@5 and MRR. NDCG becomes relevant once you have a labeled dataset with graded relevance, which takes more effort to build.

Answer Quality Scoring

Retrieval metrics require labeled data: you need to know which chunks are relevant to which queries. Answer quality metrics have the same requirement in a different form: you need to know what a correct answer looks like, or at least what properties a correct answer must have.

The two properties that matter most in production are faithfulness and relevance.

Faithfulness measures whether the generated answer is grounded in the retrieved context. An unfaithful answer introduces claims that do not appear in the source documents. This is the hallucination problem stated precisely.

Relevance measures whether the answer addresses what the query was actually asking. A faithful answer that answers the wrong question is still a bad answer.

LLM-as-Judge

The practical approach for scoring both dimensions at scale is to use a language model as the evaluator. You provide the original query, the retrieved context, and the generated answer, then ask the model to score faithfulness and relevance on a 0-1 or 1-5 scale with an explanation.

This sounds circular but works better than it sounds in practice, with two important caveats: use a different model (or at minimum different prompt) for evaluation than for generation, and validate your judge against human labels regularly.

interface EvalInput {
  query: string;
  context: string[];
  answer: string;
}

interface EvalScore {
  faithfulness: number;      // 0-1
  relevance: number;         // 0-1
  faithfulnessReason: string;
  relevanceReason: string;
}

async function scoreWithLLMJudge(
  input: EvalInput,
  callLLM: (prompt: string) => Promise<string>
): Promise<EvalScore> {
  const contextText = input.context
    .map((c, i) => `[${i + 1}] ${c}`)
    .join("\n\n");

  const prompt = `You are an evaluator for a question-answering system.

Query: ${input.query}

Retrieved context:
${contextText}

Generated answer: ${input.answer}

Evaluate the answer on two dimensions:

1. Faithfulness (0-1): Does every claim in the answer appear in the retrieved context? Score 1.0 if all claims are grounded, 0 if any claim contradicts or is absent from the context.

2. Relevance (0-1): Does the answer address what the query is asking? Score 1.0 if fully on-topic, 0 if off-topic or evasive.

Respond as JSON: { "faithfulness": <number>, "faithfulnessReason": "<string>", "relevance": <number>, "relevanceReason": "<string>" }`;

  const raw = await callLLM(prompt);

  // strip markdown code fences if present
  const json = raw.replace(/```(?:json)?\n?/g, "").trim();
  return JSON.parse(json) as EvalScore;
}

One practical note: LLM judges are expensive to run at scale. Run them on a sample (10-20% of queries in production, 100% in CI against your eval dataset). Cache results by query+context+answer hash so you do not pay for the same evaluation twice.

Building an Evaluation Dataset

You cannot measure anything without a labeled dataset. This is the part teams delay the longest and regret delaying the most.

A minimal eval dataset for a RAG pipeline has three columns: query, relevant chunk IDs, and reference answer (optional but useful for relevance scoring). Start with 50-100 examples. Fewer than that and variance dominates your metrics. More than 500 and the cost of maintenance outweighs the marginal signal.

There are three ways to build it:

Manual curation. Write queries yourself based on what you know users will ask. Label which chunks in your corpus should answer each query. This is the most reliable but the most time-consuming.

Sampling production traffic. If you have production traffic, sample real queries and have engineers or domain experts label the correct chunks. This produces the most representative distribution but requires a labeling workflow.

Synthetic generation. Use an LLM to generate (query, chunk) pairs from your corpus: for each chunk, generate two or three plausible questions that this chunk would answer. This is fast and scales with corpus size, but the queries tend to be cleaner and more well-formed than real user queries. Use it to bootstrap, then supplement with real traffic.

async function generateSyntheticQueries(
  chunk: string,
  chunkId: string,
  callLLM: (prompt: string) => Promise<string>
): Promise<Array<{ query: string; relevantChunkIds: string[] }>> {
  const prompt = `Given this document chunk, generate 3 realistic questions that a user might ask whose answer is contained in this chunk. Questions should be phrased naturally, not as search queries.

Chunk:
${chunk}

Return JSON: { "questions": ["...", "...", "..."] }`;

  const raw = await callLLM(prompt);
  const json = raw.replace(/```(?:json)?\n?/g, "").trim();
  const { questions } = JSON.parse(json) as { questions: string[] };

  return questions.map((query) => ({
    query,
    relevantChunkIds: [chunkId],
  }));
}

Store your eval dataset in version control. It is code, not data. When you add documents to your corpus, add corresponding eval examples. When a user reports a bad answer, add that query to the dataset with the correct labels. The dataset is a living record of your quality requirements.

Automated Regression Testing in CI/CD

The retrieval metrics and judge scores mean nothing if you only run them manually. Wire them into CI so every change to the retriever, prompt, or chunking strategy gets evaluated before it merges.

The structure is straightforward: a test suite that runs the full RAG pipeline against your eval dataset, computes all metrics, compares against stored baseline values, and fails the build if any metric drops beyond a defined threshold.

interface BaselineMetrics {
  recallAt5: number;
  mrr: number;
  meanFaithfulness: number;
  meanRelevance: number;
}

interface RegressionResult {
  passed: boolean;
  current: BaselineMetrics;
  baseline: BaselineMetrics;
  regressions: string[];
}

async function runRegressionSuite(
  evalDataset: RetrievalResult[],
  scoredAnswers: EvalScore[],
  baseline: BaselineMetrics,
  thresholds: { maxDropPercent: number }
): Promise<RegressionResult> {
  const current: BaselineMetrics = {
    recallAt5: meanRecallAtK(evalDataset, 5),
    mrr: meanReciprocalRank(evalDataset),
    meanFaithfulness:
      scoredAnswers.reduce((s, r) => s + r.faithfulness, 0) /
      scoredAnswers.length,
    meanRelevance:
      scoredAnswers.reduce((s, r) => s + r.relevance, 0) /
      scoredAnswers.length,
  };

  const regressions: string[] = [];
  const maxDrop = thresholds.maxDropPercent / 100;

  for (const key of Object.keys(baseline) as (keyof BaselineMetrics)[]) {
    const drop = (baseline[key] - current[key]) / baseline[key];
    if (drop > maxDrop) {
      regressions.push(
        `${key}: ${(baseline[key] * 100).toFixed(1)}% -> ${(current[key] * 100).toFixed(1)}% (dropped ${(drop * 100).toFixed(1)}%)`
      );
    }
  }

  return {
    passed: regressions.length === 0,
    current,
    baseline,
    regressions,
  };
}

Reasonable starting thresholds: fail the build if any metric drops more than 5% relative to baseline. This is tight enough to catch real regressions but loose enough to tolerate natural variance in LLM judge scores.

Update the baseline file when you intentionally improve the pipeline. Treat a baseline update as a deliberate decision that requires review, not a routine bump. The baseline file belongs in version control next to the eval dataset.

Tradeoffs

Approach	Cost	Coverage	Latency	When to use
Retrieval metrics only	Low	Retrieval only	Fast	Early stage, no labeled answers
Reference-based scoring	Medium	Both layers	Fast	When you have reference answers
LLM-as-judge (sampled)	Medium	Both layers	Slow	Production monitoring, 10-20% sample
LLM-as-judge (full CI)	High	Both layers	Slow	Small eval sets (<200 examples)
Human eval	Very high	Both layers	Very slow	Quarterly calibration of judge accuracy

The practical answer for most teams: use retrieval metrics in CI (fast, cheap, catches the most common regressions), add LLM-as-judge on a sampled basis in production for faithfulness monitoring, and do a human eval pass once a quarter to check that your judge scores still correlate with actual quality.

Detecting Quality Drift Over Time

Regression testing catches point-in-time failures. Drift is different: a slow, continuous degradation that no single CI run detects because each individual change is within threshold.

The causes are predictable: corpus updates that shift the document distribution, embedding model updates that change the vector space, prompt modifications that accumulate over time, and traffic distribution shifts where users start asking questions your eval set does not cover.

Track your metrics as a time series. Plot Recall@5, MRR, mean faithfulness, and mean relevance on a daily basis using a rolling 7-day window of production samples. Set anomaly detection thresholds based on historical variance, not just absolute values.

interface DailyMetricSnapshot {
  date: string;
  recallAt5: number;
  mrr: number;
  meanFaithfulness: number;
  meanRelevance: number;
  sampleSize: number;
}

function detectDrift(
  history: DailyMetricSnapshot[],
  current: DailyMetricSnapshot,
  windowDays: number,
  zThreshold: number = 2.5
): string[] {
  const window = history.slice(-windowDays);
  if (window.length < 7) return []; // not enough history

  const alerts: string[] = [];
  const keys: (keyof DailyMetricSnapshot)[] = [
    "recallAt5",
    "mrr",
    "meanFaithfulness",
    "meanRelevance",
  ];

  for (const key of keys) {
    const values = window.map((s) => s[key] as number);
    const mean = values.reduce((a, b) => a + b, 0) / values.length;
    const variance =
      values.reduce((a, b) => a + Math.pow(b - mean, 2), 0) / values.length;
    const stdDev = Math.sqrt(variance);

    const currentVal = current[key] as number;
    const zScore = stdDev === 0 ? 0 : (mean - currentVal) / stdDev;

    if (zScore > zThreshold) {
      alerts.push(
        `${key} z-score ${zScore.toFixed(2)} (mean ${mean.toFixed(3)}, current ${currentVal.toFixed(3)})`
      );
    }
  }

  return alerts;
}

A z-score above 2.5 on a 30-day window is a reasonable alert threshold. It will produce occasional false positives during legitimate improvements (where your score jumps up), which you can handle by only alerting on downward deviations.

The most important operational practice: when the drift alert fires, pull the specific queries that scored lowest that day and read them. Numbers tell you that quality dropped. The actual queries tell you why.

Production Considerations

A few things that only become obvious after running this in production:

Chunking changes invalidate your eval dataset. If you change chunk size or overlap, your relevantChunkIds labels become stale because the same content now lives at different IDs. Version your corpus and eval dataset together. A chunking strategy change is a breaking migration that requires re-labeling.

LLM judge calibration degrades over time. The model you use as a judge gets updated. Run a human-labeled calibration set (50-100 examples with human scores) monthly and track the correlation between human scores and judge scores. If the correlation drops, your judge is no longer trustworthy and all your historical metrics are suspect.

Retrieval metrics can look good while generation quality drops. Recall@5 of 0.9 means the right chunk is in the top 5. It does not mean the model reads and uses it correctly. Always run both layers of evaluation. A common failure mode: the retriever surfaces a chunk that partially answers the question, the model extrapolates beyond it, and faithfulness drops without any retrieval signal.

Sample size matters for statistical significance. A 5% drop in MRR on 20 queries could be one query. On 500 queries it is a signal. Keep your eval dataset large enough that a 5% metric change represents at least 10 query changes. Below that, you are chasing noise.

The measurement system does not need to be perfect to be useful. Start with a 50-query eval set, Recall@5, and a faithfulness judge running in CI. That catches the majority of regressions. Add complexity when you have evidence that the simpler setup is missing real failures.

RAG quality is a property of the system over time, not a one-time validation. The pipeline that worked three months ago may not work today if any of its dependencies changed. The only way to know is to measure it continuously, and the only way to measure it continuously is to build the infrastructure for it before you think you need it.