Building an LLM Evaluation Harness: Automated Scoring, Human-in-the-Loop Review, and Regression Testing for Prompt Changes

The pattern is predictable. A team ships an LLM feature, it works well on the examples they tested manually, and they call it done. Six weeks later a prompt change breaks a subtle behavior nobody thought to document. The regression is discovered by a user in production, not a test in CI.

The root cause is not carelessness. It is the absence of an evaluation harness. Most teams treat LLM outputs like frontend UI: they look at them and decide if they look right. That approach does not scale past a handful of test cases, and it provides zero protection against regression.

This article covers how to build an evaluation harness that actually prevents this. Specifically: structuring golden datasets that survive prompt evolution, implementing three scoring methods in TypeScript, designing a human review queue for ambiguous cases, and wiring regression gates into CI so prompt changes fail the build when quality drops.

What an Evaluation Harness Is Not

Before building one, it is worth clarifying the scope. An evaluation harness is not:

• A production monitoring system. Prod monitoring samples live traffic and watches aggregate metrics. An eval harness runs offline against a curated test set before deploy.
• A unit test suite. Unit tests assert exact outputs against deterministic functions. Eval harnesses score outputs on a quality spectrum and compare relative to a baseline.
• A benchmark. Benchmarks measure capability in isolation. Eval harnesses measure whether your specific application, with your specific prompts and your specific data, still behaves as intended after a change.

The harness sits in the CI pipeline and runs on every PR that touches a prompt, a model configuration, or a retrieval component. It produces a score, compares it to a stored baseline, and either passes or blocks the PR.

Dataset Design: The Foundation

The quality of your evaluation is bounded by the quality of your test cases. A golden dataset for an LLM application has three parts.

Inputs. The queries, documents, or conversations that the system will process. These should be representative of real production traffic, not the happy-path examples you used during development. Collect them by logging production inputs from week one, even before you have scoring infrastructure.

Expected outputs (or expected properties). For some tasks you have a ground-truth answer: the correct SQL query, the correct entity extracted, the correct classification label. For generation tasks you specify constraints: the response must mention product X, must not hallucinate a price, must be under 150 words, must include a call to action. For open-ended tasks you capture a reference response and score similarity against it.

Metadata. Tags that let you slice results. At minimum: category (what type of input this is), difficulty (hard cases that historically regressed), and source (production sample vs. synthetic).

A minimal dataset record looks like this:

interface EvalCase {
  id: string;
  input: EvalInput;
  expected: ExpectedOutput;
  metadata: {
    category: string;
    difficulty: "easy" | "medium" | "hard";
    source: "production" | "synthetic" | "adversarial";
    addedAt: string;
    notes?: string;
  };
}

interface EvalInput {
  systemPrompt: string;
  userMessage: string;
  context?: string; // for RAG cases
}

interface ExpectedOutput {
  // At least one of these must be present
  exactMatch?: string;
  referenceAnswer?: string;
  requiredSubstrings?: string[];
  forbiddenSubstrings?: string[];
  maxLength?: number;
  structuredSchema?: Record<string, unknown>; // for JSON outputs
}

Store datasets as versioned JSON files in the repo alongside the prompts they test. When a prompt changes, review whether any test cases need updating. When you find a production regression, add a test case for it immediately before fixing the bug. This creates a growing suite that directly encodes your failure history.

Three Scoring Methods

No single scoring method works for all output types. A practical harness implements all three and selects based on the output type in each test case.

Exact Match

Use this for tasks with a single correct answer: entity extraction, classification, structured JSON generation, SQL synthesis. It is cheap, fast, and deterministic.

function exactMatchScore(actual: string, expected: string): ScoringResult {
  const normalizedActual = actual.trim().toLowerCase();
  const normalizedExpected = expected.trim().toLowerCase();
  return {
    method: "exact_match",
    score: normalizedActual === normalizedExpected ? 1.0 : 0.0,
    passed: normalizedActual === normalizedExpected,
    detail: { actual: normalizedActual, expected: normalizedExpected },
  };
}

// For JSON outputs, compare parsed structure rather than raw string
function structuredMatchScore(
  actual: string,
  schema: Record<string, unknown>
): ScoringResult {
  try {
    const parsed = JSON.parse(actual);
    const violations = validateSchema(parsed, schema);
    const score = violations.length === 0 ? 1.0 : 0.0;
    return {
      method: "structured_match",
      score,
      passed: score === 1.0,
      detail: { violations },
    };
  } catch {
    return {
      method: "structured_match",
      score: 0.0,
      passed: false,
      detail: { error: "invalid_json" },
    };
  }
}

Semantic Similarity

Use this when the output is open-ended text but you have a reference answer. Two responses can be semantically equivalent while being textually different. A simple string match would false-negative constantly.

The approach: embed both the actual and reference response using a text embedding model, then compute cosine similarity.

import OpenAI from "openai";

const openai = new OpenAI();

async function semanticSimilarityScore(
  actual: string,
  reference: string,
  threshold: number = 0.85
): Promise<ScoringResult> {
  const [actualEmbedding, referenceEmbedding] = await Promise.all([
    embed(actual),
    embed(reference),
  ]);

  const similarity = cosineSimilarity(actualEmbedding, referenceEmbedding);

  return {
    method: "semantic_similarity",
    score: similarity,
    passed: similarity >= threshold,
    detail: { similarity, threshold },
  };
}

async function embed(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

The threshold needs calibration. Start at 0.85, run your existing good outputs through it, and check whether the similarity distribution matches your manual quality assessment. Adjust until false positives and false negatives are both acceptable.

LLM-as-Judge

Use this for open-ended quality criteria that cannot be expressed as substring checks or similarity scores: does the response stay on topic, does it avoid hallucinating, does it match the required tone, is the reasoning coherent?

The judge pattern works by prompting a second LLM call to score the output against a rubric. The key to making it reliable is being explicit and narrow in the rubric. Vague prompts produce vague scores.

async function llmJudgeScore(
  input: EvalInput,
  actual: string,
  rubric: JudgeRubric
): Promise<ScoringResult> {
  const judgePrompt = buildJudgePrompt(input, actual, rubric);

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: judgePrompt }],
    response_format: { type: "json_object" },
    temperature: 0, // determinism matters here
  });

  const judgment = JSON.parse(response.choices[0].message.content ?? "{}");

  return {
    method: "llm_judge",
    score: judgment.score / rubric.maxScore,
    passed: judgment.score >= rubric.passingScore,
    detail: {
      score: judgment.score,
      reasoning: judgment.reasoning,
      rubric: rubric.name,
    },
  };
}

function buildJudgePrompt(
  input: EvalInput,
  actual: string,
  rubric: JudgeRubric
): string {
  return `You are evaluating the quality of an AI assistant's response.

User message: ${input.userMessage}
${input.context ? `Context provided to assistant:\n${input.context}\n` : ""}
Assistant response: ${actual}

Evaluation criteria: ${rubric.description}

Score the response from 0 to ${rubric.maxScore} based on these criteria:
${rubric.criteria.map((c, i) => `${i + 1}. ${c}`).join("\n")}

Respond with JSON in this format:
{"score": <number>, "reasoning": "<one sentence explaining the score>"}`;
}

interface JudgeRubric {
  name: string;
  description: string;
  criteria: string[];
  maxScore: number;
  passingScore: number;
}

One practical caveat: LLM judges are not free from bias. They tend to favor longer, more confident-sounding responses. If you use a judge, use the same judge model consistently across runs so at least the bias is stable. Periodically validate the judge’s scores against human ratings to catch drift in the judge model’s behavior when it gets updated.

The Evaluation Runner

With scoring methods in place, the runner is the orchestration layer that loads a dataset, runs each case through the appropriate scorer, and produces a report.

interface EvalRunConfig {
  datasetPath: string;
  modelFn: (input: EvalInput) => Promise<string>;
  scorers: ScorerConfig[];
  outputPath: string;
  baselinePath?: string; // for regression comparison
}

interface ScorerConfig {
  method: "exact_match" | "semantic_similarity" | "llm_judge";
  rubric?: JudgeRubric;
  threshold?: number;
  appliesTo?: string[]; // category filter
}

async function runEval(config: EvalRunConfig): Promise<EvalReport> {
  const dataset = await loadDataset(config.datasetPath);
  const results: CaseResult[] = [];

  // Run in parallel with concurrency limiting to avoid rate limits
  const semaphore = new Semaphore(5);
  await Promise.all(
    dataset.cases.map(async (evalCase) => {
      await semaphore.acquire();
      try {
        const actual = await config.modelFn(evalCase.input);
        const scores = await scoreCase(actual, evalCase, config.scorers);
        results.push({
          caseId: evalCase.id,
          input: evalCase.input,
          actual,
          expected: evalCase.expected,
          scores,
          metadata: evalCase.metadata,
          passed: scores.every((s) => s.passed),
        });
      } finally {
        semaphore.release();
      }
    })
  );

  const report = buildReport(results, config);

  // Compare to baseline if provided
  if (config.baselinePath) {
    const baseline = await loadReport(config.baselinePath);
    report.regression = compareToBaseline(report, baseline);
  }

  await saveReport(report, config.outputPath);
  return report;
}

The Semaphore class here is straightforward: it limits concurrent API calls to avoid hitting rate limits during a large eval run. Keep it configurable per environment (5 for development, higher for CI with higher-tier API access).

Human-in-the-Loop Review

Automated scoring handles the clear cases. The hard cases sit in the middle: outputs that scored 0.72 on semantic similarity, where the threshold is 0.85, and you need a human to decide whether it is actually acceptable.

The pattern is a review queue: cases that fail automated scoring but are within a “maybe” band get flagged for human review instead of immediately failing.

interface ReviewQueueItem {
  caseId: string;
  runId: string;
  input: EvalInput;
  actual: string;
  expected: ExpectedOutput;
  scores: ScoringResult[];
  flagReason: string;
  status: "pending" | "approved" | "rejected";
  reviewedBy?: string;
  reviewedAt?: string;
  reviewNotes?: string;
}

function shouldFlagForReview(
  scores: ScoringResult[],
  config: ReviewConfig
): { flag: boolean; reason: string } {
  for (const score of scores) {
    if (score.method === "semantic_similarity" && score.score !== undefined) {
      const { lowerBound, upperBound } = config.ambiguityBands.semantic;
      if (score.score >= lowerBound && score.score < upperBound) {
        return {
          flag: true,
          reason: `Semantic similarity ${score.score.toFixed(3)} is in ambiguous range [${lowerBound}, ${upperBound})`,
        };
      }
    }
    if (score.method === "llm_judge" && score.score !== undefined) {
      const { lowerBound, upperBound } = config.ambiguityBands.judge;
      if (score.score >= lowerBound && score.score < upperBound) {
        return {
          flag: true,
          reason: `Judge score ${score.score.toFixed(3)} is in ambiguous range`,
        };
      }
    }
  }
  return { flag: false, reason: "" };
}

The review interface can be as simple as a static HTML page generated from the JSON queue file, where a reviewer clicks approve or reject for each item. The outcome feeds back into two places: it adjusts the final pass/fail count for the current run, and it optionally updates the dataset with a human-validated label that improves future automated scoring.

Do not block CI on human review for regular PRs. Human review is for auditing and dataset improvement, not as a blocking gate. The blocking gate is automated scoring only, because CI needs to be fast and asynchronous human review does not fit that model.

Regression Testing in CI

The evaluation harness becomes a regression gate when you store a baseline score and block merges when the current run drops below it.

interface RegressionResult {
  hasRegression: boolean;
  overallDelta: number;
  categoryDeltas: Record<string, number>;
  regressedCases: string[];
  threshold: number;
}

function compareToBaseline(
  current: EvalReport,
  baseline: EvalReport,
  threshold: number = -0.03 // allow 3% degradation
): RegressionResult {
  const overallDelta = current.passRate - baseline.passRate;
  const hasRegression = overallDelta < threshold;

  const categoryDeltas: Record<string, number> = {};
  for (const category of Object.keys(baseline.categoryPassRates)) {
    const baseRate = baseline.categoryPassRates[category] ?? 0;
    const currRate = current.categoryPassRates[category] ?? 0;
    categoryDeltas[category] = currRate - baseRate;
  }

  // Cases that passed in baseline but fail now
  const baselinePassedIds = new Set(
    baseline.results.filter((r) => r.passed).map((r) => r.caseId)
  );
  const regressedCases = current.results
    .filter((r) => !r.passed && baselinePassedIds.has(r.caseId))
    .map((r) => r.caseId);

  return {
    hasRegression,
    overallDelta,
    categoryDeltas,
    regressedCases,
    threshold,
  };
}

In the CI script, fail the job on regression:

// eval-ci.ts — run as: npx tsx eval-ci.ts
import { runEval } from "./eval-runner";
import { loadConfig } from "./eval-config";

const config = loadConfig();
const report = await runEval(config);

if (report.regression?.hasRegression) {
  console.error(
    `Eval regression detected. Pass rate delta: ${report.regression.overallDelta.toFixed(3)}`
  );
  console.error(
    `Regressed cases: ${report.regression.regressedCases.join(", ")}`
  );
  process.exit(1);
}

console.log(
  `Eval passed. Pass rate: ${(report.passRate * 100).toFixed(1)}% (baseline: ${(report.baseline?.passRate ?? 0 * 100).toFixed(1)}%)`
);
process.exit(0);

Pair this with a GitHub Actions job that runs on PRs touching prompt files or model configuration:

# .github/workflows/eval.yml
name: LLM Eval

on:
  pull_request:
    paths:
      - "src/prompts/**"
      - "src/config/models.ts"
      - "evals/datasets/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm ci
      - run: npx tsx evals/eval-ci.ts
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          EVAL_BASELINE_PATH: evals/baselines/main.json
          EVAL_OUTPUT_PATH: evals/results/pr-${{ github.event.pull_request.number }}.json

The baseline file lives in the repo and gets updated manually when you intentionally improve quality. The update process: run the harness on main, verify the scores look correct, commit the new baseline. This is a deliberate act, not an automated one. You do not want CI to auto-update the baseline on every merge or the regression gate becomes meaningless.

Tradeoffs to Know

Scoring Method	Cost	Speed	Reliability	Best For
Exact match	Free	Instant	Deterministic	Classification, extraction, SQL
Semantic similarity	Low (embedding API)	Fast	High for similar domains	Paraphrase detection, summarization
LLM-as-judge	Medium (inference cost)	Moderate	Variable (judge bias)	Open-ended quality, tone, safety
Human review	High (time)	Slow (async)	Ground truth	Ambiguous cases, dataset validation

Run exact match first. Only call the embedding API if exact match fails. Only call the LLM judge for cases that require it. This keeps eval run cost proportional to the difficulty of the test cases rather than flat across all of them.

Managing the Dataset Over Time

A dataset that does not evolve becomes stale. Track these operations explicitly:

Adding cases. Any production regression that reaches users becomes a test case. Log the input, the bad output, and the expected output. Add it to the dataset before shipping the fix. This is the only reliable way to prevent the same regression appearing twice.

Pruning cases. If your application changes behavior intentionally (you launched a new feature, you deprecated a flow), some test cases will permanently fail. Do not delete them silently. Mark them as deprecated: true with a note explaining why the behavior changed and when. This preserves the audit trail.

Versioning datasets. Store datasets as files in git alongside the prompts. When a prompt change requires updating expected outputs, the dataset diff is part of the PR. Reviewers can see both the prompt change and the corresponding test case updates in the same review.

Synthetic augmentation. When you have too few hard cases, generate synthetic ones: ask an LLM to produce adversarial inputs based on your existing cases. Treat synthetic cases with a lower weight in your pass rate calculation than production-sourced cases. They are useful for coverage, but they do not represent real user behavior.

Production Considerations

A few things that affect harness design at scale:

Cost control. A dataset with 500 cases, run on every PR, with LLM judge scoring, can cost $2-5 per run depending on prompt length. Segment your dataset: run the full suite nightly and on main merges, run a fast subset (exact match and high-confidence cases only) on every PR. The fast subset runs in under a minute and catches most regressions.

Parallelism and rate limits. The Semaphore pattern above is the right approach. Tune concurrency per environment. For OpenAI with a tier 2 account, 10-20 concurrent requests is safe. For a self-hosted model, the limit is your GPU memory.

Model version pinning. If your judge uses a model that gets updated, the judge’s behavior changes and your historical baselines become incompatible. Pin the judge to a specific model version (e.g., gpt-4o-2024-11-20) and treat a judge model upgrade as a breaking change that requires re-baselining.

Non-determinism at temperature 0. Even at temperature 0, models occasionally produce different outputs across calls due to floating-point non-determinism in GPU kernels and batching. For regression gates, run each case twice and use the lower score. This is conservative but prevents flaky CI.

The Compounding Value

The harness starts paying off immediately on the first prevented regression. But the long-term value is in the dataset. Every production bug becomes a test case. Every intentional quality improvement moves the baseline. After six months of running this, you have a precise record of what your application was supposed to do, what it actually did, and when behavior changed. That record is valuable beyond CI: it informs model upgrade decisions, prompt architecture reviews, and conversations with stakeholders about what “quality” actually means for your specific application.

Build the harness early, before you need it. The worst time to start is after the first regression reaches production.