LLM Evaluation Datasets: Building, Curating, and Versioning Gold-Standard Test Sets for Production AI

The honest state of LLM quality assurance at most companies: someone runs a few manual prompts, decides it feels better than last week, and ships. This works until a model update silently regresses a critical path, a new prompt template breaks on edge cases nobody thought to test, or a competitor publishes a benchmark that exposes what the product actually cannot do.

Eval datasets are not a research luxury. They are the difference between shipping with confidence and shipping and hoping. But building them correctly requires discipline: sourcing representative examples, getting annotations right, keeping the dataset fresh as your application evolves, and connecting it to something that actually stops bad deploys. This article covers the full pipeline.

---

Why Production Logs Are the Only Honest Source

Public benchmarks like MMLU, HellaSwag, or the GSM8K math set measure things that may have nothing to do with your application’s failure modes. A customer-facing chatbot that answers questions about a SaaS product has different failure modes than a coding assistant or a document summarizer. The distribution of inputs that matters is the one your users actually send.

Start with production logs. Every inference request your system processes is a sample from the real input distribution. The goal is to mine this for eval candidates.

interface ProductionLog {
  requestId: string;
  timestamp: Date;
  userId: string;
  sessionId: string;
  input: string;
  output: string;
  modelVersion: string;
  latencyMs: number;
  userFeedback?: "thumbs_up" | "thumbs_down" | "regenerated" | "copied";
  metadata: Record<string, unknown>;
}

interface EvalCandidate {
  sourceRequestId: string;
  input: string;
  referenceOutput?: string;
  samplingReason: SamplingReason;
  difficulty?: "easy" | "medium" | "hard" | "adversarial";
  tags: string[];
}

type SamplingReason =
  | "negative_feedback"
  | "high_latency"
  | "low_confidence"
  | "novel_pattern"
  | "stratified_random"
  | "edge_case_detected";

Prioritize negative signals first. Thumbs-down clicks, regenerations, and abandonment within 10 seconds of a response are cheap signals that something went wrong. These inputs are guaranteed to be representative and guaranteed to be where the model is failing.

After negative signals, apply stratified random sampling across input categories. If you never include easy examples, your eval set will measure nothing but tail performance. If you never include hard examples, regressions on complex queries will go undetected.

async function sampleFromLogs(
  logs: ProductionLog[],
  targetSize: number
): Promise<EvalCandidate[]> {
  const candidates: EvalCandidate[] = [];

  // Priority tier 1: explicit negative feedback
  const negativeFeedback = logs.filter(
    (l) => l.userFeedback === "thumbs_down" || l.userFeedback === "regenerated"
  );
  const negSample = reservoirSample(negativeFeedback, Math.floor(targetSize * 0.3));
  candidates.push(
    ...negSample.map((l) => toCandidate(l, "negative_feedback"))
  );

  // Priority tier 2: novel input patterns (semantic deduplication)
  const novel = await detectNovelInputs(logs, candidates.map((c) => c.input));
  const novelSample = reservoirSample(novel, Math.floor(targetSize * 0.2));
  candidates.push(...novelSample.map((l) => toCandidate(l, "novel_pattern")));

  // Priority tier 3: stratified random across difficulty bands
  const remaining = targetSize - candidates.length;
  const stratified = stratifiedSample(logs, remaining);
  candidates.push(...stratified);

  return candidates;
}

function reservoirSample<T>(items: T[], k: number): T[] {
  const reservoir = items.slice(0, k);
  for (let i = k; i < items.length; i++) {
    const j = Math.floor(Math.random() * (i + 1));
    if (j < k) reservoir[j] = items[i];
  }
  return reservoir;
}

Reservoir sampling matters here because log volumes are large and you cannot load everything into memory to sample. It gives you a uniform random sample in a single pass.

---

Annotation Guidelines and Inter-Rater Agreement

Once you have candidates, you need gold-standard labels. The temptation is to have one person annotate everything and move on. The problem is that one person’s judgment encodes their specific interpretation of quality, which may not match what your users or your team actually want.

Inter-rater agreement is how you detect that your annotation guidelines are underspecified. If two annotators who both read the same guidelines agree on 60% of examples, the guidelines are the problem, not the annotators.

interface AnnotationTask {
  id: string;
  candidate: EvalCandidate;
  rubric: EvaluationRubric;
  assignedAnnotators: string[];
  annotations: Annotation[];
  resolvedLabel?: ResolvedAnnotation;
}

interface EvaluationRubric {
  dimensions: RubricDimension[];
  scaleType: "binary" | "likert_5" | "likert_7";
  passingThreshold: number;
}

interface RubricDimension {
  name: string;
  description: string;
  examples: { input: string; output: string; score: number; rationale: string }[];
  antiExamples: { input: string; output: string; score: number; rationale: string }[];
}

interface Annotation {
  annotatorId: string;
  timestamp: Date;
  scores: Record<string, number>;
  rationale: string;
  flags: ("ambiguous_input" | "needs_context" | "policy_question")[];
}

The rubric’s examples and anti-examples section is the most important part. Abstract descriptions of what “good” means produce low agreement. Concrete examples with rationales produce high agreement. Write at least three positive and three negative examples per dimension before you start annotation.

Calculate Cohen’s Kappa or Fleiss’ Kappa (for more than two annotators) after every annotation batch:

function cohenKappa(annotator1: number[], annotator2: number[]): number {
  if (annotator1.length !== annotator2.length) {
    throw new Error("Annotator arrays must have the same length");
  }

  const n = annotator1.length;
  const categories = [...new Set([...annotator1, ...annotator2])];

  // Observed agreement
  const observedAgreement =
    annotator1.filter((v, i) => v === annotator2[i]).length / n;

  // Expected agreement by chance
  let expectedAgreement = 0;
  for (const cat of categories) {
    const p1 = annotator1.filter((v) => v === cat).length / n;
    const p2 = annotator2.filter((v) => v === cat).length / n;
    expectedAgreement += p1 * p2;
  }

  if (expectedAgreement === 1) return 1;
  return (observedAgreement - expectedAgreement) / (1 - expectedAgreement);
}

A Kappa below 0.6 means you should stop annotating, revisit the guidelines, and add more examples. A Kappa above 0.8 on the first try usually means the task is trivially easy, which raises a different question: is your rubric capturing the subtle failure modes that actually matter?

For cases where annotators disagree, establish a resolution protocol before you start. Two options that work in practice: adjudication by a third annotator who breaks ties, or a structured discussion where both annotators explain their rationale and arrive at a consensus label with notes.

---

Stratified Sampling Across Difficulty and Edge Cases

A flat random sample from production is not representative of your failure modes. The inputs where your model fails are a small fraction of traffic. If you sample uniformly, you dilute the signal.

Stratify explicitly across four buckets:

1. Easy examples: Common, well-formed inputs where any competent model should succeed. Include these because regressions can happen on easy cases too, and a failure here is a red flag.
2. Medium examples: Typical inputs that require following instructions accurately, maintaining context, or applying domain knowledge.
3. Hard examples: Long contexts, multi-step reasoning, conflicting instructions, or inputs with subtle nuances.
4. Adversarial examples: Inputs designed to probe failure modes: prompt injections, ambiguous instructions, off-topic requests, jailbreak attempts, and inputs that look like the training distribution but have unusual answers.

interface EvalDataset {
  id: string;
  version: string;
  createdAt: Date;
  examples: EvalExample[];
  metadata: DatasetMetadata;
}

interface EvalExample {
  id: string;
  input: string;
  referenceOutput: string;
  difficulty: "easy" | "medium" | "hard" | "adversarial";
  categories: string[];
  tags: string[];
  sourceType: "production_log" | "human_authored" | "llm_generated_reviewed";
  annotationMetadata: {
    annotators: string[];
    kappa: number;
    resolvedBy?: string;
  };
}

interface DatasetMetadata {
  totalExamples: number;
  difficultyDistribution: Record<string, number>;
  categoryDistribution: Record<string, number>;
  sourceDistribution: Record<string, number>;
  coverageGaps: string[];
}

function validateDistribution(dataset: EvalDataset): string[] {
  const warnings: string[] = [];
  const dist = dataset.metadata.difficultyDistribution;
  const total = dataset.metadata.totalExamples;

  if ((dist["easy"] ?? 0) / total < 0.2) {
    warnings.push("Easy examples below 20% -- regressions on simple inputs may go undetected");
  }
  if ((dist["adversarial"] ?? 0) / total < 0.1) {
    warnings.push("Adversarial examples below 10% -- robustness coverage is insufficient");
  }

  return warnings;
}

A reasonable starting distribution: 25% easy, 40% medium, 25% hard, 10% adversarial. Adjust based on your application’s risk profile. A safety-critical deployment should weight adversarial examples much higher.

---

LLM-Generated Examples and Why Human Review Is Non-Negotiable

Manually authoring hundreds of examples is expensive. Using an LLM to generate additional examples is reasonable, with a critical constraint: every LLM-generated example must pass human review before entering the eval set.

The failure mode without review is systematic blind spots. An LLM used to generate examples will reflect its own biases about what failure modes look like. It will generate plausible-looking hard examples that are actually trivial for itself to answer correctly, leaving your actual failure modes untested.

interface GenerationJob {
  seed: EvalExample[];
  targetCount: number;
  generationModel: string;
  generationPrompt: string;
  outputPath: string;
}

const GENERATION_SYSTEM_PROMPT = `You are generating evaluation examples for an LLM system.

Given seed examples as reference, generate new examples that:
1. Probe similar failure modes but with different surface forms
2. Include subtle variations that might expose edge cases
3. Cover the same difficulty level as the seed examples
4. Are realistic -- they should resemble inputs real users would send

For each example, output:
- input: the user-facing prompt
- expectedBehavior: description of what a correct response looks like (not the full response)
- difficulty: easy | medium | hard | adversarial
- rationale: why this example is useful for evaluation

Do not generate examples where the answer is obvious. Do not generate examples you are confident you can answer correctly.`;

async function generateCandidates(
  job: GenerationJob,
  llm: LLMClient
): Promise<EvalCandidate[]> {
  const response = await llm.complete({
    system: GENERATION_SYSTEM_PROMPT,
    messages: [
      {
        role: "user",
        content: `Seed examples:\n${JSON.stringify(job.seed, null, 2)}\n\nGenerate ${job.targetCount} new evaluation candidates.`,
      },
    ],
  });

  const generated = parseGeneratedExamples(response.content);

  // Every generated example is marked for mandatory human review
  return generated.map((ex) => ({
    ...ex,
    sourceType: "llm_generated_pending_review",
    reviewRequired: true,
  }));
}

Track the source of every example separately in your schema. This lets you audit whether eval set quality degrades over time if human review quality declines or if reviewers start rubber-stamping generated examples.

---

Dataset Versioning and Drift Detection

An eval dataset that never changes is a dataset that stops measuring what matters. Your application evolves, your users’ inputs shift, and edge cases you did not think of in month one become common by month six.

Version your eval dataset with the same discipline you apply to code. Use semantic versioning: patch for corrections to existing examples, minor for adding examples without removing any, major for removing or significantly reclassifying examples.

interface DatasetVersion {
  semver: string;
  parentVersion: string | null;
  createdAt: Date;
  changeType: "patch" | "minor" | "major";
  changelog: string;
  exampleCount: number;
  diffSummary: DatasetDiff;
}

interface DatasetDiff {
  added: string[];    // example IDs added
  removed: string[];  // example IDs removed
  modified: string[]; // example IDs where reference output changed
  reclassified: string[]; // example IDs where difficulty or category changed
}

async function createNewVersion(
  current: EvalDataset,
  changes: EvalDatasetChange[],
  changeType: "patch" | "minor" | "major"
): Promise<EvalDataset> {
  const diff = computeDiff(current, changes);
  const nextVersion = bumpVersion(current.version, changeType);

  return {
    ...current,
    id: generateId(),
    version: nextVersion,
    createdAt: new Date(),
    examples: applyChanges(current.examples, changes),
    metadata: {
      ...recomputeMetadata(current.examples, changes),
      parentVersion: current.version,
      changelog: summarizeDiff(diff),
    },
  };
}

Drift detection is a separate concern from versioning. Even if your eval set does not change, the production input distribution may shift such that your coverage drops. Run a periodic job that embeds recent production inputs and compares their distribution to the eval set embedding distribution.

async function detectCoverageGaps(
  recentLogs: ProductionLog[],
  evalDataset: EvalDataset,
  embedder: EmbeddingModel
): Promise<CoverageReport> {
  const evalEmbeddings = await embedder.embedBatch(
    evalDataset.examples.map((e) => e.input)
  );
  const logEmbeddings = await embedder.embedBatch(
    recentLogs.slice(0, 1000).map((l) => l.input)
  );

  // Find production inputs with low maximum cosine similarity to any eval example
  const uncoveredLogs = logEmbeddings
    .map((logEmbed, i) => {
      const maxSimilarity = Math.max(
        ...evalEmbeddings.map((evalEmbed) => cosineSimilarity(logEmbed, evalEmbed))
      );
      return { log: recentLogs[i], maxSimilarity };
    })
    .filter((item) => item.maxSimilarity < 0.75);

  return {
    coverageRate: 1 - uncoveredLogs.length / recentLogs.slice(0, 1000).length,
    uncoveredExamples: uncoveredLogs.map((item) => item.log),
    recommendation:
      uncoveredLogs.length > 50
        ? "Coverage gap detected: add examples from uncovered cluster"
        : "Coverage acceptable",
  };
}

A coverage rate below 85% is a signal to sample from the uncovered region and add examples to the eval set.

---

Storing and Serving Eval Sets in CI/CD Pipelines

Eval datasets need to be versioned artifacts, not files in a Git repository. A 2,000-example dataset with multi-turn conversations can easily exceed what belongs in source control. Use an artifact store (S3, GCS, or a dedicated eval platform) and reference datasets by version ID in your CI config.

interface EvalRunConfig {
  datasetId: string;
  datasetVersion: string;
  modelVersion: string;
  evaluatorConfig: EvaluatorConfig;
  passingThreshold: PassingThreshold;
  parallelism: number;
}

interface PassingThreshold {
  overallPassRate: number;       // e.g., 0.92 -- 92% of examples must pass
  easyPassRate: number;          // e.g., 0.98 -- near-perfect on easy examples
  adversarialPassRate: number;   // e.g., 0.75 -- lower bar for adversarial
  regressionTolerance: number;   // e.g., 0.02 -- allow at most 2% regression vs baseline
}

interface EvalResult {
  runId: string;
  datasetVersion: string;
  modelVersion: string;
  timestamp: Date;
  results: ExampleResult[];
  summary: EvalSummary;
  passedGate: boolean;
  failureReason?: string;
}

interface EvalSummary {
  overallPassRate: number;
  passRateByDifficulty: Record<string, number>;
  passRateByCategory: Record<string, number>;
  regressionVsBaseline: number | null;
  newFailures: string[];   // example IDs that passed before but fail now
  newPasses: string[];     // example IDs that failed before but pass now
}

The regressionVsBaseline field is the critical metric for CI gates. An overall pass rate of 91% is meaningless without knowing whether it was 93% last week. Track the baseline from the last promoted model version and flag when the current run regresses beyond the tolerance.

A minimal CI step looks like this in GitHub Actions:

- name: Run eval suite
  run: |
    npx tsx scripts/run-eval.ts \
      --dataset-version ${{ env.EVAL_DATASET_VERSION }} \
      --model-version ${{ github.sha }} \
      --output eval-results.json

- name: Check eval gate
  run: |
    npx tsx scripts/check-gate.ts \
      --results eval-results.json \
      --fail-on-regression

// scripts/check-gate.ts
async function checkGate(
  results: EvalResult,
  config: EvalRunConfig
): Promise<void> {
  const { summary } = results;
  const { passingThreshold } = config;

  const failures: string[] = [];

  if (summary.overallPassRate < passingThreshold.overallPassRate) {
    failures.push(
      `Overall pass rate ${(summary.overallPassRate * 100).toFixed(1)}% below threshold ${(passingThreshold.overallPassRate * 100).toFixed(1)}%`
    );
  }

  if (
    summary.regressionVsBaseline !== null &&
    summary.regressionVsBaseline < -passingThreshold.regressionTolerance
  ) {
    failures.push(
      `Regression of ${(Math.abs(summary.regressionVsBaseline) * 100).toFixed(1)}% exceeds tolerance ${(passingThreshold.regressionTolerance * 100).toFixed(1)}%`
    );
  }

  if (summary.passRateByDifficulty["easy"] < passingThreshold.easyPassRate) {
    failures.push(
      `Easy example pass rate ${(summary.passRateByDifficulty["easy"] * 100).toFixed(1)}% below threshold -- likely a prompt or model config regression`
    );
  }

  if (failures.length > 0) {
    console.error("Eval gate failed:");
    failures.forEach((f) => console.error(`  - ${f}`));
    console.error("\nNew failures (passed before, fail now):");
    summary.newFailures.forEach((id) => console.error(`  - ${id}`));
    process.exit(1);
  }

  console.log(`Eval gate passed: ${(summary.overallPassRate * 100).toFixed(1)}% overall pass rate`);
}

The newFailures list is more useful for debugging than the aggregate pass rate. It tells you exactly which examples regressed, which lets you trace the regression to a specific prompt change, model update, or configuration change.

---

Tradeoffs: Eval Dataset Approaches

Dimension	Manual Gold Set	Production Sampled	LLM Assisted
Coverage of real inputs	Low initially	High	Medium
Annotation cost	High	High	Lower, but review required
Time to first usable set	Days	Days to weeks	Hours to days
Systematic bias risk	Author’s blind spots	Reflects real distribution	Generator model’s blind spots
Adversarial coverage	Depends on author effort	Low unless deliberately seeded	Medium with good prompts
Drift over time	High (stale fast)	Low (refresh from logs)	Medium
Versioning complexity	Low	Medium	Medium

No single source is sufficient. A production eval set without manually authored adversarial examples will miss failure modes that never appeared in your logs. A purely manual set will not reflect the long tail of real user inputs. The right answer is a blend: production samples as the base, human-authored examples for adversarial coverage and specific business requirements, and LLM-generated examples to fill gaps quickly when coverage analysis flags a new cluster.

---

Production Considerations

Annotator fatigue is real. Annotation quality drops after 90-120 minutes of continuous work. Structure annotation sessions in batches of 50-100 examples maximum. Track per-annotator Kappa over time and flag if a specific annotator’s agreement with others drops.

Reference outputs age. If your application’s correct answer to a question changes because your underlying data changed (prices, policies, product features), your reference outputs are wrong. Tag examples with a validity window and re-annotate when the underlying facts change.

Separate your eval runner from your eval judge. If you use an LLM to grade open-ended outputs, the grading LLM and the model under evaluation must be different. Grading your own outputs produces inflated scores. Use a separate, pinned model version for evaluation and update it deliberately, not automatically.

Log all eval runs. Every CI eval run should produce a structured result artifact with the full per-example breakdown, stored indefinitely. This is your audit trail when you need to understand when a regression was introduced or when a product manager questions whether a model update actually improved quality.

Eval set contamination. If your model is fine-tuned on data that includes your eval examples, your eval results are meaningless. Keep a strict separation between training data and eval data. This is harder than it sounds when the source of both is production logs.

Minimum eval set size before CI gates are meaningful. Below 200 examples, the pass rate variance between runs is high enough that a 2% regression gate will produce false positives on every other deploy. Start CI gating when you have at least 300 examples with reasonable coverage across difficulty levels.

---

Eval datasets are not a one-time artifact. They are a living system that needs the same maintenance discipline as your application code. The investment is front-loaded and unglamorous, but the alternative is shipping model updates while navigating by instinct. The teams that invest in this infrastructure early are the ones that can confidently say “this is better” instead of “it feels better” when a model or prompt changes.