AI Agent Evaluation in Production: Testing Frameworks, Quality Gates, and Regression Detection

You ship a new prompt version. The agent feels better in your manual tests. You deploy. Three days later, a customer reports that the agent is truncating responses on a specific input class you never thought to test. You roll back and start debugging. The problem is not that you lack discipline. The problem is that you are applying a testing mental model designed for deterministic code to a system that is fundamentally non-deterministic. Unit tests pass or fail. Agent outputs sit on a quality spectrum, and the spectrum shifts every time you update a prompt, swap a model version, or change a tool’s return shape. Without a structured evaluation harness, you are flying blind.

Why Agent Evaluation Is Fundamentally Different

Traditional software testing rests on a simple contract: given the same input, the system produces the same output. You write assertions against that output. If the assertion passes, the test passes. This model breaks completely for AI agents.

Agents have four properties that invalidate standard testing approaches:

Non-deterministic outputs. Even at temperature 0, models can return semantically equivalent but textually different responses across providers, model versions, and prompt orderings. An exact-match assertion is useless.

Multi-step reasoning chains. An agent that uses tools executes a sequence of decisions: which tool to call, with what arguments, how to interpret the result, and when to stop. A failure at step 3 of a 7-step chain can look like a correct final answer, or a correct intermediate step can mask a wrong final answer. You need evaluation at the trace level, not just the response level.

Tool use and side effects. When an agent calls an external API, writes to a database, or sends an email, evaluation can no longer be a pure read operation. You need sandboxed environments that faithfully simulate tool behavior without triggering real side effects.

Emergent failure modes. Agents fail in ways that are hard to anticipate. Prompt injection through tool outputs. Hallucinated tool arguments. Over-reliance on a single tool when a different one would produce a better result. These failure modes do not map to any exception class or error code you can assert against.

Evaluation Harness Architecture

A production-grade evaluation harness has three layers that operate at different frequencies and costs.

┌─────────────────────────────────────────────────────┐
│                 Offline Eval Sets                    │
│  (run on every PR, deterministic + scored cases)    │
└────────────────────┬────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────┐
│               Online Monitoring                      │
│  (sampled production traces, async scoring)          │
└────────────────────┬────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────┐
│           Human-in-the-Loop Scoring                  │
│  (escalated low-confidence cases, ground truth set) │
└─────────────────────────────────────────────────────┘

Offline Eval Sets

Offline evals run against a curated dataset before any deployment. The dataset has two populations: golden cases with known-correct outputs (for regression detection) and adversarial cases that probe known failure modes. You score each case and enforce a minimum passing threshold in CI.

The typed interface for a scored test case:

interface EvalCase {
  id: string;
  description: string;
  input: AgentInput;
  expectedOutput?: string;          // optional: used for exact or fuzzy match
  expectedToolCalls?: ToolCallSpec[]; // optional: assert which tools were called
  scorers: Scorer[];                 // required: at least one scorer
  tags: string[];                    // used for sliced reporting
  weight?: number;                   // default 1.0, increase for critical cases
}

interface Scorer {
  name: string;
  type: "exact" | "contains" | "regex" | "llm-judge" | "custom";
  weight: number;                    // contribution to composite score
  config?: Record<string, unknown>;
}

interface EvalResult {
  caseId: string;
  passed: boolean;
  score: number;                     // 0.0 to 1.0
  scorerResults: ScorerResult[];
  traceId: string;
  latencyMs: number;
  toolCallSequence: ToolCall[];
  modelVersion: string;
  promptVersion: string;
  timestamp: string;
}

The LLM-as-judge scorer is the most versatile. You send the agent’s input, the agent’s output, and a rubric to a separate judge model (typically a more capable model than the one being evaluated). The judge returns a score and a rationale. Keep the rubric explicit and the scale simple: 0, 0.5, or 1.0. Granular scales introduce noise without adding information.

async function llmJudgeScorer(
  input: AgentInput,
  output: string,
  rubric: string,
  judgeModel: string = "gpt-4o"
): Promise<ScorerResult> {
  const prompt = `You are an evaluation judge. Score the following agent output.

INPUT:
${JSON.stringify(input, null, 2)}

AGENT OUTPUT:
${output}

RUBRIC:
${rubric}

Respond with JSON: { "score": 0 | 0.5 | 1, "rationale": "one sentence" }`;

  const response = await callModel(judgeModel, prompt);
  const parsed = JSON.parse(response);

  return {
    scorerName: "llm-judge",
    score: parsed.score,
    rationale: parsed.rationale,
    raw: response,
  };
}

One practical note: the judge model should not be the same model you are evaluating. If you are evaluating GPT-4o mini, judge with Claude Sonnet. If you are evaluating Claude Haiku, judge with GPT-4o. Cross-model judging reduces the risk of systematic blind spots that both the agent and its judge share.

Running the Harness

async function runEvalSuite(
  cases: EvalCase[],
  agent: AgentFn,
  options: EvalOptions
): Promise<EvalSuiteResult> {
  const results: EvalResult[] = [];

  for (const evalCase of cases) {
    const start = Date.now();
    const { output, toolCalls, traceId } = await agent(evalCase.input);
    const latencyMs = Date.now() - start;

    const scorerResults = await Promise.all(
      evalCase.scorers.map((scorer) =>
        runScorer(scorer, evalCase, output, toolCalls)
      )
    );

    const compositeScore = scorerResults.reduce(
      (sum, r, i) => sum + r.score * evalCase.scorers[i].weight,
      0
    ) / evalCase.scorers.reduce((sum, s) => sum + s.weight, 0);

    results.push({
      caseId: evalCase.id,
      passed: compositeScore >= options.passingThreshold,
      score: compositeScore,
      scorerResults,
      traceId,
      latencyMs,
      toolCallSequence: toolCalls,
      modelVersion: options.modelVersion,
      promptVersion: options.promptVersion,
      timestamp: new Date().toISOString(),
    });
  }

  const overallScore =
    results.reduce((sum, r) => {
      const evalCase = cases.find((c) => c.id === r.caseId);
      return sum + r.score * (evalCase?.weight ?? 1);
    }, 0) /
    cases.reduce((sum, c) => sum + (c.weight ?? 1), 0);

  return {
    results,
    overallScore,
    passed: overallScore >= options.minimumPassingScore,
    promptVersion: options.promptVersion,
    modelVersion: options.modelVersion,
    runAt: new Date().toISOString(),
  };
}

Online Monitoring

Offline evals catch regressions before deployment. Online monitoring catches the failure modes you never wrote test cases for. The architecture is straightforward: intercept a sample of production traces, score them asynchronously using the same scorer infrastructure, and emit the scores as metrics.

async function scoreProductionTrace(
  trace: ProductionTrace,
  scoringRules: ScoringRule[]
): Promise<void> {
  const applicableRules = scoringRules.filter((rule) =>
    rule.tagFilter.every((tag) => trace.tags.includes(tag))
  );

  if (applicableRules.length === 0) return;

  const scoredRules = await Promise.all(
    applicableRules.map(async (rule) => {
      const result = await runScorer(
        rule.scorer,
        { input: trace.input, expectedOutput: undefined, scorers: [], tags: [], id: trace.traceId },
        trace.output,
        trace.toolCalls
      );
      return { ruleName: rule.name, score: result.score };
    })
  );

  await metrics.emit("agent.production.score", {
    traceId: trace.traceId,
    scores: scoredRules,
    promptVersion: trace.promptVersion,
    modelVersion: trace.modelVersion,
    latencyMs: trace.latencyMs,
    tags: trace.tags,
  });

  // Escalate low-confidence traces for human review
  const lowestScore = Math.min(...scoredRules.map((s) => s.score));
  if (lowestScore < 0.4) {
    await escalationQueue.push({ trace, scores: scoredRules, reason: "low-confidence" });
  }
}

Sampling rate matters. At low traffic, score everything. At high traffic, stratified sampling by input category works better than uniform random sampling. You want your sample to represent the full distribution of inputs, not just the most common ones. Rare but important cases (edge inputs, adversarial inputs) should be oversampled.

Human-in-the-Loop Scoring

Automated scoring has a ceiling. LLM judges make systematic errors on certain rubrics, and exact-match scorers miss semantically valid alternatives. Human scoring is the ground truth, but it is expensive. The right approach is to use humans for two specific purposes: calibrating your automated scorers and building the ground truth dataset for your most critical cases.

Calibration workflow: sample 50-100 cases monthly, have engineers rate them, compute correlation between your automated scorer’s scores and human scores. If correlation drops below 0.7, your rubric or your judge prompt needs work.

Quality Gates in CI/CD

A quality gate blocks deployment when the eval suite score drops below a threshold or when regression is detected on a specific slice.

# .github/workflows/agent-eval.yml
- name: Run eval suite
  run: npx ts-node scripts/run-evals.ts
  env:
    PROMPT_VERSION: ${{ github.sha }}
    MODEL_VERSION: gpt-4o-mini-2024-07-18
    MIN_OVERALL_SCORE: "0.85"
    MIN_SLICE_SCORES: |
      {"adversarial": 0.75, "tool-use": 0.90, "edge-cases": 0.80}
    REGRESSION_THRESHOLD: "0.05"
    BASELINE_RUN_ID: ${{ vars.LAST_PASSING_RUN_ID }}

The gate logic needs two checks: an absolute threshold (overall score must exceed X) and a relative check (score must not drop more than Y from the previous baseline). A score of 0.86 against a baseline of 0.92 might pass the absolute threshold but still represent a significant regression.

function evaluateQualityGate(
  current: EvalSuiteResult,
  baseline: EvalSuiteResult | null,
  config: QualityGateConfig
): QualityGateResult {
  const failures: string[] = [];

  // Absolute threshold check
  if (current.overallScore < config.minimumPassingScore) {
    failures.push(
      `Overall score ${current.overallScore.toFixed(3)} below minimum ${config.minimumPassingScore}`
    );
  }

  // Per-slice threshold checks
  for (const [tag, minScore] of Object.entries(config.minimumSliceScores ?? {})) {
    const sliceResults = current.results.filter((r) =>
      current.cases?.find((c) => c.id === r.caseId)?.tags.includes(tag)
    );
    if (sliceResults.length === 0) continue;

    const sliceScore =
      sliceResults.reduce((sum, r) => sum + r.score, 0) / sliceResults.length;

    if (sliceScore < minScore) {
      failures.push(
        `Slice "${tag}" score ${sliceScore.toFixed(3)} below minimum ${minScore}`
      );
    }
  }

  // Regression check against baseline
  if (baseline) {
    const drop = baseline.overallScore - current.overallScore;
    if (drop > config.regressionThreshold) {
      failures.push(
        `Score dropped ${drop.toFixed(3)} from baseline ${baseline.overallScore.toFixed(3)} (threshold: ${config.regressionThreshold})`
      );
    }

    // Case-level regression: flag individual cases that regressed
    for (const baselineResult of baseline.results) {
      const currentResult = current.results.find(
        (r) => r.caseId === baselineResult.caseId
      );
      if (!currentResult) continue;

      const caseDrop = baselineResult.score - currentResult.score;
      if (caseDrop > config.caseLevelRegressionThreshold) {
        failures.push(
          `Case "${baselineResult.caseId}" regressed by ${caseDrop.toFixed(3)}`
        );
      }
    }
  }

  return {
    passed: failures.length === 0,
    failures,
    currentScore: current.overallScore,
    baselineScore: baseline?.overallScore ?? null,
  };
}

Keep the baseline pinned to the last production deployment, not the last passing CI run. A sequence of small regressions in CI, each below the threshold, can compound into a large production regression. The baseline should represent what users are currently experiencing.

Tradeoffs

Approach	Cost	Coverage	Latency	Maintenance
Exact match offline evals	Very low	Low (brittle)	Fast	Low
LLM-judge offline evals	Medium (API cost)	High	Medium	Medium (rubric drift)
Online sampling + async scoring	Medium (infra)	Broad (real traffic)	Zero (async)	Medium
Human scoring	High (time)	Very high	Slow	High
A/B testing agents	High (complexity)	Real-world	None	High

The right investment level depends on the cost of a wrong agent output. A customer support agent that occasionally gives slightly suboptimal answers is very different from an agent that takes financial actions. Calibrate your harness complexity to the consequence of failure.

A/B Testing Agents in Production

When you have two prompt versions and the offline evals show similar scores, production A/B testing resolves the tie. The setup is standard: route a percentage of traffic to each variant, tag traces with the variant identifier, and compare scores and user-observable outcomes across variants.

async function routeToAgentVariant(
  request: AgentRequest,
  variants: AgentVariant[],
  assignmentStrategy: "hash" | "random" = "hash"
): Promise<{ variant: AgentVariant; traceTag: string }> {
  const variantIndex =
    assignmentStrategy === "hash"
      ? hashToIndex(request.userId, variants.length) // sticky assignment per user
      : Math.floor(Math.random() * variants.length);

  const variant = variants[variantIndex];

  return {
    variant,
    traceTag: `variant:${variant.id}`,
  };
}

function hashToIndex(userId: string, buckets: number): number {
  // FNV-1a hash, deterministic per userId
  let hash = 2166136261;
  for (let i = 0; i < userId.length; i++) {
    hash ^= userId.charCodeAt(i);
    hash = (hash * 16777619) >>> 0;
  }
  return hash % buckets;
}

Hash-based assignment is important for multi-turn agents. If a user’s session spans multiple requests, you want them on the same variant throughout. Random assignment per request creates incoherent experiences and confounds your results.

For significance testing: you need enough production traces to distinguish real effects from noise. A 3% score difference on 50 traces means nothing. A 3% difference on 5,000 traces with p < 0.05 means something. Set a minimum sample size before looking at results to avoid stopping early on noise.

Production Monitoring Patterns

Once your agent is deployed, three monitoring patterns cover most of the reliability surface area.

Score distribution tracking. Plot the histogram of production scores over time, not just the mean. A mean score of 0.85 looks fine until you notice the distribution has developed a long left tail at 0.2-0.3. The mean masks the tail; the tail is where your worst user experiences live.

Prompt version attribution. Every production trace must carry a prompt version identifier. When you deploy a new prompt, you should be able to query “what is the score distribution for prompt version X vs. Y over the last 24 hours?” without joining across multiple systems.

Tool call anomaly detection. Track the distribution of which tools your agent calls and in what sequence. A sudden shift in tool call patterns is often a leading indicator of a prompt regression before the quality scores degrade visibly. If your agent normally calls search_knowledge_base in 80% of traces and that drops to 30% after a deployment, something changed.

async function detectToolCallAnomalies(
  recentTraces: ProductionTrace[],
  baselineDistribution: ToolCallDistribution,
  alertThreshold: number = 0.15
): Promise<ToolCallAnomaly[]> {
  const recentDistribution = computeToolCallDistribution(recentTraces);
  const anomalies: ToolCallAnomaly[] = [];

  for (const [toolName, baselineRate] of Object.entries(baselineDistribution)) {
    const recentRate = recentDistribution[toolName] ?? 0;
    const drift = Math.abs(recentRate - baselineRate);

    if (drift > alertThreshold) {
      anomalies.push({
        toolName,
        baselineRate,
        recentRate,
        drift,
        direction: recentRate > baselineRate ? "increase" : "decrease",
      });
    }
  }

  return anomalies.sort((a, b) => b.drift - a.drift);
}

Latency as a quality signal. Agents that are struggling often take longer. A prompt that causes the model to loop through reasoning steps before giving up manifests as a latency spike before you see a quality score drop. Track p95 and p99 latency by prompt version and use spikes as early warning signals.

Putting It Together

Agent evaluation is not a checkbox you complete before shipping. It is a system that runs continuously: offline evals on every PR, online scoring on sampled production traffic, human review of escalated cases, and anomaly detection on tool call patterns and latency. The quality gate in CI catches the regressions you can anticipate. The production monitoring catches the ones you could not. The human review loop closes the gap between your automated scorers and ground truth.

The investment scales with the risk profile of your agent. Start with a small golden set and a basic LLM judge. Add sliced thresholds as you discover which input categories matter most. Add online monitoring when you have enough production traffic to make sampling meaningful. The harness you build is not a testing artifact. It is the operational instrument that tells you whether the system you shipped is the system your users are actually experiencing.