Deploying ML Models Behind Feature Flags: Gradual Rollouts, Shadow Mode, and A/B Testing for AI in Production

Deploying a new version of your recommendation model is not the same as deploying a new version of your API. Traditional deployments fail loudly: a syntax error causes a 500, the health check fails, the rollout stops. ML model failures are quieter. The model starts serving and responses look fine until you notice conversion dropped 8% over the last 72 hours, or that latency p99 crept from 180ms to 620ms under load. By that point you have already served the degraded model to every user.

Canary releases and blue-green switches were designed for systems where correctness is binary. ML outputs exist on a spectrum. You need an approach that runs models in parallel, collects real-world metrics before committing traffic, and reverts automatically when the numbers move in the wrong direction.

Feature flags are the right primitive for this. They already solve traffic routing at the user level. Combined with model-specific observability, they give you the control surface you need.

Why ML Deployments Are Different

Before getting into the implementation, it is worth being precise about what makes ML deployments harder than regular code deployments.

Non-determinism. The same input can produce different outputs across model versions, and both can be correct. There is no diff to review. You are comparing distributions, not states.

Latency variance. A new model might be 30% slower at p50 but 200% slower at p99 under concurrency. This only surfaces under real traffic patterns. Synthetic load tests rarely capture realistic request distributions.

Cost per invocation. LLM calls have a dollar cost attached to every request. A model that generates 40% longer outputs might produce marginally better results but double your monthly OpenAI bill. That tradeoff is invisible until you route real traffic.

Metric lag. Business metrics (conversion, retention, task completion) often lag the deployment by 24 to 72 hours. You cannot declare a deployment successful after 10 minutes.

Silent regressions. A model can degrade on a specific cohort (new users, mobile clients, a particular query category) while average metrics look flat. Percentage rollouts across a random sample may not surface this.

These properties change what your deployment infrastructure needs to do.

The Model Router

The core component is a model router: a layer that sits between your application and your model providers, reads feature flag state, and dispatches each request to the right model while collecting comparison metrics.

import LaunchDarkly from "@launchdarkly/node-server-sdk";

interface ModelRequest {
  input: string;
  context: Record<string, unknown>;
  userId: string;
}

interface ModelResponse {
  output: string;
  modelVersion: string;
  latencyMs: number;
  inputTokens: number;
  outputTokens: number;
  cost: number;
}

interface ModelConfig {
  name: string;
  provider: "openai" | "anthropic" | "bedrock" | "internal";
  modelId: string;
  costPerInputToken: number;
  costPerOutputToken: number;
}

const MODEL_REGISTRY: Record<string, ModelConfig> = {
  "recommendation-v1": {
    name: "recommendation-v1",
    provider: "openai",
    modelId: "gpt-4o-mini",
    costPerInputToken: 0.00000015,
    costPerOutputToken: 0.0000006,
  },
  "recommendation-v2": {
    name: "recommendation-v2",
    provider: "openai",
    modelId: "gpt-4o",
    costPerInputToken: 0.0000025,
    costPerOutputToken: 0.00001,
  },
};

class ModelRouter {
  private ldClient: LaunchDarkly.LDClient;
  private metrics: MetricsCollector;

  constructor(ldClient: LaunchDarkly.LDClient, metrics: MetricsCollector) {
    this.ldClient = ldClient;
    this.metrics = metrics;
  }

  async route(req: ModelRequest): Promise<ModelResponse> {
    const ldUser = { key: req.userId };

    // Read which model variant this user should get
    const modelVariant = await this.ldClient.variation(
      "recommendation-model-version",
      ldUser,
      "recommendation-v1" // default
    );

    const config = MODEL_REGISTRY[modelVariant];
    if (!config) {
      throw new Error(`Unknown model variant: ${modelVariant}`);
    }

    const start = Date.now();
    const response = await this.callModel(config, req);
    const latencyMs = Date.now() - start;

    // Emit per-model metrics for comparison
    this.metrics.record({
      modelVersion: modelVariant,
      latencyMs,
      inputTokens: response.inputTokens,
      outputTokens: response.outputTokens,
      cost: response.cost,
      userId: req.userId,
    });

    return { ...response, modelVersion: modelVariant, latencyMs };
  }

  private async callModel(
    config: ModelConfig,
    req: ModelRequest
  ): Promise<Omit<ModelResponse, "modelVersion" | "latencyMs">> {
    // Provider dispatch omitted for brevity
    // Each provider returns { output, inputTokens, outputTokens }
    const result = await dispatchToProvider(config, req);
    const cost =
      result.inputTokens * config.costPerInputToken +
      result.outputTokens * config.costPerOutputToken;

    return { ...result, cost };
  }
}

The flag value is a string matching a key in MODEL_REGISTRY. LaunchDarkly percentage rollouts, Unleash gradual rollouts, or any flag system that supports string variants with traffic allocation all work here. The router does not need to know whether the flag is set to 10% or 50%; the flag system handles that.

Shadow Mode

Shadow mode is the safest way to introduce a new model. The production model continues handling all real traffic. The new model receives a copy of every request, executes in parallel, but its output is discarded. Users never see it. You collect its metrics against real inputs for as long as you want before considering a traffic shift.

class ShadowModelRouter extends ModelRouter {
  async routeWithShadow(req: ModelRequest): Promise<ModelResponse> {
    const ldUser = { key: req.userId };

    // Check if shadow mode is enabled at all
    const shadowEnabled = await this.ldClient.boolVariation(
      "recommendation-shadow-mode",
      ldUser,
      false
    );

    // Always call the production model
    const productionResponse = await this.callNamedModel(
      "recommendation-v1",
      req
    );

    if (shadowEnabled) {
      // Fire-and-forget the shadow call. Do not await, do not block.
      this.runShadow(req, productionResponse).catch((err) => {
        // Log but never surface to the caller
        console.error("Shadow model error (non-fatal):", err);
      });
    }

    return productionResponse;
  }

  private async runShadow(
    req: ModelRequest,
    productionResponse: ModelResponse
  ): Promise<void> {
    const start = Date.now();

    try {
      const shadowResponse = await this.callNamedModel(
        "recommendation-v2",
        req
      );
      const latencyMs = Date.now() - start;

      // Record shadow metrics tagged so dashboards can separate them
      this.metrics.record({
        modelVersion: "recommendation-v2-shadow",
        latencyMs,
        inputTokens: shadowResponse.inputTokens,
        outputTokens: shadowResponse.outputTokens,
        cost: shadowResponse.cost,
        userId: req.userId,
        isShadow: true,
      });

      // Optionally compute output similarity for offline comparison
      this.metrics.recordOutputDiff({
        productionOutput: productionResponse.output,
        shadowOutput: shadowResponse.output,
        userId: req.userId,
      });
    } catch (err) {
      this.metrics.recordShadowFailure({
        modelVersion: "recommendation-v2",
        error: (err as Error).message,
      });
    }
  }
}

The critical detail is that the shadow call is fire-and-forget. The production path has a deadline; the shadow path does not. If the new model times out or throws, the user never knows.

Run shadow mode for at least a week of production traffic before moving to percentage rollout. This catches latency regressions under realistic concurrency, cost anomalies from longer outputs, and tail error rates that do not appear in synthetic tests.

Percentage Rollout with Automatic Rollback

Once shadow metrics look acceptable, you move to a live percentage rollout. Start at 5%, watch for 24 hours, then progress to 25%, 50%, and 100%. The flag system handles traffic splitting per user (consistent hashing ensures the same user always gets the same variant during the ramp).

The part most teams skip is automatic rollback. Manually monitoring dashboards and reacting fast enough when metrics degrade is hard to sustain. Instead, have a monitoring job that reads your metrics store and rolls back the flag if defined thresholds are crossed.

interface RollbackConfig {
  flagKey: string;
  safeVariant: string;
  checks: MetricCheck[];
  evaluationWindowMinutes: number;
}

interface MetricCheck {
  metric: string;
  variant: string;
  threshold: number;
  direction: "above" | "below"; // "above" = alert if value exceeds threshold
}

class AutoRollbackMonitor {
  private ldApi: LaunchDarklyApiClient;
  private metrics: MetricsStore;

  constructor(ldApi: LaunchDarklyApiClient, metrics: MetricsStore) {
    this.ldApi = ldApi;
    this.metrics = metrics;
  }

  async evaluate(config: RollbackConfig): Promise<void> {
    const windowStart = new Date(
      Date.now() - config.evaluationWindowMinutes * 60_000
    );

    for (const check of config.checks) {
      const value = await this.metrics.queryPercentile({
        metric: check.metric,
        variant: check.variant,
        since: windowStart,
        percentile: 95, // use p95, not average
      });

      const triggered =
        check.direction === "above"
          ? value > check.threshold
          : value < check.threshold;

      if (triggered) {
        console.warn(
          `Rollback triggered: ${check.metric} for ${check.variant} ` +
            `is ${value} (threshold: ${check.threshold})`
        );

        await this.rollback(config.flagKey, config.safeVariant);
        await this.notifyOnCall({
          reason: `Auto-rollback: ${check.metric} exceeded threshold`,
          variant: check.variant,
          metric: check.metric,
          value,
          threshold: check.threshold,
        });

        return; // One rollback is enough; exit after the first trigger
      }
    }
  }

  private async rollback(flagKey: string, safeVariant: string): Promise<void> {
    // Set 100% of traffic to the safe variant via the flag API
    await this.ldApi.updateFlagPercentage(flagKey, {
      [safeVariant]: 100,
    });
  }
}

// Example configuration
const rollbackConfig: RollbackConfig = {
  flagKey: "recommendation-model-version",
  safeVariant: "recommendation-v1",
  evaluationWindowMinutes: 30,
  checks: [
    {
      metric: "model.latency_ms",
      variant: "recommendation-v2",
      threshold: 800, // p95 latency must stay under 800ms
      direction: "above",
    },
    {
      metric: "model.error_rate",
      variant: "recommendation-v2",
      threshold: 0.02, // error rate must stay under 2%
      direction: "above",
    },
    {
      metric: "model.cost_per_request_usd",
      variant: "recommendation-v2",
      threshold: 0.05, // cost per request ceiling
      direction: "above",
    },
  ],
};

Use p95 latency in your checks, not average. Average latency is almost always misleading for ML inference: a fast median with a heavy tail is a real problem that averages will hide.

A/B Testing Models with Statistical Significance

Shadow mode and percentage rollouts tell you whether the new model is worse. A proper A/B test tells you whether it is measurably better on the metrics that matter to your business. These are different questions and require different setups.

For an A/B test, you need:

1. Random, consistent assignment (same user always in the same variant)
2. A primary metric with a defined minimum detectable effect
3. Enough traffic to reach statistical significance before declaring results
4. Guard rail metrics that can stop the experiment early if harm is detected

interface ExperimentConfig {
  experimentId: string;
  flagKey: string;
  variants: string[];
  primaryMetric: string;
  minimumDetectableEffect: number; // e.g., 0.05 for 5% improvement
  requiredSampleSizePerVariant: number;
  guardRailMetrics: string[];
}

class ModelExperimentAnalyzer {
  async analyze(
    config: ExperimentConfig,
    since: Date
  ): Promise<ExperimentResult> {
    const variantData = await Promise.all(
      config.variants.map((v) => this.fetchVariantMetrics(v, since))
    );

    const control = variantData[0];
    const treatment = variantData[1];

    if (
      control.sampleSize < config.requiredSampleSizePerVariant ||
      treatment.sampleSize < config.requiredSampleSizePerVariant
    ) {
      return { status: "insufficient_data", variantData };
    }

    const pValue = this.twoProportionZTest(
      control.primaryMetricValue,
      control.sampleSize,
      treatment.primaryMetricValue,
      treatment.sampleSize
    );

    const relativeEffect =
      (treatment.primaryMetricValue - control.primaryMetricValue) /
      control.primaryMetricValue;

    // Check guard rails regardless of primary metric result
    const guardRailBreach = await this.checkGuardRails(
      config.guardRailMetrics,
      treatment,
      since
    );

    return {
      status: pValue < 0.05 ? "significant" : "not_significant",
      pValue,
      relativeEffect,
      guardRailBreach,
      variantData,
    };
  }

  private twoProportionZTest(
    p1: number,
    n1: number,
    p2: number,
    n2: number
  ): number {
    const pooled = (p1 * n1 + p2 * n2) / (n1 + n2);
    const se = Math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2));
    const z = (p2 - p1) / se;
    // Two-tailed p-value from standard normal
    return 2 * (1 - normalCDF(Math.abs(z)));
  }

  private normalCDF(z: number): number {
    return (1 + erf(z / Math.sqrt(2))) / 2;
  }
}

Two practical notes on running model A/B tests in production:

Sample size first. Run a power calculation before you start. For a 5% minimum detectable effect on a metric with 15% baseline conversion, you need roughly 7,000 users per variant. If you get 500 users a day, that is two weeks minimum. Plan accordingly. Cutting the experiment short because the numbers “look good” is how you ship regressions with statistical cover.

Primary metric must be a business metric. Model quality scores (BLEU, ROUGE, human preference rates) are useful but they are not the thing you care about. The primary metric should be task completion rate, conversion, return visit rate, or whatever behavior the model is supposed to drive. A model can score higher on human preference evaluations and still degrade conversion if it produces longer outputs that slow down user workflows.

Tradeoffs

Rollout Strategy	Risk	Data Quality	Time to Full Deploy	Best For
Direct cutover	High (no recovery)	None	Immediate	Internal tools, low-stakes models
Shadow mode only	None (no live exposure)	High volume, zero business risk	Days to weeks	Models where output quality matters to revenue
Percentage rollout	Medium (partial exposure)	Good if ramp is slow enough	Days	Most production ML deployments
A/B test	Low (controlled)	Highest (statistical rigor)	Weeks	Business-critical models, measurable primary metrics
Shadow then A/B	Lowest	Highest	Weeks	LLM features with cost and quality uncertainty

Observability Layer

None of this works without the right metrics. At minimum, instrument these per model variant:

• Latency: p50, p95, p99. Export as histograms, not summaries, so you can aggregate across instances.
• Error rate: Failed requests, timeouts, and provider errors, separated by type.
• Token counts: Input and output token counts per request. Output token variance is the primary driver of cost surprises.
• Cost per request: Compute from token counts and per-model pricing. Sum over a rolling 24-hour window to catch cost regressions early.
• Output length: Character count of the model response. Longer outputs affect downstream UX even when content quality improves.
• Business metrics: Tag conversion events, task completions, and engagement signals with the model variant served to each user. This requires your analytics pipeline to consume the modelVersion field from the router response.

interface ModelMetricEvent {
  modelVersion: string;
  latencyMs: number;
  inputTokens: number;
  outputTokens: number;
  cost: number;
  userId: string;
  isShadow?: boolean;
  errorType?: string;
}

class MetricsCollector {
  record(event: ModelMetricEvent): void {
    const tags = {
      model_version: event.modelVersion,
      is_shadow: String(event.isShadow ?? false),
    };

    histogram("model.latency_ms", event.latencyMs, tags);
    counter("model.input_tokens", event.inputTokens, tags);
    counter("model.output_tokens", event.outputTokens, tags);
    gauge("model.cost_usd", event.cost, tags);

    if (event.errorType) {
      counter("model.errors", 1, { ...tags, error_type: event.errorType });
    }
  }

  recordOutputDiff(event: {
    productionOutput: string;
    shadowOutput: string;
    userId: string;
  }): void {
    // Normalized edit distance as a proxy for output similarity
    const similarity = computeSimilarity(
      event.productionOutput,
      event.shadowOutput
    );
    histogram("model.shadow_output_similarity", similarity, {
      model_version: "shadow",
    });
  }
}

Build a single dashboard with production and shadow metrics side by side. When you look at this dashboard daily during a shadow run, you are making a qualitative judgment about readiness: does the new model’s latency distribution look acceptable? Is the cost per request within budget? Are the error types different or more frequent?

That judgment, made with real data from real traffic, is what makes the eventual percentage rollout much lower risk.

Production Considerations

Flag cleanup. Once the new model reaches 100% and has been stable for two weeks, the old flag key is dead weight. Build a flag audit step into your deployment process so stale model routing flags do not accumulate.

Provider rate limits during ramp. Moving from 5% to 50% traffic is a roughly 10x increase to the new model provider. If the provider has per-minute rate limits and your router lacks retry and backoff, the ramp itself causes errors that look like model quality problems. Rate limit handling belongs in the router.

Consistent assignment under flag changes. Most flag SDKs hash the user ID and flag key to assign variants consistently. When you update the percentage split, some users move from control to treatment. If your business metrics have session-level state (like a multi-step checkout), a mid-session variant change corrupts experiment data. Lock assignment at session start for high-stakes experiments.

Prompt version and model version are separate axes. A flag for “which model” and a flag for “which prompt” are different concerns. Changing both simultaneously makes it impossible to attribute a metric change to either. Track prompt versions in your observability layer independently of model versions, and change one at a time.

The infrastructure overhead is real but one-time. Build the model router, connect it to your flag system, wire up the metrics, and you have a reusable deployment primitive for every model you will ever ship. The first time automatic rollback triggers at 2am, the setup cost looks like a reasonable investment.