Designing an A/B Testing Platform: Experiment Assignment, Statistical Analysis, and Feature Flag Integration at Scale

Most teams start with a third-party A/B testing tool. That works until it doesn’t: you hit rate limits during traffic spikes, you can’t control assignment logic for server-rendered pages, or you need deeper integration with your feature flag system. Eventually someone proposes building the platform in-house. This article walks through the core design decisions you’ll face when doing that.

The Assignment Problem

The central requirement of any A/B test is deterministic, consistent assignment: a given user must see the same variant on every request, across devices if they’re authenticated, and ideally without a database lookup on the critical path.

The standard solution is hashing. Given a user identifier and an experiment identifier, you hash them together and map the result to a bucket number.

import { createHash } from "crypto";

function getBucket(userId: string, experimentId: string, bucketCount = 10000): number {
  const hash = createHash("sha256")
    .update(`${experimentId}:${userId}`)
    .digest("hex");
  // Take the first 8 hex characters, parse as a 32-bit integer
  const numeric = parseInt(hash.slice(0, 8), 16);
  return numeric % bucketCount;
}

function assignVariant(
  userId: string,
  experiment: { id: string; variants: { name: string; weight: number }[] }
): string {
  const bucket = getBucket(userId, experiment.id);
  let cumulative = 0;
  for (const variant of experiment.variants) {
    cumulative += variant.weight * 100; // weight is 0-1, buckets are 0-9999
    if (bucket < cumulative) {
      return variant.name;
    }
  }
  // Fallback to control (should not happen if weights sum to 1)
  return experiment.variants[0].name;
}

A few things to note about this approach:

Prefix the hash input with the experiment ID. If you hash just the user ID, users in the top 50% of one experiment will also be in the top 50% of every other experiment. Prefixing with the experiment ID scrambles the bucket assignments independently per experiment.

Use SHA-256, not MD5. MD5 has known collision patterns and uneven distribution in certain ranges. SHA-256 is uniformly distributed for this use case. If you want something faster, MurmurHash3 is a reasonable alternative with good distribution properties.

10,000 buckets gives you fine-grained control. With 10,000 buckets, a 1% experiment uses 100 buckets, a 50/50 split uses 5,000 each. This is enough precision for most traffic allocation needs.

Mutual Exclusion and Experiment Layers

If you run enough experiments simultaneously, some users will end up in multiple experiments that affect overlapping parts of the product. That’s a problem: you can’t cleanly attribute a metric change to one experiment if the user experienced changes from another at the same time.

The standard fix is an experiment layer system. Layers partition the user population into non-overlapping slices. An experiment can only be assigned within one layer, and a user can only be in one experiment per layer.

interface Layer {
  id: string;
  name: string;
  // Disjoint bucket ranges assigned to this layer
  bucketRange: [number, number]; // e.g., [0, 2499]
}

interface Experiment {
  id: string;
  layerId: string;
  variants: { name: string; weight: number }[];
  // Buckets within the layer range that this experiment owns
  allocatedBuckets: number; // e.g., 1000 out of 2500
}

function assignWithLayers(
  userId: string,
  layers: Layer[],
  experiments: Experiment[]
): Map<string, string> {
  const assignments = new Map<string, string>();

  for (const layer of layers) {
    const layerBucket = getBucket(userId, layer.id);
    const [start, end] = layer.bucketRange;
    const rangeSize = end - start + 1;

    // Map the global bucket to the layer's range
    const localBucket = start + (layerBucket % rangeSize);

    // Find which experiment, if any, owns this bucket within the layer
    const experiment = findExperimentForBucket(experiments, layer.id, localBucket);
    if (experiment) {
      assignments.set(layer.id, assignVariant(userId, experiment));
    }
  }

  return assignments;
}

function findExperimentForBucket(
  experiments: Experiment[],
  layerId: string,
  bucket: number
): Experiment | null {
  let cursor = 0;
  for (const exp of experiments.filter((e) => e.layerId === layerId)) {
    if (bucket >= cursor && bucket < cursor + exp.allocatedBuckets) {
      return exp;
    }
    cursor += exp.allocatedBuckets;
  }
  return null;
}

This model gives you two key properties:

1. A user participates in at most one experiment per layer, eliminating cross-contamination within a layer.
2. Experiments in different layers are independent and can run concurrently on the same users.

The tradeoff is administrative overhead: you need to decide upfront which experiments go in which layer, and you need to reserve buckets. Teams typically create layers by product area (checkout, onboarding, search) and manage bucket allocation through a configuration service.

Experiment Configuration Service

The assignment logic above is stateless, but you need a service to manage experiment state: which experiments are active, what their bucket allocations are, and when they started and ended. This is usually a small CRUD API backed by a relational database, with a caching layer in front.

interface ExperimentConfig {
  id: string;
  name: string;
  layerId: string;
  status: "draft" | "running" | "paused" | "concluded";
  allocatedBuckets: number;
  variants: { name: string; weight: number }[];
  startedAt: string | null;
  concludedAt: string | null;
  // Targeting rules applied before bucket assignment
  targetingRules: TargetingRule[];
}

interface TargetingRule {
  attribute: string; // e.g., "country", "plan", "device"
  operator: "equals" | "in" | "not_in";
  values: string[];
}

function meetsTargetingRules(
  userAttributes: Record<string, string>,
  rules: TargetingRule[]
): boolean {
  return rules.every((rule) => {
    const value = userAttributes[rule.attribute];
    if (rule.operator === "equals") return value === rule.values[0];
    if (rule.operator === "in") return rule.values.includes(value);
    if (rule.operator === "not_in") return !rule.values.includes(value);
    return false;
  });
}

The targeting rules layer sits before bucket assignment. A user who doesn’t meet the targeting criteria is excluded from the experiment entirely. This is where you implement things like “only run this experiment for users on paid plans” or “only for users in the US and Canada.”

Cache the experiment configuration aggressively. Assignment happens on every request for authenticated users. Round-tripping to a config database on each assignment will add latency you can’t afford. A TTL of 30-60 seconds on the edge or application layer is typical. Changes to experiment config propagate within that window, which is acceptable for most cases.

Feature Flag Integration

A/B testing and feature flags solve related but distinct problems. Feature flags control what code runs. A/B tests control which users see which variant and measure the outcome. In practice, experiments are almost always implemented via feature flags: the experiment decides the variant, and the feature flag gates the corresponding code path.

type FlagValue = boolean | string | number;

interface FeatureFlag {
  key: string;
  defaultValue: FlagValue;
  overrides: { experimentId: string; variantName: string; value: FlagValue }[];
}

function evaluateFlag(
  flagKey: string,
  userId: string,
  userAttributes: Record<string, string>,
  flags: FeatureFlag[],
  experiments: ExperimentConfig[]
): FlagValue {
  const flag = flags.find((f) => f.key === flagKey);
  if (!flag) return false;

  // Check if any experiment overrides this flag
  for (const override of flag.overrides) {
    const experiment = experiments.find(
      (e) => e.id === override.experimentId && e.status === "running"
    );
    if (!experiment) continue;
    if (!meetsTargetingRules(userAttributes, experiment.targetingRules)) continue;

    const variant = assignVariant(userId, experiment);
    if (variant === override.variantName) {
      return override.value;
    }
  }

  return flag.defaultValue;
}

This model keeps the concerns separated. Experiments configure what percentage of users see which variant. Feature flags resolve what value a specific user gets for a specific flag key. The linkage is explicit: an experiment override says “when user is in variant B of experiment X, flag Y returns value Z.”

One important consequence: when an experiment concludes, you update the flag’s default value to the winning variant and remove the experiment override. The flag persists; the experiment is cleaned up. This prevents the common problem of shipping a feature but leaving dead experiment code in the codebase.

Statistical Analysis Engine

Assignment is the infrastructure layer. The analysis layer is where the science lives. Two approaches dominate: frequentist hypothesis testing and Bayesian estimation.

Frequentist: Z-Test for Proportions

For binary metrics (conversion rate, click-through rate), the two-proportion z-test is the standard approach.

interface VariantMetrics {
  visitors: number;
  conversions: number;
}

interface TestResult {
  pValue: number;
  relativeUplift: number;
  significant: boolean;
  confidenceInterval: [number, number]; // 95% CI on the difference
}

function zTestForProportions(
  control: VariantMetrics,
  treatment: VariantMetrics,
  alpha = 0.05
): TestResult {
  const pControl = control.conversions / control.visitors;
  const pTreatment = treatment.conversions / treatment.visitors;

  const pooled =
    (control.conversions + treatment.conversions) /
    (control.visitors + treatment.visitors);

  const se = Math.sqrt(
    pooled * (1 - pooled) * (1 / control.visitors + 1 / treatment.visitors)
  );

  const z = (pTreatment - pControl) / se;

  // Two-tailed p-value using normal approximation
  const pValue = 2 * (1 - normalCDF(Math.abs(z)));

  const diff = pTreatment - pControl;
  const ciMargin = 1.96 * se;

  return {
    pValue,
    relativeUplift: (diff / pControl) * 100,
    significant: pValue < alpha,
    confidenceInterval: [diff - ciMargin, diff + ciMargin],
  };
}

// Standard normal CDF approximation (Abramowitz and Stegun)
function normalCDF(x: number): number {
  const a1 = 0.254829592;
  const a2 = -0.284496736;
  const a3 = 1.421413741;
  const a4 = -1.453152027;
  const a5 = 1.061405429;
  const p = 0.3275911;
  const sign = x < 0 ? -1 : 1;
  x = Math.abs(x) / Math.sqrt(2);
  const t = 1 / (1 + p * x);
  const y = 1 - ((((a5 * t + a4) * t + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);
  return 0.5 * (1 + sign * y);
}

The frequentist approach has a well-understood failure mode: peeking. If you check results every hour and stop the experiment the moment p < 0.05, you dramatically inflate your false positive rate. The solution is to pre-commit to a minimum sample size before starting and not make decisions until you’ve reached it.

Bayesian: Beta-Binomial Model

The Bayesian approach treats conversion rates as probability distributions rather than point estimates. For binary metrics, a Beta distribution is the natural conjugate prior.

interface BayesianResult {
  probabilityTreatmentWins: number;
  expectedUplift: number;
  credibleInterval: [number, number]; // 95% HDI on the difference
}

function bayesianBetaBinomial(
  control: VariantMetrics,
  treatment: VariantMetrics,
  samples = 100_000
): BayesianResult {
  // Prior: Beta(1,1) = uniform, uninformative
  const controlAlpha = 1 + control.conversions;
  const controlBeta = 1 + control.visitors - control.conversions;
  const treatAlpha = 1 + treatment.conversions;
  const treatBeta = 1 + treatment.visitors - treatment.conversions;

  // Monte Carlo sampling
  let wins = 0;
  const diffs: number[] = [];

  for (let i = 0; i < samples; i++) {
    const pControl = sampleBeta(controlAlpha, controlBeta);
    const pTreat = sampleBeta(treatAlpha, treatBeta);
    const diff = pTreat - pControl;
    diffs.push(diff);
    if (diff > 0) wins++;
  }

  diffs.sort((a, b) => a - b);
  const lower = diffs[Math.floor(0.025 * samples)];
  const upper = diffs[Math.floor(0.975 * samples)];
  const mean = diffs.reduce((s, d) => s + d, 0) / samples;

  return {
    probabilityTreatmentWins: wins / samples,
    expectedUplift: (mean / (control.conversions / control.visitors)) * 100,
    credibleInterval: [lower, upper],
  };
}

// Box-Muller-based Beta sampler via Gamma distribution approximation
function sampleBeta(alpha: number, beta: number): number {
  const x = sampleGamma(alpha);
  const y = sampleGamma(beta);
  return x / (x + y);
}

function sampleGamma(shape: number): number {
  // Marsaglia and Tsang's method
  if (shape < 1) return sampleGamma(1 + shape) * Math.random() ** (1 / shape);
  const d = shape - 1 / 3;
  const c = 1 / Math.sqrt(9 * d);
  while (true) {
    let x: number, v: number;
    do {
      x = normalSample();
      v = 1 + c * x;
    } while (v <= 0);
    v = v ** 3;
    const u = Math.random();
    if (u < 1 - 0.0331 * x ** 4) return d * v;
    if (Math.log(u) < 0.5 * x ** 2 + d * (1 - v + Math.log(v))) return d * v;
  }
}

function normalSample(): number {
  let u = 0, v = 0;
  while (u === 0) u = Math.random();
  while (v === 0) v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}

The Bayesian result is more intuitive: “there is an 87% probability that the treatment outperforms control.” It also handles continuous monitoring more gracefully since you’re updating a posterior, not accumulating toward a fixed significance threshold.

Frequentist vs. Bayesian Tradeoffs

Dimension	Frequentist (z-test)	Bayesian (Beta-Binomial)
Interpretability	p-values require careful explanation	Probability of winning is intuitive
Peeking resistance	Requires pre-committed sample size	More robust to early stopping
Prior knowledge	None incorporated	Prior can encode historical rates
Multiple variants	Requires corrections (Bonferroni)	Naturally handles multi-armed
Tooling ecosystem	Mature, well-documented	Requires more implementation work
Speed of decision	Faster with small samples if signal is strong	Needs more samples for tight credible intervals

Most platforms ship frequentist first because the null hypothesis framework is what stakeholders recognize. Bayesian is added later for teams that run many concurrent experiments and want continuous monitoring.

Sample Ratio Mismatch

Sample Ratio Mismatch (SRM) is one of the most common and damaging problems in A/B testing. It occurs when the observed ratio of users across variants differs significantly from the intended allocation.

If you assign 50/50 but your data shows 48% in control and 52% in treatment, something is wrong with assignment, logging, or both. Any metric results from that experiment are suspect.

function detectSRM(
  observed: number[],
  expected: number[],
  alpha = 0.01
): { hasSRM: boolean; chiSquare: number; pValue: number } {
  const total = observed.reduce((s, n) => s + n, 0);
  const expectedTotal = expected.reduce((s, n) => s + n, 0);

  let chiSquare = 0;
  for (let i = 0; i < observed.length; i++) {
    const expectedCount = (expected[i] / expectedTotal) * total;
    chiSquare += (observed[i] - expectedCount) ** 2 / expectedCount;
  }

  // Chi-square CDF with (k-1) degrees of freedom
  const df = observed.length - 1;
  const pValue = 1 - chiSquareCDF(chiSquare, df);

  return {
    hasSRM: pValue < alpha,
    chiSquare,
    pValue,
  };
}

Common causes of SRM:

• Bot filtering applied post-assignment: bots are assigned but filtered from analysis, and they don’t distribute evenly across variants.
• Caching asymmetry: if variant A is cacheable and variant B is not, cached responses serve variant A users without triggering an assignment log event.
• Client-side redirect loops: a variant that causes an error and redirects loses those exposure events.
• Delayed experiment start: if the experiment starts mid-session, users in active sessions may not be reassigned uniformly.

SRM detection should run automatically before any statistical results are shown. If SRM is detected, block the results view and surface a warning. Showing results from a misconfigured experiment is worse than showing nothing.

Production Considerations

Assignment logging on the exposure event, not the page load. Log an assignment record the first time a user actually sees the variant, not when they’re assigned. If the variant is behind a UI element they never scroll to, logging on page load inflates your denominator and dilutes the metric.

Holdout groups. Reserve a small slice (1-5%) of users in a holdout that is excluded from all experiments. This gives you a clean comparison point for measuring the cumulative effect of all experiments running simultaneously.

Carryover effects. When you conclude an experiment, some users have been trained by the old variant. Ramp the winner gradually and watch for regression in the days after rollout, not just during the experiment.

Metric pipelines and delayed conversions. If your conversion event happens hours or days after the assignment event (e.g., subscription purchase after a free trial), your pipeline needs to join assignment records to conversion records by user ID. Store assignment records in a queryable store (a data warehouse, not just an event stream) from day one.

Experiment SDK versioning. If assignment logic changes (e.g., you change the hash function or bucket count), in-flight experiments will reassign users and invalidate the data. Version your assignment algorithm and pin running experiments to the version active when they started.

Putting It Together

The full platform has these layers:

1. Config service: stores experiment definitions, layer allocations, and targeting rules. Consumed by the assignment SDK via a cached read.
2. Assignment SDK: runs in your application layer (server-side or edge). Pure function: user ID + experiment config = variant. Emits an exposure event to the event pipeline.
3. Feature flag resolver: maps experiment variant assignments to flag values. Used by application code to gate behavior.
4. Event pipeline: receives exposure events and conversion events. Joins them by user ID and loads into the analysis store.
5. Analysis engine: runs statistical tests on demand or on a schedule. Detects SRM before surfacing results.
6. Results UI: presents variant comparison, significance, SRM warnings, and experiment metadata.

The most common mistake is treating this as a simple feature flag system with some stats bolted on. The statistical validity of your results depends on the integrity of the assignment pipeline, the correctness of your exposure logging, and the discipline to pre-commit to sample sizes before reading results. Get those right first, then layer in the analysis sophistication.

The hash-based assignment approach described here scales to millions of users per second with no shared state and no database lookups. That’s the right foundation to build on.