Traffic Shaping in Production: Rate Limiting, Request Hedging, and Adaptive Concurrency for Resilient Services

Most services fail under load in a predictable way. A spike arrives, the service queues requests faster than it drains them, latency climbs, memory fills with waiting connections, and eventually it falls over. The failure mode is not the spike. It is the absence of any mechanism to say “no” before the queue grows unbounded.

Traffic shaping is the set of techniques that give a service that ability. Rate limiting controls request volume per caller. Adaptive concurrency caps in-flight requests based on observed performance. Request hedging reduces tail latency by firing duplicate requests when the first is slow. Priority classes let critical paths succeed under capacity constraints. Load shedding rejects traffic at the ingress before it reaches the application.

This article covers each technique with TypeScript implementations and notes on where Envoy and Cloudflare Workers apply.

Rate Limiting: Token Bucket vs Sliding Window

Token Bucket

The token bucket algorithm models a bucket that fills at a fixed rate up to a maximum capacity. Each request consumes one token. If the bucket is empty, the request is rejected or queued.

interface TokenBucketOptions {
  capacity: number;       // max tokens in bucket
  refillRate: number;     // tokens added per second
}

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(private options: TokenBucketOptions) {
    this.tokens = options.capacity;
    this.lastRefill = Date.now();
  }

  consume(count = 1): boolean {
    this.refill();
    if (this.tokens < count) return false;
    this.tokens -= count;
    return true;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;
    const added = elapsed * this.options.refillRate;
    this.tokens = Math.min(this.options.capacity, this.tokens + added);
    this.lastRefill = now;
  }
}

Token bucket handles burst traffic naturally. A user who has been idle accumulates tokens and can send a short burst. This matches real usage patterns better than a strict per-second cap.

The downside is that it allows bursts up to the full capacity. If capacity is 1000 tokens and a client fires 1000 requests at once, all 1000 pass. For abuse prevention, cap burst size separately from the sustained rate.

Sliding Window Counter

The sliding window algorithm divides time into discrete windows (often 1 second or 1 minute) and counts requests within the current window. The “sliding” variant weights the previous window based on how much of it overlaps with the current window, producing a smoother estimate.

class SlidingWindowRateLimiter {
  private currentCount = 0;
  private previousCount = 0;
  private windowStart = Date.now();

  constructor(
    private readonly limit: number,
    private readonly windowMs: number
  ) {}

  allow(): boolean {
    const now = Date.now();
    const elapsed = now - this.windowStart;

    if (elapsed >= this.windowMs) {
      // Rotate windows
      this.previousCount = elapsed < this.windowMs * 2 ? this.currentCount : 0;
      this.currentCount = 0;
      this.windowStart = now;
    }

    // Weight previous window by the fraction still in scope
    const overlap = (this.windowMs - (now - this.windowStart)) / this.windowMs;
    const estimated = this.previousCount * overlap + this.currentCount;

    if (estimated >= this.limit) return false;

    this.currentCount++;
    return true;
  }
}

Sliding window is more accurate than fixed window (which can allow 2x the limit across a boundary) and simpler to implement than token bucket in a distributed store. In Redis, use a Lua script to make the read-increment-write atomic:

const SLIDING_WINDOW_LUA = `
  local key = KEYS[1]
  local now = tonumber(ARGV[1])
  local window = tonumber(ARGV[2])
  local limit = tonumber(ARGV[3])
  local prev_key = key .. ":prev"

  local current = tonumber(redis.call("GET", key) or 0)
  local previous = tonumber(redis.call("GET", prev_key) or 0)

  local window_start = tonumber(redis.call("GET", key .. ":start") or now)
  local elapsed = now - window_start

  if elapsed >= window then
    redis.call("SET", prev_key, current, "PX", window * 2)
    redis.call("SET", key, 0, "PX", window * 2)
    redis.call("SET", key .. ":start", now, "PX", window * 2)
    current = 0
    previous = 0
    window_start = now
    elapsed = 0
  end

  local overlap = (window - elapsed) / window
  local estimated = previous * overlap + current

  if estimated >= limit then
    return 0
  end

  redis.call("INCR", key)
  redis.call("EXPIRE", key, math.ceil(window / 1000) * 2)
  return 1
`;

Where to Apply Rate Limits

Rate limiting belongs at multiple layers. At the ingress (Envoy, nginx, Cloudflare Workers), apply coarse limits: 1000 req/min per IP, 10,000 req/min per API key. At the application layer, apply fine-grained limits per user per expensive endpoint. At the service mesh, apply service-to-service limits to prevent one upstream from flooding a downstream.

Envoy’s rate limit filter delegates decisions to an external gRPC service, centralizing limit state in Redis. The Envoy config defines descriptors (headers, remote address, route) and the external service evaluates them.

In Cloudflare Workers, the rate-limiter binding (Workers Paid) handles distributed limits at the edge without managing Redis:

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const ip = request.headers.get("CF-Connecting-IP") ?? "unknown";
    const { success } = await env.RATE_LIMITER.limit({ key: ip });
    if (!success) return new Response("Too Many Requests", { status: 429 });
    return fetch(request);
  },
};

Request Hedging

A service with P50 latency of 20ms might have P99 latency of 500ms due to GC pauses, lock contention, or a slow replica. Users hitting those outliers experience a degraded product even when most requests are fast.

Request hedging (sometimes called speculative execution) sends a duplicate request to a different replica when the first is slow. The caller takes whichever response arrives first and cancels the other.

async function hedgedFetch(
  urls: string[],
  options: RequestInit,
  hedgeAfterMs: number
): Promise<Response> {
  const controllers: AbortController[] = [];

  const makeRequest = (url: string, index: number): Promise<Response> => {
    const controller = new AbortController();
    controllers[index] = controller;
    return fetch(url, { ...options, signal: controller.signal });
  };

  return new Promise((resolve, reject) => {
    let settled = false;

    const settle = (result: Response | Error) => {
      if (settled) return;
      settled = true;

      // Cancel all other in-flight requests
      controllers.forEach((c) => {
        try { c.abort(); } catch {}
      });

      if (result instanceof Error) reject(result);
      else resolve(result);
    };

    // Send first request immediately
    makeRequest(urls[0], 0).then(settle).catch(settle);

    // Hedge to second replica after delay
    if (urls.length > 1) {
      const hedgeTimer = setTimeout(() => {
        makeRequest(urls[1], 1).then(settle).catch(() => {});
      }, hedgeAfterMs);

      // Clean up timer if first request resolved quickly
      // (The settle callback handles the abort; this is just timer cleanup)
      Promise.race([makeRequest(urls[0], 0)]).finally(() => {
        clearTimeout(hedgeTimer);
      }).catch(() => {});
    }
  });
}

Set the hedge delay at roughly the P95 latency of the service. Hedging at P50 doubles request volume. Hedging at P99 adds only 1% overhead but only catches the very slowest outliers. P95 is the standard starting point.

Hedging only works on idempotent operations. Never hedge writes unless you are using an idempotency key pattern. For reads against caches, read replicas, or CDN origins, it is straightforward to apply. The cost is increased backend load, so target it at paths where tail latency is user-visible.

Adaptive Concurrency Limits

Fixed concurrency limits require manual tuning and go stale as capacity changes. Adaptive limits measure observed latency and adjust dynamically, following the same intuition as TCP congestion control.

Netflix’s concurrency-limits library popularized this with the Vegas algorithm (derived from TCP Vegas). When latency is near the observed minimum, the service has headroom and the limit can grow. When latency rises above the minimum, the service is approaching saturation and the limit should shrink.

class VegasConcurrencyLimiter {
  private limit: number;
  private inFlight = 0;
  private minRtt = Infinity;
  private readonly alpha: number;
  private readonly beta: number;
  private readonly probeInterval: number;
  private sampleCount = 0;

  constructor(
    private readonly minLimit = 5,
    private readonly maxLimit = 200,
    alpha = 3,
    beta = 6,
    probeInterval = 4
  ) {
    this.limit = minLimit;
    this.alpha = alpha;
    this.beta = beta;
    this.probeInterval = probeInterval;
  }

  tryAcquire(): (() => void) | null {
    if (this.inFlight >= this.limit) return null;

    this.inFlight++;
    const start = performance.now();

    return () => {
      const rtt = performance.now() - start;
      this.inFlight--;
      this.updateLimit(rtt);
    };
  }

  private updateLimit(rtt: number): void {
    this.minRtt = Math.min(this.minRtt, rtt);
    this.sampleCount++;

    if (this.sampleCount % this.probeInterval !== 0) return;

    const gradient = this.minRtt / rtt;
    const newLimit = Math.round(this.limit * gradient);
    const queueSize = this.limit - this.inFlight;
    const estimatedQueue = this.limit * (1 - gradient);

    if (estimatedQueue < this.alpha) {
      this.limit = Math.min(this.maxLimit, this.limit + 1);
    } else if (estimatedQueue > this.beta) {
      this.limit = Math.max(this.minLimit, newLimit);
    }
  }

  get currentLimit(): number {
    return this.limit;
  }
}

Usage:

const limiter = new VegasConcurrencyLimiter();

async function handleRequest(req: Request): Promise<Response> {
  const release = limiter.tryAcquire();
  if (!release) return new Response("Service Unavailable", { status: 503 });

  try {
    const result = await processRequest(req);
    release();
    return result;
  } catch (err) {
    release();
    throw err;
  }
}

Reset minRtt periodically (every few minutes) in long-running services to avoid anchoring to stale minimums as workload characteristics change.

Envoy’s built-in envoy.filters.http.adaptive_concurrency filter implements this at the proxy layer. If you are running behind Envoy, prefer that over an application-level implementation.

Priority-Based Traffic Classes

A checkout request matters more than a recommendation fetch. An authenticated user matters more than a crawler. Priority-based traffic shaping lets critical paths succeed when capacity is constrained.

The pattern requires tagging requests with a priority class at ingress, maintaining separate queues per class, and shedding lower-priority traffic first under saturation.

type Priority = "critical" | "high" | "normal" | "low";

interface PrioritizedRequest {
  req: Request;
  priority: Priority;
  enqueueTime: number;
}

const PRIORITY_WEIGHTS: Record<Priority, number> = {
  critical: 4,
  high: 3,
  normal: 2,
  low: 1,
};

class PriorityQueue {
  private queues: Map<Priority, PrioritizedRequest[]> = new Map([
    ["critical", []],
    ["high", []],
    ["normal", []],
    ["low", []],
  ]);

  enqueue(item: PrioritizedRequest): void {
    this.queues.get(item.priority)!.push(item);
  }

  dequeue(): PrioritizedRequest | undefined {
    // Drain higher-priority queues first
    for (const priority of ["critical", "high", "normal", "low"] as Priority[]) {
      const queue = this.queues.get(priority)!;
      if (queue.length > 0) return queue.shift();
    }
    return undefined;
  }

  shed(targetSize: number): void {
    // Drop low-priority items when queue is too deep
    const priorities: Priority[] = ["low", "normal", "high", "critical"];
    for (const priority of priorities) {
      const queue = this.queues.get(priority)!;
      while (this.totalSize() > targetSize && queue.length > 0) {
        queue.pop(); // Drop newest low-priority items first
      }
      if (this.totalSize() <= targetSize) break;
    }
  }

  totalSize(): number {
    let total = 0;
    for (const q of this.queues.values()) total += q.length;
    return total;
  }
}

Assign priority at the edge: authenticated vs. unauthenticated, endpoint criticality, customer tier, or an X-Request-Priority header set by the API gateway after auth. Never trust priority headers from external clients directly.

Load Shedding at the Ingress

Load shedding rejects requests before they consume meaningful resources. The key is shedding early with a clear 503 or 429 so callers can back off, rather than accepting requests that time out deep in the stack.

A CPU-based shedder:

import * as os from "os";

class LoadShedder {
  private cpuSamples: number[] = [];
  private readonly windowSize = 5;
  private lastSample = 0;

  shouldShed(priority: Priority): boolean {
    this.maybeSampleCpu();
    const avgCpu = this.averageCpu();

    // Always accept critical traffic
    if (priority === "critical") return false;

    // Shed in stages by priority as load increases
    if (avgCpu > 0.95) return priority !== "critical";
    if (avgCpu > 0.85) return priority === "low";
    if (avgCpu > 0.75) return priority === "low" && Math.random() > 0.5;

    return false;
  }

  private maybeSampleCpu(): void {
    const now = Date.now();
    if (now - this.lastSample < 1000) return;

    const cpus = os.cpus();
    const total = cpus.reduce((acc, cpu) => {
      const times = cpu.times;
      return acc + times.user + times.nice + times.sys + times.irq + times.idle;
    }, 0);
    const idle = cpus.reduce((acc, cpu) => acc + cpu.times.idle, 0);

    this.cpuSamples.push(1 - idle / total);
    if (this.cpuSamples.length > this.windowSize) this.cpuSamples.shift();
    this.lastSample = now;
  }

  private averageCpu(): number {
    if (this.cpuSamples.length === 0) return 0;
    return this.cpuSamples.reduce((a, b) => a + b) / this.cpuSamples.length;
  }
}

In Envoy, set max_pending_requests to limit queue depth and max_requests to cap concurrency per upstream cluster. When either limit is hit, Envoy returns 503 before the upstream sees the request.

Tradeoffs

Technique	Benefit	Cost	When to skip
Token bucket	Handles burst traffic naturally	State per rate limit key	When burst is not acceptable (strict QoS)
Sliding window	More accurate than fixed window	Slightly more complex distributed state	Low-traffic endpoints where accuracy does not matter
Request hedging	Cuts P99 latency for reads	Increased backend load	Non-idempotent operations, cost-sensitive backends
Adaptive concurrency	Self-tunes as capacity changes	Complexity, needs latency measurement	Services with highly variable request cost
Priority classes	Critical paths survive saturation	Requires priority tagging infrastructure	Systems where all traffic is equally important
Load shedding	Prevents collapse under overload	Callers must handle 503 gracefully	Backends with no retry logic in callers

A simple CRUD API behind an API gateway probably only needs ingress rate limiting and basic load shedding. Reserve adaptive concurrency and request hedging for services where tail latency is user-visible and you have the telemetry to tune them.

Production Considerations

Observability first. Instrument with P50, P95, and P99 latency histograms, in-flight request counts, queue depth, and shed/reject rates before tuning any limits. Without that telemetry, limit values are guesswork.

Graceful degradation. Rate limit and load shedding responses must include Retry-After or X-RateLimit-Reset headers. Callers that retry immediately on 429/503 amplify the very load you are trying to reduce.

Coordinated omission. When measuring latency for adaptive concurrency, do not sample only completed requests. Queued requests waiting for a slot should count toward the latency distribution, or you will underestimate the real P99.

Distributed state. Single-process token bucket and sliding window implementations do not work across replicas. Use Redis with atomic Lua scripts, or accept approximate limits where each replica enforces N/k of the global limit. The latter is simpler and sufficient for most abuse prevention scenarios.

Envoy vs. application layer. Proxy-level rate limiting and shedding reject requests before they reach the application, saving resources. Application-level implementations are necessary for limits that require auth context (per-user, per-tenant). Use both layers: coarse at the proxy, fine-grained in the application.

Closing

Traffic shaping is a layered set of controls that allow a service to degrade gracefully under pressure rather than collapse. The most important layer is whichever one is missing. Start with rate limiting at the ingress and load shedding with a queue depth limit, then work inward. Adaptive concurrency and request hedging are optimizations for services where you have the telemetry to validate their effect.