Designing a Load Shedding System: Priority Queues, Admission Control, and Graceful Degradation Under Extreme Traffic

Rate limiting protects your system from individual bad actors. Load shedding protects your system from itself.

Rate limiting is per-client: you track how many requests a given API key or IP address has sent and reject them when they exceed a threshold. It is a fairness mechanism. Load shedding is system-wide: when the service cannot sustain its current request rate regardless of who is sending, you decide which work to drop to preserve the work that matters. It is a survival mechanism.

Most systems implement rate limiting. Far fewer implement intentional load shedding. The result is predictable: under extreme traffic, every request gets a little slower until nothing gets a response at all. A properly designed load shedding layer inverts this. You sacrifice some requests deliberately so the rest complete with acceptable latency.

This article covers request classification and priority assignment, admission control algorithms, circuit breaker integration, client-side cooperation, and the observability layer that tells you whether shedding is working.

Why Load Shedding Is Different from Rate Limiting

Rate limiting answers: “Has this client sent too many requests?” The decision is made at the edge, before any work is done, and applies uniformly regardless of system load.

Load shedding answers: “Does the system have capacity to serve this request?” The decision depends on real-time state: CPU saturation, queue depth, downstream latency, memory pressure. A request that would be accepted at 40% CPU utilization might be rejected at 90%.

Rate limiting is also symmetric: it treats all requests from a given client the same. Load shedding should be asymmetric by design. A health check, a search query, and a payment confirmation are not equivalent. When capacity is scarce, you want to make explicit decisions about which work matters.

The failure mode without load shedding is a latency spiral. As the system approaches capacity, each request takes longer due to resource contention. Longer requests hold connections and threads open longer, reducing effective throughput further. Upstream services retry, compounding the load. Within minutes, median latency crosses client timeout thresholds and a system handling 10,000 requests per second ends up handling none of them.

Load shedding cuts this spiral. You accept some degradation in served request volume to preserve latency for the requests you do serve.

Request Classification and Priority Assignment

The first design decision is how to classify requests. Classification determines priority, and priority determines what gets shed first.

A simple three-tier model covers most production scenarios:

enum RequestPriority {
  CRITICAL = 0,   // payment, auth, core API
  STANDARD = 1,   // reads, searches, user content
  BACKGROUND = 2, // analytics, non-blocking batch
}

interface RequestContext {
  id: string;
  priority: RequestPriority;
  enqueuedAt: number;
  timeoutMs: number;
  clientId: string;
  endpoint: string;
}

function classifyRequest(req: IncomingRequest): RequestPriority {
  // Explicit header from trusted internal callers
  const explicit = req.headers["x-request-priority"];
  if (explicit === "critical") return RequestPriority.CRITICAL;

  // Endpoint-based classification
  if (CRITICAL_ENDPOINTS.has(req.path)) return RequestPriority.CRITICAL;
  if (BACKGROUND_ENDPOINTS.has(req.path)) return RequestPriority.BACKGROUND;

  return RequestPriority.STANDARD;
}

const CRITICAL_ENDPOINTS = new Set([
  "/api/payments",
  "/api/auth/token",
  "/api/orders/confirm",
]);

const BACKGROUND_ENDPOINTS = new Set([
  "/api/analytics/events",
  "/api/recommendations/refresh",
  "/api/exports",
]);

Classification should happen at the edge, before any queuing. The priority level travels with the request context through every layer.

Do not trust client-supplied priority headers from external callers. Classify based on authenticated identity and endpoint, then stamp the priority internally. If you let clients self-declare priority, every client declares CRITICAL.

Admission Control: The Token Bucket with Priority Tiers

A token bucket is the right base primitive for admission control. Tokens refill at a constant rate. Each request consumes tokens. When the bucket is empty, new requests are rejected rather than queued.

The key extension for load shedding is separate buckets per priority tier with different refill rates and capacities:

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private readonly capacity: number,
    private readonly refillRate: number, // tokens per millisecond
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  tryConsume(count: number = 1): boolean {
    this.refill();
    if (this.tokens < count) return false;
    this.tokens -= count;
    return true;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    const added = elapsed * this.refillRate;
    this.tokens = Math.min(this.capacity, this.tokens + added);
    this.lastRefill = now;
  }

  get fillRatio(): number {
    this.refill();
    return this.tokens / this.capacity;
  }
}

class PriorityAdmissionController {
  private readonly buckets: Map<RequestPriority, TokenBucket>;

  constructor(private readonly systemCapacity: number) {
    this.buckets = new Map([
      [RequestPriority.CRITICAL, new TokenBucket(
        Math.floor(systemCapacity * 0.5),  // 50% of capacity reserved
        systemCapacity * 0.5 / 1000,       // refill over 1 second
      )],
      [RequestPriority.STANDARD, new TokenBucket(
        Math.floor(systemCapacity * 0.4),
        systemCapacity * 0.4 / 1000,
      )],
      [RequestPriority.BACKGROUND, new TokenBucket(
        Math.floor(systemCapacity * 0.1),
        systemCapacity * 0.1 / 1000,
      )],
    ]);
  }

  admit(ctx: RequestContext): boolean {
    const bucket = this.buckets.get(ctx.priority)!;

    // Under extreme load, allow CRITICAL to borrow from STANDARD bucket
    if (!bucket.tryConsume()) {
      if (ctx.priority === RequestPriority.CRITICAL) {
        const standardBucket = this.buckets.get(RequestPriority.STANDARD)!;
        return standardBucket.tryConsume();
      }
      return false;
    }

    return true;
  }

  // Global fill ratios for observability
  getBucketState(): Record<string, number> {
    return {
      critical: this.buckets.get(RequestPriority.CRITICAL)!.fillRatio,
      standard: this.buckets.get(RequestPriority.STANDARD)!.fillRatio,
      background: this.buckets.get(RequestPriority.BACKGROUND)!.fillRatio,
    };
  }
}

The bucket ratios (50/40/10 above) are a starting point. Your actual split depends on traffic mix and what “critical” means for your system. Make these configurable without a deploy.

Weighted Fair Queuing for Burst Absorption

Token buckets handle steady-state admission. For services with an internal work queue before processing, weighted fair queuing (WFQ) ensures high-priority work is dequeued first without completely starving lower tiers.

interface QueuedRequest {
  ctx: RequestContext;
  resolve: (result: ProcessingResult) => void;
  reject: (err: Error) => void;
}

class WeightedFairQueue {
  private readonly queues: Map<RequestPriority, QueuedRequest[]>;
  private readonly weights: Map<RequestPriority, number>;
  private readonly maxSize: number;
  private currentSize: number = 0;

  constructor(maxSize: number) {
    this.maxSize = maxSize;
    this.queues = new Map([
      [RequestPriority.CRITICAL, []],
      [RequestPriority.STANDARD, []],
      [RequestPriority.BACKGROUND, []],
    ]);
    this.weights = new Map([
      [RequestPriority.CRITICAL, 5],
      [RequestPriority.STANDARD, 3],
      [RequestPriority.BACKGROUND, 1],
    ]);
  }

  enqueue(item: QueuedRequest): boolean {
    if (this.currentSize >= this.maxSize) {
      // Shed the lowest-priority item to make room for higher-priority ones
      if (!this.shedLowest(item.ctx.priority)) {
        return false;
      }
    }

    const queue = this.queues.get(item.ctx.priority)!;
    queue.push(item);
    this.currentSize++;
    return true;
  }

  // Weighted round-robin dequeue
  dequeue(): QueuedRequest | undefined {
    const priorities = [
      RequestPriority.CRITICAL,
      RequestPriority.STANDARD,
      RequestPriority.BACKGROUND,
    ];

    for (const priority of priorities) {
      const weight = this.weights.get(priority)!;
      const queue = this.queues.get(priority)!;

      if (queue.length > 0) {
        // Drain up to `weight` items from this tier before moving on
        const item = queue.shift()!;
        this.currentSize--;
        return item;
      }
    }

    return undefined;
  }

  private shedLowest(incomingPriority: RequestPriority): boolean {
    // Walk tiers from lowest upward, find something to evict
    const tiersToShed = [
      RequestPriority.BACKGROUND,
      RequestPriority.STANDARD,
      RequestPriority.CRITICAL,
    ].filter((p) => p > incomingPriority);

    for (const tier of tiersToShed) {
      const queue = this.queues.get(tier)!;
      if (queue.length > 0) {
        const evicted = queue.pop()!; // shed the tail (oldest item in this tier)
        evicted.reject(new Error("REQUEST_SHED_CAPACITY"));
        this.currentSize--;
        return true;
      }
    }

    return false; // nothing lower-priority to shed
  }

  get depth(): number {
    return this.currentSize;
  }
}

shedLowest evicts the lowest-priority tail item when the queue is full. A new CRITICAL request displaces a BACKGROUND item. A new BACKGROUND request is rejected outright if nothing lower-priority exists. Under sustained overload, background work drains first, standard second, and critical is protected for as long as possible.

Circuit Breaker Integration

A circuit breaker tracks downstream health and trips when error rates cross a threshold. Load shedding and circuit breakers are complementary, not alternatives.

The integration point: when a circuit is open, requests routed to that downstream should be shed at the admission layer rather than allowed to queue and fail. This preserves the queue capacity for requests that can actually complete.

type CircuitState = "closed" | "open" | "half-open";

class CircuitBreaker {
  private state: CircuitState = "closed";
  private failures: number = 0;
  private successes: number = 0;
  private lastTrip: number = 0;

  constructor(
    private readonly failureThreshold: number,
    private readonly recoveryMs: number,
    private readonly halfOpenProbes: number,
  ) {}

  canProcess(): boolean {
    if (this.state === "closed") return true;

    if (this.state === "open") {
      if (Date.now() - this.lastTrip > this.recoveryMs) {
        this.state = "half-open";
        this.successes = 0;
        return true; // allow one probe
      }
      return false;
    }

    // half-open: allow limited probes
    return this.successes < this.halfOpenProbes;
  }

  recordSuccess(): void {
    if (this.state === "half-open") {
      this.successes++;
      if (this.successes >= this.halfOpenProbes) {
        this.state = "closed";
        this.failures = 0;
      }
    } else {
      this.failures = Math.max(0, this.failures - 1);
    }
  }

  recordFailure(): void {
    this.failures++;
    if (this.state === "half-open") {
      this.trip();
    } else if (this.failures >= this.failureThreshold) {
      this.trip();
    }
  }

  private trip(): void {
    this.state = "open";
    this.lastTrip = Date.now();
    this.successes = 0;
  }

  get currentState(): CircuitState {
    return this.state;
  }
}

// In the admission controller
class AdmissionGate {
  constructor(
    private readonly admission: PriorityAdmissionController,
    private readonly breakers: Map<string, CircuitBreaker>,
  ) {}

  admit(ctx: RequestContext, downstream: string): "accept" | "shed" | "open-circuit" {
    const breaker = this.breakers.get(downstream);
    if (breaker && !breaker.canProcess()) {
      return "open-circuit"; // immediate shed, no queuing
    }

    if (!this.admission.admit(ctx)) {
      return "shed";
    }

    return "accept";
  }
}

The key difference between shed and open-circuit in the response: you may want to return different HTTP status codes (503 vs 429) and different Retry-After headers. An open circuit suggests a longer backoff. A capacity shed suggests a shorter retry.

Client-Side Cooperation

Load shedding only works well when clients cooperate. A client that ignores 503 responses and retries immediately converts a controlled shed into a retry storm.

The contracts you need from clients:

Respect Retry-After. Return this header on every shed response. For capacity sheds, a value between 1 and 5 seconds is typical. For open circuits, consider 30 to 60 seconds.

Implement exponential backoff with jitter. Fixed-interval retries synchronize across thousands of clients and create regular load spikes. Jitter breaks the synchronization.

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  opts: { maxAttempts: number; baseMs: number; maxMs: number },
): Promise<T> {
  let attempt = 0;

  while (attempt < opts.maxAttempts) {
    try {
      return await fn();
    } catch (err: any) {
      if (err.status === 503 || err.status === 429) {
        const retryAfterMs = parseRetryAfter(err.headers?.["retry-after"]) ?? 0;
        const backoff = Math.min(
          opts.maxMs,
          opts.baseMs * Math.pow(2, attempt),
        );
        const jitter = Math.random() * backoff * 0.3;
        const delay = Math.max(retryAfterMs, backoff + jitter);

        await sleep(delay);
        attempt++;
        continue;
      }
      throw err;
    }
  }

  throw new Error(`Exhausted ${opts.maxAttempts} retry attempts`);
}

function parseRetryAfter(header: string | undefined): number | null {
  if (!header) return null;
  const seconds = parseInt(header, 10);
  return isNaN(seconds) ? null : seconds * 1000;
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

Implement client-side circuit breakers. A client that tracks its own error rate and stops sending when it crosses a threshold reduces load before the server has to enforce shedding. Server-side circuit breakers are the enforcement mechanism; client-side ones are a layer of defense before you need enforcement.

Observability for Shedding Decisions

You cannot tune a load shedding system you cannot observe. The metrics you need:

Shed rate by tier: How many requests are being shed per second, broken down by priority tier? A rising shed rate in BACKGROUND is expected under load. A rising shed rate in CRITICAL means your capacity reservation is too small or the incoming critical volume is genuinely overloading the system.

Admission decision breakdown: Record the admission decision (accepted, shed-capacity, shed-circuit, shed-timeout) as a metric dimension. This separates capacity problems from dependency problems.

Queue depth per tier: Depth spikes indicate the admission controller is accepting work faster than the system can process it. A sustained depth in CRITICAL means your downstream is underprovisioned or degraded.

Token bucket fill ratio: A bucket perpetually near zero means the refill rate is too low for incoming traffic. You need this before you can reason about any configuration change.

class InstrumentedAdmissionGate {
  private counters = {
    accepted: 0,
    shedCapacity: 0,
    shedCircuit: 0,
  };

  admit(
    ctx: RequestContext,
    downstream: string,
    emit: (metric: string, tags: Record<string, string>, value: number) => void,
  ): "accept" | "shed" | "open-circuit" {
    const decision = this.gate.admit(ctx, downstream);
    const priorityLabel = RequestPriority[ctx.priority].toLowerCase();

    if (decision === "accept") {
      this.counters.accepted++;
      emit("admission.accepted", { priority: priorityLabel }, 1);
    } else if (decision === "shed") {
      this.counters.shedCapacity++;
      emit("admission.shed", { reason: "capacity", priority: priorityLabel }, 1);
    } else {
      this.counters.shedCircuit++;
      emit("admission.shed", { reason: "circuit", downstream, priority: priorityLabel }, 1);
    }

    return decision;
  }

  constructor(private readonly gate: AdmissionGate) {}
}

The most useful alert is not “shed rate is above X.” It is “CRITICAL shed rate is non-zero.” That is always worth investigating. Background and standard shed rates being non-zero under high traffic is expected behavior.

Production Tradeoffs

Approach	Throughput under overload	Request fairness	Complexity	When to use
No shedding	Collapses (latency spiral)	None	Low	Never, intentionally
Uniform drop (random)	Stable	Equal degradation for all	Low	Stateless services without priority differences
Priority-tier shedding	Stable, differentiated	Proportional to tier	Medium	Services with clear request importance hierarchy
Weighted fair queuing	Stable, burst-tolerant	Weighted by tier	Medium-high	Services with variable processing time per request
Adaptive shedding (CPU/latency signal)	Optimal	Depends on signal quality	High	Services with reliable real-time load signals

A few production-specific observations:

Start with conservative bucket ratios. You can always increase capacity allocation for a tier; decreasing it during an incident is harder to reason about. Start CRITICAL at 50%, STANDARD at 40%, BACKGROUND at 10%.

Make everything configurable at runtime. Changing the threshold that triggers shedding during an incident should not require a deploy. Store these in your config service.

Shedding is not a replacement for capacity planning. If you are shedding STANDARD traffic at baseline load, you have an underprovisioning problem. Shedding should be an emergency measure, not a steady-state traffic management tool.

Handle the timeout problem. Requests that are admitted but never complete because a downstream is slow still consume queue slots. Add a maximum queue time per request and reject with a specific error if the deadline is exceeded before processing begins.

Shed before you queue, not after. Reject at the entry point, before the request body is read or any work is done. Every millisecond of work before shedding is wasted under overload.

Closing

Load shedding is not about dropping traffic. It is about making explicit, intentional decisions about which traffic you drop so the rest completes with acceptable behavior.

The systems that survive extreme load events degrade gracefully: health checks keep returning, critical paths keep working, and shed decisions are logged with enough context to understand what happened. That requires classification at the edge, admission control with real state, circuit breaker integration, and an observability layer to close the feedback loop.

The alternative is a system that tries to serve everything and ends up serving nothing.