How Load Balancers Work: Algorithms, Health Checks, and Scaling Patterns

Most systems hit traffic spikes before the team has time to design anything elegant. You spin up more servers, point a load balancer at them, and move on. It works until it doesn’t: one server silently starts failing, sessions start dropping, your “balanced” traffic is 90% on one node, or your load balancer itself becomes the bottleneck.

Load balancing is one of those topics that looks simple from the outside and gets progressively more interesting the closer you get to production. This article covers the core algorithms, how to design health checks that actually work, the tradeoffs between L4 and L7 balancing, session persistence patterns, and how to scale the load balancer itself when it runs out of headroom.

The Algorithms

Round-Robin

The simplest algorithm: each new request goes to the next server in a circular list. No state required, trivially fast.

class RoundRobinBalancer {
  private servers: string[];
  private index = 0;

  constructor(servers: string[]) {
    this.servers = servers;
  }

  next(): string {
    const server = this.servers[this.index];
    this.index = (this.index + 1) % this.servers.length;
    return server;
  }
}

Round-robin breaks down when requests have wildly different costs. A 10ms health check and a 30-second report export both count as “one request” to the scheduler. The server running the exports accumulates queued connections while the others stay idle.

Least Connections

Route each new request to the server with the fewest active connections. This handles heterogeneous workloads better because expensive requests naturally reduce a server’s share of incoming traffic until they complete.

class LeastConnectionsBalancer {
  private connections: Map<string, number>;

  constructor(servers: string[]) {
    this.connections = new Map(servers.map((s) => [s, 0]));
  }

  acquire(): string {
    let best = "";
    let min = Infinity;

    for (const [server, count] of this.connections) {
      if (count < min) {
        min = count;
        best = server;
      }
    }

    this.connections.set(best, min + 1);
    return best;
  }

  release(server: string): void {
    const current = this.connections.get(server) ?? 0;
    this.connections.set(server, Math.max(0, current - 1));
  }
}

The catch: you need accurate connection tracking. If the load balancer loses state (crash, failover), you’re back to guessing. For stateless HTTP requests where the load balancer proxies and counts, this works well. For persistent connections (WebSockets, gRPC streams), it’s the right default.

Weighted Round-Robin and Weighted Least Connections

When your servers have different capacities, weight them accordingly. A 32-core machine should handle more load than an 8-core one.

interface WeightedServer {
  address: string;
  weight: number;
}

class WeightedRoundRobin {
  private servers: WeightedServer[];
  private currentWeights: number[];
  private totalWeight: number;

  constructor(servers: WeightedServer[]) {
    this.servers = servers;
    this.currentWeights = new Array(servers.length).fill(0);
    this.totalWeight = servers.reduce((sum, s) => sum + s.weight, 0);
  }

  // Nginx smooth weighted round-robin algorithm
  next(): string {
    let best = -1;
    let bestWeight = -Infinity;

    for (let i = 0; i < this.servers.length; i++) {
      this.currentWeights[i] += this.servers[i].weight;
      if (this.currentWeights[i] > bestWeight) {
        bestWeight = this.currentWeights[i];
        best = i;
      }
    }

    this.currentWeights[best] -= this.totalWeight;
    return this.servers[best].address;
  }
}

This is the smooth weighted round-robin algorithm Nginx uses internally. It distributes traffic more evenly than naive weight-based selection and avoids bursty allocation patterns where high-weight servers receive many requests in a row.

Consistent Hashing

Round-robin and least-connections both assume any server can handle any request. Consistent hashing changes the model: a request is always routed to the same server (or small group of servers) based on a hash of the request key. Common keys: user ID, session token, or a cache key.

The classic use case is distributed caches. If you route /user/123 to any of ten servers with round-robin, every server needs a copy of that user’s data. With consistent hashing, /user/123 always goes to server 4 (for example), so only one server needs it in memory.

import { createHash } from "crypto";

class ConsistentHashRing {
  private ring: Map<number, string> = new Map();
  private sortedKeys: number[] = [];
  private readonly replicaCount: number;

  constructor(servers: string[], replicaCount = 150) {
    this.replicaCount = replicaCount;
    for (const server of servers) {
      this.addServer(server);
    }
  }

  private hash(key: string): number {
    const hex = createHash("md5").update(key).digest("hex").slice(0, 8);
    return parseInt(hex, 16);
  }

  addServer(server: string): void {
    for (let i = 0; i < this.replicaCount; i++) {
      const virtualNode = `${server}:${i}`;
      const key = this.hash(virtualNode);
      this.ring.set(key, server);
    }
    this.sortedKeys = [...this.ring.keys()].sort((a, b) => a - b);
  }

  removeServer(server: string): void {
    for (let i = 0; i < this.replicaCount; i++) {
      const key = this.hash(`${server}:${i}`);
      this.ring.delete(key);
    }
    this.sortedKeys = [...this.ring.keys()].sort((a, b) => a - b);
  }

  getServer(requestKey: string): string {
    if (this.ring.size === 0) throw new Error("No servers available");

    const hash = this.hash(requestKey);
    // Find first virtual node with key >= hash
    const idx = this.sortedKeys.findIndex((k) => k >= hash);
    const ringKey =
      idx === -1
        ? this.sortedKeys[0] // wrap around
        : this.sortedKeys[idx];

    return this.ring.get(ringKey)!;
  }
}

Virtual nodes (the replicaCount replicas per server) solve the uneven distribution problem. Without them, adding or removing servers shifts large chunks of keys to a single neighbor. With 150 virtual nodes per server, key remapping is distributed across the ring.

When a server is removed, only its portion of the keyspace migrates to neighbors, not the entire dataset.

Health Check Design

Routing to a dead server is worse than no routing at all. Health checks need to be sensitive enough to catch failures quickly without producing false positives that cause unnecessary failovers.

Active vs Passive Checks

Active checks are synthetic probes the load balancer sends on a schedule: hit /health on each backend every 5 seconds, and if it fails three times in a row, mark the server down.

Passive checks observe real traffic. If 50% of responses to a server return 5xx errors in a 30-second window, mark it down.

Active checks detect failures faster because you control the frequency, but they add baseline traffic. Passive checks have no overhead but only trigger when real requests are already failing.

Production systems use both: active for hard failures (server down, port not listening), passive for soft failures (server up but returning errors or high latency).

interface HealthCheckConfig {
  interval: number;      // ms between checks
  timeout: number;       // ms before marking a check failed
  healthyThreshold: number;   // consecutive successes to mark healthy
  unhealthyThreshold: number; // consecutive failures to mark unhealthy
}

type ServerState = "healthy" | "unhealthy" | "draining";

interface ServerHealth {
  state: ServerState;
  consecutiveSuccesses: number;
  consecutiveFailures: number;
  lastCheck: number;
}

class HealthChecker {
  private health: Map<string, ServerHealth> = new Map();
  private config: HealthCheckConfig;

  constructor(servers: string[], config: HealthCheckConfig) {
    this.config = config;
    for (const server of servers) {
      this.health.set(server, {
        state: "healthy",
        consecutiveSuccesses: 0,
        consecutiveFailures: 0,
        lastCheck: 0,
      });
    }
  }

  async check(server: string): Promise<void> {
    const h = this.health.get(server)!;
    const controller = new AbortController();
    const timer = setTimeout(
      () => controller.abort(),
      this.config.timeout
    );

    try {
      const res = await fetch(`http://${server}/health`, {
        signal: controller.signal,
      });
      clearTimeout(timer);

      if (res.ok) {
        h.consecutiveFailures = 0;
        h.consecutiveSuccesses++;
        if (
          h.state === "unhealthy" &&
          h.consecutiveSuccesses >= this.config.healthyThreshold
        ) {
          h.state = "healthy";
          console.log(`[health] ${server} recovered`);
        }
      } else {
        this.recordFailure(server, h);
      }
    } catch {
      clearTimeout(timer);
      this.recordFailure(server, h);
    }

    h.lastCheck = Date.now();
  }

  private recordFailure(server: string, h: ServerHealth): void {
    h.consecutiveSuccesses = 0;
    h.consecutiveFailures++;
    if (
      h.state === "healthy" &&
      h.consecutiveFailures >= this.config.unhealthyThreshold
    ) {
      h.state = "unhealthy";
      console.log(`[health] ${server} marked unhealthy`);
    }
  }

  isHealthy(server: string): boolean {
    return this.health.get(server)?.state === "healthy";
  }
}

The unhealthy threshold matters more than people expect. Setting it to 1 (fail once, mark down) causes flapping under transient network hiccups. Setting it to 5 means 25 seconds of failures (at 5s intervals) before traffic shifts. Most production systems use 2-3 failures for unhealthy, 2 successes for recovery.

What to health check: Don’t just verify that the HTTP port is open. A minimal health endpoint should check the database connection and any critical downstream dependency. If your app can’t serve requests without the database, a passing health check while the DB is down is actively harmful.

Session Persistence (Sticky Sessions)

Some applications store session state in local process memory. If a request lands on the wrong server, the session is gone. Sticky sessions solve this by ensuring a client always hits the same backend.

Cookie-based stickiness is the most common approach. The load balancer sets a cookie (often called SERVERID or similar) on the first response. Subsequent requests include that cookie, and the load balancer routes based on it.

IP-based stickiness hashes the client IP to a server. Simpler, but breaks for clients behind NAT (many users sharing one IP all land on the same backend) and fails when clients switch networks.

The tradeoff with sticky sessions is that they reintroduce imbalance. If one server gets assigned a batch of heavy sessions, it bears disproportionate load. They also complicate rolling deployments: you can’t take a server down without evicting sessions.

The better long-term answer is externalizing session state to Redis or a database so any server can serve any request. Sticky sessions are a stopgap, not an architecture.

L4 vs L7 Load Balancing

Dimension	L4 (TCP/UDP)	L7 (HTTP/HTTPS)
Visibility	IP, port, TCP flags	HTTP method, path, headers, body
Routing granularity	Per connection	Per request
Latency overhead	~0.1ms	1-5ms (TLS termination, parsing)
Health checks	TCP connect, basic	HTTP status, response body
Session affinity	IP hash, TCP session	Cookie, header
Content routing	No	Yes (path, host, A/B)
TLS termination	No (pass-through)	Yes
WebSocket support	Native	Requires explicit upgrade handling
Observability	Connection counts	Request rates, status codes, latency

L4 balancers operate at the transport layer. They see IP addresses and ports but not HTTP. They’re faster because there’s less to parse, and they can handle any TCP or UDP protocol (not just HTTP). The tradeoff is less flexibility: you can’t route based on URL path, you can’t do header-based canary deployments, and health checks are limited to “can I open a connection.”

L7 balancers terminate the connection, inspect the HTTP request, make a routing decision, and open a new connection to the backend. This adds latency (typically 1-5ms for TLS termination and parsing) but unlocks content-aware routing, granular health checks, better observability, and features like request retries and circuit breaking.

Most modern deployments use both: an L4 balancer at the network edge for raw throughput, and an L7 balancer (often in-cluster, like an ingress controller) for application-layer routing.

Scaling the Load Balancer

The irony of load balancers is that they can become the bottleneck themselves. Here are the three main patterns for scaling past a single node.

DNS-Based Load Balancing

Return multiple A records for the same hostname. Clients pick one (usually the first, though behavior varies by resolver and OS). Simple to set up, no single point of failure.

The problem: DNS TTLs mean clients cache the IP for minutes or hours. When a server goes down, traffic keeps hitting it until the TTL expires. You can set short TTLs (30-60s) but many resolvers ignore them anyway. DNS-based balancing is better suited for coarse geographic distribution than fine-grained failover.

Anycast

Multiple servers worldwide advertise the same IP via BGP. The network routes each client to the nearest one. Cloudflare and Google use this extensively for their edge infrastructure.

This is how DDoS mitigation at scale works: traffic from an attack is automatically distributed across PoPs (points of presence) rather than concentrating on one target. Implementing anycast yourself requires controlling BGP routes, which typically means your own AS and relationships with upstream providers. For most teams, this means using a CDN or managed DNS provider that does anycast on your behalf.

Active-Passive and Active-Active

Active-passive: One load balancer handles all traffic. A standby is ready to take over via a floating IP (using keepalived/VRRP). Failover takes seconds to tens of seconds. Simple operationally, but the standby is idle capacity.

Active-active: Multiple load balancers handle traffic simultaneously, typically behind a DNS round-robin or anycast. If one fails, the others absorb its share. More complex (you need to synchronize state for session persistence, health check results, and rate limit counters) but no wasted capacity.

For stateless load balancers doing pure L4 or stateless L7 routing, active-active is straightforward. The complexity compounds when you need shared state.

Practical Tradeoffs

Scenario	Recommended Approach
Stateless HTTP microservices	Round-robin or least connections, L7, no sticky sessions
WebSocket connections	Least connections, L4 or L7 with upgrade support, IP or cookie affinity
Distributed in-memory cache	Consistent hashing keyed by cache key
Mixed server capacity	Weighted least connections
Multi-region active-active	Anycast at edge, per-region L7 inside
Legacy app with local session state	Cookie-based sticky sessions short-term, externalize sessions long-term
gRPC microservices	L7 with per-RPC balancing (connection-level L4 balancing doesn’t work with HTTP/2 multiplexing)

One note on gRPC: HTTP/2 multiplexes many RPCs over a single TCP connection. An L4 balancer sees one connection per client and routes all that client’s traffic to the same backend. You need L7 balancing to distribute individual RPCs across servers. This catches teams by surprise when they migrate from REST to gRPC and their traffic distribution gets worse.

Production Considerations

Connection draining: When removing a server from rotation (deployment, scaling down), don’t drop active connections. Mark the server as draining, stop sending new connections to it, and wait for active connections to complete before terminating. Typical drain timeout is 30-60 seconds.

Slow start: When a new server comes up, ramp up its traffic share gradually rather than immediately sending it full load. Many load balancers support a “warm-up” period. This avoids the cold-start problem where a fresh server hasn’t populated its caches yet and takes longer to respond, which can compound under full load.

Circuit breaking at the load balancer: If a backend consistently returns errors or times out, stop sending it traffic for a period (say, 30 seconds) and try again with a reduced traffic share. This is distinct from health checks, which are active probes. Circuit breaking is passive and responds to real error rates.

Observability: Track per-backend request rates, error rates, and latency percentiles (p50, p95, p99). Aggregate metrics hide problems. If one backend is at 500ms p99 while the others are at 20ms, round-robin will keep sending it traffic because the aggregate looks acceptable.

Timeouts: Set connection, read, and write timeouts at the load balancer. A backend that accepts connections but never responds will accumulate open connections and exhaust the load balancer’s connection pool. Timeouts are not optional.

Closing

Load balancers solve traffic distribution, but the decisions you make about algorithms, health checks, and session handling ripple through the rest of your architecture. The right algorithm depends on your workload shape. Health check thresholds are a reliability dial you tune per environment. Session persistence is a constraint you should try to eliminate, not accommodate permanently. And the load balancer itself needs a scaling plan before it becomes the ceiling.

Pick the simplest setup that fits your traffic pattern, instrument it well, and revisit the algorithm choice when the data shows it’s not distributing evenly.