How Envoy Proxy Works Internally: xDS Configuration, Connection Pooling, and the Filter Chain Architecture Behind Modern Service Meshes

Every service mesh you encounter today, whether Istio, Consul Connect, or AWS App Mesh, runs Envoy at its data plane. The control plane sends configuration; Envoy enforces it. Understanding what happens inside Envoy between “receive a request” and “forward it upstream” is what separates engineers who tune service meshes by feel from those who tune them by understanding.

This article walks through Envoy’s internals at a level that makes production behavior predictable: the threading model, the listener and filter chain pipeline, the xDS protocol family and how incremental ADS works, hot restart mechanics, the HTTP connection manager, circuit breaking and outlier detection, and the connection pool lifecycle. At the end, there is a tradeoffs table comparing Envoy against HAProxy, Nginx, Traefik, and Caddy across the dimensions that matter when picking a proxy.

The Threading Model: One Event Loop Per Core

Envoy runs a single main thread and N worker threads, where N defaults to the number of hardware threads. Each worker thread runs its own event loop backed by libevent (or libuv on some builds). No shared mutable state crosses thread boundaries during the request path. That is the design principle that lets Envoy scale linearly with CPU count without lock contention.

The main thread owns:

• xDS client connections and configuration updates
• Admin API handling
• Cluster membership updates from EDS responses

Worker threads own:

• Accepted connections
• All I/O for those connections
• Filter chain execution for every request on their connections

Connection pinning is the consequence of this model. When a downstream client opens a TCP connection, a worker accepts it. Every request on that connection is processed exclusively by that worker for its lifetime. There is no handoff. This means CPU affinity is good and cache locality is high, but it also means a slow worker cannot shed load to a faster neighbor. Load imbalance across workers can surface in services with persistent HTTP/2 connections where one client opens a single stream-multiplexed connection that hammers a single worker while others sit idle.

// Illustrating the mental model in TypeScript terms:
// Each worker owns a set of connections.
// Requests on connection X always execute on worker W.
interface WorkerThread {
  connections: Map<ConnectionId, DownstreamConnection>;
  eventLoop: EventLoop;
}

interface DownstreamConnection {
  workerId: number; // pinned at accept time
  upstreamPools: Map<ClusterName, ConnectionPool>;
}

Worker threads maintain their own upstream connection pools per cluster. When a request on worker 3 needs to reach the payments cluster, it uses worker 3’s pool for that cluster. If the pool has no idle connection, worker 3 opens a new one. Workers do not share upstream connections, which means N workers can hold up to N × pool_size connections to each upstream host.

The Listener, Filter Chain, and Cluster Architecture

Every Envoy instance has at least one listener. A listener binds to an IP:port, accepts connections, and runs them through a filter chain. The filter chain is a sequence of network filters. The most common network filter is the HTTP connection manager (HCM), but raw TCP proxying, TLS inspection, and custom WASM filters all plug in at this level.

The processing pipeline looks like this:

Downstream TCP connection
  → Listener (IP:port binding)
    → Filter Chain Match (SNI, ALPN, source IP)
      → Network Filters (e.g., HTTP Connection Manager)
        → HTTP Filters (router, rate limit, JWT auth, ...)
          → Cluster selection (via route table)
            → Load balancing (pick upstream host)
              → Connection Pool (reuse or open connection)
                → Upstream TCP/HTTP2/HTTP3 connection

Filter chain matching happens before any filter code runs. Envoy evaluates the chain match criteria (SNI for TLS, ALPN for protocol negotiation, source IP CIDR) and selects the right filter chain for the connection. This is how a single listener on port 443 can route gRPC traffic to one HCM config and REST traffic to another.

xDS: Dynamic Configuration Without Restarts

Static configuration in YAML is fine for experimentation. Production deployments use xDS, the discovery service API family. xDS is gRPC-streaming: the control plane pushes configuration updates to Envoy; Envoy applies them atomically without dropping connections.

The xDS resource types:

API	Resource	What It Controls
LDS	Listeners	IP:port bindings, filter chains
RDS	Routes	Virtual host and route table per listener
CDS	Clusters	Upstream cluster definitions, circuit breaker thresholds
EDS	Endpoints	Individual upstream host addresses and weights
SDS	Secrets	TLS certificates, SPIFFE-issued SVIDs

Aggregated Discovery Service and Incremental xDS

Early xDS used per-resource-type gRPC streams. Managing five streams with independent versioning led to ordering hazards: a CDS update could reference a cluster that EDS had not yet populated, causing a brief traffic blackhole. ADS (Aggregated Discovery Service) solves this by multiplexing all resource types over a single ordered stream, giving the control plane explicit control over sequencing.

Incremental xDS (also called delta xDS) goes further. Classic xDS sends the full state of a resource type in each response. At thousands of clusters, a single endpoint change triggers a full EDS response listing every endpoint in every cluster. Incremental xDS sends only the diff: added, modified, and removed resources. The Envoy-side API looks like this at the protobuf level:

// Simplified representation of a delta xDS response:
interface DeltaDiscoveryResponse {
  typeUrl: string;           // e.g., "type.googleapis.com/envoy.config.cluster.v3.Cluster"
  resources: Resource[];     // added or updated
  removedResources: string[]; // resource names to delete
  nonce: string;             // for ACK/NACK flow control
}

// Envoy ACKs with:
interface DeltaDiscoveryRequest {
  typeUrl: string;
  responseNonce: string;     // must match the response nonce
  errorDetail?: Status;      // populated on NACK
}

When Envoy receives a configuration update it cannot apply (a malformed route, an unknown filter), it sends a NACK with an errorDetail. The control plane can retry with a corrected response. This explicit ACK/NACK loop is what makes xDS suitable as a correctness-critical configuration protocol rather than a best-effort push.

Hot Restart: Zero-Downtime Binary Upgrades

Upgrading a proxy binary in production without dropping connections is a classic hard problem. Envoy’s hot restart forks a new process, passes listening socket file descriptors from the old process to the new one via Unix domain socket, and then drains the old process over a configurable drain window.

The sequence:

1. Operator sends SIGUSR1 to the old Envoy process.
2. Old process forks a new Envoy child, passing inherited FDs for all listen sockets.
3. New process starts, loads new configuration, and begins accepting new connections on the inherited FDs.
4. Old process enters drain mode: it stops accepting new connections but finishes existing ones.
5. After the drain window (default 600 seconds), the old process exits.

The two processes communicate over a Unix domain socket for stat forwarding (Envoy’s hot restart protocol merges stats from both processes during the overlap window). This means your monitoring dashboards do not see a stats gap during upgrade.

Hot restart is why Kubernetes operators that manage Envoy sidecars can roll out new sidecar binaries with zero dropped requests when drain windows are tuned correctly. The most common mistake is setting the drain window shorter than the P99 request latency of long-running gRPC streams.

The HTTP Connection Manager in Depth

The HCM is itself a network filter, but it contains an entire sub-pipeline of HTTP filters. Understanding its internals is essential for debugging routing problems.

The HCM receives a downstream TCP connection and speaks HTTP/1.1 or HTTP/2 (or HTTP/3 with QUIC) on it. For HTTP/2, it maintains a single TCP connection that multiplexes many streams. Each stream is an independent request.

HTTP filters execute in order on every request, then in reverse order on every response:

Request direction  →  [JWT Auth] → [Rate Limit] → [Router]
Response direction ←  [Router]   ← [Rate Limit] ← [JWT Auth]

The router filter is always last on the request path. It is the filter that performs route table lookup and selects the upstream cluster. Route table lookup evaluates virtual hosts by authority header (or SNI for TLS), then matches route predicates (prefix, path, header conditions, query parameters) in order, and picks the first match.

// A route configuration as a TypeScript interface (matching Envoy's xDS model):
interface RouteConfiguration {
  name: string;
  virtualHosts: VirtualHost[];
}

interface VirtualHost {
  name: string;
  domains: string[];   // ["api.example.com", "api.example.com:443"]
  routes: Route[];
}

interface Route {
  match: RouteMatch;
  action: RouteAction;
}

interface RouteMatch {
  prefix?: string;      // "/api/v1/"
  path?: string;        // exact path
  headers?: HeaderMatcher[];
}

interface RouteAction {
  cluster: string;      // target cluster name
  timeout: Duration;
  retryPolicy?: RetryPolicy;
  weightedClusters?: WeightedCluster[]; // for traffic splitting
}

The weightedClusters field is how Envoy implements canary deployments and A/B traffic splits at the routing layer, without any upstream-side changes.

Circuit Breaking and Outlier Detection

These two mechanisms are often conflated. They protect different things.

Circuit breaking limits resource consumption to prevent cascading overload. It is configured per cluster and operates on counters:

• max_connections: maximum active TCP connections to the cluster
• max_pending_requests: queue depth for requests waiting for a connection
• max_requests: maximum active HTTP/2 streams or HTTP/1.1 requests in flight
• max_retries: maximum concurrent retries across all connections

When a threshold is exceeded, new requests are rejected with a 503 immediately, without touching the upstream. The circuit does not “open” and “close” on a timer the way Hystrix does. Envoy’s circuit breakers are pure concurrency limits. The practical implication: if you see upstream_rq_pending_overflow incrementing in Envoy stats, your pending requests threshold is too low or your upstream is genuinely overloaded.

Outlier detection watches individual hosts within a cluster and ejects hosts that are returning errors or responding slowly. It operates on observed behavior, not on configuration thresholds:

interface OutlierDetection {
  consecutive5xx: number;           // eject after N consecutive 5xx responses
  interval: Duration;               // evaluation interval (default 10s)
  baseEjectionTime: Duration;       // initial ejection duration
  maxEjectionPercent: number;       // cap on ejected hosts (e.g., 50%)
  successRateMinimumHosts: number;  // need at least N hosts for statistical ejection
  successRateRequestVolume: number; // need at least N requests per host per interval
}

Ejected hosts are placed into a penalty box for baseEjectionTime * ejectionCount. Each subsequent ejection of the same host doubles the penalty duration up to maxEjectionTime. After the penalty expires, the host is unejected and receives traffic again. If it continues to fail, it is ejected again with a longer penalty.

The interaction between circuit breaking and outlier detection matters: if outlier detection ejects half your upstream hosts and your circuit breaker max_connections is set based on a full healthy cluster, you may hit the connection limit with half the normal traffic.

Connection Pooling and Upstream Management

Each worker thread maintains connection pools per cluster per upstream host. Envoy has separate pool implementations for HTTP/1.1, HTTP/2, and HTTP/3.

For HTTP/1.1, connections are not multiplexed. Each in-flight request needs its own connection. The pool maintains:

• An active connection list (connections with in-flight requests)
• An idle connection list (connections ready for reuse)
• A pending request queue (requests waiting for an available connection)

For HTTP/2, a single connection multiplexes many streams. The pool maintains active connections and tries to spread load across them:

interface Http2ConnectionPool {
  connections: Http2Connection[];
  maxRequestsPerConnection: number; // drain connection after N requests

  // Selects the connection with the fewest active streams.
  selectConnection(): Http2Connection | null;
}

The max_requests_per_connection setting on a cluster is how you implement connection draining for HTTP/2. Set it to 1000, and after a connection processes 1000 requests it is drained and replaced. This prevents any single connection from accumulating state across deployments or configuration changes.

Upstream health checking in Envoy is separate from outlier detection. Active health checks (HTTP, TCP, gRPC) are configured per cluster and run from every worker independently. Passive health checking is outlier detection. Using both together gives you fast failure detection: active checks catch dead hosts that never return errors (e.g., a hung process not accepting TCP), while outlier detection catches hosts that accept connections but return bad responses.

Load Balancing Algorithms

Envoy implements several load balancing policies selectable per cluster:

• Round robin: distributes requests sequentially. Good baseline for homogeneous upstreams.
• Least request: routes to the upstream with the fewest active requests. Uses a two-random-choice algorithm: pick two hosts at random, send to the one with fewer requests. This approximates the optimal least-loaded choice with O(1) complexity.
• Random: routes to a uniformly random host. Useful when connection-level distribution is already handled elsewhere.
• Ring hash (consistent hash): hashes a configurable key (header, cookie, source IP) to a virtual ring. Used for session affinity and cache locality. The virtual node count controls ring granularity and rebalancing spread on membership changes.
• Maglev: Google’s consistent hash table with better distribution properties than ring hash. More uniform slot assignment reduces rebalancing impact when hosts are added or removed.
• Weighted least request: like least request but weights are applied from EDS endpoint weights.

For most east-west gRPC traffic in a service mesh, least request is the right default. Round robin distributes connections evenly but not requests; a slow upstream host accumulates requests because the balancer keeps sending new ones at the same rate.

Tradeoffs: Envoy vs HAProxy, Nginx, Traefik, and Caddy

Dimension	Envoy	HAProxy	Nginx	Traefik	Caddy
Threading model	Main + N workers, connection-pinned, no shared state	Multi-process or multi-thread, shared state via locks	Master + N workers, connection-pinned	Single process, goroutines	Single process, goroutines
Configuration model	xDS gRPC streaming (dynamic); static YAML fallback	Static file with reload via SIGUSR1; no live push API	Static file + optional dynamic modules; reload drops no connections	Kubernetes-native CRDs, Docker labels, or file; polling	File-based; Kubernetes via CRD; no streaming push
Protocol support	HTTP/1.1, HTTP/2, HTTP/3 (QUIC), gRPC, raw TCP, DUBBO	HTTP/1.1, HTTP/2, TCP, QUIC (HAProxy 2.6+)	HTTP/1.1, HTTP/2, HTTP/3 (1.25+), TCP stream	HTTP/1.1, HTTP/2, TCP, UDP, gRPC	HTTP/1.1, HTTP/2, HTTP/3, reverse proxy
Observability	First-class: stats, tracing headers, access logs, admin API	Stats page, Prometheus exporter, access logs	Stub status module, access logs; Prometheus via exporter	Built-in Prometheus, tracing via middleware	Prometheus plugin, access logs
xDS / control plane	Native; designed for Istio, Consul, custom control planes	None; manual or via dataplaneapi for HAProxy RANA	None natively; OpenResty or Nginx Plus add APIs	Pilot provider for Kubernetes; no xDS	No xDS support
Hot restart	Built-in via fork + FD passing	SIGUSR1 graceful reload, no FD passing	Master process graceful reload, no FD passing	Signal-based reload	Signal-based reload
Operational complexity	High: xDS, filter chain model, stats cardinality	Low to medium: battle-tested, simple config	Low: widely understood, extensive documentation	Low: declarative, Kubernetes-native	Low: automatic HTTPS, minimal config
Sweet spot	Service mesh data plane, complex L7 routing, programmatic config	High-throughput TCP/HTTP, simple routing, extreme reliability	Static content, simple reverse proxy, high concurrency	Kubernetes ingress, developer-friendly routing	Simple HTTPS reverse proxy, automatic certificate management

The sweet spot column tells the real story. Envoy’s strength is programmable configuration at runtime via a well-specified API. If you need a control plane to push routing rules without restarting the proxy, Envoy is the only mainstream option. If you need a high-throughput TCP load balancer with a configuration file and minimal moving parts, HAProxy’s reliability and predictability are hard to beat.

Production Considerations

A few patterns that trip up teams running Envoy in production:

Stats cardinality explosion. Envoy emits per-cluster, per-listener, per-filter stats. If your cluster names include dynamic identifiers (tenant IDs, deployment timestamps), you can generate millions of unique metric time series. Use static cluster names and rely on routing weights to differentiate traffic.

Connection pool exhaustion under H2 multiplexing. With HTTP/2 upstreams, a single slow request does not block others on the same connection. But if max_requests_per_connection is unset and you have long-lived streaming RPCs, connections accumulate state across upstream deployments. Set a reasonable ceiling and monitor cx_active per cluster.

xDS NACK propagation. When Envoy NACKs a configuration update, the control plane sees the NACK but the rest of your fleet may have ACKed it successfully. You now have configuration skew across instances. Build your control plane to treat NACKs as hard failures and alert on them. Do not silently retry indefinitely.

Drain window vs request latency. During hot restart, the drain window must exceed your P99 request latency for long-running RPCs. A 600-second default is conservative. If your gRPC streams are session-scoped and can last hours, hot restart will terminate them at drain window expiry. Design long-running streams to reconnect gracefully.

Outlier detection and small clusters. With fewer than successRateMinimumHosts endpoints (default 5), statistical outlier detection is disabled. In a 3-pod deployment, consecutive_5xx-based detection still works but success-rate detection does not. Account for this when sizing small services.

The filter chain model, xDS protocol, and connection-pinned threading model are not incidental design choices. They are the reason Envoy can be safely programmed at runtime by a control plane across thousands of replicas. Once you understand the pipeline from listener match through filter execution to cluster selection to pool management, production behavior becomes predictable rather than mysterious.