Designing a Service Mesh: Traffic Routing, Observability, and mTLS for Microservices

Most teams running microservices hit the same wall. Service A needs to retry failed calls to Service B with exponential backoff. Service C needs mutual TLS to Service D. Every new service needs distributed tracing headers propagated correctly. You start writing the same library, the same middleware, the same configuration across a dozen codebases. Then someone updates the retry logic in three of them but not the other nine.

A service mesh moves this cross-cutting infrastructure out of application code and into the network layer. But the decision is not binary, and the operational cost is real. This article covers how service meshes actually work, what they give you, and when the tradeoff makes sense.

The Sidecar Proxy Pattern

The core primitive in any data-plane service mesh is the sidecar proxy. Each service instance gets a proxy (typically Envoy) running alongside it in the same pod or VM. All inbound and outbound traffic routes through that proxy. The application code does not change; the network stack changes.

[Service A Pod]                  [Service B Pod]
  App (port 8080)                  App (port 8080)
  Envoy sidecar (port 15001)       Envoy sidecar (port 15001)
        |                                |
        +------------ mTLS -------------+

The control plane (Istio’s Istiod, Linkerd’s control plane, Consul Connect) pushes configuration down to these proxies via xDS APIs. The proxies report telemetry back up. Your application code has no awareness of any of this.

In Kubernetes, injection happens automatically via a mutating admission webhook. When a pod is created in a namespace with injection enabled, the webhook patches an initContainer that redirects iptables rules so all traffic flows through Envoy on port 15001.

The tradeoff you accept immediately: every network call now has two proxy hops instead of zero. Envoy is fast, but you will measure an added 1-5ms per call in P99 latency for simple pass-through traffic. For services with tight latency budgets, this is relevant.

Traffic Management at the Mesh Layer

Routing Rules

The control plane lets you express routing intent declaratively. Instead of deploying a load balancer configuration per service, you write VirtualService and DestinationRule resources (Istio) or ServiceProfile resources (Linkerd).

// This TypeScript represents the shape of an Istio VirtualService spec
// as you would construct it with a Kubernetes client library

import { KubeConfig, CustomObjectsApi } from "@kubernetes/client-node";

interface VirtualServiceSpec {
  hosts: string[];
  http: HttpRoute[];
}

interface HttpRoute {
  match?: RouteMatch[];
  route: RouteDestination[];
  retries?: RetryPolicy;
  timeout?: string;
  fault?: FaultInjection;
}

interface RouteDestination {
  destination: {
    host: string;
    subset: string;
    port?: { number: number };
  };
  weight: number;
}

interface RetryPolicy {
  attempts: number;
  perTryTimeout: string;
  retryOn: string;
}

interface FaultInjection {
  delay?: {
    percentage: { value: number };
    fixedDelay: string;
  };
  abort?: {
    percentage: { value: number };
    httpStatus: number;
  };
}

async function applyCanaryRoute(
  client: CustomObjectsApi,
  namespace: string,
  stableVersion: string,
  canaryVersion: string,
  canaryWeight: number
): Promise<void> {
  const virtualService = {
    apiVersion: "networking.istio.io/v1beta1",
    kind: "VirtualService",
    metadata: { name: "payment-service", namespace },
    spec: {
      hosts: ["payment-service"],
      http: [
        {
          route: [
            {
              destination: {
                host: "payment-service",
                subset: stableVersion,
              },
              weight: 100 - canaryWeight,
            },
            {
              destination: {
                host: "payment-service",
                subset: canaryVersion,
              },
              weight: canaryWeight,
            },
          ],
          retries: {
            attempts: 3,
            perTryTimeout: "2s",
            retryOn: "connect-failure,refused-stream,5xx",
          },
          timeout: "10s",
        },
      ],
    } satisfies VirtualServiceSpec,
  };

  await client.patchNamespacedCustomObject(
    "networking.istio.io",
    "v1beta1",
    namespace,
    "virtualservices",
    "payment-service",
    virtualService,
    undefined,
    undefined,
    undefined,
    { headers: { "Content-Type": "application/merge-patch+json" } }
  );
}

This gives you weighted traffic splitting, header-based routing, fault injection for testing, and retry policies without touching application code. The retry policy on the VirtualService applies to every caller of payment-service automatically.

Circuit Breaking

Envoy implements circuit breaking natively. You configure it on the DestinationRule as outlier detection. When a host fails enough consecutive requests, Envoy ejects it from the load-balancing pool for a configurable interval.

const destinationRule = {
  apiVersion: "networking.istio.io/v1beta1",
  kind: "DestinationRule",
  metadata: { name: "payment-service", namespace },
  spec: {
    host: "payment-service",
    trafficPolicy: {
      connectionPool: {
        tcp: { maxConnections: 100 },
        http: {
          h2UpgradePolicy: "UPGRADE",
          http1MaxPendingRequests: 50,
          maxRequestsPerConnection: 10,
        },
      },
      outlierDetection: {
        consecutive5xxErrors: 5,
        interval: "10s",
        baseEjectionTime: "30s",
        maxEjectionPercent: 50,
      },
    },
    subsets: [
      { name: "stable", labels: { version: "v1" } },
      { name: "canary", labels: { version: "v2" } },
    ],
  },
};

The maxEjectionPercent: 50 is important. Without it, all hosts in a subset could be ejected simultaneously if they all fail at once, which turns a partial failure into a total outage. This is one of those settings that matters only when things are already on fire.

Where in-library circuit breakers (Hystrix, resilience4j, or a hand-rolled implementation) break down: each service instance maintains its own circuit breaker state. Instance A does not know that Instance B has already opened its circuit. Envoy maintains the state at the proxy level, so the decision is consistent across all instances of a calling service.

Mutual TLS and Zero-Trust Networking

mTLS is where the security posture of a service mesh becomes concrete. In a flat network with no mTLS, any workload that can reach Service B’s port can call it, including a compromised pod. With mTLS enforced at the mesh layer, every connection requires both sides to present a valid certificate issued by the mesh’s CA (certificate authority).

Istio handles certificate rotation automatically. Each workload gets a SPIFFE identity embedded in the certificate’s SAN (Subject Alternative Name) field: spiffe://cluster.local/ns/production/sa/payment-service. Certificates rotate every 24 hours by default; you can lower this to 1 hour in high-security environments with minimal operational overhead since rotation is fully automated.

From your application code’s perspective, nothing changes. You still open a plain HTTP connection to the service hostname. The sidecar intercepts the connection and upgrades it to mTLS transparently.

// Application code: plain HTTP, no TLS knowledge required
import { fetch } from "undici";

async function chargeCard(
  paymentServiceUrl: string,
  payload: ChargeRequest
): Promise<ChargeResponse> {
  // The sidecar intercepts this, wraps it in mTLS, and forwards it.
  // The receiving sidecar unwraps mTLS and delivers plain HTTP to the target app.
  const response = await fetch(`${paymentServiceUrl}/v1/charge`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    throw new Error(`Charge failed: ${response.status}`);
  }

  return response.json() as Promise<ChargeResponse>;
}

Authorization policy then lets you express access control in terms of SPIFFE identities, not IP addresses:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/production/sa/checkout-service"
              - "cluster.local/ns/production/sa/subscription-service"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charge"]

Only checkout-service and subscription-service can call POST /v1/charge on payment-service. Every other workload gets a 403, regardless of network-level access. This is zero-trust in practice: identity is cryptographic, not topological.

Observability Integration

Distributed Tracing Propagation

This is where application code does need to participate. Envoy generates spans automatically for every proxied request and adds trace headers (B3, W3C Trace Context, or Jaeger format) to outbound requests. But when your application makes an outbound call, it needs to forward the incoming trace headers so the spans form a connected trace rather than disconnected fragments.

import { IncomingMessage, ServerResponse } from "http";
import { fetch } from "undici";

// W3C Trace Context headers that Envoy uses by default in Istio
const TRACE_HEADERS = [
  "traceparent",
  "tracestate",
  // B3 headers (used by Zipkin, Jaeger in B3 mode)
  "x-b3-traceid",
  "x-b3-spanid",
  "x-b3-parentspanid",
  "x-b3-sampled",
  "x-b3-flags",
  // Datadog
  "x-datadog-trace-id",
  "x-datadog-parent-id",
  "x-datadog-sampling-priority",
] as const;

type TraceHeader = (typeof TRACE_HEADERS)[number];

function extractTraceHeaders(
  req: IncomingMessage
): Partial<Record<TraceHeader, string>> {
  const headers: Partial<Record<TraceHeader, string>> = {};
  for (const header of TRACE_HEADERS) {
    const value = req.headers[header];
    if (typeof value === "string") {
      headers[header] = value;
    }
  }
  return headers;
}

async function handleCheckout(
  req: IncomingMessage,
  res: ServerResponse
): Promise<void> {
  const traceHeaders = extractTraceHeaders(req);

  // Forward trace context to downstream services
  const inventoryResponse = await fetch(
    "http://inventory-service/v1/reserve",
    {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        ...traceHeaders, // Envoy on the receiving end will see these
      },
      body: JSON.stringify({ items: [] }),
    }
  );

  // ... rest of checkout logic
}

The minimum viable change is forwarding traceparent and tracestate (W3C) or the B3 header set. If you do not propagate these, every service hop creates a new root span, and your trace view in Jaeger or Tempo shows a graveyard of disconnected single-hop traces instead of the full request flow.

Metrics Collection

Envoy emits a dense set of metrics per-route without any application instrumentation: request rate, error rate, latency histograms (P50/P95/P99), connection pool saturation, retry counts, and circuit breaker state changes. Prometheus scrapes these from the sidecar directly.

// You get these automatically in Prometheus without any application code changes
// envoy_cluster_upstream_rq_total{cluster_name="payment-service", response_code="200"}
// envoy_cluster_upstream_rq_total{cluster_name="payment-service", response_code="503"}
// envoy_cluster_upstream_rq_time_ms{cluster_name="payment-service", quantile="0.99"}
// envoy_cluster_upstream_cx_active{cluster_name="payment-service"}

// What you still need application-level instrumentation for:
// - Business metrics (orders processed, revenue, conversion rate)
// - Internal function latency within a service
// - Queue depths and consumer lag
// - Cache hit rates at the application layer

import { Counter, Histogram, Registry } from "prom-client";

const registry = new Registry();

const ordersProcessed = new Counter({
  name: "orders_processed_total",
  help: "Total orders successfully processed",
  labelNames: ["payment_method", "region"],
  registers: [registry],
});

const checkoutDuration = new Histogram({
  name: "checkout_duration_seconds",
  help: "End-to-end checkout latency",
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [registry],
});

The mesh handles infrastructure-level observability (request rates, error rates, latency by service pair). Application code handles business-level observability. These two layers compose without overlap.

Tradeoffs: Mesh vs. Application-Level Implementation

Dimension	Service Mesh	Application Libraries
Consistency across services	Uniform: same config, same behavior	Depends on library adoption and version discipline
Language support	Language-agnostic	Per-language library needed
Latency overhead	1-5ms per hop (proxy hops)	Near-zero for local in-process calls
mTLS / cert rotation	Automatic, transparent	Requires application changes, cert management per service
Traffic splitting	VirtualService weight config	Requires external LB or feature flag system
Debugging complexity	Two extra layers to understand	Failure is in application code, easier to trace
Ops dependency	Control plane is a critical component	No new infrastructure to operate
Incremental adoption	All-or-nothing per namespace (mostly)	Add to one service at a time

The mesh wins when you have many services in many languages that all need the same resilience and security primitives. The application library approach wins when you have a small, homogeneous fleet and you want to keep infrastructure simple.

The latency number is the one that surprises teams the most. Two proxy hops on a service-to-service call inside a datacenter adds 2-10ms in real deployments. For a checkout flow that fans out to 8 services, that is 16-80ms of added latency purely from proxy overhead. Run your own benchmarks before committing to a mesh in a latency-sensitive path.

Production Considerations

Control plane availability. The control plane (Istiod) pushes config to proxies. If Istiod is down, existing proxies continue operating with their last known config. New pods cannot receive config until Istiod recovers. Run Istiod with at least two replicas and a PodDisruptionBudget.

Certificate rotation in high-churn environments. In environments where pods scale rapidly (thousands of pods per hour), the CA may become a bottleneck. Istio’s CA handles ~1,000 certificate signings per second per control plane replica. Measure this under load before assuming it is safe.

iptables redirection and init containers. The iptables rules that redirect traffic to Envoy run in an initContainer. If your nodes use strict seccomp or AppArmor policies, you may need to explicitly allow NET_ADMIN and NET_RAW capabilities. Missing this manifests as pods stuck in Init:Error with cryptic iptables permission failures.

Envoy xDS config propagation delay. When you apply a VirtualService change, it takes time for the control plane to push the new config to all proxy instances. In a large cluster (500+ pods), this can take 10-30 seconds. During that window, you have inconsistent routing behavior. For zero-downtime deploys, use readiness probes that wait for the proxy to be ready, not just the application.

Headless services and mesh behavior. Headless Kubernetes services (ClusterIP: None) bypass the mesh’s load balancing. If you use headless services for stateful workloads like Kafka or databases, the mesh cannot apply retry or circuit-breaking policies to those connections. This is expected but surprises teams when they assume all TCP traffic is mesh-managed.

When NOT to use a service mesh. Small fleets (under 10 services) where application-level libraries are consistent. Services with sub-millisecond latency requirements where 2ms overhead is a hard budget miss. Teams that do not have operational capacity to debug Envoy configurations and xDS state. The mesh is infrastructure, and infrastructure requires maintenance.

Choosing a Framework

Start here: if you are running a mixed-language fleet on Kubernetes and need mTLS, distributed tracing, and traffic management consistently across all services, a service mesh is the right call. The consistency guarantee is the primary value: every service behaves the same way regardless of what language it is written in or which team owns it.

Are you running Go services only with net/http and want retry logic? Use a library. Are you adding mTLS to three services? Write a shared TLS config helper. Are you managing 40 services in six languages with different teams owning each one? The mesh pays for itself.

Are you on a deadline with one team and three services? Skip it. Come back when you have 15 services and are tired of maintaining retry logic in five codebases.

---

A service mesh does not simplify your system. It moves complexity from application code into infrastructure. The bet is that infrastructure-level complexity is more manageable than scattered, inconsistent application-level implementations of the same primitives. That bet pays off at scale. At small scale, you are paying the operational cost without getting the consistency benefit.