Designing a Service Discovery System: Client-Side, Server-Side, and DNS-Based Patterns for Microservices

Static configuration works until it doesn’t. You deploy your first handful of services, drop IP addresses and ports into config files, and everything runs. Then you start horizontal scaling. Pods reschedule on different nodes. Instances come and go during deployments. Blue-green cutover happens mid-request. Suddenly, your static config is pointing at dead addresses, and your services are throwing connection refused errors before you’ve even shipped the feature.

Service discovery is the infrastructure that removes humans from the loop of “what address do I use to call service X right now.” It’s a solved problem in the sense that there are multiple well-understood patterns. It’s still a design decision in the sense that the three main patterns have meaningfully different operational costs, consistency properties, and failure modes.

This article covers those three patterns, when each breaks down, health checking strategies, the consistency vs. availability tradeoff inside the registry itself, and a complete TypeScript implementation of client-side discovery with health-aware routing.

Why Static Configuration Breaks

In a static environment, service locations don’t change often. You configure once, restart rarely, and the pain is manageable. In a dynamic environment, four things happen that invalidate that model:

Auto-scaling changes instance counts. When traffic spikes, your orchestrator adds instances. When it drops, instances are removed. Any static address list is stale within minutes.

Pod rescheduling changes addresses. In Kubernetes, a pod restart typically gets a new IP. If you’ve hardcoded the old IP anywhere in config, those callers are broken.

Rolling deployments run multiple versions concurrently. During a deployment, old instances and new instances coexist. Traffic needs to route to healthy instances regardless of version.

Failures need to be routed around. If an instance crashes, callers need to stop sending it traffic. With static config, that doesn’t happen automatically.

The fundamental requirement is: callers need a way to ask “give me a healthy address for service X” at call time, not at config-write time.

Pattern 1: Client-Side Discovery with a Service Registry

In client-side discovery, each service instance registers itself with a central registry on startup and deregisters on shutdown. When service A wants to call service B, it queries the registry directly, gets the list of healthy instances, picks one using a load balancing algorithm, and makes the call directly.

interface ServiceInstance {
  id: string;
  serviceId: string;
  address: string;
  port: number;
  tags: string[];
  health: "passing" | "warning" | "critical";
  lastChecked: Date;
  metadata: Record<string, string>;
}

interface ServiceRegistry {
  register(instance: Omit<ServiceInstance, "lastChecked">): Promise<void>;
  deregister(instanceId: string): Promise<void>;
  getHealthyInstances(serviceId: string): Promise<ServiceInstance[]>;
  watch(serviceId: string, callback: (instances: ServiceInstance[]) => void): () => void;
}

The registry itself is typically Consul or etcd. Both use Raft consensus for strong consistency within the registry cluster. Consul adds a built-in health checking API. etcd is more of a generic key-value store that you layer your own health checking on top of.

Here’s a realistic client-side discovery implementation with weighted round-robin and health-aware routing:

interface DiscoveryClientConfig {
  registryUrl: string;
  cacheTtlMs: number;
  healthCheckIntervalMs: number;
}

class ClientSideDiscovery {
  private instanceCache = new Map<string, ServiceInstance[]>();
  private cacheTimestamps = new Map<string, number>();
  private roundRobinCounters = new Map<string, number>();

  constructor(private config: DiscoveryClientConfig) {}

  async resolve(serviceId: string): Promise<ServiceInstance> {
    const instances = await this.getInstances(serviceId);
    const healthy = instances.filter((i) => i.health === "passing");

    if (healthy.length === 0) {
      // Fall through to warning-state instances before giving up
      const degraded = instances.filter((i) => i.health === "warning");
      if (degraded.length === 0) {
        throw new Error(`No available instances for service: ${serviceId}`);
      }
      return this.roundRobin(serviceId, degraded);
    }

    return this.roundRobin(serviceId, healthy);
  }

  private roundRobin(serviceId: string, instances: ServiceInstance[]): ServiceInstance {
    const counter = this.roundRobinCounters.get(serviceId) ?? 0;
    const index = counter % instances.length;
    this.roundRobinCounters.set(serviceId, counter + 1);
    return instances[index];
  }

  private async getInstances(serviceId: string): Promise<ServiceInstance[]> {
    const cached = this.instanceCache.get(serviceId);
    const timestamp = this.cacheTimestamps.get(serviceId) ?? 0;
    const age = Date.now() - timestamp;

    if (cached && age < this.config.cacheTtlMs) {
      return cached;
    }

    const instances = await this.fetchFromRegistry(serviceId);
    this.instanceCache.set(serviceId, instances);
    this.cacheTimestamps.set(serviceId, Date.now());
    return instances;
  }

  private async fetchFromRegistry(serviceId: string): Promise<ServiceInstance[]> {
    const response = await fetch(
      `${this.config.registryUrl}/v1/health/service/${serviceId}?passing=false`
    );

    if (!response.ok) {
      // On registry failure, return stale cache rather than hard error
      const stale = this.instanceCache.get(serviceId);
      if (stale) return stale;
      throw new Error(`Registry unavailable and no stale cache for: ${serviceId}`);
    }

    const data = await response.json();
    return this.parseConsulResponse(data);
  }

  private parseConsulResponse(data: unknown[]): ServiceInstance[] {
    return (data as any[]).map((entry) => ({
      id: entry.Service.ID,
      serviceId: entry.Service.Service,
      address: entry.Service.Address || entry.Node.Address,
      port: entry.Service.Port,
      tags: entry.Service.Tags ?? [],
      health: this.mapConsulHealth(entry.Checks),
      lastChecked: new Date(),
      metadata: entry.Service.Meta ?? {},
    }));
  }

  private mapConsulHealth(checks: any[]): ServiceInstance["health"] {
    if (checks.every((c) => c.Status === "passing")) return "passing";
    if (checks.some((c) => c.Status === "critical")) return "critical";
    return "warning";
  }

  // Subscribe to registry changes using long-polling (Consul blocking queries)
  watchService(serviceId: string, callback: (instances: ServiceInstance[]) => void): () => void {
    let index = "0";
    let active = true;

    const poll = async () => {
      while (active) {
        try {
          const response = await fetch(
            `${this.config.registryUrl}/v1/health/service/${serviceId}?index=${index}&wait=30s`
          );
          index = response.headers.get("X-Consul-Index") ?? index;
          const instances = this.parseConsulResponse(await response.json());
          this.instanceCache.set(serviceId, instances);
          this.cacheTimestamps.set(serviceId, Date.now());
          callback(instances);
        } catch {
          // Back off on error, don't spin
          await new Promise((r) => setTimeout(r, 5000));
        }
      }
    };

    poll();
    return () => {
      active = false;
    };
  }
}

The cache with stale fallback is important. If the registry becomes unavailable, you want to keep routing traffic using the last known good instance list, not fail all calls immediately. Stale data is usually better than no data, with appropriate observability to alert on stale age.

Where client-side discovery breaks down: every service now has a dependency on the registry client library. Polyglot environments get painful because you need to implement or import this logic in every language. Registry client bugs propagate to every service. Load balancing policy changes require updating every service.

Pattern 2: Server-Side Discovery with a Load Balancer

In server-side discovery, the caller sends requests to a load balancer (or sidecar proxy). The load balancer is responsible for querying the registry and routing to a healthy instance. The caller doesn’t know about the registry at all.

Caller --> Load Balancer --> Registry query --> Healthy instance

This is how Kubernetes Services work. The service object provides a stable virtual IP. kube-proxy (or eBPF-based alternatives) maintains iptables or BPF rules that route traffic to healthy pods in the service’s endpoint set. The caller just calls the service’s ClusterIP.

For non-Kubernetes environments, this pattern shows up as HAProxy or Envoy with dynamic endpoint discovery via xDS APIs. The load balancer subscribes to the registry and keeps its routing table current without callers needing any knowledge of the registry.

The advantage is centralization: load balancing logic, health checking, and routing policies are managed in one place. Adding mTLS, circuit breaking, or retries at the proxy layer doesn’t require changes to any service.

The cost is an extra network hop and a new point of failure. If the load balancer is unavailable, all traffic is blocked. You mitigate this with redundancy (multiple proxy instances) and local caching of routing tables in the proxy itself.

Where server-side discovery breaks down: you need to trust the proxy’s health model. If the proxy thinks an instance is healthy but it’s actually returning 500s for a specific endpoint, traffic still goes there. The proxy sees TCP-level health, not application-level health. This is where combining server-side routing with application-level circuit breakers makes sense.

Pattern 3: DNS-Based Discovery

DNS-based discovery uses the DNS protocol as the service registry. Each service gets a DNS name that resolves to the current set of healthy instances. CoreDNS in Kubernetes handles this natively: a service named payments in the billing namespace resolves to payments.billing.svc.cluster.local.

For non-Kubernetes setups, Consul also exposes a DNS interface. A service registered as payments resolves via payments.service.consul. Consul updates the DNS records as instances register, deregister, and pass or fail health checks.

Route 53 with health checks covers the cross-region case. You configure a weighted routing policy with health check associations. Unhealthy endpoints are automatically removed from DNS responses.

import { promises as dns } from "dns";

async function resolveService(serviceName: string): Promise<string[]> {
  try {
    // SRV records give both address and port
    const records = await dns.resolveSrv(`${serviceName}.service.consul`);
    return records
      .filter((r) => r.priority < 100) // filter out de-prioritized records
      .map((r) => `${r.name}:${r.port}`);
  } catch {
    // Fall back to A records if SRV isn't configured
    const addresses = await dns.resolve4(`${serviceName}.service.consul`);
    return addresses.map((addr) => `${addr}:8080`);
  }
}

DNS-based discovery is operationally simple. Your services already use DNS. No new client libraries. No additional protocol to implement. It works across every language and runtime without changes.

The sharp edge is TTL. DNS responses are cached by clients and intermediate resolvers for the duration of the TTL. If an instance goes down, DNS propagation takes as long as the TTL (commonly 10 to 30 seconds). During that window, some callers still resolve to the dead instance. Setting TTL to 1 second helps, but aggressive short TTLs increase load on your DNS server and make DNS a higher-risk bottleneck.

A second sharp edge: DNS gives you addresses, not health status. You can exclude unhealthy instances from DNS responses (Consul does this), but callers don’t get real-time health signals. They need their own retry logic or circuit breakers to handle the gap between DNS update and failure detection.

Where DNS-based discovery breaks down: at high connection rates with short TTLs, your authoritative DNS server can become a bottleneck. At scale, you’re doing discovery work on every new connection. Connection pooling reduces this, but it also means you hold connections to instances longer, increasing the window during which a failed instance still receives traffic.

Health Checking Strategies

All three patterns depend on health signals from instances. The common approaches are:

Active HTTP health checks. The registry or load balancer polls each instance on a designated endpoint at a fixed interval. If the check fails three times consecutively, the instance is marked unhealthy. This is the most common pattern and the easiest to reason about.

// Health check endpoint example
app.get("/health", async (req, res) => {
  const checks = await Promise.allSettled([
    checkDatabaseConnection(),
    checkCacheConnection(),
    checkDependencyReachability("payments-api"),
  ]);

  const failed = checks.filter((c) => c.status === "rejected");
  if (failed.length > 0) {
    return res.status(503).json({
      status: "unhealthy",
      checks: checks.map((c, i) => ({
        name: ["database", "cache", "payments-api"][i],
        status: c.status === "fulfilled" ? "ok" : "error",
        error: c.status === "rejected" ? c.reason?.message : undefined,
      })),
    });
  }

  res.json({ status: "healthy" });
});

TTL-based heartbeat. The instance is responsible for sending a heartbeat to the registry at regular intervals. If the registry doesn’t receive a heartbeat within the TTL window, it marks the instance unhealthy. This pattern inverts the direction: push rather than pull. It works well for batch jobs and workers that don’t expose HTTP endpoints.

TCP checks. The registry attempts a TCP connection to the instance’s port. Faster and lighter than HTTP, but only confirms the socket is accepting connections. An instance that accepts connections but returns errors for all requests will pass a TCP check.

gRPC health checks. The gRPC Health Checking Protocol is a standardized way to expose health over gRPC. The registry calls the grpc.health.v1.Health/Check method. Cleaner than a side-channel HTTP endpoint for services that only expose gRPC.

For any health check that depends on downstream services, be careful about what you include. A health check that fails when your database is slow will pull the instance from rotation even if the instance itself is fine. This amplifies cascading failures: database slowness removes all instances simultaneously, destroying the service even though the application could still handle degraded-mode traffic.

A more robust pattern is a tiered health model: /health/live (is the process running), /health/ready (can it serve traffic now), and /health/deep (full dependency check for debugging). Use ready for load balancer routing decisions and live for restart decisions.

Consistency vs. Availability in the Registry

The registry itself is a distributed system with its own consistency tradeoffs.

Consul and etcd use Raft consensus: writes are only acknowledged when a majority of registry nodes confirm them. This gives you strong consistency. If you read from a quorum-aware endpoint, you always get current data. The tradeoff is that a minority partition loses the ability to serve writes. During a network partition where fewer than half the registry nodes are reachable, the registry refuses writes. Instances can’t register or deregister.

ZooKeeper follows the same model.

For service discovery specifically, what does a write failure mean? If an instance comes up but can’t register, it won’t receive traffic. If an instance goes down but can’t deregister, callers will continue sending it traffic until health checks mark it failed. Both are bad, but they’re bounded failures. Your health checks are your safety net.

The alternative is AP behavior (availability over consistency in a partition). Some discovery systems allow stale reads and eventually-consistent state. You might see a recently-deregistered instance for a few seconds after deregistration. Your callers need to handle connection failures via retry and circuit breaker regardless, so a short window of stale data often doesn’t matter in practice.

If you’re self-managing a Consul cluster, run 5 nodes and spread them across availability zones. 5 nodes tolerates 2 failures. 3 nodes (a common starting point) tolerates only 1. Use a dedicated registry cluster, don’t colocate registry nodes with application services.

Tradeoffs Table

Dimension	Client-Side	Server-Side	DNS-Based
Caller complexity	High (registry client per language)	Low (just make the call)	Low (use DNS natively)
Infrastructure complexity	Medium (registry cluster)	High (proxy cluster + registry)	Low to medium (DNS + health checks)
Load balancing flexibility	High (arbitrary client logic)	Medium (proxy policy)	Low (round-robin or weighted only)
Failure isolation	Medium (registry failure degrades routing)	Medium (proxy failure blocks traffic)	Low (TTL-bounded stale data)
Health signal latency	Low (watch-based push)	Low (proxy subscription)	High (TTL-bound)
Polyglot support	Poor (library per language)	Excellent	Excellent
Extra network hop	None	Yes	None
Observability	Good (client emits routing decisions)	Good (proxy metrics)	Poor (DNS resolution is opaque)

Production Considerations

Deregistration on shutdown. Your instance must deregister before the process exits. Register a shutdown hook that calls the registry. In Kubernetes, handle SIGTERM and deregister before the process exits. If the process is killed with SIGKILL, health checks will remove it within their interval, but that window matters.

Startup order. Don’t register until your service is ready to serve traffic. If you register on process start before the HTTP server is listening, you’ll get traffic routed to an instance that isn’t ready yet. Register after the server is bound and initial setup (database migrations, cache warm-up) is complete.

Circuit breakers at the caller. No discovery pattern eliminates the need for circuit breakers. Health checks have detection latency. DNS has TTL lag. Even with client-side discovery and watch-based updates, there’s a short window during which a failed instance is still in the routing table. Circuit breakers at the caller protect against that window and handle application-level failures that infrastructure health checks don’t detect.

Registry connection failure handling. Design for the registry being temporarily unreachable. Client-side: cache the last known instance list and continue routing from stale data, with an alert on stale age. Server-side: the proxy should maintain its routing table in memory and continue serving from stale state. DNS: the OS resolver cache provides natural stale fallback.

Metadata and versioning. Use instance metadata to carry version information. This enables version-aware routing: callers can filter instances to a compatible version when you’re running a mixed-version deployment. It also enables canary routing: tag instances with canary: true and route a percentage of traffic to them.

// Version-aware instance selection
async function resolveCompatibleInstance(
  serviceId: string,
  requiredMajorVersion: number,
  discovery: ClientSideDiscovery
): Promise<ServiceInstance> {
  const instance = await discovery.resolve(serviceId);
  const instanceVersion = parseInt(instance.metadata.version?.split(".")[0] ?? "0", 10);

  if (instanceVersion !== requiredMajorVersion) {
    throw new Error(
      `Instance version mismatch: expected v${requiredMajorVersion}, got v${instanceVersion}`
    );
  }

  return instance;
}

Observability. Track registry query latency, stale cache hits (age of cached data at query time), health check failure rates per instance, and discovery failures (no healthy instances found). These metrics let you catch registry performance degradation before it causes routing failures.

Choosing a Pattern

Start with server-side discovery if you’re on Kubernetes and don’t need cross-cluster routing. Kubernetes Services give you it for free, with kube-proxy or Cilium handling the routing table. No extra infrastructure.

Move to client-side discovery if you need fine-grained routing control (version-aware, latency-based, weighted), are operating in a polyglot environment where you want one team owning the routing library, or need sub-second failover detection.

Use DNS-based discovery for cross-cloud or cross-region scenarios where you need a routing layer that isn’t tied to any one orchestration platform, or for environments where operational simplicity matters more than failover speed.

None of these patterns eliminates the need for retries, timeouts, and circuit breakers in the caller. Discovery tells you where to send traffic. It doesn’t guarantee the destination is working correctly when the request arrives. Build callers that handle failure independently of what the discovery layer says.

The registry is infrastructure. Treat it with the same SRE rigor you apply to your database: dedicated cluster, automated backup, runbook for partition scenarios, and an alert when the cluster loses quorum.