Designing a Health Check System: Liveness Probes, Readiness Gates, and Deep Health Checks for Distributed Services

Most services ship a /health endpoint that returns { status: "ok" } and call it done. That works until a database connection pool silently exhausts itself, a downstream service starts timing out at the 99th percentile, or a newly deployed pod tries to serve traffic before its in-memory cache is warm. At that point, your orchestrator sees a healthy process that is actually producing errors for every request.

The problem is that “healthy” means different things depending on who is asking and what they intend to do with the answer. Kubernetes asking whether to kill and restart a pod needs a different answer than a load balancer asking whether to route traffic to it, which needs a different answer than a human operator asking whether the service is degraded.

This article covers how to design a health check system that answers each of those questions correctly, without introducing the failure modes that naive implementations create.

Three Different Questions

Before writing any code, get clear on the three probe types and what each one is actually asking.

Liveness: Is this process in a broken state it cannot recover from on its own? If yes, kill it and start a new one. The liveness probe should only fail if the process is genuinely stuck: deadlocked, out of memory, in an infinite loop that prevents the event loop from processing anything. It should almost never check downstream dependencies. If your database is down, the pod is not broken. Killing it will not fix the database. It will just trigger a thundering herd on restart.

Readiness: Is this instance ready to receive traffic right now? A pod can be alive without being ready. This includes startup scenarios (cache warming, connection pool initialization, schema migration checks) and runtime degradation scenarios (a circuit breaker is open, the upstream is too slow, the connection pool is at capacity). When a pod fails its readiness probe, the orchestrator stops routing traffic to it. The pod stays alive.

Startup: Has the process finished its initial boot sequence? This probe exists to give slow-starting applications a grace period without triggering the liveness probe’s faster failure threshold. Once the startup probe passes, Kubernetes hands off to the liveness and readiness probes.

The common mistake is collapsing these into a single endpoint with a single response, or worse, making the liveness probe check everything the readiness probe does. That creates a situation where a database outage causes every pod to restart in a cascade, amplifying the outage rather than tolerating it.

Shallow vs Deep Health Checks

Within readiness checks, there is a useful split between shallow and deep checks.

A shallow check verifies that the process is in a state where it can attempt to handle requests: the HTTP server is listening, the connection pool has at least one available connection, the configuration has loaded. These checks are fast (sub-millisecond) and safe to call frequently.

A deep check actually exercises a dependency end-to-end: run a SELECT 1 against the database, publish and consume a test message from the queue, hit the cache with a GET. These checks are slow and expensive. Running them on every probe interval at scale will generate non-trivial load on your dependencies.

The right approach is to separate them. Expose the shallow check on your liveness and readiness probe endpoints, and run deep checks on a background schedule, caching the result. The probe endpoints return the cached result, not a live check.

interface HealthResult {
  status: "healthy" | "degraded" | "unhealthy";
  latencyMs: number;
  checkedAt: number;
  error?: string;
}

interface DependencyHealthCache {
  database: HealthResult | null;
  cache: HealthResult | null;
  messageQueue: HealthResult | null;
}

const healthCache: DependencyHealthCache = {
  database: null,
  cache: null,
  messageQueue: null,
};

async function checkDatabase(pool: DatabasePool): Promise<HealthResult> {
  const start = Date.now();
  try {
    await pool.query("SELECT 1");
    return {
      status: "healthy",
      latencyMs: Date.now() - start,
      checkedAt: Date.now(),
    };
  } catch (err) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      checkedAt: Date.now(),
      error: err instanceof Error ? err.message : "unknown",
    };
  }
}

// Run this on a 15-second interval, not on every probe hit
async function refreshHealthCache(deps: Dependencies): Promise<void> {
  const [database, cache, messageQueue] = await Promise.allSettled([
    checkDatabase(deps.pool),
    checkCache(deps.redis),
    checkMessageQueue(deps.mq),
  ]);

  healthCache.database =
    database.status === "fulfilled" ? database.value : makeErrorResult();
  healthCache.cache =
    cache.status === "fulfilled" ? cache.value : makeErrorResult();
  healthCache.messageQueue =
    messageQueue.status === "fulfilled"
      ? messageQueue.value
      : makeErrorResult();
}

Running Promise.allSettled here matters. If the database check throws, you still get results for the cache and queue. Promise.all would short-circuit and leave you with a partial picture.

Implementing the Probe Endpoints

With the health cache in place, the probe endpoints become simple aggregations.

Here is a complete implementation using Hono, though the logic maps directly to Express or any other Node.js framework:

import { Hono } from "hono";

const app = new Hono();

// Liveness: is the process itself broken?
// Only fails if the event loop is processing requests but in a bad internal state.
// Does NOT check external dependencies.
app.get("/healthz/live", (c) => {
  // Check internal state only: memory usage, event loop lag, etc.
  const memoryUsage = process.memoryUsage();
  const heapUsedMb = memoryUsage.heapUsed / 1024 / 1024;

  if (heapUsedMb > 1500) {
    // Adjust threshold for your service
    return c.json(
      { status: "unhealthy", reason: "heap_exhaustion", heapUsedMb },
      503
    );
  }

  return c.json({ status: "healthy" }, 200);
});

// Readiness: should traffic be routed to this instance?
// Returns 503 if any critical dependency is unhealthy.
app.get("/healthz/ready", (c) => {
  const { database, cache, messageQueue } = healthCache;

  // If the cache hasn't been populated yet (startup), report not ready
  if (!database || !cache) {
    return c.json({ status: "starting", reason: "health_cache_not_ready" }, 503);
  }

  const isReady =
    database.status !== "unhealthy" && cache.status !== "unhealthy";

  if (!isReady) {
    return c.json(
      {
        status: "unhealthy",
        dependencies: {
          database: database.status,
          cache: cache.status,
          messageQueue: messageQueue?.status ?? "unknown",
        },
      },
      503
    );
  }

  return c.json(
    {
      status: "healthy",
      dependencies: {
        database: database.status,
        cache: cache.status,
        messageQueue: messageQueue?.status ?? "unknown",
      },
    },
    200
  );
});

// Deep health: human-readable diagnostic endpoint.
// Not used by orchestrators. Used by operators and dashboards.
// Can be protected behind internal auth.
app.get("/healthz/deep", async (c) => {
  return c.json({
    status: aggregateStatus(healthCache),
    uptime: process.uptime(),
    dependencies: {
      database: healthCache.database,
      cache: healthCache.cache,
      messageQueue: healthCache.messageQueue,
    },
    memory: process.memoryUsage(),
    pid: process.pid,
  });
});

function aggregateStatus(cache: DependencyHealthCache): string {
  const statuses = [cache.database?.status, cache.cache?.status, cache.messageQueue?.status];
  if (statuses.some((s) => s === "unhealthy")) return "unhealthy";
  if (statuses.some((s) => s === "degraded")) return "degraded";
  return "healthy";
}

The /healthz/deep endpoint is the one you wire up to your internal dashboards and runbook links. It gives operators a snapshot without requiring them to dig through logs. It should not be called by your orchestrator: it is too slow and too detailed for that purpose.

Startup Probes and the Warm-Up Problem

Some services take more than a few seconds to be genuinely ready: they load a large lookup table into memory, warm a compiled regex cache, or wait for a database migration to complete before accepting connections. Kubernetes liveness probes have a failureThreshold and a periodSeconds, and if your service takes 90 seconds to start but your liveness probe gives up after 30 seconds, you will see pods in a restart loop that never stabilizes.

Startup probes solve this cleanly. Configure the startup probe with a generous failureThreshold * periodSeconds budget (for example, 30 attempts at 5 second intervals gives 150 seconds). The startup probe endpoint should check whether initialization is complete:

let initialized = false;

async function initialize(deps: Dependencies): Promise<void> {
  await loadLookupTable(deps.pool);
  await warmConnectionPool(deps.pool, { minConnections: 10 });

  // Start the background health check loop only after initialization
  setInterval(() => refreshHealthCache(deps), 15_000);
  await refreshHealthCache(deps); // Run once immediately so the cache is populated

  initialized = true;
}

app.get("/healthz/startup", (c) => {
  if (!initialized) {
    return c.json({ status: "starting" }, 503);
  }
  return c.json({ status: "ready" }, 200);
});

Once the startup probe passes, Kubernetes transitions to using the liveness and readiness probes. The startup probe is never called again during that pod’s lifetime.

The Thundering Herd Problem

Here is the failure mode nobody talks about until it bites them in production.

You have 20 pods. A dependency becomes temporarily unavailable for 30 seconds. Your readiness probe is configured to check that dependency directly on every probe hit. The load balancer stops routing traffic to all 20 pods. Traffic stops.

Worse: if you have the same check on your liveness probe, your orchestrator starts restarting all 20 pods simultaneously. They all come back up at the same time, all try to establish connections, all hit the database at once. If the database was already under stress, this amplifies the problem.

There are three mitigations:

First: Never put dependency health checks on your liveness probe. The liveness probe only fails for conditions that require a restart to fix. Dependency unavailability is not one of those conditions.

Second: Use a background health check loop with cached results, as shown above. Probe endpoints read from the cache. This decouples probe latency from dependency check latency and prevents probe calls from adding load to already-stressed dependencies.

Third: Add jitter to your background health check interval. If all pods refresh their health cache on the same 15-second interval, you still get a coordinated burst of requests to dependencies. Use a random jitter:

function startHealthCheckLoop(deps: Dependencies): void {
  const baseInterval = 15_000;
  const jitter = Math.random() * 5_000; // 0-5 seconds of random offset

  setTimeout(function run() {
    refreshHealthCache(deps).finally(() => {
      const nextInterval = baseInterval + (Math.random() * 4_000 - 2_000);
      setTimeout(run, nextInterval);
    });
  }, jitter);
}

This spreads the dependency check load across a window rather than concentrating it.

Dependency Health Aggregation

Not all dependencies are equal. Your service might be able to handle requests correctly even if the message queue is down (writes go to a fallback, or writes are not in the critical path). Treating every dependency as equally critical leads to over-aggressive readiness failures.

Model your dependencies with criticality levels:

type Criticality = "critical" | "degraded" | "optional";

interface DependencyConfig {
  name: string;
  criticality: Criticality;
}

const dependencies: DependencyConfig[] = [
  { name: "database", criticality: "critical" },
  { name: "cache", criticality: "degraded" },    // degraded without it, but functional
  { name: "messageQueue", criticality: "optional" }, // writes queue locally if unavailable
];

function computeReadiness(
  cache: DependencyHealthCache,
  config: DependencyConfig[]
): { ready: boolean; serviceStatus: string } {
  const results: Record<string, HealthResult | null> = {
    database: cache.database,
    cache: cache.cache,
    messageQueue: cache.messageQueue,
  };

  for (const dep of config) {
    const result = results[dep.name];
    if (dep.criticality === "critical" && result?.status === "unhealthy") {
      return { ready: false, serviceStatus: "unhealthy" };
    }
  }

  const hasDegraded = config.some(
    (dep) =>
      dep.criticality === "degraded" &&
      results[dep.name]?.status === "unhealthy"
  );

  return {
    ready: true,
    serviceStatus: hasDegraded ? "degraded" : "healthy",
  };
}

A “degraded” status means the service accepts traffic but is not operating at full capacity. You can surface this to clients via a response header (X-Service-Status: degraded) or include it in the /healthz/deep response. The service stays in the load balancer rotation, but the degraded signal is available for dashboards and alerting.

Health Check Design Tradeoffs

Dimension	Shallow probe on every request	Cached background checks	Direct deep check on probe
Dependency load	High at scale	Predictable, low	Spiky, unpredictable
Probe latency	Tied to dependency latency	Constant (cache read)	High
Result freshness	Real-time	15-30 second lag	Real-time
Thundering herd risk	High	Low with jitter	High
Suitable for liveness	Never	Yes	Never
Suitable for readiness	Avoid	Yes	No
Suitable for diagnostics	No	Supplement	Yes

The freshness lag in cached checks is the real cost. If a dependency fails and recovers in under 15 seconds, your service might report unhealthy for a window after recovery. For most production workloads, this is acceptable. If it is not, reduce the cache interval and add jitter to smooth the load.

Production Considerations

Timeout every dependency check. A hung database connection can make your health check hang indefinitely. Wrap each check with an explicit timeout:

async function withTimeout<T>(
  fn: () => Promise<T>,
  timeoutMs: number,
  fallback: T
): Promise<T> {
  const timeout = new Promise<T>((resolve) =>
    setTimeout(() => resolve(fallback), timeoutMs)
  );
  return Promise.race([fn(), timeout]);
}

// Usage in the health check loop
const dbResult = await withTimeout(
  () => checkDatabase(deps.pool),
  3_000,
  { status: "unhealthy", latencyMs: 3000, checkedAt: Date.now(), error: "timeout" }
);

Separate health check traffic from application traffic. Run your health check endpoints on a different port from your application endpoints. This means a backpressured application port does not block the orchestrator from reading liveness or readiness probes. In Node.js, this means a second http.createServer instance. In Hono with a Bun runtime, you can start two separate serve calls on different ports.

Do not authenticate probe endpoints. Kubernetes probes make unauthenticated HTTP requests. If you put your liveness probe behind auth middleware, you will see healthy pods appear unhealthy to the orchestrator. Use a different port (not exposed externally) for probe endpoints if you are concerned about unauthorized access.

Alert on prolonged degraded status, not on individual probe failures. A single readiness probe failure is noise. A pod staying in a degraded state for more than 5 minutes is a signal. Wire your alerting to the dependency health cache results, not to the probe endpoints directly.

The Architecture Layered Out

The health check system as described has three layers, each serving a different consumer:

Process layer (liveness probe, /healthz/live): serves the orchestrator’s kill decision. Checks only process-internal state. Should almost never fail in a healthy deployment.

Traffic layer (startup probe, /healthz/startup, readiness probe, /healthz/ready): serves the orchestrator’s routing decision. Reads from cached dependency health. Returns fast. Fails when the service cannot handle requests correctly.

Diagnostic layer (deep health, /healthz/deep): serves human operators and dashboards. Returns rich dependency state with latencies and error details. Not called by automated systems.

The background health check loop sits underneath all three layers, polling dependencies on a jittered 15-second interval and populating the cache that the traffic layer reads from.

Get the boundaries between these layers wrong and you get cascading restarts, thundering herds, or a false sense of security from a health endpoint that never reports anything other than 200 OK. Get them right and your orchestrator can make confident, correct decisions with minimal load on the dependencies it is actually checking.