DevOps ·

Graceful Shutdown in Node.js: Draining Connections, Finishing Jobs, and Avoiding Data Loss

Most guides show process.on('SIGTERM', () => process.exit()) and call it done. This guide covers the full shutdown lifecycle: signal handling, HTTP drain, database cleanup, job queue stop, health check integration, Kubernetes preStop hooks, timeout handling, and testing shutdown behavior in TypeScript.

Graceful Shutdown in Node.js: Draining Connections, Finishing Jobs, and Avoiding Data Loss

Most Node.js shutdown guides show you this and move on:

process.on('SIGTERM', () => process.exit(0));

In production, that single line kills in-flight HTTP requests, aborts database transactions mid-write, and abandons queued jobs that will either duplicate or disappear depending on your queue’s delivery guarantee. You’ll see 502s during deploys, corrupted records after pod restarts, and jobs that silently vanish.

Graceful shutdown is not a single event handler. It’s a coordinated sequence: stop accepting new work, drain what’s in flight, clean up resources in dependency order, then exit. This guide walks through that full lifecycle with TypeScript examples you can drop into a real service.

What “graceful” actually means

A process shuts down gracefully when it satisfies three conditions:

  1. No new requests or jobs are accepted after the shutdown signal arrives.
  2. All in-flight work finishes (or is explicitly handed back to the queue).
  3. Resources are released in safe order: jobs before DB connections before the process exit.

The constraint that makes this hard is time. Kubernetes, systemd, and most container orchestrators give you a grace period (30 seconds by default in Kubernetes) before they send SIGKILL. Your shutdown logic has to complete within that window. If it doesn’t, the kernel kills your process anyway, and you’re back to data loss.

Signal handling foundation

Node.js exposes OS signals through process.on. The two you care about are SIGTERM (graceful stop, sent by orchestrators) and SIGINT (Ctrl+C in development). Treat them identically.

// shutdown.ts
let isShuttingDown = false;

async function shutdown(signal: string): Promise<void> {
  if (isShuttingDown) {
    console.log(`${signal} received again, forcing exit`);
    process.exit(1);
  }

  isShuttingDown = true;
  console.log(`${signal} received, starting graceful shutdown`);

  const SHUTDOWN_TIMEOUT_MS = 25_000; // stay under k8s terminationGracePeriodSeconds

  const timeoutHandle = setTimeout(() => {
    console.error('Graceful shutdown timed out, forcing exit');
    process.exit(1);
  }, SHUTDOWN_TIMEOUT_MS);

  // Don't let this timer prevent the process from exiting if everything else is done
  timeoutHandle.unref();

  try {
    await runShutdownSequence();
    console.log('Graceful shutdown complete');
    process.exit(0);
  } catch (err) {
    console.error('Error during shutdown:', err);
    process.exit(1);
  }
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

The isShuttingDown guard handles double-signal scenarios. The unref() call is important: without it, the timeout itself keeps the event loop alive even after everything else has finished.

HTTP server drain

An Express or Fastify server has two separate concerns during shutdown: stopping the acceptance of new TCP connections, and waiting for existing connections to finish their current request.

server.close() handles the first part. It stops the listening socket so no new connections can be accepted. But it does not close existing keep-alive connections that are idle between requests. Those connections stay open indefinitely, blocking your shutdown.

import http from 'http';

function closeHttpServer(server: http.Server): Promise<void> {
  return new Promise((resolve, reject) => {
    server.close((err) => {
      if (err) reject(err);
      else resolve();
    });
  });
}

For idle keep-alive connections, you need to track them and destroy them on shutdown. Node.js 18.2+ has server.closeAllConnections() and server.closeIdleConnections() built in. If you’re on an older version, you need to track sockets manually:

// For Node.js < 18.2
const connections = new Set<import('net').Socket>();

server.on('connection', (socket) => {
  connections.add(socket);
  socket.on('close', () => connections.delete(socket));
});

function destroyIdleConnections(): void {
  for (const socket of connections) {
    // _httpMessage is null on sockets not currently serving a request
    const res = (socket as any)._httpMessage;
    if (!res) {
      socket.destroy();
      connections.delete(socket);
    }
  }
}

For Node.js 18.2+, the shutdown sequence is cleaner:

async function shutdownHttp(server: http.Server): Promise<void> {
  // Stop accepting new connections
  await closeHttpServer(server);

  // If server.close() didn't resolve yet (active requests still in flight),
  // close idle connections to help drain faster
  if ('closeIdleConnections' in server) {
    (server as any).closeIdleConnections();
  }
}

The right sequence is: call server.close(), wait for it to resolve (meaning all active requests finished), and only then move to database cleanup. Active requests get their full time to complete. Idle connections get destroyed immediately so they don’t hold up the process.

Database connection cleanup

The database connection pool should be the last thing you close, because HTTP handlers and job workers depend on it. Close it after those layers have drained.

For a typical PostgreSQL setup with a pool:

import { Pool } from 'pg';

async function shutdownDatabase(pool: Pool): Promise<void> {
  console.log('Draining database connection pool');
  await pool.end();
  console.log('Database pool drained');
}

pool.end() waits for all checked-out connections to be returned before closing them. If a connection is in the middle of a transaction when pool.end() is called, the transaction will not be rolled back automatically. This is why you must drain HTTP and job layers first. By the time you call pool.end(), no application code should be acquiring new connections.

For Prisma, the equivalent is:

import { PrismaClient } from '@prisma/client';

async function shutdownPrisma(prisma: PrismaClient): Promise<void> {
  await prisma.$disconnect();
}

If you’re using multiple data stores (primary DB, read replica, Redis, Elasticsearch), shut them down in the same order: application layer first, then each data store, with the most critical one last.

Job queue graceful stop

Job queues need special handling because a worker might be mid-execution on a job that takes 10-30 seconds. The wrong behavior is abandoning the job mid-run. The right behavior is:

  • Stop picking up new jobs immediately.
  • Let currently-executing jobs run to completion (or to a safe checkpoint).
  • Acknowledge or nack jobs appropriately before the connection closes.

For a BullMQ-style worker:

import { Worker } from 'bullmq';

async function shutdownWorker(worker: Worker): Promise<void> {
  console.log('Stopping job worker');
  // close(true) = force close without waiting; close() = wait for current job
  await worker.close();
  console.log('Worker stopped');
}

worker.close() without arguments waits for the currently-executing job to complete before closing. That’s usually what you want. If you need to abort long-running jobs on shutdown, you need to implement cooperative cancellation using an AbortSignal:

// In your job processor
async function processJob(
  job: Job,
  token: string | undefined,
  signal: AbortSignal
): Promise<void> {
  for (const chunk of job.data.items) {
    if (signal.aborted) {
      // Move job back to waiting so another worker can pick it up
      await job.moveToDelayed(Date.now() + 5000, token);
      return;
    }
    await processChunk(chunk);
  }
}

The signal.aborted check is your escape hatch. Check it at logical boundaries in your job logic, not inside tight loops. When you detect abort, move the job back to the queue rather than letting it fail. This prevents duplicate side effects if the job was partially complete.

Health check integration with load balancers

Load balancers poll a health endpoint to decide whether to send traffic to a pod. During shutdown, you want the load balancer to stop sending new requests before you actually start draining. The sequence should be:

  1. SIGTERM arrives.
  2. Health endpoint starts returning 503.
  3. Load balancer detects 503, stops routing to this pod.
  4. In-flight requests drain.
  5. Server closes.
import express from 'express';

let healthy = true;

const app = express();

app.get('/health', (req, res) => {
  if (!healthy) {
    return res.status(503).json({ status: 'shutting_down' });
  }
  res.json({ status: 'ok' });
});

async function runShutdownSequence(): Promise<void> {
  // Step 1: signal unhealthy immediately
  healthy = false;

  // Step 2: wait for load balancer to detect and drain its side
  // This should match or exceed the LB's health check interval
  await sleep(5_000);

  // Step 3: stop accepting connections and drain active requests
  await shutdownHttp(server);

  // Step 4: drain job workers
  await shutdownWorker(worker);

  // Step 5: close database connections
  await shutdownDatabase(pool);
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

The 5-second sleep after marking unhealthy is a deliberate delay. It gives the load balancer time to poll, detect the 503, and stop routing. Without this, you’ll still receive requests during the early seconds of shutdown even though you’ve already started the drain sequence.

Kubernetes preStop hook

Kubernetes sends SIGTERM and simultaneously removes the pod from the service’s endpoint list. But those are not atomic. There’s a race: traffic can still be routed to the pod for a few seconds after SIGTERM arrives, because the endpoint removal propagates through kube-proxy with some latency.

The preStop hook runs before SIGTERM is sent, giving you a way to inject a delay before the process signal arrives:

# deployment.yaml
spec:
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]
      terminationGracePeriodSeconds: 30

The sleep 5 in preStop gives kube-proxy time to propagate the endpoint removal before SIGTERM hits. This means by the time your Node.js process starts seeing SIGTERM, the load balancer has already stopped routing new traffic to this pod. You can remove (or reduce) the in-process health check delay because the preStop hook has already bought you that time.

Keep terminationGracePeriodSeconds larger than preStop sleep plus your expected drain time. A reasonable formula: terminationGracePeriodSeconds = preStop sleep (5s) + max expected job duration (15s) + buffer (10s) = 30s.

Full shutdown sequence in order

Putting it all together, the complete shutdown sequence looks like this:

async function runShutdownSequence(): Promise<void> {
  // 1. Stop health checks (if no preStop hook, add sleep here)
  healthy = false;

  // 2. Stop HTTP server (stops new connections, drains active requests)
  await shutdownHttp(server);

  // 3. Stop job workers (waits for current job to finish)
  await shutdownWorker(worker);

  // 4. Close database pool (safe now that no code is using connections)
  await shutdownDatabase(pool);

  // 5. Flush any remaining logs/traces before process exit
  await flushTelemetry();
}

The order matters. HTTP drains first because handlers call workers and the database. Workers drain second because they call the database. Database closes last because everything else depends on it. Telemetry flushes last so you capture any shutdown-related errors before the process exits.

Tradeoffs and edge cases

ScenarioSafe approachRisk if ignored
Long-running HTTP SSE or WebSocket connectionsSet a max connection duration; close at shutdown with a protocol-level close frameThese connections never “finish,” so server.close() never resolves
Jobs longer than grace periodImplement checkpointing and cooperative abortJob is killed mid-run, causing partial writes or duplicate side effects on retry
Database in middle of transactionDrain HTTP and jobs before closing poolpool.end() called while transaction is open; behavior is driver-dependent
Multiple replicas during rolling deploypreStop + health check 503Traffic hits draining pods; intermittent 502s visible to end users
Uncaught exception during shutdownTop-level try/catch with forced exitShutdown hangs indefinitely; container is eventually killed by orchestrator

Testing shutdown behavior

Shutdown logic is easy to write and easy to forget to test. Here’s a minimal test approach:

// shutdown.test.ts
import { createServer } from './server';
import { once } from 'events';

test('server drains in-flight requests before closing', async () => {
  const { app, shutdown } = createServer();
  const server = app.listen(0);
  const port = (server.address() as any).port;

  // Start a slow request
  let requestResolved = false;
  const slowRequest = fetch(`http://localhost:${port}/slow`).then((r) => {
    requestResolved = true;
    return r;
  });

  // Trigger shutdown while the slow request is in flight
  const shutdownPromise = shutdown();

  // The slow request should still complete
  await slowRequest;
  expect(requestResolved).toBe(true);

  // Then shutdown completes
  await shutdownPromise;
  server.close();
});

The test pattern: start a slow request, trigger shutdown, verify the request completes before the shutdown resolves. If your shutdown logic is broken, the request will be cut off and the test will fail.

For job queue tests, use a test queue backed by an in-memory store. Trigger shutdown mid-job and assert the job ends up back in the queue with the correct state.

Production considerations

A few things that bite teams after they think they’ve solved this:

Stdio and file descriptors: console.log goes to stdout. If stdout is piped to a log aggregator that closes its side of the pipe when the container stops, writes to stdout can throw after shutdown starts. Catch these errors or switch to a logger that buffers internally.

Graceful shutdown under load: At high request rates, draining might take longer than expected. Monitor the time between SIGTERM and process exit in your observability stack. If p99 drain time approaches your grace period, you need to either increase terminationGracePeriodSeconds or reduce max request duration.

Third-party SDK clients: SDKs for analytics, error tracking, and feature flags often have their own flush methods. Call them explicitly in your shutdown sequence. Many of these SDKs batch events and will lose the final batch if the process exits before flushing.

Worker thread and child process cleanup: If you spawn workers with worker_threads or child_process, send them a shutdown message and wait for them to exit before the parent exits. They don’t receive SIGTERM by default when the parent does.

The reliable pattern in one sentence

Stop accepting work at the boundary, drain inward from the edge to the core, close resources in reverse dependency order, and always set a hard timeout that forces exit before the orchestrator’s kill timer fires.

The complexity in graceful shutdown comes not from any individual step, but from the composition: each layer has to complete before the layer beneath it closes. Get the order right, instrument it well, and your rolling deploys stop producing 502s.

More in DevOps

How Terraform Works Internally: HCL Parsing, Dependency Graph Execution, and the Provider Protocol Behind Infrastructure as Code
DevOps ·

How Terraform Works Internally: HCL Parsing, Dependency Graph Execution, and the Provider Protocol Behind Infrastructure as Code

A deep dive into Terraform's internal architecture covering HCL parsing, the expression evaluation engine, DAG-based dependency resolution, the gRPC provider plugin protocol, state mechanics, and the plan/apply lifecycle.

How Docker Works Internally: Namespaces, Cgroups, Union Filesystems, and the OCI Runtime Behind Every Container
DevOps ·

How Docker Works Internally: Namespaces, Cgroups, Union Filesystems, and the OCI Runtime Behind Every Container

A deep dive into what happens when you run docker run: Linux namespaces for process isolation, cgroups v2 for resource enforcement, overlay2 union filesystem for layered images, the OCI runtime spec and how runc spawns containers, content-addressable storage, the containerd shim architecture, and networking with veth pairs.

How Kubernetes Works: Pods, Scheduling, and the Reconciliation Loop
DevOps ·

How Kubernetes Works: Pods, Scheduling, and the Reconciliation Loop

A deep dive into Kubernetes internals covering the control plane architecture, etcd watch mechanics, the scheduler's filter and score phases, controller manager reconciliation loops, kubelet pod assignment, the CRI and CNI plugin models, and production considerations for resource requests, pod disruption budgets, and cluster autoscaler behavior.

How Docker Works: Namespaces, Cgroups, Union Filesystems, and the Container Runtime from Build to Execution
DevOps ·

How Docker Works: Namespaces, Cgroups, Union Filesystems, and the Container Runtime from Build to Execution

A deep dive into Docker internals covering Linux namespace isolation, cgroup resource constraints, OverlayFS copy-on-write layer mechanics, Dockerfile build cache invalidation rules, the containerd and runc OCI runtime stack, and production considerations for image hardening and startup latency.