DEV Community

subashthiruppathy
subashthiruppathy

Posted on

Graceful Shutdown in Node.js: Preventing Dropped Requests and Broken Database Transactions

#kubernetes #devops #node #backend

Every time you deploy, autoscale, or restart a node, your orchestrator does the same thing to your app: it asks it to stop.

If your app handles that request well, nobody notices. If it doesn't, you get a brief spike of 502s on every release, users who got "payment failed" on a payment that actually went through, background jobs that ran twice, and database transactions cut off mid-flight.

The frustrating part is that it's easy to miss. Locally you press Ctrl-C and the process dies instantly, with no one watching. In production, with real traffic, the same behavior drops real requests.

This post covers how to shut a Node.js service down gracefully: stop taking new work, finish the work in flight, release resources in the right order, and exit before the orchestrator loses patience.

What actually happens when your app is stopped

On Docker, Kubernetes, ECS, systemd, and most platforms, stopping a process is a two-step sequence:

  1. The platform sends SIGTERM: "please shut down."
  2. After a grace period, if you're still running, it sends SIGKILL, which can't be caught or handled. The process is gone immediately.
Platform Default grace period
Docker (docker stop) 10 seconds (--stop-timeout / stop_grace_period to change)
Kubernetes 30 seconds (terminationGracePeriodSeconds)

The grace period is your time budget. Everything in this post has to fit inside it.

If you don't handle SIGTERM, Node's default behavior is to exit immediately. Every in-flight request is cut mid-response, and every open transaction loses its connection.

(Ctrl-C sends SIGINT, which behaves similarly, so handle both.)

Failure mode 1: The naive version 

// ❌ No signal handling at all
const server = app.listen(3000);
Enter fullscreen mode Exit fullscreen mode

On deploy, in-flight requests are severed. Clients see ECONNRESET or a 502 from the load balancer.

The first fix everyone tries:

// ⚠️ Better, but still wrong
process.on('SIGTERM', () => {
  server.close();
  process.exit(0);   // kills everything still in flight, immediately
});
Enter fullscreen mode Exit fullscreen mode

server.close() stops accepting new connections, but process.exit(0) on the next line terminates the process before any in-flight request can finish. This is the most common "I handle SIGTERM" bug.

The correct shutdown sequence

Order matters. The principle is: stop work coming in, let work in progress finish, then tear down what that work depended on, and not before.

SIGTERM
  │
  1. Mark unready          → load balancer stops sending new traffic
  2. Wait briefly          → let routing updates propagate
  3. Stop accepting HTTP   → server.close(), close idle keep-alive connections
  4. Drain in-flight work  → requests finish (with a deadline)
  5. Stop background work  → job workers, queue consumers, cron
  6. Close dependencies    → DB pool, Redis, tracing/log flush
  7. Exit                  → code 0 if clean, 1 if forced
Enter fullscreen mode Exit fullscreen mode

Closing the database pool at step 6 and not step 3 is the key insight for transactions: requests that are still running need their connections.

Step 1 and 2: Fail readiness first, then wait

On Kubernetes, termination is not a clean sequence. When a pod is deleted, two things happen concurrently: the pod is removed from Service endpoints, and the container receives SIGTERM. Routing updates take a moment to reach every proxy and load balancer. If you close your listener the instant SIGTERM arrives, some traffic is still being routed to you and hits a closed port.

The fix is to make your app report not ready, then pause briefly before closing the listener.

let shuttingDown = false;

app.get('/healthz', (_req, res) => res.sendStatus(200));            // liveness: process is alive
app.get('/readyz', (_req, res) => res.sendStatus(shuttingDown ? 503 : 200)); // readiness
Enter fullscreen mode Exit fullscreen mode

You can do the pause in the app (await sleep(5000) before server.close()), or with a Kubernetes preStop hook:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
Enter fullscreen mode Exit fullscreen mode

Either works. Just make sure it's counted in your time budget (see below).

Step 3 and 4: Stop accepting, drain in-flight requests

Node's server.close() stops the server from accepting new connections and fires its callback once all existing connections have ended. Two details trip people up:

HTTP keep-alive. Idle keep-alive connections count as open connections, so close() can appear to hang until they time out. In Node 19+, server.close() closes idle connections for you. On Node 18.2+, call server.closeIdleConnections() explicitly. For connections that are busy during shutdown, tell clients not to reuse them by sending Connection: close on responses:

app.use((req, res, next) => {
  if (shuttingDown) res.set('Connection', 'close');
  next();
});
Enter fullscreen mode Exit fullscreen mode

A hard deadline. One stuck request (a client that never finishes reading) would hold shutdown open until SIGKILL. Set your own deadline, a bit shorter than the platform's, and force-close the stragglers when it hits:

function closeServer(server: http.Server, deadlineMs: number): Promise<void> {
  return new Promise((resolve, reject) => {
    const force = setTimeout(() => {
      log.warn('HTTP drain deadline hit; closing remaining connections');
      server.closeAllConnections?.();             // Node 18.2+
    }, deadlineMs);

    server.close((err) => {
      clearTimeout(force);
      err ? reject(err) : resolve();
    });
    server.closeIdleConnections?.();              // Node 18.2+ (no-op need on 19+)
  });
}
Enter fullscreen mode Exit fullscreen mode

Long-lived connections need special handling. server.close() doesn't close WebSockets or SSE streams for you. Track them and close them explicitly (for WebSockets, send a close frame with code 1001, "going away", so clients know to reconnect elsewhere).

Step 5: Stop background work

If the same process runs queue workers or consumers, stop them before closing the resources they use.

import { Worker } from 'bullmq';

// worker.close() waits for currently active jobs to finish by default
await worker.close();
Enter fullscreen mode Exit fullscreen mode

With BullMQ, worker.close() stops fetching new jobs and waits for active ones to complete. If a job outlasts your grace period and the process is killed, BullMQ will eventually consider that job stalled and retry it, which means your job handlers must be idempotent. Graceful shutdown reduces duplicate runs. It doesn't eliminate them. The same applies to any at-least-once consumer (Kafka, SQS, RabbitMQ): stop consuming, finish or negotiate the in-flight message, and commit or ack appropriately before disconnecting.

Stop timers and cron schedulers too, so no new work starts while you're draining.

Step 6: Close dependencies, and why order protects your transactions

Now that nothing is using them, close the shared resources.

await pool.end();          // pg: ends idle clients, and checked-out clients as they're released
await redis.quit();        // ioredis: graceful (waits for pending replies); disconnect() is abrupt
await otelSdk.shutdown();  // flush traces/metrics
await logger.flush?.();    // flush buffered logs, if your logger buffers
Enter fullscreen mode Exit fullscreen mode

Why this order protects database transactions: a request in the middle of BEGIN ... COMMIT holds a checked-out connection. If you close the pool (or exit) first, that transaction is cut off. Because we waited for requests to drain in step 4, those transactions have committed or rolled back by the time we reach pool.end().

What if you are killed mid-transaction anyway (SIGKILL, OOM, node failure)? PostgreSQL has your back on atomicity: when the connection drops, the server rolls back the uncommitted transaction automatically. Your data won't be half-written inside the database.

The danger is everything outside the database. If the handler already charged a card, published a message, or sent an email before the commit, killing the process leaves the world in an inconsistent state: payment taken, no order recorded. Graceful shutdown narrows that window, but it can't close it. The durable fixes are:

  • Idempotency keys on external calls and on the endpoints themselves, so a retry is safe
  • The outbox pattern: write the event in the same transaction as the data, and publish it from a separate relay
  • Keeping transactions short, with no slow network calls inside them (which also protects your connection pool)

Don't let a deadline become a mid-commit kill

If drain is taking too long, you'd rather abort work cleanly than be SIGKILLed at a random point. Tie request work to a shutdown signal so handlers can stop and roll back on their own terms:

const shutdownController = new AbortController();

// in shutdown(), when the drain budget is nearly spent:
shutdownController.abort(new Error('server shutting down'));

// in a handler / withTransaction callback:
shutdownController.signal.throwIfAborted();   // before starting the next step
Enter fullscreen mode Exit fullscreen mode

An explicit abort leads to an explicit ROLLBACK and a clean error response, which is always better than a severed connection.

Putting it together 

// server.ts
import http from 'node:http';
import express from 'express';

const GRACE_MS = Number(process.env.SHUTDOWN_GRACE_MS ?? 25_000); // < platform grace period
const READINESS_DELAY_MS = Number(process.env.READINESS_DELAY_MS ?? 5_000);
const HTTP_DRAIN_MS = Number(process.env.HTTP_DRAIN_MS ?? 15_000);

const app = express();
let shuttingDown = false;

app.use((_req, res, next) => {
  if (shuttingDown) res.set('Connection', 'close');
  next();
});
app.get('/healthz', (_req, res) => res.sendStatus(200));
app.get('/readyz', (_req, res) => res.sendStatus(shuttingDown ? 503 : 200));
// ...your routes

const server = http.createServer(app).listen(Number(process.env.PORT ?? 3000));

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function shutdown(signal: string) {
  if (shuttingDown) {                       // second signal: operator is impatient, stop waiting
    log.warn({ signal }, 'second signal received; forcing exit');
    process.exit(1);
  }
  shuttingDown = true;
  log.info({ signal }, 'shutdown started');

  // Absolute backstop: never outlive the platform's grace period
  setTimeout(() => {
    log.error('graceful shutdown timed out; forcing exit');
    process.exit(1);
  }, GRACE_MS).unref();

  let exitCode = 0;
  try {
    await sleep(READINESS_DELAY_MS);              // 1-2. unready, let routing catch up
    await closeServer(server, HTTP_DRAIN_MS);     // 3-4. stop accepting, drain requests
    await worker.close();                         // 5.   finish active jobs
    await pool.end();                             // 6.   DB pool (nothing is using it now)
    await redis.quit();
    await otelSdk.shutdown();
    log.info('shutdown complete');
  } catch (err) {
    log.error({ err }, 'error during shutdown');
    exitCode = 1;
  }
  process.exit(exitCode);                         // 7.
}

process.on('SIGTERM', () => void shutdown('SIGTERM'));
process.on('SIGINT', () => void shutdown('SIGINT'));
Enter fullscreen mode Exit fullscreen mode

Notes on the details:

  • process.exit() at the end is fine once cleanup is done. The problem earlier was calling it instead of cleanup. Without it, a stray open handle (a forgotten interval, a socket) can keep the process alive until SIGKILL.
  • The .unref() on the backstop timer means it never keeps the process alive by itself.
  • The second-signal guard makes Ctrl-C twice do what developers expect, and prevents running the shutdown sequence twice.

Crashes should shut down, too

An uncaughtException or unhandledRejection means your process is in an unknown state. Don't try to carry on, but do try to exit cleanly:

process.on('unhandledRejection', (reason) => {
  log.error({ reason }, 'unhandled rejection');
  void shutdown('unhandledRejection');
});
process.on('uncaughtException', (err) => {
  log.fatal({ err }, 'uncaught exception');
  void shutdown('uncaughtException');
});
Enter fullscreen mode Exit fullscreen mode

For crash paths, consider a shorter drain budget and a non-zero exit code so your orchestrator records a failure.

Budget your time

The pieces must fit inside the platform's grace period. For a Kubernetes default of 30 seconds:

Phase Budget
Readiness delay / preStop 5s
HTTP drain 15s
Workers, pool, flush 5s
Safety margin 5s

Then set terminationGracePeriodSeconds to match what you actually need. If your longest legitimate request takes 60 seconds, a 30-second grace period guarantees you'll kill some of them, so either raise the grace period or shorten the work. And keep your in-app deadline (GRACE_MS) comfortably below the platform's.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec: { command: ["sleep", "5"] }
      readinessProbe:
        httpGet: { path: /readyz, port: 3000 }
Enter fullscreen mode Exit fullscreen mode

Container gotchas that silently break all of this

Signals never reach your app

This one is brutal because your shutdown code is perfect and never runs.

# ❌ npm and shell-wrapped commands often don't forward signals to node
CMD npm start

# ✅ exec form, running node directly
CMD ["node", "dist/server.js"]
Enter fullscreen mode Exit fullscreen mode

If npm or a shell is PID 1, SIGTERM goes to it, and it frequently doesn't pass it on. Docker then waits out the full grace period and SIGKILLs you. Use the exec form of CMD/ENTRYPOINT and run node directly (or use --init / tini as a minimal init process).

Node as PID 1

On Linux, PID 1 gets special treatment: signals without an explicit handler are ignored, not defaulted. Once you register process.on('SIGTERM', ...) this is fine, but it's another reason the handler must actually be installed.

Dev tooling

Watchers such as nodemon or tsx watch restart your process on file change using signals as well. They're good for exercising your shutdown path locally.

Frameworks and libraries that help

You don't have to hand-roll everything:

  • Fastify: await app.close() runs onClose hooks for plugins, and stops the server
  • NestJS: call app.enableShutdownHooks() and implement OnApplicationShutdown / OnModuleDestroy
  • http-terminator or stoppable: utilities that track connections for you and implement drain-with-deadline for plain http servers

Whatever you use, still verify the behavior with a test. Defaults vary by version.

Test it, don't trust it

Shutdown bugs only show up during deploys, so reproduce them on purpose. Spawn the real server, start a slow request, send SIGTERM, and check that the request completes and the process exits cleanly:

// shutdown.test.ts
import { spawn } from 'node:child_process';
import { it, expect } from 'vitest';

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

it('finishes in-flight requests on SIGTERM, then exits 0', async () => {
  const child = spawn('node', ['dist/server.js'], {
    env: { ...process.env, PORT: '4123', READINESS_DELAY_MS: '0' },
    stdio: 'inherit',
  });
  const exited = new Promise<number | null>((resolve) => child.on('exit', resolve)); // attach before kill

  await waitForOk('http://localhost:4123/readyz');

  const slow = fetch('http://localhost:4123/slow?ms=2000');   // a test route that sleeps
  await sleep(200);                                           // make sure it's in flight
  child.kill('SIGTERM');

  const res = await slow;
  expect(res.status).toBe(200);                               // request was NOT dropped
  expect(await exited).toBe(0);                               // clean exit
});

it('exits non-zero if drain exceeds the deadline', async () => {
  // start with a tiny SHUTDOWN_GRACE_MS, hit /slow?ms=60000, send SIGTERM,
  // and assert the process exits with code 1 well before 60s
});
Enter fullscreen mode Exit fullscreen mode

For a stronger check, run a load test (autocannon, k6) against a staging deployment while performing a rolling restart, and watch for non-2xx responses. A well-behaved service shows zero errors during the roll.

Checklist

  • ✅ Handle SIGTERM and SIGINT; install the handlers before listening
  • ✅ Fail readiness first and pause briefly for routing to catch up
  • ✅ server.close() plus closing idle connections; send Connection: close while draining
  • ✅ Track and close WebSockets/SSE explicitly
  • ✅ Stop job workers and consumers before closing their dependencies
  • ✅ Close the DB pool and other clients after in-flight work has finished
  • ✅ Hard deadline shorter than the platform's grace period; force-exit beyond it
  • ✅ Idempotency and the outbox pattern for side effects outside the database
  • ✅ CMD ["node", ...] in exec form so signals reach your process
  • ✅ A test that sends SIGTERM mid-request

Key takeaways

  1. Deploys are shutdowns. Every release sends SIGTERM to your app, then SIGKILL after the grace period.
  2. server.close() then process.exit() is a bug. Wait for the close to complete before exiting.
  3. Order is everything: unready, stop accepting, drain requests, stop workers, close the pool, exit.
  4. Close the database pool last, so in-flight transactions can finish. If the process dies anyway, Postgres rolls back uncommitted work, but side effects outside the database need idempotency or an outbox.
  5. Always have a deadline, shorter than the platform's grace period, so a stuck request can't turn a clean shutdown into a SIGKILL.
  6. Check your container. If signals never reach node, none of the above matters.
  7. Test it by sending SIGTERM during a slow request.

What's the worst shutdown bug you've seen in production? Share it in the comments.

Further reading: the Node.js docs for http.Server.close() and process signal events, and the Kubernetes documentation on pod termination.

Top comments (0)