Drain In-Flight HTTP Before Kubernetes Sends SIGKILL

Kubernetes’s default terminationGracePeriodSeconds is 30. That is not a suggestion and it is not process.exit(0). SIGTERM means: stop taking new work, finish what you already accepted, then leave. Miss the window and kubelet sends SIGKILL. The in-flight request you already 200-OK’d on the client side never gets a body.

Google autocomplete for graceful shutdown currently offers golang, spring boot, kubernetes, and nodejs. terminationGracePeriodSeconds fills in with default, default value, 30, k8s, openshift, karpenter. kubernetes sigterm fills in with grace period, vs sigkill, timeout, pid 1. People search the signal. The interview is whether you spend the 30 seconds or throw them away.

What is the interviewer actually asking?

Most answers stop at “I listen for SIGTERM.” Node’s own process docs make that the trap. 'SIGTERM' and 'SIGINT' have default handlers that exit with 128 + signal number. The moment you install a listener, that default goes away. Node will no longer exit. If your handler calls process.exit(0) on the first tick, you did the same thing as having no handler: you tore down sockets that still had a request on them.

The Kubernetes pod-lifecycle page is more specific than a blog post. On delete, the Pod is marked Terminating. The kubelet runs any preStop hook. Then the container runtime sends TERM to process 1. When the grace period expires, remaining processes get SIGKILL. The default grace period in the Pod API is 30 seconds. preStop is not extra time. It comes out of the same 30.

sequenceDiagram
  participant K as kubelet
  participant P as PID 1
  participant S as in-flight request
  K->>P: SIGTERM (grace = 30s)
  P->>P: server.close()
  P->>S: still writing the body
  Note over K,P: preStop + drain share the 30s
  alt in-flight hits 0
    P->>K: exit 0
  else still busy at 30s
    K->>P: SIGKILL
  end

The other half of the contract sits in EndpointSlice. Terminating endpoints report ready: false, so new regular traffic should stop arriving. That does not cancel the POST that already made it onto this process. You still have to drain it.

Why doesn't server.close() by itself finish the job?

It stops accepting new connections. Since Node 19.0.0 it also reaps idle keep-alive sockets before the callback fires. The Node docs are explicit: server.close() “closes all connections connected to this server which are not sending a request or waiting for a response.” Active requests stay. That is the correct default. You want those to finish.

Two follow-ups the interviewer will poke.

First: you are on Node 18 in production. Then close() does not reap idle keep-alive. server.close() never fires because a phone from 40 seconds ago is still sitting in keep-alive. The documented escape is closeIdleConnections(), called after close(), because a connection accepted between the two calls would otherwise race. Node 19 made that automatic. Libraries that still support 18 call it anyway. Harmless on 24.

Second: a request that will not finish inside the grace period. A 2-minute report download. A webhook you forwarded to a partner who is slow. close() waits forever. Kubernetes does not. At second 30 you get SIGKILL and the client sees a TCP reset. The Node API for that case is closeAllConnections(), again after close(). It is forceful. The docs say so. Use it as the deadline, not as step one.

How small is the pad version?

A gate, not an http.Server. Clock and grace are injected so the tests do not sleep. wrap is the request. begin is SIGTERM. wait is “I am allowed to exit.” tickForce is second 30.

const REFUSED = "refused";
const FORCED = "forced";
const DRAINED = "drained";

function createShutdownGate({ now, graceMs = 30_000 } = {}) {
  let draining = false;
  let forceAt = null;
  let inFlight = 0;
  const waiters = [];
  const events = [];

  function maybeResolve() {
    if (inFlight === 0) {
      for (const w of waiters.splice(0)) w({ ok: true, reason: DRAINED });
    }
  }

  function begin() {
    if (draining) return;
    draining = true;
    forceAt = now() + graceMs;
    events.push("begin");
  }

  function wrap(work) {
    if (draining) {
      events.push("refuse");
      return Promise.reject(Object.assign(new Error(REFUSED), { code: REFUSED }));
    }
    inFlight += 1;
    events.push("accept");
    return Promise.resolve()
      .then(work)
      .finally(() => {
        inFlight -= 1;
        maybeResolve();
      });
  }

  function wait() {
    if (!draining) throw new Error("wait_before_begin");
    if (inFlight === 0) return Promise.resolve({ ok: true, reason: DRAINED });
    if (forceAt !== null && now() >= forceAt) {
      events.push("force");
      return Promise.resolve({ ok: false, reason: FORCED, inFlight });
    }
    return new Promise((resolve) => waiters.push(resolve));
  }

  function tickForce() {
    if (!draining || forceAt === null || now() < forceAt) return null;
    if (waiters.length === 0 && inFlight === 0) {
      return { ok: true, reason: DRAINED };
    }
    events.push("force");
    const leftover = inFlight;
    for (const w of waiters.splice(0)) {
      w({ ok: false, reason: FORCED, inFlight: leftover });
    }
    return { ok: false, reason: FORCED, inFlight: leftover };
  }

  return { begin, wrap, wait, tickForce, isDraining: () => draining, inFlight: () => inFlight, events: () => events.slice() };
}

Production wires begin to process.on("SIGTERM"), wrap around the request listener, wait to server.close()'s callback, and tickForce to closeAllConnections(). The pad exists so you can fail the contracts in 20 seconds without standing up kubelet.

Which four contracts do you have to show?

Contract 1: a request accepted before SIGTERM still completes. New work after begin is refused. Event order is accept, begin, refuse.

const clock = { t: 0 };
const gate = createShutdownGate({ now: () => clock.t, graceMs: 30_000 });

const slow = gate.wrap(
  () => new Promise((resolve) => setImmediate(() => resolve("ok-inflight"))),
);
gate.begin();
assert.equal(gate.inFlight(), 1);
await assert.rejects(() => gate.wrap(async () => "should-not-run"), { code: REFUSED });
assert.equal((await gate.wait()).reason, DRAINED);
assert.equal(await slow, "ok-inflight");
assert.deepEqual(gate.events(), ["accept", "begin", "refuse"]);

That is the whole drain. If you cannot show this, you do not have graceful shutdown. You have a signal handler that prints a log line.

Contract 2: process.exit on the first tick drops the in-flight request. The work never settles. The process is gone anyway.

function createNaiveExit() {
  const events = [];
  return {
    wrap: (work) => {
      events.push("accept");
      return Promise.resolve().then(work);
    },
    exitNow: () => {
      events.push("exit");
      return { ok: false, reason: "dropped" };
    },
    events: () => events.slice(),
  };
}

const naive = createNaiveExit();
naive.wrap(() => new Promise(() => {})); // still on the wire
assert.equal(naive.exitNow().reason, "dropped");

This is the rolling-deploy 502. The replica coming up is healthy. The replica going down reset a connection the load balancer already handed it.

Contract 3: a request that outlives the grace period is forced. inFlight stays 1. SIGKILL does not wait for your await.

clock.t = 0;
const late = createShutdownGate({ now: () => clock.t, graceMs: 30_000 });
late.wrap(() => new Promise(() => {}));
late.begin();
clock.t = 30_000;
const forced = late.tickForce();
assert.equal(forced.reason, FORCED);
assert.equal(forced.inFlight, 1);

The spoken version: I would not pretend a 2-minute export fits in 30 seconds. Either I fail that request with 503 on SIGTERM and let the client retry another replica, or I raise terminationGracePeriodSeconds to a number I measured. I would not sit in close() and hope kubelet is patient.

Contract 4: preStop spends the same budget. At t=10_000 the force window has not opened. At t=29_999 it still has not. At t=30_000 an idle drain is allowed to finish. There is no bonus 30.

clock.t = 0;
const prestop = createShutdownGate({ now: () => clock.t, graceMs: 30_000 });
prestop.begin();
clock.t = 10_000;
assert.equal(prestop.tickForce(), null);
clock.t = 29_999;
assert.equal(prestop.tickForce(), null);
clock.t = 30_000;
assert.equal(prestop.tickForce().reason, DRAINED);

The Kubernetes docs even give a 2-second one-off extension if preStop is still running when the grace period expires. That is kubelet covering the hook, not extra drain time for your HTTP server. If the hook sleeps 25 seconds, you have about 5 seconds of SIGTERM left. I have seen a preStop: sleep 20 copied from a nginx snippet into a Node service. Then people wonder why in-flight GraphQL requests reset during every rollout.

I ran those four with node sigterm-drain.mjs. All green.

What do you say out loud in the backend / SRE round?

Something like:

I treat SIGTERM as a drain, not an exit. First I stop being a Service endpoint: fail readiness, then server.close() so we do not accept new TCP. In-flight requests keep running. I wait on a counter, not on a sleep(30). If the counter hits zero I exit 0. If the grace period is about to expire I call closeAllConnections(), log how many were still open, and exit. preStop time is subtracted from the same 30 seconds, so I would not sleep in the hook. TERM has to reach PID 1. If the container PID 1 is a shell that does not forward signals, none of this runs and kubelet SIGKILLs a process that never knew it was dying.

The AceRound DevOps engineer interview guide puts rolling deploys and probe failures in the production-scenario bucket. aceround.app — AI interview assistant is useful when the next hour is talking through the 30-second window while someone interrupts you. It does not replace running the four contracts above.

A few follow-ups I would expect:

Follow-up	What I would not say
Can't we just raise the grace period to 300?	"Then we never have to drain." Long grace periods make rollouts slow. Measure p99 in-flight duration, set grace to that plus a buffer, still refuse new work immediately.
What about HTTP/2 or WebSockets?	closeAllConnections() does not destroy upgraded sockets. Those need their own shutdown. I would say so instead of waving server.close().
Why fail readiness before SIGTERM?	Because EndpointSlice removal is not instantaneous. A preStop that only sleeps, with no readiness fail, still takes traffic for part of the window. Fail the probe first.
process.on("SIGTERM", () => process.exit(0))?	That uninstalls Node's default handler and then exits anyway. Same dropped request, extra confidence.

I would not ship this Map as the process manager. Production is http.Server plus a readiness flag plus a deadline timer. The pad exists so you can fail “new work after SIGTERM” in 20 seconds instead of drawing a rectangle labelled “graceful” on a whiteboard.

FAQ

Does server.close() wait for in-flight requests?

It stops new connections and, since Node 19, idle keep-alive. Active requests keep running until they end or you call closeAllConnections().

Is SIGTERM the same on Windows?

Node can listen for 'SIGTERM' on Windows. Kubernetes Linux nodes send TERM to PID 1. Do not demo this with Ctrl+C and assume it is the same path. Ctrl+C is SIGINT.

Why 30 seconds?

That is the Pod API default when terminationGracePeriodSeconds is nil. kubectl delete uses the same default. You can set it. You cannot ignore SIGKILL after it.

If my handler never calls process.exit, what happens?

Node no longer exits on SIGTERM once a listener exists. You sit in Terminating until kubelet SIGKILLs you at second 30. The log line that says “received SIGTERM” is not a shutdown.

Should the drain reject new HTTP with 503 or just close the listen socket?

Both. Close the listen socket so the kernel refuses new TCP. Return 503 for anything already accepted that you choose not to run. Clients retry the next replica.

If you already handle SIGTERM in production, does the handler wait for inFlight === 0, or does it process.exit on the first tick?

Drafted with AI assistance, then edited. The Node contracts were run locally before publishing. Grace period default of 30 seconds and TERM-then-KILL: Kubernetes Pod lifecycle and the Pod API. server.close / closeIdleConnections / closeAllConnections: Node.js HTTP docs, including the v19.0.0 idle-connection change. SIGTERM listener replacing the default exit: Node.js process signal events.