Kubernetes graceful shutdown coordinates SIGTERM handling, traffic draining, rollout capacity, and disruption controls into one production contract.
Original article: Kubernetes graceful shutdown: drain traffic before processes exit
A reliable Pod must stop accepting new work, give routing components time to converge, finish or safely release in-flight work, close connections, and exit before its single termination budget expires. Kubernetes can send signals and update endpoints, but it cannot complete a transaction, acknowledge a queue message, or make a side effect idempotent for your application.
This distinction matters during rollouts and node maintenance. A Deployment can look healthy while users still see connection resets, workers duplicate jobs, or containers end with forced SIGKILL.
Start with a shutdown contract
Treat shutdown as an application state machine with explicit invariants:
- The instance changes from accepting to draining exactly once.
- It rejects or redirects new long-running work after draining starts.
- Work already acknowledged is completed, released with a lease, or made safe to retry.
- Listeners, workers, buffers, and database clients reach a known state before exit.
- Every cleanup step has a deadline shorter than the remaining Pod grace period.
The exact contract depends on the workload:
- An HTTP API stops accepting new connections and waits for active requests.
- A queue consumer stops polling before it waits for handlers.
- A WebSocket service asks clients to reconnect, then closes within a bounded window.
- A leader releases or expires its lease before exiting.
Do not hide these different semantics behind one generic delay. The shutdown path should also be idempotent: a second signal or callback must observe the existing drain instead of starting another flush or acknowledging the same job twice.
Understand the real termination timeline
When a Pod gets a deletion timestamp, multiple activities begin close together:
- The kubelet starts the local termination flow.
- If a preStop hook exists, it runs before the main process receives TERM.
- The control plane marks the Pod as terminating and updates Service endpoint state.
- The application drains work after receiving the signal.
- Anything still running when the grace period expires is forcibly stopped.
The important detail is that the grace countdown starts before preStop. A 15-second hook inside a 60-second budget does not leave another 60 seconds for the application; it leaves roughly 45.
Kubernetes EndpointSlices expose terminating, serving, and ready conditions. A terminating endpoint is normally no longer ready for ordinary Service traffic, but external load balancers, Ingress controllers, service meshes, DNS caches, and keep-alive connections may converge at different speeds. Endpoint removal is necessary; it is not proof that no new packet can arrive.
Official reference: Pod termination flow and EndpointSlice conditions.
Make the process own graceful shutdown
Register signal handling before the application reports readiness. A practical handler follows this order:
- set an internal draining flag and make readiness fail;
- stop accepting new connections or queue messages;
- tell long-lived clients to reconnect when the protocol supports it;
- wait for in-flight work until a bounded deadline;
- release leases, flush telemetry, and close database clients;
- exit with a result that distinguishes clean completion from timeout.
The container's process 1 must actually receive and forward the stop signal. Shell wrappers that do not use exec, detached child processes, and custom supervisors can swallow or delay SIGTERM. Test the built container image, not only the framework's local development server.
Readiness is a traffic-admission signal, not a shutdown executor. Changing readiness to false helps remove the Pod from normal routing, but it does not close existing connections or finish current work.
Use preStop as a small propagation buffer
A short preStop hook can help when routing components need time to observe endpoint changes before the application exits. Keep it deterministic and small because its runtime is charged against terminationGracePeriodSeconds.
Avoid putting business cleanup into a large shell script that depends on DNS or an unauthenticated admin endpoint. Native application signal handling is easier to test and can apply workload-specific deadlines.
If the hook only waits for routing convergence, measure that delay. Copying a 20- or 30-second sleep from another manifest is cargo-cult reliability.
Official reference: Container lifecycle hooks.
Calculate one explicit grace budget
Size the termination period from measured components:
grace budget >=
routing propagation buffer
+ longest accepted request or job slice
+ connection close time
+ durable flush or lease release
+ safety margin
Do not use average latency. Use a high percentile, the product deadline, and the retry contract. If requests may run for two minutes, a 30-second grace period cannot make shutdown graceful. Either shorten the request deadline, make the work resumable, transfer ownership, or reserve a larger budget.
Long grace periods are not free. Terminating Pods continue consuming CPU, memory, connections, and sometimes temporary rollout capacity. Often the better solution is a bounded unit of work and a faster checkpoint rather than an arbitrarily large timeout.
A PDB does not control your rollout
A PodDisruptionBudget limits concurrent voluntary disruptions performed through the Eviction API. kubectl drain respects it, so a PDB is useful for planned node maintenance and cooperating eviction clients.
It does not prevent involuntary failures such as node loss or a network partition. It also does not control a Deployment rolling update; rollout capacity is defined by the workload strategy.
| Control | What it protects |
|---|---|
| minAvailable / maxUnavailable in a PDB | Voluntary evictions |
| maxUnavailable in a Deployment | Serving capacity removed during rollout |
| maxSurge | Temporary replacement capacity |
| minReadySeconds | A replacement must remain ready before counting as available |
| progressDeadlineSeconds | Detects a stalled rollout; it does not automatically roll back |
A strict PDB cannot compensate for all replicas living on one node, and a perfect shutdown handler cannot help when the scheduler has no capacity for a replacement.
Official reference: Disruptions and PodDisruptionBudgets and Deployment rolling updates.
A complete starting manifest
The following example keeps a short propagation buffer while leaving most of a 45-second budget for the application:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 4
minReadySeconds: 10
progressDeadlineSeconds: 300
selector:
matchLabels:
app: orders-api
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: orders-api
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api
image: registry.example/orders-api:2026-10-18
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /ready
port: http
periodSeconds: 3
timeoutSeconds: 1
failureThreshold: 2
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: orders-api
spec:
minAvailable: 3
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app: orders-api
This manifest is a starting point, not a guarantee. The application still needs bounded SIGTERM handling, retry-safe work, and enough cluster capacity. Also verify that the image contains the shell and sleep; a distroless image often does not.
Observe shutdown as a first-class operation
Useful signals include:
- graceful_shutdown_duration_seconds;
- forced_shutdown_total;
- in_flight_requests or active_jobs;
- rejected_during_drain_total;
- connection resets and rollout-time 5xx errors;
- PDB disruptionsAllowed;
- EndpointSlice terminating and serving conditions.
Correlate them with the deployment revision, node-drain events, container exit reason, and resource pressure. Exit code 137 alone is ambiguous: forced termination and memory pressure can both lead to it.
Test failure instead of trusting YAML
Before calling the design reliable:
- Send requests that span the shutdown window and start a rollout.
- Verify that accepted responses finish without resets.
- Drain one node under load and confirm the PDB blocks an unsafe simultaneous eviction.
- Make one Pod unhealthy and verify the selected unhealthy eviction policy.
- Exceed the cleanup deadline deliberately and ensure the timeout is visible.
- Test involuntary cases a PDB cannot prevent: node loss, network interruption, and insufficient replacement capacity.
- For workers, kill the process after receiving a message but before acknowledgement and prove retry does not duplicate side effects.
Production checklist
- Handle SIGTERM in process 1.
- Stop new work before waiting for current work.
- Make cleanup idempotent and deadline-aware.
- Keep preStop short and count it inside the same grace budget.
- Measure routing convergence and request duration.
- Configure rollout capacity separately from the PDB.
- Spread replicas across failure domains.
- Exercise graceful and forced failures under load.
A shutdown is successful only when users see no unexplained errors and acknowledged work remains correct.
The expanded article, including additional decision guidance and references, is available at nooryasser.com.
Top comments (2)
One wrinkle worth adding: endpoint removal propagates with a delay, so requests can still land on a terminating Pod after SIGTERM. Keeping the app serving them normally until the drain deadline, instead of failing fast, avoids a 503 storm during every rollout. It is also worth tracking how many requests get served after SIGTERM, since that tells you how wide the convergence window really is.
This is can be solved by adding preStop hook, add sleep in prestop hook which delays the SIGTERM and in meantime pod will get removed from active endpoints