At 02:00 the deployment script reported success. The log showed the worker finishing, the health check passing, and the script exiting with code zero. What the log did not show was the child process it had started ten minutes earlier, still alive, still holding the port, still appending to a log file that had already been rotated out from under it. The next deploy failed because the port was taken, and the postmortem said the obvious thing: the previous run did not clean up. Nobody had typed a wrong command. The script had simply killed the wrong thing.
Every language that starts a subprocess ships a kill button, and in every language that button is a lie half of the time. In Python the lie has three parts, and they fail in a fixed order: what terminate() actually sends, who is listening when the signal arrives, and what happens to the process after the signal lands. Most cleanup bugs live in the gap between the call and the corpse, and most of them survive code review because the happy path — a child that exits on its own — never exposes them. This article walks that gap from the first signal to the reaped process, and ends with a termination protocol that leaves nothing running, plus tests that prove it without a single sleep.
What terminate() Actually Sends
When you call process.terminate() on a subprocess.Popen object, Python sends the child a SIGTERM. SIGTERM is not a kill. It is a request to exit, delivered to a signal handler that the child may have replaced, may be ignoring, or may not be ready to handle yet. A well-behaved process exits. A process with a handler that flushes buffers, saves state, or finishes a network call can take seconds or minutes. A process with no handler at all exits immediately. And a process with a handler that decides, in code you do not control, not to exit simply keeps running.
process.kill() sends SIGKILL, which the kernel delivers without asking the process anything. Nothing can ignore it, nothing can defer it, and the process gets no chance to clean up. The two calls look like a toggle between gentle and brutal. In reality they are the two endpoints of a negotiation that the parent must manage, because the parent cannot know in advance which kind of child it is talking to.
import signal
import subprocess
proc = subprocess.Popen(["worker", "--queue", "high"])
proc.terminate() # SIGTERM: a polite request
try:
proc.wait(timeout=5) # give the handler time to clean up
except subprocess.TimeoutExpired:
proc.kill() # SIGKILL: the kernel takes over
proc.wait()
Who Actually Receives the Signal
The second lie is delivery. With default settings, the signal goes to exactly one process: the direct child. If that child is a shell — and it is whenever you pass shell=True, and it often is when the executable you named is a wrapper script, a Makefile, or a language runner that spawns its own workers — the child forwards nothing. Your SIGTERM reached the shell sitting on top of a tree of three more processes, and the shell has its own ideas about what to do with signals meant for its children.
Even without a shell, real children spawn grandchildren. A build tool starts a compiler, the compiler starts a linker, a media encoder fans out to worker processes. The parent holds a handle only to the top of that tree, and killing the top leaves the rest running. The rest is the part holding the port, the lock file, or the half-written database row.
# shell=True wraps the command in /bin/sh -c
proc = subprocess.Popen(
"ffmpeg -i in.mp4 out.mp4", # sh spawns ffmpeg as its own child
shell=True,
)
proc.terminate() # kills the shell; ffmpeg keeps transcoding
Killing the Whole Family
The fix is to stop addressing the process and start addressing the group. POSIX systems group processes precisely so that a signal can be aimed at a tree. Start the child in its own session with start_new_session=True, which makes it a process-group leader, and then signal every member of the group with os.killpg. The child you started, the grandchildren it spawned, and the shell in between all receive the signal in one call.
proc = subprocess.Popen(
["worker", "--queue", "high"],
start_new_session=True, # the child leads its own process group
)
os.killpg(proc.pid, signal.SIGTERM) # the whole tree, politely
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
os.killpg(proc.pid, signal.SIGKILL) # the whole tree, for real
proc.wait()
Grouping changes the meaning of the kill from "tell the top to stop" to "tell everyone in this unit to stop". It is the difference between phoning the manager and evacuating the floor, and for a process tree it is the only way to be sure the floor is actually empty.
The Timeout That Lets the Child Keep Running
The third lie is about time. Popen.communicate(timeout=N) is the most common way to bound a subprocess, and its failure mode is quietly famous: the method raises TimeoutExpired, the caller catches it, and the child keeps running. The timeout bounded the parent's patience, not the child's life. The trap is extra nasty because it is not silent — the code clearly knows the call timed out — and yet the natural reaction, log and continue, is exactly the reaction that leaves the orphan behind.
try:
out, err = proc.communicate(timeout=10)
except subprocess.TimeoutExpired:
logger.error("worker exceeded 10 seconds") # logged, and...
# ...the worker is still running, still holding the queue
There is a second trap inside the same method. After a timeout, communicate() has left the pipes in an inconsistent state, and the standard library documentation is explicit about the recovery: kill the process first, then call communicate() again to drain whatever remains in the buffers. The second call is not optional bookkeeping. A child that was writing a large payload when it died can leave the parent blocked on a full pipe buffer unless the remaining bytes are read.
except subprocess.TimeoutExpired:
proc.kill()
out, err = proc.communicate() # drain the pipes, then reap
A Termination Protocol That Ends
Put the pieces together and a reliable termination is a fixed sequence of four steps: signal the group politely, wait with a deadline, signal the group forcefully, and wait again until the corpse is collected. The deadline between the two signals is what gives a well-behaved process the chance to do its cleanup, and the second wait is what guarantees the slot in the process table is actually free before the parent moves on.
def terminate_tree(proc: subprocess.Popen, grace: float = 5.0) -> int:
os.killpg(proc.pid, signal.SIGTERM)
try:
return proc.wait(timeout=grace)
except subprocess.TimeoutExpired:
os.killpg(proc.pid, signal.SIGKILL)
return proc.wait()
The grace period is a policy decision, not a tuning knob: it is the amount of cleanup time you are willing to pay before you accept the risk of a half-written file. The protocol works because each step is unconditional — no step asks whether the child looks cooperative, and no step re-checks the process state before deciding. The only conditional in the whole flow is the timeout, and a timeout is a deadline, not a guess about the child's mood.
Zombies, Orphans, and Who Reaps Them
A process that exits is not finished. It becomes a zombie: an entry in the process table carrying an exit status, waiting for its parent to read that status with wait(). A zombie costs almost nothing — no CPU, no memory worth naming — but it occupies a process-table slot, and a system with enough zombies eventually refuses to spawn new processes. The kernel keeps the corpse around because the parent might want to know how the child died. The parent's job is to collect the body.
Popen.wait() and communicate() do the reaping for you. The subtle part is what happens when the parent never calls them. A long-lived worker that spawns one-shot children and forgets to wait on each one accumulates zombies until the machine notices. A child whose parent died first is adopted by the nearest surviving ancestor, which usually means it stops being your problem at the same moment your code stops running — but only if your code actually stops. In a process that catches errors and keeps going, an un-reaped child stays un-reaped.
The asyncio world has the same shape with different plumbing. create_subprocess_exec returns a Process with the same wait() and communicate() methods, and the same escalation applies. The extra hazard is that a bare exception handler around an un-awaited subprocess leaves the child running until the event loop itself finishes, and the loop will happily finish while the child keeps working. Termination is not a detail the event loop manages for you.
Testing Lifecycles Without Sleep
Lifecycle bugs are invisible to the tests people usually write, because the usual test starts a process, asserts on its output, and lets it exit naturally. Every failure mode in this article involves a child that refuses to exit, so the test has to create one deliberately. The trick is to use Python itself as the unruly child: a short script that catches SIGTERM, prints that it received it, and keeps running until something stronger arrives.
STUB = """
import signal
import sys
import time
def hold(signum, frame):
print("got SIGTERM, ignoring", flush=True)
signal.signal(signal.SIGTERM, hold)
print("ready", flush=True)
while True:
time.sleep(0.05)
"""
def test_terminate_tree_reaches_everyone():
proc = subprocess.Popen(
[sys.executable, "-c", STUB],
start_new_session=True,
stdout=subprocess.PIPE,
)
assert proc.stdout.readline().strip() == "ready"
os.killpg(proc.pid, signal.SIGTERM)
assert proc.poll() is None # the polite step was ignored
os.killpg(proc.pid, signal.SIGKILL)
assert proc.wait(timeout=5) == -signal.SIGKILL
Asserting the middle state — alive after SIGTERM, dead after SIGKILL — pins the test exactly where the bug lives. The same stub works for the shell case: start it with shell=True, terminate the top, and verify that the grandchild survives a plain terminate() but not a killpg. Every assertion is about process state, not wall-clock timing, so the test is deterministic and never sleeps.
Terminating a subprocess is not a call. It is a protocol with four obligations: know what your signal asks for, know who is listening, address the whole tree, and collect the corpse. Python provides every piece — SIGTERM, start_new_session, os.killpg, wait — and none of it works if the pieces run in the wrong order, and none of it is checked by the tests that let the child exit on its own. The next time a deploy fails because a port is taken, the question is not whether the previous run killed its child. The question is whether the child was killed for real.
Originally published on Dispatch.
Top comments (0)