48-Hour Field Notes: SIGTERM Stopped the Container. atexit Never Touched the Lockfile.

Have you ever watched a container exit after docker stop, then found yesterday's lockfile still sitting on the volume? I spent two days chasing that leftover file, because every local Ctrl+C run flushed it like a good citizen. The remote process never called the cleanup I had registered with atexit, and the volume kept the stale lock. Why would the same Python file behave like two different programs once it left my laptop?

What I thought was happening

I treated atexit like a universal finally block for process shutdown, which felt reasonable after years of local development. The script wrote a lockfile at start, then registered a small callback that unlinked the path on the way out. On my laptop I interrupted with Ctrl+C, watched the print from the callback, and called the design done. Who would expect Docker's default stop signal to skip that entire atexit path without a single log line?

The original script

This labeled example is the smallest worker that shows the trap. It is not production code, and it is not a framework wrapper.

# lock_worker.py — labeled reproduction example
from __future__ import annotations

import atexit
import os
import time
from pathlib import Path

LOCK = Path("/tmp/lock_worker.lock")


def acquire() -> None:
    if LOCK.exists():
        raise SystemExit(f"stale lock: {LOCK}")
    LOCK.write_text(str(os.getpid()), encoding="utf-8")
    print(f"acquired {LOCK} pid={os.getpid()}", flush=True)


def release() -> None:
    print("atexit: release()", flush=True)
    try:
        LOCK.unlink()
    except FileNotFoundError:
        pass


def main() -> None:
    acquire()
    atexit.register(release)
    print("working; send SIGTERM to me", flush=True)
    while True:
        time.sleep(1)


if __name__ == "__main__":
    main()

Hour 0–8: the laptop lied to me

I started the script in a terminal and pressed Ctrl+C, which raises KeyboardInterrupt inside CPython. The interpreter then unwinds main, runs atexit callbacks, and the lockfile disappears like I expected. I repeated that loop a few times, because I wanted to believe the handler was solid. Does a successful local interrupt prove anything about a container stop? It does not.

Then I wrapped the same file in a tiny image and stopped the container from another terminal. The process vanished. The lockfile stayed. I stared at docker stop for a while, because the logs showed the working line and nothing from atexit: release().

# labeled example; pin whatever interpreter you actually run
FROM python:3-slim
WORKDIR /app
COPY lock_worker.py .
CMD ["python", "-u", "lock_worker.py"]

python lock_worker.py
# Ctrl+C → atexit: release() on the laptop

docker build -t lock-worker .
docker run --name lock-worker --rm -v /tmp:/tmp lock-worker
docker stop lock-worker
# no atexit line; /tmp/lock_worker.lock can still be present

Is the volume mount confusing the story, or is the process dying on a path Python never maps to shutdown? That question ate the rest of the morning.

Hour 8–24: reading the docs instead of the vibes

CPython's atexit documentation is blunt once you stop skimming it. Registered functions run on normal interpreter shutdown, including a clean SystemExit from main. They do not run when the process dies from a signal that Python does not handle, when os._exit() is called, or when a fatal internal error occurs. Docker's default stop signal is SIGTERM, which the docker stop reference still describes as SIGTERM first, then SIGKILL after the grace period. CPython does not install a SIGTERM handler that turns that signal into SystemExit. So the kernel tears the process down, and atexit never gets a vote.

SIGINT is the odd one that had been spoiling me. The interpreter handles SIGINT by raising KeyboardInterrupt, which often looks like a normal unwind if you do not swallow it. SIGKILL cannot be caught at all, which is what arrives after the stop grace period. Are we testing the signal we think we are testing? Most local sessions are not.

Shutdown decision table

How the process dies	Default CPython behavior	atexit runs?	Typical source
Reaching the end of main	Normal shutdown	Yes	Script finishes
sys.exit() / SystemExit	Normal shutdown	Yes	Intentional stop
SIGINT (Ctrl+C)	KeyboardInterrupt	Usually yes, if uncaught	Interactive terminal
SIGTERM, unhandled	Default OS terminate	No	docker stop, kill -TERM
os._exit()	Immediate C-level exit	No	Child processes, some daemons
SIGKILL	Cannot be caught	No	kill -9, grace period expiry

I printed that table into the notes because I kept mixing SIGTERM with Ctrl+C in conversation. They are not cousins. They are different exits wearing the same hoodie.

Hour 24–36: reproduce on a real Linux box

My laptop and Docker Desktop still argued about volume paths, so I wanted a boring Linux PID I could signal with kill. I used MonkeyCode's free server option for that Linux-side check, because I needed a shell that was not my laptop's terminal emulator.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I also used the free model access there to draft a tiny harness around signal.signal, then I rewrote most of it by hand after reading the signal docs. The draft logged from inside the raw SIGTERM handler, which is a tempting and unsafe habit. Signal handlers in Python are not a friendly place for file I/O, locks, or anything that might re-enter the interpreter. Did I still paste the first draft into the server? Unfortunately yes, then I watched it print half a line and hang.

Minimal reproduction on Linux

# labeled reproduction steps
python lock_worker.py &
pid=$!
sleep 1
ls -l /tmp/lock_worker.lock
kill -TERM "$pid"
wait "$pid"
echo "exit=$?"
ls -l /tmp/lock_worker.lock || true

You should see the process die, and you should still see the lockfile. Exit status 143 is 128 + 15, which is the conventional shell encoding for SIGTERM. No atexit banner. No unlink. That is the whole bug, and it is documented behavior rather than a Docker mystery.

Want an even smaller probe that does not involve a lockfile at all? This one-liner is enough to prove the callback never runs.

python -u - <<'PY' &
import atexit, os, time
atexit.register(lambda: print("atexit ran", flush=True))
print("pid", os.getpid(), flush=True)
time.sleep(60)
PY
pid=$!
sleep 1
kill -TERM "$pid"
wait "$pid"
# "atexit ran" should be missing

If that print appears on your machine, you are not sending unhandled SIGTERM, or something else already installed a handler. Check signal.getsignal(signal.SIGTERM) before you blame the volume again.

Hour 36–48: make SIGTERM look like a shutdown

The fix I will actually repeat is small. Handle SIGTERM by requesting a Python-level exit, then keep the real cleanup in atexit or in a finally block. That way SIGINT, SIGTERM, and a normal return from main share one release path. I still refuse to put the unlink directly in the signal handler, because I do not want to debug a half-written file next time.

# lock_worker_term.py — labeled example
from __future__ import annotations

import atexit
import os
import signal
import sys
import time
from pathlib import Path

LOCK = Path("/tmp/lock_worker.lock")
_stopping = False


def acquire() -> None:
    LOCK.write_text(str(os.getpid()), encoding="utf-8")
    print(f"acquired {LOCK} pid={os.getpid()}", flush=True)


def release() -> None:
    print("atexit: release()", flush=True)
    try:
        LOCK.unlink()
    except FileNotFoundError:
        pass


def _ask_exit(signum: int, frame: object) -> None:
    global _stopping
    if _stopping:
        return
    _stopping = True
    print(f"signal {signum}; requesting sys.exit", flush=True)
    sys.exit(128 + signum)


def main() -> None:
    acquire()
    atexit.register(release)
    signal.signal(signal.SIGTERM, _ask_exit)
    print("working; SIGTERM should now run atexit", flush=True)
    while True:
        time.sleep(1)


if __name__ == "__main__":
    main()

The _stopping flag is there because a second SIGTERM can arrive while sys.exit is already unwinding. I want one exit, one atexit run, and logs that a human can read. Is sys.exit from a signal handler elegant? No. Is it clearer than mixing cleanup into the handler itself? For this worker, yes.

What I would repeat as a checklist

1. Print a unique line from atexit, not just delete a file, so logs can prove the callback ran.
2. Send SIGTERM with kill -TERM before you ever involve Docker or a scheduler.
3. Confirm the shell exit code, because 143 is a clue and 0 is a different story.
4. Route SIGTERM through sys.exit, and keep the real work in atexit or finally.
5. Treat SIGKILL as a separate problem: persist state continuously, not only on shutdown.
6. Inspect signal.getsignal(signal.SIGTERM) if a parent process or framework already stole the handler.

After the handler was in place, the same kill -TERM printed both the signal line and atexit: release(), then the lockfile was gone. docker stop followed that path too, as long as I finished within the grace period. That last part still matters, because a slow flush can lose the race against SIGKILL. Have you measured how long your cleanup actually takes, or did you just hope the default ten seconds would be enough?

Limitations, and who should skip this

This pattern is not signal-safe in the POSIX C sense, and it will not save you from SIGKILL. If your cleanup must survive kill -9, you need an external lease, a TTL on the lock, or a supervisor that removes the file. Windows service stop events are not SIGTERM, so this article is the wrong map for a Windows service. I also would not register heavy network calls in atexit, because shutdown is already racing the clock.

Threads and C extensions complicate the picture, since a signal can arrive on a thread you did not expect. If you already use a framework that installs SIGTERM handlers, compose with that framework instead of overwriting it. And if your process is PID 1 in a container without an init, you may need a tiny init anyway, because PID 1 has extra signal quirks that Python will not paper over.

Would I still use atexit? Yes, for the shared cleanup path, once SIGTERM is translated into SystemExit. Would I trust a laptop Ctrl+C as a container stop test? Never again. The next time a lockfile survives a "clean" stop, I will send kill -TERM first, then argue with Docker only if the callback already ran.