48 Hours Chasing a Duplicate Nightly Job: SIGTERM Was Landing on the Wrapper

Two consumers were draining the same queue, and neither of them knew the other existed. My nightly job usually takes eleven minutes, so when the metrics showed the same batch processed twice, I assumed the queue had duplicate messages. It took 48 hours to accept that my deploy script had been killing the wrong process for months.

This is a field note, not a tutorial about signal theory. I want to show you what I tried, what actually broke, and which parts of the workflow I would repeat tomorrow.

The symptom: double processing after every deploy

The duplicate always appeared within fifteen minutes of a deployment, never during a quiet night. That timing was the only clue I trusted, because it pointed at the shutdown path instead of the message broker. When I checked the worker count on the host, pgrep -f worker.py returned two PIDs while the supervisor claimed zero.

So the sequence looked like this:

1. Deploy script reads a pidfile and sends SIGTERM to the recorded PID.
2. The supervisor reports the service as stopped and starts a fresh instance.
3. The old Python process is still alive, reparented, and still holding a queue connection.

Why would a process ignore a term signal that everyone agrees should end it? Because the signal never reached it.

What I tried first, and why it wasted a day

My first theory was a missing signal handler, so I added signal.signal(signal.SIGTERM, handler) and logged every delivery. That changed nothing, which should have been decisive evidence instead of a puzzle. My second theory was a stuck network read, so I added timeouts everywhere and eventually a forced os._exit. Still two processes.

Here is what I kept getting wrong: I was reading the pidfile as ground truth. The wrapper script wrote echo $$ > worker.pid before starting Python, so the recorded PID belonged to the shell, not the consumer. Killing the shell left the child orphaned, reparented to init, and perfectly healthy.

The 40-second inspection that should have been step one

# What is actually running, and who owns the process group?
ps -o pid,ppid,pgid,sid,cmd -C python3
cat worker.pid
ps -o pid,ppid,pgid,sid,cmd -p "$(cat worker.pid)"

If the PID in the file is a sh process and the Python PID has a different PPID, you already have your answer. Process group and session IDs tell you whether kill -TERM -PGID would be a safe broadcast or a small disaster.

The repro I should have written on day one

I finally stopped guessing and built a three-file harness that reproduces the whole failure in about five seconds. The worker records every event as JSON lines, including the PID, PPID, and process group at the moment of each event.

# worker.py — a consumer that tells you whether it ever saw a signal
import json, os, signal, time
from pathlib import Path

LOG = Path(os.environ.get("WORKER_LOG", "worker_events.jsonl"))

def note(event, **fields):
    rec = {"ts": time.time(), "pid": os.getpid(), "ppid": os.getppid(),
           "pgid": os.getpgid(0), "event": event, **fields}
    with LOG.open("a") as fh:
        fh.write(json.dumps(rec) + "\n")

def on_term(signum, frame):
    note("signal", signum=signum)
    raise SystemExit(0)          # cooperative shutdown, main thread only

for sig in (signal.SIGTERM, signal.SIGINT):
    signal.signal(sig, on_term)

note("start")
while True:
    time.sleep(0.5)

#!/bin/sh
# start.sh — deliberately does NOT exec, mimicking the deploy wrapper
cd "$(dirname "$0")"
python3 worker.py

#!/bin/sh
# repro.sh — kill the wrapper and look for survivors
set -eu
rm -f worker_events.jsonl
./start.sh &
wrapper=$!
sleep 1
echo "--- before ---"
ps -o pid,ppid,pgid,sid,cmd -p "$wrapper" --ppid "$wrapper" || true
kill -TERM "$wrapper"
sleep 1
echo "--- after ---"
ps -o pid,ppid,pgid,sid,cmd -C python3 || echo "no python3 survivors"
cat worker_events.jsonl

If worker_events.jsonl has a start record and no signal record, the handler never ran. That single missing line is the difference between a logging problem and a supervision problem.

Decoding who has handlers installed

On Linux, /proc/<pid>/status publishes a signal bitmap, which tells you whether a process is even set up to catch a term signal.

# sigset.py — which signals does this process have handlers for?
import signal, sys
from pathlib import Path

for line in Path(f"/proc/{sys.argv[1]}/status").read_text().splitlines():
    if line.startswith(("SigCgt", "SigBlk", "SigIgn")):
        name, hexbits = line.split(":")
        bits = int(hexbits.strip(), 16)
        names = [s.name for s in signal.Signals if bits >> (s.value - 1) & 1]
        print(name, names)

Run it against the wrapper and against Python, and you will usually see SIGTERM in SigIgn or absent from SigCgt on one of them. Remember the special case for PID 1: the kernel does not apply default terminating actions to PID 1 without an installed handler, which is why containers with a shell entrypoint often wait for SIGKILL instead.

Where MonkeyCode's free options fit in my loop

I ran this experiment roughly forty times, and each run produced ps snapshots, SigCgt bitmaps, and JSONL rows that I had to compare by hand. MonkeyCode offers free model access and a free server option, both of which I used here: the free model access absorbed the repetitive triage of those dumps, and the free server option let me run the harness on Linux, since my laptop is macOS and has no /proc to read. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability and current terms are operator-supplied, so verify them yourself before you plan around them.

The honest part is that the model's explanation was wrong twice. It proposed a race in my signal handler before I had shown it the reparented PPID, and the ps output settled the argument both times. Use the model to summarize evidence you already collected, not to replace the evidence.

The fix, and the test that keeps it fixed

The smallest correct change was to replace the child launch with exec, so the wrapper stops existing and the recorded PID becomes the Python process.

#!/bin/sh
cd "$(dirname "$0")"
exec python3 worker.py

Then I made the shutdown assertable instead of observable:

1. Start the wrapper, capture $!, send SIGTERM, and poll for at most ten seconds.
2. Fail the test if pgrep -f worker.py returns anything after the deadline.
3. Fail the test if worker_events.jsonl contains no signal record, because that means shutdown was never cooperative.
4. Run the loop twenty times in CI to catch intermittent exec behavior across shells.

Signals alone never give you exactly-once, so I added the boring compensating control as well:

# claim_once.py — makes a duplicate consumer harmless
import sqlite3

def claim(conn: sqlite3.Connection, key: str) -> bool:
    with conn:
        cur = conn.execute(
            "INSERT OR IGNORE INTO claims(key, claimed_at) VALUES (?, datetime('now'))", (key,))
        return cur.rowcount == 1   # True means you own this job

Decision table for the wrapper you already shipped

Wrapper style	Who receives SIGTERM	Survivor risk	What I use now
sh start.sh, no exec	the shell that was recorded	high, and intermittent	add exec, or log the child PID
exec python worker.py	the Python process itself	low	my default
setsid plus kill -TERM -PGID	the whole group	low, but broad	only with grandchildren
systemd unit, default settings	the main process only	medium	KillMode=mixed
container with a shell as PID 1	PID 1 rules apply	high	exec form, or an init shim

Limitations, and who should skip this

SIGKILL cannot be caught or forwarded, so every plan here still needs idempotent job handling. /proc is Linux-only, and on macOS you are left with ps -o pid,ppid,pgid,sid,command -ax plus lsof for the port check. Whether a wrapper replaces itself with exec also varies with shell and syntax, which is exactly why the repro beats the documentation.

If your jobs are short-lived batches that always run to completion, this whole investigation is probably wasted effort. If your platform only ever sends SIGKILL, skip the wrapper surgery and spend the time on the claim table instead.

If you want to reproduce this without polluting your main machine, the free server option is what I used to run the harness on Linux, and the free model access is what kept forty rounds of log triage from eating my afternoon.