Who Kills Your Background Worker, and How Soon? A Shutdown Test Plan

Most teams spend their testing budget on the first second of a process and almost none on the last one. That asymmetry is expensive, because a worker that starts cleanly but gets killed mid-write can leave a half-applied batch, duplicated side effects, or a queue item that looks in-flight forever. The plan below is about measuring the shutdown path you already have: which signal arrives, how long the platform waits, and what your process leaves behind when the window closes early.

To be explicit about the evidence: everything here is an illustrative testing procedure, not a report of an experiment. Any number you end up with comes from your own platform and your own workload, so treat the values below as placeholders you must replace.

What actually happens when a platform stops your process

Shutdown usually has two phases: a polite request (SIGTERM), then an unblockable one (SIGKILL) after some interval. The interval is a platform policy, and it is rarely the value people assume it is. The signal(7) man page documents the underlying semantics if you want the kernel-level detail.

• SIGTERM has a default action of terminating the process, so installing a handler turns it into a request you can act on.
• SIGKILL cannot be caught or delayed, so anything not durable before it arrives is simply lost.
• When your process runs as PID 1 inside a container, signals with default dispositions may not terminate it at all, so verify that for your runtime instead of assuming.
• In bash, an exit status of 143 usually means the default action fired (128 + 15), which is a hint that your handler never ran.

Three timestamps are worth capturing on every run: when the handler observed the signal, when the last durable write was acknowledged, and when the process actually exited. With those three numbers you can reason about the budget instead of guessing at it.

The artifact: a small shutdown-budget harness

Prerequisites are modest: Python 3, a POSIX shell, and a worker whose side effects are keyed by an item ID. The code below is illustrative and unexecuted; replace the queue API with your own and keep the shape.

# worker.py — illustrative, not run. Adapt before use.
import json, os, signal, time

BUDGET_S = float(os.environ.get("SHUTDOWN_BUDGET_S", "8"))  # platform policy: measure it
MARGIN_S = float(os.environ.get("SHUTDOWN_MARGIN_S", "2"))  # room for ack + flush

stop_requested = False
t_signal = None

def on_term(signum, frame):
    # Keep the handler boring: set flags only, log in the main loop.
    global stop_requested, t_signal
    stop_requested = True
    t_signal = time.monotonic()

signal.signal(signal.SIGTERM, on_term)

def emit(event, **fields):
    print(json.dumps({"mono": time.monotonic(), "event": event, **fields}), flush=True)

def deadline():
    return t_signal + max(0.0, BUDGET_S - MARGIN_S)

def run(queue, process, ack):
    while not stop_requested:
        item = queue.take()
        ack(item["id"], process(item))  # process() must be idempotent per item id

    emit("signal_seen", budget_s=BUDGET_S)
    while queue.pending() and time.monotonic() < deadline():
        item = queue.take()
        ack(item["id"], process(item))

    leftover = queue.pending()
    emit("exit", leftover=len(leftover), drained=not leftover)

One honest caveat: print() inside a signal handler is not async-signal-safe, which is why the handler above only flips flags. If you need logging from inside the handler, write to a pre-opened pipe or defer the log line to the loop.

Measuring locally before you trust a platform

#!/usr/bin/env bash
# illustrative: SIGTERM a worker, then record exit latency and status
set -u
python3 worker.py > worker.log 2>&1 &
pid=$!
sleep 2

start=$(date +%s%N)
kill -TERM "$pid"
wait "$pid"; code=$?
end=$(date +%s%N)

awk -v s="$start" -v e="$end" -v c="$code" \
  'BEGIN { printf "exit_code=%s elapsed_ms=%.0f\n", c, (e-s)/1000000 }'
tail -n 5 worker.log

Run it twice: once with the handler installed, once with the handler removed as a control. The difference between the two runs is the only proof that your handler is actually wired to the process the supervisor is signalling.

The test matrix you can run in one sitting

Scenario	What to try	What to observe	What to verify
Graceful path	SIGTERM mid-batch	signal_seen, acks after signal	no new batch starts after the signal
Control	SIGTERM with no handler	exit status, last write	default action really fires
Hard stop	SIGKILL mid-batch	durable store + queue state	restart path recovers cleanly
Oversized unit	one item longer than the budget	leftover count	your margin is realistic
Restart loop	start, kill, restart, repeat	duplicate or missing keys	idempotency keys hold

Add one row for the failure you actually fear, because the matrix exists to surface it before production does.

Turning the numbers into a policy

• If elapsed time is far below your assumed budget and the last ack is missing, you are being hard-killed; split work into smaller durable units.
• If elapsed time exceeds the assumed budget and the process still exits cleanly, the platform is more generous than you thought, but do not build on that.
• If leftover is consistently non-zero, shrink the batch or checkpoint mid-batch, and never acknowledge before the write is durable.

Notice that every one of these decisions reduces the blast radius of a kill rather than trying to prevent it, which is the only approach that survives an unblockable signal.

Where free model access and a free server genuinely fit

Two operator-supplied conveniences are relevant here. MonkeyCode offers free model access and a free server option, and both map onto this workflow without pretending to replace your platform's job.

• Use the free model access to enumerate failure modes for the drain loop, and to normalise your JSON log lines into a table you can paste into a review.
• Review anything the model produces with the async-signal-safety caveat above in hand, because generated handlers often log or allocate inside the handler.
• Use the free server option as one observation point for how a stop request reaches a process you do not control; it is a single data point, not a guarantee.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations worth stating plainly: a free tier may sleep, may stop instances on a schedule, and may not let you control how a stop request is delivered. Do not extrapolate one observation into a production grace-period guarantee, and keep your budget honest by treating the platform value as measurable input rather than a constant.

Who should skip this plan

Skip it if your work is already acknowledged by a managed queue with visibility semantics, because the queue owns the redelivery contract and your shutdown path matters far less. Skip it if you cannot shell into the environment to send signals at all. Skip it too if your side effects are not idempotent yet, because measuring a drain loop over non-idempotent work only tells you how fast you produce duplicates.

If you already have a shutdown log from one of these runs, the three timestamps are more useful than another anecdote, so paste them into the comments and say which platform produced them.