48-Hour Field Notes: `kill -0` Said My Batch Job Was Alive. The PID Belonged to Someone Else.

How do you know a background job is still running when you are not looking at it? For two days I thought I knew, and for two days I was wrong in a way that cost me about ninety minutes of silent nothing.

I had moved a small batch job off my laptop — a few hundred prompts, each expected to come back as JSON matching a schema — and onto a free server instance so that closing the lid would stop being a scheduled outage. I used MonkeyCode's free model access for the endpoint and its free server option for the host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What follows is the harness I built, the check that lied to me, and the version I would keep.

The design constraint I started from

I did not want a sophisticated deployment. I wanted one property: if the process disappears, I should be able to start it again and not pay twice for work already finished. Everything else — throughput, parallelism, pretty logs — was negotiable.

So I treated the free server as disposable. Anything that mattered had to land in a file I could copy off the box in one command, and the runner had to be resumable from that file alone. No hidden state in memory, no "just don't restart it" assumption.

Three pieces I built before touching the model

1. A content-addressed key per item. The key is a hash of the item payload, not a line number, so reordering the input file cannot cause a double charge or a skipped item.
2. An append-only ledger with fsync. One JSON object per line, written after each item finishes. A torn final line is expected and tolerated on read.
3. A heartbeat that answers "is this the same process", not "is something running". This is the piece I got wrong first.

# runner.py — standard library only, no framework
import json, os, hashlib, time, pathlib

STATE = pathlib.Path("state")
LEDGER = STATE / "ledger.jsonl"
HEARTBEAT = STATE / "heartbeat.json"

def item_key(item: dict) -> str:
    blob = json.dumps(item, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def done_keys() -> set[str]:
    if not LEDGER.exists():
        return set()
    keys = set()
    for line in LEDGER.read_text(encoding="utf-8").splitlines():
        try:
            row = json.loads(line)
        except json.JSONDecodeError:
            continue  # torn tail from an interrupted write
        if row.get("status") == "ok":
            keys.add(row["key"])
    return keys

def append(row: dict) -> None:
    STATE.mkdir(exist_ok=True)
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")
        fh.flush()
        os.fsync(fh.fileno())

How I checked liveness, and why it was the wrong question

My first heartbeat stored a PID and a timestamp. My check was one line in a shell:

kill -0 "$(jq -r .pid state/heartbeat.json)" && echo alive || echo dead

kill -0 exits 0 when a process with that PID exists and I am allowed to signal it. That is all it means. It does not mean the process is mine, and it certainly does not mean it is making progress. PIDs are recycled; on a small box, every other process that starts and exits feeds the same counter.

On day one I got exit 0 back, went to dinner, and returned to a ledger whose newest row was ninety minutes old. The job had died on an unsent request. Something unrelated had inherited its number. My monitor had been reporting on a stranger for an hour and a half.

Isn't that the worst kind of monitoring failure — the one that keeps saying everything is fine?

The fix: pin identity, then judge freshness

A PID is a hint. Boot ID plus process start time is an identity. Linux exposes both cheaply, and neither requires a daemon or a dependency.

import pathlib, os, time

def read_boot_id() -> str:
    return pathlib.Path("/proc/sys/kernel/random/boot_id").read_text().strip()

def read_starttime(pid: int) -> str | None:
    try:
        stat = pathlib.Path(f"/proc/{pid}/stat").read_text()
    except FileNotFoundError:
        return None
    # Field 22 of /proc/<pid>/stat is starttime. Split on the LAST ')' because
    # the comm field (field 2) may itself contain spaces and parentheses.
    return stat.rsplit(")", 1)[1].split()[19]

def write_heartbeat() -> None:
    STATE.mkdir(exist_ok=True)
    pid = os.getpid()
    HEARTBEAT.write_text(json.dumps({
        "pid": pid,
        "starttime": read_starttime(pid),
        "boot_id": read_boot_id(),
        "written_at": time.time(),
        "host": os.uname().nodename,
    }, indent=2))

def is_alive(hb: dict, max_stale_s: int = 180) -> tuple[bool, str]:
    if not pathlib.Path("/proc", str(hb["pid"])).exists():
        return False, "process gone"
    if read_boot_id() != hb["boot_id"]:
        return False, "host rebooted"
    starttime = read_starttime(hb["pid"])
    if starttime is None:
        return False, "process gone"
    if starttime != hb["starttime"]:
        return False, "pid recycled"
    age = time.time() - hb["written_at"]
    if age > max_stale_s:
        return False, f"heartbeat stale ({age:.0f}s)"
    return True, "alive"

The runner calls write_heartbeat() between items, not on a timer thread. That way the heartbeat also measures progress: if one request hangs, the file goes stale and I learn about it, instead of watching a thread happily rewrite a timestamp while the work sits still.

The decision table I actually use now

A boolean is not enough to act on. Two signals give me six states, and each one has an obvious next move.

Heartbeat verdict	Newest ledger row	Action
alive	under 5 minutes old	leave it alone and stop refreshing the terminal
alive	older than 15 minutes	inspect the in-flight item, add a per-item timeout
dead — process gone	any	resume; the ledger makes re-entry safe
dead — pid recycled	any	resume, and rewrite the heartbeat file first
dead — heartbeat stale, identity matches	any	treat as hung, save the log, then restart
dead — host rebooted	any	copy the ledger off the box before anything else

The model side needed the same suspicion

Free model access is genuinely useful for a batch job like this, but a schema is a promise made by me, not by the endpoint. So every response gets recorded with the item key, a hash of the request, whatever model identifier the response carries, and the raw text. If validation fails, the row is written with status: "invalid" and the item stays eligible for one retry with jitter.

That record paid for itself on day two. Two items came back with a field wrapped in a string instead of an object. Because the raw text was on disk, I could tell that my prompt was ambiguous rather than the output being random, and I fixed the instruction instead of adding a retry loop that would have hidden the problem forever.

What I would repeat, and what I would delete

• Repeat: the content-addressed key. It survived an input file reorder without a single duplicate call.
• Repeat: fsync on every ledger line. Slow, boring, and the reason a mid-run kill cost me nothing.
• Repeat: heartbeat-on-progress rather than heartbeat-on-timer.
• Delete: the PID-only check. It answered a question I never needed to ask.
• Delete: my instinct to add retries before reading the raw responses.

Limitations, and who should not do this

This is single-host tooling. There is no distributed lock, so two runners pointed at the same state directory will happily duplicate work; use a lock file or, better, a real queue if you need concurrency. The rsplit(")", 1) parse assumes the process name does not end with a closing parenthesis, which is rare but not impossible. And inside containers, PID namespaces mean a heartbeat written by one namespace can look wrong — or worse, plausible — from another, so verify identity from the same namespace that runs the job.

If you already have systemd, Supervisor, or a container orchestrator with health checks, you should use those instead of hand-rolling any of this. The harness above is for the case I was in: one small box, one batch job, no cluster, and a strong preference for not babysitting it.

If you want to try the same loop, the free server option is enough to keep the runner going while your laptop sleeps — just make sure the ledger can leave the box before you trust it with anything.