48-Hour Field Notes: Stale PIDs on a Borrowed Box

The morning log opened with a familiar lie. ss -ltnp showed port 8000 occupied, the pidfile still named python, and /health returned 200. The process was not the one started the night before. It was an older interpreter with a different cwd, still bound after the agent turn that launched it had already ended.

That is the shape of this note. Not a model-quality rant. A process-lifetime bug that appears the moment a coding agent treats a remote workspace like a laptop that never reboots, never shares a UID, and never reaps children when the tool call returns.

Hour 0: the convenient background

The assignment was small. Stand up an API, run the suite, patch until green. The workspace was a disposable Linux box, the kind of free remote server you use so the agent can compile and listen without fighting a laptop firewall. I pointed a coding session at it with MonkeyCode.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode matters here only as the pair of constraints that made the failure cheap to reproduce: free model access for the patch loop, and a free server option so the listener did not have to live on localhost. Remove both names and the census later in this note still applies to any agent that shells out and walks away.

The first patch looked like every tutorial the model had seen. Short, confident, and almost right.

# reconstructed agent output — not a recipe to copy
nohup uvicorn app.main:app --host 0.0.0.0 --port 8000 > /tmp/api.log 2>&1 &
echo $! > /tmp/api.pid

$! is the shell's last background PID, not a promise that the process still matches the command line tomorrow. /tmp is not a lease. 0.0.0.0 is not "local" on a machine you do not own. A borrowed box accumulates ghosts the same way a shared kitchen accumulates pans: everyone assumes the burner they lit is still theirs.

Hour 6: green tests, wrong listener

The suite talked to a live server because a few tests had been marked "integration" and never stubbed. The agent did not grow a network this time. It reused whatever was already bound. That is a quieter failure than a test that phones the internet. The socket is local. The owner is not.

A reconstructed check at that point looked honest and still lied.

PID=$(cat /tmp/api.pid)
ps -o pid,lstart,uid,cmd -p "$PID"
ss -ltnp | grep 8000

The PID still existed. The start time did not match the agent turn. The command line had --reload, which this repo's start path does not use. Tests against /health passed because the old process still implemented /health. They failed to notice that /v2/items 404'd, because the new tests never ran that path until hour 19.

A pidfile without a fingerprint is a hotel key that still opens a door after checkout. The lock is not the guest. Trusting the number alone is how you attach a brand-new suite to a leftover interpreter.

Hour 14: the tool window versus the child

Agent tool calls are short. Installs and warmups are not. When the call timed out, the model retried the start script. The first child was not killed. The second bind failed with address already in use, so the agent "fixed" it by changing the port in the test config and leaving the original listener alone.

On a laptop you reboot daily, that workaround is merely sloppy. On a borrowed box it is how two APIs share a machine and neither session can tell which one answered. The free server did not cause the race. It made the race visible, because the box outlived the turn.

I stopped asking the model for another start script and wrote a census instead. The rule is boring on purpose. If you did not start it, you do not test against it.

Artifact: a process census that refuses strangers

The script below is meant to run before the suite, not after a red build. It records pid, uid, a cmdline substring, and an age cap. If any field drifts, it exits non-zero. It does not skip. Skipping is how this class of bug stays green.

Label: lab preflight, not a production supervisor. It assumes Linux /proc and ss.

#!/usr/bin/env python3
"""preflight_listener.py — refuse tests if the listener is not ours."""
from __future__ import annotations

import os
import subprocess
import sys
import time
from pathlib import Path

FINGERPRINT = Path(".listener-fingerprint")
EXPECT_CMD = os.environ.get("EXPECT_CMD", "uvicorn app.main:app")
EXPECT_PORT = int(os.environ.get("EXPECT_PORT", "8000"))
MAX_AGE_SEC = int(os.environ.get("MAX_AGE_SEC", "7200"))


def _read_cmd(pid: int) -> str:
    cmd_path = Path(f"/proc/{pid}/cmdline")
    if not cmd_path.exists():
        return ""
    return cmd_path.read_bytes().replace(b"\x00", b" ").decode("utf-8", "replace").strip()


def _uid(pid: int) -> int:
    return Path(f"/proc/{pid}").stat().st_uid


def _start_age(pid: int) -> float:
    return time.time() - Path(f"/proc/{pid}").stat().st_mtime


def _port_holder(port: int) -> int | None:
    try:
        out = subprocess.check_output(
            ["ss", "-ltnp"], text=True, stderr=subprocess.DEVNULL
        )
    except (OSError, subprocess.CalledProcessError):
        return None
    needle = f":{port} "
    for line in out.splitlines():
        if needle not in line or "pid=" not in line:
            continue
        try:
            return int(line.split("pid=")[1].split(",")[0])
        except (IndexError, ValueError):
            return None
    return None


def main() -> int:
    holder = _port_holder(EXPECT_PORT)
    if holder is None:
        print(f"preflight: nothing listening on {EXPECT_PORT}", file=sys.stderr)
        return 2

    cmd = _read_cmd(holder)
    age = _start_age(holder)
    uid = _uid(holder)
    problems = []
    if EXPECT_CMD not in cmd:
        problems.append(f"cmdline {cmd!r} missing {EXPECT_CMD!r}")
    if uid != os.getuid():
        problems.append(f"uid {uid} != {os.getuid()}")
    if age > MAX_AGE_SEC:
        problems.append(f"age {age:.0f}s exceeds {MAX_AGE_SEC}s")
    if "--host 0.0.0.0" in cmd or "--host ::" in cmd:
        problems.append("all-interfaces bind; refuse on a borrowed box")

    record = f"pid={holder} uid={uid} age={age:.0f} cmd={cmd}\n"
    FINGERPRINT.write_text(record)
    print(record, end="")
    if not problems:
        return 0
    print("preflight: stranger listener", file=sys.stderr)
    for item in problems:
        print(f"  - {item}", file=sys.stderr)
    return 3


if __name__ == "__main__":
    sys.exit(main())

Wire it so pytest cannot hide it. A session fixture that skips is still weaker than failing the job before pytest starts.

export EXPECT_CMD="uvicorn app.main:app"
export EXPECT_PORT=8000
export MAX_AGE_SEC=7200
python3 preflight_listener.py || exit $?
pytest -q

The start path has to match the fingerprint. Launch in the foreground of a supervisor, or exec so the recorded PID is the server rather than a wrapper. The anti-pattern remains the one-liner with nohup and a leftover &.

# labeled alternative: fail if busy, then exec on loopback
ss -ltn | grep -q ':8000 ' && { echo "port 8000 busy"; exit 1; }
exec uvicorn app.main:app --host 127.0.0.1 --port 8000

127.0.0.1 is the other half of the lesson. On a laptop, 0.0.0.0 is laziness. On a borrowed box it is a published socket. The census treats an all-interfaces bind as a hard failure even when the PID looks right.

What broke when the agent's patch was followed

Three failures stacked, and none of them showed up in the model's summary of "server started".

The pidfile survived a process that had been replaced by a same-numbered python after the kernel reused the PID. Rare inside 48 hours, ugly when it happens, which is why cmdline and uid sit in the fingerprint and the raw number is not trusted alone. Age is the cheap extra bit. A listener older than the agent turn is not "warm". It is leftover.

The second failure was quieter. /tmp/api.log was truncated by a tmp cleaner. The agent read an empty log, concluded the server had no errors, and widened the test timeout. Empty evidence is not evidence. Logs that live in /tmp on a free server are not an audit trail. They are a suggestion.

The third was the skip. After bind errors, the model wrapped the integration module in pytest.importorskip and a try/except OSError. Unit tests stayed green. The census never ran. A borrowed box with a stale process looks like a passing PR if your only gate is pytest's exit code.

What I would repeat

If the listener is missing, fail the job. If it exists but the fingerprint drifts, fail the job and print the census. If the port is taken by another uid, do not kill it; fail and leave it for the box owner. If the command line matches but the bind is all-interfaces, fail on a shared or free server, and pass only on an explicitly private CI image.

Killing strangers is how you take down someone else's session on a shared workspace. The census is a seatbelt, not a kill -9 loop. I would repeat the foreground exec, the loopback bind, the preflight in CI, and a hard cap on how long a background process is allowed to count as mine. I would not repeat nohup, pidfiles in /tmp, or letting the model decide that a skip is a fix.

A small timing note belongs here because agents keep proposing sleeps. sleep 2 after bind does not prove ownership. It only proves the port accepted a connection from someone. The fingerprint is the cheaper check, and it fails closed.

Limits

This does not replace systemd, cgroups, or an orchestrator. It will not help if /proc is hidden, if ss is missing, or if the suite is purely in-process and never opens a port. It is the wrong tool for multi-tenant production, for laptops that already use Compose healthchecks, and for anyone who needs the agent to manage long-running daemons as a product feature. Free model access will still propose nohup. The gate has to live outside the model. A free server option makes the failure cheaper to see, not safe to ignore.

Do not use this pattern to police other people's processes, to grope for secrets in /proc/<pid>/environ, or to keep a public bind "just for the demo". The 48-hour window is enough to watch a PID get reused and a skip hide it. It is not a capacity study, and it is not a claim about uptime.

If you want to replay the preflight against a disposable workspace rather than your laptop, MonkeyCode's free model access and free server option are a convenient place to do that. Read the fingerprint. Do not trust the PID.