The Stop Flag Was Real. The Worker Had Never Shared the Agent's cwd

The worker would not stop, and that single refusal swallowed a weekend I had reserved for quieter work. I had a tiny agent-style loop that promised to exit when it noticed a stop flag on disk. Have you ever trusted a log line more than the process that printed it, even after it lied twice? I did that for almost two days before I asked where the process was actually standing.

This is a 48-hour field note, reconstructed as a lab you can break on purpose. I am not claiming production metrics, customer names, or a ranked model bake-off. I am claiming a cwd mismatch that made a perfectly real file invisible to a perfectly healthy loop.

Hour 0–8: A stop file that looked like adulthood

I wanted a cheap loop I could halt without a second SSH session and without inventing a control plane. A flag file felt boring, auditable, and honest enough for a weekend experiment. I drafted the first loop with MonkeyCode's free model access, then ran the same script on the free server option so I could watch a real process instead of a notebook cell. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The sketch looked like every tutorial I have skimmed too quickly. Poll the disk, print a heartbeat, leave when the flag appears, and pretend relative paths are a shared language. Why would a six-line loop ignore a file that ls could already see?

# reconstructed lab example, not the original script
from pathlib import Path
import time

while True:
    if Path("stop.flag").exists():
        print("seen stop.flag; exiting")
        break
    print("heartbeat")
    time.sleep(2)

I started it under nohup, wrote stop.flag from the directory I believed was the repo root, and waited for silence. The heartbeats kept arriving, cheerful and evenly spaced, which is the worst kind of failure. A crash would have been kinder, because a crash points at a stack. A healthy loop that refuses your stop file points at your story about the machine.

What I tried while the heartbeat kept printing

I did the respectable debugging first, because respectable debugging usually works and therefore wastes the most hours. None of these checks were foolish. They were simply aimed at the wrong process story.

1. I sent SIGTERM once, then again, then felt guilty about SIGKILL.
2. I ran ls -l stop.flag in the directory I had cloned, and the file was really there.
3. I checked mode bits, ownership, and whether the process user could read the directory.
4. I added flush=True because I have been burned by buffered stdout on remote hosts.
5. I grepped the script for a second loop, a thread, and a child I had forgotten to reap.

Would you have started with /proc either? I would like to say yes, but I started with feelings about the code. The code was doing exactly what it was told. It was looking beside its own feet, and I was dropping the flag beside mine.

Hour 8–24: Signals, permissions, and other adult dead ends

By the afternoon I had a theory that the process was ignoring signals because it spent too long inside sleep. That theory is almost true in other bugs, which is why it survived longer than it deserved. I wrapped the sleep, caught SIGTERM, and logged the signal number with a timestamp. The process still lived, because I had not actually asked it to die. I had asked a file to appear in a directory the worker did not occupy.

Permissions were the next respectable villain. I ran namei -l on the flag and convinced myself that a sticky bit or a shifted uid was hiding in the path. The listing was boring in the way correct listings are boring. Same user, same group, world-readable, written two minutes ago. If the worker and I had shared a working directory, the loop would have exited before I finished the coffee that I used as a timer.

I also blamed the agent side of the workflow, because blaming the writer is easier than blaming the reader. The writer script used Path(__file__).resolve().parent and therefore dropped stop.flag next to the generator, not next to the running worker. Two Python files, two honest path helpers, zero shared cwd. If two processes do not share a working directory, what does the sentence "write stop.flag" even mean?

Hour 24–40: The log was honest, and that made it worse

The heartbeat line said alive, and I treated that word like a location. It was only a mood. I had not printed os.getcwd(), Path(__file__), or the absolute path of the flag the loop intended to honor. Without those three lines, a remote log is a novel with the place names ripped out.

I added timestamps and thought I was doing science. I added a counter and thought I was doing reliability. I even added a JSON line so I could pretend this was observability. None of that answers the only question a relative path can ask: relative to what, right now, in this process, after whoever launched it changed directories?

nohup did not help me think. A systemd unit would not have helped either if WorkingDirectory= pointed at a folder I never opened. Launchers are cwd machines. I keep forgetting that, then relearning it in public.

Hour 40–48: /proc told me the working directory

The useful hour was the one where I stopped reading my script and started reading the kernel's view of the process. Linux will tell you the cwd if you ask without poetry.

# replace PID with the worker you actually launched
PID=$(pgrep -f 'worker.py' | head -n 1)
echo "pid=${PID}"
readlink "/proc/${PID}/cwd"
pwdx "${PID}"
tr '\0' '\n' < "/proc/${PID}/cmdline"
ls -l "$(readlink /proc/${PID}/cwd)"

The cwd was not the repo root I had been stroking with ls. It was a work directory I had created so the agent could scribble drafts without dirtying git. The worker had been started from that work directory on purpose, hours earlier, by a command I had already scrolled away. The flag file was real. It was simply next door.

I wrote the flag into the cwd /proc had named, and the loop exited on the next poll. No mystery thread. No ignored signal. No broken exists(). Just two directories that looked identical in my head and different on disk. Have you noticed how often "the server is hung" is cover language for "I do not know this process's cwd"?

A reconstructed worker you can actually break

Label this as a lab reproduction. It is not a claim about a hosted image, a quota, or a named model. Copy both files, then start the worker from a different directory than the writer.

#!/usr/bin/env python3
"""lab_worker.py — reconstructed example, unexecuted until you run it."""
from __future__ import annotations

import os
import sys
import time
from pathlib import Path

FLAG_NAME = "stop.flag"
POLL_SECONDS = 2.0


def main() -> None:
    cwd = Path.cwd().resolve()
    script_dir = Path(__file__).resolve().parent
    print(f"pid={os.getpid()}", flush=True)
    print(f"cwd={cwd}", flush=True)
    print(f"script_dir={script_dir}", flush=True)
    print(f"looking_for={cwd / FLAG_NAME}", flush=True)

    while True:
        flag = cwd / FLAG_NAME
        if flag.exists():
            print(f"seen {flag}; exiting", flush=True)
            return
        print(f"heartbeat looking_for={flag}", flush=True)
        time.sleep(POLL_SECONDS)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""lab_writer.py — reconstructed example. Writes beside __file__, not cwd."""
from pathlib import Path

flag = Path(__file__).resolve().parent / "stop.flag"
flag.write_text("stop\n", encoding="utf-8")
print(f"wrote {flag}")

mkdir -p /tmp/flag-lab/work
cp lab_worker.py lab_writer.py /tmp/flag-lab/
cd /tmp/flag-lab/work
python3 /tmp/flag-lab/lab_worker.py
# other terminal
python3 /tmp/flag-lab/lab_writer.py
ls -l /tmp/flag-lab/stop.flag /tmp/flag-lab/work/stop.flag

The writer succeeds. The worker continues. That is the whole incident, stripped of my weekend narration. If you want the loop to stop, either start both tools from the same directory or stop using relative names as if they were UUIDs.

The inspection workflow I will keep

I am keeping a short checklist because I will forget this again the next time a log sounds confident. Print identity first. Then ask the kernel. Then decide whether the flag should be relative to cwd, relative to the script, or absolute because you are tired.

1. Print pid, cwd, Path(__file__).resolve(), and the absolute flag path at boot, with flush=True.
2. Confirm the live cwd with readlink /proc/$PID/cwd or pwdx $PID, not with memory.
3. Confirm the live command line with /proc/$PID/cmdline so you know which copy started.
4. Write a probe file into that cwd (echo probe > "$(readlink /proc/$PID/cwd)/probe.txt") before you trust ls in your shell.
5. Only after those four match should you reach for signals, permissions, or a rewrite of the loop.

# boot banner I now want in every long-running script
from pathlib import Path
import os
print(
    {
        "pid": os.getpid(),
        "cwd": str(Path.cwd().resolve()),
        "script": str(Path(__file__).resolve()),
        "flag": str((Path.cwd() / "stop.flag").resolve()),
    },
    flush=True,
)

If the banner and /proc disagree, believe /proc. If they agree and the flag still "missing," you are finally allowed to talk about permissions. Relative paths are not a protocol. They are a rumor two processes tell about a directory.

Decision table: where the flag goes

Writer does	Worker looks	Typical landing zone	Stops the loop?
Path("stop.flag").write_text(...) from shell cwd A	Path("stop.flag") from cwd B	directory A	No
Path(__file__).parent / "stop.flag"	Path("stop.flag") from a work dir	beside the writer script	No
Path(__file__).parent / "stop.flag"	same __file__ parent	beside both scripts	Yes, if they are the same file
Absolute path both sides, e.g. /tmp/flag-lab/work/stop.flag	that same absolute path	exactly one file	Yes
Flag next to script, worker uses Path(__file__).parent	script directory	stable across launch cwd	Yes

I now prefer the last row for solo workers and the absolute-path row for anything started by a launcher. Shared relative names are fine in a demo. They are a footgun the moment an agent, a Makefile, and a human shell all believe they are "in the project."

What I would repeat next time

I would still use a flag file for a single-host loop that I intend to kill with touch. I would still draft the boring version first and run it on a throwaway server before I wrap it in hopes. I would not treat the model's first relative path as a contract, and I would not treat my shell cwd as the process cwd.

I would repeat the boot banner, the /proc check, and the probe file in the live cwd. I would repeat using flush=True on those lines, because a banner that arrives after the mystery is just a memoir. I would not repeat blaming sleep, signals, or permissions before I can point at one resolved path both tools name out loud.

Would I trust a relative stop flag after this? Not without printing both directories first, and not when an agent writes files beside __file__ while a worker polls Path.cwd(). That sentence is the whole lesson, and it does not need a product chorus around it.

Limitations, and who should skip this

This approach is for one machine, one user, and one process you can inspect with /proc. It is not a cluster lock, not a job queue, and not a substitute for a real supervisor when you need restart budgets and health checks. Flag files also fail in the usual disk ways: leftover flags from a previous crash, NFS rename oddities, and containers whose cwd is an anonymous workdir you never mounted.

Skip this if you already have systemd WorkingDirectory=, a container entrypoint that cds for you, or multiple replicas that must stop together. Skip it if your "agent" and your worker do not share a filesystem. Skip it if you cannot print cwd in production logs. In those cases a relative stop.flag is not a control signal. It is a coincidence waiting for a weekend.

The 48 hours were not about a hung free server, a closed stdin, or a health probe lying about a port. The process was alive, the file was real, and my map of the directories was fiction. If you are already using free model access to draft a loop like this, add the cwd banner before you add another retry. That is the only change I still trust.