The YAML on Disk Was Fresh. The Worker Still Held an Unlinked Inode.

Have you ever replaced a config file, grepped the new timeout, and still watched the old number come back from a live process? I spent a long weekend doing exactly that, and the directory listing kept lying with a straight face. This is a set of 48-hour field notes, not a postmortem with fake dashboards. I am writing down what I tried, what actually broke, and the checks I would run again before I trust an agent that only edited the working tree.

The setup looked boring on purpose. A long-running Python worker read config.yaml once during startup, then served a health endpoint that echoed the loaded timeout. An assistant on a spare box kept rewriting that YAML because the timeout in the response never moved. Why would it move, if nobody reopened the file descriptor?

What I thought was happening

I assumed three comfortable stories, and each one wasted a few hours. Does that sound familiar if you have ever debugged with an agent that only sees the filesystem?

• The worker must be reading a copied tree, so the file I edited was a decoy.
• The YAML parser must be caching the mapping object somewhere clever inside PyYAML.
• The process must have crashed and been replaced by a parent that still held old flags.

None of those stories survived /proc. The worker was alive, the path was correct, and the parser was not the villain. The process had opened an inode on Monday, and every later write created a different inode with the same name.

Hour 0–6: I kept proving the file was new

I started with the rituals that make you feel busy. I printed timestamps, hashed the file, and asked the agent to rewrite the document from scratch. The bytes on disk were honest. The HTTP body was not.

stat -c '%n inode=%i mtime=%y size=%s' config.yaml
sha256sum config.yaml
curl -s localhost:8080/health

The stat line kept changing. The curl body kept saying timeout_s: 30 after I had typed timeout_s: 5 into the file. How long would you keep diffing a file that already matches the patch? I kept going because the assistant kept reporting success. It had written the path I named. It had not asked whether any process still held the previous inode.

I also let a coding assistant take a pass on the same box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access on the free server option to draft the replacement script and to narrate candidate causes. That pairing is useful when you want a second pair of eyes without standing up a paid cluster, but it still only sees files, not open descriptors.

Hour 6–18: the agent assumed write equals apply

The assistant did the thing agents love. It wrote to a temporary file, then renamed the temporary file over config.yaml, because atomic replace is the textbook way to avoid a half-written document. That textbook is correct for the next reader. It is wrong for a process that already called open() and cached the mapping in memory.

Here is the pattern it emitted, which I am labeling as a reconstructed example rather than a production dump:

# reconstructed example: atomic replace looks safe, then surprises a live worker
from pathlib import Path
import os
import tempfile

def atomic_write(path: Path, body: str) -> None:
    path = path.resolve()
    fd, tmp_name = tempfile.mkstemp(dir=path.parent, prefix='.cfg-', suffix='.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as handle:
            handle.write(body)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp_name, path)
    except Exception:
        try:
            os.unlink(tmp_name)
        except OSError:
            pass
        raise

After that function returns, ls is thrilled. lsof is not. Have you checked whether your worker still points at a (deleted) path?

PID=$(pgrep -f 'python -m worker')
ls -l /proc/$PID/fd | grep config.yaml
# often looks like: 3 -> /home/dev/app/config.yaml (deleted)

That (deleted) suffix is the whole story. The directory entry now names a new inode. The running process still holds the old one, and Linux will keep that data alive until the last file descriptor closes. Restarting is the apply step. Rewriting is only a promise to the next open().

A tiny worker you can actually run

I wanted a reproduction that does not depend on a private service. The snippet below is a complete toy, not a claim about anyone's production traffic. Run it in one terminal, then replace the YAML from another terminal without killing the process.

# worker.py — toy process that opens config once
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
import json
import os
import yaml

CONFIG_PATH = Path('config.yaml').resolve()

with CONFIG_PATH.open('r', encoding='utf-8') as handle:
    CONFIG = yaml.safe_load(handle)
    INODE = os.fstat(handle.fileno()).st_ino

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        payload = {
            'timeout_s': CONFIG.get('timeout_s'),
            'inode_at_boot': INODE,
            'pid': os.getpid(),
        }
        body = json.dumps(payload).encode('utf-8')
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        return

if __name__ == '__main__':
    HTTPServer(('127.0.0.1', 8080), Health).serve_forever()

Seed the file, start the worker, then perform the same atomic replace the agent loved.

printf 'timeout_s: 30\n' > config.yaml
python -m pip install pyyaml
python worker.py &
curl -s localhost:8080/health
python - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('config.yaml').resolve()
fd, tmp = tempfile.mkstemp(dir=path.parent)
with os.fdopen(fd, 'w', encoding='utf-8') as h:
    h.write('timeout_s: 5\n')
    h.flush(); os.fsync(h.fileno())
os.replace(tmp, path)
PY
stat -c 'disk inode=%i' config.yaml
curl -s localhost:8080/health
ls -l /proc/$(pgrep -f 'python worker.py')/fd | grep yaml

The health payload should still show 30 and the boot inode. The stat line should show a different inode. That mismatch is the bug, and it is also the lesson.

Decision table I wish I had printed first

I now keep this table next to the terminal when an agent claims a config change is live. It is boring on purpose.

What you did to the path	Same inode?	Open fd sees	Process that cached YAML
Truncate and rewrite in place (open(path, 'w'))	Yes	New bytes if it seeks and rereads	Still old until it reloads
Write temp file, then os.replace / mv	No	Old bytes on the unlinked inode	Still old until restart or reopen
chmod / chown only	Yes	Same bytes	Same cache
Edit via a symlink that now points elsewhere	New target inode	Old target if fd already open	Still old
Kill and restart the worker after any of the above	Fresh open()	Whatever the directory names now	New mapping

Would you still blame the parser after reading that last column? I would not, but I did, because the file content matched the patch and the process was still answering HTTP.

Hour 18–36: what actually broke

Three separate mistakes stacked, and only one of them was Python.

1. The worker treated config as immutable after boot, which is a reasonable choice until somebody expects hot reload.
2. The assistant used atomic rename, which is the right durability trick and the wrong apply trick.
3. I verified the directory entry and never verified the descriptor.

The combination is mean because every local check you run from a shell opens the file again. Your shell is a new reader. The worker is an old reader. Asking an agent to cat the file just creates more new readers that agree with each other and disagree with the process that matters.

I also tried sending SIGHUP because some daemons reload on hangup. This toy worker did not register a handler, so the signal either did nothing useful or died depending on the shell. That is another assumption agents make: Unix services share a secret reload protocol. They do not.

Hour 36–48: checks I will repeat

If I am going to let a free remote assistant touch a long-lived process again, I want a short, mean checklist that does not care about the brand of the model. The commands are the product.

# 1. Identity of the live process, not the file name you like
pgrep -a -f 'python worker.py'

# 2. Descriptor vs directory entry
PID=$(pgrep -f 'python worker.py')
readlink /proc/$PID/fd/* 2>/dev/null | grep yaml || true
stat -c 'dir inode=%i' config.yaml
python - <<PY
import os, yaml
from pathlib import Path
with Path('config.yaml').open() as h:
    print('shell inode', os.fstat(h.fileno()).st_ino)
    print('shell yaml', yaml.safe_load(h))
PY

# 3. Apply step that is actually an apply step
kill $PID
python worker.py &
curl -s localhost:8080/health

I will also make the worker print the inode it captured at boot, the way the toy handler does. That one integer turns a religious argument into a comparison. If the health payload and stat disagree, you do not have a YAML bug. You have a lifetime bug.

When I used the free server option, I asked the assistant to generate the readlink loop instead of another rewrite. That was the first useful turn. The model can draft the replace helper in seconds, but it cannot feel the difference between a path and a descriptor unless you put /proc in the prompt.

What I would repeat, and what I would not

I would repeat the reproduction, the inode print, and the rule that atomic rename is for the next open(). I would repeat asking any assistant, including one on a free server, for inspection commands before patch commands. I would not repeat trusting a successful write as evidence of a live config. I would not repeat sending HUP at a process I had not instrumented. I would not repeat hashing the file as a proxy for hashing the process memory.

A small reload path is worth the lines if you truly want agents to tune timeouts without a restart. The sketch below is a proposal, not something I rolled out everywhere.

# proposal: explicit reopen, still not magic
def load_config(path):
    with path.open('r', encoding='utf-8') as handle:
        data = yaml.safe_load(handle)
        inode = os.fstat(handle.fileno()).st_ino
    return data, inode

# later, on a deliberate signal or admin endpoint
CONFIG, INODE = load_config(CONFIG_PATH)

Even then, you must decide what happens to work already in flight. Reloading a timeout does not rewind a request that already started sleeping.

Limitations, and who should skip this

This workflow assumes a Unix-like /proc, a worker that holds a file descriptor or an in-memory snapshot, and a writer that replaces files by rename. It does not translate cleanly to Windows, where the sharing rules and the idea of an unlinked inode are different. It does not help if your process re-reads the path on every request. It does not help if you are chasing a wrong host, a wrong container layer, or a config client that talks to a remote store instead of a YAML file.

Do not use this as a reason to disable atomic writes. Truncating in place can give you a torn file when a crash lands mid-save, and that failure mode is worse than a delayed apply. Do not use a free remote coding box as a substitute for knowing how your supervisor restarts the worker. A model that can edit files will happily keep editing them while the old inode answers traffic.

Skip this approach if you already have a documented reload endpoint, a Kubernetes rolling restart, or a config watcher you have actually tested. Skip it if you cannot inspect /proc because the process is on a sealed appliance. Skip it if the file is a secret mounted as a projected volume that rotates by replacing the directory; that is a cousin of this bug, but the checks live in the mount namespace, not in tempfile.mkstemp.

The 48-hour version of me wanted a smarter parser. The version of me that would repeat the work wanted readlink on the descriptor, an inode in the health payload, and a restart that I could point at. The file on disk can be perfect and still be the past.