I Tailed worker.log for 48 Hours. The File Had Opened Under a Different cwd.

Have you ever watched tail -f sit there, blinking, while ps insists your worker is alive and busy? I did that for two days on a cheap remote box, and I was completely sure logging was broken. The scaffolded worker looked fine in a local terminal, and the free server showed a healthy PID the entire time. So why was the log path I trusted still empty at hour forty-eight?

This is a field notebook, not a victory lap or a platform tour. I will walk through what I tried, what actually broke, and the small checklist I would repeat tomorrow. If you already SSH into a scratch machine and start Python with nohup, you have probably been one reconnect away from the same trap.

Hour 0: A Worker That "Just Logs to a File"

I needed a tiny batch worker that pulled a directory of JSON files, summarized each one, and left an audit trail. Locally, that is a Saturday afternoon job if you already know the shape of the data. I did not want to hand-write boilerplate logging, argument parsing, and a retry loop around file IO.

I used MonkeyCode's free model access on a free server option to scaffold the worker, then I edited the business logic by hand. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The generated snippet was ordinary Python, which is exactly why I trusted it without reading the path twice.

It configured logging.basicConfig with filename="worker.log" and a reasonable format string. I ran it once in my laptop repo, saw lines appear beside the script, and shipped the same file with scp. Does that look wrong to you on first glance, or does it look like every tutorial you have ever skimmed?

# worker.py — the version I shipped first
import json
import logging
from pathlib import Path

logging.basicConfig(
    filename="worker.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def handle(path: Path) -> None:
    payload = json.loads(path.read_text())
    logging.info("handled %s keys=%s", path.name, list(payload))

def main(inbox: Path) -> None:
    for path in sorted(inbox.glob("*.json")):
        handle(path)

if __name__ == "__main__":
    import sys
    main(Path(sys.argv[1] if len(sys.argv) > 1 else "./inbox"))

Hours 1–12: The Ritual That Wastes a Day

On the free server I cloned the layout I already had locally: a ~/jobs/summarizer directory, a virtualenv, and a nohup line I have typed too many times. I started the process from that directory during an SSH session, confirmed the PID, and tailed ~/jobs/summarizer/worker.log like a person who still believed working directories were sticky.

Here is the exact ritual, because the commands were not the bug and I do not want that part to sound mysterious.

cd ~/jobs/summarizer
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
mkdir -p inbox
nohup python worker.py ./inbox > /tmp/summarizer.out 2>&1 &
echo $!
tail -f worker.log

tail -f printed nothing. /tmp/summarizer.out was empty too, which felt like a clue and was actually a red herring. I dropped a few JSON files into inbox, watched CPU twitch for a moment, and assumed the handler was swallowing exceptions. Would you have reached for lsof this early, or would you have "just added more logs" like I did?

I added more logs. Of course I added more logs. That is the move that feels like progress when the file you are watching cannot change.

Hours 12–30: The Fixes That Could Not Possibly Fail

I wrapped handle() in a broad except Exception and logged the traceback at ERROR. I switched from basicConfig to an explicit FileHandler so I could set delay=False and feel professional. I even printed to stderr, which nohup should have captured in /tmp/summarizer.out if the process still had that cwd.

Still nothing I could tail in the repo directory. The inbox files were disappearing, so some code path was running, which made the empty log feel supernatural. Then I did the unhelpful thing that feels productive at 1 a.m.: I asked the same free model to "make logging more reliable" and pasted the new snippet over the old one.

The new snippet still used a relative filename. I had asked the wrong question, so I received a confident answer to a different problem. What broke was not Python's logging module, and it was not the free server "dropping writes." What broke was my belief that the process cwd would remain the directory where I first typed nohup.

Hour 36: lsof Finally Told the Truth

A remote box does not owe you a stable working directory after you detach, reconnect, or restart a worker from a login shell. I finally ran lsof against the PID I had been bragging about in my notes, and the open file list was ruder than any stack trace.

ps aux | grep '[w]orker.py'
# pid 18421
lsof -p 18421 | grep -E 'cwd|txt|worker.log'
readlink -f /proc/18421/cwd
tr '\0' '\n' < /proc/18421/environ | grep -E '^(PWD|HOME)='
ls -l /proc/18421/fd | grep log

The open file was /home/taylor/worker.log, not ~/jobs/summarizer/worker.log. The cwd had quietly become $HOME because I later started a "cleaner" one-liner from the login directory after the first SSH session dropped. Relative filename="worker.log" followed the process, not the repo. I had been tailing a file the process never opened.

Is that embarrassing? Yes. Is it rare? Not on a free server you treat like a scratch laptop. Cron would have been worse, because the default cwd is often /, and then you get a permission error you never see if stderr is also pointed at a relative path.

A tiny reproducer you can run in five minutes

Label this as a local demonstration, not a production benchmark and not a claim about any hosted runtime. Create two directories, start the worker from the wrong one, and watch where the file lands.

# cwd_trap.py
"""Reproducer: relative log paths follow os.getcwd(), not the script path."""
from __future__ import annotations

import argparse
import logging
import os
from pathlib import Path


def setup_broken() -> None:
    logging.basicConfig(
        filename="worker.log",
        level=logging.INFO,
        format="%(asctime)s %(message)s",
        force=True,
    )


def setup_pinned(base: Path) -> Path:
    base.mkdir(parents=True, exist_ok=True)
    log_path = base / "worker.log"
    logging.basicConfig(
        filename=str(log_path),
        level=logging.INFO,
        format="%(asctime)s %(message)s",
        force=True,
    )
    return log_path


def boot_banner(log_path: Path | None = None) -> None:
    print(f"pid={os.getpid()}")
    print(f"cwd={Path.cwd()}")
    print(f"script={Path(__file__).resolve()}")
    if log_path is not None:
        print(f"log={log_path.resolve()}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--pin", action="store_true")
    args = parser.parse_args()

    if args.pin:
        target = Path(__file__).resolve().parent / "logs"
        path = setup_pinned(target)
        boot_banner(path)
        logging.info("pinned hello")
    else:
        setup_broken()
        boot_banner()
        logging.info("relative hello")

Run it like this and then hunt for the file instead of trusting the directory you meant.

mkdir -p /tmp/repo /tmp/elsewhere
cp cwd_trap.py /tmp/repo/
cd /tmp/elsewhere
python3 /tmp/repo/cwd_trap.py
ls -l /tmp/elsewhere/worker.log /tmp/repo/worker.log || true
python3 /tmp/repo/cwd_trap.py --pin
ls -l /tmp/repo/logs/worker.log

The first run creates /tmp/elsewhere/worker.log. The second run creates /tmp/repo/logs/worker.log. That is the entire outage, minus the forty-eight hours of storytelling I wrapped around it. If you want a failing test instead of a visual check, this assertion is enough.

# test_cwd_trap.py — unexecuted until you run pytest in a throwaway dir
from pathlib import Path
import runpy
import sys

def test_pin_writes_beside_the_script(tmp_path, monkeypatch):
    script = tmp_path / "cwd_trap.py"
    script.write_text(Path("cwd_trap.py").read_text())
    elsewhere = tmp_path / "elsewhere"
    elsewhere.mkdir()
    monkeypatch.chdir(elsewhere)
    monkeypatch.setattr(sys, "argv", ["cwd_trap.py", "--pin"])
    runpy.run_path(str(script), run_name="__main__")
    assert (tmp_path / "logs" / "worker.log").is_file()
    assert not (elsewhere / "worker.log").exists()

What I would repeat

I would print a boot banner on every long-lived process, including pid, cwd, script path, and the resolved log path. I would pin logs to Path(__file__).resolve().parent unless a config file supplies an absolute directory. I would treat nohup plus a relative filename as a footgun on any box where I SSH in twice.

Here is the checklist I now keep next to the worker, because memory is not a logging strategy.

1. Print Path.cwd(), Path(__file__).resolve(), and the log path during main(), not after the first failure.
2. Prefer FileHandler with an absolute path, then confirm the service user can write that directory.
3. After start, run readlink -f /proc/$PID/cwd and lsof -p $PID | grep '.log'.
4. Smoke-test by asserting the file exists and grows after one logging.info call.
5. If you daemonize from cron or a login shell, set WorkingDirectory or cd in the unit, then still pin the path.
6. Pin ./inbox, ./outbox, and any state.json the same way, because relative data paths fail more loudly than logs.

Decision table

Situation	Relative worker.log	Pinned to __file__	stdout + journald
Interactive SSH in the repo	Accidentally works	Works	Works
nohup from $HOME after reconnect	Silent miss	Works	Works
cron with default cwd /	File in / or a permission error	Works	Works
systemd without WorkingDirectory	Depends on the unit	Works	Best
Kubernetes / 12-factor	Do not do this	Maybe	Best

The table is the artifact I actually wanted at hour six. A model can scaffold a handler in one pass, but it will not tell you which row you are about to live in after the SSH session drops.

Limitations, and who should skip this

This workflow is for a single-box experiment where you still SSH in and read files by hand. It is not a logging architecture, and it is not a reason to add files on a platform that already captures stdout. If you already run Kubernetes, systemd with StandardOutput=journal, or a hosted worker that only exposes process output, do not introduce a local file just because a scaffold suggested filename="worker.log".

I also would not use a free remote server as the place you first discover cwd behavior if the job can delete data. Relative paths affect more than logs: they affect ./inbox, SQLite files, and any open("state.json") a generated snippet might include. Pin those too, or inject them from the environment with absolute values you print at boot.

MonkeyCode's free model access and free server option were useful for scaffolding and for having a second machine that did not share my laptop's cwd. They did not inspect /proc. The model will happily keep emitting a relative log name unless you ask it to pin paths, and even then you should read the snippet. I am not attaching quotas, hardware claims, or timings, because I did not measure a product benchmark; I measured one mistaken tail.

If you already have a scratch box and a worker that runs but never logs, start with lsof and the reproducer above before you rewrite the handler again.