I Debugged the Sample Defaults for 48 Hours. cwd Never Entered the Repo.

I spent forty-eight hours tuning retry counts on a parser that kept returning the sample defaults. The stack traces looked clean, the file name was right, and every local pytest run stayed green. So why did the remote job keep loading a config I did not remember writing? Relative paths were the entire bug, and the interactive agent shell had been hiding them.

These field notes reconstruct that debugging workflow as a labeled example you can rerun. Treat the helper and tests as a recipe, not as production telemetry from a fleet. If your process already receives an absolute config path, you can skip most of this and still steal the three-line startup log.

Hour 0: the job looked too simple to fail

The pipeline was boring on purpose: load YAML, walk a data directory, and emit one summary JSON document. I let a coding assistant fill the path handling because the happy path looked like every tutorial I still have bookmarked. It wrote cfg = Path("config.yaml") and data = Path("data"), matching my pytest fixtures almost exactly. Those fixtures called monkeypatch.chdir(repo), so the tests never asked where the process actually lived.

Was that lazy, or was it just the usual confidence that a green suite gives you? I still think green tests are necessary, but they are not evidence about the working directory. The suite never started the module the way a batch job starts it, from some other folder.

What I tried for the first eight hours

I did what I always do when defaults leak through: I assumed the loader was too forgiving. I pinned PyYAML, printed the parsed dictionary, and compared keys until the scrollback looked ridiculous. Nothing important was missing from the in-memory dict, which made the bug feel like a logic error. The remote log showed sample_batch: true, a demo flag I had left in the fixture file.

How did a demo flag survive a job that was supposed to read the production-shaped config instead? I kept rereading the YAML, as if another hyphen would explain a boolean I had typed myself. The file on disk in the repo was correct; the process simply never opened that file. Here is the loader I started with, reconstructed as a small example you can paste.

# job.py — labeled example, not a shipped service
from pathlib import Path
import yaml


def load_config():
    path = Path("config.yaml")  # this follows cwd, not the script
    with path.open(encoding="utf-8") as handle:
        return yaml.safe_load(handle)


def main():
    config = load_config()
    print("loaded", config)
    data_dir = Path(config.get("data_dir", "data"))
    records = list(data_dir.glob("*.json"))
    print("records", len(records))


if __name__ == "__main__":
    main()

That Path("config.yaml") call is not a filename in the way people casually describe it. It is a promise that os.getcwd() already points at the repository root you care about. Break that promise, and the rest of the program will look disciplined while it reads nonsense.

I also tried a pile of nearby hypotheses that wasted real clock time without moving cwd:

1. I reinstalled PyYAML and pinned the loader, because defaults felt like a library quirk rather than a path quirk.
2. I dumped json.dumps(config, indent=2) on startup, which only proved I was parsing the wrong document cleanly.
3. I added retry counts around the parser, as if another attempt would find keys the first parse had missed.
4. I asked the assistant to make startup more robust, which is how a stub file landed outside the repo.

What actually broke

The remote run eventually raised FileNotFoundError for config.yaml, which honestly felt like real progress. I pasted the traceback into the assistant and asked it to make startup more robust. It created config.yaml in whatever directory the process currently occupied and filled it with sample defaults. Do you see the trap in that patch, even before the next run went green again?

I used MonkeyCode's free model access to iterate on a stricter helper after that stub-file detour. Then I reran the same commands on its free server option, which does not inherit my laptop cwd. That clean job was the first environment that did not start inside the git checkout at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One difference in the starting directory was enough for relative paths to aim at the wrong tree. The model had not failed at English; it had succeeded at removing the only honest error. Green logs after a generated stub are not a fix, and I should have treated them as contamination.

Hours 20–32: three print statements ended the mystery

I stopped asking the model for another patch and printed a triple on startup instead. The first line is cwd, the second is the script path, and the third is the resolved config. If those three paths do not share a repository root, I no longer trust the parsed YAML.

python -c "import os, pathlib; print(os.getcwd()); print(pathlib.Path('job.py').resolve()); print(pathlib.Path('config.yaml').resolve())"

On my laptop, after I had already changed into the repo, all three lines agreed perfectly. On the clean job they disagreed, and that disagreement was almost insultingly easy to read aloud. The script lived in the checkout, while the config path resolved beside the process, not the script.

Example output from a clean job, not a measurement of any host:

/home/job
/home/job/work/demo/job.py
/home/job/config.yaml

A quick search made the duplicate file obvious, which is embarrassing after two days of parser archaeology. I had been editing one YAML document, and the job had been reading a different document with the same name. That is a forty-eight hour bug with a one-line explanation, which is why I am writing these notes.

pwd
ls -l config.yaml ./config.yaml
find "$HOME" -name 'config.yaml' 2>/dev/null
python -c "import os; print(repr(os.getcwd()))"

The artifact: refuse relative configs unless cwd is the repo

I now treat an unexpected working directory as a startup error, not as a reason to write stubs. The helper below is a complete recipe for a throwaway repo, not a library I have shipped anywhere. Copy it, break it, and run the tests from /tmp before you believe a single relative path again.

# pin_repo.py — labeled recipe
from __future__ import annotations

from pathlib import Path
import os


class WrongWorkingDirectory(RuntimeError):
    """Raised when relative paths would silently read the wrong tree."""


def repo_root_from(anchor: Path) -> Path:
    start = anchor.resolve().parent if anchor.is_file() else anchor.resolve()
    for candidate in [start, *start.parents]:
        if (candidate / "pyproject.toml").is_file() or (candidate / ".git").exists():
            return candidate
    raise WrongWorkingDirectory(f"no repo root above {start}")


def require_repo_cwd(*, anchor: Path) -> Path:
    root = repo_root_from(anchor)
    cwd = Path.cwd().resolve()
    if cwd != root:
        raise WrongWorkingDirectory(
            f"cwd={cwd} expected_root={root}; "
            "pass an absolute CONFIG_PATH or chdir before opening files"
        )
    return root


def config_path(*, anchor: Path, env: dict[str, str] | None = None) -> Path:
    environ = env if env is not None else os.environ
    override = environ.get("CONFIG_PATH")
    if override:
        path = Path(override)
        if not path.is_absolute():
            raise WrongWorkingDirectory(
                f"CONFIG_PATH must be absolute, got {override!r}"
            )
        return path
    require_repo_cwd(anchor=anchor)
    return repo_root_from(anchor) / "config.yaml"

The job uses __file__ as the anchor, never the current directory, when it wants a sibling file.

# job.py — labeled example
from pathlib import Path
from pin_repo import config_path

HERE = Path(__file__).resolve()


def load_config() -> str:
    path = config_path(anchor=HERE)
    print(f"cwd={Path.cwd()} file={HERE} config={path}")
    return path.read_text(encoding="utf-8")

Notice that __file__ is the anchor when the job wants a sibling file, never the current directory. An absolute CONFIG_PATH still wins, and that keeps schedulers and containers from fighting the guard. A relative override is rejected, because it recreates the original bug with slightly fancier clothing.

A test that fails when cwd drifts

The tests should fail when cwd drifts, not when the YAML schema forgets a demo boolean flag. I put the repo markers in tmp_path so the helper cannot accidentally climb into my real checkout. If you already have a monorepo, pass an explicit env override instead of trusting parent walking alone.

# test_pin_repo.py — labeled example
from pathlib import Path
import pytest

from pin_repo import WrongWorkingDirectory, config_path, require_repo_cwd


@pytest.fixture
def repo(tmp_path: Path) -> Path:
    (tmp_path / "pyproject.toml").write_text("[project]\nname = 'demo'\n")
    (tmp_path / "config.yaml").write_text("sample_batch: false\n")
    (tmp_path / "job.py").write_text("# sentinel\n")
    return tmp_path


def test_require_repo_cwd_ok(repo: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    monkeypatch.chdir(repo)
    assert require_repo_cwd(anchor=repo / "job.py") == repo.resolve()


def test_require_repo_cwd_rejects_elsewhere(
    repo: Path, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
    elsewhere = tmp_path / "not-the-repo"
    elsewhere.mkdir()
    monkeypatch.chdir(elsewhere)
    with pytest.raises(WrongWorkingDirectory):
        require_repo_cwd(anchor=repo / "job.py")


def test_absolute_override_skips_cwd(
    repo: Path, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
    elsewhere = tmp_path / "not-the-repo"
    elsewhere.mkdir()
    monkeypatch.chdir(elsewhere)
    absolute = repo / "config.yaml"
    path = config_path(
        anchor=repo / "job.py",
        env={"CONFIG_PATH": str(absolute)},
    )
    assert path == absolute


def test_relative_override_is_rejected(
    repo: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
    monkeypatch.chdir(repo)
    with pytest.raises(WrongWorkingDirectory):
        config_path(
            anchor=repo / "job.py",
            env={"CONFIG_PATH": "config.yaml"},
        )

Run the suite, then run the module again after changing into /tmp, and keep both commands. The second command should raise WrongWorkingDirectory instead of inventing a new YAML file at the filesystem root. If it writes /config.yaml, you have reproduced the assistant's stub-file move on your own machine.

pytest test_pin_repo.py -q
python -c "import os; os.chdir('/tmp'); from job import load_config; load_config()"

Decision table I keep next to the helper

I keep a small decision table next to the helper because the symptoms rhyme and the fixes do not. Most of these rows showed up in the same forty-eight hours, just wearing different log messages. If your row is missing, add it before you ask a model to silence the exception again.

What you see	Likely cwd story	Do not do this	Do this instead
Tests pass, remote uses sample defaults	The shell started in the repo; the job did not	Let the model write a stub config	Print cwd, __file__, and the resolved path together
FileNotFoundError: config.yaml	The process started in $HOME or /tmp	Path("config.yaml").write_text(...)	Resolve from __file__ or require an absolute env override
Two files named config.yaml	A previous "fix" already polluted another directory	Delete only the file in the repo	find the extra copies and treat them as evidence
Nested pyproject.toml	Walking parents grabbed the wrong root	Trust the first marker file blindly	Set CONFIG_PATH or a REPO_ROOT env var explicitly
Frozen binary or zipapp	__file__ is not a real filesystem path	Call Path(__file__).parent and hope	Use importlib.resources or an explicit argv path

What I would repeat

I would log that cwd, file, and config triple before any YAML document hits the parser. I would fail closed when a relative CONFIG_PATH sneaks in through a compose file or a dashboard field. I would rerun the job once from a directory that is not the git root, on purpose. Would I still let a model draft the helper text? Yes, but I would not let it create files.

This is the short checklist I now paste into the pull request before I call the path handling done. It is boring on purpose, because boring checks are the ones that survive a late-night generated patch. If any box is unchecked, I assume the next environment will start somewhere I have not seen.

1. Print os.getcwd(), Path(__file__).resolve(), and the resolved config path on the first startup line.
2. Reject relative CONFIG_PATH values, and allow only absolute overrides to skip the cwd guard.
3. Run pytest in the repo, then run the same module again after changing into /tmp.
4. Search the home directory for extra config.yaml files that an assistant may already have planted.
5. Resolve data directories from the config file's parent directory, not from whatever cwd happens to be.

Limitations, and who should skip this

This guard is not a security boundary, and it will not save you from a wrong absolute path. Remember that __file__ is unreliable inside zipapps, frozen PyInstaller binaries, and some namespace-package layouts. Those apps need importlib.resources or an explicit command-line flag, not a pathlib parent walk. Monorepos with nested marker files can resolve inward when you wanted the outer project root.

Jobs that intentionally use a scratch working directory should skip require_repo_cwd and pass CONFIG_PATH. You should not adopt this pattern if your entrypoint already receives absolute paths from a scheduler. A container workingDir or a well-tested CLI already solves the problem without another helper module. You should not use it as an excuse to skip an integration run on a clean machine either.

You should not ask a model to make the error go away when the error is the only honest signal. The standard library is doing exactly what it promised with every relative Path object you construct. Path("config.yaml") is relative to the process, not relative to the source file, and that will not change. My tests had been lying by changing directory for me, and the sample defaults were the receipt.

Once I made a cwd mismatch a loud failure, the demo flags disappeared from the remote logs. The retry knobs I had been twisting for two days became irrelevant, which is a humbling kind of relief. Would I spend another forty-eight hours on YAML booleans next time? Only after I print cwd first.