48-Hour Field Notes: The Settings File Was in Git. open() Used Another CWD.

I spent forty-eight hours convinced a remote process could not see settings.json, even though the file was committed. Every local run succeeded because I launched python app.py from the repo root, and I never questioned that habit. The failure appeared only when a scheduler, a test runner, or a remote shell started the same file from another directory. Have you ever blamed a missing file when the real bug was the working directory under your feet?

What I thought was broken

I treated this like an inventory problem, not a path problem, which sent me down the wrong list of checks.

• I re-cloned the repository and confirmed settings.json was not hiding in .gitignore.
• I printed the filename in logs, which still showed the relative string settings.json.
• I checked permissions, encodings, and case-sensitive filesystems, none of which were the fault.
• I stared at packaging docs as if a wheel had eaten a file that git ls-files could still see.

None of those checks can save you when open("settings.json") is documented to use os.getcwd(), not the script location. Relative paths look complete in a log line until you remember they are incomplete by design. Why do we keep logging the name we intended instead of the path we actually resolved?

Hours 0-12: the false fixes

I tried os.path.abspath("settings.json") first, because the word absolute sounds like it should pin the file down. It does not pin anything; abspath still starts from the current working directory and only then makes the string look fully qualified. The log then showed /tmp/settings.json or /home/runner/settings.json, which felt more serious to me and was still completely wrong. Have you noticed how a longer path can make a bad assumption look like a finished investigation?

I then passed a --config flag, but the default value inside argparse was still that same relative name. Defaults are evaluated against whatever cwd exists at parse time, not against the directory that contains your module. So the flag made the code look configurable while the failure mode stayed identical, and I was only polishing the interface of a bug.

The command that should have ended hour two

I wish I had printed three strings before I blamed Git, Docker, or a “missing” checkout.

python -c "import os, pathlib, sys; print('cwd', os.getcwd()); print('argv0', sys.argv[0]); print('home', pathlib.Path.home())"
pwd
ls -la

argv0 can be a relative name too, so it is a witness, not a verdict. pwd and os.getcwd() can also disagree with __file__, and that disagreement is the whole bug. Would you still re-clone a repo after seeing those three lines point at different trees?

Hours 12-24: the environment that finally disagreed with my laptop

My laptop hid the bug because my shell history always cd'd into the project before running anything. A clean process does not inherit that ritual, so the same entry point can fail without any code change. I reproduced the miss on a free remote environment where the default working directory was not my repo root.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free server option was useful here because it did not start in the folder my fingers expect. The free model access sat next to that runtime, so I asked for a patch without pasting cwd or __file__. It suggested os.path.abspath first and, in another pass, a hardcoded /Users/... path copied from a laptop-shaped example. Both answers compile, and both answers fail the moment the working directory changes under the process.

Would you have caught that without a second machine that refuses to share your local shell habits? I am not claiming a particular model name, quota, or benchmark here, because those details are not the point. The point is that a second cwd is a test fixture, and a suggested patch still needs a test that changes directory on purpose.

The artifact: force cwd to lie, then assert the real path

The following example is a small local reproduction you can run yourself, and it is not a production settings stack. Treat it as a test harness rather than a framework, and label the loaders as deliberate contrasts. I want the cwd loader to fail in the test, because a green test in the module directory would lie to me.

# resolve_settings.py
from __future__ import annotations

import json
import os
from pathlib import Path


def load_from_cwd(name: str = "settings.json") -> dict:
    # Deliberately wrong for app config shipped next to the module.
    with open(name, encoding="utf-8") as handle:
        return json.load(handle)


def load_from_module(name: str = "settings.json") -> dict:
    here = Path(__file__).resolve().parent
    path = here / name
    with path.open(encoding="utf-8") as handle:
        return json.load(handle)


def boot_report() -> dict:
    file_path = Path(__file__).resolve()
    return {
        "cwd": os.getcwd(),
        "file": str(file_path),
        "module_dir": str(file_path.parent),
        "cwd_settings": str(Path.cwd() / "settings.json"),
        "module_settings": str(file_path.parent / "settings.json"),
    }

# test_resolve_settings.py
from __future__ import annotations

import json
from pathlib import Path

import pytest

import resolve_settings


@pytest.fixture
def isolated_settings(tmp_path, monkeypatch):
    settings = tmp_path / "settings.json"
    settings.write_text(json.dumps({"source": "module"}), encoding="utf-8")
    stub = tmp_path / "resolve_settings.py"
    stub.write_text("# stub for __file__\n", encoding="utf-8")
    elsewhere = tmp_path / "elsewhere"
    elsewhere.mkdir()
    monkeypatch.setattr(resolve_settings, "__file__", str(stub))
    monkeypatch.chdir(elsewhere)
    return tmp_path


def test_cwd_loader_breaks_when_we_chdir(isolated_settings):
    with pytest.raises(FileNotFoundError):
        resolve_settings.load_from_cwd()


def test_module_loader_survives_chdir(isolated_settings):
    payload = resolve_settings.load_from_module()
    assert payload["source"] == "module"


def test_boot_report_shows_disagreement(isolated_settings):
    report = resolve_settings.boot_report()
    assert Path(report["cwd"]).name == "elsewhere"
    assert Path(report["file"]).name == "resolve_settings.py"
    assert report["cwd"] != report["module_dir"]
    assert report["cwd_settings"] != report["module_settings"]

Run the harness from a directory that is not the module directory, or the relative loader will fake a pass.

pytest test_resolve_settings.py -q
python -c "import json, os, sys; sys.path.insert(0, '.'); os.chdir('/tmp'); import resolve_settings; print(json.dumps(resolve_settings.boot_report(), indent=2))"

If the cwd loader passes while you are sitting in the module directory, you have not tested the bug. You have only tested your habit, which is the same trap that burned the original forty-eight hours. Run pytest after adding the chdir, or the suite will keep blessing the relative open() call.

Commands I now print before I blame Git

I want the process to confess its location before I invent another packaging or permissions story.

python -c "import resolve_settings, json; print(json.dumps(resolve_settings.boot_report(), indent=2))"
readlink -f resolve_settings.py || realpath resolve_settings.py

Print cwd, __file__, and the resolved config path on boot, even when the script still feels too simple for that. A one-line boot report is cheaper than another clone from a worried operator at two in the morning. If those three strings disagree with each other, stop debugging the contents of the file and fix the resolver first.

A decision table I wish I had on hour two

Not every relative path is a bug, and some programs should read the directory the user is standing in. I do not want to fix a formatter into module-relative lookups, because that would ignore the user's workspace. The table below is the filter I now apply before I touch a path helper in application code.

Kind of file	Resolve against	Why
User-supplied input, ./output/, globs in a CLI	Current working directory	The user is pointing at a workspace, not at your package
Default settings.json shipped next to the code	Path(__file__).resolve().parent	Install location is stable; cwd is not
Secrets, machine names, feature flags	Environment variables or a secrets manager	Those values should not ride along in git
Optional operator override	--config with no relative default	A flag without a cwd-based default cannot hide this bug
User-level edits	XDG or Path.home() config dir	Home is not cwd, and it is not the repo either

If you only remember one row, remember this one: application defaults belong next to __file__, and user workspaces belong to os.getcwd(). Mixing those rows is how I lost two days, and it is also how the tests stayed green on my laptop. Read the table twice if you are about to change every open() call in a mixed CLI and library repo.

What actually broke, in named failure modes

I want the failure modes named clearly, because every symptom above looked like a simple file-not-found error. Numbering them kept me from retesting permissions after I already knew the working directory was the real fault. You can reuse the list as a checklist the next time a remote run cannot see a committed file.

1. open("settings.json") consulted cwd, so a scheduler starting in / could not see the repo file.
2. os.path.abspath("settings.json") still consulted cwd, then dressed the wrong location in an absolute costume.
3. A model-suggested /Users/me/project/settings.json could not exist on a remote server, and it also could not exist for anyone else.
4. pytest sometimes kept me in the project root, so a test without monkeypatch.chdir never exercised the miss.
5. python -c and some frozen entry points do not define __file__, so a loader must have a fallback.

That last point matters more than it looks, because __file__ is the right default for a normal module, not a universal law. If you ship a zipapp, a namespace package, or a -c snippet, keep an explicit --config or SETTINGS_PATH override. Do not pretend one path trick covers every runtime, especially when the entry point is generated or frozen.

What I would repeat

I would repeat the unglamorous parts of this hunt, because those parts are what actually ended the loop. The glamorous theory about missing files did not end it, and neither did a prettier absolute string. A chdir test and a boot report did, which is why they sit at the top of my repeat list.

• I would print a boot report with cwd, __file__, and the resolved settings path before any parse happens.
• I would add one test that chdirs into a temp directory and still loads the shipped defaults.
• I would refuse patches that only call abspath on a relative name, even when they look cleaner in review.
• I would keep a second working directory around, including a free remote shell, so laptop muscle memory cannot hide the miss.
• I would ask any model for a fix only after I paste the boot report, otherwise it will optimize the story I told.

Would I still use a model to draft the loader after this mess, knowing it likes abspath as a default answer? Yes, but I would make the chdir test the acceptance gate, not the prose of the patch. The test is the artifact I trust, and the model is only a stenographer until that test is red then green.

Who should not copy this

Do not blindly switch every open() to __file__ when you are writing a CLI that formats user files. Formatters, linters, compilers, and find-like tools should honor cwd, and users get angry when they do not. Notebooks, python -c snippets, and some frozen binaries should not assume that __file__ exists in every runtime. And please do not park production secrets in a JSON file beside the module just because the path problem is now solved.

This approach also will not help if the file truly is missing from the image or the clone. A volume mount that shadows the directory will still win after you fix the resolver, so keep that on the list. Path resolution is not packaging, and packaging is not path resolution, even when the error string is identical. After the boot report agrees with itself, go back to inventory checks with a much shorter search space.

Limitations I am not going to paper over

Calling resolve on __file__ follows symlinks, which is usually what I want and is occasionally the wrong parent. If your deploy uses a symlink farm, print both the unresolved and resolved parents during boot. resolve() can also complain if a path does not exist yet, which is rare for __file__ but possible in generated layouts. I have not measured performance here because a single path resolve at process start is not the budget that matters for this bug.

I also have not claimed that any hosted runtime stays free forever, or that a remote shell matches production. A free server is a different cwd for your process, not a replica of your busiest production cluster. Use it to disagree with your laptop, and do not use it as the only staging environment you have.

Closing the notes

The settings file was in git the whole time, and the process was simply standing somewhere else. I spent forty-eight hours arguing with the map instead of asking the process where it was standing. Relative open() calls are honest about that once you remember they never promised to follow the module. Absolute strings are not a fix unless they start from __file__, an env var, or a user flag with no cwd-based default.

If this spares you a clone-and-pray cycle, tell me which three paths you now print on boot. I am curious whether you include argv[0], because that string lies almost as often as a relative filename. A comment with your boot report shape is more useful to me than another generic note that this also bit you.