A Characterization Harness for CWD-Sensitive Settings Paths

Path helper extracts fail when the opened file changes. Freeze every opened path before any extract. Mixed CWD, ENV, and __file__ sources need tests.

Value checks do not catch a silent file swap. Two files can return identical JSON bytes. Path identity is the contract to freeze.

Why mixed path sources break extracts

Most messy loaders open files from three sources. CWD relative names follow the current process. ENV paths override the choice without logging.

Package relative paths follow the module __file__. An extract often changes only one source. Callers then observe a different opened file.

Assertions on parsed values miss that swap. Defaults can mask a missing file entirely. Golden opened paths expose the miss immediately.

This article treats path identity as a test seam. It does not parse JSON in the first change. Parsing stays behind the recorded open.

The messy module under test

The module below is a labeled example. It is not a production snapshot. Do not treat the names as a real service.

# settings_load.py — labeled messy example, not live code
from __future__ import annotations

import json
import os
from pathlib import Path

_HERE = Path(__file__).resolve().parent
_OPENED: list[str] = []


def load_settings() -> dict:
    # Three sources, one return. Order is the real bug.
    env_path = os.environ.get("APP_SETTINGS")
    cwd_path = Path.cwd() / "settings.json"
    pkg_path = _HERE / "defaults" / "settings.json"

    chosen = pkg_path
    if cwd_path.is_file():
        chosen = cwd_path
    if env_path:
        chosen = Path(env_path)

    _OPENED.append(str(chosen))
    with chosen.open("r", encoding="utf-8") as handle:
        return json.load(handle)

The global _OPENED list is a temporary probe. Production code should not keep that list. Tests will replace it with a wrapper.

Freeze this decision table first

Record winners before touching function boundaries. The table below is the contract. Empty cells mean the source is absent.

ENV APP_SETTINGS	CWD settings.json	Package default	Expected opened path
set, file exists	present or absent	present	exact ENV path
set, file missing	anything	present	ENV path, then open error
unset	present	present	CWD file
unset	absent	present	package default
unset	absent	absent	package path, then open error

Note the missing-file rows with care. A later extract may swallow FileNotFoundError. That swallow is a contract change. Characterization tests must keep the exception type.

Workflow

1. Capture opens without refactoring

Wrap Path.open on the loader module only. Do not patch all of pathlib. Module-local wraps keep unrelated I/O free.

# test_settings_paths.py — labeled harness
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

import pytest

import settings_load


@pytest.fixture
def opened_paths(monkeypatch: pytest.MonkeyPatch) -> list[str]:
    seen: list[str] = []
    original = Path.open

    def tracking_open(self: Path, *args: Any, **kwargs: Any):
        seen.append(str(self))
        return original(self, *args, **kwargs)

    monkeypatch.setattr(Path, "open", tracking_open)
    return seen

This fixture records every Path.open call. It still performs the real open. Fake filesystem layers can wait.

2. Build one golden case per table row

Each test sets CWD, ENV, and files. Each test asserts the opened path string. Parsed JSON is a secondary assertion only.

def write_json(path: Path, payload: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(payload), encoding="utf-8")


def test_env_path_wins_over_cwd(
    tmp_path: Path,
    monkeypatch: pytest.MonkeyPatch,
    opened_paths: list[str],
) -> None:
    cwd = tmp_path / "cwd"
    cwd.mkdir()
    write_json(cwd / "settings.json", {"src": "cwd"})

    env_file = tmp_path / "from-env.json"
    write_json(env_file, {"src": "env"})

    monkeypatch.chdir(cwd)
    monkeypatch.setenv("APP_SETTINGS", str(env_file))

    data = settings_load.load_settings()

    assert opened_paths == [str(env_file)]
    assert data["src"] == "env"


def test_cwd_wins_when_env_unset(
    tmp_path: Path,
    monkeypatch: pytest.MonkeyPatch,
    opened_paths: list[str],
) -> None:
    cwd = tmp_path / "cwd"
    cwd.mkdir()
    write_json(cwd / "settings.json", {"src": "cwd"})

    monkeypatch.chdir(cwd)
    monkeypatch.delenv("APP_SETTINGS", raising=False)

    data = settings_load.load_settings()

    assert opened_paths == [str(cwd / "settings.json")]
    assert data["src"] == "cwd"


def test_package_default_when_cwd_missing(
    tmp_path: Path,
    monkeypatch: pytest.MonkeyPatch,
    opened_paths: list[str],
) -> None:
    cwd = tmp_path / "empty-cwd"
    cwd.mkdir()
    monkeypatch.chdir(cwd)
    monkeypatch.delenv("APP_SETTINGS", raising=False)

    expected = settings_load._HERE / "defaults" / "settings.json"
    expected.parent.mkdir(parents=True, exist_ok=True)
    if not expected.is_file():
        write_json(expected, {"src": "pkg"})

    data = settings_load.load_settings()

    assert opened_paths == [str(expected)]
    assert data["src"] == "pkg"

Keep ENV missing-file behavior in a dedicated test. The open must fail with FileNotFoundError. Do not replace that with a default.

def test_env_missing_file_still_raises(
    tmp_path: Path,
    monkeypatch: pytest.MonkeyPatch,
    opened_paths: list[str],
) -> None:
    missing = tmp_path / "nope.json"
    monkeypatch.chdir(tmp_path)
    monkeypatch.setenv("APP_SETTINGS", str(missing))

    with pytest.raises(FileNotFoundError):
        settings_load.load_settings()

    assert opened_paths == [str(missing)]

3. Run the harness before any extract

Use one command and a clean working tree. Do not mix formatting with this run.

git status --porcelain
python -m pytest test_settings_paths.py -q

A dirty tree hides the later diff. Green tests are the extract permit. Red tests mean the table is wrong.

4. Extract one resolver only

The smallest safe change is path selection. Leave json.load in load_settings. Leave encoding and error text alone.

def resolve_settings_path() -> Path:
    env_path = os.environ.get("APP_SETTINGS")
    if env_path:
        return Path(env_path)
    cwd_path = Path.cwd() / "settings.json"
    if cwd_path.is_file():
        return cwd_path
    return _HERE / "defaults" / "settings.json"


def load_settings() -> dict:
    chosen = resolve_settings_path()
    _OPENED.append(str(chosen))
    with chosen.open("r", encoding="utf-8") as handle:
        return json.load(handle)

Re-run the same pytest file after the extract. Opened paths must match the golden list. Exception types must match the missing-file row.

5. Delete the probe after the extract

Remove _OPENED once tests wrap Path.open. Dual probes drift and lie. One recording seam is enough.

Where a hosted model helps this harness

Drafting row fixtures by hand is slow. A free coding model can propose missing rows. It cannot certify the opened path contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access for drafting cases. It also offers a free server option. Use that server to run this pytest file when local Python is blocked.

Paste the messy loader and the table only. Ask for tests that assert opened paths. Reject patches that also rewrite JSON parsing.

Treat model output as untrusted scaffolding. Keep the decision table as the review artifact. Discard cases that invent extra ENV names.

What this harness does not prove

This harness does not prove value equality. It only locks which path the loader opens. Two different files can still parse alike.

Normalized real paths can still hide symlinks. Windows short names also break naive equality. Compare the string the loader passed to open.

It does not lock encoding error text. It does not lock key order inside JSON. Those contracts need a later characterization pass.

It does not lock import-time work. Path(__file__).resolve() runs at import. Import the module after chdir in a subprocess if needed.

# labeled follow-up, not required for the first extract
import subprocess
import sys

def test_import_does_not_open_settings() -> None:
    code = (
        "import settings_load, json, sys; "
        "sys.stdout.write(json.dumps(settings_load._OPENED))"
    )
    out = subprocess.check_output([sys.executable, "-c", code], text=True)
    assert out == "[]"

Skip that subprocess test if import stays pure. Add it when module import reads ENV. Import-time I/O is a separate extract.

Who should not use this approach

Do not use this on secret-bearing config files. Do not commit opened paths that include home directories. Redact absolute prefixes in failure messages.

Do not apply this to packed binary resources. Zip imports and namespace packages change __file__. Those loaders need a different seam.

Do not extract resolver and parser together. Dual extracts hide which contract moved. One function boundary per green run.

Teams without pytest fixtures should not copy this file. The method needs tmp_path and monkeypatch. A weaker runner will leak CWD.

Review checklist after the extract

1. git diff shows resolve_settings_path plus call-site only.
2. Every table row still has a named test.
3. Missing ENV files still raise FileNotFoundError.
4. No test asserts JSON keys except as a weak extra.
5. No new ENV names appear outside the table.

If step 1 fails, revert and shrink the patch. If step 3 fails, the extract changed errors. Error text is out of scope here.

Closing constraint

Opened path identity is the first refactor lock. Extract one resolver after those goldens stay green. Leave merge logic and parsing for later.