Extracting config path logic is unsafe without a process input matrix.
Unit tests that mock Path objects hide cwd and env coupling.
Record cwd, env keys, and relative opens first.
Only then extract one config path helper.
Messy CLIs resolve files from several process inputs.
Those inputs include cwd, HOME, XDG vars, and argv flags.
A coding model often rewrites that resolution during cleanup.
The diff looks small while observed file opens shift.
Green tests still hide process coupling
Typical unit tests inject a fake Path or tmp_path.
They never start the process with a foreign cwd.
They also inherit the developer's full exported environment.
HOME, XDG_CONFIG_HOME, and EDITOR leak into results.
Relative opens then resolve against the test runner cwd.
That cwd is pytest's directory, not the user shell.
The production entrypoint still calls os.getcwd() internally.
It still indexes os.environ for keys nobody listed.
Config extracts fail in those hidden branches first.
A helper that looks cleaner can change search order.
Artifact: a portable process input matrix
The artifact is a JSONL matrix, not a mocked unit test.
Each matrix row captures one controlled process execution.
Columns record cwd class, env subset, and argv.
They also record observed env keys and relative opens.
Exit code belongs in the same row as those opens.
A later extract must keep every row byte-stable.
Store opens relative to CFGMATRIX_ROOT, not host prefixes.
That keeps goldens portable across laptops and remote runners.
Label this harness as a proposed local example.
Do not point it at credentials or live user homes.
1. Inventory the messy entrypoint
Pick one CLI module that builds config paths inline.
Do not extract anything during this inventory step.
python -c "import messy_app.cli as c; print(c.__file__)"
Note every Path, open, and os.environ access you see.
Static notes remain hypotheses until the tracer confirms them.
2. Build a hermetic fixture tree
Create three fixture directories with known files only.
Keep tokens, keys, and user secrets out of this tree.
mkdir -p /tmp/cfgmatrix/{repo,home,xdg}/.config/messy
mkdir -p /tmp/cfgmatrix/empty
printf 'source=repo\n' > /tmp/cfgmatrix/repo/messy.ini
printf 'source=home\n' > /tmp/cfgmatrix/home/.config/messy/config.ini
printf 'source=xdg\n' > /tmp/cfgmatrix/xdg/messy/config.ini
Map each directory to one cwd class in the matrix.
The repo class starts the process inside the project tree.
The empty class starts inside a directory with no config.
The xdg class sets XDG_CONFIG_HOME over a dedicated tree.
3. Install an env and open tracer
The tracer wraps os.environ access and builtins.open.
It also wraps pathlib.Path.open when that method exists.
Place the file on PYTHONPATH for matrix runs only.
Do not install the tracer as a global site module.
This is a proposed example, not a production tracer.
Replace package names before any real matrix run.
# tracer_cfg.py — proposed example, not a production agent
from __future__ import annotations
import builtins
import json
import os
import pathlib
import sys
from typing import Any
_KEYS: list[str] = []
_OPENS: list[str] = []
_SKIP = {
"PATH", "PYTHONPATH", "LANG", "LC_ALL",
"SYSTEMROOT", "COMSPEC", "CFGMATRIX_ROOT",
"PYTHONNOUSERSITE",
}
def _note_key(key: object) -> None:
k = str(key)
if k in _SKIP:
return
if k not in _KEYS:
_KEYS.append(k)
class _EnvProxy:
def __init__(self, real: os._Environ) -> None:
object.__setattr__(self, "_real", real)
def __getitem__(self, key: str) -> str:
_note_key(key)
return self._real[key]
def get(self, key: str, default: Any = None) -> Any:
_note_key(key)
return self._real.get(key, default)
def __contains__(self, key: object) -> bool:
_note_key(key)
return key in self._real
def __getattr__(self, name: str) -> Any:
return getattr(self._real, name)
def __setitem__(self, key: str, value: str) -> None:
self._real[key] = value
def __delitem__(self, key: str) -> None:
del self._real[key]
def __iter__(self):
return iter(self._real)
def __len__(self) -> int:
return len(self._real)
def _record_open(path: object) -> None:
text = os.fspath(path)
abs_text = text if os.path.isabs(text) else os.path.abspath(text)
root = os.environ.get("CFGMATRIX_ROOT", "")
if root and abs_text.startswith(root):
_OPENS.append(os.path.relpath(abs_text, root))
return
_OPENS.append(text)
_real_open = builtins.open
def _open(path, *args, **kwargs):
_record_open(path)
return _real_open(path, *args, **kwargs)
_real_path_open = pathlib.Path.open
def _path_open(self, *args, **kwargs):
_record_open(self)
return _real_path_open(self, *args, **kwargs)
def install() -> None:
os.environ = _EnvProxy(os.environ) # type: ignore[misc]
builtins.open = _open # type: ignore[misc]
pathlib.Path.open = _path_open # type: ignore[method-assign]
def dump(exit_code: int) -> None:
payload = {
"argv": sys.argv[1:],
"env_keys": _KEYS,
"relative_opens": _OPENS,
"exit_code": exit_code,
}
sys.stderr.write("CFGMATRIX " + json.dumps(payload) + "\n")
install()
Wrap the entrypoint so dump always runs on exit.
Dump writes one JSON object to stderr for the runner.
# run_traced.py — proposed example
from __future__ import annotations
import runpy
import sys
import tracer_cfg
mod = sys.argv[1]
sys.argv = [mod, *sys.argv[2:]]
code = 0
try:
runpy.run_module(mod, run_name="__main__")
except SystemExit as exc:
raw = exc.code
code = int(raw) if isinstance(raw, int) else 1
finally:
tracer_cfg.dump(code)
4. Execute the six-row matrix
Six rows are enough for a first lock.
Add rows only after a real failure mode appears.
This proposed runner keeps env values inside the process.
The golden stores names, opens, and exit codes only.
# matrix_run.py — proposed example
from __future__ import annotations
import json
import os
import subprocess
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
ROOT = Path("/tmp/cfgmatrix")
ROWS = [
{
"id": "repo-default",
"cwd": ROOT / "repo",
"env": {"HOME": str(ROOT / "home")},
"argv": ["messy_app.cli"],
},
{
"id": "empty-home",
"cwd": ROOT / "empty",
"env": {"HOME": str(ROOT / "home")},
"argv": ["messy_app.cli"],
},
{
"id": "empty-xdg",
"cwd": ROOT / "empty",
"env": {
"HOME": str(ROOT / "home"),
"XDG_CONFIG_HOME": str(ROOT / "xdg"),
},
"argv": ["messy_app.cli"],
},
{
"id": "flag-abs",
"cwd": ROOT / "empty",
"env": {"HOME": str(ROOT / "home")},
"argv": ["messy_app.cli", "--config", str(ROOT / "repo" / "messy.ini")],
},
{
"id": "flag-rel",
"cwd": ROOT / "repo",
"env": {"HOME": str(ROOT / "home")},
"argv": ["messy_app.cli", "--config", "messy.ini"],
},
{
"id": "empty-nohome",
"cwd": ROOT / "empty",
"env": {},
"argv": ["messy_app.cli"],
},
]
BASE_ENV = {
"PATH": os.environ.get("PATH", ""),
"PYTHONPATH": str(HERE),
"PYTHONNOUSERSITE": "1",
"CFGMATRIX_ROOT": str(ROOT),
}
def run_row(row: dict) -> dict:
env = dict(BASE_ENV)
env.update(row["env"])
proc = subprocess.run(
[sys.executable, str(HERE / "run_traced.py"), *row["argv"]],
cwd=row["cwd"],
env=env,
capture_output=True,
text=True,
check=False,
)
marker = ""
for line in proc.stderr.splitlines():
if line.startswith("CFGMATRIX "):
marker = line[len("CFGMATRIX ") :]
body = json.loads(marker) if marker else {
"argv": row["argv"],
"env_keys": [],
"relative_opens": [],
"exit_code": proc.returncode,
"tracer_missing": True,
}
return {
"id": row["id"],
"env_names": sorted(row["env"]),
"argv": row["argv"],
"env_keys": body.get("env_keys", []),
"relative_opens": body.get("relative_opens", []),
"exit_code": body.get("exit_code", proc.returncode),
}
def main() -> None:
rows = [run_row(r) for r in ROWS]
Path("cfg_matrix.golden.jsonl").write_text(
"".join(json.dumps(r, sort_keys=True) + "\n" for r in rows),
encoding="utf-8",
)
if __name__ == "__main__":
main()
5. Freeze the JSONL as the golden
Write the golden beside the messy module, not in /tmp.
Treat any drift as a failed extract, not as noise.
python matrix_run.py
mkdir -p tests/goldens
cp cfg_matrix.golden.jsonl tests/goldens/cfg_matrix.golden.jsonl
git add tests/goldens/cfg_matrix.golden.jsonl
Compare with a strict byte check in CI later.
Do not pretty-print in a way that reorders keys.
The runner already uses sort_keys=True for stability.
Keep that flag when you re-run after the extract.
6. Allow one extract only
The only legal diff extracts one resolve_config_path helper.
Call sites must pass the same strings they used before.
Avoid logging changes, default-value edits, and extra files.
Re-run the matrix after that single diff.
If any row changes keys, opens, or exit code, revert.
The helper is wrong even if unit tests pass.
Proposed shape after the extract. Do not copy it blindly.
# proposed extract — keep search order identical to the golden
def resolve_config_path(cli_flag: str | None) -> str:
if cli_flag:
return cli_flag
xdg = os.environ.get("XDG_CONFIG_HOME")
if xdg:
return os.path.join(xdg, "messy", "config.ini")
home = os.environ.get("HOME")
if home:
return os.path.join(home, ".config", "messy", "config.ini")
return os.path.join(os.getcwd(), "messy.ini")
That order is a hypothesis until the golden agrees.
If the golden shows repo cwd files winning, keep that order.
Fill the review table from JSONL
Do not invent expected opens before the first freeze.
Fill this table from the recorded golden rows.
| id | cwd class | env names | argv shape | env keys | relative opens | exit |
|---|---|---|---|---|---|---|
| repo-default | repo | HOME | module only | from golden | from golden | from golden |
| empty-home | empty | HOME | module only | from golden | from golden | from golden |
| empty-xdg | empty | HOME, XDG_CONFIG_HOME | module only | from golden | from golden | from golden |
| flag-abs | empty | HOME | abs --config | from golden | from golden | from golden |
| flag-rel | repo | HOME | rel --config | from golden | from golden | from golden |
| empty-nohome | empty | (none) | module only | from golden | from golden | from golden |
The table is documentation for reviewers, not a second oracle.
The JSONL file remains the only machine-checked contract.
Search order bugs show up as a changed relative_opens list.
Missing HOME reads show up as a shorter env_keys list.
Free model edits still need this gate
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
Those matter when laptop environments are already polluted.
A model can propose the extract on that server.
The JSONL golden remains the only acceptance gate.
Do not treat model commentary as evidence of safety.
Re-run the matrix in the same hermetic fixture tree.
Keep recorded keys as names only in every log.
Never persist env values from a developer shell.
Limitations
Python tracers miss opens performed inside C extensions.
ctypes loaders and some SSL paths will not appear.
The tracer does not wrap os.open or os.stat.
Native file probes can therefore miss the golden.
Windows drive letters and case folding are not covered here.
Ship Windows binaries only after a Windows matrix run.
Env allowlists that are too broad leak secrets into logs.
This golden stores key names, never key values.
The matrix does not prove parsed config values are correct.
It only pins how the process locates files.
Threads can open files after import but before dump.
Install the tracer before importing the messy package.
sitecustomize hooks can fight this PYTHONPATH-based tracer.
Unset PYTHONSTARTUP and user site during matrix runs.
Who should not use this approach
Do not use this on CLIs that must touch live credentials.
Do not use this if the app is not primarily Python.
Do not extract path logic during an incident hotfix.
Do not skip the golden because the diff looks tiny.
Operators without a hermetic fixture directory should wait.
Shared CI runners with leftover HOME files will poison rows.
Close
Process inputs are the contract for config path code.
Fixture them, trace them, then extract one helper.
Run the matrix on a free server if the laptop env is dirty.
Keep the allowlist to key names, and reject any extra diff.
Top comments (0)