Green Tests, Shadow Package: 48-Hour Notes on sys.path[0]

Have you ever watched an agent rewrite a package, then watched every test pass, then watched production still call last week's code? I have, and the mismatch hid in import order rather than in the git history I had already reviewed. This is a 48-hour field notebook about that class of failure, not a victory lap. I am writing it down so the next run prints an origin check before I trust a green bar.

Hour 0–6: the symptom that looked like a win

The agent produced a tidy diff against src/billing/totals.py, and pytest answered with a wall of dots. The summary claimed zero failures, and I came very close to merging on that feeling alone. Then I printed one extra line from a smoke script that lived outside the test folder, and the old rounding bug walked right back in. How can a suite stay green while a one-liner still sees the previous function body?

I did the cheap checks first, because they usually catch the mistakes I make with my own hands.

• git status and git diff both showed the new rounding helper under src/
• rg "def round_cents" found only the new body in the tree I thought I was testing
• python -c "import billing; print(billing.__file__)" printed a path I did not recognize at all

That third line was the entire incident, and I still spent the next two days proving it to myself. The import succeeded without a traceback, yet the loaded file was not the file I had just reviewed. Have you trained yourself to treat a successful import as proof of the right file? I had, and that habit is expensive.

Hour 6–18: what I tried that did not help

I restarted the shell, deleted every __pycache__ tree I could find, and reinstalled the editable package. None of those steps changed the smoke script's __file__, which kept pointing at a temp path. Why would cache clearing fail when the source on disk was already the version I wanted? Because the interpreter never looked at that source tree in the first place.

The generated runner stole index zero

The agent had generated a runner script under a scratch directory, which is a reasonable way to try a patch. Python then put that directory at sys.path[0], and that behavior is documented, not mysterious. A folder named billing/ had been copied there so the agent could experiment without touching git. The test process imported that copy, while my repo package sat later on the path, untouched and almost insultingly correct.

Here is the command sequence I actually ran, and you should run it before arguing with the agent about test quality.

python -c "import sys, pprint; pprint.pp(list(enumerate(sys.path)))"
python -c "import billing, inspect; print(billing.__file__); print(inspect.getsource(billing.round_cents)[:200])"
find /tmp -maxdepth 3 -type d -name billing 2>/dev/null | head

The first command showed scratch at index zero, which already explained the rest of the night. The second printed a file under /tmp and a function body that only existed in the copy. The third confirmed a shadow package the agent had left behind for convenience. Three observations, one process, and no agreement with src/.

Hour 18–30: an origin check I now require

I wanted a check that fails when the imported file is not under the repo I think I am testing. Assertions about return values cannot catch this failure, because the scratch copy can implement the same tests the agent just wrote. The origin check has to be about identity, not about behavior, or the suite will keep congratulating itself. Would a perfect assertion on round_cents(199) have saved me? No, because the shadow module can return 2.00 as cheerfully as the real one.

The script below is the artifact I now keep in tools/import_origin.py. It is ordinary Python with no framework magic, and that is the point. Run it with the same interpreter the tests use, or the printed origin is just another story.

"""Print where a module really came from. Fail if it is not under repo/src."""

from __future__ import annotations

import argparse
import importlib
import json
import sys
from pathlib import Path


def describe(modname: str) -> dict:
    payload = {
        "cwd": str(Path.cwd()),
        "executable": sys.executable,
        "sys_path": [str(p) for p in sys.path],
        "module": modname,
    }
    module = importlib.import_module(modname)
    file_path = Path(getattr(module, "__file__", "") or "").resolve()
    spec = module.__spec__
    payload.update(
        {
            "file": str(file_path) if file_path else None,
            "package": module.__package__,
            "origin": getattr(spec, "origin", None),
            "search_locations": [
                str(p) for p in (getattr(spec, "submodule_search_locations", None) or [])
            ],
        }
    )
    return payload


def assert_under_src(payload: dict, src_root: Path) -> None:
    file_path = payload.get("file")
    if not file_path:
        raise SystemExit(
            f"{payload['module']} has no __file__; namespace packages need a different check"
        )
    resolved = Path(file_path).resolve()
    src_root = src_root.resolve()
    if src_root not in resolved.parents and resolved != src_root:
        raise SystemExit(
            f"imported {payload['module']} from {resolved}, expected a path under {src_root}"
        )


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("module")
    parser.add_argument("--src", type=Path, default=Path("src"))
    parser.add_argument("--json", action="store_true")
    args = parser.parse_args()
    payload = describe(args.module)
    if args.json:
        print(json.dumps(payload, indent=2))
    else:
        print(f"cwd={payload['cwd']}")
        print(f"executable={payload['executable']}")
        for i, entry in enumerate(payload["sys_path"]):
            print(f"sys.path[{i}]={entry}")
        print(f"file={payload['file']}")
        print(f"origin={payload['origin']}")
    assert_under_src(payload, args.src)


if __name__ == "__main__":
    main()

I also added a tiny pytest guard so the suite cannot go green on a shadow copy. Keep the path comparison boring, because clever string prefixes fail on Windows and on bind mounts.

from pathlib import Path
import billing


def test_billing_comes_from_repo_src():
    src = Path(__file__).resolve().parents[1] / "src"
    imported = Path(billing.__file__).resolve()
    assert src in imported.parents, f"{imported} is not under {src}"

Would I still believe a coverage number after this kind of miss? Only if the coverage tracer printed the same __file__ the origin check printed. Coverage of a shadow module is still coverage of the wrong tree.

Hour 30–42: isolating the interpreter from my laptop

Local machines lie in a special way that I keep underestimating after a long week. Mine had an editable install, a leftover PYTHONPATH export, and a sitecustomize.py I had forgotten in a user directory. An agent that runs on my box inherits all of that folklore, then reports a green suite as if the environment were empty. Does your laptop still have a .pth file from a tool you uninstalled last quarter? Mine did, and it quietly extended sys.path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode for a narrow job: free model access to draft the origin script, and the free server option so the same commands ran away from my laptop's import path. I am not claiming a model name, a quota, or a hardware spec I cannot verify from this chair. I am claiming that a second machine with a stock interpreter made the scratch-directory import obvious in a single run. If you already have a clean CI image, use that image instead, because the workflow does not depend on a vendor.

The isolated run printed sys.path[0] as the project root I expected from the docs. The laptop run printed a /tmp/agent-scratch-... entry first, then my user site-packages, then the repo. Same git commit, different first path entry, different module file. That split is the whole incident, and it does not show up in a diff.

A decision table I wish I had on hour one

What you see	What I check next	What would fool me
Tests pass, smoke script fails	module.__file__ versus src/	Agent-written tests that only import the scratch copy
Source looks right, behavior looks old	sys.path[0] and the working directory	A generated runner living outside the repo
Deleting __pycache__ changes nothing	A shadow package earlier on sys.path	Namespace packages with no __file__
Editable install listed, old code runs	Whether tests launched via a generated script	PYTHONPATH set in a wrapper the agent owns
CI green, laptop red	Working directory and -P / PYTHONSAFEPATH	Different first path entries, identical command text

Locking sys.path[0] when the runner is generated

Python 3.11 added -P and PYTHONSAFEPATH so you can refuse to prepend the script directory. I now launch agent-driven tests with that flag when the runner is a generated file sitting in temp space. The flag is not a personality test for the agent, and it will not make a bad patch correct. It is a lock on index zero, which is the only index the shadow package needed.

python -P tools/import_origin.py billing --src src
python -P -m pytest -q

If your platform is older than 3.11, do not export PYTHONSAFEPATH and assume the interpreter honors it. Confirm the version first, then decide whether you need a wrapper that changes into the repo and launches python -m pytest instead of a scratch script. I prefer -m because it keeps sys.path[0] closer to the working directory I chose on purpose.

Why grep is not an import

Would I trust an agent that confirms imports by grepping the source tree after a patch? Not anymore, because grep proves a file exists on disk. It does not prove the interpreter loaded that file during the test process you are about to ship. rg is a map of the repo. module.__file__ is a map of the process. Those two maps are allowed to disagree, and they will.

What broke when I got sloppy

I once pointed --src at the repo root instead of src/, and a naive prefix check would have passed on a longer temp path. That is why the guard uses Path.parents, not str.startswith, even though the latter looks friendlier in a screenshot. I also learned that namespace packages can leave __file__ empty, so the origin check must fail closed rather than skip the assertion and smile.

Another break cost me an evening I will not get back. I ran the origin script with a different Python than pytest, and the printed path was honest for the wrong process. Same trap, smaller costume, still a lie about the suite. Always print sys.executable in the test session itself, not in a neighboring shell that happens to share a prompt color.

What I would repeat, and who should skip this

These are the steps I will run again the next time a generated suite looks suspiciously perfect after a busy night.

1. Print sys.executable, sys.path, and module.__file__ in the same process as the tests.
2. Fail the build if the imported file is not under the expected src/ directory.
3. Launch generated runners with python -P so scratch directories cannot steal index zero.
4. Re-run the origin check on a clean interpreter, not on the laptop that accumulated PYTHONPATH folklore.

Skip this approach if you ship only frozen binaries with no import graph, or if your tests are allowed to import generated stubs by design. Skip it if you cannot name the package you believe is under test, because an identity check needs a name. Skip it if you are mid-incident on a production pager and you need a rollback, not a lecture about sys.path.

The limitation is blunt: this origin check does not prove behavior, thread safety, or that the dependency graph is pinned. It only proves which file was loaded into that one process. That is a small claim, and it would have saved me most of the 48 hours I spent comparing function bodies that were never in the same tree.

If you are already printing __file__ in CI, you do not need another ritual around it. If you are not, start with the script above and keep the output next to the test log, then argue with the agent after you know which file actually ran.