User Site-Packages Passed Every Import. The Clean Server Had None.

Have you ever watched an import succeed on your laptop and then fail on a machine that had never met your home directory? I just spent two days writing field notes around that exact mismatch, and the villain was not a missing wheel. It was Python's user site-packages path, quietly enabled on my laptop and completely absent on a clean Linux interpreter. Why do we keep treating a local import as proof that the package was actually declared?

The problem I actually had

I was packaging a tiny HTTP worker that only needed the standard library plus one JSON schema helper. On the laptop, the import returned immediately, the tests passed, and I felt ready to copy the directory. Then the clean interpreter raised ModuleNotFoundError, even though I had a requirements file I swore was complete. Was the server missing pip, or was I quietly invoking a different minor version of Python?

The worker itself was boring on purpose, so the failure could not hide inside a framework. I ran the same three commands in both places and wrote down the interpreter identity first. That sounds painfully obvious, and I still skipped it for the first hour like an overconfident fool. Field notes only work if you record the executable path before you record your feelings about pip.

python3 -c "import sys, site; print(sys.executable); print(sys.version); print(site.ENABLE_USER_SITE); print(site.getusersitepackages()); print('---'); print(chr(10).join(sys.path))"
python3 -c "import jsonschema; print(jsonschema.__file__)"
python3 -m pip show jsonschema
python3 -m sysconfig | sed -n '1,40p'

Please treat those commands as a reproduction ritual, not as telemetry from a billed outage or a customer cluster. I wanted the same questions answered on two machines that did not share a home directory. If the second machine cannot print jsonschema.__file__, the first machine has been covering for you.

Hours 0–8: what I tried

I copied requirements.txt again and stared at it as if repetition could invent a missing pin. I ran python3 -m pip install -r requirements.txt on the server and watched pip claim satisfaction. Did I mention the file only listed httpx and my own package name, with no schema library at all? The undeclared dependency had come from a leftover pip install --user during a previous experiment on the laptop.

Here is the trimmed requirements file I started with, labeled as a broken fixture rather than a production lockfile. Please treat it as a reproduction input, because I did not pull this from a private backlog. The only honest claim I can make is that this file is enough to recreate the false confidence.

# fixture: requirements.broken.txt — intentionally incomplete
httpx

I should be explicit about the assistant I actually used during that drafting step on the dumps. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the first diagnostic script from those two interpreter dumps. I ran that script on the free server option so the laptop could not cheat with leftover user packages.

Does that mean a coding assistant can replace a careful reading of site.getusersitepackages()? No, and I will not pretend a generated checklist is a substitute for the path dump. It wrote a decent checklist, and the server gave me a path list without ~/.local inside it. My laptop list still included ~/.local/lib/python3.12/site-packages, which is where jsonschema actually lived.

Commands that looked useful and were not

I tried several supposed fixes that only made the notes longer and the environment even dirtier. Each attempt taught me something concrete, even when the import still failed on the clean interpreter. Which of these have you already done at one in the morning with too much confidence?

1. Run python3 -m pip install jsonschema on the laptop again, which wrote into user site and made local imports look even healthier than before.
2. Copy ~/.local/lib/python3.12/site-packages/jsonschema by hand, which is how you invent an unmaintainable snowflake and a second source of truth.
3. Export PYTHONPATH=/usr/lib/python3/dist-packages, which mixed distro packages with my script and then broke optional extras I had not pinned.
4. Create a virtualenv after the fact, forget to reinstall from the file, and then blame the empty venv for being empty.

I have done three of the four, and the copy-by-hand version is the one I still feel slightly ill about. Have you noticed how every failed fix still changes the laptop, so later notes become harder to trust? That is why the clean interpreter matters more than another local install.

Hours 8–24: what actually broke

The break was not a missing compiler, a yanked wheel, or a mysterious DNS problem on the remote box. User site-packages were enabled on the laptop interpreter, so site.ENABLE_USER_SITE printed True and a previous --user install had dropped the module there. The clean server used a stock CPython whose user site was enabled in theory and empty in practice, so the same import name resolved to nothing. Virtual environments disable user site by default, which is why this bug hides for people who always work inside a venv.

I also hit the Debian-style externally-managed-environment error when I tried to install into system Python on the server. That PEP 668 guard was honest, and my laptop Python had never shown it because that interpreter is not owned by the distro the same way. This mismatch is not a pip bug, and it is not a reason to pass --break-system-packages. Did I want a flag that turns a clean server into a second laptop, complete with shadow packages?

error: externally-managed-environment

This environment is externally managed

No. I wanted a venv, a declared requirement, and a test that fails when user site is the only reason an import works. I also listed *.pth files under user site, because those files can extend sys.path without looking like a normal package install. That extra directory was the second lie, sitting beside the undeclared jsonschema folder.

python3 - <<'PY'
from pathlib import Path
import site
user_site = Path(site.getusersitepackages())
print("user_site", user_site, "exists", user_site.exists())
print("ENABLE_USER_SITE", site.ENABLE_USER_SITE)
if user_site.exists():
    for pth in sorted(user_site.glob("*.pth")):
        print("PTH", pth)
        print(pth.read_text(encoding="utf-8", errors="replace"))
PY

The failing check I wish I had at hour one

This script is the original artifact for these notes, and it is meant to run twice. Run it once on the machine you trust, then again on a machine that has never seen your home directory. I am labeling it as a field-notes helper, not as a published benchmark, and not as evidence of production traffic.

#!/usr/bin/env python3
"""site_shadow_check.py — fail if an import only exists in user site."""

from __future__ import annotations

import argparse
import importlib
import site
import sys
from pathlib import Path


def path_is_under(path: str | None, roots: list[Path]) -> bool:
    if not path:
        return False
    resolved = Path(path).resolve()
    return any(root in resolved.parents or resolved == root for root in roots)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("modules", nargs="+")
    parser.add_argument(
        "--allow-user-site",
        action="store_true",
        help="Do not fail when the module file lives under user site-packages.",
    )
    args = parser.parse_args()

    user_roots: list[Path] = []
    for raw in (site.getusersitepackages(), site.getuserbase()):
        if raw:
            user_roots.append(Path(raw).resolve())

    print(f"executable={sys.executable}")
    print(f"version={sys.version.split()[0]}")
    print(f"ENABLE_USER_SITE={site.ENABLE_USER_SITE}")
    print(f"user_site={site.getusersitepackages()}")
    print(f"prefix={sys.prefix}")
    print(f"base_prefix={sys.base_prefix}")
    print(f"in_venv={sys.prefix != sys.base_prefix}")

    failures = 0
    for name in args.modules:
        try:
            module = importlib.import_module(name)
        except ImportError as exc:
            print(f"MISSING {name}: {exc}")
            failures += 1
            continue
        origin = getattr(module, "__file__", None)
        shadowed = path_is_under(origin, user_roots)
        flag = "USER-SITE" if shadowed else "ok"
        print(f"{flag} {name} -> {origin}")
        if shadowed and not args.allow_user_site:
            failures += 1
    return 1 if failures else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it like this, and notice the -s and -I pair sitting beside the default invocation. Isolated mode ignores user site, PYTHONPATH, and PYTHONSTARTUP, which is the closest cheap replica of a stranger's machine. On my laptop the first command printed a USER-SITE line, and the -s run printed MISSING jsonschema.

python3 site_shadow_check.py jsonschema httpx
python3 -s site_shadow_check.py jsonschema httpx
python3 -I site_shadow_check.py jsonschema httpx

That single difference is the whole incident, and it still makes me wince. Why did I not run -s before I copied anything onto the clean interpreter? ENABLE_USER_SITE can be True, False, or None, and I keep forgetting that None means the site module declined to enable it. Isolated mode and -S are not the same switch, so mixing them in notes will confuse future me.

Hours 24–48: what I would repeat

I would repeat the clean-room step before I repeat any install-from-memory step on a laptop that already knows too much. A throwaway Linux interpreter is enough for this, because you are testing import paths rather than training anything. I would also freeze the rule into a unit test so CI cannot share user site with the job by accident. The test below is a seatbelt, not a performance claim.

# test_no_user_site_imports.py
from pathlib import Path
import site

import jsonschema


def test_jsonschema_is_not_served_from_user_site():
    origin = Path(jsonschema.__file__).resolve()
    user_site = Path(site.getusersitepackages()).resolve()
    assert user_site not in origin.parents, f"import resolved under user site: {origin}"

Is that assertion a little blunt for a helper package that might be installed several ways? Yes, and that is fine for field notes that exist to make a hidden path visible. I want the test to fail when someone runs pytest with system Python and a stuffed ~/.local directory. Inside a venv, ENABLE_USER_SITE is typically False, and the assertion becomes quiet, which is what I want during normal work.

Decision table I actually used

I kept this table in the same directory as the script so I would not reinvent the branch next time. It is a decision aid, not a scoreboard, and it does not contain timings.

Observation	Likely cause	What I do next	What I refuse to do
Import works, python3 -s fails	User site or PYTHONPATH	Add the package to requirements, recreate the venv	Copy ~/.local onto a server
Import fails, PEP 668 error	System Python is distro-managed	python3 -m venv .venv, then install from the file	--break-system-packages
Import works only in venv A	The package was never declared	Rebuild venv B from the declared file	pip freeze from a dirty laptop
__file__ points at .../python3/dist-packages	Debian package, not your pin	Decide whether apt or pip owns that name	Mix both and hope the extra wins
ENABLE_USER_SITE is None	Isolated mode or site disabled	Good for diagnosis, risky as a hidden production entrypoint	Ship python -S without reading the flags

Would I still run pip freeze on the laptop and call that a lockfile? Not after this, because freeze captures whatever user site and leftover edits happened to import. A declared file plus a clean venv is slower to type and much harder to lie with. The table exists so I stop arguing with a satisfied pip message that never saw the second machine.

A tiny workflow I am keeping

This is the sequence I would run again tomorrow, in this order, without improvising around a half-copied bin/python. Record identities first, then compare -s, then build a venv on the clean machine. Install only from the declared file, and do not congratulate yourself until the check prints ok for every name you actually import.

1. Record sys.executable, sys.version, and site.ENABLE_USER_SITE on every machine before changing anything.
2. Run site_shadow_check.py normally, then again with python3 -s, and keep both transcripts.
3. Create a fresh venv on the clean server; do not reuse a copied interpreter from the laptop.
4. Install only from a declared file with python -m pip install -r requirements.txt.
5. Run the check inside that venv until every needed import prints ok instead of USER-SITE.
6. Add the pytest guard so user site cannot silently save a future pull request.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip
python -m pip install -r requirements.txt
python site_shadow_check.py jsonschema httpx
python -m pytest test_no_user_site_imports.py

If you need a throwaway Linux interpreter for the same clean-room run, the free server option I already mentioned was enough for these notes. I did not need a specialized GPU box to print sys.path and fail an import on purpose. The value was the empty home directory, not a slogan about models.

Limitations, because this approach is not universal

Do not use this workflow as a security audit, because user site-packages are a dependency hygiene problem rather than a sandbox. I also would not run python -I as the production entrypoint without reading how isolated mode ignores environment variables you may actually need. That includes a PYTHONPATH you might set on purpose in a container image you control.

Skip this entire ritual if you already enforce lockfiles and ephemeral CI images that never mount a home directory. You already have the clean room, and these notes would only add ceremony. This write-up is for the laptop-shaped middle, where pip install --user still happens during debugging and then survives for months. Windows paths differ too, because user site lives under %APPDATA%\Python\PythonXY\site-packages, and I would retest Path.resolve() there instead of assuming Unix layouts.

I did not measure install times, and I will not invent a speedup or a quota. The only result I trust is the boolean question: does python3 -s still import the module after a fresh venv install? If yes, you probably declared it. If no, your laptop has been covering for you, and the clean server is the first reviewer that refused to play along.

What I am taking into the next 48 hours

Will I still forget -s next month when a tiny script looks green on the first try? Probably, which is why the check is a file now and not a memory of feeling careful. The surprise was not that Python has a user site; the docs have said that for years. The surprise was how many of my green local tests were really tests of ~/.local, plus one .pth file I had not looked at since last season.

If a future import fails only on a machine without my home directory, I will dump sys.path before I dump the package manager. That is the whole lesson, and it still fits on a sticky note beside the venv command. A clean interpreter is a harsh reviewer, and that is exactly what I wanted once I stopped arguing with pip's satisfied message.