I Trusted python3 for 48 Hours. The Process Never Entered the venv.

Have you ever trusted a python3 shebang because the command existed on both machines? I did that, and the two interpreters were not even distant cousins on disk. One lived inside a virtualenv that my shell had activated hours earlier without ceremony. The other was distro Python, marked externally managed, and quietly uninterested in my requirements file.

This write-up is a 48-hour field notebook, not a victory lap or a benchmark chart. I am recording what I tried, what broke, and what I would run again tomorrow. If you generate scripts with a free model and execute them on a free server, pin interpreter identity first.

Hour 0–6: The symptom looked like a bad patch

The generated script imported httpx and printed a short JSON summary to standard output. Locally that import worked, and the summary looked sane enough for me to trust. On the remote box, the same file raised ModuleNotFoundError, then later imported a system requests I never requested. Why would the same file disagree with itself across two otherwise similar login shells?

I blamed the model first, because that is the cheap story whenever outputs diverge. Was the patch incomplete, or did I copy the wrong buffer into the remote file? I hashed the script on both sides, and the hashes matched with no leftover newline drama. The file was not the bug, because the process that executed the file was the bug.

Here is the first command I should have run, and the one I postponed until evening:

python3 -c "import sys; print(sys.executable); print(sys.prefix); print(sys.base_prefix); print(sys.version)"
which python3
type python3
python3 -m pip -V
pip -V || true

Locally, sys.prefix and sys.base_prefix differed, which is the quiet signature of a virtualenv. Remotely they were identical, and sys.executable pointed at /usr/bin/python3. Same command name, different worlds, and a pip binary that did not belong to that interpreter. Have you noticed how often those two prefixes disagree only on the machine that feels healthy?

Hour 6–18: pip said no, and I heard server broken

The model, being helpful in the usual way, emitted a bare install one-liner. You know that line. It looks innocent until a modern distro Python answers with PEP 668.

error: externally-managed-environment

Did I read the error as a closed door with a sign on it? Not really, not at first. I tried pip install --user, then sudo pip install, then pip3 instead of pip. Every variant either failed closed or wrote into a user site that the shebang would never import. Have you noticed how many generated READMEs still lead with bare pip install against whatever python3 happens to be?

PEP 668 exists so distro Python stays owned by the operating system package manager. Debian and Ubuntu ship an EXTERNALLY-MANAGED marker for that reason, and the marker is not a broken scratch server. I finally printed the path instead of negotiating with pip like a stubborn tourist.

python3 -c "import sys, pathlib; p = pathlib.Path(sys.prefix) / 'EXTERNALLY-MANAGED'; print(p, p.exists())"

When that file exists, I no longer bargain. I create a virtualenv with that same interpreter, and I never call bare pip again. I call python -m pip from the venv executable, every time, including upgrades of pip itself.

Hour 18–30: the venv existed, the job never entered it

This is the part that still makes me wince when I reread the scrollback. I created .venv. I installed the requirements into it. Then I ran python3 app.py from a fresh SSH session that had never sourced activate. Guess which Python answered the import? The distro one, with a clean sys.path and no httpx in sight.

Would you bet a night on source .venv/bin/activate being present in every future runner? I will not, not for cron, not for CI, and not for a pasted SSH command. Activation is a shell story. The only path I trust is the absolute interpreter, spelled out so a new session cannot silently fall back.

./.venv/bin/python -c "import sys; print(sys.executable)"
./.venv/bin/python -m pip install -r requirements.txt
./.venv/bin/python app.py

If a runner cannot see that executable, the job should fail before it imports anything interesting. Silent fallback to /usr/bin/python3 is how you donate the next morning to a fake model-quality debate.

Hour 30–48: I stopped comparing app output first

Once both machines printed a JSON contract, the argument got smaller and much less mystical. I was no longer asking whether the draft had drifted. I was asking whether executable, in_venv, and the requirements hash agreed. They did not, and that was enough.

The template below is a preflight I keep beside the app; treat it as a runnable checklist, not as a claimed production incident log. Run it with the same interpreter you will use for the job, or the report is theater.

preflight.py

#!/usr/bin/env python3
"""preflight.py — fail closed before a generated script wastes a day."""
from __future__ import annotations

import hashlib
import json
import os
import pathlib
import site
import sys


def sha256_file(path: pathlib.Path) -> str | None:
    if not path.is_file():
        return None
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()[:16]


def in_venv() -> bool:
    return sys.prefix != sys.base_prefix or hasattr(sys, "real_prefix")


def externally_managed() -> bool:
    marker = pathlib.Path(sys.prefix) / "EXTERNALLY-MANAGED"
    return marker.is_file()


def main() -> int:
    root = pathlib.Path.cwd()
    req = root / "requirements.txt"
    venv_python = root / ".venv" / "bin" / "python"
    running_venv = False
    if venv_python.is_file():
        running_venv = os.path.realpath(sys.executable) == os.path.realpath(
            venv_python
        )

    report = {
        "cwd": str(root),
        "executable": sys.executable,
        "prefix": sys.prefix,
        "base_prefix": sys.base_prefix,
        "version": sys.version.split()[0],
        "in_venv": in_venv(),
        "externally_managed": externally_managed(),
        "user_site": site.getusersitepackages(),
        "requirements_sha256_16": sha256_file(req),
        "venv_python_exists": venv_python.is_file(),
        "running_venv_python": running_venv,
        "PATH_head": os.environ.get("PATH", "").split(os.pathsep)[:3],
    }
    print(json.dumps(report, indent=2))

    errors: list[str] = []
    if not in_venv():
        errors.append("not running inside a virtualenv")
    if externally_managed() and not in_venv():
        errors.append("distro Python is externally managed; refuse bare pip")
    if not req.is_file():
        errors.append("requirements.txt missing")
    if venv_python.is_file() and not running_venv:
        errors.append("a .venv exists but this process is not that interpreter")

    if errors:
        print("PREFLIGHT_FAIL", file=sys.stderr)
        for item in errors:
            print(f"- {item}", file=sys.stderr)
        return 2
    print("PREFLIGHT_OK", file=sys.stderr)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

run.sh

I also keep a runner that does not depend on a previously activated shell. Copy it, then change nothing about the Python selection rules until preflight prints PREFLIGHT_OK.

#!/usr/bin/env bash
# run.sh — create or reuse .venv, then fail closed on preflight.
set -euo pipefail
cd "$(dirname "$0")"
PY="${PYTHON:-python3}"
if [[ ! -x .venv/bin/python ]]; then
  "$PY" -m venv .venv
fi
./.venv/bin/python -m pip install -U pip
./.venv/bin/python -m pip install -r requirements.txt
./.venv/bin/python preflight.py
./.venv/bin/python app.py "$@"

Is this glamorous? No, and it should not be. Does it catch the class of bugs where pip succeeded and the job still imported another site-packages? Yes, before you start comparing generated summaries like they were the root cause.

How I read the JSON

• in_venv false on the remote box, true on the laptop: stop, do not compare app output.
• externally_managed true and in_venv false: do not call bare pip; make a venv.
• venv_python_exists true and running_venv_python false: you created a venv and then ignored it.
• requirements_sha256_16 mismatch: you are not even arguing about the same lock-in file.
• PATH_head starting with a user bin you forgot: env python3 may not be the binary you think.

Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I still let a free model draft the first version of app.py and the requirements list, because that draft is cheap and reversible. I do not let the draft choose the interpreter, the pip binary, or the working directory. MonkeyCode's free model access is useful for that draft, and the free server option is useful as the second machine that never inherited my laptop's activated venv. If you already work in that setup, run the preflight on the free server before you argue with the draft.

I paste the preflight JSON from laptop and server into the same note as the prompt transcript. If executable, in_venv, or requirements_sha256_16 disagree in a way I cannot explain, I stop. Have you seen how fast a wrong sys.path impersonates a bad model and a flaky host at the same time?

Decision table I keep above the keyboard

Observation	What I used to do	What I do now
ModuleNotFoundError after a successful pip	Re-run the model and tweak imports	Print sys.executable and sys.path
externally-managed-environment	sudo pip or --break-system-packages	python3 -m venv .venv, then ./.venv/bin/python -m pip
Script works in this SSH session only	Export more PATH entries	Stop activating; call .venv/bin/python
Hashes of app.py match, behavior does not	Blame temperature and retry the prompt	Diff the two preflight JSON documents
pip and python3 report different prefixes	Install again to be sure	python3 -m pip -V and refuse mixed tools

What broke, in order

• Bare pip install against distro Python, blocked by PEP 668, which I treated as a host outage.
• A --user install that the shebang interpreter never imported, because user site was not on sys.path.
• A .venv I created and then ignored by typing python3 in a new SSH session.
• A generated shebang of #!/usr/bin/env python3 that followed PATH, not the venv.
• Comparing application output before comparing interpreter identity, which made the model look guilty.

What I would repeat

1. Hash the script and the requirements file on both machines before talking about model quality.
2. Create the venv with the intended python3, then refuse every other binary for that job.
3. Run preflight.py with that binary until it prints PREFLIGHT_OK on laptop and remote.
4. Only then run the generated app, and only with .venv/bin/python plus set -euo pipefail.
5. Keep the two preflight JSON files next to any model transcript, because that pair is the receipt.

Would I skip the preflight because the laptop already looks fine in my current shell? Not after this notebook. Local success is a hint, not a receipt, and a receipt is the only thing I want to diff.

Limitations, and who should not use this

This workflow assumes you can create a virtualenv on the remote box and write a .venv directory. If the server forbids python -m venv, or the filesystem is read-only outside a temp dir, stop and use an image you control. The preflight also assumes a Unix-style .venv/bin/python path, so Windows runners need .venv/Scripts/python.exe instead of the POSIX layout.

It does not pin CPU, GPU, or model quality, and it does not prove the generated code is correct. It only proves which interpreter ran, whether PEP 668 applies, and whether the process entered the venv you think it entered. If you need production isolation, use a container or a locked image, not a scratch venv on a shared box you do not own.

Do not use this approach if you are installing into a distro Python on purpose, or if you must use system packages exclusively. Do not use --break-system-packages to silence PEP 668 on a machine you did not provision. And do not treat a free remote box as a secret store; keep credentials off that filesystem and out of the prompt.

Closing the notebook

I used to debug prompts when the process was the real diff, and that was the expensive habit. The cheap habit is a JSON preflight and an explicit interpreter path, run on both machines before I argue with a model. Two environments, one script, one contract: that is the whole field note I still trust.