48-Hour Field Notes: I Diffed the Env Allowlist Before I Trusted the Patch

I spent the last forty-eight hours on an import that passed in my laptop shell and failed under the service account. The traceback named a missing module, but the same three lines imported when I pasted them into an interactive session. Was the package actually absent, or was my shell quietly lending a path the service account never received? I had already chased encoding and locale ghosts on earlier jobs, so I refused to reopen that drawer this week.

What I thought the failure was

I first assumed the lockfile had dropped a dependency, because that is the story most import errors want you to believe. The suggested patch added an install line, a short retry, and a broader try block that swallowed the original exception. Would you merge a wider try just because the traceback got shorter and the shell looked calm? I would not, and I did not, because a quieter error is not the same thing as a located module.

The interactive check passed only after my shell had exported PYTHONPATH for an unrelated checkout earlier that morning. A user-site wheel also sat on the laptop, which meant pip show looked healthy while the service unit never saw that wheel. Have you watched an install suggestion fight a path bug that no lockfile change can honestly fix? I had that exact argument with myself before lunch on the first day of this note.

Hour 0 to 12: I captured the wrong kind of same

I copied the command, the working directory, and the interpreter version, then told myself the two runs were the same run. They were not the same run, because the service unit started from a near-empty environment and my terminal inherited a long startup file. Did matching the interpreter version convince you that the process environment matched on both sides too? It convinced me for about an hour, which was long enough to almost apply the install line.

I stopped reading the patch and started listing keys instead, because the import machinery consults variables before it consults your prose. I cared about those path variables, plus the user-site switches, and whether a virtualenv was actually active. Anything else was allowed to differ across the two machines without explaining this particular import miss. If a key could not change module lookup, why was I letting that key block the rest of the review?

What broke on the clean replay

I built a tiny fixture so I could fail on purpose without touching a real service account. A one-file package lived under a temporary checkout, and the sample script imported it with no install step at all. The loud command printed a checkout marker, and the scrubbed command raised ModuleNotFoundError in the same second. Why would an install line be the honest fix when the module was never meant to be installed on that box?

A synthetic failure you can rerun

Here is the fixture I would re-run before I trust the next import patch from any assistant. Treat the temporary paths as a synthetic local reproduction, not as a log pulled from a production incident. I am marking the commands as a reproducible example you should execute locally, rather than as a borrowed production log. If your loud shell cannot fail the scrubbed run, you do not have this bug, and you should stop here.

mkdir -p /tmp/loud/sibling_pkg
printf 'VALUE = "from-checkout"\n' > /tmp/loud/sibling_pkg/__init__.py
printf 'import sibling_pkg\nprint(sibling_pkg.VALUE)\n' > /tmp/app.py

PYTHONPATH=/tmp/loud python3 /tmp/app.py
env -u PYTHONPATH -u PYTHONHOME -u PYTHONUSERBASE \
  PYTHONNOUSERSITE=1 python3 /tmp/app.py

The first command succeeds only because the shell injects a checkout that the service account will never see. The second command is the service-shaped run, and it fails even when your laptop site-packages look complete. A suggested exception handler followed by a network install would hide that split behind a new side effect. Do you really want a service unit installing packages during startup because a laptop variable leaked into the prompt?

The allowlist I would run again

I did not want raw values in the manifest, because environment dumps wander into tickets, chats, and paste buffers. The script below records presence and path-entry counts for a blocking set, and it never prints the values themselves. Exit status is non-zero when blocking presence drifts, and zero when the two sides agree on those keys. Would you paste a full environment into a model just to learn that one variable was set?

#!/usr/bin/env python3
"""Compare Python-relevant env presence. Values are never printed."""
from __future__ import annotations

import json
import os
import sys
from pathlib import Path

BLOCKING = (
    "PYTHONPATH",
    "PYTHONHOME",
    "PYTHONSAFEPATH",
    "PYTHONNOUSERSITE",
    "PYTHONUSERBASE",
    "PIP_USER",
    "VIRTUAL_ENV",
)

def classify(key: str, env: dict[str, str]) -> str:
    if key not in env:
        return "unset"
    value = env[key]
    if value == "":
        return "empty"
    if key in {"PYTHONPATH", "PYTHONHOME", "PYTHONUSERBASE", "VIRTUAL_ENV"}:
        return f"set:{len(value.split(os.pathsep))}entries"
    return "set"

def manifest(env: dict[str, str] | None = None) -> dict[str, str]:
    source = os.environ if env is None else env
    found = {key: classify(key, source) for key in BLOCKING}
    found["safe_path"] = str(bool(getattr(sys.flags, "safe_path", False)))
    return found

def main(argv: list[str]) -> int:
    if len(argv) == 1:
        print(json.dumps(manifest(), indent=2, sort_keys=True))
        return 0
    if len(argv) != 3:
        print("usage: env_allowlist.py [left.json right.json]", file=sys.stderr)
        return 2
    left = json.loads(Path(argv[1]).read_text(encoding="utf-8"))
    right = json.loads(Path(argv[2]).read_text(encoding="utf-8"))
    keys = sorted(set(left) | set(right))
    drift = [key for key in keys if left.get(key) != right.get(key)]
    for key in drift:
        print(f"drift {key}: {left.get(key)!r} != {right.get(key)!r}")
    return 1 if drift else 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

python3 env_allowlist.py > laptop.json
env -u PYTHONPATH -u PYTHONHOME -u PYTHONUSERBASE \
  PYTHONNOUSERSITE=1 python3 env_allowlist.py > scrubbed.json
python3 env_allowlist.py laptop.json scrubbed.json
echo "exit=$?"

On a loud shell the manifest should report PYTHONPATH as set, while the scrubbed file should report that key unset. The scrubbed side may also show PYTHONNOUSERSITE as set, and that extra drift is a control, not a new bug. Either drift is enough to reject an install patch when the code never declared a real dependency at all. Would a green interpreter banner have shown you that split without the manifest sitting in front of the diff?

Where the free replay seat fit

I still wanted a second opinion that was not my own startup file talking back to me in a circle. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used the free model access to list blocking keys I might forget, then kept only keys I could explain. I used the free server option as a clean replay seat, and I refused to paste a full environment dump into the chat.

The model proposed a startup install, which is the same patch I had already refused after the loud shell failed the scrub. Seeing that idea arrive from a cleaner seat did not make the idea safer, and the allowlist drift stayed decisive. I am not treating free model access or the free server as a quota, a hardware match, or a permanent lab. Those details were not verified for this note, so a free seat is a convenience rather than a twin of production.

What I would repeat next time

These are the steps I would repeat on the next import miss, and I would not skip the first one because I felt rushed. A fast merge is how the loud shell wins, and the service account is the one that pages you later. Ask yourself whether the patch still makes sense after the extra variables are gone from the process. If it does not, the patch was describing your laptop, not the program you thought you were fixing.

1. Dump the presence manifest before I read any suggested diff, even when the traceback already looks obvious to me.
2. Re-run the import under a scrubbed env with user site disabled before I add any dependency line.
3. If a clean replay seat is already available, repeat that scrubbed command there and keep every secret out of the prompt.
4. Accept a patch only after the scrubbed run fails for the claimed reason and a second scrubbed run shows the fix.
5. Prefer python3 -P on interpreters that honor a safe path, so the current directory cannot impersonate an installed library.

How I decide before applying

What I see	What I do	What I refuse
Blocking key set on the laptop only	Scrub and replay before any edit	An install line written from the loud shell
Both sides unset, and the import still fails	Treat it as a real missing dependency	A bare exception handler that hides the name
User site or PIP_USER set on one side	Replay with PYTHONNOUSERSITE=1	Copying a wheel path into the unit file
Drift only in unrelated notes	Record it, and do not block this bug class	Declaring the two environments identical
Patch needs the network during startup	Reject it unless the unit already installs on boot	Letting a model fix a path bug with pip

One more command I keep beside the manifest

I still ask the interpreter whether user site is enabled, because a key can be unset while a path file is already loaded. This one-liner is a companion check, not a replacement for the allowlist diff you just ran. If user site is enabled only on the laptop, I scrub again before I believe any install suggestion. Would you trust a wheel that only the interactive session can see from that extra site directory?

python3 -c 'import site, sys; print("user_site", site.ENABLE_USER_SITE); print("safe_path", sys.flags.safe_path)'

Limits, and who should skip this

This check will not save you from a bad wheel tag, a different libc, or a path configuration file you forgot to inventory. Startup customization modules can still change imports without setting any key that this allowlist knows how to name. A clean replay seat is not a production twin, and I would not cite it as evidence about latency, capacity, or permanence. If those are the questions you have, you need the real runtime, not a convenient replay seat that happened to be free.

Skip the whole approach when the bug is a data race, a lockfile hash mismatch, or anything that never consults the process environment. If you already ship a hermetic image, replay inside that image instead of adding another machine to an already long story. If the unit file holds production credentials, do not upload an environment dump to any model, whether the seat is free or paid. A redacted presence manifest is the most I want in a ticket, and it stays out of a prompt unless I need the drift.

If a free server seat is already on your account, park one scrubbed replay there before you merge the import fix. If that seat is not available, the scrubbed run on your own machine teaches the same habit without borrowing a second box. Either way, I would rather reject a calm patch than explain a startup installer to the person who owns the service account. Would you bet a deploy on a shell that was louder than the process you are actually shipping this week?