I Trusted pip show for 48 Hours. The Job Ran Another Interpreter.

Have you ever stared at a successful pip install and still watched ModuleNotFoundError on the next command? I have, and the mismatch ate a reconstructed 48-hour lab before I treated interpreters as first-class evidence. This notebook is not a production postmortem with invented uptime numbers or customer counts. I am walking the commands I would run again on a throwaway Linux box, labeled as a lab from the first hour.

The Question I Should Have Asked First

Was the installer talking to the same Python that later executed the scheduled job? That sounds obvious when you write it on a whiteboard in calm daylight. It is not obvious when an assistant, a login shell, and a unit file each resolve python3 through a different PATH. I assumed one binary because the hostname was one hostname. The machine never shared that assumption, and it did not owe me a warning.

Hour 0–7: Comfort Commands That Agreed With Each Other

I reproduced a tiny scheduled runner that imported a helper, wrote a summary file, and exited zero when healthy. Locally the import worked, so I copied the same pip line onto the remote shell and watched it print success. Then the scheduled job raised ModuleNotFoundError: No module named 'orjson' after the installer had already said the wheel was in place. Why would pip lie to a person who had just watched it succeed?

It was not lying. It was describing a different interpreter than the process that later ran the job. Here is the first batch I ran, labeled as a lab transcript rather than a one-liner fix.

# lab transcript — confirmation theater, not a diagnosis
python3 -m pip show orjson
python3 -c "import orjson; print(orjson.__file__)"
command -v python3
command -v pip
type python3
ls -l "$(command -v python3)"

Those commands agreed with each other, which made the next morning worse instead of better. pip show found the dist-info directory, and the interactive import printed a file under /usr/local. So I blamed the scheduler, then the lockfile, then the package index. I still had not printed sys.executable from inside the job itself, and that omission is the entire notebook.

Hour 8–18: PATH Stopped Being a Single Answer

The login shell and the scheduled environment were never the same process, so why did I treat their python3 names as synonyms? One inherited my interactive profile with user-local bins prepended. The other started from a unit file that cleaned the environment and then pinned a system path. I dumped env and still missed the binary, because several names on PATH were not the same inode.

I started comparing identities instead of comparing the strings my fingers like to type.

# still a lab transcript
readlink -f "$(command -v python3)"
python3 -c "import sys; print(sys.executable); print(sys.prefix); print(sys.base_prefix)"
python3 -m pip --version
pip --version
head -n 1 ./run_job.py

The shebang said #!/usr/bin/env python3. The unit file said ExecStart=/usr/bin/python3 /opt/job/run_job.py. Those two lines are not synonyms on a box that also has a user-local interpreter. env python3 followed my login PATH, while /usr/bin/python3 did not follow it at all. Which of those files had actually received the wheel I kept celebrating?

Hour 19–27: pip Was Honest About the Wrong Target

A bare pip executable is a separate program, and it can bind to an interpreter you never intended to run. I had been reading pip show as if it were a property of the job. It was a property of whichever pip binary landed first on PATH. Once I forced the module form against each candidate, the story split into two install layouts that could not see each other.

/usr/bin/python3 -m pip show orjson
/usr/bin/python3 -c "import orjson"
"$HOME/.local/bin/python3" -m pip show orjson

The user-local interpreter had the package, and the unit file's interpreter did not. PEP 668 made the plot noisier on Debian-family images, because the system Python refused an unmanaged install and I had "fixed" that by installing somewhere else. The assistant's free-form pip install orjson never named an interpreter, which is exactly how this class of bug is born.

I replayed the layout on a disposable host so I could break PATH on purpose without touching a real workload. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and the free server option as a scratch interpreter layout and a second reviewer of the commands, not as an authority on which binary a scheduler would start.

The model, like me at hour 11, kept saying the package was installed because a shell import succeeded. That sentence can be true and still be useless to the process that actually runs at 02:00.

The Artifact: An In-Process Interpreter Audit

Names on PATH are rumors, and the running process is the only evidence I now trust. I print identity from inside the job, and I keep a tiny auditor that I point at whichever binary I am about to believe.

#!/usr/bin/env python3
"""interpreter_audit.py

Label: reproducible lab helper. Pass the binary you believe the job uses:
    /usr/bin/python3 interpreter_audit.py orjson
"""
from __future__ import annotations

import os
import site
import sys
from importlib import metadata


def lines() -> None:
    print(f"pid:        {os.getpid()}")
    print(f"executable: {sys.executable}")
    print(f"real exe:   {os.path.realpath(sys.executable)}")
    print(f"version:    {sys.version.split()[0]}")
    print(f"prefix:     {sys.prefix}")
    print(f"base:       {sys.base_prefix}")
    print(f"in_venv:    {sys.prefix != sys.base_prefix}")
    print("sys.path:")
    for entry in sys.path:
        print(f"  - {entry!r}")
    try:
        print("site-packages:")
        for entry in site.getsitepackages():
            print(f"  - {entry}")
    except Exception as exc:  # embedded or stripped builds may omit this layout
        print(f"site-packages: <unavailable: {exc}>")


def dist_info(name: str) -> None:
    try:
        dist = metadata.distribution(name)
    except metadata.PackageNotFoundError:
        print(f"{name}: NOT installed for this interpreter")
        return
    print(f"{name} version: {dist.version}")
    print(f"{name} locate:  {dist.locate_file('')}")


if __name__ == "__main__":
    pkg = sys.argv[1] if len(sys.argv) > 1 else "orjson"
    lines()
    dist_info(pkg)

Run that helper with every candidate binary, not with the one your fingers prefer after coffee.

/usr/bin/python3 interpreter_audit.py orjson
/usr/bin/env python3 interpreter_audit.py orjson
# if the job is already running on Linux:
pid=$(pgrep -n -f run_job.py)
readlink -f "/proc/${pid}/exe"
tr '\0' '\n' < "/proc/${pid}/cmdline"

If /proc/<pid>/exe and command -v python3 disagree, stop arguing with pip about metadata. You are already looking at two runtimes that will never share a site-packages directory.

Decision Table I Wish I Had at Hour 4

Observation	What it usually means	Next move
pip show finds the package, job import fails	pip is not python -m pip for the job binary	Re-run show with /path/to/job-python -m pip
Interactive import works, timer or unit fails	scheduler PATH or ExecStart is another inode	Compare readlink /proc/<pid>/exe
System pip refuses the install	PEP 668 externally managed environment	Create a venv owned by the job, then python -m pip
Console script runs, python -c import fails	script shebang points at a third interpreter	head -1 $(command -v the-cli)
venv exists, job still misses the dist	unit file never used venv/bin/python	Point ExecStart at the venv binary

Notice that none of those rows mention the package index, the CDN, or a flaky wheel. The index was innocent the entire time I was yelling at it.

Hour 28–41: What Actually Broke When I "Fixed" It

I pointed the unit file at a virtualenv and felt briefly clever, which should have been a warning. The venv's pyvenv.cfg still named a base interpreter I had replaced during image cleanup, so activation looked healthy while imports resolved against a ghost. Recreating the environment with an explicit binary was the unglamorous fix, and it was also the only fix that survived a reboot.

/usr/bin/python3 -m venv /opt/job/.venv
/opt/job/.venv/bin/python -m pip install -U pip
/opt/job/.venv/bin/python -m pip install orjson
# ExecStart=/opt/job/.venv/bin/python /opt/job/run_job.py
/opt/job/.venv/bin/python interpreter_audit.py orjson

Would I trust a chat suggestion that says "just pip install it again" without an executable line from the job's own stdout? I would not, because assistants optimize for a shell that already works. Schedulers do not give you that shell, and they will not apologize for the PATH you thought you exported.

I also logged identity from the job, because audits you run by hand will always use the comfortable binary.

# inside run_job.py — tiny breadcrumb, not a framework
import json
import sys

print(
    json.dumps({
        "executable": sys.executable,
        "version": sys.version.split()[0],
    }),
    flush=True,
)

What I Would Repeat

• Treat python3, pip, and sys.executable as three suspects until they resolve to the same real path.
• Install only with that_python -m pip, never with a bare pip on a shared PATH.
• Point process managers at a venv binary, not at /usr/bin/env python3.
• Print interpreter identity from inside the running job, not from the debugging shell.
• When an agent claims a dependency is present, ask which executable it actually probed.

What I Would Not Repeat

I would not "confirm" an install from an interactive notebook kernel and then ship that confirmation to a service account. I would not copy a user-local ~/.local/bin tree onto a daemon user and call the result packaging. I would not let a model rewrite a shebang unless the new path is an absolute venv binary I just created and audited.

Limitations, and Who Should Skip This

This notebook is about interpreter identity, not about ABI mismatches, manylinux tags, or compiled extension crashes after a successful import. If import orjson works and then the process dies in native code, you need a different lab with ldd and wheel tags. The live /proc/<pid>/exe check assumes Linux; the in-process sys.executable print still works on other systems, and that print is the part I would keep everywhere.

Skip this approach if you already ship a locked image with one Python and user site-packages disabled. You do not need a 48-hour PATH autopsy on a single-binary container that cannot drift. Skip it if you cannot recreate the venv from a lockfile, because pinning by folklore only moves the skew to the next host. Skip any workflow that treats a free scratch server as production, since a disposable box exists so you can break PATH on purpose.

Did the job print its own executable, or did you only ask pip in a friendlier shell? If you skipped that print, every successful pip show was theater.