DEV Community

Taylor Wang
Taylor Wang

Posted on

I Trusted the Remote Green Bar for 48 Hours. pip freeze Told Another Story.

The same pytest node was red on my laptop and green on a remote runner, which felt personal. Have you ever trusted the quieter log just because it did not live on your desk? I spent forty-eight hours hunting a product bug that never existed, and the lockfile never boarded that server. This is the field notebook I wish I had opened at hour zero.

Hours 0–8: I treated the assistant like a second author

I pasted the failing assertion into a coding assistant and asked for a patch, not a diagnosis. The model rewrote a set comparison as a sorted list, and the local test went quiet for one commit. Why did that feel like progress when the remote job had already been green? I now had two passing stories and zero proof they were the same Python.

I copied only the test module onto a second machine and reran the exact node id. The remote session stayed green, and my laptop still carried a dirty site-packages tree I had not printed. I kept lecturing the test file because tests argue back in a smaller font than environments.

Hours 8–24: a free server is not a clone of your laptop

I needed a shell that was not mine, so I used MonkeyCode as a second interpreter. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful part for this bug was free model access plus a free server option, not a slogan about intelligence. I still had to type the same commands I would type on any rented box.

I asked the model to “make the test stable,” and it happily sorted every collection it could see. That hid a real dependency drift, because the remote install had resolved packages from an unlocked pyproject.toml. My laptop was still running last month’s transitive graph, and the test was asserting on iteration order that no API had promised.

Here is the command trail I actually ran, in the order I should have run it on both machines:

python -c "import sys; print(sys.executable); print(sys.version)"
echo "PYTHONHASHSEED=${PYTHONHASHSEED-unset}"
echo "PYTHONPATH=${PYTHONPATH-}"
python -m pip freeze | sha256sum
ls -l uv.lock poetry.lock Pipfile.lock requirements.lock requirements.txt 2>/dev/null
python -m pytest tests/test_labels.py::test_unique_labels -vv
Enter fullscreen mode Exit fullscreen mode

The hashes disagreed before pytest even imported my package. Would you keep rewriting assertions after that printout? I did, for several more hours, because the green remote bar felt like authority.

Hours 24–40: what actually broke

The failing test looked like a product rule about unique labels. It was a set of strings turned into a list, then compared against a golden list from a fixture.

# tests/test_labels.py — the version that wasted a day
def unique_labels(rows: list[dict]) -> list[str]:
    return list({row["label"] for row in rows})


def test_unique_labels():
    rows = [{"label": "beta"}, {"label": "alpha"}, {"label": "beta"}]
    assert unique_labels(rows) == ["beta", "alpha"]
Enter fullscreen mode Exit fullscreen mode

On one interpreter the set hashed beta first. On the other machine alpha won, and PYTHONHASHSEED was unset in both shells. The remote job had also installed a newer packaging and an older HTTP stack because no lockfile was in the upload. I was debugging two variables at once and calling them one bug.

The assistant’s “fix” was this, and it compiled, and it lied:

def unique_labels(rows: list[dict]) -> list[str]:
    return sorted({row["label"] for row in rows})
Enter fullscreen mode Exit fullscreen mode

Sorted output is a product change, not a flake quarantine. Did the API promise alphabetical labels, or first-seen labels? I had not checked the callers. The remote green bar could not answer that question, because it had never seen my lockfile or my intended order.

Hours 40–48: the fingerprint I now run first

I stopped asking the model for patches until both machines printed the same environment document. The artifact is a tiny script that records the interpreter, the hash seed, lockfile digests, and a stable hash of importlib.metadata.

# env_fingerprint.py
from __future__ import annotations

import hashlib
import json
import os
import platform
import sys
from importlib import metadata
from pathlib import Path

LOCK_CANDIDATES = (
    "uv.lock",
    "poetry.lock",
    "Pipfile.lock",
    "requirements.lock",
    "requirements.txt",
)


def _dist_lines() -> list[str]:
    lines: list[str] = []
    for dist in metadata.distributions():
        name = dist.metadata["Name"]
        if not name:
            continue
        lines.append(f"{name.lower()}=={dist.version}")
    return sorted(set(lines))


def fingerprint() -> dict[str, object]:
    dist_lines = _dist_lines()
    blob = "\n".join(dist_lines).encode("utf-8")
    locks = {}
    for name in LOCK_CANDIDATES:
        path = Path(name)
        locks[name] = (
            hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else None
        )
    return {
        "python_version": sys.version.replace("\n", " "),
        "executable": sys.executable,
        "implementation": platform.python_implementation(),
        "platform": platform.platform(),
        "hashseed": os.environ.get("PYTHONHASHSEED", "unset"),
        "pythonpath": os.environ.get("PYTHONPATH", ""),
        "dist_count": len(dist_lines),
        "dists_sha256": hashlib.sha256(blob).hexdigest(),
        "lock_files": locks,
    }


def main() -> None:
    payload = fingerprint()
    Path("env_fingerprint.json").write_text(
        json.dumps(payload, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )
    print(json.dumps(payload, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Commit env_fingerprint.json from a known-good local install after you sync the lockfile. Then fail fast on the remote side before any “smart” test rewrite.

# tests/test_env_fingerprint.py
import json
from pathlib import Path

import env_fingerprint


def test_interpreter_and_dists_match_committed_fingerprint():
    committed = json.loads(Path("env_fingerprint.json").read_text(encoding="utf-8"))
    current = env_fingerprint.fingerprint()
    assert current["dists_sha256"] == committed["dists_sha256"], (
        "site-packages drifted; copy the lockfile and reinstall before editing tests"
    )
    local_py = committed["python_version"].split()[0]
    remote_py = current["python_version"].split()[0]
    assert remote_py == local_py
    assert current["lock_files"] == committed["lock_files"]
Enter fullscreen mode Exit fullscreen mode

Run it like this on both sides, and do not skip the install step that actually reads the lock:

python env_fingerprint.py
python -m pytest tests/test_env_fingerprint.py tests/test_labels.py -vv
Enter fullscreen mode Exit fullscreen mode

If you must pin iteration order for a single module, set the seed in the test command, not in a model-generated sort. I use PYTHONHASHSEED=0 only where the test already leaked set order, and I still fix the assertion to match the documented contract.

Decision table: trust the remote green bar?

Signal on the two machines What I do now What I no longer do
dists_sha256 differs Reinstall from the lockfile, then rerun Ask a model to “stabilize” assertions
Lockfile digest is null remotely Copy uv.lock / poetry.lock first Install from unlocked pyproject.toml
PYTHONHASHSEED is unset and a set leaked Pin seed in tests or assert a set Sort output and call it a product rule
Python X.Y differs Match the interpreter, then re-fingerprint Compare pytest traces as if they were peers
Fingerprints match and the test still splits Then debug the code Blame the runner because it is remote

What I would repeat

  1. Print sys.executable, pip freeze hash, and lockfile names before the first assistant prompt.
  2. Upload the lockfile with the tests, even when the model only asked for the failing module.
  3. Keep PYTHONHASHSEED explicit in the test runner when any test touches a set or dict order.
  4. Treat a model-generated sort as a product change, and read the callers before landing it.
  5. Re-run the fingerprint after every pip install, because that is when the story changes.

Limitations, and who should skip this

This fingerprint does not prove ABI sameness for compiled wheels, and it will not catch a broken sitecustomize that only loads under a login shell. It also treats requirements.txt as a lock, which is only true if that file was compiled and pinned. If your install path uses extra indexes or local wheels, hash those files too, or you will fake agreement.

Skip this workflow if you already have a single CI image that installs from a real lockfile and you never run pytest on a laptop. Skip it if the bug is timing, network, or a GPU driver, because a dist hash will not tell that story. And skip asking a free model to edit tests until the two fingerprints match, or you will spend forty-eight hours polishing the wrong file.

A second machine is still worth having when your laptop has gone native and a little strange. I only trust that remote green bar after both sides print the same dists_sha256; that is the entire reason a free server stays in my loop.

Top comments (0)