The Agent Called the Suite Deterministic. Dict Order Had Never Been Frozen.

Have you ever let an agent rewrite a Python test until it passed once on your laptop? I did that recently, and the remote failure looked like flaky data instead of an unfrozen runtime. The patch was tiny, the assertion read cleanly, and my local pytest output stayed green. So why did the same file stumble as soon as another process started?

This writeup is a 48-hour field notebook, not a launch post or a benchmark claim. I will record what I tried, what broke, and the receipt I now print before trusting agent-authored tests. If you strip every product name out of this article, the debugging method should still be useful. That is the bar I am holding myself to here.

What actually failed

The symptom was embarrassingly small, which is why I misread it for most of a day. A helper loaded a JSON object, called .items(), and glued the pairs into one golden string. On my laptop the order matched the comment the agent had written in the test. On the other host, every key and value was present, but the concatenation had rotated.

Was this a race in the application code, or a stale cache in the runner? I spent the first evening on those stories because the chat log had called the test idempotent. It was not idempotent. CPython still randomizes hash seeds unless you freeze them, and naive datetime fixtures still inherit the host timezone.

The three knobs I had never printed

I finally wrote the missing prints by hand, and they were not exotic at all:

1. PYTHONHASHSEED — dict and set iteration order across processes.
2. TZ / time.tzname — any fixture that calls datetime.now() without a timezone.
3. LC_ALL / locale.getlocale() — parsing, sorting, and a few CSV edge paths.

If you skip those three, an agent can keep editing the golden string until it matches one machine. That is a snapshot of a single process, not a test of your code. Would you merge a snapshot just because the chat UI drew a green check?

The 48-hour timeline

I am writing the hours down so I can repeat the useful parts and skip the rest next time.

Hours 0–6: I trusted the agent's summary

The agent said the suite was deterministic after one green local run. I asked it to explain the assertion, and it simply repeated the golden string. I did not ask which interpreter, which working directory, or which environment variables were in scope. Repeating pytest on the same laptop twice more taught me nothing, because a biased sample is not reproduction.

Hours 6–18: I chased the wrong layer

I compared lockfiles. I compared git diffs. I even recopied the test file byte for byte onto the other host. Those checks matter for other bugs, but they were the wrong layer here. The source was identical, and the process identity was not. Matching sys.version made me feel clever for ten minutes, until I noticed the hash seed was still unset on one side.

Hours 18–36: I finally froze the process

I added a tiny receipt module that every test session has to print first. It is boring on purpose. If a remote job cannot print this block, I do not read the rest of the log, and I do not let an agent propose another patch. Freezing PYTHONHASHSEED=0 and TZ=UTC turned the "flake" into a stable failure, which is the only kind of failure I can fix honestly.

Hours 36–48: I changed the assertion, not the environment story

With the seed frozen, the golden string was simply the wrong kind of assertion. I updated the test to compare mappings, not concatenated items() output. The agent could have done that first, if I had refused to accept a pass without a receipt. That is the habit I am keeping.

A receipt you can actually run

Treat the following as an in-repo workflow, not as a published benchmark. Copy it, run it, and keep the output next to any agent patch. I am labeling it as a method I now keep locally, not as a claim about someone else's infrastructure.

# receipt.py
"""Print a runtime receipt before tests. This is not a security boundary."""
from __future__ import annotations

import hashlib
import locale
import os
import platform
import sys
import time
from pathlib import Path


def _short_hash(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]


def build_receipt() -> dict[str, str]:
    lock = Path("requirements.lock")
    lock_hash = "missing"
    if lock.exists():
        lock_hash = _short_hash(lock.read_text(encoding="utf-8"))

    return {
        "python_executable": sys.executable,
        "python_version": sys.version.split()[0],
        "platform": platform.platform(),
        "cwd": str(Path.cwd().resolve()),
        "hashseed": os.environ.get("PYTHONHASHSEED", "unset"),
        "tz": os.environ.get("TZ", "unset"),
        "tzname": ", ".join(name for name in time.tzname if name),
        "lc_all": os.environ.get("LC_ALL", "unset"),
        "locale": str(locale.getlocale()),
        "lock_hash": lock_hash,
        "sys_path_head": " | ".join(sys.path[:5]),
    }


def print_receipt() -> None:
    print("=== runtime receipt ===")
    for key, value in build_receipt().items():
        print(f"{key}: {value}")
    print("=== end receipt ===")


if __name__ == "__main__":
    print_receipt()

Wrap pytest so the receipt is mandatory. I am deliberately not putting this in conftest.py, because I want the print even when pytest is not the first command in the job.

#!/usr/bin/env bash
# run_tests.sh — proposed wrapper, not a production orchestrator
set -euo pipefail
export PYTHONHASHSEED="${PYTHONHASHSEED:-0}"
export TZ="${TZ:-UTC}"
export LC_ALL="${LC_ALL:-C.UTF-8}"
python receipt.py
pytest -q "$@"

Add a guard so CI-like runs cannot silently forget the freeze. Skip it on a laptop if you must, but do not skip it on the second host.

# test_receipt_guard.py
import os
import pytest


@pytest.mark.skipif(os.environ.get("CI") != "1", reason="guard is for CI-like runs")
def test_hash_seed_is_frozen_when_ci_is_set() -> None:
    assert os.environ.get("PYTHONHASHSEED") == "0"


@pytest.mark.skipif(os.environ.get("CI") != "1", reason="guard is for CI-like runs")
def test_timezone_is_utc_when_ci_is_set() -> None:
    assert os.environ.get("TZ") == "UTC"

Local freeze, then the same command on the other process:

export CI=1
export PYTHONHASHSEED=0
export TZ=UTC
export LC_ALL=C.UTF-8
chmod +x run_tests.sh
./run_tests.sh tests/test_receipt_guard.py

If two receipts disagree, do not let an agent "fix" the assertion yet. Align the process first, then decide whether the test is wrong. That ordering saved me the second day.

Decision table I now keep above the keyboard

Receipt field disagrees	Do this first	Do not do this yet
hashseed unset versus 0	rerun both sides with PYTHONHASHSEED=0	rewrite golden strings
TZ or tzname	set TZ=UTC and use timezone-aware datetimes	patch datetime.now only in the test
locale	set LC_ALL and compare structured data	assert printed sentences
cwd	print Path.cwd() and the test file path	trust relative imports from the chat
lock_hash	reinstall from the same lock file	ask the model to make it pass
python_executable	refuse the log and start over	keep comparing pytest summaries

What I would repeat, and what I would not

I would repeat the receipt before any agent patch, every single time. I would repeat comparing mappings instead of glued strings. I would repeat refusing a log that cannot print PYTHONHASHSEED. I would not repeat merging because a chat transcript said the suite was deterministic.

After I had a receipt, I still needed a second process that did not inherit my laptop timezone. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is relevant when you want a first draft of the receipt helper, and the free server option is relevant when you want that second process; if you are already comparing two hosts, that pair is one place to park throwaway reruns.

I am not attaching model names, token budgets, hardware, or uptime claims I cannot verify from primary docs sitting in front of you. Availability can change, so read the current product page before you plan a pipeline around it. The receipt still matters if you use a completely different runner tomorrow.

Limitations, and who should not use this

This receipt is a flashlight, not a lock. It does not prove the remote host is clean, and it does not stop a process from mutating env vars after startup. It also does not replace a real CI system with artifact storage, because a printed block in a scrollback buffer can still be lost.

Do not put secrets, production credentials, or customer data on a shared free server just to get a second process. Do not treat a free model draft as a reviewed patch. Do not freeze PYTHONHASHSEED=0 in a long-lived production web process if you were relying on hash randomization as a minor hardening measure; this freeze is for tests.

Skip this approach if you cannot keep a lockfile, if your tests must assert rendered prose in a specific locale, or if you need a compliance boundary rather than a debugging habit. An agent that cannot see the receipt will still invent a calm story. Your job is to make the process visible before you believe it.