48-Hour Field Notes: The Laptop Stayed Green. A Clean Host Shuffled the Set.

I kept treating my laptop as the source of truth, and that habit is what burned the first evening. The suite was green in the terminal I always reuse, yet the same file on a clean host printed tokens in another order. Was the parser suddenly wrong, or had my shell pinned a world the other machine never inherited? These notes are the 48-hour loop I want future me to repeat whenever a local green run and a fresh box disagree.

I am not reporting a private outage, a benchmark, or a quota I measured on someone else's infrastructure. I am walking a labeled reproduction: an assertion on set order is not a product requirement, and a shell profile can hide that fact for days. If your real failure is a race, a clock jump, or a missing file, this loop will not save you. Would you still spend one clean run on the seed before you rewrite the parser?

What looked guilty

The function split a short string, dropped duplicates with a set, and returned whatever iteration order that process happened to use. My local check compared that list with a string I had copied from an earlier print. It matched every time I pressed the up arrow, which felt like evidence and was only familiarity. Have you ever promoted a printout into an expected value because it sat still for an afternoon?

CPython randomizes the hash seed for str and bytes unless you pin PYTHONHASHSEED, and that seed changes set iteration from process to process. Dict insertion order is a different contract, and it does not make a set stable, even when the dict beside it looks calm. The primary note I want next to this claim is the environment-variable section for the interpreter you actually run: PYTHONHASHSEED. Read that page against your build before you quote it in a ticket, because a vendor Python can still surprise you.

A debugging shell often exports a fixed seed so yesterday's session will replay. A clean machine does not inherit that export, so the same file and the same input can lead with a different token. That is not a mystery parser, and it is not proof that the host is cursed. It is two processes, two seeds, and one frozen string that was copied from only one of them.

The loop I would actually repeat

I split the two days into four blocks so I would not "fix" the code before I had two comparable runs. Each block has a question, a command, and a stop rule. If I cannot answer the question, I do not open a refactor, no matter how tidy the patch looks.

Hours 0 to 8: freeze the command

I write the invocation on one line and stop editing the fixture. The question is whether a second person can run it without my terminal history. I dump version, seed, timezone, and locale before I touch any logic. If those four lines differ, I am not debugging a function yet, and another clever assertion will only paint over the mismatch.

Hours 8 to 24: make the seed the suspect

I run the same file twice and change only PYTHONHASHSEED. If the signatures diverge, the assertion was environment-shaped, and the copied expected string is the bug. If they stay identical, I drop this theory and look at locale, timezone, or a different Python build. Why burn a day inserting a sort if two seeds already agree on the output?

Hours 24 to 40: leave the laptop profile behind

A clean server matters because it does not load my rc file, my warm virtualenv, or a copied .env. I copy the single repro file and nothing else, then I run the frozen command before I form a new opinion. I do not copy secrets, and I do not pretend the image matches production. The question is whether the failure survives a host that has never heard of my pin.

This is the first step where a hosted toolchain actually belongs in the method. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this loop I would use MonkeyCode's free server option as that clean room, and later its free model access as a draft reader for the diff. Those are availability claims supplied for this draft, not measurements of hardware, duration, model name, or quota. If either option is absent on the account you have, a spare virtual machine and a teammate still fit the same checks.

Hours 40 to 48: draft, then rerun

I paste the two stdout blocks and ask for the smallest contract change, not a new framework. A useful draft names the unstable assumption and proposes an explicit order. A noisy draft adds a retry, a sleep, or a third argument nobody asked for, and that noise goes in the trash. I keep only the line I can justify, then I rerun both seeds on the clean host. A suggestion that has not been rerun is still a suggestion, isn't it?

The artifact you can run tonight

The file below is a labeled example, not a log pulled from a production incident. Unstable mode prints set order as the process produced it. Stable mode sorts before joining, which is the contract I would keep. Do not hard-code a signature you happened to see under seed 1, because that repeats the original mistake on a new host.

# repro_order.py — labeled example, not a production reporter
import os
import sys

def signature(raw: str, stable: bool) -> str:
    tokens = set(raw.split())
    ordered = sorted(tokens) if stable else list(tokens)
    return ",".join(ordered)

def main() -> int:
    stable = os.environ.get("STABLE", "0") == "1"
    got = signature("beta alpha beta", stable)
    print(
        f"py={sys.version.split()[0]} "
        f"seed={os.environ.get('PYTHONHASHSEED', 'unset')} "
        f"stable={int(stable)} sig={got}"
    )
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

# Two pinned seeds. Differing sig lines mean the check was order-shaped.
PYTHONHASHSEED=1 python repro_order.py
PYTHONHASHSEED=2 python repro_order.py

# Explicit contract: both seeds should now print the same signature.
PYTHONHASHSEED=1 STABLE=1 python repro_order.py
PYTHONHASHSEED=2 STABLE=1 python repro_order.py

# Dump the world you ran, then stop arguing from memory.
python -c "import os,sys; print(sys.version); print('PYTHONHASHSEED', os.environ.get('PYTHONHASHSEED')); print('TZ', os.environ.get('TZ')); print('LANG', os.environ.get('LANG'))"

I would commit the unstable printer beside the sorted helper for one day. Deleting the failure too fast turns a ten-line repro into folklore. Would you rather reread a command, or reread a paragraph that only says the suite felt flaky?

A decision table, not a vibe

What you observed	What I would trust	What I would not do
Local run matches a copied print, clean host does not	The env dump, especially PYTHONHASHSEED	Rewrite the parser from a hunch
Two pinned seeds disagree, sorted runs agree	An explicit order contract in the test	Add a retry, a sleep, or a new dependency
Two pinned seeds agree, hosts still disagree	Locale, timezone, or a different Python build	Blame the set and stop looking
Draft patch adds an argument nobody requested	The rerun on the clean host	Merge from the chat transcript
Failure needs private rows to show up	A redacted five-word fixture	Upload the raw log or a token

What I would repeat, and what I would drop

1. Write the command in the ticket before I write a theory in the ticket.
2. Dump version, seed, timezone, and locale on every host I mention out loud.
3. Run two pinned seeds before I accept a local green check as evidence.
4. Copy only the repro to a host that does not load my shell profile.
5. Ask for the smallest diff, then delete every line I cannot explain.
6. Rerun that original command on the clean host before I call the note done.

I would drop the habit of pasting a green local screenshot into the ticket as if it were a second machine. I would also drop any patch that "stabilizes" the run by catching the mismatch and continuing. A swallowed failure is not a contract, and it will wait until the next host to embarrass you. If the sorted runs still diverge, I stop this theory instead of stacking more flags on it.

Limits, and who should walk away

This loop does not prove production safety, and a clean room is not a clone of your cluster. A free server will not share your production glibc, your patch level, or your network policy, so a pass there is only a pass there. A free model can propose a sort that fits this fixture and still miss the next input shape. I am not stating that those options remain free, which model name appears, or how much capacity they include. Check the product surface you actually have before you plan a weekend around it.

Skip this approach if the bug needs an identical production image, a licensed dataset, or a private network path. Skip it if policy forbids third-party hosts or prompts that leave your laptop. Skip it when the flake is threads, timers, or filesystem latency, because a hash seed will look innocent and you will leave with the wrong lesson. Skip pasting tokens, customer rows, or internal hostnames into either the server or the model. The five-word fixture above was enough to expose the contract bug.

If you already have a MonkeyCode login, the free server is a reasonable shelf for this repro, and the free model is only the colleague who suggests the next command you still have to rerun yourself.