I Sorted One Assertion for 48 Hours. The Cache Key Still Followed Set Order.

I have a bad habit of trusting a laptop that has been running the same shell for weeks. Have you ever watched pytest stay green through a whole afternoon, then fail the moment a clean process starts? I spent forty-eight hours on a cache key that looked deterministic in my terminal and became a different string on every fresh interpreter. The quiet part is that my local environment had already chosen the iteration order for me.

The helper joined a set of feature flags into a string that looked like an identifier. A unit test compared that string to a literal I had copied from one lucky run. Why did that feel safe when the set was only a literal in the test file? I still treat source order like a contract, even though Python never promised one for sets.

What I thought was broken

I started where most of us start, which is the test runner and not the process that actually hashes strings. Was pytest-xdist shuffling cases across workers and leaking a mutated set into later tests? I turned the workers off, deleted .pytest_cache, and reran the file until the laptop fan complained. The helper under test was short enough that I refused to suspect it at all.

def feature_cache_key(flags: set[str], user_id: str) -> str:
    # Looks stable because the set literal is written in one order.
    return f"{user_id}:" + ",".join(flags)

The assertion that kept me busy looked equally innocent on the first several readings.

def test_feature_cache_key_is_stable():
    flags = {"search", "billing", "beta-ui"}
    key = feature_cache_key(flags, "user-42")
    assert key == "user-42:search,billing,beta-ui"

On my laptop that assertion passed so often that I stopped reading the expected string. On a clean interpreter the joined flags arrived in a different order, and I still blamed the runner. Does that sound like a plugin problem, or like a runtime I had tamed by accident?

Hours 0–8: workers, caches, and a very confident laptop

I treated the failure like a parallel-test bug because those fail in the same sometimes-yes pattern. I ran pytest with workers disabled and verbose output until I was thoroughly bored of the green dots. Then I looped the same file with --looponfail while the kettle boiled in the other room. The suite stayed green, which made me more certain about the runner and more wrong about the process.

I printed worker names, fixture scopes, and the id() of the flags set. That is what you print when you would rather not open the language reference. None of those values moved during the reruns, which should have been a clue. The set still looked boring, the string still looked boring, and my shell still carried a seed from an older flake hunt.

Commands from that first night, copied out of the scrollback:

pytest -n0 -vv tests/test_cache_key.py
pytest --looponfail tests/test_cache_key.py
rm -rf .pytest_cache __pycache__
python -c "import os, sys; print(os.getpid(), sys.flags)"

Hours 8–24: I asked a model to make the suite green

This is the part I am not proud of, because the model did exactly what I asked and nothing that I needed. I pasted the failing assertion from a CI log and said make this test pass without rewriting the product. It sorted the expected literal on one side and left the production helper mixed. Have you noticed how quickly a flake disappears when you edit the assertion instead of the contract?

That is a local-only fix dressed up as engineering, and it felt productive because pytest went quiet. The test became a tautology on my machine, and the cache key in application code still followed set iteration order. I had not changed the string that leaves the process. I had only changed the string I was willing to accept.

I wanted a second opinion from a process that did not inherit my shell, so I copied the tiny module onto a clean box. I used MonkeyCode's free model access and free server option for that rerun, because I needed an interpreter that had never loaded my direnv. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The remote session suggested the same assertion patch until I pasted both joined strings side by side and asked which order was the API. Then the useful question finally showed up in the transcript. Who promised that a set would remember the order I typed into the editor?

The twelve-line reproduction I should have run first

Do not take my word for the seed, because this is cheaper to prove than it is to debate. Save this file and run it in two terminals while keeping the rest of the environment equal.

# repro_hash_seed.py
from __future__ import annotations
import sys

def feature_cache_key(flags: set[str], user_id: str) -> str:
    return f"{user_id}:" + ",".join(flags)

if __name__ == "__main__":
    flags = {"search", "billing", "beta-ui"}
    print("seed", sys.hash_info.seed)
    print("key ", feature_cache_key(flags, "user-42"))
    print("hash", {name: hash(name) for name in flags})

Commands that actually moved the story, instead of moving the assertion:

PYTHONHASHSEED=1 python repro_hash_seed.py
PYTHONHASHSEED=2 python repro_hash_seed.py
PYTHONHASHSEED=random python repro_hash_seed.py
PYTHONHASHSEED=0 python repro_hash_seed.py
python -c "import sys; print(sys.hash_info.seed)"

You should see the joined flags rearrange when the seed changes, even though the set literal is written in the same order every time. PYTHONHASHSEED=0 disables hash randomization and makes those strings look stable, which is exactly how a laptop lies to you. Unset or random restores the shuffle that a fresh production process will see. Hash randomization has been the default since Python 3.3, and salted str hashes are why two interpreters can disagree about set order.

Dicts have been insertion-ordered since Python 3.7 as a language guarantee, but sets never got that promise. Building dict.fromkeys(some_set) will inherit the shuffle, and so will ",".join(some_set). Any cache key, filename, or log fingerprint you derive from that join will split across processes that look identical in ps.

I finally asked the laptop whether it was cheating, which is a question I should have asked before blaming xdist.

env | grep -E 'PYTHONHASHSEED|PYTHONPATH|VIRTUAL_ENV'
python -c "import os; print(repr(os.environ.get('PYTHONHASHSEED')))"
test -f .envrc && grep -n HASH .envrc || true
test -f pytest.ini && cat pytest.ini
test -f pyproject.toml && grep -A6 pytest pyproject.toml || true

There it was, sitting in a file I no longer thought about. PYTHONHASHSEED=0 lived in a .envrc I had added during a previous flake hunt and never deleted. My stable suite was a seeded suite, and the clean server was a different process with a different salt.

What I would repeat

I would not start with the test runner next time, and I would not start by asking a model to silence an assertion. I would print the process seed, the joined key, and a sorted key on one line. Then I would run that line under two seeds before pytest gets a vote.

The helper I now keep next to the reproduction is boring on purpose, which is the point of a contract.

def feature_cache_key(flags: set[str], user_id: str) -> str:
    return f"{user_id}:" + ",".join(sorted(flags))

If I need a hashed blob instead of a readable string, I still sort first and then hash the canonical form. Sorting only the assertion, and leaving the helper mixed, is how you hide a production cache split behind a green suite. Would you rather have a quiet laptop or one string per logical input?

Decision table I wish I had on day one

What you are building	Sort first?	Pin PYTHONHASHSEED?	Why
Cache key, filename, idempotency material	Yes	No, except to replay one flake	Callers need one string per logical input
In-memory set membership tests	No	No	in does not care about iteration order
A CI-only assertion you cannot read yet	Maybe	Yes, temporarily, to replay one process	You want the same shuffle twice
Public JSON built from a set	Yes, or dump a list you already sorted	No	Clients will diff the payload
An assertion-only patch from a model	Do not	Do not	You moved the bug into production

Numbered workflow I will actually reuse:

1. Print sys.hash_info.seed and the raw joined value in the failing process.
2. Rerun the same module with PYTHONHASHSEED=1 and PYTHONHASHSEED=2 before touching plugins.
3. Diff laptop env against a clean server env, including direnv and pytest addopts.
4. Canonicalize at the boundary that leaves your process: keys, files, URLs, log fingerprints.
5. Treat assertion-only patches as suspects until the helper and the test describe the same contract.

What broke when I fixed it the wrong way

After the model sorted the expected string, local pytest went quiet for the rest of the afternoon and I almost shipped the diff. Production still built keys from an unsorted set, so two app processes wrote two cache entries for one user. That is no longer a flake in the test runner. That is a split brain that presents as a random cache miss.

I also tried freezing the seed in pytest.ini as a permanent policy, which made the suite deterministic and left production random. Have you seen teams export PYTHONHASHSEED=0 in CI and then wonder why staging still misses? You reproduced one process when you pinned the seed. You did not define a contract that surviving processes can share.

Turning randomization off in production to make sets iterate politely is the wrong trade, and it still does not make ",".join(flags) a documented API. The feature exists so hostile input cannot predict insertion layout. I do not want my cache key design depending on that switch staying off.

Limitations, and who should skip this

This write-up is about set iteration and salted string hashes, not about dict insertion order after Python 3.7. If your key is built from a list you already control, the hash seed is probably not your bug. If your flake involves clocks, DNS, filesystems, or network retries, do not start here. Those failures deserve their own reproductions, and this script will not catch them.

Do not pin PYTHONHASHSEED=0 in production to make sets iterate politely, even if a model suggests it as a one-line fix. That hides data-dependent order bugs and throws away a defense you did not pay for. Do not paste production secrets into a model prompt while you debug, and do not treat a shared box as a vault for .env files.

I am not claiming a benchmark, a quota, a model name, or a hardware profile for the remote rerun. The useful part of that server was the empty environment, not a scoreboard. If you cannot isolate a twelve-line script, a longer chat session will not isolate it either.

Field notes I am keeping

• My laptop is a contaminated runtime after a week of debugging, even when the repo looks clean.
• A green assertion can still describe the wrong contract, especially when a model edited only the test.
• Sets are bags. If a string leaves the process, sort it or otherwise canonicalize it first.
• A clean server is a control, not a personality test for pytest-xdist.
• If a model patches the test and not the helper, I ask it to print both seeds before I accept the diff.

Would I repeat the forty-eight hours? Only the last six, the ones where I ran the same file under two seeds and stopped arguing with the runner. The rest was me negotiating with a shell that had already chosen an order.

If you need a throwaway interpreter that does not inherit your direnv, a free server is a reasonable place to rerun repro_hash_seed.py before you rewrite the suite.