My Migration Generator Shuffled Itself for Two Days. PYTHONHASHSEED Was the Coin Flip.

Two consecutive runs of the same command emitted the same SQL statements in a different order, and I did not believe my own terminal. I re-ran it four times, redirected each run into a file, and compared them the way you check whether the fridge light really goes off. The statement set was identical, the sequence was not, and the applied-version check that consumed it failed on roughly every other run.

These are my field notes from those two days: the hypotheses I burned, the reproduction I should have written in the first hour, and the seed matrix I now run before I accuse anyone's framework of being flaky.

The evidence, before any theory

I forced myself to write down only what I could observe, because my first three explanations were stories rather than measurements.

• python -m myapp.migrate --print produced 41 statements on every run, with byte-identical content and different ordering.
• Diffing run A against run B showed the same lines moved around, never added or removed.
• The ordering changed between processes, not between calls inside one process.
• Re-running inside a single long-lived REPL session never reproduced it, which is the detail that eventually cracked the case.

That last bullet is the one I kept skimming past, and it was the whole answer. Anything that varies per process but stays fixed within a process is not a scheduling bug, a race, or a merge issue.

Suspects I burned hours on

Here is the honest list, with the reason each one was wrong.

1. pytest-xdist worker merge order. Plausible, because the generator ran under a parallel test harness. It was wrong: the reorder reproduced under a plain interpreter with no pytest involved.
2. Filesystem glob order. I rewrote every glob into sorted(glob...) out of superstition. os.scandir order was stable on both machines, and the fix changed nothing.
3. SQLAlchemy metadata ordering. I assumed the mapper was reshuffling my dependency graph. Dumping the graph before render showed a stable list and an unstable render.
4. A stale __pycache__ or a leftover .pyc. I nuked caches, reinstalled the package in editable mode, and recreated the virtualenv. The shuffle survived all of it.
5. An unset locale or collation difference. LC_ALL and collation were identical on both boxes, which I confirmed with locale and with sort --version before moving on.

Five confident explanations, zero of them true. That is the tax you pay for debugging by intuition.

A 25-line reproduction I should have written on day one

The bug lived in a dependency resolver that walked a set of module names and popped them one at a time. Set iteration order for strings depends on the string hashes, and those hashes are salted per process by default.

# repro_seed.py — runnable on CPython 3.8+
import json
import os

NAMES = ["core", "billing", "auth", "reports", "audit", "search"]


def dependency_order(names):
    # BUG: set iteration order follows the salted hash table layout,
    # so pop() hands back a different module on every fresh process.
    pending = set(names)
    out = []
    while pending:
        out.append(pending.pop())
    return out


if __name__ == "__main__":
    print(json.dumps({
        "seed": os.environ.get("PYTHONHASHSEED", "<unset>"),
        "order": dependency_order(NAMES),
    }, indent=2))

Run it twice with different seeds and watch the order move:

PYTHONHASHSEED=1 python repro_seed.py
PYTHONHASHSEED=2 python repro_seed.py

The pending.pop() call is the villain. A set gives you fast membership tests and no ordering contract whatsoever, which is precisely why it is such a pleasant place for an ordering bug to hide.

Why my laptop stayed quiet and the clean server screamed

Salted hashing of str, bytes, and datetime keys has been the default since Python 3.3, so a fresh interpreter already has a randomized seed. The CPython command line documentation covers PYTHONHASHSEED, the -R switch, and the accepted random value if you want to read the exact contract.

My laptop never reproduced it because I had PYTHONHASHSEED=0 exported in my shell profile, a leftover from an old profiling session I had long forgotten about. Setting os.environ["PYTHONHASHSEED"] = "0" inside the script does not help either, since the seed is read during interpreter startup and my assignment lands far too late. I learned that the embarrassing way, eight hours in, after writing a fix that did literally nothing.

I also reached for two things from MonkeyCode here, and I want to be clear about which parts were load-bearing. The free server option gave me a disposable Linux box with an unpinned interpreter, which is exactly what I needed to see the shuffle while my laptop insisted everything was fine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used the free model access to draft the first version of the seed matrix below, and its initial answer confidently blamed pytest-xdist — the same wrong guess I had made. A model is a decent rubber duck for generating candidate hypotheses and a poor authority for confirming them, so every claim in this post was re-checked against the interpreter and the docs.

The seed matrix: six seeds and one hash comparison

Stability is a property you can assert, not a vibe you can feel. This script renders the same artifact under several seeds and fails when the outputs disagree.

#!/usr/bin/env bash
# seed-matrix.sh — fail if generated output depends on the interpreter seed
set -euo pipefail

OUT_DIR=$(mktemp -d)
trap 'rm -rf "$OUT_DIR"' EXIT

for seed in 0 1 2 3 4 5 random; do
  PYTHONHASHSEED="$seed" python -m myapp.migrate --print > "$OUT_DIR/$seed.sql"
done

# Note: use `shasum -a 256` instead of sha256sum on macOS.
UNIQUE=$(sha256sum "$OUT_DIR"/*.sql | awk '{print $1}' | sort -u | wc -l)
if [ "$UNIQUE" -ne 1 ]; then
  echo "FAIL: output is seed-dependent"
  diff "$OUT_DIR/0.sql" "$OUT_DIR/random.sql" || true
  exit 1
fi
echo "OK: output is seed-stable across 7 seeds"

Wire that into CI as a separate job so a failure names the real problem instead of surfacing as a mysterious flake. The diff output is the part that actually teaches you something, because it shows which collection leaked its ordering into your artifact.

The fix is boring, and that is the point

I sorted at the boundary where iteration order stopped being an implementation detail and became part of the output contract.

def dependency_order(names):
    # Deterministic: sort before iterating, never trust set order.
    return sorted(set(names))

If real topological constraints exist, use graphlib.TopologicalSorter and break ties explicitly with a sorted() heap rather than hoping the set cooperates. A regression test pins the contract so a future refactor cannot quietly reintroduce the ambiguity:

def test_dependency_order_is_stable():
    assert dependency_order(NAMES) == [
        "audit", "auth", "billing", "core", "reports", "search",
    ]

Where else seed sensitivity hides

• Any hash set of strings used to serialize config, snapshots, or lockfiles.
• Cache keys built by iterating a set of tags, where two logically identical inputs produce two entries.
• Pretty-printed JSON of a set-derived list, which makes snapshot tests flap in CI and pass locally.
• Deduplication loops that emit "first seen" records, because "first" was never defined.

Integers are a different story, since small ints hash to themselves and a set of them tends to iterate identically for the same insertion history. That asymmetry is why a mixed test suite can look half-stable and lead you toward entirely the wrong theory.

Symptom	Seed-sensitive?	Cheapest check
Output differs across fresh processes only	Likely, if sets of str exist	Two runs with PYTHONHASHSEED=0 and =1
Order changes inside one process	No	Look for threads, async tasks, or mutation
Only integer keys involved	Rarely	Re-run the matrix once to confirm
Flaps in CI, green on your laptop	Very often	Diff local against CI output verbatim

Who should skip this approach

If your output is a genuinely unordered payload consumed by a client that never inspects order, adding a seed matrix is ceremony. If you already sort every serialization boundary, you have effectively solved this and can spend the hour elsewhere. And if the instability comes from real concurrency rather than hashing, the matrix will pass while your race stays hidden, which is arguably worse than no check at all.

Field notes: what I would repeat

• Write the minimal reproduction before the third hypothesis, not after the fifth.
• Distrust any bug that survives a fresh process but dies inside a REPL session.
• Run important generators on a clean box, because your shell profile is part of the environment.
• Assert determinism with a script, then delete the theories.

Two days for a one-line sorted() call stings, but the seed matrix is now the first thing I run when an artifact misbehaves across machines. If you want a throwaway environment to run it in without touching your laptop's pinned interpreter, the MonkeyCode free server option is the one I reached for, and it earned its place in this workflow.