Shuffle the Input 200 Times Before You Extract One Module

A legacy credits module returned its rows in a different order after every restart. Tests passed locally. CI failed on byte comparisons. No logic had changed.

You cannot pin a nondeterministic output. You pin a canonical projection of it. You quarantine everything else. Only then do you extract code.

Every snippet below runs as written. Copy four small files, run one sweep, read two numbers.

1. Reproduce the flake in one minute

The order came from iterating a set of SKUs. CPython randomizes string hashing per process. Restart the process and the iteration order changes.

# credits/legacy.py
def assign_credit_notes(lines, note_skus, note_cents):
    """lines: list[dict]; note_skus: set[str]; note_cents: dict[str, int]"""
    out = []
    for sku in note_skus:                      # set iteration order is not stable
        for line in lines:
            if line['cents'] > 0 and line['sku'] == sku:
                out.append({'sku': sku,
                            'cents': min(note_cents[sku], line['cents'])})
                break
    return out

One shared fixture keeps both snapshot scripts honest.

# credits/fixture.py
LINES = [{'sku': 'A-1', 'cents': 1200},
         {'sku': 'B-2', 'cents': 800},
         {'sku': 'C-3', 'cents': 300}]
SKUS = {'A-1', 'B-2', 'C-3'}
CENTS = {'A-1': 500, 'B-2': 900, 'C-3': 100}

Now two snapshots: raw order, and a sorted projection.

# credits/snapshot_raw.py
import json
from credits.fixture import LINES, SKUS, CENTS
from credits.legacy import assign_credit_notes

print(json.dumps(assign_credit_notes(LINES, SKUS, CENTS),
                 separators=(',', ':'), sort_keys=True))

# credits/snapshot.py
import json
from credits.fixture import LINES, SKUS, CENTS
from credits.legacy import assign_credit_notes

def canonical(rows):
    ordered = sorted(rows, key=lambda r: (r['sku'], r['cents']))
    return json.dumps(ordered, separators=(',', ':'), sort_keys=True)

print(canonical(assign_credit_notes(LINES, SKUS, CENTS)))

2. Sweep the hash seed and count unique outputs

Each seed needs a fresh interpreter. That is the whole point of the sweep.

mkdir -p /tmp/sweep/raw /tmp/sweep/canon
for s in $(seq 1 24); do
  PYTHONHASHSEED=$s python -m credits.snapshot_raw > /tmp/sweep/raw/$s.json
  PYTHONHASHSEED=$s python -m credits.snapshot     > /tmp/sweep/canon/$s.json
done

echo "raw   unique: $(sha256sum /tmp/sweep/raw/*.json   | awk '{print $1}' | sort -u | wc -l)"
echo "canon unique: $(sha256sum /tmp/sweep/canon/*.json | awk '{print $1}' | sort -u | wc -l)"

On macOS, swap sha256sum for shasum -a 256. Hash randomization is on by default, and PYTHONHASHSEED=0 disables it.

Read the two numbers as a gate, not as a score.

• raw > 1, canon = 1 — order is unstable, projection is stable. Pin the projection.
• raw = 1 — output is already stable. Pin the bytes directly and skip the rest.
• canon > 1 — you have a real bug. Stop refactoring and fix it first.

That third case is the one people skip. A canonical projection must be a pure function of business inputs. If it is not, the extraction will hide a defect behind a passing test.

3. Turn the projection into a 200-case test

A parametric shuffle test costs nothing and catches accidental dependence on input position.

# tests/test_order_invariance.py
import json, random
import pytest
from credits.fixture import LINES, SKUS, CENTS
from credits.legacy import assign_credit_notes

def canon(rows):
    ordered = sorted(rows, key=lambda r: (r['sku'], r['cents']))
    return json.dumps(ordered, separators=(',', ':'), sort_keys=True)

GOLDEN = canon(assign_credit_notes(LINES, SKUS, CENTS))
GOLDEN_TOTAL = sum(r['cents'] for r in json.loads(GOLDEN))

def shuffled(seq, seed):
    items = list(seq)
    random.Random(seed).shuffle(items)
    return items

@pytest.mark.parametrize('seed', range(200))
def test_line_order_is_irrelevant(seed):
    rows = assign_credit_notes(shuffled(LINES, seed), SKUS, CENTS)
    assert canon(rows) == GOLDEN
    assert sum(r['cents'] for r in rows) == GOLDEN_TOTAL

Shuffling input rows is only safe when duplicate SKUs cannot change which line matches. Check that before you trust the test. If duplicates exist, deduplicate the fixture or annotate the expected first-match rule.

4. Run the sweep where the environment cannot drift

The full run is 48 short-lived processes against one pinned interpreter. Environment drift is the usual reason a local sweep disagrees with CI. That is the job the free server option is meant to absorb: throwaway processes, no local state, same interpreter for every seed. MonkeyCode offers both a free server option and free model access, per the operator.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model access did one narrow job here. It drafted the first version of the canonical serializer and proposed the PYTHONHASHSEED sweep after I described the failing comparison. I rewrote the serializer to use an explicit tuple sort key, because the draft sorted on the string form of the row. A model can suggest a projection. It cannot tell you whether that projection is business-meaningful.

5. Make the smallest safe change

Now extract one function. Nothing else moves in the same commit.

- # credits/legacy.py
- def assign_credit_notes(lines, note_skus, note_cents):
-     ...
+ # credits/notes.py
+ def assign_credit_notes(lines, note_skus, note_cents):
+     ...

Keep call sites untouched by re-exporting from the old path.

# credits/legacy.py
from credits.notes import assign_credit_notes  # noqa: F401

Then re-run both sweeps. The acceptance condition is exact: the canonical count stays 1, and the raw count stays above 1 with an unchanged multiset. If the raw count drops to 1 as a side effect, you changed behavior. Revert and split the change.

6. Gate the extraction with a decision table

Observation	Pinned?	What the test asserts
Raw JSON line order	No, quarantined	Nothing
Canonical JSON	Yes	Exact string equality
Money total in cents	Yes	Exact integer equality
Row count and SKU set	Yes	Exact set equality
Log lines	Partial	Count plus level multiset

The quarantine list matters as much as the pinned list. It documents what the team already knows is unstable, so nobody re-adds a byte assertion next sprint.

7. What this proves, and what it does not

It proves the canonical projection survives a refactor across hash seeds and input permutations. That is all it proves.

Three gaps are worth stating plainly.

1. Order can be contractual. If customers see invoice line order, sorting it away is a bug, not a fix. Fix the ordering instead of quarantining it.
2. Hash seeds are one source only. Threads, wall-clock time, filesystem glob order, locale collation, and database row order all break determinism too. Each needs its own seam.
3. Canonicalization is not free. Sorting a large output on every assertion slows the suite. Cache the projection when the fixture set is big.

8. Who should skip this approach

• Teams whose output order is part of a contract with a downstream system.
• Single-caller scripts where a rewrite is cheaper than a harness.
• Jobs with side effects only, no snapshot-able output. Pin the outbound calls instead of the result.

The method is cheap. The discipline is not. Measure first, quarantine explicitly, then move one function.

If you want a second opinion on which fields belong in your canonical projection, brainstorming that list with free model access is fine. Keep the final sort key and the assertion yours.