48-Hour Field Notes: The Shard Map Looked Stable. hash() Salted Every Process.

Have you ever watched a shard function pick a new bucket after every restart, with no deploy in the logs? I spent two days blaming Redis eviction, a sticky load balancer, and a cache that was not guilty. The user ids in the logs looked identical, and the traffic shape on the workers never really changed. So why did user-42 keep landing on a different partition after each boring process start?

What I thought was broken

I had a tiny routing helper that turned a string user id into one of eight buckets. It lived in a Python worker, and it looked far too boring to fail under review. Locally the same id kept hitting bucket seven, so I treated the helper as a constant. After a restart on another box, that same id showed up in bucket two, then five.

Could the config file be truncated, or was whitespace sneaking into the identifier during parsing? I printed repr(user_id) on every worker, and the logged value stayed exactly 'user-42'. Nothing in the request path looked like a mutation, which made the bucket jump feel supernatural.

The helper that looked harmless

Here is the function I kept staring at, because it contained no network calls at all. You can paste it into a file named shard.py and run it in two fresh processes. If the printed bucket changes, you are not looking at Redis, and you are looking at hash().

# shard.py — illustrative repro you can run locally
def bucket_for(user_id: str, buckets: int = 8) -> int:
    return hash(user_id) % buckets


if __name__ == "__main__":
    print(bucket_for("user-42"))

Run that file twice without exporting PYTHONHASHSEED, and watch the integer instead of the dashboards.

python3 shard.py
python3 shard.py

Do you get the same integer from both processes, or does the bucket jump while the string stays put? On a long-lived local interpreter I sometimes saw a stable number, which is how the bug hid. Start a fresh process, and that number can jump, because hash() salts str values per process.

What I tried for 48 hours

I went down the wrong list first, because the symptom looked like a distributed cache problem.

1. I flushed Redis completely and watched the same user land in a new bucket anyway after restart.
2. I grepped for strip(), lower(), and hidden carriage returns on the user id, and found nothing.
3. I logged id(user_id) like an amateur, which cannot explain a disagreement across separate processes.
4. I compared sys.version on both machines, and they were in the same series with different buckets.
5. I asked a coding assistant why the shard map drifted, and it confidently blamed cache TTLs.

That last detour cost me real time, because a plausible story is worse than a loud crash. Have you noticed how easy it is to accept an infrastructure narrative when the function looks pure?

The moment the salt showed up

The useful check was not another dashboard, and it was not a packet capture on the workers. It was a pair of commands, each launched in a brand new interpreter with a clean environment.

python3 -c "print(hash('user-42'))"
python3 -c "print(hash('user-42'))"

Then I pinned the seed, and the printed number finally stopped twitching between process starts.

PYTHONHASHSEED=0 python3 -c "print(hash('user-42')); print(hash('user-42') % 8)"
PYTHONHASHSEED=0 python3 shard.py
PYTHONHASHSEED=1 python3 shard.py

PYTHONHASHSEED=0 disables randomization, while a numeric seed makes the salt reproducible for a debug session. Neither setting is a sharding strategy I want sitting in a production systemd unit file. Python randomizes hash() for str, bytes, and datetime as a defense against hash-flooding attacks. If your partition function rides on that integer, every new interpreter is a brand new map.

This is still current behavior, not a dusty 3.3 trivia item you can ignore on a 2026 worker image. The interpreter documents PYTHONHASHSEED as an environment switch for that salt, not as a public routing API. I wanted a second Linux process that was not my laptop's long-running shell, because that shell had been lying.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to review the helper, then ran the script on the free server option. The model did what models do with incomplete logs: it told a clean story about Redis eviction. The free server started a new Python process, printed a different hash('user-42'), and ended the debate.

I did not need a named model or a hardware story; I needed two fresh interpreters and one integer. If you remove that product from this write-up, the debugging method still holds on any two processes. Pin PYTHONHASHSEED only while you confirm the diagnosis, then stop using hash() as a router.

The artifact I actually wanted

I replaced the shard function with an explicit digest, and that is the version I would actually keep.

# stable_shard.py — run this in two processes and compare
import hashlib


def bucket_for(user_id: str, buckets: int = 8) -> int:
    digest = hashlib.sha256(user_id.encode("utf-8")).digest()
    n = int.from_bytes(digest[:8], "big")
    return n % buckets


if __name__ == "__main__":
    print(bucket_for("user-42"))

Run it under different seeds, and the bucket should stay put while the environment variable changes.

PYTHONHASHSEED=0 python3 stable_shard.py
PYTHONHASHSEED=1 python3 stable_shard.py
python3 stable_shard.py

Decision table I wish I had on hour one

Need	Use	Avoid
In-process dict or set membership	Built-in hash()	Depending on the numeric value across processes
Stable shard or cache key across processes	hashlib.sha256 / blake2b, or a specified keyed digest	hash(str) % n
Reproducing a bug that involves hash()	PYTHONHASHSEED=<int> for that run only	Exporting PYTHONHASHSEED=0 on every host
Security-sensitive token or password	A real MAC or KDF	hash(), or md5 as a password hash
Fast non-crypto partition of bytes you control	blake2b with an explicit key= and digest_size=	A random per-process salt you do not own

blake2b with a key you own is a reasonable keyed partition if you document how that key is stored. hash() is not keyed by you, because the interpreter owns the salt and will not advertise it. Redis never sees that salt, which is why the keyspace looked haunted after a routine restart.

A test that actually crosses processes

A same-process unit test will never catch this, because hash() is stable inside one interpreter. You need a child process with a different PYTHONHASHSEED, which is the boundary the production workers already cross. The script below is a pytest-style check you can save next to stable_shard.py.

# test_stable_shard.py — run with: python3 -m pytest test_stable_shard.py
import os
import subprocess
import sys

HELPER = r"from stable_shard import bucket_for; print(bucket_for('user-42'))"


def test_bucket_independent_of_hashseed():
    values = []
    for seed in ("0", "1", "42"):
        env = os.environ.copy()
        env["PYTHONHASHSEED"] = seed
        out = subprocess.check_output(
            [sys.executable, "-c", HELPER],
            env=env,
            text=True,
        )
        values.append(out.strip())
    assert len(set(values)) == 1

If that assertion fails, your shard function is still reading the per-process salt, and you should keep looking. A passing test does not prove even distribution, and it does not prove cryptographic strength. It only proves the bucket no longer follows the interpreter salt, which is the bug I actually had.

What broke, in one list

• hash() on strings stays stable inside one process, and it is not a portable identifier across process starts.
• A long-lived local interpreter hid the jump, while fresh workers on another box revealed it immediately.
• Restarting Redis reset keys, which rhymed with "the map changed," so I followed the rhyme instead of the integer.
• An assistant filled the log gap with cache folklore instead of asking me to print hash('user-42') twice.
• Pinning PYTHONHASHSEED=0 everywhere would freeze shards and weaken hash-flooding protection at the same time.

What I would repeat

Print the integer first, and do not theorize about the cluster until two processes disagree in front of you. Ask whether the value is even supposed to be portable across process boundaries and language runtimes.

python3 - <<'PY'
import os, sys
print("seed", os.environ.get("PYTHONHASHSEED", "<unset>"))
print("hash", hash("user-42"))
print("bucket", hash("user-42") % 8)
print("version", sys.version.split()[0])
PY

If two processes disagree, stop looking at the network and look at the salt on str hashes. I would also keep a one-file repro like stable_shard.py in the ticket, because reviewers can run it. Would I ask a model again? Yes, but I would paste the two printed hashes before any infrastructure theory. Models guess at caches and load balancers; they rarely guess PYTHONHASHSEED from a narrative alone.

Limitations, and who should skip this

This write-up is about partition functions and cache keys derived from hash(str), not about Redis internals. It is also not a review of consistent hashing rings, rendezvous hashing, or Python's dict insertion order. Dict order has been insertion order for years, so do not confuse that guarantee with a stable hash() integer.

Do not disable hash randomization globally just to make a shard function look deterministic in production. That trade weakens protection against hash-flooding, and it treats a security feature as a configuration nuisance. Do not use hashlib.md5 as a password hash, and do not copy PYTHONHASHSEED=0 into every host image. Do not assume hash(7) is randomized, because integers are not in the same category as strings.

If you already shard with a documented digest, you do not need this workaround, and you can stop here. If you need cryptographic authentication of the identifier, you need a MAC, not a bucket index. If your mapping only lives inside one process and never crosses the network, built-in hash() is fine. The forty-eight hour version of this bug is almost always the same shape, hiding behind a pure-looking function.