I Trusted hash() for 48 Hours. Two Workers Never Shared a Cache Key.

Have you ever watched two worker processes miss the same in-memory cache and blamed Redis anyway? I did that during this stretch, then I blamed the serializer, then I blamed a load balancer health check. The miss rate looked random, which made every dashboard feel like evidence instead of noise. Forty-eight hours later those keys were never going to match, because I treated Python's built-in hash() as a stable digest.

The problem I walked into

I had a small request coalescer sitting in front of a slow metadata call that workers loved to stampede. Each process hashed a query dict and stored the inflight future under that integer key. One process on my laptop looked perfectly fine, which is the worst kind of green dashboard. Under gunicorn with four workers, every process recomputed the same payload and none of them shared work.

Why would a hash of the same string disagree across processes that I had just started? I assumed a race. I assumed dict iteration order had bitten me again after a recent scare. I even assumed the reverse proxy was retrying with a mutated query string. None of those theories survived a print of the full key.

Field notes from the forty-eight hours

Hours 0–8: I chased the wrong layer

I dumped Redis MONITOR output until the terminal felt haunted and slightly personal. Every GET was a miss, which I treated as proof that the cache client was lying. I bounced the replica, lowered the TTL, and added a prefix like a person rearranging furniture during a power outage. The miss graph did not care about any of that activity.

Then I logged the cache keys themselves instead of logging my feelings about Redis. That is when the integer keys started looking unhinged in a very consistent way. Same payload, different numbers, every worker, every restart. Have you stared at two supposedly identical keys that were not even close?

What I tried during that first block:

• Restarted Redis and flushed a logical database I should not have flushed
• Pinned gunicorn to one worker, which made the bug vanish and wasted a morning
• Printed id(payload) as if object identity could explain a cross-process miss

What actually broke in that block:

• Single-worker mode hid the disagreement, so I called it a concurrency bug
• Redis was a spectator, because the coalescer never left process memory
• Debug logs truncated the integers, so I thought the values were almost equal

Hours 8–24: I blamed dict order, then JSON, then UTF-8

I still had a scar from json.dumps and insertion order, so I walked straight back into that room. I sorted keys, switched to separators=(",", ":"), and encoded to UTF-8 before hashing. The integers still jumped between processes like they had never met. That should have been the clue, and I still looked at the network path instead.

Commands I actually ran, in two separate interpreter sessions on purpose:

python -c 'print(hash("user:42:meta"))'
python -c 'print(hash("user:42:meta"))'

Two sessions produced two different numbers on the same laptop with no Redis in sight. No gunicorn, no proxy, no clever dict, just Python doing what it has done since hash randomization became the default. I laughed, then I felt a little sick, then I checked whether my shell had pinned a seed I had forgotten. How did I ship a cache key that cannot survive fork and cannot survive a second process?

Python documents this in the PYTHONHASHSEED note and in the data model for __hash__. The interpreter salts string and bytes hashes per process unless you override that salt. I had read that page years ago and then written production keys as if it were trivia.

Hours 24–40: I tried to "fix" PYTHONHASHSEED

Setting PYTHONHASHSEED=0 made the numbers stable, and it felt like unbuckling a seatbelt because the warning light was annoying. I still needed a machine that was not my shell history, not direnv, and not a leftover export from 2 a.m. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I reran the same script on MonkeyCode's free server option and used the free model access only to review the coalescer diff; the useful part was a stubborn refusal to treat hash() as a digest.

A clean Linux shell matters here because local debugging exports lie with a straight face. If your laptop already has PYTHONHASHSEED=0 in a profile file, every local run will look fixed while gunicorn workers disagree. I wanted a box that had not inherited my panic. That was the whole reason to leave the laptop, not a benchmark and not a shopping trip.

Hours 40–48: I replaced the key function

I switched the coalescer to hashlib.sha256 over canonical JSON and I stopped asking Redis for forgiveness. I kept hash() only for in-process set membership that never hits disk, never hits Redis, and never crosses a worker boundary. Then I added a regression check that starts two subprocesses and compares the keys they print. The check is ugly. The check is also the first honest test this coalescer had.

The reproducible artifact

Save this as hash_key_repro.py and run it as a script. You want two processes, not two lines that share one already-seeded interpreter. Pasting both calls into the same REPL will hide the bug the same way one gunicorn worker hid it.

# hash_key_repro.py
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from typing import Any


def builtin_hash_key(payload: dict[str, Any]) -> str:
    # Looks cheap. Dies across processes.
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return str(hash(blob))


def stable_digest_key(payload: dict[str, Any]) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()


def worker(kind: str) -> None:
    payload = {"user": 42, "fields": ["name", "email"], "flag": True}
    key = builtin_hash_key(payload) if kind == "builtin" else stable_digest_key(payload)
    seed = os.environ.get("PYTHONHASHSEED", "<unset>")
    print(f"{kind} pid={os.getpid()} seed={seed} key={key}")


def spawn_pair(kind: str) -> list[str]:
    lines: list[str] = []
    for _ in range(2):
        completed = subprocess.run(
            [sys.executable, __file__, "--worker", kind],
            check=True,
            capture_output=True,
            text=True,
        )
        lines.append(completed.stdout.strip())
    return lines


if __name__ == "__main__":
    if len(sys.argv) == 3 and sys.argv[1] == "--worker":
        worker(sys.argv[2])
        raise SystemExit(0)

    print("== builtin hash() across two processes ==")
    builtin_lines = spawn_pair("builtin")
    for line in builtin_lines:
        print(line)

    print("== sha256 across two processes ==")
    digest_lines = spawn_pair("digest")
    for line in digest_lines:
        print(line)

    builtin_keys = [line.rsplit("key=", 1)[1] for line in builtin_lines]
    digest_keys = [line.rsplit("key=", 1)[1] for line in digest_lines]
    print("builtin_keys_match=" + str(builtin_keys[0] == builtin_keys[1]))
    print("digest_keys_match=" + str(digest_keys[0] == digest_keys[1]))

Run it under an unset seed, then under a pinned seed, so you can see which knob actually matters:

python hash_key_repro.py
PYTHONHASHSEED=0 python hash_key_repro.py
PYTHONHASHSEED=random python hash_key_repro.py

Expected shape, not a specific integer you should copy into a wiki:

• hash() keys disagree across the two child processes unless the seed is forced
• sha256 keys match in every process and every seed this script can reach
• Pinning PYTHONHASHSEED=0 hides the bug the same way "run one worker" hid it
• A lucky match under a random seed is coincidence, not a design, so rerun before celebrating

I am not pasting my laptop's integers here because they are not a fact about your runtime. The disagreement is the fact, and the docs above are the reason. If both builtin keys match while PYTHONHASHSEED is unset, rerun before you invent a new theory.

A decision table I wish I had taped to the monitor

Need	Use	Do not use
In-process set / dict keys that never leave this interpreter	hash() is fine; Python already uses it	A hex digest, unless you are debugging
Cache keys shared by workers, containers, or languages	hashlib.sha256 over canonical bytes, or another specified digest with an explicit seed	hash(), id(), or repr() of a dict
Idempotency keys persisted in Redis or a database	Canonical bytes, then a real digest	hash() even with PYTHONHASHSEED=0
Security-sensitive comparison	hmac.compare_digest on a MAC	hash() or string == on a secret
"Make it stable on my laptop" under time pressure	Fix the key function	Export PYTHONHASHSEED=0 in production

Numbered checks I now run before I blame infrastructure:

1. Print the exact cache key, not a slice, and not a log line that casts to int.
2. Spawn two subprocesses; do not compare two calls inside one REPL session.
3. Run once with PYTHONHASHSEED unset, once with it pinned, and compare the keys.
4. If the key must travel, hash canonical bytes, never a Python hash code.
5. If a single worker "fixes" it, treat that as a clue about process isolation, not a victory.

What I would repeat, and what I would not

I would repeat the two-process repro before I open Redis MONITOR again. I would repeat a clean-server run when my shell is already polluted with debug exports. I would repeat the decision table in review, because the next reader will also think hash() means "hash function" in the cryptographic sense. Would you have caught this from a metric that only showed miss rate?

I would not repeat PYTHONHASHSEED=0 as a deploy flag, because that turns a local convenience into a process-wide footgun. I would not repeat blaming gunicorn worker count without printing keys. I would not repeat asking an assistant to optimize the cache before I can say whether the key must be stable off-box. The model review helped only after the repro existed.

Limitations, and who should skip this

This write-up is about process-local hash() randomization, not about Redis eviction, HTTP validators, or filesystem hashing. hashlib.sha256 is not free in CPU terms, so if you coalesce millions of tiny keys per second, measure that function on your workload. Canonical JSON still needs a policy for floats, NaN, and key types that json.dumps cannot encode. I am not claiming a universal speedup, and I am not claiming a production incident beyond this coalescer.

Do not use this approach if:

• You need a password hash or a password KDF; that is hashlib.scrypt or Argon2 territory, not sha256 of JSON
• You already persisted hash() integers and cannot migrate those keys
• You are debugging a language that does not salt hash() per process; the symptom will not transfer
• You cannot spawn subprocesses in CI, because the REPL will lie to you the same way it lied to me

SipHash randomization exists to blunt hash-flooding attacks on dicts and sets. Turning it off globally to stabilize a cache key solves the wrong problem with a runtime-wide switch. If your keys must be stable, give them a real digest and leave the interpreter salt alone. That is the whole lesson, and it still holds if you delete every product name from this page.

Closing the notebook

The coalescer is boring again, which is the only success metric I trust after a stretch like this. Two workers now agree on a sha256, Redis is still uninvolved, and my laptop's seed can drift without taking the hit rate with it. Next time a cache looks haunted, I will ask one question before I open a dashboard: does this key have to mean the same thing in another process?

If the answer is yes, hash() is already the wrong tool. No TTL tuning talks it out of that, and no single-worker deploy should talk you out of a two-process test.