I Trusted json.dumps for 48 Hours. The Idempotency Hash Followed Insertion Order.

Have you ever watched two JSON bodies print the same story and still fail a hash check? I did, and I spent forty-eight hours blaming a gateway that never rewrote a byte. The payloads compared equal as Python dicts, so every pretty-printed log looked completely honest to me. The idempotency key still drifted because json.dumps follows insertion order unless you pass sort_keys=True.

This field note keeps the repro I wanted on hour two, not a scoreboard. It is not a war story about traffic scale, private customers, or a stack I cannot show. I am writing down what I tried, what broke, and what I would repeat. If you hash JSON for idempotency, you already live next to this trap.

The question I started with

Why would a retry look like a brand-new request when the body seemed unchanged in the logs? I had a client that hashed the JSON body and sent an Idempotency-Key header so duplicates could collapse. Local checks passed because both sides built the dict from one shared fixture. A second builder on the retry path inserted the same keys in another order. Do you glance at pretty JSON and assume the underlying bytes match?

I did, and that glance sent me into the wrong layer for a full day. The dicts were equal in Python, and the logs were indented for humans. The digests were still different, which is how you lose a quiet forty-eight hours.

What I tried during the first day

I treated the failure like a network problem, because that story feels productive when you are tired.

1. I captured both HTTP requests with curl -v and a tiny Python listener, then compared request lines.
2. I dumped headers looking for Content-Length drift, a Via hop, or a charset surprise from a proxy.
3. I forced UTF-8, stripped spaces around colons, and hashed pretty text copied from logs.
4. I added sleeps and extra retries, because I wanted a race that I could name out loud.

None of those moves changed the digest on either side of the comparison. The bodies stayed valid JSON, and Python still said the dicts were equal. So why were the hex strings different after all of that busywork? The question was already pointing at serialization, not transit.

What actually broke

Python 3.7 made dict insertion order a language guarantee, which is helpful until you hash the serialized form. json.dumps(payload) walks keys in that insertion order by default, not in some imaginary canonical order. Two dicts can compare equal in memory and still serialize to different byte strings. My retry path built items before user_id, while the original path did the opposite.

Logs used indent=2, so my eyes never noticed the swapped keys on the first tired pass. I was hashing unstable bytes and calling that fingerprint a stable identity. Was this really a cryptography problem in disguise, or just a messy encoder default? It was a canonicalization bug wearing a security-shaped header.

Primary docs I should have opened sooner: the json.dumps parameters and RFC 8785 for actual JSON canonicalization.

The reproducible artifact

I stopped guessing and wrote a script that builds one logical payload in two insertion orders. Run it on a current CPython 3 interpreter and read the printed hashes. The first pair of digests should differ, and the second pair should match.

# labeled example: idempotency_hash_repro.py
# Reconstructs the field note. Not a production client.
import hashlib
import json


def dump_default(payload: dict) -> str:
    return json.dumps(payload, separators=(",", ":"))


def dump_sorted(payload: dict) -> str:
    return json.dumps(payload, separators=(",", ":"), sort_keys=True)


def sha256_hex(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


built_left = {"user_id": 42, "items": [{"sku": "a", "qty": 1}]}
built_right = {"items": [{"sku": "a", "qty": 1}], "user_id": 42}

assert built_left == built_right
assert list(built_left.keys()) != list(built_right.keys())

raw_left = dump_default(built_left)
raw_right = dump_default(built_right)
print("equal dicts?", built_left == built_right)
print("raw left :", raw_left)
print("raw right:", raw_right)
print("raw hashes equal?", sha256_hex(raw_left) == sha256_hex(raw_right))
print("left hash :", sha256_hex(raw_left))
print("right hash:", sha256_hex(raw_right))

canon_left = dump_sorted(built_left)
canon_right = dump_sorted(built_right)
print("sorted left :", canon_left)
print("sorted right:", canon_right)
print("sorted hashes equal?", sha256_hex(canon_left) == sha256_hex(canon_right))
print("sorted hash:", sha256_hex(canon_left))

Commands after I closed the notebook

I ran these once the file existed on disk and the long-lived kernel was gone.

python3 idempotency_hash_repro.py

python3 -c 'import json,sys; print(json.dumps(json.load(sys.stdin), sort_keys=True, separators=(",", ":")))' < body.json

python3 -m json.tool --sort-keys body.json

The middle command is the cheap canonicalizer I now keep in shell history. The last command is only for human eyes. Please do not hash json.tool output if separators or a trailing newline matter to your contract.

A decision table I should have drawn at hour two

Check	What it proves	What it hides
left == right on dicts	Same keys and values	Serialization order
Pretty indent=2 logs	Human-readable shape	Key order, separators, ASCII escaping
json.dumps without sort_keys	Exact builder bytes	Any other insertion path
json.dumps(..., sort_keys=True)	Stable key order	Float encoding, default=, key types
Hash of the HTTP body as sent	Wire identity	Logical JSON equality

If you need a real canonical JSON rule, read RFC 8785 instead of inventing one from sort_keys. Python's encoder still is not JCS, even with sorted keys. Floats, NaN, and non-string keys remain sharp edges across languages. I labeled the script a reconstructed repro because it is not a payment-grade canonicalizer.

Hour twenty-something: I took the hasher off my laptop

My notebook had already polluted the investigation in a quiet way. Cells ran in an order that rebuilt the dict the lucky way, so the retry path looked fixed until I restarted the kernel. I needed a clean interpreter, and I needed a process that had never imported the fixture module. Have you ever "fixed" a hash bug by rerunning one cell?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to list hypotheses I had not tested, including Unicode normalization, ensure_ascii, and key order. I used the free server option to run the hasher in a process that did not share notebook state. The model did not discover the bug for me, and the script only did after I discarded confident proxy guesses.

Would I paste a production secret into that loop? No. The repro uses fake identifiers on purpose, and that boundary stays.

Nested lists almost tricked me again

After the hashes matched with sort_keys=True, I almost shipped the helper and walked away happy. Then a nested cart reminded me that key sorting is recursive, which is what I wanted. List order stays significant, which is also what I wanted. Shuffling items must change the hash, while swapping key insertion must not.

# labeled example: extra assertions I added after the first green run
nested_a = {"user_id": 42, "items": [{"sku": "a"}, {"sku": "b"}]}
nested_b = {"items": [{"sku": "a"}, {"sku": "b"}], "user_id": 42}
nested_c = {"user_id": 42, "items": [{"sku": "b"}, {"sku": "a"}]}

assert dump_sorted(nested_a) == dump_sorted(nested_b)
assert dump_sorted(nested_a) != dump_sorted(nested_c)

That tiny triangle is the whole contract I care about here. Object keys are noise for identity, and sequences are data. Mix those two rules and you will merge two different carts into one key. Would you rather collapse duplicates, or collapse two orders?

What I would repeat

• Print repr(raw_json) before printing a pretty blob, every single time I doubt a hash.
• Hash both builders in one process before blaming a proxy, a gateway, or a retry loop.
• Keep a unit test that inserts keys backwards on purpose and still expects one digest.
• Avoid debugging JSON identity inside a long-lived notebook kernel that still holds fixtures.

I would also refuse to treat sort_keys=True as a substitute for a shared wire format. If both services can send the same bytes, then hash those transmitted bytes. A reserialized cousin is a different document, even when the dicts compare equal in Python.

Limitations, and who should not copy this blindly

This workflow is for developers who control both JSON builders and who can add a focused unit test. It is not a general guide that makes hashing safe for every API.

• Do not use this writeup as a cryptography design. SHA-256 is only a fingerprint of a canonical string here.
• Do not hash pretty-printed logs, because indent and newlines are not part of your real contract.
• Do not ignore floats, since 1.0 and 1 become different JSON tokens after mixed type dumps.
• Do not feed non-string keys to json.dumps and expect the same bytes from another language.
• If you need interoperable canonical JSON, implement RFC 8785. Do not pretend sort_keys is that specification.
• Skip this approach when the HTTP body is not JSON, or when the client already supplies the key.

The hypothesis list was useful for covering Unicode and escaping in one sitting. It was not a substitute for running the script. If your payload includes secrets, keep the hasher on a machine you already control.

Closing the notebook

Forty-eight hours is a long time to relearn that equal dicts are not equal bytes. I still catch myself trusting a pretty log when I am tired. The cure is boring and local: dump with explicit separators, sort keys when the contract is logical JSON, and hash the string you will send. Would I start with the gateway again next time? Probably, because that reflex is human.

I would still run the short repro before I opened a packet capture. The script is the takeaway, not a slogan. Run it in a clean interpreter you already trust.