48-Hour Field Notes: json.loads Never Flagged the Duplicate Key

Have you ever stared at two config files that looked identical in the editor, then watched production pick the wrong credential? I did exactly that last week, and I blamed the deploy pipeline for two full days afterward. The payload passed every schema check I owned, because those checks never walked object keys for duplicates at all. Why would those checks bother, when the RFC already told every parser that object names should stay unique?

This is a field notebook from a forty-eight hour chase, not a dashboard postmortem with invented incident numbers. I will show what I tried, what actually broke, and the small test I now rerun on every merge. You can copy the artifact without buying anything, and the lesson still holds if every product name disappears.

Hour zero: the payload looked boring

I received a flattened settings blob from an upstream service that merged two sources into one object. My local pretty-printer showed a single api_key field, and the unit tests loaded the same fixture through json.loads. I printed the dict, hashed the dict, and even dumped it back with json.dumps, which still looked round-trippable. Did I open the raw bytes before hour thirty, or did I trust the highlighter like a coward?

Here is the kind of body I was actually dealing with, reconstructed as a fixture rather than a live secret:

RAW = b'''
{
  "service": "billing",
  "api_key": "from-defaults",
  "api_key": "from-override",
  "retries": 3
}
'''

Open that fixture in most highlighters and you will see one key, colored once, because editors collapse the story. Ask json.loads what it thinks, and it will smile and hand you only the second value without comment. That is not a CPython bug so much as a compatibility choice that RFC 8259 never forbade parsers from making. Section 4 still says object names SHOULD be unique, which is advice, not a runtime exception in the decoder you already ship.

What I tried first

I did the usual forty-eight hour tour of the wrong neighborhoods, and I wrote them down this time. Does any of this list feel like your last outage, or am I the only one who trusts pretty printers? The parsed object was honest about its contents; my questions were dishonest about what the parser had thrown away.

1. I compared Git SHAs on the config repo, because duplicate names feel like a merge artifact that git should have shown.
2. I dumped process environment variables, assuming an overlay had replaced the file after the image was built.
3. I logged sorted(settings.items()) after parse, which can never reveal a key that already vanished into the dict.
4. I reran the schema validator with extra debug flags, which still walked a Python dict, not the original pairs.
5. I blamed the remote merge service, because blaming the network is cheaper than reading the CPython json docs.

Pretty printers, schema jobs, and repr(settings) all consume the winner. None of them are a hex dump. That is the whole first day in one paragraph, and I am still annoyed at myself.

What actually broke

CPython's json module builds objects from an ordered list of pairs, then stores those pairs in a dict. When two pairs share a name, the dict assignment wins last, and no exception is raised for the loser. JSON Schema validators that receive the dict also never see the discarded pair, so they stay green. My tests asserted on the winning key, which is how a live default credential survived code review.

I finally reproduced it with a pairs hook, which is the extension point the standard library documents for this case. The hook receives every pair before the dict is built, so a duplicate name can still speak in time. Nested objects work as well, because the decoder calls the hook once per object, including objects inside arrays.

import json
from typing import Any


class DuplicateKeyError(ValueError):
    """Raised when one JSON object contains the same name twice."""


def reject_duplicate_pairs(pairs: list[tuple[str, Any]]) -> dict:
    seen: dict[str, Any] = {}
    for key, value in pairs:
        if key in seen:
            raise DuplicateKeyError(
                f"duplicate object key {key!r}; "
                f"first={seen[key]!r} second={value!r}"
            )
        seen[key] = value
    return seen


def loads_strict(text: str | bytes) -> Any:
    if isinstance(text, bytes):
        text = text.decode("utf-8")
    return json.loads(text, object_pairs_hook=reject_duplicate_pairs)

Run that against the fixture above and you get a traceback instead of a silent override. That is the whole fix. Everything else in these notes is how long I took to stop asking the dict what the file had said.

A tiny command I wish I had run at hour one

python -c "import json,sys; print(json.loads(sys.stdin.read()))" < fixture.json
python -c "
from pathlib import Path
raw = Path('fixture.json').read_text()
print(raw.count('\"api_key\"'))
"
xxd fixture.json | head

The second command is crude, but a count greater than one is a smell worth chasing before you trust the dict. It false-positives on keys nested under different parents, which is why the pairs hook remains the real check. Would you have run the hex dump first, or would you have grepped the pretty JSON like I did?

The forty-eight hour timeline, compressed

• Hours 0–8: I trusted logs of the parsed dict and a green schema job on the merged blob.
• Hours 8–20: I chased deploy order, environment overlays, and a suspected CDN cache of the settings file.
• Hours 20–32: I hex-dumped the payload and finally saw the second "api_key" sitting in plain UTF-8.
• Hours 32–40: I learned object_pairs_hook exists, then wrote the reject function in the block above.
• Hours 40–48: I turned the hook into a pytest parametrize list and reran it on a clean interpreter.

Would I call this a parser bug now that I have watched the pairs list move through a dict? No, I would call it a contract I never wrote down, even though last-wins is completely predictable behavior. Silent last-wins is still a footgun for anything that merges documents from more than one writer.

A reproducible test plan, not a vibe

I now keep a table next to the loader, because I forget the nested edge cases after a quiet week. Label the pytest module as a local example until you have wired it into your own tree on purpose. I ran it on a stock CPython interpreter with the standard json module, not a rewritten third-party decoder.

Case	Input sketch	Expect
Unique keys	{"a": 1, "b": 2}	dict, no error
Sibling duplicates	{"a": 1, "a": 2}	DuplicateKeyError
Nested duplicate	{"n": {"a": 1, "a": 2}}	DuplicateKeyError
Array of objects	[{"a": 1}, {"a": 2}]	ok, different objects
Same key, different parents	{"l": {"a": 1}, "r": {"a": 2}}	ok
Empty object	{}	ok
Duplicate in list item	{"k": [{"a": 1, "a": 9}]}	DuplicateKeyError

import pytest

SAMPLES = [
    ('{"a": 1, "b": 2}', True),
    ('{"a": 1, "a": 2}', False),
    ('{"n": {"a": 1, "a": 2}}', False),
    ('[{"a": 1}, {"a": 2}]', True),
    ('{"l": {"a": 1}, "r": {"a": 2}}', True),
    ('{}', True),
    ('{"k": [{"a": 1, "a": 9}]}', False),
]


@pytest.mark.parametrize("text, ok", SAMPLES)
def test_duplicate_keys(text: str, ok: bool) -> None:
    if ok:
        loads_strict(text)
        return
    with pytest.raises(DuplicateKeyError):
        loads_strict(text)

If your stack uses orjson, ujson, or a Go service in front, re-check that stack before you copy this hook. Last-wins is a family tradition across JSON parsers rather than a single CPython function you can patch once. Have you verified your sidecar parser, or are you assuming every language throws the way your linter wishes?

Where a clean box and a second brain helped

My laptop had a sitecustomize.py that pretty-printed failed tests, and that extra import path made every local result feel cursed. I wanted a boring interpreter, no plugins, no editor-injected PYTHONPATH, and nastier fixtures than I would type by hand at 2 a.m.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to brainstorm nested duplicate-key fixtures I kept forgetting, like duplicates inside arrays of arrays. I used the free server option to run loads_strict in a clean process that did not inherit my shell aliases. That pairing is optional; the hook above is the actual fix, and it runs anywhere you already have Python. If you try that path, treat it as a clean runner plus a fixture generator, not as a substitute for rejecting pairs yourself.

I am not going to invent model names, quotas, or hardware claims for that environment, because I did not measure them here. If you already have a spare container, use that instead and keep the same tests against the pairs hook. Is a clean box glamorous? No, but it is how I stopped arguing with my own sitecustomize.py file.

What I would repeat

• I would dump the raw UTF-8 before I dump the parsed dict, every time a config merge is involved.
• I would count quotes around candidate keys only as a smoke test, then rely on object_pairs_hook for the verdict.
• I would parametrize the seven cases in the table so a future merge cannot regress silently on review.
• I would still ask a model to generate extra fixtures, because I am bad at inventing nested duplicates after midnight.

I would not repeat blaming the CDN for a byte that was sitting in the file the whole time. I would not repeat asserting on settings["api_key"] without proving the source object had one pair. I would not repeat trusting a syntax highlighter to render two identical keys as two identical keys. Those three mistakes cost me the first thirty-two hours, and they were free to avoid.

Limitations, and who should skip this

This hook is not a full JSON linter, and it will not catch Unicode lookalike names that are not the same Python string. It will not catch YAML merge keys, TOML dotted redefinitions, or environment overlays that happen after the document is already a dict. It loads the whole document, so it is the wrong tool for multi-gigabyte feeds that need a streaming parser. Last-wins may be a compatibility promise you already shipped to clients, and flipping to raise will break those clients on purpose.

Skip this approach if you control both writers and you already emit canonical JSON with unique names through a single serializer. Skip it if you are parsing untrusted gigantic payloads and cannot afford a Python object graph in memory. Skip it if your decoder is not CPython json and you have not verified that object_pairs_hook exists there with the same pair order. In those cases, fix the writer or pick a parser API that exposes pairs on that runtime.

Schema libraries that validate after json.loads will not save you, which is the whole point of these notes. Need a second pass over pairs, or do not bother pretending uniqueness is enforced by the RFC alone.

Closing the notebook

Forty-eight hours was a long time to learn that uniqueness in JSON is a SHOULD, not a runtime guarantee your decoder owes you. The reproducible part is small: reject pairs, parametrize the table, run it on an interpreter that does not inherit your laptop. What would your pretty printer have shown you on that fixture? I already know my answer, and I still do not like it.