The Payload Looked Like JSON for 48 Hours. Then a Strict Parser Hit NaN.

Have you ever watched a downstream service reject a body that every local Python test called perfectly valid JSON? I just burned forty-eight hours on that mismatch, and the encoder was not lying to me in the way I expected. It was json.dumps, doing the documented default, while I silently assumed "JSON" meant RFC 8259 and nothing else. These field notes cover what I tried, what actually broke, and the small gate I will keep.

Hour zero: the body looked fine

The symptom was almost boring, which is how these things usually start in production traffic. A Python worker posted a small metrics object, tests printed the body, and the fixture matched a golden string that still contained numbers. The HTTP client reported 400, and the peer logged a JSON syntax error without pointing at a field name I recognized. Why would a one-line dump fail after the unit tests pretty-printed the same dict?

I did what I always do when a payload "looks like JSON" and still dies at the border. I blamed the proxy, then gzip, then a charset, then a truncated write, then the other team's parser. None of those theories survived a hex dump of the complete UTF-8 body that was sitting on disk. The bytes were not truncated at all; they were ugly in one very specific token.

What I tried for the first day

Here is the messy list, because field notes that skip the dead ends teach the wrong lesson:

1. I replayed the POST with curl --data-binary @body.json and still got a parse error from a strict client.
2. I checked Content-Type and BOM markers, because a previous encoding bug had trained me to start there.
3. I ran python -m json.tool body.json and watched CPython accept the file with a smile.
4. I opened the same bytes in a browser console and finally saw JSON.parse throw a SyntaxError.
5. I diffed Python's dump against Node's JSON.stringify on the same logical object and stared at one token.

That fifth item was the whole bug, and I should have reached it before blaming the gateway. Before I paste the dump, can you guess which single token made a strict parser give up? The rest of this note is the reproduction I should have written during hour one, plus the CI gate I will not skip again.

The reproduction I should have written at hour one

Runnable example on a stock CPython interpreter, with no extra packages:

import json
import math

payload = {
    "name": "p95_latency",
    "value": float("nan"),
    "ok": 1.0,
}

print(json.dumps(payload))
print("isnan:", math.isnan(payload["value"]))

When the value is float("nan"), json.dumps prints an illegal document that looks harmless in a terminal. On my machine that print is not null for the missing sample. It is this document, which RFC 8259, Section 6 does not allow:

{"name": "p95_latency", "value": NaN, "ok": 1.0}

RFC 8259 is blunt about numbers: tokens such as Infinity and NaN are not permitted in JSON text. Python's json module still emits them unless you pass allow_nan=False, which is not the default. The standard library even says that default is not spec-compliant, and I had never highlighted that sentence.

I now keep three one-liners in a scratch file so I stop arguing with the wrong parser:

python -c "import json; print(json.dumps({'x': float('nan')}))"
python -c "import json; json.dumps({'x': float('nan')}, allow_nan=False)"
node -e "JSON.parse('{\"x\": NaN}')"

The first command prints {"x": NaN}, and that token already should have ended the debate. The second command raises ValueError: Out of range float values are not JSON compliant on purpose. The third throws in Node, which is much closer to the parser my downstream actually resembled. Python's json.loads('{"x": NaN}') round-trips happily, so CPython-only tests were a mirror looking at itself.

A gate I can run in CI

I wanted a helper that fails on encode and on decode, because I do not control every producer. The next block is a labeled local example, not a framework and not a published package:

import json
from typing import Any


def _reject_constant(token: str) -> None:
    raise ValueError(f"non-JSON number token: {token}")


def dump_strict(obj: Any) -> str:
    return json.dumps(obj, allow_nan=False, separators=(",", ":"))


def load_strict(text: str) -> Any:
    return json.loads(text, parse_constant=_reject_constant)

And here is a tiny unittest file I re-run when someone adds another metrics field to the payload:

import unittest

class StrictJsonTests(unittest.TestCase):
    def test_finite_float_roundtrip(self):
        body = dump_strict({"v": 1.25})
        self.assertEqual(load_strict(body), {"v": 1.25})

    def test_dump_rejects_nan(self):
        with self.assertRaises(ValueError):
            dump_strict({"v": float("nan")})

    def test_dump_rejects_inf(self):
        with self.assertRaises(ValueError):
            dump_strict({"v": float("inf")})

    def test_load_rejects_nan_token(self):
        with self.assertRaises(ValueError):
            load_strict('{"v": NaN}')

    def test_load_rejects_infinity_token(self):
        with self.assertRaises(ValueError):
            load_strict('{"v": Infinity}')

    def test_string_that_looks_like_nan_is_fine(self):
        body = dump_strict({"v": "NaN"})
        self.assertEqual(load_strict(body), {"v": "NaN"})


if __name__ == "__main__":
    unittest.main()

Notice the last test, because a sloppy regex over raw text would have punished a legal string. parse_constant only fires for the bare tokens, which is exactly the behavior I want here. A helper like this belongs in CI next to the encoder, not in a wiki page nobody opens.

Decision table from the forty-eight hours

I collapsed the forty-eight hours into a table so I would stop relearning the same six rows.

Value in the dict	json.dumps default	allow_nan=False	RFC 8259	json.loads of that text
1.0	1.0	1.0	valid	1.0
float("nan")	NaN	ValueError	invalid	nan
float("inf")	Infinity	ValueError	invalid	inf
float("-inf")	-Infinity	ValueError	invalid	-inf
None	null	null	valid	None
"NaN" (string)	"NaN"	"NaN"	valid	"NaN"

If you only remember one row from that table, make it the float("nan") row. It is the row that sneaks out of pandas, NumPy, and a lonely division inside a rolling window. Once that value exists in a dict, default json.dumps will smuggle it past every CPython unit test.

Where a second environment earned its keep

I needed a checker that would not accidentally call Python's lenient loader and then declare victory. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the parse_constant helper from the failing dump. I then ran the unittest file on the free server option so my laptop's site-packages could not rescue me.

Was a second environment required for the finding? A local venv plus Node's JSON.parse would have caught the token too. I used it anyway because I no longer trust a green bar from the same interpreter that emitted NaN. The pairing did not invent a new JSON spec; it kept my hands off the shell where I had already installed convenience wrappers.

What broke when I tightened the encoder

Turning on allow_nan=False did not make the original bug vanish without leaving leftovers in CI. A few tests had been using math.nan as a sentinel for "not enough samples," and those dumps started raising. Pandas objects were worse, because a single empty group turned a float column into NaN and poisoned the payload. I had to decide field by field: omit the key, emit null, or fail the job immediately.

My current rule is boring on purpose:

• Finite numbers serialize as numbers, and that is the only numeric case the wire format should see.
• Missing samples omit the key, or use null if the schema requires the field to exist.
• NaN and infinities are bugs in the producer, not values the wire format must preserve.
• Golden fixtures are parsed with load_strict, never with default json.loads.

If you need a numeric hole in a scientific file, JSON was the wrong container for that hole. Use a format that actually has NaN, or write a documented sidecar, but do not ask RFC 8259 to carry it. Do not smuggle an IEEE 754 quiet NaN inside a metrics POST and call the result interoperable JSON.

Limitations, said out loud

This gate does not serialize datetime, Decimal, UUID, or set objects without an explicit hook. Those types still need a default callable, and that callable can hide other bugs if it is just str. It does not defend against huge payloads, cyclic references, or ensure_ascii surprises in non-ASCII logs. parse_constant will not save you if a proxy already stripped the body or a Java parser rejects null.

Who should not copy this approach into a codebase that already owns both ends of the wire? Anyone who intentionally ships Python-to-Python json with allow_nan=True as an internal cache is in a different boat. That is still a footgun, but pickle or msgpack would be the more honest tool for those process-local blobs. Also skip this if your "JSON" is actually JSON Lines mixed with debug prints; fix the stream first.

I did not benchmark encoders, and I will not pretend a free server is a load-test cluster. The only cost claim I can stand behind is simple and a little embarrassing to write down. The extra allow_nan=False argument is cheaper than another forty-eight hours of blaming nginx for a NaN token.

What I would repeat

If I am honest about the forty-eight hours, the useful loop was small and slightly humiliating.

1. Dump the raw bytes, not the pretty repr of a dict that Python already knows how to print.
2. Parse those bytes with a second implementation that is stricter than CPython's json module.
3. Fail closed on encode with allow_nan=False before inventing sanitizers that turn NaN into null quietly.
4. Keep one test that feeds the literal tokens NaN and Infinity into the loader.
5. Write down the sentinel policy for missing samples so pandas does not decide it for me.

Would I still start by blaming the proxy when the status code is a generic 400? Probably yes, because I am human and a gateway error is the story that fits the dashboard. I would just add the Node one-liner an hour sooner, and I would stop treating python -m json.tool as proof of interoperability.

If a strict parser has ever rejected a body your tests still call JSON, what token did it choke on?