A JSONL record split in two: U+2028, U+0085, and the separator I missed

I keep a ledger as JSONL — one JSON object per line, written by my own importer, read by half a dozen small checkers. One day the checkers started reporting a line as unparseable, while json.loads on the same line was perfectly happy.

The character in the middle was U+2028 LINE SEPARATOR.

Why one line became two

Two facts, both documented, that only bite when they meet:

1. str.splitlines() treats U+2028 as a line boundary — along with \n, \r, U+2029, U+0085 and a few C0 controls (str.splitlines).
2. json.dumps(..., ensure_ascii=False) emits non-ASCII characters as themselves. U+2028 is not in the set JSON requires you to escape, so it stays raw (json).

So the file has exactly one \n per record, json.loads round-trips fine, and any reader that iterates with splitlines() sees a broken fragment.

import json

rec = {"id": 1, "body": "before\u2028after"}
line = json.dumps(rec, ensure_ascii=False) + "\n"

len(line.split("\n")) - 1     # 1   … one newline
len(line.splitlines())        # 2   … splits here
json.loads(line) == rec       # True … valid JSON the whole time

json.loads(line.splitlines()[0])
# JSONDecodeError: Unterminated string starting at: line 1 column 19 (char 18)

The fix I shipped, and why it was on the writer

There were several readers and one writer, so I fixed the writer:

def dumps_line(rec):
    return json.dumps(rec, ensure_ascii=False).replace("\u2028", "\\u2028").replace("\u2029", "\\u2029")

The invariant I should have written down on day one, and eventually did:

The serializer must not emit a raw code point that any of its readers treats as a line boundary.

Stated that way, the fix isn't "escape U+2028" — it's "keep the emitted set and the readers' boundary set disjoint," and the list of characters becomes something you derive rather than remember.

Three deliberate choices:

• Keep ensure_ascii=False. Setting it to True would also solve this, but then every Japanese character becomes \uXXXX and the ledger stops being readable by eye. Being readable is the point of this file, so I targeted the separators only.
• The value doesn't change. \u2028 is a standard JSON escape, so json.loads gives back the original character. I changed the representation, not the data.
• Put the incident in the docstring. The reason a replace() like this exists is exactly the thing a future me deletes during cleanup.

Existing rows still held the raw character at that moment. They got fixed sideways: the ledger tool's update path rewrites every row through dumps_line, so the next update swept them. Counted today across all 973 rows: 0 raw U+2028, 0 raw U+2029, 0 raw U+0085. That the sweep happened at all was luck, not design — worth saying out loud.

Four months later, I counted

I wrote "escaping the two separator characters" in the docstring and in the article. Neither the code nor I had ever asked the obvious question: how many characters can actually do this?

Of the characters splitlines() splits on, the ones that survive json.dumps(ensure_ascii=False) as raw text are:

Character	Raw in dumps output?	splitlines() splits?
U+2028 LINE SEPARATOR	yes	yes
U+2029 PARAGRAPH SEPARATOR	yes	yes
U+0085 NEXT LINE	yes	yes
U+000B, U+000C, U+001C–U+001E	no (escaped as \u000b, …)	no

Three, not two. The C0 controls are harmless here precisely because json.dumps is required to escape them; U+0085 is above 0x20, so nothing escapes it for you.

My guard had been green for four months on a hole it was never given a chance to see. The cases it covered worked; the scope was incomplete.

def dumps_line(rec):
    _s = json.dumps(rec, ensure_ascii=False)
    for _ch, _esc in (("\u2028", "\\u2028"),
                      ("\u2029", "\\u2029"),
                      ("\u0085", "\\u0085")):
        _s = _s.replace(_ch, _esc)
    return _s

The check that would have caught it on day one

Not more test data. A positive control: feed in a case that must be caught and require the failure.

Injected	splitlines()	value round-trips
U+2028	1 line	yes
U+2029	1 line	yes
U+0085	1 line	yes
all three at once	1 line	yes

Before the fix, the U+0085 row returned 2 lines. That single red is the whole value of the exercise: a test that only ever passes has told you nothing about the cases you didn't hand it.

There's a pattern here that generalizes past Unicode. When you write a guard against a class of input, the number that matters is not "did it pass" but how many members of the class you enumerated. I had enumerated two out of three, and the guard could not tell me that, because enumeration is the part outside the guard.

What this doesn't cover

• This is about the writer. A reader that uses splitlines() is still free to split on anything; I chose one place to fix instead of six.
• Other producers writing to the same file (a different script, a hand edit) bypass it entirely.
• The shape isn't Python-specific: the gap exists wherever a line splitter treats some code point as a boundary and the serializer emits that same code point raw. Which characters those are depends on the pair you're using — enumerate the intersection for your own splitter and encoder rather than copying my three.

What I'd do differently

1. When writing a guard against "characters of type X", print the enumerated list in the code, not in prose. ("\u2028", "\u2029", "\u0085") next to the replace is checkable; "the separator characters" in a docstring is not.
2. Add one poison case per member, and require red before the fix.
3. When the guard's scope changes, fix the message and the docstring in the same commit. Mine still said "only two" while the array said three, one revision later — the code was right and the text was lying.

Counted, not estimated: 973 rows with 0 raw U+2028 / U+2029 / U+0085 today, 3 characters that can split a line through this encoder, 1 of which was unguarded for four months.