Python json.dumps ensure_ascii: 24% More Claude Tokens, Wrong Kanji

The Python port of my support-ticket classifier used 24% more input tokens than the TypeScript version it replaced. Same model. Same system prompt. Same 2,000 tickets in the replay set.

I spent an afternoon diffing prompts before I saw it. The TypeScript prompt said 東京. The Python prompt said 東京.

One default argument in json.dumps, ensure_ascii=True, was quietly rewriting every non-ASCII character in my prompts into hex escapes. It cost tokens, and worse, it made Claude misread Japanese customer names.

TL;DR

• Python's json.dumps defaults to ensure_ascii=True, which turns é into é and 東 into 東. JavaScript's JSON.stringify doesn't do this.
• If you paste json.dumps(...) output into a prompt string, the model sees literal backslash-u hex, not the characters.
• On my 2,000-ticket replay set, that raised average input tokens from 1,180 to 1,461 per request (+24%). Japanese ticket payloads alone grew about 3.1x.
• Extraction quality dropped too: 29 of 340 Japanese tickets came back with a wrong character in a name or address, versus 3 of 340 with raw UTF-8.
• Fix: json.dumps(obj, ensure_ascii=False), plus force_ascii=False in pandas to_json, plus a test that fails if \u shows up in a rendered prompt.

What does json.dumps ensure_ascii actually do?

ensure_ascii=True (the default) makes json.dumps escape every character outside ASCII as a \uXXXX sequence, and characters outside the Basic Multilingual Plane (most emoji) as a surrogate pair of two escapes. The output is valid JSON and decodes back to the same data. It's a transport-safety feature from an era when you couldn't trust a pipe to carry UTF-8.

>>> import json
>>> json.dumps({"city": "München", "name": "東京"})
'{"city": "M\\u00fcnchen", "name": "\\u6771\\u4eac"}'
>>> json.dumps({"city": "München", "name": "東京"}, ensure_ascii=False)
'{"city": "München", "name": "東京"}'

For a JSON file or an HTTP body, this is harmless. The receiver parses the JSON and gets 東京 back. The problem only starts when the JSON is not the transport but the content.

Why does ensure_ascii only break prompts and not API calls?

Because a prompt is a string that nobody parses as JSON before the model reads it. Here's the line that bit me:

prompt = f"""Classify this ticket and extract the customer's name and city.

<ticket>
{json.dumps(ticket, indent=2)}
</ticket>"""

client.messages.create(
    model=MODEL,
    max_tokens=512,
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": prompt}],
)

The SDK then serializes the whole request body to send over HTTP. That outer layer is fine. But the inner 東 is now part of a string value, so the SDK escapes the backslash, the API decodes it once, and the model receives the six literal characters \, u, 6, 7, 7, 1.

The model doesn't get 東. It gets a hex puzzle and has to solve it in its head.

The TypeScript version did the same thing with JSON.stringify(ticket, null, 2), which leaves non-ASCII characters alone. That's why nobody noticed during the port: the code looked identical line for line.

How many extra tokens does ensure_ascii cost?

On my data, 24% more input tokens per request on average, and about 3x more on the ticket payload for Japanese text. I measured it with Anthropic's count_tokens endpoint against the same 2,000 tickets rendered both ways:

Ticket language	Share of set	Payload tokens, escaped vs raw
English	38%	1.00x
German	22%	1.06x
Spanish / Portuguese	13%	1.09x
Japanese	17%	3.1x
Other (French, Polish, emoji-heavy)	10%	about 2x

Average input per request went from 1,180 tokens to 1,461. About 600 of those are a fixed system prompt, so the payload part grew by almost half.

The pattern makes sense once you look at what the tokenizer is fed. A German ticket only escapes the occasional ü or ß, so the damage is a few percent. A Japanese ticket escapes every single character, and each one becomes a dozen-ish ASCII characters of hex that tokenize badly. Emoji are the worst per character: one 😀 becomes 😀, twelve characters of surrogate pair.

At about 40,000 tickets a month, the extra 281 tokens per request comes to roughly 11 million input tokens I was paying for so the model could read hex. Multiply by your own per-token price; mine was not a fun number.

Does escaping non-ASCII hurt LLM accuracy?

Yes, at least for extraction. Classification barely moved, but copying exact characters out of escaped text got noticeably worse.

The bug actually surfaced through accuracy, not cost. A support agent flagged a ticket where the extracted customer name contained a kanji that appeared nowhere in the original message. I grepped the raw ticket for the extracted name: zero hits.

So I scored the 340 Japanese tickets from the replay set. A result counted as wrong if the extracted name or city string didn't appear verbatim in the original ticket text:

• Escaped prompts: 29 of 340 had at least one wrong character (8.5%)
• Raw UTF-8 prompts: 3 of 340 (0.9%)
• Category labels: within one point of each other across both versions

My working theory: common characters like 東京 show up escaped all over the training data, so the model decodes them reliably. Rare kanji in personal names don't, and the model lands on a character whose code point is close but whose meaning and shape are unrelated. Hex nearness is not visual nearness. A one-digit slip in 髙 gives you a completely different character, and nothing in the surrounding text looks wrong to the model.

Caveats, since I'd want them if I were reading this: one model, one run per version, 340 tickets, exact-match scoring. It's a strong signal on my data, not a benchmark. But the direction is not subtle.

Where else does ensure_ascii hide in a Python LLM pipeline?

Anywhere you serialize data into prompt text. I found three more spots after fixing the first one:

1. pandas.DataFrame.to_json() defaults to force_ascii=True. I used it to inline a few rows of order history into prompts. Same bug, different parameter name.
2. Few-shot examples loaded from a file and re-dumped. The loader read UTF-8 JSON correctly, then a helper called json.dumps(example) to format it. Every Japanese few-shot example was escaped.
3. Tool results. My agent returned tool outputs as json.dumps(result) in tool_result content. Search results with French product names went to the model as café.

Places that were already fine: Pydantic v2's model_dump_json() writes raw UTF-8, and orjson.dumps() always outputs UTF-8. If you're on either, you're probably safe. Check anyway; I was sure about the TypeScript-to-Python port too.

How do you fix it and keep it fixed?

Pass ensure_ascii=False everywhere JSON ends up in a prompt, then add a test so the next helper function can't bring it back.

I centralized prompt serialization in one function:

def to_prompt_json(obj) -> str:
    # Compact separators, raw UTF-8. This goes to a model, not a wire.
    return json.dumps(obj, ensure_ascii=False, separators=(",", ":"))

Dropping indent=2 for compact separators saved another 6% on payload tokens. I checked extraction accuracy on the same set and saw no difference, but test that on your own data before copying it; some prompts read better indented.

Then a guard that runs in CI against every prompt template with a fixture that includes Japanese, German, and an emoji:

import re

ESCAPE = re.compile(r"\\u[0-9a-fA-F]{4}")

def test_prompts_contain_no_unicode_escapes():
    for name, render in PROMPT_TEMPLATES.items():
        prompt = render(MULTILINGUAL_FIXTURE)
        assert not ESCAPE.search(prompt), f"{name} leaks \\u escapes"

It caught the pandas call on its first run. And a grep for the lazy version of the audit:

grep -rn "json.dumps(" src/ | grep -v "ensure_ascii=False"
grep -rn "to_json(" src/ | grep -v "force_ascii=False"

After the fix, the Python version came in at 1,148 average input tokens: 3% under the TypeScript original, thanks to the compact separators. Japanese extraction errors dropped back to 3 of 340.

The lesson I'm keeping

The scary part isn't that ensure_ascii exists. It's that every layer looked correct. The JSON was valid. The request succeeded. The logs printed fine, because I was logging the ticket, not the rendered prompt. The model answered confidently. The only visible symptoms were a slightly bigger bill and a wrong character in a name that a human happened to check.

Log the exact string the model receives, at least for a sample. Read it once with your own eyes. Most prompt bugs I've had were visible in about five seconds of looking at the real thing.

So does json.dumps ensure_ascii waste LLM tokens?

Yes. Python's json.dumps escapes every non-ASCII character as \uXXXX by default, and when that output is pasted into a prompt, the model reads literal hex instead of the original text. In my 2,000-ticket test with Claude, it raised average input tokens by 24%, tripled the payload size for Japanese tickets, and raised Japanese name-extraction errors from 3 to 29 out of 340. Use json.dumps(obj, ensure_ascii=False) (and force_ascii=False for pandas to_json) anywhere JSON becomes prompt text, then add a test that fails if \u escapes appear in a rendered prompt.

---

Written by the developer behind Preterview, an interview prep platform.