Digit Tokenization: Why Commas Fix LLM Arithmetic

Take the number 1234567. Now take 234567. To you, the second is the first with the leading digit removed — the trailing six digits are identical, and their place values are identical.

To a GPT-lineage tokenizer, they share almost nothing. The first splits as 123|456|7. The second splits as 234|567. Not one token boundary lines up. The model has to relearn "these are the same trailing digits" from scratch, per length class.

That is digit tokenization, and it is the single most under-discussed reason LLMs fumble arithmetic on numbers longer than three digits.

TL;DR

• Digit tokenization groups digit runs into chunks (up to 3 for tiktoken-lineage tokenizers) using a left-to-right greedy pre-tokenizer regex. Chunk boundaries therefore depend on total number length, not on place value.
• Left-to-right grouping means the token 456 sits in the thousands place in one number and the units place in another. Carries in addition cross token boundaries at unpredictable offsets.
• Formatting numbers with thousands separators (1,234,567) forces right-to-left, power-of-1000-aligned chunks — the comma is a hard token boundary, so every group is a stable place-value unit. This consistently helps arithmetic and numeric copying.
• Other tokenizer families (Llama 2, Gemma, DeepSeek-family) split digits individually, which is cleaner for math but costs ~3x the tokens on numeric-heavy text.
• Production rule: normalize number formatting at prompt-build time, and route real arithmetic to a tool. Formatting hygiene is damage control, not a calculator.

How do LLM tokenizers actually split numbers?

Modern BPE tokenizers do not run BPE over raw text. They first apply a pre-tokenizer regex that carves text into fragments; BPE merges never cross those fragment boundaries. The digit clause in the cl100k_base and o200k_base patterns is essentially:

\p{N}{1,3}

"Match one to three digits." Regex alternation is greedy and scans left to right, so a 7-digit run is consumed as 3+3+1, a 6-digit run as 3+3, a 5-digit run as 3+2.

This was a deliberate improvement over GPT-2, whose tokenizer had no digit clause at all. There, BPE merged whatever digit strings happened to be frequent in the training corpus — 2017 was one token because it appeared everywhere, while a random 4-digit number fragmented arbitrarily. Capping at three at least made tokenization deterministic by length. It did not make it aligned by place value.

Verify your own tokenizer rather than trusting anyone's blog post, this one included:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

for s in ["1234567", "234567", "1,234,567", "1234.56", "1,234.56"]:
    ids = enc.encode(s)
    print(f"{s:>12}  {len(ids)} tok  {[enc.decode([i]) for i in ids]}")

Watch what the comma does. 1234567 fragments as three chunks whose boundaries depend on the leading digit. 1,234,567 cannot fragment any other way: the comma is not a digit, so the regex is forced to emit 1, ,, 234, ,, 567. Every digit group is exactly one power-of-1000 band. The formatting you already use for humans happens to be the alignment the tokenizer needs.

Anthropic does not publish Claude's tokenizer, so you cannot inspect splits directly for Opus 4.x or Sonnet 4.x. You can measure the effect: client.messages.count_tokens() will show you that 1,234,567 and 1234567 cost different token counts, which tells you the boundaries differ. And the downstream accuracy effect is measurable with an eval regardless of what the tokenizer does internally.

Why does left-to-right digit grouping break place value?

Because addition is a right-aligned algorithm and the tokens are left-aligned.

Consider 7654321 + 654321. The operands tokenize as:

7654321  ->  765 | 432 | 1
 654321  ->  654 | 321

The units digit of the first operand lives in a 1-digit token; the units digits of the second live inside a 3-digit token. To add them, the model must, inside the residual stream, decode 432 into (hundreds=4, tens=3, units=2), decode 321 into (4,3,2)-analogous components, figure out that the offsets differ by two decimal places, add, and propagate carries across token boundaries that sit at different offsets in each operand.

Two things make this hard rather than merely tedious:

1. The representation burden is combinatorial. With 3-digit chunks there are 1000 distinct digit-triples, each of which the model must be able to decompose into place components — and it must do so relative to a per-number offset it has to infer from the run length. Single-digit tokenization gives 10 embeddings whose place value is just the position index.
2. Carries are non-local. A carry out of 321 has to land inside 432 at a specific internal digit. Attention can route information between tokens, but the within-token place arithmetic has to happen in MLP layers, and the offset varies per example. This is exactly the kind of variable-alignment task that transformers learn poorly and generalize on worst — accuracy degrades sharply as digit count rises, and degrades further when the two operands have different lengths (and therefore different chunk offsets).

That last point is the practical tell. If your model's arithmetic errors cluster on length-mismatched operands, you are looking at a tokenization alignment failure, not a reasoning failure.

Single-digit tokenizers (Llama 2, Gemma, DeepSeek family) sidestep this entirely at the cost of sequence length: a 12-digit number is 12 tokens instead of 4. On numeric-heavy documents — financial tables, log dumps, telemetry — that is a real 2-3x context cost. It is a genuine trade-off, not a free win, and it is one reason the tiktoken lineage kept the 3-digit cap.

Does adding commas actually help, or is it folklore?

It helps, and the mechanism is not mysterious: the comma is a non-digit character, so it terminates the digit-run match. Grouping becomes right-to-left by construction, and identical trailing place-value bands get identical tokens across numbers of different magnitude.

1,234,567 and 234,567 now share the tokens 234, ,, 567 — with the same place-value meaning in both. That is the invariance the raw form destroys.

The same logic explains a second, more common production bug: decimal formatting inconsistency. 1234.56, 1,234.56, and 1234.560 are three different token sequences for one quantity. If half your RAG corpus uses one convention and your query uses another, your lexical retrieval scores and your embedding vectors both drift for no semantic reason.

What breaks in production because of this?

Five failure modes I have actually watched happen:

• Extraction off-by-a-digit on long IDs. Order numbers, account numbers, and IBANs get copied with one wrong digit. Copying a 3-digit chunk is atomic and reliable; copying across a boundary where the source chunking differs from the natural output chunking is where digits get dropped.
• Silent unit drift in aggregation. Ask a model to sum a column of unformatted numbers with mixed digit lengths and errors correlate with length mismatch, not with magnitude.
• Retrieval misses on numeric queries. "revenue of 1234567" fails to match a chunk containing "1,234,567" more often than semantic similarity would predict, because neither the sparse nor the dense path sees shared subunits.
• Meaningless numeric confidence. If you use logprobs to gauge confidence in an extracted number, remember the logprob is over a chunk, not a digit. P(token="456") conflates uncertainty about three digits into one scalar. Per-digit calibration is not recoverable from it.
• Cache-key churn. If your prompt builder formats numbers inconsistently between turns, you break prompt-cache prefix matches on otherwise-identical context.

How should I format numbers in prompts for Claude and GPT-5?

Normalize once, at prompt-build time, and make it a pure function you can test:

from decimal import Decimal

def norm_number(x, decimals: int = 2) -> str:
    """Stable, place-value-aligned numeric rendering for LLM prompts."""
    d = Decimal(str(x)).quantize(Decimal(10) ** -decimals)
    return f"{d:,}"                      # 1234567.5 -> '1,234,567.50'

def norm_id(s: str) -> str:
    """IDs are identifiers, not quantities: group in fixed 4s, never comma them."""
    s = "".join(ch for ch in s if ch.isalnum())
    return "-".join(s[i:i+4] for i in range(0, len(s), 4))

The rules behind it:

1. Quantities get thousands separators. Always. Both in prompts and in retrieval corpora — normalize at ingest, not just at query time.
2. Fix decimal places per field. Never let 1234.5 and 1,234.50 coexist for the same field. Round with Decimal, not float, so you do not emit 0.30000000000000004 into a prompt.
3. Identifiers are not quantities. Do not comma-group an order number — that implies magnitude. Use fixed-width dashed groups so chunking is stable regardless of length.
4. Emit units as separate tokens: 1,234 ms, not 1234ms. Gluing a unit onto a digit run changes the boundary behavior of the fragment.
5. Keep the format identical between ingest and query, or you pay for it in retrieval recall and in cache misses.

When does none of this matter?

When you route the arithmetic to a tool — which you should, for anything load-bearing.

A code-execution or calculator tool turns the model's job from "compute" into "extract operands and call a function." Extraction is a copying task, and copying is where 3-digit chunks are actually an advantage: fewer tokens to emit, fewer chances to drop one. Formatting hygiene still helps the extraction step, but the correctness guarantee comes from the tool.

Treat it as a hierarchy: tool-call for real arithmetic; formatting normalization for the numbers the model must read, compare, or copy; digit-level tokenizers only if you control the model and your workload is math-dominant enough to pay the 3x sequence cost.

How do I measure this on my own stack?

Do not take the effect size on faith — it varies by model and by task. Build a 15-minute eval:

import random

random.seed(0)
cases = [(random.randint(10**6, 10**9), random.randint(10**3, 10**6))
         for _ in range(200)]

def prompt(a, b, fmt):
    f = (lambda n: f"{n:,}") if fmt == "comma" else str
    return f"Compute {f(a)} + {f(b)}. Reply with only the number."

# Run both arms, strip separators from the reply, compare to a + b.
# Bucket accuracy by digit-length delta: len(str(a)) - len(str(b)).

The bucketing is the point. A single aggregate accuracy number hides the effect; the gap between formatted and raw shows up most sharply in the high-length-delta buckets, which is exactly what the alignment theory predicts. If your data does not show that shape, your bottleneck is somewhere else and you should stop optimizing formatting.

So why do commas fix LLM arithmetic?

Because LLM tokenizers group digits left-to-right in runs of up to three, which makes token boundaries a function of a number's total length rather than of place value — so the same trailing digits get different tokens in numbers of different magnitude, and carries in addition have to cross boundaries at offsets the model must infer per example. A thousands separator is a non-digit character that forces the split right-to-left, so every group becomes a stable power-of-1000 band that means the same thing across numbers. The fix is nearly free: normalize number formatting once at prompt-build and ingest time, keep it identical on both sides of retrieval, and send anything that actually has to be correct to a tool.