Auditing Token Budget Assumptions Baked Into Chunking Logic

#ai #debugging #llm #machinelearning

This model’s maximum context length is 8192 tokens, however you requested 8451 tokens. You changed a model string, not a chunker, and now the ingest job is failing on documents it processed last week. The chunker was never measuring what you thought it was measuring; the old model was just forgiving enough to hide it.

The error, and the version with no error

The loud failure is an over-length rejection at the embedding or completion endpoint. The exact wording differs by provider and has changed within providers, so match on the status and the error type rather than on the sentence, but the shape is always the same: a declared maximum, a requested count, and a difference.

The quiet failure is worse and far more common. Some endpoints do not reject an over-length input; they truncate it and return a normal result. Nothing errors. Your chunk went in at 9,000 tokens, 8,192 of it was embedded, and the last ninth of the text — which may have been the part with the answer in it — is in no vector anywhere. Retrieval quality drops by an amount nobody can attribute, months later, with no log line to point at.

Whether a given endpoint truncates or rejects is a per-provider, per-endpoint behaviour and it is not stable across versions. Do not infer it from one successful call. Test it deliberately with a deliberately over-length input and record what happened, because the audit below depends on knowing which one you are dealing with.

Where the assumption was made

Chunkers acquire wrong budgets in a small number of recognisable ways.

Sized in characters with a divisor. The “four characters per token” rule of thumb is an average over English prose under one particular byte-pair encoding. It is wrong for code (dense in punctuation, which fragments), wrong for JSON and XML (same), wrong for languages written in scripts the vocabulary covers thinly, and wrong for text with long identifiers or URLs. A chunker that splits at 4,000 characters and assumes 1,000 tokens can be handing over 2,000. The size of that gap by language is the subject of the tokenizer language tax.
A hardcoded encoding name. Code that asks a tokenizer library for a specific encoding by name, rather than asking it which encoding a given model uses, keeps counting with the old vocabulary after the model moves to a new one. It does not error. It returns confidently wrong numbers. This is the single most common cause and it is the one covered in detail by the tokenizer mismatch bug.
A tokenizer from a different family entirely. A pipeline that counts with one vendor’s tokenizer and sends to another’s is not approximately right; the two vocabularies were built from different corpora and the discrepancy is content-dependent, so it is largest exactly on your unusual documents.
A budget that forgot a term. The constant was derived once, correctly, for a prompt that has since grown a longer system message, a tool schema block, or a second retrieved chunk. The chunk size is unchanged and the total no longer fits.
Overlap counted once. A chunker with a 200-token overlap emits chunks of size, but the retrieval step that concatenates k of them pays for the overlap k times.

Finding every budget in the code

This is a search problem before it is a maths problem, because the assumption is rarely in one place. Sweep for all of these:

# numeric constants that are almost always a token budget
rg -n '\b(256|384|512|768|1000|1024|1536|2000|2048|4000|4096|8000|8191|8192|16384|32000|32768|100000|128000|200000)\b' \
   --glob '!**/node_modules/**' --glob '!**/*.lock'

# the chars-per-token heuristic, in its usual disguises
rg -n '(/\s*4|\* *0\.25|CHARS_PER_TOKEN|approx.*token|estimate.*token)'

# a pinned encoding rather than a model lookup
rg -n '(get_encoding|cl100k|o200k|p50k|r50k|AutoTokenizer\.from_pretrained)'

# where the budget is spent
rg -n '(chunk_size|chunkSize|max_tokens|maxTokens|max_completion_tokens|top_k|topK|overlap)'

For each hit, write down three things: what the number is supposed to bound, which model or endpoint it was derived for, and who enforces it at runtime. Most audits find the same number written in four places — the chunker, the retriever, the prompt builder and a test fixture — and only one of them was updated.

Re-deriving the budget

Replace the constant with an equation that names every term. For a retrieval-augmented completion:

W        = model context window, in tokens
R        = tokens reserved for the completion (your output cap)
S        = system prompt, measured
T        = tool schema block, measured (often the forgotten term)
H        = conversation history you intend to keep
k        = number of chunks retrieved
o        = overlap tokens per chunk
M        = safety margin

usable_for_chunks = W - R - S - T - H - M
chunk_size        = usable_for_chunks / k

# and because retrieved neighbours repeat their overlap:
effective_cost_of_k_chunks = k * chunk_size - (k - 1) * o   # best case, adjacent
                           = k * chunk_size                 # worst case, scattered

# always plan against the worst case.

Two terms are routinely omitted. T, the serialised tool definitions, is invisible in the prompt you wrote but real in the request body, and it grows every time someone adds a tool — its size is worth measuring rather than guessing, as the token cost of a tool schema sets out. M matters because you are counting with a tokenizer and the provider is counting with theirs; even when they agree on the vocabulary they may not agree on the exact overhead per message. Reserve a few percent rather than aiming at the cap.

Two edge cases fall out of the equation and are worth handling explicitly. Documents shorter than chunk_size produce a single chunk and no overlap, so a corpus of short records never exercises the overlap path at all — which is why the bug ships. And a chunker that splits on structure rather than length can emit one oversized chunk from a single unbreakable element: a wide table, a base64 blob, a minified asset. Give the splitter a hard fallback that splits mid-element when no boundary exists inside the budget, and log every time it fires, because a frequently-firing fallback means the boundary rules are wrong for that document type.

For the embedding side, the equation is simpler and the term that bites is different: there is no completion to reserve for, so the budget is the model’s input cap, but the correct target is usually well below it for reasons that have nothing to do with fitting — see migrating chunk size when you change embedding models.

Making it fail loudly next time

Delete every literal token budget and derive it from one module that takes the model identifier as input. One source, one place to change.
Make the counter model-aware: ask the tokenizer library which encoding the target model uses rather than naming an encoding, and for providers that expose a token-counting endpoint, prefer it for the pre-flight check on the real request body.
Assert at the boundary. Before dispatch, count the assembled request and raise if it exceeds the derived budget. An assertion you own gives you a stack trace pointing at the assembler; a provider 400 gives you a string.
Add a fixture that is deliberately pathological — a document of CJK text, one of minified JSON, one of source code — and assert that the chunker’s emitted token counts stay under budget for all three. A test using only English prose will pass forever regardless of how wrong the divisor is.
Log the counted and the billed token totals per request and alert on divergence, which is the general technique described in token count mismatch. A drift that appears the day a provider ships a new tokenizer is then a graph, not an incident.