Token Healing: Why a Trailing Space Wrecks LLM Completions

Prompt a code model with response = requests.get("https: and watch it emit // followed by mush. Prompt it with The capital of France is — trailing space — and watch a model that knows the answer perfectly hedge into the city of Paris... or worse. Neither failure is a knowledge failure. Both are tokenizer failures, and the fix is called token healing.

The prompt ends in the middle of what would normally be a single token. The model has never seen that boundary during training, so you've handed it an off-distribution prefix and asked it to recover.

TL;DR

• Token healing is backing the prompt up by its last token, then constraining the first generated token to candidates whose string starts with the text you removed. The model re-chooses the boundary instead of inheriting yours.
• The root cause is BPE boundary bias: greedy merges mean ":" at the end of https: is a different token than the "://" the model saw a million times in training. Same characters, different token IDs, different distribution.
• The most common production form is a trailing space. Tokenizers attach leading whitespace to words (Paris is one token), so a trailing space in your prompt forces the model to emit the rare space-less variant.
• Fixes are local-inference-only for the token-level version (HF transformers, vLLM, llama.cpp, guidance). For hosted APIs like Claude Opus 4.x or GPT-5.x you cannot touch token IDs — you avoid the failure by never ending a prompt mid-token.
• Anthropic's API enforces part of this for you: an assistant prefill with trailing whitespace is rejected outright.

What is token healing, exactly?

Token healing is a decoding-time correction: before generation, drop the final token(s) of the tokenized prompt, then restrict the first sampled token to the set of vocabulary entries that begin with the dropped characters. Generation resumes from a boundary the model chose rather than one your string concatenation imposed.

It costs one shortened prefill and a vocabulary prefix lookup. It is not a model change, not a fine-tune, and not a prompt trick.

Why does a trailing partial token break completions?

Because BPE tokenization is greedy and context-sensitive at the right edge. Consider a typical GPT-family or Llama-family vocabulary:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# What the model saw during training:
print(tok.tokenize("The capital of France is Paris"))
# [..., 'ĠFrance', 'Ġis', 'ĠParis']        <- ' Paris' is ONE token

# What your prompt gives it:
print(tok.tokenize("The capital of France is "))
# [..., 'ĠFrance', 'Ġis', 'Ġ']             <- a lonely space token

The Ġ prefix is the tokenizer's leading-space marker. Every strong continuation the model learned — Paris, the, located — carries that space inside the token. Your trailing space already consumed it. The model must now emit a token starting with P, and the bare Paris token (no leading space) is comparatively rare in training data: it shows up mid-word, in identifiers, in URLs. You have redirected probability mass into a corner of the vocabulary that mostly encodes weird contexts, and the continuation gets weird to match.

The URL case is the same bug wearing different clothes. "://" is a single token in essentially every modern code-heavy vocabulary. End your prompt at https: and you've split it. The model sees a standalone : after https, which in training usually means a dict key, a type annotation, or a label — not a URL scheme. It continues accordingly.

This generalizes: any prompt whose final characters are a strict prefix of a longer token is off-distribution. Trailing space, trailing _ in get_, trailing ( before a known (" merge, trailing < before </div>, trailing digits mid-number.

How do you implement token healing?

Pop the last token, then constrain the first step. Here is a working single-token implementation for Hugging Face transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, LogitsProcessorList

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto")

# Build once: decoded string for every vocab id.
VOCAB_STRS = [tok.convert_tokens_to_string([t]) for t in
              tok.convert_ids_to_tokens(range(len(tok)))]

class FirstTokenPrefix:
    """Force the first generated token to start with `prefix`."""
    def __init__(self, prefix, start_len):
        self.prefix, self.start_len = prefix, start_len
        self.allowed = torch.tensor(
            [i for i, s in enumerate(VOCAB_STRS) if s.startswith(prefix)],
            dtype=torch.long,
        )

    def __call__(self, input_ids, scores):
        if input_ids.shape[-1] != self.start_len:      # only the first step
            return scores
        mask = torch.full_like(scores, float("-inf"))
        mask[:, self.allowed.to(scores.device)] = 0.0
        return scores + mask


def generate_healed(prompt: str, **kw):
    ids = tok(prompt, add_special_tokens=True).input_ids
    trimmed, last_id = ids[:-1], ids[-1]
    prefix = VOCAB_STRS[last_id]                       # text we removed

    proc = FirstTokenPrefix(prefix, start_len=len(trimmed))
    if len(proc.allowed) == 0:                         # nothing extends it
        trimmed, proc = ids, None                      # fall back, no healing

    out = model.generate(
        torch.tensor([trimmed], device=model.device),
        logits_processor=LogitsProcessorList([proc] if proc else []),
        **kw,
    )
    return tok.decode(out[0][len(trimmed):])

Three details that matter and are easy to get wrong:

Use convert_tokens_to_string, not decode, for the vocab table. decode strips or normalizes whitespace on some tokenizers, and whitespace is precisely the thing you are trying to preserve. Get this wrong and healing silently becomes a no-op on the trailing-space case — the one you most need it for.

The allowed set must include the exact-match token. If the healed choice is the same token you popped, that's a legitimate outcome: the model is telling you your boundary was right. startswith covers this since a string is a prefix of itself.

One pop is not always enough. https:// may tokenize as ['https', '://']; popping :// leaves https, and https itself may be a prefix of https://-style merges in some vocabularies. The greedy variant keeps popping while the accumulated suffix remains a strict prefix of some longer vocabulary entry, capped at 2–3 tokens. Beyond that you're re-generating text the user typed, which is a worse bug than the one you fixed.

Where does this actually bite in production?

Four places, in rough order of how often I've seen it cause real damage:

Code autocomplete. The cursor is by definition mid-token. def get_ , self., import n — every one of these splits a merge. This is why serious FIM setups heal; without it, completion quality at intra-word cursor positions falls off a cliff relative to at-whitespace positions, and the gap is invisible in aggregate benchmarks that evaluate at clean boundaries.

Assistant prefill / response prefilling. You seed the assistant turn with {"name": to force JSON. That trailing space is the classic failure. Prefill with {"name": instead and let the model choose whether the next token carries a space.

Constrained decoding with grammars. A grammar-constrained decoder walks characters, but sampling emits tokens. When the grammar forces a partial token at a boundary, you get the same distribution shift — and now it compounds, because every subsequent constrained step inherits a prefix the model considers unlikely. Good grammar engines handle this internally via token-prefix tries; if you rolled your own constrained decoder, this is probably why your JSON is syntactically valid and semantically drunk.

Stop-sequence resume. You stop on \n\n, do something, then resume by re-sending the truncated text. Truncation rarely lands on a token boundary that matches how the full string would tokenize.

Does token healing apply to Claude and GPT-5 APIs?

Not as an implementation — you cannot pass logit masks or partial-token prefixes over a hosted API. It applies as a constraint on how you build prompts, and the underlying tokenizers have the same boundary bias.

Practical rules for Claude Opus 4.x / Sonnet 4.x and GPT-5.x:

• Never end a prompt or assistant prefill with whitespace. Anthropic's API rejects an assistant prefill with trailing whitespace with an explicit error — treat that as a hint about the whole class, not just the one case it catches.
• End on a natural boundary: complete word, complete punctuation cluster, end of line. Let the model produce the space.
• Prefer structure over surgical prefixes. Instead of prefilling The answer is to force terseness, prefill {"answer": or use a tool/schema. Structure gets you determinism without a mid-token cut.
• Watch templated prompts. f"Context: {ctx}\nQuestion: {q}\nAnswer:" is fine. Adding one trailing space after Answer: — which people do reflexively, because it looks nicer — is the bug.

How much does this actually cost you?

Qualitatively: large where it applies, zero where it doesn't. A prompt ending at a clean boundary is unaffected — healing changes nothing because the popped token is already the model's preferred merge. A prompt ending mid-token can shift the entire continuation, and the effect is worst on short, high-confidence completions where a single first-token error determines the whole answer.

Don't take my word for the magnitude — measure it on your own traffic, because it depends entirely on how often your prompts cut mid-token:

# A/B on your real eval set. Two variants of the SAME prompt.
clean  = template.rstrip()          # ends at a boundary
broken = template.rstrip() + " "    # trailing space

# Score both. If the gap is near zero, your prompts already end cleanly.
# If it isn't, you found free accuracy.

Instrument it: log the final token of every prompt you send, and count how many distinct vocabulary entries have that token's string as a strict prefix. High counts mark exactly the requests that need healing.

The direct answer

Token healing removes the last token of a tokenized prompt and constrains the first generated token to vocabulary entries that start with the removed characters, so the model picks its own token boundary rather than inheriting a broken one. It matters because BPE merges bind leading whitespace and punctuation into single tokens — Paris, ://, get_ — and a prompt that stops partway through one of those merges is a sequence the model effectively never saw in training. The most frequent trigger is a trailing space, which forces the rare no-leading-space variant of the next word. Implement it with a first-step logits mask when you control decoding; when you're on the Claude or GPT-5 API and can't, just make sure every prompt and every assistant prefill ends at a token boundary a human would also consider a natural stopping point.