71.53% of Prompt Cut Positions Are Not a Token Boundary of the Text That Follows

Day 73 was about where the reply gets cut. This is about where the prompt does.

The last thing you type is not the last thing the model reads. Your string is turned into tokens first, and if it stops in the middle of one, the token sequence handed to the model is a sequence that cannot occur in front of the text you wanted.

Read it: https://dev48.infy.uk/prompt/day74-token-healing.html

The measurement

Nothing here simulates language ability. The tokenizer, the boundary test, the extension-set lookup and nine healing rules are real string code, and every one of 1,029 cut positions across 20 target texts is enumerated — so every rate is a count, never a sample.

71.53% of cut positions are not a token boundary of the text that follows.

That is the default state of prompt construction, not an edge case. Any time you build a prompt by concatenating a prefix you control with a continuation you do not, you are usually cutting mid-token.

you type:        "The answer is 12"
tokeniser sees:  ["The", " answer", " is", " ", "12"]   <- fine
you type:        "The answer is 1"
tokeniser sees:  ["The", " answer", " is", " ", "1"]
the model has never seen " 1" followed by "2" here; it has seen " 12".

What healing actually does

Token healing backs up to the last real boundary and re-tokenises forward, constraining generation to tokens that extend what you cut. Nine rules are implemented and measured against each other on the same 1,029 positions.

Exactly two numbers on the page are declared rather than measured — both are on sliders, and both are labelled where they are used. Everything else is enumerated.

Why it matters more than it sounds

The failure is silent. You get fluent output; it is simply conditioned on a token sequence that never appears in training. There is no error, no warning, and no way to see it from the outside — which is why the rate matters more than any individual example.

Independently verified, 0 failures.