GPT-4o’s Stop Parameter Fails on Multi-Token Sequences

#ai #debugging #llm #machinelearning

You set stop: ["###END###"], the model produces ###END### in the middle of its output, and generation carries straight on past it. The response comes back with finish_reason: "stop" or "length" and your delimiter sitting in the middle of the text. Nothing errored. This page is about why, and there are three distinct causes with three different fixes.

The symptom

The request looks like this and is not malformed:

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "List three fruits, then write ###END###"}
    ],
    "stop": ["###END###"]
  }'

The first diagnostic to run is not about stop at all. Look at finish_reason on the returned choice. If it is "stop", generation ended by the model’s own end token or by your stop sequence and you cannot tell which from that field alone. If it is "length", your stop never fired and the run hit max_tokens. The full vocabulary of that field is in what finish_reason can return, and reading it first saves debugging the wrong layer.

What stop is documented to do

OpenAI documents stop in the Chat Completions API reference as up to four sequences at which the API will stop generating further tokens, with the returned text not containing the stop sequence. Three things follow from that description and all three matter here.

It is a limit of four. A fifth is rejected, and the limit is on the array, not on total length. See the stop parameter’s documented limits.
The sequence is removed from the output. So if your stop fired, you will not see the string in the text. Seeing the string in the text is proof it did not fire.
It stops generation, not the model. The model is not told about your stop sequences. They are not in the prompt, they do not bias sampling, and the model has no incentive to produce them. stop is a server-side check applied to output the model was going to produce anyway.

That last point is the one that reframes the whole problem. A stop sequence is a scissors, not an instruction. If you want the model to emit a delimiter, you have to ask for it in the prompt, and then the scissors can cut there.

It follows that stop can only ever save you money and time, never buy you structure. The tokens before the delimiter were generated and billed; the ones after it were not. On a request where the model would have rambled for another three hundred tokens after finishing, that is a real saving. On a request where you need the output to have a shape, it is not a mechanism at all — and the cases where people are most frustrated by it are almost always cases where it was being asked to do the second job.

Cause 1: the string is never emitted exactly

This is the most common cause by a wide margin and it has nothing to do with tokenization. The check is for an exact substring. The model, asked for ###END###, produces one of:

### END ###          spaces inserted
###END###.           trailing punctuation
**###END###**        markdown emphasis wrapped around it
###end###            case changed
### END              the closing hashes dropped

None of those contains ###END### as a substring except the second, which does — and that one stops correctly, leaving nothing after it. The others sail through. The model is a next-token predictor; it produced the most plausible continuation of a request to write a delimiter, and plausible continuations of markdown-ish text include markdown decoration around it.

Fix. Choose a delimiter the model is likely to reproduce byte-exactly, and check what it actually produced before blaming the parameter. Run the same request once without stop and read the raw output. If your string is not in there verbatim, the parameter was never going to fire and no amount of adjusting it will help.

Cause 2: whitespace is inside the token

This is the tokenization one, and it is real but narrower than the folklore suggests. GPT-4o uses the o200k_base BPE vocabulary, in which leading whitespace is part of the token rather than separate from it — the word END at the start of a line and the same word after a space are different tokens. Newlines behave the same way: \n\n is frequently a single token rather than two of \n.

The consequence for stop is specific. A stop of "\n" asks to cut at a single newline. If the model produced the double-newline token, the emitted text does contain a newline and the server-side substring check will find it — but the unit of generation was the pair, so the cut lands inside a token boundary and what you get back is trimmed in a way that can look arbitrary. More practically, a stop of "Observation:" will not match output where the model emitted " Observation:" with a leading space if you are comparing the trimmed strings yourself downstream.

Fix. Include both variants in the array — you have four slots — and never write a stop sequence that begins mid-word:

"stop": ["\nObservation:", "Observation:", "\n\nObservation:"]

If you want to see exactly how your delimiter segments, run it through the o200k_base encoding. The tokenizer is the ground truth and it is a two-line check; see what o200k_base changed.

Cause 3: stop is ignored on this endpoint

Before debugging any of the above, confirm the parameter applies at all. There are several documented cases where it does not, and in most of them the request succeeds and the field is simply ignored, which reproduces the symptom exactly.

Reasoning models. OpenAI’s o-series does not accept the same sampling parameters as the GPT line. If you switched model and the stop sequence stopped working, this is the first thing to check. See what the o-series does and does not accept.
Structured outputs. With response_format set to a JSON schema, the output is constrained by a grammar and cutting it at an arbitrary string would produce invalid JSON. Do not combine the two. See how the schema constraint works.
Tool calls. When the model is emitting a tool_calls block rather than content, your stop string is not being matched against the arguments JSON in the way you expect.
A gateway or SDK dropping the field. If you are not talking directly to api.openai.com, log the outbound body and confirm stop is in it.

Which parameters each model family accepts is versioned and changes. OpenAI’s API reference marks unsupported parameters per model family; treat the list above as the shape of the problem and the reference as the current answer.

The fix that does not depend on tokenization

stop is a convenience, and every cause above is a way for a string match to miss. If the boundary matters to your program, do not make correctness depend on it.

Ask for the delimiter explicitly in the prompt, in the exact form you will match: “When you have finished, output the line ###END### and nothing after it.” The model has to produce it before anything can cut on it.
Set stop as the optimisation it is — it saves you the tokens after the delimiter, and those tokens are billed. Include the whitespace variants.
Parse defensively on your side regardless. Split on the delimiter, take the first segment, and strip. If the delimiter is absent, you still have the whole output rather than an exception.
Check finish_reason on every response and treat "length" as a failure to handle rather than a result to use — it means the output is truncated mid-thought.
If the output must have a fixed shape, use structured outputs instead of a delimiter. A schema-constrained response cannot run past its closing brace, which is the guarantee a stop sequence never offered.