A 200 OK Can Still Be a Broken Answer: Check Why Your LLM Stopped

Here's a bug that never shows up in your error dashboard: the LLM call succeeds, returns HTTP 200, your code takes the text and moves on... and the text is half an answer. Or half a JSON object. Or a polite refusal your app renders as if it were the result.

Nothing threw. Nothing retried. Your monitoring says everything is green.

The fix is one field that every major LLM API already gives you, and that a lot of code never reads: the reason the model stopped generating.

(This morning's post was about RAG chunking, what goes into the model. This one is about the other end: what you do with what comes out.)

The field: finish_reason / stop_reason

Every response tells you why generation ended.

OpenAI Chat Completions puts it on each choice as finish_reason (API reference):

Value	Meaning
stop	Natural stop point, or one of your stop sequences
length	The maximum number of tokens you allowed was reached
content_filter	Content was omitted because of a content-filter flag
tool_calls	The model called a tool

Anthropic Messages returns stop_reason on every successful response (Handling stop reasons):

Value	Meaning
end_turn	Claude finished naturally
max_tokens	Hit your max_tokens limit
stop_sequence	Emitted one of your stop_sequences
tool_use	Claude is calling a tool
pause_turn	A server-tool loop hit its iteration limit; send the content back to continue
refusal	Claude declined to respond
model_context_window_exceeded	The response filled the model's context window

Only one value in each table means "this is a complete, normal answer": stop and end_turn. Everything else needs a branch in your code.

Why your try/except doesn't catch this

Anthropic's docs make the distinction explicit: stop reasons are part of successful responses. Errors are HTTP 4xx/5xx. A truncated answer is not an error, so it never reaches your except block.

That includes refusals. Per the same docs, a refusal is returned "as a normal HTTP 200 response, not an error." If your code only checks the status code, a refusal looks exactly like a success.

The four cases worth handling

1. Truncation (length / max_tokens)

The output hit your cap mid-sentence. This bites hardest with structured output: a JSON object cut off at the token limit won't parse, and if your parser is lenient or you fall back to "best effort," you can end up storing garbage.

What to do:

• Don't use the output as if it were complete.
• Raise the limit, or continue the response. Anthropic's docs show a loop that appends the partial text as an assistant message and asks the model to continue, stopping once stop_reason is no longer max_tokens.
• If what got cut off was a tool call, Anthropic recommends simply retrying with a higher max_tokens. A half-written tool call is useless.
• If you just show the text to a user, at least mark it. The docs' example appends a note like [Response truncated due to max_tokens limit].

2. Reasoning models eating the budget

On OpenAI, max_completion_tokens is "an upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens." With a reasoning model, a tight cap can be consumed by thinking before much of the visible answer gets written. The response still finishes with length.

If you migrated from a non-reasoning model and kept the same small cap, check your finish_reason distribution. The usage object also reports completion_tokens_details.reasoning_tokens, so you can see how much of the budget went to reasoning.

3. Refusals and content filters (refusal / content_filter)

These deserve their own path: log them, show the user something honest, and decide whether a retry makes sense. Anthropic's docs suggest reading stop_details (which names the policy category) and retrying on a fallback model for some of their models.

4. Tool and agent loops (tool_calls / tool_use / pause_turn)

In an agent loop, the stop reason is the control signal: run the tool and continue on tool_use, send the content back on pause_turn, and only treat end_turn as "done." A loop that stops on anything other than these is a loop that silently quits halfway.

A small handler

Here's the shape I use, written for the Anthropic Python SDK (the OpenAI version is the same idea on response.choices[0].finish_reason):

import logging

log = logging.getLogger("llm")

class Truncated(Exception): ...
class Refused(Exception): ...

def text_or_raise(response):
    reason = response.stop_reason
    log.info("llm_call", extra={
        "stop_reason": reason,
        "output_tokens": response.usage.output_tokens,
    })

    if reason in ("end_turn", "stop_sequence"):
        return "".join(b.text for b in response.content if b.type == "text")
    if reason in ("max_tokens", "model_context_window_exceeded"):
        raise Truncated(reason)        # caller decides: raise limit, continue, or fail
    if reason == "refusal":
        raise Refused(response.stop_details)
    # tool_use / pause_turn belong to your agent loop, not here
    raise ValueError(f"unexpected stop_reason: {reason}")

The point isn't the exceptions. It's that "not a normal stop" becomes loud instead of flowing into your database or UI.

Streaming: the reason arrives last

When you stream, you don't get the stop reason up front. On Anthropic, stop_reason is null in the initial message_start event and is provided in the message_delta event. On OpenAI, each chunk carries finish_reason: null until the final chunk.

So if you stream to a UI, record the stop reason when the stream ends and decide what to do then (for example, show a "response was cut off" notice, or mark the stored message as incomplete).

Make it a metric

Once you log it, the stop reason is one of the cheapest health signals you have:

• Share of length / max_tokens per endpoint. If it's above zero for something that should always finish, your cap is too low or your prompt is producing longer output than you designed for.
• Share of refusal / content_filter. A jump after a prompt change is a regression worth catching.
• Alert on unexpected values, not just on HTTP errors.

If you already trace LLM calls (I wrote about Langfuse for LLM observability a couple of weeks ago), add the stop reason as a field on every generation so you can filter by it.

TL;DR

1. Read finish_reason / stop_reason on every call.
2. Only stop / end_turn (and your own stop sequences) mean "complete."
3. Truncation and refusals come back as HTTP 200, so status-code checks miss them.
4. Reasoning tokens count toward OpenAI's max_completion_tokens.
5. When streaming, the reason arrives at the end; handle it there.
6. Log it and chart it.

References

• Anthropic, Handling stop reasons: https://docs.anthropic.com/en/api/handling-stop-reasons
• OpenAI, Chat Completions API reference (finish_reason, max_completion_tokens, usage): https://platform.openai.com/docs/api-reference/chat/object
• OpenAI, Reasoning models guide: https://platform.openai.com/docs/guides/reasoning
• Anthropic, Streaming messages: https://docs.anthropic.com/en/docs/build-with-claude/streaming

Does your code check why the model stopped, or only whether the HTTP call succeeded? What's the worst truncated output you've shipped?