We alerted on errors. The silent failure was a truncated answer.

Every monitor was green. HTTP 200s, latency inside SLO, zero exceptions. Meanwhile users were getting half an answer and cut-off JSON, and nothing on our side had noticed, because a truncated completion is not an error.

The page never fired. That was the whole problem. Support forwarded a handful of complaints that answers were "getting cut off," and I opened the dashboards expecting to see the cause. Error rate flat at zero. Latency well under budget. No exceptions in the logs, no failed calls, no 5xx. By every signal we watched, the service was healthy. The signals were wrong. It was handing back broken output with a 200 stamped on it, and none of our monitors were built to notice.

What the monitors were watching, and what they missed

We had inherited the standard web-service monitoring shape: watch HTTP status, watch latency, watch exceptions. That shape assumes failure looks like an error. For a normal API that holds. For an LLM call it does not, because the most common user-facing failure throws nothing at all.

The failure was truncation. When a completion hits the max_tokens ceiling, the model stops mid-thought and hands back what it had so far. The provider returns a perfectly valid response object with a 200. The only tell is a field in the payload: finish_reason comes back as length instead of stop. No exception, no error code, no latency anomaly. A response that stopped because it ran out of room looks identical, at the transport layer, to one that finished naturally.

So the answer that got cut in half was a 200. The JSON blob that ended after {"status": "app was a 200. Our completeness was failing and our monitors were structurally blind to it, because they were counting the wrong kind of failure.

What it cost before we caught it

This ran for the better part of two weeks before the pattern was clear enough to act on. Best I can reconstruct, somewhere around 3 to 4 percent of completions were truncating, and a big share of those were the long-output ones: multi-step answers, long summaries, and the structured JSON responses another service consumed downstream.

The JSON case was the expensive one. A downstream service parsed those completions. A truncated blob is invalid JSON, so that service caught a parse exception, swallowed it, and fell back to a default. No page there either. So one silent failure (truncation with a 200) fed a second silent failure (a swallowed parse error and a default value), and the only place reality surfaced was a slow trickle of user complaints. Two weeks of that is a lot of quietly wrong answers.

Treat finish_reason as a first-class signal

The fix starts with promoting finish_reason from a field nobody reads to a metric you alert on. Every completion, check why it stopped, and emit that. A rising rate of length finishes is a truncation incident in progress, and it will show up here long before support forwards the first complaint.

from collections import Counter

_finish_reasons = Counter()

def record_finish_reason(response) -> str:
    # OpenAI-style: response.choices[0].finish_reason
    # "stop" = natural end, "length" = hit max_tokens (truncated), plus others
    reason = response.choices[0].finish_reason or "unknown"
    _finish_reasons[reason] += 1
    return reason

def truncation_rate() -> float:
    total = sum(_finish_reasons.values())
    return _finish_reasons["length"] / total if total else 0.0

That is the signal we never had. truncation_rate() is the number that should have paged us on day one.

Monitor completeness, not just finish_reason

Finish reason tells you the model ran out of room. It does not tell you whether the answer the user got was usable. For that, check the output itself. For structured responses, the sharpest signal is whether the payload parses and carries the fields you require. A valid-JSON-parse rate and an expected-fields-present rate turn "the answer looks complete" into a number you can alert on.

import json

_completeness = {"parse_ok": 0, "parse_fail": 0, "fields_ok": 0, "fields_missing": 0}
REQUIRED_FIELDS = ("status", "summary", "items")

def check_completeness(text: str, structured: bool) -> dict:
    result = {"truncated_json": False, "missing_fields": []}
    if not structured:
        return result
    try:
        obj = json.loads(text)
        _completeness["parse_ok"] += 1
    except json.JSONDecodeError:
        # a cut-off JSON blob lands here: valid 200, unusable payload
        _completeness["parse_fail"] += 1
        result["truncated_json"] = True
        return result
    missing = [f for f in REQUIRED_FIELDS if f not in obj]
    if missing:
        _completeness["fields_missing"] += 1
        result["missing_fields"] = missing
    else:
        _completeness["fields_ok"] += 1
    return result

def json_parse_rate() -> float:
    total = _completeness["parse_ok"] + _completeness["parse_fail"]
    return _completeness["parse_ok"] / total if total else 1.0

Now a truncated JSON response is not a downstream default swallowed in silence. It is a parse_fail, counted, and it moves a rate you can wake someone up over.

Set max_tokens from the data, not a guess

The last piece is the setting that caused it. Our max_tokens was a round number somebody picked once and never revisited. It was too low for the long-output traffic, which is exactly the traffic that truncates. The right value is not a guess, it is the measured distribution of actual output lengths with headroom on top.

def recommend_max_tokens(observed_output_lengths, headroom=1.2, pct=99):
    # size the ceiling off the p99 real output, not a round number
    s = sorted(observed_output_lengths)
    p99 = s[min(len(s) - 1, int(pct / 100 * len(s)))]
    return int(p99 * headroom)

Feed it a few days of real completion lengths and it hands back a ceiling that clears your p99 output with room to spare, instead of a number that clips your longest and most important answers. Reserving output budget you rarely use costs a little on the concurrency side, so this is a trade to make with eyes open, not a free lever. But clipping real answers costs correctness, and correctness is not a thing to save money on quietly.

Once truncation rate, JSON parse rate, and expected-fields-present were on the wall, the incident stopped being a support-ticket archaeology exercise. The next time a prompt change pushed outputs longer and truncation started climbing, the rate moved within minutes and we caught it before a single user did.

What I'd page on

Different checklist from the token-cost write-up. This one is about a failure that never throws.

• Truncation rate (finish_reason == "length"). The headline. Fraction of completions that stopped because they hit max_tokens. Warn at a low single-digit percent, page on a sharp climb. This fires on the exact failure that green error dashboards hide.
• JSON parse-success rate. For any structured output, the fraction that parses. A drop is truncated or malformed payloads reaching consumers. Page on it, because the downstream service will swallow the failure and you will not hear about it otherwise.
• Expected-fields-present rate. Of the payloads that do parse, the fraction carrying every required field. Catches the answer that parsed but came back half-filled.
• Output-length distribution vs max_tokens. Plot p50 / p95 / p99 output length against the ceiling. When p99 marches toward the ceiling, truncation is about to start. This is the leading indicator.
• Downstream default-fallback rate. How often the consuming service fell back to a default because it could not use the response. A silent failure feeding a silent failure is the worst case, so surface the second one too.
• Completeness by route. Truncation is not uniform. Break the truncation rate out by endpoint or prompt template so the long-output routes, the ones that actually clip, are not averaged into invisibility by the short ones.