The Reply Looks Finished. Record the Finish Reason.

The agent returned only a half-written patch file.
The dashboard still painted a green success check.
Observed latency sat inside the usual service band.
No tool span in the trace had failed.
The model had simply hit max tokens.

Nobody stored a generation finish_reason field.
The exported trace still looked fully complete.
The visible reply was not actually finished.
This is a reconstructed incident, not a customer study.
Treat the numbers below as example fixtures only.

What broke in the run

A coding agent started a small refactor patch.
It opened one search_replace tool call next.
The argument JSON ended in the middle of a key.
The runtime classified that break as a model bug.
The real stop condition was a length cap.

Generation spans still carried status: ok flags.
They omitted finish_reason on every record.
They omitted output_tokens and max_tokens too.
Prompt diffs then blamed the wrong system text.
The budget, not the prompt, cut the bytes.

Why green traces hide length stops

Most agent collectors copy HTTP success only.
The provider returned a normal 200 response.
The token stream closed without a socket error.
Your collector stored the visible output text.
It never stored why generation actually stopped.

Four stop classes matter in daily debugging:

• stop: the model chose a natural end
• length: the configured token cap fired
• tool_calls: it requested tools, not prose
• content_filter: the provider blocked the text

Without those labels, later diffs start lying.
A truncated tool call looks like invalid JSON.
A filtered answer looks like a silent model.
A tool_calls stop looks like empty chat output.
Retry counters will not explain any of this.
Clock skew checks will not explain it either.

Artifact: a generation budget record

Do not treat this record as full product telemetry.
Treat it as a required span contract instead.
Every LLM generation writes exactly one record.
Keep the payload small, typed, and portable.

{
  "span_type": "generation",
  "span_id": "gen_8f2c",
  "parent_run_id": "run_19",
  "model_slot": "unspecified-free-slot",
  "finish_reason": "length",
  "max_tokens": 1024,
  "output_tokens": 1024,
  "prompt_tokens": 3188,
  "truncated": true,
  "tool_call_open": true,
  "text_sha256": "c0ffee",
  "ended_at": "2026-09-16T14:02:11Z"
}

Do not invent provider model names in spans.
Store a slot identifier that you control locally.
Map slots outside the trace if names change.
Keep the span format stable across exporters.

truncated must stay a derived boolean field.
Set it when finish_reason equals length.
Also set it when output reaches max_tokens.
Never infer truncation from string length alone.
Unicode, tools, and clippers break that heuristic.

Checker you can run on JSONL

Label this script as an unexecuted example first.
Point it at captured JSONL traces before production.
Fail the run when required fields are missing.

#!/usr/bin/env python3
"""Fail a trace if generation spans omit finish_reason."""
from __future__ import annotations

import hashlib
import json
import sys
from collections import Counter
from typing import Any

REQUIRED = (
    "span_id",
    "finish_reason",
    "max_tokens",
    "output_tokens",
    "truncated",
)

ALLOWED_REASONS = {
    "stop",
    "length",
    "tool_calls",
    "content_filter",
    "unknown",
}


def load_jsonl(path: str) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    with open(path, encoding="utf-8") as handle:
        for line_no, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"bad jsonl:{line_no}:{exc}") from exc
    return rows


def check_span(span: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    span_id = span.get("span_id", "?")
    for key in REQUIRED:
        if key not in span:
            errors.append(f"{span_id}: missing {key}")
    reason = span.get("finish_reason")
    if reason not in ALLOWED_REASONS:
        errors.append(f"{span_id}: bad reason {reason!r}")
    max_tokens = span.get("max_tokens")
    output_tokens = span.get("output_tokens")
    truncated = span.get("truncated")
    if isinstance(max_tokens, int) and isinstance(output_tokens, int):
        hit_cap = output_tokens >= max_tokens and max_tokens > 0
        if hit_cap and reason not in {"length", "unknown"}:
            errors.append(f"{span_id}: cap without length")
        if truncated is True and reason != "length":
            errors.append(f"{span_id}: truncated flag mismatch")
        if truncated is False and reason == "length":
            errors.append(f"{span_id}: length without truncated")
    return errors


def summarize(spans: list[dict[str, Any]]) -> None:
    reasons = Counter(s.get("finish_reason") for s in spans)
    print("finish_reason counts:")
    for key, value in sorted(reasons.items(), key=lambda kv: (-kv[1], str(kv[0]))):
        print(f"  {key}: {value}")
    truncated = sum(1 for s in spans if s.get("truncated") is True)
    print(f"truncated_generations: {truncated}/{len(spans)}")


def main() -> None:
    if len(sys.argv) != 2:
        raise SystemExit("usage: check_finish_reason.py trace.jsonl")
    rows = load_jsonl(sys.argv[1])
    gens = [r for r in rows if r.get("span_type") == "generation"]
    if not gens:
        raise SystemExit("no generation spans")
    errors: list[str] = []
    for span in gens:
        errors.extend(check_span(span))
    summarize(gens)
    if errors:
        print("contract failures:")
        for item in errors:
            print(f"  {item}")
        raise SystemExit(2)
    print("generation contract ok")


if __name__ == "__main__":
    main()

Run the checker against one captured file.

python3 check_finish_reason.py traces/run_19.jsonl

A failing export often prints this shape.

finish_reason counts:
  None: 4
truncated_generations: 0/4
contract failures:
  gen_8f2c: missing finish_reason
  gen_8f2c: missing max_tokens

Repair the exporter before touching prompts.
Missing fields are collector bugs, not model bugs.
Prompt edits cannot restore an omitted stop class.

Decision table after the contract is green

Use this table before anyone rewrites prompts.

finish_reason	truncated	Next debug move
stop	false	Diff prompt text and tool schemas
length	true	Raise cap or shrink context first
tool_calls	false	Validate tool JSON against schema
content_filter	false	Inspect the blocked span text
unknown	any	Repair the exporter, then rerun

Most incident reviews reverse this exact order.
They rewrite the system prompt under time pressure.
They add another retry around the same cap.
They never raise max_tokens on that slot.
They never shrink the stale context window.
The next trace still ends in the same cut.

How length stops corrupt tool JSON

Tool calls fail in a specific, repeatable way.
The model starts a JSON object for arguments.
The cap fires before the closing brace arrives.
Your parser raises JSONDecodeError immediately.
The tool span is then marked as a model failure.

That classification sends people to prompt surgery.
The cheaper fix is often a smaller context pack.
Drop unused files from the prompt first.
Lower retrieval count before raising the cap.
Only then change max_tokens on that slot.

Keep a digest even when logs clip the body.
Many collectors truncate long tool payloads.
The digest still proves the bytes changed.
Store the first two kilobytes plus the hash.

def text_digest(text: str) -> str:
    blob = text.encode("utf-8")
    return hashlib.sha256(blob).hexdigest()

If the digest changes and reason is length,
the model did not quietly change its plan.
The configured cap cut the output bytes.
Diff the budget fields before diffing prose.

Attach the fields in one adapter

Keep provider adapters intentionally thin.
Map native stop fields in a single function.
Write the contract inside your collector path.

Proposed mapping notes for an exporter:

1. Read native finish_reason or stop_reason.
2. Normalize those values to the five tokens.
3. Copy usage counts without rounding tricks.
4. Derive truncated only from those fields.
5. Hash visible text with SHA-256 as shown.
6. Persist the record before the next tool call.

Unknown must remain a first-class stored value.
Do not coerce missing reasons into stop.
That coercion recreates the original incident.
Empty usage objects should stay empty, not zero.
Zero tokens look like a finished empty answer.

Where spare compute actually helps

Local JSONL is enough for the checker script.
A shared box helps when trace files grow large.
MonkeyCode is an open-source coding project.
It provides free model access for agent runs.
It also provides a free server option for replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use spare compute to replay traces, not scoreboards.
Do not publish unnamed model league tables from this.
Run the checker, then inspect finish_reason counts.
Decide whether the cap, not the prompt, is wrong.
If traces already live in your own store, stay there.
The span contract does not require a new vendor.
The free server is optional batch compute only.
A laptop still runs the same JSONL check cleanly.

Limitations and who should skip this

This contract does not measure answer quality.
A clean stop span can still be factually wrong.
Token counters can lag a partial stream flush.
Some providers omit usage on failed requests.
unknown needs a ticket, not a prompt tweak.

Do not compare models using these raw counts.
Do not turn one-run ratios into ranking charts.
A single trace is a sample size of one.
Clock skew still needs a separate measurement.
Retry counts still need their own span fields.

Skip this approach in these four cases:

• Teams that still have no generation logs
• Apps that never issue tool calls at all
• Brokers that forbid storing token fields
• Reviews that only hunt writing style issues

If token counts are blocked by policy rules,
still store finish_reason on every generation.
That one field already splits the debug queue.
Length stops stop looking like prompt failures.

Reusable debug loop

Reuse this loop on the next truncated patch.

1. Export JSONL that includes generation spans.
2. Run check_finish_reason.py on that file.
3. If the contract fails, fix export mapping.
4. If length dominates, cut context or caps.
5. If tool_calls dominates, test tool schemas.
6. Only then diff prompts against prior runs.

The order is the method, not a slogan list.
Skip a step and you will debug ghost failures.
Green HTTP checks are not finish reasons yet.
Record the reason, then change the budget.