Textbook: The Cliff Edge of Streaming Output
Source version of CodeSmith:
v0.5.0(commit3a74c82f). All paths are relative to the repo root; line numbers refer to this version.
Intended audience: readers who have written streaming parsers and assume a dropped connection always raises an error.
When model output slams into the token limit, the most dangerous part is not what got cut off — it is what didn't get cut cleanly: a half-written tool call whose argument JSON breaks off mid-string, looking "close enough to work." v0.5.0 handles this cliff with two commits landed within the same hour of the same day: P0-2 refuses to repair, and refuses to execute, truncated tool calls; P0-3 establishes a proof of termination for streaming. In total, 20 files, +1418/−55 lines, 22 new tests.
The Execution Ground of Half a Sentence
First, look at where the old code was dangerous. CodeSmith has an "argument repair ladder" — Article 16 covered its five rungs: close the brackets, drop the trailing commas, and rescue "almost-legal" JSON back to life. For a turn the model finished writing and merely fumbled, this is a handrail.
But when the turn was severed by the output limit, the handrail turns into a weapon. The module documentation is blunt about it (crates/agent-runtime/src/engine/turn/truncation.rs:1-13): at that point, tool arguments that fail to parse are almost certainly half-written and cut off mid-way, not a case of format fumbling; "repairing" the fragments into executable arguments means finishing the model's half-sentence on its behalf — for shell-type tools, what gets executed is a command the model never finished writing, and that is a security problem, not a quality problem.
A concrete example. Suppose this fragment of an argument arrives in the stream: {"command": "rm -rf /tmp/leg — the string is unclosed, the right brace missing. The repair ladder would eagerly supply the quote and the brace, and an executable command is born. Yet what the model actually meant to write might have been /tmp/legacy-cleanup — or an entirely different sentence.
The deciding predicate is short (truncation.rs:45): normalize the stop_reason, and if it hits length or max_tokens (the two wire spellings), the turn is judged "truncated by the output limit." There is also a provider exemption table — some vendors misreport length even on turns that ended normally, and the exemption table keeps the gate open for them. For now the table is deliberately empty, waiting for actual repeat offenders to be caught before entries are added (the mechanism is pinned down by tests).
The "arguments truncated" sentinel is even simpler: the engine's stream reducer parses the argument buffer strictly, and on failure sets it to Value::Null (crates/agent-runtime/src/engine/turn/stream.rs:190-198). The dispatch-side verdict is then just the AND of two conditions: the turn was truncated, and this call's arguments are Null.
The repair ladder was not deleted; it was parameterized into an explicit policy (crates/agent-runtime/src/engine/dispatch.rs:149-168): RepairAllowed runs the full ladder as before, for turns that were fully written; StrictOnly does strict parsing only and yields None on failure — mandatory for truncated turns.
The tests pin this divide down: given the same fragmentary input, RepairAllowed rescues it, while StrictOnly must return a parse failure.
Reject and Resend
The gate sits at the batch-dispatch stage (crates/agent-runtime/src/engine/turn/batches.rs:164-189), and its placement carries three deliberate choices.
First, rejection happens before the loop guard does its bookkeeping — the model never finished writing this call, so it shouldn't count as one of its attempts. Second, speculatively early-started background tasks are canceled on the spot; fragmentary arguments may not slip into any execution path. Third, the comment states explicitly: like a guard interception, this counts as an "intervention," not an "execution," and stays out of the error-escalation circuit-breaker statistics.
The feedback handed back to the model is a three-part notice in house-rules style (batches.rs:49-73): state plainly NOT executed, and name the stop_reason; demand a complete resend of this call; then append one self-help suggestion — if the arguments are too long, shorten or split them so the call fits inside the output limit. The manuals of the three shell tools had this contract written into them at the same time: teach first, block second.
The exemptions run three layers: calls in the same turn whose arguments parsed completely execute as usual; providers on the exemption table don't trip the gate; turns without length/max_tokens keep the old lenient dispatch path. The blast radius is cut precisely to the "beheaded and fragmentary" calls.
The Evidence Gate: No Proof, No Ending
P0-2's gate carries a hidden premise: the engine must be able to see the true stop_reason. P0-3 uncovered an awkward fact — it can't.
The problem lies in the rig-core 0.39 middle layer (crates/providers/src/rig_adapter/stream.rs:28-45): the OpenAI-compatible path compresses the wire's finish_reason into a crate-private two-value enum and skips [DONE] outright; the Anthropic path reads stop_reason and then throws the string away.
Worse, when an SSE connection dies silently, the adapter forges MessageDelta and MessageStop, disguising a broken stream as a clean, normal ending.
The one signal that survived through rig's public API is the Final payload, which rig produces only when the SSE loop closes without error. So P0-3 enthroned it as the "proof of termination": if the stream ends without a Final ever having been seen, treat it as a connection interruption across the board and throw a retryable error upward — never, under any circumstances, forge a MessageStop.
The engine side carries one more isomorphic defense (crates/agent-runtime/src/engine/turn/stream.rs:460-486): for any other LlmClient implementation, if the stream runs dry without a MessageStop arriving, the old "treat it as a clean completion" fallback has been deleted, and the interruption exit is taken instead.
The exit splits into two branches: not a single byte received, Empty — a transparent resend, no content, no billing; part of it received, Partial — presented as-is, no retry, because the model has already been billed, and retrying would mean double billing plus losing half a turn.
Error wording is itself an exam question: two error strings are pinned verbatim by tests and must land in the engine's text classifier's "retryable Network" category — let the phrasing drift, and a broken stream degrades from automatic retry to hard failure. The test comment confesses: the providers crate isn't on the dependency graph, so the strings can only be copied into the test and flagged keep in sync.
The Derived stop_reason
So what about the stop_reason rig swallowed? P0-3's answer: if this layer cannot prove the wire value, then derive what this layer can prove (crates/providers/src/rig_adapter/convert.rs:278-304):
pub(crate) fn derive_stop_reason(saw_tool_use: bool, usage: &Usage, max_tokens: u32) -> String {
if max_tokens > 0 && usage.output_tokens >= max_tokens {
"max_tokens".to_string()
} else if saw_tool_use {
"tool_use".to_string()
} else {
"end_turn".to_string()
}
}
In plain language: if billed output has pushed right up against the requested output limit, that is hard evidence of truncation — providers, when truncating, always stop exactly at the limit; if a tool block was submitted, it's tool_use; otherwise, end_turn. The CHANGELOG spells out why this step matters: it arms P0-2's truncation gate on real traffic — before this, rig-based providers never reported max_tokens, and the gate existed in name only.
The rare false positive — a turn that happens to end naturally exactly at the limit — is harmless: the gate rejects only calls whose arguments fail to parse, and a normal call with intact arguments walks through unscathed.
Two Pieces of Engineering Picked Up Along the Way
The same commit tucked in two other items. One is the h2→h1 fallback (crates/providers/src/rig_adapter/http_fallback.rs): certain CDN edge nodes negotiate HTTP/2 via ALPN and then crash at the protocol layer.
Detection walks the entire error chain doing string matching (five markers including http2, goaway, preface) — the h2 crate isn't a direct dependency, so structured matching isn't possible; and the cost of a false positive is merely one harmless HTTP/1.1 replay. One hit sets a sticky flag, and from then on all requests go straight to h1.
The other is the structuring of the retry banner: which retry attempt it is, the countdown to how many seconds, and why — being process-level globals is deliberate: a subagent's background retries must not light up the foreground's status bar.
Conclusion: Railings on Both Sides of the Cliff
String the mechanisms into a single chain:
- The stream reducer parses the argument buffer strictly and sets Null on failure — the truncation sentinel;
- The turn's stop_reason hits length/max_tokens and the arguments are Null: no repair, no execution, no counted attempt; the model is asked to resend the call in full;
- If the stream ends without a proof of termination (no Final, no MessageStop), treat it as an interruption across the board: zero content means a transparent resend, half content is presented as-is;
- The stop_reason is derived from billed output touching the limit, and that is what arms the truncation gate on real traffic;
- The leftover rough edges — the h2 protocol crash, retry visibility — get closed off in the same sweep.
Three features, and their prices:
- Refusing to finish the model's sentences: half a sentence never reaches the shell — the price is that a legitimate truncated call can only be resent in full, costing long-argument turns an extra round;
- Proof before ending: a broken stream no longer masquerades as completion — the price is dependence on one public signal from the middle layer; if the middle layer changes, the evidence hunt starts over;
- Half-presented, not retried: respect for the already-billed half turn — the price is that if a Partial turn's crucial content happened to sit in the lost tail, the model has to discover that on its own the next round.
Top comments (0)