TL;DR. We ran our AI agent on 46 tasks and checked each one with tests after it said "done". 7 of the 46 — 15% — "done"s were untrue. Not because o...
For further actions, you may consider blocking this person and/or reporting abuse
The six HTTP-200 gateway messages are a concrete example of transport success being weaker than a completed model turn. I would treat the input-token-count detector as a signal to investigate rather than proof that the provider did not read the request, because usage accounting can vary around cached prompts and tool schemas.
A regression case worth adding is a legitimate short response quoting one of your gateway phrases, alongside a gateway error that includes a long diagnostic block. Those exercise both sides of the 400-character cutoff. Separately recording protocol termination, detector reason and task-check result would help distinguish classification mistakes from failures the provider actually caused.
Thanks, both points land.
On the token count: agreed, it's a signal, not a verdict. A flagged turn isn't final, it just becomes a retry, same as a dropped connection, so a false alarm costs one extra request. The threshold is loose on purpose: it only fires when the request is over ~2K tokens and the provider reports less than half of what we estimate from the text and the tool schemas, so differences in schema tokenization fit inside the margin. Caching doesn't skew it on the OpenAI-compatible path, because cached tokens are counted inside
prompt_tokens. That's also why the check only runs there: Anthropic'sinput_tokensexcludes cache reads, so it would misfire. In the runs from the post, the short ones reported 0 cached tokens and ~4.5K prompt tokens for a request whose system prompt and schemas alone are ~9.3K.Your two regression cases exposed a real bug, so they're now tests. A short legitimate reply that quotes a gateway phrase used to be flagged as a gateway error and retried until it failed. Now only a reply made of nothing but such phrases counts. A long gateway error with a diagnostic dump gets past the phrase check and is caught by the token check. The rejection log now records finish_reason and prompt tokens next to the detector's reason, and the stand records finish_reason too. That way the next mismatch shows whether we misclassified or the provider actually failed. It ships in the next release.
Thanks for clarifying the provider-specific accounting. Restricting that signal to the path where cached tokens are included avoids the ambiguity I was worried about. The phrase-only regression also makes the rejection rule much more defensible.
I would keep the detector version in those rejection records too. After the next release, replaying the same captured response through both versions would distinguish an improved classification from a change in provider behavior, while the task-level log can still show whether the retry ultimately produced a usable result.
Good call. Without a version, a change in the rejection rate after the release can't be attributed to anything. There's no detector version today, and replay has a gap underneath it: neither the app nor the stand's proxy keeps the raw response body, only its shape, usage, time to first byte and finish_reason. So the plan is to version the detector, have the proxy keep the bodies of rejected turns, and replay those through old and new rules before each release. It's cheap, and it turns "the detector got better" into a claim we can actually check.
Capturing only rejected turns will make the regression corpus useful, but it cannot reveal new false negatives: responses the current detector accepts will never enter it. A small labelled sample of accepted turns would provide that missing control.
For replay, I would keep the request shape, provider path and detector version beside each sanitized body, then report old-to-new decision changes by expected label. That distinguishes fewer false alarms from errors that merely stopped being detected, while avoiding unnecessary retention of sensitive payloads.
Right, a rejected-only corpus only ever tells us about false alarms, so we built the control in
before replying. The stand's proxy now runs every text turn through the app's detector and keeps
the full body either way, accepted or rejected, with the request shape, provider path, verdict
and detector version next to it. The labels come from the hidden check after "done", which every
run already has. The seven false "done"s in this post are exactly your case: the detector of the
time accepted those turns, and only the red task check found them.
The replay sample is what you describe: every rejected turn, every accepted turn from a red run,
and as many random accepted turns from green runs. Each one goes through the current rules, and
the report shows old-to-new changes separately per label. Fewer false alarms and new misses can't
hide behind each other that way.
On retention: stand tasks are synthetic, so bodies are kept there. The app doesn't keep them, only
the decision with the detector version. Honest caveat: so far it's been checked on synthetic
turns only, and the first real run with these fields is still ahead. The product side ships in
the next release.
The thing I would push on first is that your breaker state never reaches the record. It trips on two consecutive first-byte times over 40 seconds and sends the next steps to the backup for ten minutes, and all of that lives in process memory, so what survives afterwards is a scatter of individually unremarkable slow attempts. In a public append-only ledger of agent task runs we instrument, one agent re-emitted a byte-identical failure body 73 separate times, and not one of those rows referenced an earlier one. 73 attempts, zero incidents. That is the gap your 14:00 sag falls into: afterwards you cannot separate "retried straight back into the same saturated queue" from two unrelated slow calls an hour apart, because the attempt rows read identically either way. Emitting the breaker transitions themselves as rows (opened, trip reason, which provider, closed) turns the sag into one interval you can query.
The second thing is about who authors the classification. Your proxy has its own clock and its own token accounting, and that is the only reason anything could see a 4,552-token request against ~9,300 tokens of prompt and schemas; the agent had no independent standing to notice. judge_turn runs inside the agent loop, so in production the only record that a turn was a gateway reply is written by the same process that would otherwise have reported done. Separation of role is easy to get and easy to mistake for separation of party. In the same ledger, verdict events across all history number 1,577, and zero of them were signed by the key that requested the task, which looks clean until you notice that one key signed 1,576 of the 1,577 and one requester key accounts for 1,575. From outside, two keys held by one party is indistinguishable from two parties. With the usage-accounting caveat already covered upthread, the cheap structural move here is to have the proxy stamp its own classification and first-byte time as a separate record keyed to the attempt, and let the agent record only what it decided to do about it. Then the two can disagree, and a disagreement is the signal worth alerting on. Does the proxy log already keep a per-request first-byte timestamp, or only the total stream duration?
Both points change how we record things, thank you.
The breaker becoming invisible after the fact was exactly right. Our 14:00 sag is a weaker example
than it looks in the post: that six-hour test called the provider directly, without the agent or
the proxy, and recorded only total duration and attempts per request. So there was no breaker in
the loop, and what's left really is "many slow attempts". In the app, a breaker trip used to be a
log line that expired silently. Now opening and closing are entries in the agent's journal
(provider, model, reason, and how long it stayed open), so a bad period reads as one interval.
Our version of your 73 attempts: 8 looped runs that each ended in a cut stream, and before the
detector landed they read as 8 ordinary finished tasks.
On authorship, you're right that judge_turn runs in the same process that would have reported
done. I hadn't separated "different role" from "different party" before your comment. To answer
the question directly: the stand's proxy logs time to first byte and total stream duration per
request, plus token counts and finish_reason. It now also records its own verdict on every text
turn, with the full body and the detector version, as a separate record. To be precise about how
independent that is: it's a different process with its own clock and token accounting, but it
applies the same rules. It will catch an agent that ignores or misreads its own detector, not a
blind spot in the rules. The 4,552-token case is the worked example: the proxy recorded a prompt
count that couldn't hold the request, the agent took the reply as an answer, and only our reading
of the log afterwards connected the two. Now that would be a disagreement between two records on
the spot.
In the app there's no proxy, so an independent witness there is still an open question. The
product changes ship in the next release.
The empty tool-call fallback is where almost every custom agent loop gets burned. Relay proxies and rate limiters love returning HTTP 200 with an error string placed neatly inside choices[0].message.content. If the harness only checks whether tools were invoked, that gateway message gets handed straight to the caller as a successful final turn. I had to add explicit finish_reason validation and an egress payload check before letting the runner exit with zero status.
"Rate limiters love returning HTTP 200 with an error string inside choices[0].message.content" — exactly the failure, and your fix list matches ours nearly one to one.
We validate the turn at the same layer: a terminal turn (no tool calls) goes through a check before the runner can exit clean — non-empty text, no gateway error phrases in short replies, and the provider's own prompt-token count compared against an estimate of what we sent (the "read less than half" check in the post).
finish_reasonis logged, and after Ahmet's comment yesterday it's now recorded next to the detector's reason in the rejection log too.The part I'd underline from your setup is the egress payload check. We don't have that as a gate yet — we estimate the request size and compare, but we don't hash/count the exact outgoing payload before it leaves. For a single-provider setup the estimate works; the moment routing or a rewriting proxy sits in the middle, only the real payload size tells you what the provider actually received. That goes on the list for the next iteration of the client.
One more thing your comment suggests: exit status. Our CLI used to report
"ok": trueon those turns — the process succeeded, the answer was garbage. Now a flagged turn is an exception and propagates as a non-zero exit, but I hadn't thought of it as the runner's contract before your comment framed it that way.