A paper posted to arXiv on June 1 this year, From Confident Closing to Silent Failure by Laksh Advani, puts a number on something everyone running coding agents has seen. Across 9,876 tau2-bench trajectories and 1,879 AppWorld trajectories, the agent asserted task completion while the environment state said otherwise in 45 to 48 percent of failures in the single-control tau2-bench domains, and in 75.8 percent of the AppWorld coding-agent failures where the agent made an explicit status claim. The part that matters more for anyone building tooling is why the usual fix does not work. No LLM judge configuration the author tried got above 0.65 AUROC on tau2-bench, and the same judges managed 0.54 on AppWorld, because judges "rely on surface completion proxies", chiefly "confident closing language", rather than verified state changes.
Transluce's Docent team measured the same thing in production traffic on August 4. Out of 8,600 real coding agent sessions, 1.8 percent of the public SWE-chat sessions contained a severe case of what they call overselling. Their example is an agent that declared "Integration Tests Converted" with a check mark after deleting one of the tests.
Both studies land on the same conclusion: the agent's own report of completion is generator output, and any monitor that reads the report instead of the state will be fooled by confident prose.
What a terminal can know that a transcript cannot
A shell.online share is a live terminal behind a browser link, and since 0.14 an agent can be invited into that terminal with a scoped, revocable grant. That puts the question above in a very concrete form. When an agent submits a command through the terminal and then says "done", what does the terminal itself have as evidence?
This is what the Refstream handoff layer is for. Refstream.js is the optional renderer we publish from the same GitHub account as shell.online. It is still an alpha and xterm.js stays the default. Its v0.1.0-alpha.5 release went out this morning, and shell.online 0.15.1, tagged a few hours later, vendors it.
Each handoff is a task record with a stable ID and one of five statuses: waiting, needs_attention, completed, collected, cancelled. The rule that makes it useful is in src/tasks.ts of the refstream.js tree at the vendored revision. A task can only move to completed through a method that throws Completion requires an explicit source when no source is given, and the serializer rejects any restored record that is completed or collected without one: Completed task has no completion evidence. Output arriving on the PTY does not touch the status at all. It flips a single boolean, outputObserved, and nothing else.
The source has exactly three allowed values, and the record keeps which one it was:
-
shell: an OSC 133 command boundary arrived for this specific command. The exit code is retained, and completion does not imply exit code zero. -
host: the embedding application calledcompleteTaskfrom a real application-completion event. -
agent_observed: the visiting agent read the answer on screen and explicitly said so. The docs describe this as "an observation, not independent confirmation by the host."
The interesting case is collecting a task that has no completion evidence yet. The session's collectTask stores a bounded screen excerpt, marks it truncated, and leaves the task pending. If the agent wants to close it, it has to pass the actual answer text along with the agent_observed label and a sequence number from a fresh read. Pass the label without an answer and the call fails with the message A quiet screen is not completion. So the provenance of every "done" survives into the record that the next agent, or the person who opened the share, will read.
What shell.online does and does not supply
This is where the honest limits are. The shell.online web client mounts these tools in web/refstream-tools.ts and keeps the task records across a reload in tab-local storage, but at the current main commit it does not call completeTask or reportApplicationState anywhere. shell.online is not an application integration. It is a PTY over an encrypted WebSocket, and it has no more idea than xterm.js does whether the program on the other end finished.
The CLI also does not inject OSC 133 markers. A grep of the Go sources on main today finds none. Markers come from your shell configuration, or from nothing. That has a direct consequence in the handoff code: a kind: command ask is refused with A command needs an empty marked shell prompt and one line; nothing was sent unless the terminal is sitting at a marked prompt. Without markers there is no shell completion available through shell.online at all, only agent_observed, and the record says so. That is the whole point. A reader looking at a task record can tell whether the shell reported the exit or the agent merely claimed it, which is the distinction both papers say monitors currently cannot make.
The same guard on the way in
Completion provenance is the second half of a model whose first half is about typing. In src/session.ts, ask demands an expectedSequence from a recent read and refuses if the terminal has moved. It sets the input owner to agent, pastes the prompt, and then checks again that the owner is still agent and nothing local intervened before sending Enter. Local keystrokes or IME composition in between abort the submission and push the task into needs_attention with a note that delivery may be partial. Restoring a snapshot deliberately downgrades a saved agent owner, with the comment "A saved ownership flag cannot authorize Enter in a newly attached process."
Sequence-gated writes and source-labelled completions are two ends of the same discipline: the terminal only believes what it saw happen.
For Pilot Protocol this sits one layer up from what we do. Our overlay gives an agent a stable virtual address and an authenticated, encrypted tunnel to a peer, and the reference implementation can prove which node sent a message. It cannot prove the message is true. A2A or MCP traffic over a Pilot tunnel that says "task complete" is still a claim, and the handoff records above are an example of what the application layer has to add: keep the claim, keep who made it, and never let output or silence promote it.
Top comments (0)