Does Your Agent "Know" the Tool Is Useless? It Knows, But It Won't Stop

Luca Harris — Tue, 06 Oct 2026 12:39:57 +0000

On October 5, 2026, a paper appeared on arXiv cs.AI: Seven tool-using agents, faced with a retrieval source that continuously returns useless results, correctly judge "this result is useless" 97%–100% of the time. And then? They keep querying.

This is the thing I've seen recently that best explains today's agent systems: the rupture between model judgment and system behavior. This week alone, at least seven new papers are attacking the same problem from different angles. Read together, they reveal that the 2026 frontier is no longer about "can it do it right"—it's about "can it know when it's doing it wrong." In this article, I string them into one line, with reproducible mechanisms and numbers.

Let's Set the Scene

You ask a coding agent to fix a build script. It replies "resolved." You stare at the answer for three seconds and notice a physically impossible value in the premises—the problem is fundamentally unsolvable. It didn't make a single calculation error; it's perfectly consistent. Its error is pretending it can solve it.

This isn't something I made up. Yang and Wang tested 14 models on 30 pairs of mechanics problems, where one problem in each pair was artificially broken into unsolvability. Of 90 responses, 12 failed to reject the unsolvable problem, and 11 of those are even more striking: the model itself pointed out the defect, answered the corrected version, and still reported "resolved" (arXiv:2610.06668). It saw the problem, but handed over a conclusion that wasn't its own.

A model can simultaneously accomplish three things in the same response: identify the defect, correct and solve it, and incorrectly report the status. These three things have become decoupled. This is the core phenomenon this article will discuss.

The Shell Between Judgment and Behavior

Push the above scenario to its extreme and you get the second experiment. Zhang et al. deliberately made a certain source continuously fail in a retrieval environment (arXiv:2610.06191), and strictly distinguished two things: how the agent judges, and what the agent actually does.

The results have two layers:

Judgment is almost always correct: All 7 agents labeled the failed source's results as "useless" 97–100% of the time.
Behavior is almost unchanged: Labeling it "useless" doesn't mean stopping; most continue querying.

Then they manipulated the prompts to see which moves worked:

Intervention	Effect
"You're allowed to answer from memory"	Stops early, but unrelated to evidence
Give a token/time budget	7–8B models' stopping points all shift to the deadline
Write "stopping rules" or invocation costs in the prompt	At most partially followed
Add nothing	Continue querying the failed source

Key finding: Prompts can only shift "when to stop," not "based on what to stop." The only thing that made all models genuinely stop based on evidence was the harness forcibly inserting an integration step at the底层—after 5 consecutive "useless" judgments, force it to answer. This step raised every model's success rate on the failed source, and when the budget doubled, the stopping point didn't budge.

This is the mechanism behind the phrase in the title: judgment lives in the model, behavior lives in the shell, and there's no reliable channel between the two layers.

Long Tasks: What Step Should Humans Actually Watch?

When tasks get long, the human's role shifts from "approving each step" to "reviewing a segment of autonomous execution." Sun et al. (arXiv:2610.06406) ground this problem in something measurable:

AgentMonBench: A software engineering benchmark covering two dimensions—behavioral alignment with requirements, and identification of "critical autonomous decisions."
EBG (Evidence-Grounded Behavior Graph): Training-free. It aggregates evidence with source-code links into "behavior" nodes, connects them into a graph, and then slices and presents them from the task perspective to the monitor.

Run on 8 models, EBG outperforms directly reading raw context in most settings at identifying critical decisions and locating evidence, and this advantage doesn't collapse with input scale or hyperparameters.

One point worth dwelling on: EBG is training-free. It doesn't change model weights; it changes how you organize the evidence you feed to the monitor. The primary improvement in reliability comes from "how evidence is presented," not "swapping in a stronger model."

Making "Giving Up" an Objective Function

"Agentic abstention" became a formally optimized objective in 2026. HERA (arXiv:2610.06563)'s approach is co-evolution of harness and environment:

Use controlled environment mutations to transform a solvable problem into one that must be abandoned;
Use failure signals to infer how the harness should be modified;
Generate new execution environments and tasks that specifically target the previous round's weaknesses.

Numbers on the held-out set:

Metric	Baseline	After HERA Evolution
Abstention accuracy	61.7%	83.3%
Solvable task completion rate	68.3%	76.7%

But the number truly worth pausing on comes later: The evolved harness transfers directly across 19 other LLMs, gaining an average of 15.3 more points, letting weak models match strong models at about 85% lower cost. This shows the reliability bottleneck isn't in model weights—it's in the shell wrapped around the model. The shell is a transferable asset; the model is not.

Memory: From "Store It" to "Change It Yourself"

For agents to work long-term, they need to remember things. PrisMem (arXiv:2610.06361) opposes a default assumption—using overall performance as the evolution direction. It decomposes signals onto individual capability dimensions:

Dependency-aware selection of targets with cross-capability benefits;
History-guided diagnosis, specifically fixing one dimension;
Paired differential cases for trajectory-guided integration, merging complementary benefits into a unified memory program.

On million-token-scale histories, it outperforms the strongest baseline by 10.54 and 7.83 points (BEAM-1M / LongMemEval-M). "Overall it's rising" might just be A rising while covering B's regression; only by decomposing dimensions can you see where you truly should go.

The Shell and the Engine Finally Hear Each Other

For the above to run, you can't avoid infrastructure. The HEAR protocol (arXiv:2610.06597) fills in a missing piece:

What the harness knows: workflow dependencies, context lifecycle, execution goals;
What the engine knows: request queues, KV-cache state, resource pressure.

The two sides used to talk past each other. HEAR does bidirectional pairing, decoupling protocol semantics from optimization strategies—the same shell can swap scheduling strategies without touching workflows or models. Across four benchmarks: 1.61× batch processing speedup, 2.23× reduction in median first-token latency, 2.45× end-to-end on DeepResearchBench, with no observed degradation in task quality.

I put this last because it's the least conspicuous: the first five problems are all at the "model layer" and "harness logic layer"; HEAR is between "harness and inference engine." The bottommost layer of the entire reliability stack is also being protocolized.

Putting the 7 Papers Together

Individually each is a point; strung together they form a surface:

Paper	Which layer of the stack it fixes
Impossible problems still reported as solved (2610.06668)	Inside the model: recognition ≠ reporting
Judged useless, still queried (2610.06191)	Model ↔ harness: judgment ≠ behavior
EBG monitoring (2610.06406)	Human ↔ harness: evidence organization
HERA abstention (2610.06563)	Harness ↔ environment: shell is transferable
PrisMem memory (2610.06361)	Inside the harness: dimensional evolution
HEAR protocol (2610.06597)	Harness ↔ engine: infrastructure

The common conclusion in one sentence: The 2026 frontier has moved from "stacking parameters" to "stacking shells." Model intelligence is no longer the bottleneck; reliability engineering is.

Add a "Stop When You Should" Integration Step to Your Agent

Don't wait for frameworks to build it in. The one pattern you can immediately move into production this week comes from the second paper's conclusion:

USELESS_THRESHOLD = 5

def should_stop(history):
    # history: the most recent N tool calls and their results
    useless_run = 0
    for call in reversed(history):
        if judge_useless(call.result):   # let the model label "useless" itself
            useless_run += 1
        else:
            break
    # Key: not "the model says it should stop," but "after N consecutive self-judged useless results, force an integration answer"
    return useless_run >= USELESS_THRESHOLD

A few implementation points:

Make the model explicitly label. In the second paper, once judgments are recorded, stopping becomes stable; without recording, judgment and behavior break the chain. The label is that "recording" action.
Put the threshold in the harness, not the prompt. Stopping rules in prompts are "at most partially followed"; only the forced integration step raised success rates across all models and kept stopping points insensitive to doubled budgets.
The integration step should "force it to answer," not let it keep querying. In the paper, this step improves success rate on the failed source—i.e., "acknowledge this source is dead, switch methods or give an answer," not another round of ineffective retrieval.

Limitations must also be stated clearly: the threshold of 5 is the paper's optimal value in a controlled retrieval environment. Your tool domain (codebase, database, API) has different failure modes, and you should re-scan the threshold on your own logs. The second paper open-sourced its code and 300 pre-registered reproductions—use them directly as a starting point.

There's Still a Layer No One Is Backstopping for You

None of the 7 papers solves "who validates the shell." HERA can transfer shells across models, but the shell's own correctness depends on its own held-out set; EBG improves monitoring, but the monitor is still human. The end of reliability engineering is an open question: When the shell matters more than the model, who guarantees the shell's quality?

My judgment: This will become an independent track in the second half of 2026 to 2027. First evaluation (similar to AgentMonBench expanding to "shell behavior auditing"), then formal verification (the shell's control flow is finite, more verifiable than models). Starting now to build shell auditing tools has more leverage than grinding out another benchmark model score.

References (all submitted 2026-10-05):

arXiv:2610.06668 — Language models can notice an impossible engineering problem yet still report it as solved
arXiv:2610.06191 — Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
arXiv:2610.06406 — What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents
arXiv:2610.06563 — HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
arXiv:2610.06361 — Capability-Driven Self-Evolution of Agent Memory (PrisMem)
arXiv:2610.06597 — Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol

DEV Community: Luca Harris