DEV Community: Igor Eduardo

Your Agent's Failure Predictor Is Grading the Task, Not the Run

Igor Eduardo — Mon, 09 Nov 2026 08:23:21 +0000

Your Agent's Failure Predictor Is Grading the Task, Not the Run

By Igor Eduardo · Austin, TX · https://igoreduardo.com

A short new paper, Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents (arXiv 2611.08285, posted 2026-11-05), asks a question most agent monitoring skips: when a predictor flags a run as "going wrong", has it learned anything about that run?

The setup is familiar. A coding agent works on a repository issue for many steps. A predictor reads the partial trajectory (reasoning, commands, tool output) and raises an alarm. On an alarm you stop the run, retry it, or hand the issue to another model.

The authors point out that a run can fail for three different reasons, and each one calls for a different action:

Hard issue: most models fail it. Stopping or asking a human is the better move.
Weak model: this model fails many issues. Switching to a stronger one is the better move.
Bad run: the same model solves the same issue on other attempts. Only here does a retry help.

Their finding is that current trajectory-based predictors mostly can't tell these apart.

What they report

They used public trajectories on a widely used benchmark of 500 real repository issues, where each model attempts each issue once: 3,492 runs from seven models, 994 of them failures, plus outcomes for 23 models and a separate public corpus with repeated runs.

A score that never sees the run wins. A reference that only knows how hard an issue is for other models reaches 0.92 pooled AUROC. The trajectory predictors reach 0.62–0.63.
Pooled AUROC is mostly a between-task number. Of 494,109 failed/successful pairs in the test folds, 99.66% compare different issues. A score that recognizes hard issues can look excellent without reading a single step.
Within an issue, knowing the model beats reading the run. Model identity alone reaches 0.72 within-issue AUROC, against 0.60–0.61 for the predictors. On "upset" pairs, where the overall weaker model won, the predictors score 0.42, below chance. They are reproducing the model ranking, not watching the run.
What the trajectory adds is statistically zero. Adding a predictor's score on top of issue difficulty and model identity changes AUROC by an amount equivalent to zero in all 18 configurations they tested.
The "switch models" story mostly dissolves. The best single model solves 76.8% of issues and some model solves 86.8%, a 10.0-point "oracle gap". That gap sits inside the range chance alone produces. 45% of failures (448 of 994) happen on 64 issues no model solves, so there is nothing to switch to. Picking a model per issue from its past runs gains +0.09 points on new runs; a router for new issues gains +0.41.
The gain people attribute to routing comes from a second attempt. Escalating every failure to one fixed second model gains +6.91 points, and that assumes a perfect failure detector. The win is the extra try, not a better-suited model.
Bad runs are real. In the repeated-run pilot (522 issues, at least four runs per model), outcome variation splits into 53% issue difficulty, 42% run-to-run variation, and only 4.6% model-issue specialization. So retries have real room to help, but 64–82% of predictor score variance lies between issues, which means the alarm mostly fires on the cases where a retry is less likely to work.

Scope, in their words: mainly one benchmark and one scaffold, trajectory-based predictors only, a repeated-run pilot with older models, and a router with shallow issue features. Not detecting specialization is not proof it is absent. Those are their numbers, not mine. I have not re-run the experiments, and I am deliberately not naming the models, the benchmark, or the prior systems the paper discusses.

The stake

Teams will read "our failure predictor has a good AUROC" as "our agent knows when a run is going wrong".

That is the same mistake as reading a leaderboard as a forecast. A pooled score mixes three things into one number: how hard the tasks were, how strong the model is, and what happened in this attempt. Only the third is something a run-time monitor can act on. If the number is carried by the first two, you have built an expensive difficulty estimate and wired it to a retry button.

The cost shows up in the action layer. A predictor that fires on hard issues triggers retries on exactly the failures a retry rarely fixes, and stays quiet on the bad runs a retry would rescue.

Opinion (one sentence)

Any claim that a monitor, router or retry rule makes an agent more reliable should be backed by repeated runs and run-blind baselines at equal budget, because a single-attempt leaderboard can't tell a hard task from a weak model from a bad run.

That is a preference with a limiter, not a benchmark result. If your agent runs short, cheap, deterministic tasks, single runs may be enough. Once runs are long, stochastic, and you act on an alarm mid-run, the cause of failure decides whether the action helps.

What a cause-aware agent eval reports (menu, not recipe)

Column	Question it answers	Failure if skipped
Run-blind baselines	Does the monitor beat "how hard is this task" and "which model ran"?	A difficulty estimate sold as run monitoring
Within-task score	Can it rank two attempts at the same task?	Pooled score carried by task mix
Repeated runs	How often does the same model pass and fail the same task?	Bad runs invisible, luck read as skill
Chance reference for gaps	Is the "best of several models" gap bigger than noise?	Routing built on a coin-flip artifact
Equal-budget comparison	Does the strategy beat simply trying again with the same model?	A second attempt credited to cleverness
Action outcome	Did the triggered stop, switch or retry actually fix the failure?	An alarm graded on itself, not on what it changed

None of that requires a private harness or new model. It needs repeated runs, two baselines that never look at the trajectory, and the discipline to grade the decision, not the alarm.

Why "the alarm was right" is not a pass

The detail I would steal first is the toy example the authors open with. If a model succeeds 90% of the time on easy issues and 10% on hard ones, a score that only knows which issues are hard reaches 0.90 pooled AUROC, yet it ranks two runs of the same issue at 0.50, a coin flip. A great-looking metric with zero information about the run.

That is the eval-as-contract problem in miniature. The metric was computed correctly. It was just answering a different question from the one the product needs answered at run time.

Where this connects to habits I already keep public

My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit is the same: before a component gets credit, compare it against the dumbest baseline that could have produced the same number. In retrieval, that means checking whether a gain survives query-class slices. For agents, this paper names the baselines for run-time monitoring: task difficulty, model identity, and "just try again". A monitor that can't beat them isn't monitoring the run.

Steal this check

Pick one agent with a stop, retry or routing rule and add four things before your next release:

Run each task more than once. Even a small repeated slice tells you how much of failure is the run.
Score two run-blind baselines. Task difficulty and model identity, next to your monitor, on the same cases.
Report within-task ranking. Can the monitor tell a good attempt from a bad one on the same task?
Compare at equal attempts. Does your clever strategy beat a plain second try with the same model?

Then ask whether the monitor is still worth its cost once the baselines are on the same chart.

What I am not claiming

I am not reporting this paper's results as my own, and I have not reproduced them.
I am not ranking models, agents or vendors.
I am not saying failure prediction is useless. The authors call for predictors that explain why a run fails, evaluated on repeated runs, in closed-loop, equal-budget settings.
I am not publishing alarm thresholds or retry rules. Those depend on what a wasted run and a missed failure cost you.

The point

A failure alarm that mostly knows the task is hard is not watching the run. Run tasks more than once, put run-blind baselines on the same chart, and grade the action the alarm triggers, not the alarm itself.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Your Agent Used 6% of Its Context Window and Still Lost a Third of Its Tool Output

Igor Eduardo — Mon, 02 Nov 2026 08:20:24 +0000

Your Agent Used 6% of Its Context Window and Still Lost a Third of Its Tool Output

By Igor Eduardo · Austin, TX · https://igoreduardo.com

A short new paper, Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses (arXiv 2610.36114, posted 2026-10-26), measures something most agent dashboards never show: how much of a tool's output is lost before the model ever sees it.

The authors instrumented a real tool-using agent that diagnoses injected faults by reading Linux kernel traces. Their point is simple. An agent never reads its context window directly. It reads whatever the harness, the code wrapped around the model, decided to put there. Between "the tool produced a result" and "the model saw it", the harness serializes, cuts to a budget, keeps or drops history, and passes findings between workers. Each of those steps can lose information.

What they report:

The window was nearly empty. Across 720 runs, the single largest prompt used 5.75% of the window.
A third of tool results were cut anyway. 3,827 of 11,189 tool results were truncated before reaching the model, and every cut result became broken JSON.
The loss followed position, not importance. The harness sliced each result's text at a fixed length, so whatever sat near the end disappeared. One list of containers reached the model in 4.7% of calls; a second list, in the very same payloads, reached it in 99.3%.
Accuracy split along the same line. Faults that showed up in the surviving list scored 40–55 out of 60; faults that showed up only in the deleted list scored 0–24 out of 60. The authors are explicit that this field split is correlation, not proven cause.
On a production-scale trace, a whole evidence class vanished. Replaying the shipped cut on one real trace delivered 0 of 128 records of the fault's evidence.
The first cost was extra work, not wrong answers. In a controlled experiment, when the agent had another route to the answer it still found it, but tool calls rose from 53 to 76 (the paper's "43% more"), computations roughly doubled (18 to 36), and yield, findings per computation, fell from 0.61 to 0.31.
It hides well. Only 2.5% of calls were affected in their replay of 864 recorded results, and in traces from two production coding agents, waiting on the model took a median 84% of wall-clock time, so latency dashboards barely move.

The authors also surveyed seven agent frameworks and found two that cut tool output by position in the same way. I am not naming them here; the paper does, and the point is the pattern, not a scoreboard.

Scope, in their words: one harness, one domain, one model, small samples, and the causal link to the accuracy drop is still an open question. Those are their numbers, not mine. I have not re-run the experiments.

The stake

Teams will read "context window: mostly empty" as "the model saw everything".

Context size and final accuracy are the two numbers agents are usually judged on. Neither tells you whether the evidence the model needed was delivered intact. A log that records what the harness sent cannot show what the tool produced, so this class of loss is invisible in the place people look for it.

And it fails quietly. When the agent has a second route to the answer, the loss shows up as more calls and more tokens. When it does not, as with a quiet fault that lives in exactly one field, the evidence is simply gone and the agent answers anyway.

Opinion (one sentence)

For any agent that acts on tool output, "what reached the model" should be a measured, reported number next to accuracy, because a harness can delete the evidence that matters while every capacity and latency metric looks healthy.

That is a preference with a limiter, not a benchmark result. If your tools return small, flat outputs that never approach a budget, this may cost you nothing. Once tools return lists, logs or traces that can grow, the delivery step becomes part of the system under test.

What a delivery-aware agent eval looks like (menu, not recipe)

Column	Question it answers	Failure if skipped
Produced vs. delivered	Did each field of the tool result reach the model?	Evidence lost upstream looks like model error
Payload validity	Did the delivered result still parse?	Broken structure passed off as "the data"
Loss by position	Do late fields disappear more than early ones?	Importance decided by key order
Work per finding	How many calls and computations per useful finding?	A silent cost rise read as "harder task"
Hand-off between workers	Did computed findings reach whatever decides?	Silence read as "nothing found"
Quiet-case accuracy	Accuracy on cases with only one route to the evidence	Redundant routes masking the loss

None of that requires a private harness or a new model. It requires logging the tool's full result next to what was actually sent, and refusing a release that only reports window size and final accuracy.

Why "the answer was still right" is not a pass

The finding I would steal first is the compensation tax. In the controlled runs, accuracy on the easy, loud cases stayed flat while the agent did far more work to get there. That is exactly the kind of result that passes a pass-rate gate and then fails in production on the one case where the second route does not exist.

The authors' own prediction was wrong in an instructive way: they expected that deleting the culprit's list would drop accuracy, and the agent recovered the name from another table. Their sharper rule is that the cut hurts only where there is no second route, and that its cost appears as extra work before it appears as errors. If your eval set is full of loud cases, you will measure the work and miss the risk.

Where this connects to habits I already keep public

My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit is the same: grade each stage on its own before letting the final answer vouch for the pipeline. In retrieval, that means checking that the right evidence came back before grading the answer. For agents, this paper isolates the next stage: whether the evidence that came back actually reached the model. A system can retrieve correctly and still never show the model what it retrieved.

Steal this check

Pick one agent that reads tool output and add four things before your next release:

Log produced and delivered side by side. Keep the full tool result and the text the model actually received, per call.
Count what didn't arrive. Per field, per run: how often was it cut, and did the payload still parse?
Track work per finding. Calls and computations per useful result, next to accuracy.
Keep a quiet-case slice. Cases where the evidence lives in exactly one place. Grade accuracy there separately.

Then ask whether the release is still an upgrade once delivery is visible.

What I am not claiming

I am not reporting this paper's results as my own, and I have not reproduced them.
I am not ranking frameworks, models or vendors.
I am not claiming truncation caused the accuracy split. The authors say that is still open.
I am not publishing budgets or cut rules to use. Those depend on your tools and what a missed field costs you.

The point

A mostly empty context window tells you there was room. It does not tell you what arrived. Measure the path from tool to model, report loss next to accuracy, and keep a slice of cases where the agent has only one way to find the truth.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Your Agent's Pass Rate Doesn't Tell You Whether It Can Notice Its Own Mistake

Igor Eduardo — Mon, 26 Oct 2026 07:18:30 +0000

By Igor Eduardo · Austin, TX · https://igoreduardo.com

A new benchmark paper, CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents (arXiv 2610.27645, posted 2026-10-20), asks a question most agent scoreboards skip: when an agent inherits a workflow that already went wrong, does it notice, and can it fix it?

The setup is careful. The authors compose long, multi-application desktop workflows with an executable "gold lineage", the correct value at every step. They run six agents, collect the failures those agents actually produce, locate the root-cause step of each failure, and replay the run to a point 0 to 25 steps after that root cause. Then an agent takes over from that state without being told anything went wrong. The test suite has 2,250 human-verified erroneous states over 100 tasks, drawn from 619 failed rollouts. They score three things separately: whether the agent says something is wrong (Error-Awareness Rate, EAR), how much of the task it completes (Rubric Score), and whether it fully completes the task in any of three runs (Pass@3).

What they report:

Recovery is rare. Across six agents and six takeover depths, EAR reaches at most 44.0% and Pass@3 at most 25.5%.
Clean-start skill does not predict noticing. One agent solves 66.9% of the source tasks from a clean start but notices at most 18.5% of the errors in its own failures. Another solves 5.1% from a clean start but notices 14.1–26.1%.
Noticing is not fixing. Macro-averaged across the six agents, runs that recognized the inherited error passed the verifier 15.6% of the time, versus 12.2% for runs that did not. Agents often name the problem and then leave the affected state unchanged.
Errors surface late. In their annotated failures, 78.1% of errors became explicit three or more steps after the root cause, with a median delay of 17 steps. In 361 of 602 failures where the error became explicit, the first evidence appeared only at the final step.
Most root causes are decisions, not clicks. Planning and perception account for 57.3% of root causes; termination errors, including reporting success while requirements remain unmet, account for 23.5%.

The authors are explicit about scope: one desktop environment, agents run with their own scaffolds, and an LLM judge that they checked against human labels and found slightly strict. They also show that fine-tuning on their recovery trajectories raises one open model's average EAR from 19.9% to 35.2%. Code is public at CUA-Recovery.

Those are their numbers, not mine. I have not re-run the benchmark, and I am not using it to rank anyone. What I want to formalize is the eval habit it exposes.

The stake

Teams will ship agents that are good at starting and bad at noticing.

A clean-start pass rate measures one thing: given a correct initial state, does the agent reach the goal? Production does not hand agents correct states. A previous step, a previous run, a human, or a flaky tool has often already written something wrong into a file, a spreadsheet or a record. If the eval never starts from a broken state, the agent's ability to notice is never measured, and the number you ship on is silent about the failure that hurts most.

AI failures rarely fail loud. An agent that builds 17 confident steps on top of a wrong record looks, on a completion dashboard, a lot like one that did the job.

Opinion (one sentence)

Any agent that writes to persistent state should be graded on takeovers from broken states, with "noticed" and "repaired" scored as separate columns, because a clean-start pass rate cannot tell you whether the agent will catch its own inherited mistake.

That is a preference with a limiter, not a benchmark result. For read-only agents that only answer questions, clean-start evals may be enough. Once an agent writes, sends or orders, the inherited-error case is the one that compounds.

What a recovery-aware agent eval looks like (menu, not recipe)

Column	Question it answers	Failure if skipped
Clean-start completion	Can it do the task from a correct state?	No baseline at all
Takeover completion	Can it finish from a state that is already wrong?	Shipping a system that only works when nothing went wrong before
Error awareness	Did it say something earlier is wrong?	Silent agents look identical to careful ones
Repair, scored apart from awareness	Did it actually fix the affected state?	Crediting "I see a problem" as a fix
Depth since root cause	How far back was the mistake?	Missing that late takeovers behave differently
Completion claim vs. state	Does the final state match what the agent reported?	Premature "done" passing as success

None of that needs a private harness. It needs someone to refuse an agent release that only reports the first column.

Why "noticed" and "repaired" must be different columns

The finding I would steal first is the gap between awareness and recovery. A system that says "it looks like the earlier step used the wrong year" and then carries on with the wrong year has produced a better transcript, not a better outcome. If one metric lumps the two together, it will reward agents that narrate problems over agents that fix them.

The paper also shows the two can move in opposite directions. As the takeover point moves further from the root cause, some agents became more likely to notice something was off while becoming less likely to complete the task, because the step that caused the problem had left their context window. Later is not simply harder or easier. It is different, so it needs its own row.

Where this connects to habits I already keep public

My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit is the same: grade each part on its own before letting the final answer vouch for it. For agents, the "part" this paper isolates is the moment the agent receives state it did not create. I would treat that moment as a first-class test case, next to the clean start.

Steal this check

Pick one agent workflow that writes to persistent state and add four things before your next release:

Collect real failures. Keep the runs where your agent actually went wrong, not just synthetic faults.
Replay to a broken state and hand over. Start a fresh run from a point after the mistake, without warning it.
Score awareness and repair separately. "Said something is wrong" and "fixed what was wrong" go in different columns.
Check the claim against the state. If the agent says done, verify the final state, not the final message.

Then ask whether the release is still an upgrade once the takeover columns are visible.

What I am not claiming

I am not reporting CUA-Recovery results as my own, and I have not reproduced the benchmark.
I am not ranking models or vendors. The paper's per-agent tables exist; this post deliberately does not use them as a leaderboard.
I am not publishing thresholds for acceptable awareness or recovery rates. Those depend on what a wrong write costs in your product.
One desktop environment is not every environment. The authors scope their claims the same way.

The point

A pass rate from a clean start tells you the agent can begin well. It does not tell you whether the agent will catch a mistake already sitting in the state it inherits. Start some evals from broken states, score noticing and repairing separately, and check every "done" against the state it claims to have produced.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Extra Retrieval Rounds Can Make RAG Worse at Saying "I Don't Know"

Igor Eduardo — Fri, 23 Oct 2026 07:24:51 +0000

A new benchmark paper, DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge (arXiv 2610.28340, posted 2026-10-20), measures something most RAG scoreboards skip: how often a system admits that neither its documents nor its weights can answer the question.

The setup is clever. The authors build a retrieval corpus from Minecraft Wiki pages as they stood at a 31 March 2025 cut-off, then write unanswerable questions about game features documented only after that date. The benchmark has 700 answerable and 300 unanswerable questions, and they evaluate 27 LLMs. Their headline metric is Correct Abstention Rate (CAR): the share of unanswerable questions where the model acknowledges it lacks the answer. They score "premise denial", where the model asserts without support that the item does not exist, separately from an honest "I don't know".

What they report:

Single-shot retrieval helps. Compared with no retrieval, it raises CAR significantly in 38 of 40 model/thinking configurations. Mean CAR goes from about 17.1% to 56.4%.
Iterative retrieval gives part of that back. When the model can request up to two more retrieval rounds, CAR drops significantly in 30 of 40 configurations, to a mean of about 46.4%. Answer Accuracy on the answerable questions barely moves (about 68.5% â†’ 69.3%).
Adding query rewriting plus reranking to hybrid retrieval lowered CAR for all 12 models tested, from a mean of about 54.7% to 49.8%. The authors note that only 3 of the 60 per-model paired comparisons against plain hybrid are individually significant, so the direction is consistent but each per-model gap is small.
Retrieving more chunks also lowered it. On an 11-model subset, going from 5 to 10 chunks dropped mean CAR from about 56.8% to 51.7%.

The authors are careful about what this does and does not show. Their aggregate comparison does not isolate which part of iterative RAG causes the drop: chunk count, context length, follow-up queries and self-reflection all change together. Their corpus is one domain with one chunking scheme. Code and data are public at declinecraft-benchmark.

Those are their numbers, not mine. I have not re-run the benchmark. What I want to formalize is the eval habit the paper forces into the open.

The stake

Teams will ship the wrong upgrade.

The usual path for improving a RAG system is to add things: more chunks, a reranker, query rewriting, a second retrieval round. Each one is justified by an accuracy chart on answerable questions. In this paper, accuracy on answerable questions stays roughly flat while the system gets measurably worse at declining the questions it cannot answer. If your eval set has no unanswerable questions, that regression is invisible. You will ship it as an improvement.

AI failures rarely fail loud. A confident answer to a question your corpus cannot support looks exactly like a correct one on a dashboard that only grades answerable items.

Opinion (one sentence)

Every change to a retrieval pipeline should be graded on unanswerable questions too, because "more context" can buy a little accuracy while quietly spending the system's ability to say "I don't know."

That is a preference with a limiter, not a benchmark result. I am not saying iterative retrieval, reranking or query rewriting are mistakes. I am saying the contract for adding them must include the column that can go down.

What an abstention-aware eval contract looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Answerable accuracy	Did we answer the questions the corpus supports?	Shipping on vibes
Correct abstention rate	Did we decline the questions nothing supports?	Confident answers with no evidence
Premise denial, counted separately	Did we claim something does not exist without being able to verify it?	"No such thing" passing as caution
Attempt rate on answerable items	Did we refuse questions we could have answered?	A system that looks safe because it answers nothing
Per-change deltas	Which single change moved each column?	Crediting the whole pipeline for one component's regression

None of that needs a private harness. It needs someone to refuse a pipeline change that only reports the accuracy column.

Why "I don't know" is not the same as "no such thing"

The distinction I would steal first is premise denial.

When a corpus is older than the question, "I can't find this" and "this doesn't exist" are different claims. The first is honest. The second is a statement about the world that the system cannot back. A metric that lumps both into "declined" will reward the second. The paper's own example is a model that, asked about an item added after the cut-off, says the item is not recognized and then answers about a different item instead.

If you run retrieval over documentation that changes, such as product docs, internal wikis or release notes, this is the failure to watch. The question can be newer than your index.

Where this connects to habits I already keep public

My public work is mostly retrieve-first systems, and the same habit applies there: measure the retrieval half on its own before letting the answer vouch for it. My hybrid retrieval work is built around that habit. This week's paper adds a column I would put next to it: what the system does when the right evidence is not in the index at all.

Steal this check

Pick one RAG pipeline you are about to "improve" and add four things before you merge:

Write a small unanswerable slice. Questions about things newer than, or missing from, your corpus. Even a few dozen is better than none.
Grade abstention next to accuracy. Report both columns for the before and after versions of the pipeline.
Split premise denial from abstention. "I can't find it" and "it doesn't exist" go in different buckets.
Change one thing at a time. Extra chunks, reranking, rewriting and extra rounds each get their own before/after row.

Then ask whether the change is still an upgrade once the abstention column is visible.

What I am not claiming

I am not reporting DeclineCraft results as my own, and I have not reproduced the benchmark.
I am not saying iterative RAG, reranking or query rewriting should be removed. The paper's own caveat is that it cannot isolate which component drives the drop.
I am not publishing thresholds. What counts as an acceptable abstention rate depends on the product and on what a wrong answer costs.
One domain (a game wiki) is not every domain. The authors say so too.

The point

An accuracy chart on answerable questions cannot tell you whether your upgrade made the system more willing to bluff. Put unanswerable questions in the eval, grade "I don't know" as a real outcome, and make every pipeline change earn its place on both columns.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

An 8-Point nDCG Bump That Costs Two Minutes Is Not Free

Igor Eduardo — Mon, 12 Oct 2026 16:12:42 +0000

A new NVIDIA NeMo Retriever paper, Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks (arXiv 2610.05750, posted around 2026-10-05), puts numbers on a trade everyone arguing about agentic RAG already feels. With the same embedding model, their agentic retrieval setup lifts nDCG@10 by about 8.7 points over standard dense retrieval. It also averages roughly 107.4 seconds of latency against about 0.67 seconds for the standard path, and burns on the order of 764.1K input tokens plus 5.8K output tokens per query. They also report competitive results on ViDoRe v3 and BRIGHT without task-specific specialization. Their retrieval-bench notes live in the public NeMo-Retriever tree.

Those are their numbers, not mine. I have not re-run the bench. What I want to formalize is the eval shape the paper forces into the open.

Here is what I would require before I trusted an "agentic retrieval win," and what I would not claim yet.

The stake

Teams will ship the wrong default.

Most retrieval scoreboards still answer one question: "did we find better documents?" That question is real. It is incomplete. An 8-point nDCG bump that costs two minutes and three-quarters of a million input tokens is a different product than a sub-second dense call. If your dashboard only shows the quality column, the expensive path looks like a free upgrade. In production it is a budget, a timeout policy, and a fallback story.

AI failures rarely fail loud. They fail as bills that looked like wins on the eval sheet.

Opinion (one sentence)

An eval contract that reports only retrieval quality without latency, token cost, and failure/fallback rate is grading a different product than the one you will ship.

That is a preference with a limiter, not a benchmark result. The NeMo numbers stay theirs. What I am formalizing is the contract: quality, cost, and reliability sit in the same report, or you are not evaluating the system you will actually run.

What a cost-aware retrieval contract looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Retrieval quality	Did we find the right evidence under the stated metric?	Shipping on vibes
Latency distribution	How long did the path take, including agent loops?	A "win" that times out in UX or SLA
Token / dollar cost	What did one query cost on average and at the tail?	Quality gains that bankrupt the budget
Failure / fallback rate	How often did the agentic path abort, stall, or hand off?	A green mean that hides a red path
Same-model / same-corpus controls	Did we change one variable at a time?	Crediting the agent loop for an embedding upgrade

None of that needs a private harness. It needs someone to refuse a scoreboard that only prints nDCG.

Why the latency column belongs next to nDCG

The paper's latency gap is the part I would steal first as a reporting habit.

A sub-second dense path and a ~two-minute agentic path are not two settings of the same product. They are different products with different failure modes. One fails by missing evidence. The other can also fail by being too slow to use, too expensive to run at volume, or too brittle when a tool call hangs. If you only publish the quality delta, you are arguing about a system your users will never wait for.

I would report latency and token cost next to the quality metric, not bury them in an appendix. The same goes for failure and fallback rate: every run that aborted or handed back to a cheaper path belongs in its own column.

Where this connects to retrieval habits I already keep public

My public work is mostly retrieve-first systems, and the same split shows up there. A fluent answer can look like a retrieval win the same way a slow agent loop can look like a quality win. In both cases the fix is the same move: measure the retrieval half on its own, and refuse to let one number vouch for the whole product.

The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure retrieval separately before letting generation, or orchestration, speak for it. This week's paper is a cost-shaped version of the same lesson.

Steal this check

Pick one retrieval path you already call "agentic" or "multi-hop" and add four things:

Keep the quality metric. Report nDCG (or your usual ranking metric) with the same embedding model and corpus controls you already trust.
Add latency next to it. Mean is not enough; look at the tail your users will feel.
Add token cost next to it. Input and output, average and heavy queries, in the same table as quality.
Count failure / fallback separately. Every abort, timeout, or handoff to a cheaper path goes in its own column. That column is the one that becomes an incident or a surprise bill.

Then ask whether the quality bump is still the headline once those columns are visible.

What I am not claiming

I am not reporting NeMo Retriever results as my own, and I have not reproduced their bench.
I am not saying agentic retrieval is a mistake. The paper shows a real quality gain on hard tasks. Cost is part of the grade, not a reason to ignore the gain.
I am not publishing thresholds. What counts as acceptable latency or token spend depends on the product and the query class.

The point

An 8-point nDCG bump that costs two minutes is not free. Grade the product you will ship: quality, cost, and the rate at which the expensive path fails closed.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Done Is Testimony. Terminal State Is the Grade.

Igor Eduardo — Mon, 05 Oct 2026 15:54:00 +0000

Two things landed in the same week and they are the same argument.

Microsoft and Hugging Face published ThinkingBox, a benchmark that grades agents on the backend state and side effects they leave behind instead of the sentences they generate, and then asks whether they can do it twenty times in a row. Their opening example is an agent that makes nine clean tool calls, closes a ticket as resolved, and is wrong: the carrier exception was still open, so the required end state was "on hold." A grader reading tool calls sees nine well-formed calls. The database disagrees.

On Dev.to, The Witness Was the Suspect made the same point from the incident side: when the agent writes its own success log, the record of the incident was authored by the cause of the incident.

Here is what I would require before I trusted an agent's "done," and what I would not claim yet.

The stake

Teams will ship the wrong default.

Most agent dashboards still answer "did it say it succeeded?" Status strings, tool-call traces, and a tidy final message are all produced by the same run that took the action. They are useful for debugging. They are not a grade. When the eval and the agent share a witness, a confident wrong action scores the same as a correct one, and the failure shows up later as a customer, a ledger, or an auditor.

AI failures rarely fail loud. They fail plausible. That is exactly why the grade has to come from somewhere the agent cannot write.

Opinion (one sentence)

An agent's "done" is testimony, not evidence. The eval contract should grade the terminal state through a read path the agent does not control, and it should grade it more than once.

That is a preference with a limiter, not a benchmark result. I have not run ThinkingBox, and its numbers (507 stateful workflows, 20 runs each) are theirs. What I am formalizing is the shape: separate the actor from the grader, and treat "said done while the state is red" as a named failure, not as noise.

What separating the witness looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Required end state	What should the record look like when the task is truly finished?	"Resolved" accepted when "on hold" was correct
Independent read path	Who reads the final state, and can the agent influence that read?	The agent's own log grades the agent
Side-effect ledger	What else changed (tickets, refunds, emails, rows)?	Correct primary record, wrong collateral writes
Done-while-red	Did the run claim success while the state check failed?	Confident wrong actions blend into the pass rate
Repeat consistency	Does the same task reach the same end state across N runs?	One lucky green run treated as a capability

None of that needs a private harness. It needs someone to write down the required end state before the run, and a grader that reads the system of record after the run.

Why repeats belong in the contract

ThinkingBox's "twenty times in a row" framing is the part I would steal first.

A single green run tells you the agent can reach the right state. Production needs to know whether it will. If the same ticket ends "resolved" on some runs and "on hold" on others, you do not have a capability with a small error rate. You have a coin with a nice transcript. I would report consistency next to the pass rate, not bury it in an appendix.

Where this connects to retrieval

My public work is mostly retrieve-first systems, and the same split shows up there. A RAG answer can cite cleanly and still be wrong about the source, the same way an agent can log cleanly and still leave the wrong record. In both cases the fix is the same move: grade against something the generator did not produce. For retrieval that is a gold source set; for agents it is the terminal state.

The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure the retrieval half on its own before letting a fluent answer vouch for it.

Steal this check

Pick one agent workflow you already run and add three things:

Write the required end state first. Not "the agent responds," but "the ticket is on hold, no refund issued, customer told why."
Grade from the system of record. Read the final state with a query the agent never touches, and ignore the agent's own status message for scoring.
Count done-while-red separately. Every run where the agent claimed success but the state check failed goes in its own column. That column is the one that becomes an incident.

Then run it more than once and report how often it lands in the same place.

What I am not claiming

I am not reporting ThinkingBox results or reproducing their benchmark.
I am not saying traces and logs are useless. They are how you debug. They are just the wrong thing to score.
I am not publishing thresholds. What counts as acceptable consistency depends on what the action costs when it is wrong.

The point

The process that acted should not be the only witness that it worked. Grade the record, not the report.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Green Scores Without a Pinned Harness Are Screenshots

Igor Eduardo — Mon, 28 Sep 2026 16:09:46 +0000

Everyone is talking about Evaluation Cards and reproducible eval benches — the EvalEval × UK AISI write-up on Hugging Face, plus a week of discourse where green numbers still hide the wrong harness (and DeepEval-style “which half is broken?” chatter keeps landing). That heat is a live market object, not a niche footnote.

Here is what I would require before I trusted a green eval score for high-stakes retrieval or generation — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A dashboard can show a comforting pass rate while the report never pins how the score was earned: which protocol, which compute budget, which dataset slice, and whether the number measured retrieval, generation, or a blended soup. Celebrate that number and you optimize a screenshot. The next engineer cannot reproduce the gate, and the next incident cannot tell which half failed.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after an unpinned harness is not a win. It is a silent miss with a pretty badge.

Opinion (one sentence)

A green score without a pinned harness — protocol, compute, dataset slice, and which half of the pipeline it measures — is not an eval contract; it is a screenshot.

That is a preference with a limiter, not a SOTA claim. I have not run EvalEval in production. AISI card fields and anyone else’s leaderboard numbers stay theirs. What I am formalizing is the harness pin as a first-class part of eval-as-contract, from the same reliable-systems lane I already publish on.

What a pinned harness looks like (menu, not recipe)

Evaluation Cards earned attention because they make the boring fields visible: protocol, compute, data cut, scoring rules. You do not need their exact schema to steal the shape.

Field	Question it answers	Failure if skipped
Protocol	What procedure produced this number?	“We eval’d it” with no replay path
Compute	What budget / hardware / call limits?	Incomparable scores across teams
Dataset slice	Which queries / docs / holdouts?	Student-subset vibes framed as prod
Pipeline half	Retrieval, generation, or joint?	Green faithfulness while recall is broken (or the reverse)

The DeepEval-style lesson that keeps circulating is the same shape: a single RAG score can hide which half failed. Pin the half. Prefer two honest columns over one blended crown.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Pin the harness block before the score. Same report header every time: protocol name/version, compute class, dataset slice id, and pipeline half (retrieval / generation / joint). If any field is missing, the number is not ready to celebrate.
Split halves when you report. For the same query set, publish retrieval quality and generation/faithfulness as separate columns. One green blended score is not enough — “which half is broken?” should be answerable from the table alone.
Freeze the evaluator before you chase labels. If the judge, metric, or prompt changed mid-board, say so. Moving the goalposts mid-week and claiming progress is theater.
Keep a missing-evidence arm. Same task with the gold document removed (or retrieval forced empty). If generation still narrates a confident answer, the eval contract is incomplete — regardless of how green the happy-path column looks.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retrieval as a failure in the harness, not as a soft success.

Why query-class honesty transfers

On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to.

The lesson that transfers to Evaluation Cards is not “clone our fusion.” It is eval honesty on query classes and pipeline halves. If your harness cannot say which class and which half produced the green cell, you are not ready to ship the number into a decision.

What I will not claim

That EvalEval, UK AISI cards, or any one vendor harness is universally correct for every workload. Workload-honest eval first.
That we “ran EvalEval inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
That a pinned harness alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy remain separate gates.
Internal thresholds, prompts, or clone guides for private products. Menu only: pin the fields, split the halves — not the spice blend.

Why this formalizes who we are

The market is amplifying Evaluation Cards and green eval dashboards. The thesis I want indexed next to that heat:

Eval as contract — protocol, compute, slice, and pipeline half are gates, not vibes.
Reliable systems — production AI fails on harness mismatch and wrong-half celebration — not only on model choice.
Retrieve-first — when the retrieval half is unpinned, generation fluency is not evidence.

If a sharp reader walks away believing this engineer will not celebrate a green score without a pinned harness, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Site / builder trail: https://igoreduardo.com
Market object that sparked this note: EvalEval × UK AISI Evaluation Cards (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Igor Eduardo — Thu, 24 Sep 2026 17:58:50 +0000

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Everyone is talking about typed judgment models for RAG — Jev-class scorers that keep or drop passages, not only reorder them. Hugging Face just got a clean write-up of jev-reranker: hybrid pool in, relevance filter out, fewer documents reaching the generator. That is a live market object, not a niche blog.

Here is what I would require before I trusted “we upgraded ranking to judging” for high-stakes retrieval — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A hybrid lexical + dense pool can look fine on ranking metrics while still shipping distracting neighbors into the prompt. Rerank asks in what order? Relevance filtering asks should this context reach the generator at all? Those are different contracts. Collapsing them into one score hides the broken half.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after a weak retained set is not a win. It is a silent miss.

Opinion (one sentence)

Ranking answers closeness; judging needs an explicit leave-out policy plus a faithfulness gate that is scored separately — never a single blended “quality” number.

That is a preference with a limiter, not a SOTA claim. I have not run Jev in production. Their NanoHotpotQA numbers stay theirs. What I am formalizing is the judgment type split, from the same retrieve-first / eval-as-contract lane I already publish on.

What ranking gets right — and where it stops

Hybrid retrieval earned its keep for a reason. On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms (scores, drug names, identifiers) and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to, Zenodo preprint under CC BY.

Reranking on top of that pool is a natural next step. Cross-encoders and decision models both try to push the useful passages up. Fine — as a ranking layer.

Judging is different work:

Judgment	Question	Failure if skipped
Rank	Which of these hits is closer?	Wrong order; still often has something in context
Filter / leave-out	Should this hit reach the generator?	Distractors in the prompt; token waste
Faithfulness	Does the final answer’s claim live in the cited source?	Fluent invention with a real cite list

A leave-out policy is not “sort harder.” When the filter clears the deck, the system needs a first-class missing-evidence outcome — not a silent fallback into guessing.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Split the report. For the same query set, publish (a) ranking quality on the candidate pool and (b) retained-set size / leave-out rate after the filter. One number is not enough — the HF jev-reranker table that pairs nDCG with documents retained is the right shape of honesty, independent of their thresholds.
Pin a missing-evidence arm. Same task with the filter forced to retain nothing (or with the gold document removed from the pool). If the generator still narrates a confident answer, the eval contract is incomplete.
Keep faithfulness separate. For every quoted span, verify membership against the cited source id. Fail closed on miss. Ranking/relevance can still look fine while the quote-span check fails — I have seen that failure mode in public discourse this week, and it matches retrieve-first practice.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retained sets as failures in the harness, not as soft successes.

Hybrid pools make the split sharper

Once you run BM25 and dense in parallel, the candidate set is intentionally broad. That is a feature: complementary methods catch different query classes. It is also why a leave-out layer matters more, not less.

A wide hybrid pool without a filter ships more distractors. A filter without a missing-evidence policy ships fluent guesses when nothing survived. Rerank alone does not fix either failure — it only reorders the same set.

So when the market says “just add a judgment model,” I hear two separate jobs: upgrade the order when you still want generation, and upgrade the admit/deny policy when generation should be allowed to refuse. Publish both outcomes in the eval report. Prefer workload-honest tables over a universal crown for any one store, model, or scorer.

What I will not claim

That Jev (or any typed decision model) is universally better than a cross-encoder or a lexical filter. Workload-honest eval first.
That filter quality alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy are separate gates.
That we “shipped Jev inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
Internal thresholds, prompts, or clone guides for private products. Menu only: what we serve is the judgment split and the eval shape — not the spice blend.

Why this formalizes who we are

The market is amplifying judgment models. The thesis I want indexed next to that heat:

Retrieve-first — evidence path before orchestration theater.
Hybrid / complementary search — lexical and dense cover different query classes; fusion needs honest eval, not a universal crown.
Eval as contract — exact-match, faithfulness, context precision/recall, paired leave-out tests — gates, not vibes.

If a sharp reader walks away believing this engineer will not collapse ranking into judging, and will fail closed when evidence is missing, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Preprint (CC BY): https://doi.org/10.5281/zenodo.19686739
Market object that sparked this note: Introducing jev-reranker (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Two Retrieval Methods Are Better Than One: Evidence from 500 Clinical Queries

Igor Eduardo — Wed, 13 May 2026 19:14:30 +0000

When I set out to evaluate retrieval configurations for Portuguese clinical text, I expected one method to dominate. Instead, I found something more interesting: BM25 and dense retrieval solve different questions. Neither is a substitute for the other.

This post summarizes the methodology and results from a 500-query empirical study of hybrid retrieval for clinical question answering. All code is open source: https://github.com/nomad-link-id/hybrid-rag-pipeline

The Setup

500 clinical queries across 6 medical specialties (cardiology, endocrinology, infectology, nephrology, neurology, oncology). Each query has a single reference answer grounded in a specific passage from clinical documentation.

Four retrieval configurations were evaluated:

Config	Method
BM25-only	BM25 with Portuguese stopword removal
Dense-only	BioBERTpt embeddings, cosine similarity
Hybrid-RRF	BM25 + dense via Reciprocal Rank Fusion
Hybrid-Rerank	RRF candidates re-ranked with cross-encoder

What Is Reciprocal Rank Fusion?

RRF combines ranked lists from multiple retrievers without requiring score normalization:

def rrf_score(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

Results

Config	Recall@5	MRR	Citation F1
BM25-only	0.71	0.64	0.82
Dense-only	0.68	0.61	0.78
Hybrid-RRF	0.84	0.77	0.91
Hybrid-Rerank	0.86	0.79	0.93

The Complementarity Finding

McNemar's test on BM25-only versus dense-only:

BM25 correct, dense incorrect: 89 queries
Dense correct, BM25 incorrect: 57 queries
McNemar chi2 = 39.55, p < 0.001

The asymmetry is statistically significant. Dense-only missed 22.2% of queries that BM25 solved. You need both.

Citation Verification

Deterministic approach (BM25 score threshold + exact n-gram overlap): 461/500 citations verified.

Prompt-based LLM approach (same passages, ask LLM "does this support the answer?"): 1/500.

The difference is task design, not model quality. A deterministic check measures actual textual overlap; a prompt check measures the model's opinion of the overlap.

Inter-Annotator Agreement

100 query-response pairs independently annotated by two reviewers. Cohen's kappa = 0.954 — near-perfect agreement on what constitutes correct retrieval for clinical text.

Practical Takeaway

Run both BM25 and dense retrieval
Use RRF to merge results
Implement deterministic citation verification
Measure complementarity with McNemar's test on your domain

Code and Data

Preprint: https://doi.org/10.5281/zenodo.19686739

Igor Eduardo | igoreduardo.com | ORCID: 0009-0005-6288-1135