DEV Community: Igor Eduardo

Done Is Testimony. Terminal State Is the Grade.

Igor Eduardo — Mon, 05 Oct 2026 15:54:00 +0000

Two things landed in the same week and they are the same argument.

Microsoft and Hugging Face published ThinkingBox, a benchmark that grades agents on the backend state and side effects they leave behind instead of the sentences they generate, and then asks whether they can do it twenty times in a row. Their opening example is an agent that makes nine clean tool calls, closes a ticket as resolved, and is wrong: the carrier exception was still open, so the required end state was "on hold." A grader reading tool calls sees nine well-formed calls. The database disagrees.

On Dev.to, The Witness Was the Suspect made the same point from the incident side: when the agent writes its own success log, the record of the incident was authored by the cause of the incident.

Here is what I would require before I trusted an agent's "done," and what I would not claim yet.

The stake

Teams will ship the wrong default.

Most agent dashboards still answer "did it say it succeeded?" Status strings, tool-call traces, and a tidy final message are all produced by the same run that took the action. They are useful for debugging. They are not a grade. When the eval and the agent share a witness, a confident wrong action scores the same as a correct one, and the failure shows up later as a customer, a ledger, or an auditor.

AI failures rarely fail loud. They fail plausible. That is exactly why the grade has to come from somewhere the agent cannot write.

Opinion (one sentence)

An agent's "done" is testimony, not evidence. The eval contract should grade the terminal state through a read path the agent does not control, and it should grade it more than once.

That is a preference with a limiter, not a benchmark result. I have not run ThinkingBox, and its numbers (507 stateful workflows, 20 runs each) are theirs. What I am formalizing is the shape: separate the actor from the grader, and treat "said done while the state is red" as a named failure, not as noise.

What separating the witness looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Required end state	What should the record look like when the task is truly finished?	"Resolved" accepted when "on hold" was correct
Independent read path	Who reads the final state, and can the agent influence that read?	The agent's own log grades the agent
Side-effect ledger	What else changed (tickets, refunds, emails, rows)?	Correct primary record, wrong collateral writes
Done-while-red	Did the run claim success while the state check failed?	Confident wrong actions blend into the pass rate
Repeat consistency	Does the same task reach the same end state across N runs?	One lucky green run treated as a capability

None of that needs a private harness. It needs someone to write down the required end state before the run, and a grader that reads the system of record after the run.

Why repeats belong in the contract

ThinkingBox's "twenty times in a row" framing is the part I would steal first.

A single green run tells you the agent can reach the right state. Production needs to know whether it will. If the same ticket ends "resolved" on some runs and "on hold" on others, you do not have a capability with a small error rate. You have a coin with a nice transcript. I would report consistency next to the pass rate, not bury it in an appendix.

Where this connects to retrieval

My public work is mostly retrieve-first systems, and the same split shows up there. A RAG answer can cite cleanly and still be wrong about the source, the same way an agent can log cleanly and still leave the wrong record. In both cases the fix is the same move: grade against something the generator did not produce. For retrieval that is a gold source set; for agents it is the terminal state.

The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure the retrieval half on its own before letting a fluent answer vouch for it.

Steal this check

Pick one agent workflow you already run and add three things:

Write the required end state first. Not "the agent responds," but "the ticket is on hold, no refund issued, customer told why."
Grade from the system of record. Read the final state with a query the agent never touches, and ignore the agent's own status message for scoring.
Count done-while-red separately. Every run where the agent claimed success but the state check failed goes in its own column. That column is the one that becomes an incident.

Then run it more than once and report how often it lands in the same place.

What I am not claiming

I am not reporting ThinkingBox results or reproducing their benchmark.
I am not saying traces and logs are useless. They are how you debug. They are just the wrong thing to score.
I am not publishing thresholds. What counts as acceptable consistency depends on what the action costs when it is wrong.

The point

The process that acted should not be the only witness that it worked. Grade the record, not the report.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Green Scores Without a Pinned Harness Are Screenshots

Igor Eduardo — Mon, 28 Sep 2026 16:09:46 +0000

Everyone is talking about Evaluation Cards and reproducible eval benches — the EvalEval × UK AISI write-up on Hugging Face, plus a week of discourse where green numbers still hide the wrong harness (and DeepEval-style “which half is broken?” chatter keeps landing). That heat is a live market object, not a niche footnote.

Here is what I would require before I trusted a green eval score for high-stakes retrieval or generation — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A dashboard can show a comforting pass rate while the report never pins how the score was earned: which protocol, which compute budget, which dataset slice, and whether the number measured retrieval, generation, or a blended soup. Celebrate that number and you optimize a screenshot. The next engineer cannot reproduce the gate, and the next incident cannot tell which half failed.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after an unpinned harness is not a win. It is a silent miss with a pretty badge.

Opinion (one sentence)

A green score without a pinned harness — protocol, compute, dataset slice, and which half of the pipeline it measures — is not an eval contract; it is a screenshot.

That is a preference with a limiter, not a SOTA claim. I have not run EvalEval in production. AISI card fields and anyone else’s leaderboard numbers stay theirs. What I am formalizing is the harness pin as a first-class part of eval-as-contract, from the same reliable-systems lane I already publish on.

What a pinned harness looks like (menu, not recipe)

Evaluation Cards earned attention because they make the boring fields visible: protocol, compute, data cut, scoring rules. You do not need their exact schema to steal the shape.

Field	Question it answers	Failure if skipped
Protocol	What procedure produced this number?	“We eval’d it” with no replay path
Compute	What budget / hardware / call limits?	Incomparable scores across teams
Dataset slice	Which queries / docs / holdouts?	Student-subset vibes framed as prod
Pipeline half	Retrieval, generation, or joint?	Green faithfulness while recall is broken (or the reverse)

The DeepEval-style lesson that keeps circulating is the same shape: a single RAG score can hide which half failed. Pin the half. Prefer two honest columns over one blended crown.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Pin the harness block before the score. Same report header every time: protocol name/version, compute class, dataset slice id, and pipeline half (retrieval / generation / joint). If any field is missing, the number is not ready to celebrate.
Split halves when you report. For the same query set, publish retrieval quality and generation/faithfulness as separate columns. One green blended score is not enough — “which half is broken?” should be answerable from the table alone.
Freeze the evaluator before you chase labels. If the judge, metric, or prompt changed mid-board, say so. Moving the goalposts mid-week and claiming progress is theater.
Keep a missing-evidence arm. Same task with the gold document removed (or retrieval forced empty). If generation still narrates a confident answer, the eval contract is incomplete — regardless of how green the happy-path column looks.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retrieval as a failure in the harness, not as a soft success.

Why query-class honesty transfers

On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to.

The lesson that transfers to Evaluation Cards is not “clone our fusion.” It is eval honesty on query classes and pipeline halves. If your harness cannot say which class and which half produced the green cell, you are not ready to ship the number into a decision.

What I will not claim

That EvalEval, UK AISI cards, or any one vendor harness is universally correct for every workload. Workload-honest eval first.
That we “ran EvalEval inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
That a pinned harness alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy remain separate gates.
Internal thresholds, prompts, or clone guides for private products. Menu only: pin the fields, split the halves — not the spice blend.

Why this formalizes who we are

The market is amplifying Evaluation Cards and green eval dashboards. The thesis I want indexed next to that heat:

Eval as contract — protocol, compute, slice, and pipeline half are gates, not vibes.
Reliable systems — production AI fails on harness mismatch and wrong-half celebration — not only on model choice.
Retrieve-first — when the retrieval half is unpinned, generation fluency is not evidence.

If a sharp reader walks away believing this engineer will not celebrate a green score without a pinned harness, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Site / builder trail: https://igoreduardo.com
Market object that sparked this note: EvalEval × UK AISI Evaluation Cards (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Igor Eduardo — Thu, 24 Sep 2026 17:58:50 +0000

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Everyone is talking about typed judgment models for RAG — Jev-class scorers that keep or drop passages, not only reorder them. Hugging Face just got a clean write-up of jev-reranker: hybrid pool in, relevance filter out, fewer documents reaching the generator. That is a live market object, not a niche blog.

Here is what I would require before I trusted “we upgraded ranking to judging” for high-stakes retrieval — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A hybrid lexical + dense pool can look fine on ranking metrics while still shipping distracting neighbors into the prompt. Rerank asks in what order? Relevance filtering asks should this context reach the generator at all? Those are different contracts. Collapsing them into one score hides the broken half.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after a weak retained set is not a win. It is a silent miss.

Opinion (one sentence)

Ranking answers closeness; judging needs an explicit leave-out policy plus a faithfulness gate that is scored separately — never a single blended “quality” number.

That is a preference with a limiter, not a SOTA claim. I have not run Jev in production. Their NanoHotpotQA numbers stay theirs. What I am formalizing is the judgment type split, from the same retrieve-first / eval-as-contract lane I already publish on.

What ranking gets right — and where it stops

Hybrid retrieval earned its keep for a reason. On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms (scores, drug names, identifiers) and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to, Zenodo preprint under CC BY.

Reranking on top of that pool is a natural next step. Cross-encoders and decision models both try to push the useful passages up. Fine — as a ranking layer.

Judging is different work:

Judgment	Question	Failure if skipped
Rank	Which of these hits is closer?	Wrong order; still often has something in context
Filter / leave-out	Should this hit reach the generator?	Distractors in the prompt; token waste
Faithfulness	Does the final answer’s claim live in the cited source?	Fluent invention with a real cite list

A leave-out policy is not “sort harder.” When the filter clears the deck, the system needs a first-class missing-evidence outcome — not a silent fallback into guessing.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Split the report. For the same query set, publish (a) ranking quality on the candidate pool and (b) retained-set size / leave-out rate after the filter. One number is not enough — the HF jev-reranker table that pairs nDCG with documents retained is the right shape of honesty, independent of their thresholds.
Pin a missing-evidence arm. Same task with the filter forced to retain nothing (or with the gold document removed from the pool). If the generator still narrates a confident answer, the eval contract is incomplete.
Keep faithfulness separate. For every quoted span, verify membership against the cited source id. Fail closed on miss. Ranking/relevance can still look fine while the quote-span check fails — I have seen that failure mode in public discourse this week, and it matches retrieve-first practice.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retained sets as failures in the harness, not as soft successes.

Hybrid pools make the split sharper

Once you run BM25 and dense in parallel, the candidate set is intentionally broad. That is a feature: complementary methods catch different query classes. It is also why a leave-out layer matters more, not less.

A wide hybrid pool without a filter ships more distractors. A filter without a missing-evidence policy ships fluent guesses when nothing survived. Rerank alone does not fix either failure — it only reorders the same set.

So when the market says “just add a judgment model,” I hear two separate jobs: upgrade the order when you still want generation, and upgrade the admit/deny policy when generation should be allowed to refuse. Publish both outcomes in the eval report. Prefer workload-honest tables over a universal crown for any one store, model, or scorer.

What I will not claim

That Jev (or any typed decision model) is universally better than a cross-encoder or a lexical filter. Workload-honest eval first.
That filter quality alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy are separate gates.
That we “shipped Jev inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
Internal thresholds, prompts, or clone guides for private products. Menu only: what we serve is the judgment split and the eval shape — not the spice blend.

Why this formalizes who we are

The market is amplifying judgment models. The thesis I want indexed next to that heat:

Retrieve-first — evidence path before orchestration theater.
Hybrid / complementary search — lexical and dense cover different query classes; fusion needs honest eval, not a universal crown.
Eval as contract — exact-match, faithfulness, context precision/recall, paired leave-out tests — gates, not vibes.

If a sharp reader walks away believing this engineer will not collapse ranking into judging, and will fail closed when evidence is missing, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Preprint (CC BY): https://doi.org/10.5281/zenodo.19686739
Market object that sparked this note: Introducing jev-reranker (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Two Retrieval Methods Are Better Than One: Evidence from 500 Clinical Queries

Igor Eduardo — Wed, 13 May 2026 19:14:30 +0000

When I set out to evaluate retrieval configurations for Portuguese clinical text, I expected one method to dominate. Instead, I found something more interesting: BM25 and dense retrieval solve different questions. Neither is a substitute for the other.

This post summarizes the methodology and results from a 500-query empirical study of hybrid retrieval for clinical question answering. All code is open source: https://github.com/nomad-link-id/hybrid-rag-pipeline

The Setup

500 clinical queries across 6 medical specialties (cardiology, endocrinology, infectology, nephrology, neurology, oncology). Each query has a single reference answer grounded in a specific passage from clinical documentation.

Four retrieval configurations were evaluated:

Config	Method
BM25-only	BM25 with Portuguese stopword removal
Dense-only	BioBERTpt embeddings, cosine similarity
Hybrid-RRF	BM25 + dense via Reciprocal Rank Fusion
Hybrid-Rerank	RRF candidates re-ranked with cross-encoder

What Is Reciprocal Rank Fusion?

RRF combines ranked lists from multiple retrievers without requiring score normalization:

def rrf_score(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

Results

Config	Recall@5	MRR	Citation F1
BM25-only	0.71	0.64	0.82
Dense-only	0.68	0.61	0.78
Hybrid-RRF	0.84	0.77	0.91
Hybrid-Rerank	0.86	0.79	0.93

The Complementarity Finding

McNemar's test on BM25-only versus dense-only:

BM25 correct, dense incorrect: 89 queries
Dense correct, BM25 incorrect: 57 queries
McNemar chi2 = 39.55, p < 0.001

The asymmetry is statistically significant. Dense-only missed 22.2% of queries that BM25 solved. You need both.

Citation Verification

Deterministic approach (BM25 score threshold + exact n-gram overlap): 461/500 citations verified.

Prompt-based LLM approach (same passages, ask LLM "does this support the answer?"): 1/500.

The difference is task design, not model quality. A deterministic check measures actual textual overlap; a prompt check measures the model's opinion of the overlap.

Inter-Annotator Agreement

100 query-response pairs independently annotated by two reviewers. Cohen's kappa = 0.954 — near-perfect agreement on what constitutes correct retrieval for clinical text.

Practical Takeaway

Run both BM25 and dense retrieval
Use RRF to merge results
Implement deterministic citation verification
Measure complementarity with McNemar's test on your domain

Code and Data

Preprint: https://doi.org/10.5281/zenodo.19686739

Igor Eduardo | igoreduardo.com | ORCID: 0009-0005-6288-1135