Today, one call for /deep-research burned 6.57M Fable tokens. If you translate that through rough data-center energy estimates, it is on the order of hours or days of a desk lamp, depending on the model and serving setup. The task was a real system design problem: how do I stop a local 10B MoE model from hallucinating citation relevance when evaluating medical transcripts?
The insights were worth it. The useful direction was clear: move citation relevance upstream into quote-first selection, then use a specialized NLI/cross-encoder filter as the verification layer.
But, I was still intrigued: what was the logic orchestrating all this compute? I reconstructed the .jsonl trace and the local .js orchestrator file to see what happened.
The runtime shape is a strict JavaScript file executing a deterministic Map-Reduce pipeline: fan-out search, fetch, adversarial verify, and synthesize.
A comment on line 9 gives the lineage:
// Ported from bughunter architecture. WebSearch/WebFetch instead of git/grep.
It appears this is a port of their automated bug-hunting loop. When hunting bugs, you generate failure hypotheses and then adversarially try to disprove them against the code. That lineage explains the shape of the system: generate hypotheses, then try to kill them.
Modeling the domain: The anatomy of a harness
A software harness is a controlled environment designed to run code under strict constraints. In practice, this script gives the model a small box each time: scope the question, return ranked sources, extract claims with quotes, vote on one claim, or synthesize verified evidence.
Before the first search runs, the script defines what can exist in the research process: SCOPE, SEARCH, EXTRACT, VERDICT, and REPORT. Every LLM call is restricted to a micro-task with a forced, typed output. The schema prevents the model from answering with free-form prose because the orchestration forces structured JSON at the tool-call level. The repeated root question handles drift later in the pipeline.
Once the domain is modeled and typed, you can write orchestration like standard code: sort, filter, and count quorums.
For example, the extractor schema requires claims[].quote: every extracted claim must carry a verbatim quote from the source. In one source, the extractor mapped the quote "Simply switching stronger models cannot significantly improve..." to the claim about model capacity. That is the harness pattern in miniature: narrow task, typed output, evidence attached.
Editorial policy in four constants
const VOTES_PER_CLAIM = 3
const REFUTATIONS_REQUIRED = 2
const MAX_FETCH = 15
const MAX_VERIFY_CLAIMS = 25
A lot of the system's research policy sits in four variables at the top of the file. How many opinions it takes to kill a claim. How many sources we can afford to read. How many claims are worth the expensive verification step.
This is the second half of the harness. The schemas constrain the shape of work; the constants constrain the judgment policy. If you want stricter verification, you bump VOTES_PER_CLAIM to 5. The policy becomes visible in a diff.
Turning one question into search jobs
Once the schemas are defined, the pipeline begins with Scope. The first agent takes the research question and decomposes it into narrow search jobs.
In my run, the question was about quote relevance in medical transcript evaluation: a local 10B MoE judge could produce substring-grounded quotes, yet 41% of those quotes were irrelevant to the rubric criterion. A separate LLM verifier had already stalled at ~0.55 precision, with a target of 0.85.
Scope turned that into concrete angles:
Selection-time techniques (extractive / constrained citation)NLI / cross-encoder classifier for evidence relevance (low-label)Attribution evaluation practice (ALCE, RARR, AIS, RAG citation systems)UX degradation: showing weak or missing evidence honestlySkeptical: why small-LLM self-verification fails; distillation label economics
This compresses a vague systems problem into small research surfaces: quote-first generation, NLI classifiers, attribution benchmarks, product degradation, and a skeptical pass over the whole premise.
The search results followed those surfaces. The selection angle found Attribute First, FRONT, and ReClaim. The classifier angle found mDeBERTa, AttrScore, domain NLI checkpoints, SetFit, AttributionBench, and CTKFactsNLI. The attribution angle found ALCE, RARR, AIS, and related citation-quality work. The skeptical angle later mattered because it gave the verifier counter-evidence against claims that plain LLM self-verification would solve the problem.
Turning pages into claims
Search jobs return pages. Pages are still not research results.
The next step is Fetch/Extract: each selected page is fetched and converted into small claim objects.
At this point, the system is doing relevance extraction. It checks whether the page contains concrete, quotable statements that matter for the original question.
If yes, the extractor emits:
{
claim: "...",
quote: "...",
importance: "central | supporting | tangential"
}
If no, it returns an empty list.
The original research brief is repeated in the extractor prompt so the model pulls claims that help answer the root problem.
In my run, a model card for mDeBERTa-v3-base-xnli became useful only because the extractor connected it to the actual task: checking whether a rubric criterion is supported by a quote from a consultation. AttrScore was mapped to the same criterion description ↔ quoted evidence structure.
The final synthesis was careful here: it treated the UX path as under-covered and kept it out of decision-grade evidence. The evidence budget left that branch partially answered.
Building and ranking the claim pool
The midsection of the script runs a Map-Reduce pipeline. It coordinates the parallel search queries, filters duplicate URLs, and fetches sources.
Plain code owns the budget management. The harness sets MAX_FETCH = 15. It has a strict rule: if the budget is exhausted, medium and low relevance sources are discarded, while high relevance sources are let through.
In my run, the extractor built a pool of 122 raw claims from 25 unique sources. Sending 122 claims to a 3-vote verifier panel would cost 366 expensive LLM calls.
This step is triage. It decides which extracted claims are important enough to spend verifier calls on.
The harness uses a two-step sorting logic written in plain code:
- Sort claims by importance (
central>supporting>tangential). - If equal, sort by source quality (
primary>secondary>blog>forum>unreliable).
The top 25 claims are sent to the verifiers. The other 97 are dropped. The selection of what to check is a database query.
Adversarial verification
This is where the system finally judges support. Every claim in the top 25 is checked by three independent agents. The prompt acts as a failure checklist: checking for overreach, contradictory sources, or cherry-picked data. The explicit instruction is: "Default to refuted=true if uncertain." The burden of proof is entirely on the claim.
Here is how the adversarial panel caught an overreach in my run. The model extracted a claim: "Even fine-tuned GPT-3.5 is capped well below 0.85 precision."
Two verifier agents voted refuted: true. Their evidence: the source paper (AttributionBench) only reported a static ~80% macro-F1. Because macro-F1 is an aggregate metric, you can tune the operating threshold to hit $\ge$0.85 precision by trading off recall. The claim was killed 2-1 for overreaching the data.
Meanwhile, the claim "stronger models alone leave attribution unsolved" survived 3-0, verified against the primary PDF showing zero-shot GPT-4 underperforming a fine-tuned 770M FLAN-T5.
The implementation also separates verifier failures from refutations. If the verification panel crashes, it returns unverified. An infrastructure failure is classified as inconclusive, then surfaced in the final report.
Treating the web as untrusted input
The harness also protects the runtime around the model. Once agents scrape the open web, page titles, URLs, and snippets become untrusted input flowing into logs and progress labels.
The middle of the file contains a compact regex block for sanitizing URL hosts and terminal labels. The comments explain why \\ is a path separator in URLs, and why they strip ANSI escape sequences, bidirectional text overrides, and IDN homographs (like a Cyrillic 'a' in amazon.com).
That is part of the harness too. It constrains the model's outputs and external inputs before they reach the runtime.
The takeaway
To make an LLM do a hard task reliably, start by modeling the task outside the model. Define the objects that can exist. Split the work into small typed steps. Bind every extracted claim to evidence. Rank before you verify. Treat verifier failures as their own state. Treat external inputs as untrusted.
The model does the local reasoning. The harness decides what it is allowed to see, emit, verify, and ignore.
Top comments (0)