Most code-retrieval tools have exactly one voice: confident. You ask, they return their top-k, and the agent on the other end has to guess whether to trust it. That guess fails worst in one specific case: the empty result.
An empty search result is ambiguous. It can mean the thing doesn't exist, or it can mean the index couldn't see it. Those are wildly different facts, and an agent that treats every empty result as "doesn't exist" will confidently hallucinate the negative. "There's no retry logic in this codebase" is a strong claim to make off a search that ran against a half-built index.
We think retrieval responses should carry their own trust signals, machine-readable, so the agent can gate its next action on data instead of vibes. Here's the contract jCodeMunch ships, offered as a shape we'd like to see more of the ecosystem adopt.
The four-state verdict
Every retrieval response carries a verdict:
-
ok: confident matches returned. -
low_confidence: matches exist but score below the confidence threshold. Verify before relying on them. -
absent: the corpus was genuinely scanned and the answer isn't there. The verdict carries the scan counts (how many symbols and files were actually examined), so "absent" is a claim with evidence, not a shrug. -
degraded: the index itself is impaired (stale, partial, or mid-rebuild). A missing or thin result may be truncation, not absence. Don't conclude anything from silence in this state.
The absent-versus-degraded distinction is the whole point. "Absent" is a positive claim: we looked, here's how much we looked at, it isn't there. "Degraded" is the index admitting it can't be trusted right now. Collapsing those two into one empty list is lying by omission, and it's what most retrieval surfaces do today.
For an agent, the practical difference is enormous. On absent, stop re-querying; the code provably isn't indexed there. On degraded, don't conclude anything; refresh the index or fall back to a raw read. Without the distinction, agents burn tokens re-asking questions the index already answered, or worse, they build plans on a negative that was never proven.
Signal two: calibrated confidence
Verdicts cover the empty case. For non-empty results, every response carries a 0 to 1 confidence score composed from measurable components: the top-1 versus top-2 score gap (is the winner dominant or a coin flip?), absolute top-1 strength, whether an exact identity match is present, and index freshness.
One number, gateable. An agent can decide "above 0.8, use it; below, fetch the source and check" without parsing prose.
Signal three: per-symbol freshness
Every result entry is stamped fresh, edited_uncommitted, or stale_index, derived from comparing the index's recorded git HEAD against the working tree's actual HEAD plus per-file mtime checks. The index doesn't pretend the world stopped at indexing time. It tells you, per symbol, whether what it's showing you still matches the bytes on disk.
The contract is machine-checkable
None of this is prose in a README. The verdict contract is published as a JSON Schema (schemas/retrieval-verdict.schema.json in the repo), alongside schemas for the confidence provenance and the ranked-context response shape. A CI pipeline, yours and not just ours, can validate live responses against it mechanically. If a response ever ships without the honesty fields, that's a schema violation, not a style regression.
Why bother
Retrieval quality gets all the benchmark attention. Retrieval honesty gets almost none, and it's the thing agents actually break on. A tool that's right 95% of the time but can't tell you which 5% it's unsure about forces the agent to treat every answer with the same suspicion, which means re-verifying everything, which means the retrieval tool saved you nothing.
Three honest signals (verdict, confidence, freshness) let the agent spend verification effort only where the tool itself flagged doubt. That's the trade we think the whole ecosystem should be making, and the schema is there for anyone who wants to make it.
The full mechanics are in Chapter 1 of the technical manual: https://github.com/jgravelle/jcodemunch-mcp/blob/main/UNDER_THE_HOOD.md
If a claim in this post doesn't match the source, that's a bug in the post. File an issue and I'll fix it.
-jjg
Top comments (1)
The verdict shape is useful, but
absentneeds a coverage contract as well as scan counts. Examining 10,000 symbols proves little if generated files, unsupported languages, ignored paths, parser failures, or an uncommitted rename were excluded. I would return the corpus/index generation, included and excluded scopes, parser error counts, and unsupported-file counts with the verdict. The 0–1 confidence also needs empirical calibration rather than only a weighted heuristic: keep planted positive/negative queries, publish reliability curves or ECE/Brier by query class, and version the calibration with the scorer. Then an agent's 0.8 threshold has measured meaning instead of merely looking precise.