RAG for codebases is hard. Trusting the answer is harder.

Blas Rodriguez Irizar — Mon, 29 Jun 2026 17:16:36 +0000

There's a good post making the rounds — "RAG for Codebases Is Harder Than It Looks" — on why naive retrieval falls apart on code. Filtering out node_modules, chunking on AST boundaries instead of token counts, preserving file-path metadata, treating architecture questions as graph traversals rather than nearest-neighbor lookups. All correct. If you're building code RAG, read it.

But notice where the author lands at the very end:

Developer trust depends on retrieval quality, honest uncertainty acknowledgment, and verifiable sources — not just algorithmic sophistication.

That's the part I want to pull on. Because everything before it — better chunking, better embeddings, repo maps — makes the model's answer more likely to be right. None of it makes the answer checkable. And once an LLM is writing your code, "more likely to be right" is not the same problem as "did it actually do what it just told me it did."

The retrieval is upstream of a bigger trust gap

Good RAG gets the model better context. The model then makes an edit and reports back: "I added the /v1/refund route, set MAX_RETRIES to 5, only touched the parser, and the tests pass."

Every one of those is a factual claim about the repo as it exists right now. And here's the uncomfortable measurement: across 100 real SWE-agent runs on real GitHub issues, 30% of the attempts that failed the test suite still claimed they fixed the issue — "the method has been successfully added," "the issue has been resolved." Ground truth from the SWE-bench eval, claims judged by an LLM, reproducible.

Better retrieval doesn't fix that. A model with perfect context can still over-claim what it did. The hallucination moves from "wrong fact about the codebase" to "wrong fact about my own change" — which is harder to catch, because it sounds like a status update, not a guess.

What "verifiable sources" actually requires

The article's three closing words — honest uncertainty, verifiable sources, no speculation — are exactly the design constraints I've been building truth around. But I'd argue you can't get them from a better RAG pipeline, because they're not retrieval properties. They're verification properties:

Honest uncertainty isn't "the model hedges." It's a system that can say Refused to "this is cleaner" because there's no measurable subject — and mean it. A verifier that bluffs is worse than none.
Verifiable sources isn't "cite a chunk." It's a verdict that rests on a binary fact you read directly: a file present/absent in the git diff, an AST symbol that exists or doesn't, an exact value, a command's exit code. Cited to file:line.
No speculation isn't a prompt instruction the model can ignore. It's a hard architectural rule: a fixed-rule engine decides the verdict, not a model. The agent cannot talk it into a different answer.

So truth does the inverse of RAG. RAG retrieves context to help the model answer. truth retrieves evidence to check what the model claimed — against the real code, the working-tree git diff, recorded command runs, and logs — and returns Supported / Contradicted / Refused, every verdict cited.

$ truth verify-turn "I added the /v1/refund endpoint, set MAX_RETRIES to 3, I only changed src/api.rs, and tests pass"

  ✓ Supported     I added the /v1/refund endpoint  (src/api.rs)
  ✗ Contradicted  set MAX_RETRIES to 3             (src/config.rs:1)
  ✗ Contradicted  I only changed src/api.rs        (src/api.rs)
  ✗ Contradicted  tests pass                       (recorded 2026-06-11 09:14 UTC)

An LLM only parses the sentence into a structured claim. The verdict comes from a deterministic engine.

The asymmetry the article gestures at, made explicit

"Honest uncertainty acknowledgment" has a sharp edge the article doesn't dwell on: a verifier that false-accuses gets uninstalled on day one. Crying wolf on a truthful agent is the one unforgivable failure.

So truth only ever says Contradicted on a structured binary fact — never on a noisy count (a text scan over-counts comments; a log window samples). "Nobody uses this," "updated all 4 call sites," "X is unused" — those surface their evidence as Inconclusive: a suspicion to act on, not a verdict that blocks. And it's measured, not asserted: a labeled corpus of real agent over-claims holds the false-contradiction rate at 0 as a hard CI gate. When truth blocks, it means it.

This is the same instinct the article has about retrieval noise — "if chunks are too large, retrieval becomes noisy" — applied to verdicts. A noisy verifier is as useless as noisy retrieval. Worse, because it erodes the exact trust it was supposed to provide.

Where they meet

The author is right that code RAG needs preprocessing, structure-awareness, and citations. I'd add one thing to the conclusion: those buy you a better-informed model, not a trustworthy one. When the model is editing your repo, the last mile isn't retrieval quality — it's whether you can independently confirm what it told you, with a citation, deterministically, and without it ever crying wolf.

truth is that last mile. It runs entirely locally (a single SQLite file in .truth/; your code never leaves the machine), exposes one MCP tool an agent calls on its own message before reporting done, and can be a hook the agent can't skip.

If you're building code RAG, your model's answers are about to get a lot better. Make them checkable too.

→ github.com/blasrodri/truth

DEV Community: Blas Rodriguez Irizar

RAG for codebases is hard. Trusting the answer is harder.

The retrieval is upstream of a bigger trust gap

What "verifiable sources" actually requires

The asymmetry the article gestures at, made explicit

Where they meet