RAG Without the Hype: Make Retrieval Observable, Testable, and Replaceable

How my agent actually finds answers — and what happens when it doesn't

Part 5 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.

---

"What's your refund policy?"

Something has to know the answer. The model doesn't. Not reliably.

The answer lives in documents the company wrote. Getting the right one in front of the model at the right moment has an intimidating name: retrieval-augmented generation (RAG). And most explanations make it sound like magic.

It's a pipeline. Score the documents, rank them, hand back the best few. That's all. The interesting part is what you do with the score.

Retrieval is a tool, not context stuffing

Fuzzy results behind a hard contract — that's the split this system is built on, and here it is made real.

The agent doesn't get knowledge silently injected into its prompt. It gets a tool, the same way it gets customer lookup:

// dev/tonal/support/knowledge/KnowledgeBase.java
public interface KnowledgeBase {

    /** Returns up to query.topK() articles, best match first. */
    List<ScoredArticle> search(Query query);
}

The agent decides when to search and what to ask. It never redefines what searching means, and every call is visible: query in, ranked articles with scores out.

flowchart LR
    A["Agent needs an answer"] --> B["Query: text + topK"]
    B --> C{"Scorer"}
    C --> D["Ranked articles + scores"]
    D --> E["Top-k back to the agent<br/>as tool result"]
    C -.-> F["keyword overlap (shipped)"]
    C -.-> G["embeddings (same port)"]
    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
    classDef alt fill:#f7f9fb,stroke:#c5d1dc,color:#24313f
    class A,B,D,E step
    class C decision
    class F,G alt

Deterministic first, semantic later

Here's the part that breaks with convention: the shipped implementation scores articles by keyword overlap — plain code, no embeddings, no API key.

// dev/tonal/support/knowledge/KeywordScoringKnowledgeBase.java
public List<ScoredArticle> search(Query query) {
    Set<String> queryTokens = tokens(query.text());
    return articles.stream()
            .map(article -> new ScoredArticle(article, score(article, queryTokens)))
            .filter(scored -> scored.score() > 0)
            .sorted(Comparator.comparingDouble(ScoredArticle::score).reversed())
            .limit(query.topK())
            .toList();
}

Why ship the dumb version? Because it's fully assertable.

Four tests pin the whole behaviour:

• the right article ranks first for a policy question
• topK actually limits results, zero token overlap returns empty (not "closest guess")
• ordering is strictly by score.

When an embedding-backed scorer replaces this class — same port, better matching on paraphrases — those tests define what honouring the contract means. Swap the implementation, keep the guarantees.

Scores are also why retrieval is debuggable. Every match carries its number:

$ java ... dev.tonal.support.knowledge.KnowledgeMain
# GET http://localhost:8080/rag/search?q=refund&k=3
[1.00] Refund Policy (billing)

# GET http://localhost:8080/rag/search?q=xylophone
No articles matched.

When the agent later cites a policy, you can replay the exact query and see exactly what it was shown. No black box between the corpus and the answer.

What wrong looks like

Retrieval being probabilistic means sometimes the ranker surfaces the wrong document — a rate-limit page for an SLA question. That's a failure mode like any other in this system: enumerated, mitigated, measured.

The mitigation starts with honesty about scores (a 0.2 match should be treated differently from a 1.0), continues through grounding answers in what was actually retrieved rather than what the model remembers, and ends with the eval suite scoring whether answers follow from sources. A wrong document isn't a bug you fix once. It's a quality property you track.

The pattern generalizes past support bots:

• enterprise search ranks but humans decide what's authoritative;
• clinical guideline systems surface candidates while physicians own the prescription;
• legal research tools find precedents while counsel argues them.

Ranked candidates plus human-or-rule judgment beats either pure search or pure generation everywhere it matters.

---

Comments

[Kane Lim]

Hello Antonio, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.

I strongly agree with treating retrieval as an explicit, observable contract rather than hidden prompt context. The interface separation is especially valuable because it makes the retrieval layer independently testable and lets you evolve from lexical scoring to embeddings without coupling the agent to the ranking implementation.

I would take this one step further with a retrieval evaluation harness. Store every query, candidate set, score distribution, selected documents, latency, and final grounding decision as structured telemetry. Then measure Recall@K, MRR, nDCG, zero result rate, citation coverage, and answer faithfulness across a versioned evaluation corpus.

For production, I would also introduce a confidence gate before generation. If the top score is below a calibrated threshold or the margin between rank one and rank two is too small, the system should abstain or trigger query expansion rather than manufacture an answer.

The architecture becomes particularly powerful when retrieval is treated like an interchangeable infrastructure component with regression tests and measurable SLOs. That gives you a controlled path from deterministic lexical search to hybrid retrieval, reranking, and eventually domain specific semantic models without sacrificing debuggability.

Really enjoyed the engineering discipline behind this approach. I would be happy to exchange ideas on retrieval evaluation and production RAG architecture.