RAG Isn't Dying. It's Doing the One Thing Agents Can't.

Where Should the Knowledge Live? Building a Cite-or-Refuse RAG (and Weighing the Alternatives)

---

I built a compliance RAG on one rule — every claim carries a real document reference, or the system refuses. Then I went looking at the alternatives. Here's the honest accounting: what I built, what exploring PageIndex taught me, and what I found when I evaluated fine-tuning.

Before any architecture, one requirement drove everything: an answer is only trustworthy if it points at a real source. Not a paraphrase, not "the model is fairly sure" — a specific document, a specific page, an exact quote you can go read yourself.

In any regulated or high-stakes domain, a confident wrong answer isn't an embarrassing bug — it's a liability. So the choice is stark: either be confident and cite the source, or refuse to answer. There's no safe middle ground where the system guesses and hopes. The system either returns an answer with a citation attached to every claim, or it returns "Insufficient evidence found." Refusing is not a failure mode. It's the core feature. A citation you can verify is what separates a tool a domain expert can defend from a chatbot that merely sounds authoritative.

That one requirement — cite a real reference, or refuse — turns out to decide almost every downstream choice. This post is the honest version of how those choices went: the RAG I actually built, the reasoning-tree approach I seriously explored, and the fine-tuning path I evaluated and set aside. I'll put real costs and real advantages next to each, and I'll end genuinely unsure about one thing and hoping you'll tell me what I'm missing.

---

1 · What I built — vector RAG with a hard refusal gate

A fairly classic retrieval pipeline with one opinionated addition. Documents are parsed and split into chunks; each chunk is embedded and stored in a vector database. At query time I embed the question, pull the closest chunks, and — this is the part that matters — score the strength of that top evidence and gate on it.

# the whole guarantee, in one line
if confidence < threshold:
    return refuse()          # "Insufficient evidence found."
else:
    return generate(chunks)  # answer, each claim cited to a chunk

Because the answer is generated from the retrieved chunks, every claim has somewhere to point: the source document, the page, the quoted span. And because refusal is a threshold on a real number, it's mechanical and auditable — I can tell an auditor "the best evidence scored below our floor, so the system declined," and show them the number. That legibility is the whole reason the design exists.

A few choices inside that pipeline mattered more than I expected:

• Parsing with Docling. Chunk quality caps retrieval quality — you can't cite a table the parser mangled. I use Docling, IBM's layout-aware parser, as the default loader because regulatory PDFs are full of tables, columns, and reading-order traps that a naive text extractor flattens into nonsense. (A lightweight pypdf path stays available for simple text PDFs.)
• Parent-child chunking. This is the pattern I settled on: embed small child chunks so retrieval stays precise, but hand the LLM the larger parent chunk for context. Search on the needle, answer from the paragraph around it. It's meaningfully better than one flat chunk size trying to do both jobs.
• A swappable embedding model. Embeddings decide what "close" even means, so I kept the model behind an interface. I default to a self-hosted BGE model (bge-small-en-v1.5) to avoid shipping documents to a third party, with an OpenAI embedding path as a drop-in alternative for quick experiments. Different embedding models genuinely change which chunks surface — a dial worth testing, not set-and-forget.

Cost profile

Phase	When you pay	Order of magnitude
Chunking & ingest	Once per corpus (re-run only when docs change)	CPU-minutes to parse + split + embed the document set. Re-ingesting the whole corpus is minutes, not hours.
Retrieval	Every query	~1 embedding call for the question + a millisecond-scale vector lookup. Cheap enough to be interactive.
Generation	Every answered query	One LLM call over the retrieved context. Refusals skip it entirely — refusing is nearly free.

Advantages

• Citations are native. The answer is built from chunks, so every claim already has an address.
• Refusal is a hard signal. Weak retrieval → low score → decline. No model judgment required.
• Updates are cheap. A rule changes, you re-ingest one file in minutes. The knowledge lives outside the model, on disk.
• Fully auditable. Every answer and every refusal traces to an observable step.

The honest weakness

Retrieval is only as good as similarity. Embedding-based search matches on topical closeness, not on whether a passage truly supports the answer — so a question about one threshold can surface a high-scoring chunk about a different threshold in the same family. High score, wrong rule. That single weakness is what sent me exploring the next two approaches.

---

2 · What I explored — reasoning-tree retrieval (PageIndex)

The near-miss weakness above is a known limitation of pure similarity search, and reasoning-tree approaches like PageIndex attack it head-on. Instead of chunk-and-embed, they build a hierarchical map of each document — a table of contents with summarized nodes — and let an LLM navigate it at query time, asking "which section most likely contains this answer?" the way a person uses an index. There's no embedding-similarity step in the core path at all.

I explored this seriously because the fit for structured regulatory text is genuinely strong: regs are deeply hierarchical, and a reasoner asking "does this section actually address the question?" is far less likely to be fooled by the vocabulary-overlap near-miss that trips up cosine similarity.

Cost profile

Phase	When you pay	Order of magnitude
Tree build / index	Once per corpus	More than chunking — you're summarizing nodes with an LLM, so it's LLM-calls at build time, not just CPU-minutes. Still a one-time, re-runnable cost.
Retrieval	Every query	Multiple LLM calls per query — each navigation hop is a model call. Meaningfully slower and pricier per query than a vector lookup, and it grows with tree depth.
Generation	Every answered query	One LLM call over the selected nodes, same as RAG.

Advantages

• Structure-aware. It exploits the document hierarchy that vector search flattens away.
• Fewer near-misses. Reasoning about relevance beats measuring distance on structured text.
• Still citable. Answers point at real tree nodes — the cite-a-source property survives.

Why I didn't just swap it in

The catch is what it does to refusal. My gate thresholds a scalar — the strength of the top evidence. A navigation trace doesn't natively produce that number; refusal would become "the model decided the tree had no answer," which is a softer, model-judged signal than a threshold I can defend line-by-line. Add the per-query latency and cost of multiple LLM hops on an interactive system, and the tradeoff isn't obvious.

---

3 · What I evaluated — fine-tuning a base model on the PDFs

An honest disclaimer first: fine-tuning is not my area of hands-on expertise. I haven't trained a model on this corpus, and I won't pretend the cost figures below come from my own training runs — they're my best understanding of the shape of the work, not measured numbers. I evaluated this path on paper, reasoned about how it collides with the cite-or-refuse requirement, and decided not to build it. If you've actually done this and I've got something wrong, I especially want to hear from you.

With that said: the most tempting idea, and the one I spent the most time reasoning through before deciding not to build it, is to stop retrieving and just train a base model on the documents so it "knows" them. It sounds like the more advanced move. Evaluated against the cite-or-refuse requirement, it comes apart — and the failure looks structural rather than a matter of quality.

Cost profile

Phase	When you pay	Order of magnitude
Dataset build	Before every training run	Turning raw PDFs into training pairs is real, manual, error-prone work — and it has to be redone as the corpus grows.
Training	Per update	GPU-hours (and the dollars that come with them) for each run. Not interactive, not incremental — a batch job.
A rule changes	Per change	Retrain to unlearn a stale figure. Compare: RAG re-ingests one file in minutes. This is the cost that compounds most painfully.
Inference	Every query	One model call — genuinely cheap and fast per query. The cheapest column, and the only one where fine-tuning wins.

Advantages (and I want to be fair to them)

• Fast, self-contained inference. No retrieval hop, no vector store to run.
• Fluent, in-domain phrasing. The model internalizes the vocabulary and reads as a native of the domain.
• No retrieval infrastructure to build, host, or keep healthy.

Why it fails the requirement anyway

Every advantage above is real, and none of them matter if the system can't do the one thing it exists to do. Three problems that look structural to me, in order of how badly they seem to hurt:

• No source to cite. Knowledge in the weights is generated from parametric memory — there's no chunk, no page, no quote to attach. The answer requires a citation and the model has nothing to fill it with. That alone disqualifies it here.
• Confident exactly where it's dangerous. Regulations turn on precise figures, and a fine-tuned model produces them fluently whether or not it memorized the right one. The near-miss confusion doesn't get fixed — it gets baked into the weights, invisible and un-inspectable.
• It can't refuse. A retrieval system has a real "found nothing" state. A generative model always generates — there's no mechanical place for the gate to stand, and no honest way for it to say "the rule is silent on this."

Where I landed: Fine-tuning is the wrong tool for knowledge and the right tool for behavior. I won't train facts into a generator I need to cite and audit — that just produces a more fluent, less honest answer. If I fine-tune anything, it'll be the retriever — a domain-tuned embedder or reranker sharpens which evidence surfaces while every guarantee stays intact.

---

The pattern underneath all three

Line the three up and they're answers to one question — where does the knowledge live? Vector RAG and reasoning-trees keep it outside the model, where it can be quoted, swapped when a rule changes, audited by a stranger, and reported as absent when it's missing. Fine-tuning moves it inside, where it can't be cited, goes stale silently, and never admits it doesn't know.

Property	Vector RAG (I built)	Reasoning tree (I explored)	Fine-tune (I evaluated)
Cite a real reference	✅ from the chunk	✅ from tree nodes	❌ no address
Hard, auditable refusal	✅ score < threshold	⚠️ soft, model-judged	❌ always generates
Cost to update a rule	✅ minutes (re-ingest)	✅ minutes (re-index)	❌ retrain (GPU-hours)
Per-query cost	✅ low (1 embed + lookup)	⚠️ high (multi-hop LLM)	✅ low (1 model call)
Confident-hallucination risk	✅ low (gated)	⚠️ low–med	❌ high (baked in)

Read down the columns: the two that keep knowledge external can do what a compliance system must; the one that internalizes it can't. For my requirement this isn't a close call decided by retrieval accuracy — it's decided by topology, upstream of any benchmark.

And the discipline that held across all three was the same: don't optimize a number you can't see. Build the instrument — a set of questions that should be refused, and one honest metric for how often the system answers when it should have stayed silent — then let measurements settle the debates that demos can't.

---

Where I'm genuinely unsure — tell me what I'm missing

I've committed to external knowledge and a hard refusal gate because the compliance domain seems to demand it. But I hold that conviction loosely, and there are a few directions I haven't resolved. I'd genuinely like your read on any of these:

• Hybrid over either/or? Is the real answer a reasoning-tree retriever feeding a threshold-able confidence signal — PageIndex's navigation quality with RAG's hard gate? Has anyone made tree-retrieval produce a defensible refusal number?
• Fine-tuning the retriever, not the generator. A domain-tuned embedder or reranker on hard near-miss pairs seems like the one place training helps without costing me a guarantee. Worth it, or is a good reranker enough?
• Is a threshold too blunt? "Confidence < X → refuse" is legible, but a single scalar is a crude proxy for "does this evidence actually support the claim." What's a better refusal signal that stays auditable?
• Am I wrong about fine-tuning entirely? Is there a citation-preserving way to put knowledge in weights that I'm dismissing too fast?

If you've built something in this space — or you think I've drawn a line in the wrong place — I'd love to hear it. Comments, disagreements, and "you missed an approach" are all very welcome.