DEV Community: Raj Kumar

RAG Is Dead. Who Killed It?

Raj Kumar — Tue, 21 Jul 2026 12:45:45 +0000

Your RAG pipeline just returned the wrong answer. Again.

The query was simple. The doc had the answer. You watched the top-5 chunks come back and none of them held it. The right paragraph sat two pages away from anything your embedding model thought was "similar." So the LLM hallucinated something plausible, your user trusted it, and you spent the afternoon tuning chunk_size from 512 to 1024 and back.

This is not a tuning problem. This is the architecture.

Vector RAG was a clever hack for a 2022 problem: LLMs had tiny context windows, so we shredded documents into chunks, embedded them, and prayed cosine similarity would find the right shred. It worked well enough on FAQ pages and Notion dumps that we called it a pattern and shipped it into production at every company on earth.

Then we pointed it at a 200-page 10-K. A Kubernetes operator manual. A pharma SOP. A legal contract. And it fell apart.

The Chunking Lie

Vector RAG rests on four assumptions. All four break on real documents.

Assumption 1: Similarity equals relevance.
It doesn't. "revenue declined" and "revenue grew" sit next to each other in embedding space — opposite meanings, near-identical vectors. Domain jargon drifts even worse. "margin" means one thing in finance, another in typography, another in ML. Your general-purpose embedding model doesn't know which doc it's in.

Assumption 2: The right answer fits in k chunks.
Why is k=5? Because someone's tutorial said so. Real answers span sections, reference earlier definitions, sit inside tables that got shredded on ingest. Top-k is a guess dressed up as a hyperparameter.

Assumption 3: Chunk boundaries preserve meaning.
They don't. A 512-token window slices mid-sentence, mid-table, mid-recipe. The chunk that contains the answer loses the heading that gives it context. The LLM sees "...bake at 425°F for 40 minutes" and has no idea which dish "it" belongs to.

Assumption 4: Documents are bags of text.
They aren't. A 10-K has a structure. A manual has chapters. A contract has clauses that reference other clauses. Vector RAG throws all of that away on ingest, then acts surprised when retrieval can't find its way back.

Every RAG failure I've debugged in the last two years traces back to one of these four. Bigger embedding models don't fix them. Reranking papers over them. Hybrid BM25 just postpones them. The problem isn't the retriever — it's the premise.

Humans don't read a 200-page doc by cosine similarity. They open the table of contents.

How You Actually Find an Answer

Forget databases for a second. Picture a 400-page cookbook, and you want the roast chicken recipe.

You don't read all 400 pages looking for the passage most "similar" to chicken. You flip to the contents, find Poultry, turn to that chapter, run your finger down to Roast Chicken, and read the whole recipe — ingredients, oven temp, and timing all together, in order.

Now imagine doing it the vector-RAG way: tear every page out, shuffle them into a bin, and pull the five loose pages whose words look closest to your question. You might grab the ingredients list but miss the oven temperature on the next page. You might grab a chicken stock recipe by mistake. And you'd have no idea which chapter any page came from.

That's the difference. Humans use reasoning over structure; vector RAG uses similarity over shredded text. The table of contents isn't decoration — it's a map of meaning the author already drew for you, and chunking throws it in the trash.

So what if retrieval kept the map?

Meet the Killer

It's called PageIndex.

Instead of embedding chunks, PageIndex builds a tree from the document's structure — chapters, sections, sub-sections — mirroring the table of contents. (No table of contents? An LLM generates the tree from the page text.) It keeps the cookbook intact instead of shredding it. Each node knows what it contains. No vectors. No cosine similarity. No k.

At query time, an LLM walks the tree the way a human walks a TOC. It starts at the root, reasons about which branch is most likely to hold the answer, descends, re-evaluates, and either drills deeper or backs out. When it lands on the right node, it reads the full, intact section — headers, tables, and context still attached.

Two things fall out of this for free:

1. Context stays whole. The answer arrives with the section that frames it. That orphaned "...bake at 425°F for 40 minutes" from earlier now carries its heading — Roast Chicken — instead of floating free.

2. Retrieval is explainable. You get the reasoning trace — the exact path from root down to the answer node. When it's wrong, you see where it turned wrong. Try debugging a cosine score.

Does it actually work?

PageIndex is built and open-sourced by Vectify AI (docs). And the numbers on structured docs are hard to argue with. On FinanceBench — QA over real financial filings — Vectify's Mafin 2.5, built on PageIndex, reports:

98.7% accuracy — vs 31% for ChatGPT-4o with search, and 45% for Perplexity, on the same documents.

Worth being honest: that's a vendor-reported number, not an independent replication. But the pattern holds — structured documents are exactly where similarity search hurts most, and where a tree wins biggest.

Vector RAG searches for text that looks like the answer. PageIndex reasons about where the answer lives.

Same Question, Same Doc

Enough theory. Back to that 400-page cookbook, one real query: "What temperature do I roast the chicken at, and for how long?"

Vector RAG returns the top-5 chunks by similarity. Four are about chicken stock, chicken salad, chicken stir-fry — they all say "chicken." One catches "425°F for 40 minutes" but lost its heading on chunking, so the model can't tell if that's the roast chicken or the roast potatoes two pages over. Answer: confident, wrong.

PageIndex walks the tree — root → Poultry → Roast Chicken → the intact recipe, temperature and timing under their own heading, ingredients and steps still attached. It reads the answer in context and gets it right.

Same book. One retriever guessed at surface text; the other navigated to where the answer actually lived — the same way you would.

The Part Everyone Misses: Cross-References

Real documents are full of pointers. "Prepare the brine as described on page 12." "See the pan-searing technique in Chapter 2." "Use the spice blend from the recipe above." The roast chicken recipe only makes sense alongside a brine three chapters back.

Vector RAG is structurally blind to this. It fetches the top-k chunks that look like your query — and a cross-reference looks like nothing. The chunk that says "brine the bird first" scores well; the chunk three chapters away that actually explains how to make the brine doesn't, so it never comes back. The model answers with half the recipe and no idea the other half exists.

PageIndex sees the whole tree at once. When the node it lands on points elsewhere — "see the brine on page 12" — the navigating LLM can follow that pointer and pull the referenced node too. It isn't a dedicated link-resolver; it's a side effect of reasoning over the full structure instead of a bag of disconnected fragments. But the result is what matters: the dependency gets read, not dropped.

A chunk is an island. A tree node knows its neighbors.

But I Have a Whole Shelf, Not One Book

Fair — real systems aren't one cookbook, they're a library of thousands. The instinct is to reach for vectors again here: embed every book, do a similarity search to pick candidates, then navigate inside. And you can — a vector first pass narrows the shelf, then PageIndex walks each book to the exact recipe. It's fast, and it's the current default in Vectify's own API, mostly because walking a giant tree with an LLM costs more calls than a millisecond vector lookup.

But notice what just happened: you dragged the shredder back into your pipeline for the one job it was never bad at — a coarse "which books are roughly about Italian food?" And now you're maintaining an embedding store, a vector index, and a reasoning layer to answer one question.

The cleaner idea is to go structure all the way up: the whole shelf becomes one big tree. Cuisine → book → chapter → recipe, one hierarchy the LLM navigates top-down — no embeddings, no separate vector store, no re-indexing when documents change. Vectify ships this as their "file system" layer for corpus-scale search. It's the newer path and still maturing, but it's the honest end state of the argument. If structure beats similarity inside a document, it beats it across a collection too.

The lesson isn't "add vectors for scale." It's don't reach for a similarity index at every level as a reflex — that's overhead masquerading as architecture. Use it as a temporary speed shortcut if you must, and drop it the moment structure can do the job. Either way, the shredder never comes back.

Which One, When

PageIndex is not free. A vector lookup is ~50ms and costs cents; walking a tree with an LLM takes seconds — sometimes tens of seconds — and multiple model calls. You're trading milliseconds and cents for accuracy, whole-context answers, and an audit trail. Whether that's worth it is entirely about your workload:

Workload	Reach for	Why
Dense filings, contracts, SOPs, manuals	PageIndex	real structure to navigate; answers need whole-section context + an audit trail
Steps or clauses that reference other steps	PageIndex	follows cross-references vector search is blind to — the difference between a right answer and a confidently wrong one
A million structured docs	PageIndex	one hierarchy, navigated top-down — vectors optional as a speed prefilter, not a requirement
100k tickets, FAQs, chat logs	Vector RAG	genuinely flat — no structure to navigate, sub-second latency, cents per query
Real-time chatbot autocomplete	Vector RAG	~50ms lookup; a tree-walk is far too slow

Vectors still win where the data is genuinely flat and latency is everything — chat logs, FAQs, autocomplete. But the moment your documents have structure, that's PageIndex's turf, one book or a million.

So Is RAG Actually Dead?

What's dead is the reflex to reach for vector-only RAG as the default for every retrieval problem. That default was never right for structured documents; we just didn't have a better option worth the effort. Now we do.

Retrieval is turning from a similarity-search problem into a reasoning problem. Vectors were the hack we needed when models were small and context was scarce. Neither is true anymore.

Point your worst-performing RAG query — the one you've been re-chunking for a week — at a tree instead. The code is open source: github.com/VectifyAI/PageIndex. Then tell me what breaks. Change my mind: what breaks PageIndex?

If you're building at the seam between real apps and LLMs, that's most of what I write about here — follow along if it's your kind of problem.

Raj Kumar is a Toronto-based tech lead. Building software since 2011, AI since 2014 (before the buzzword). Ruby, Python, and system architecture; currently working the seam between production apps and LLM agents.

Your Rails App Is Already AI-Ready. You Just Don't Know It.

Raj Kumar — Tue, 05 May 2026 12:00:00 +0000

I've spent the last several months watching a Claude-based agent read a production Rails codebase — opening files, tracing scopes, following background jobs, identifying the right place to investigate a bug I'd written years ago. The strange thing isn't that it works. It's how little I had to do to make it work.

I went in expecting to write a lot of glue. Entity maps. Canonical module summaries. Hand-built manifests telling the model where things live. I wrote none of it. The agent reads config/routes.rb, follows the conventions Rails apps have agreed on for fifteen years, and finds what it needs.

That disconnect is what this post is about. Not "AI for Rails." Rails, seen through an LLM's eyes.

The agent is a new hire who reads fast

The mental model that helped me most is this: the agent is a new hire on day one. It has never seen your code. It has no institutional memory. But it has general knowledge of Ruby and Rails, the way a new hire does. It reads quickly and forgets nothing — and it believes what it reads.

Under that frame, every decision you've made about naming, layout, and structure is either cheap for the agent to navigate or expensive. And "cheap to navigate" almost always means "looks like every other codebase of this kind." That's where Rails wins, and wins in a way that surprised me even after a decade and a half of writing it.

Here are five things I kept noticing. None of them are features. All of them are conventions.

1. `app/services/` is a keyword, even to the model

We work in regulated clinical software. Tickets often look like: "Customer reports the nightly audit export for study X came back empty." An ordinary support engineer would know, without thinking, to start at app/services/exports/. They don't know because someone documented it. They know because that's where Rails apps put services.

The agent knows the same thing, for the same reason. It was trained on millions of Rails apps following the convention. Ask it where the logic for X lives; it opens app/services/ and is often right on the first read.

Imagine the alternative. Seventeen repositories with seventeen philosophies — hexagonal architecture here, DDD bounded contexts there, lib/ somewhere else. All defensible. Taken together, each new ticket forces the agent to spend its first five minutes learning your idiosyncrasies before it can do any work. Those minutes are tokens. Tokens are budget. Budget is what decides whether a per-ticket analysis costs $0.30 or $5.

Shared conventions compound. Clever per-repo conventions don't.

2. Active Record scopes are self-documenting in a way no repository pattern is

Here's a scope close in shape to one that lives in our codebase, with domain nouns swapped:

scope :visible_to, ->(sponsor) {
  joins(:study).where(
    studies: { sponsor: sponsor, archived_at: nil }
  )
}

One line. It tells you: this model is tenant-scoped to a sponsor, through a study association, and archived studies don't count. The agent reads that scope and has absorbed three things it would otherwise need to discover by tracing across files: tenancy rule, association path, lifecycle rule.

Compare to the equivalent in a codebase using a repository-pattern abstraction over an ORM — a StudyRepository.find_visible_to(sponsor) method that delegates to a query builder that composes predicates defined somewhere else. Same behavior; three files to read; each file requires inference to connect to the next.

Agents reason the way tired humans do: by pattern-matching against the nearest legible evidence. Active Record puts the legible evidence at the definition site. The agent reads once and knows. No tracing, no inference, no token budget spent on reconstruction.

This is why, when I watch the agent answer questions about multi-tenant access, it almost never gets it wrong. The tenancy rules are inside the model, at the scope. It can't miss them.

3. Background jobs that name themselves

AuditTrailReplayJob.perform_later(study_id, since: 24.hours.ago)

Tell me what that does. You told me without reading the class.

Now picture the same operation as a publish to a generic queue with a JSON payload and a routing consumer on the other end that dispatches based on an event type string. Same behavior. The agent has to find the producer, follow the payload, find the consumer, match the event type, read the handler. Four files, three inferences, and one place where a typo in the event type silently breaks everything.

Sidekiq, GoodJob, Solid Queue — doesn't matter which — gave us a convention where background work has a name and a signature that describes itself. For an agent trying to trace "what happens after a study closes?", that's an enormous gift. It finds the controller that receives the close action, sees three Job.perform_later calls with self-describing names, and is done. It doesn't spelunk through an event bus.

I've watched this specifically make the difference on root-cause analysis. The agent's reasoning chains are shorter. Shorter chains mean fewer hallucination opportunities. Fewer hallucinations mean I trust the output.

4. Release branches are how Rails ships, and that matters more than you think

Our platform ships on a quarterly cadence. Customers run a pinned version — let's call it 26.1.0 or 26.2.0. Our agent's first concrete task on any new ticket isn't to analyze the error. It's to resolve which branch the customer is actually running, and to read that branch.

If it reads main, every downstream conclusion is drifting. "This function handles X" — yes, on main, not on the release the customer is on.

This sounds like an ops detail, not a Rails thing. It matters because Rails-shop culture expects release branches. Most of the Rails teams I've worked in or read about branch from main for releases and maintain those branches for the release's lifetime. When you build tooling that assumes this, you're building on ground that already holds the weight.

Not every ecosystem works this way. I've watched teams attempt the same pattern against a polyglot stack where some services cut tags, others branched, others shipped trunk-based with feature flags. Every service had a different answer to "what code is the customer running right now." Each variation was a bug waiting to happen. Rails shops tend to vary less on this axis, and that homogeneity is worth real money when you're trying to pin an agent to the truth.

5. `config/routes.rb` is the densest map of application structure I've ever handed to an LLM

When a ticket says "the customer is seeing a 500 on the study-closeout page," the agent does this:

Reads config/routes.rb.
Identifies the controller action.
Opens the controller.
Follows to the service.
Lands on the specific method where the exception is raised.

I've watched this exact chain complete in under a minute on a real ticket, zero guidance from a human. No custom index, no pre-built call graph, no LSP server in the loop. One routes file, read top to bottom.

That file tells the truth. It's one place. It reads like a table of contents. No custom dispatcher, no decorators rewriting paths at runtime, no middleware hijacking the first matching route. Just resources :studies, with nested routes for the actions that matter.

This is the single highest-leverage file in a Rails app for an agent to read. If you've been putting off cleaning up your routes file, now you have a new reason.

What I was wrong about

I expected to spend the first month of this project writing helpers for the agent — summaries, structured manifests, entity graphs, canonicalized module docs. The classic "AI-friendly codebase requires extra scaffolding" thesis you see a lot of right now.

I wrote none of it.

The agent reads config/routes.rb, navigates app/services/, reads Active Record scopes and follows associations, reads named jobs and understands them. The scaffolding I'd pre-written to help it turned out to be noise the model had to weigh against the code itself. The most useful thing I did was delete the "helpful" entity summaries I'd generated up front and trust the agent to read the code instead.

If you take one concrete thing from this post, let it be that. Don't pre-chew the codebase for the model. Let it read what your new-hire Rails engineer reads.

The broader point

There's a common framing that AI-native development will require "AI-friendly" codebases written in some new way. My experience points the other direction.

The AI-friendly codebase is the one that was already legible to the tenth engineer who joined the team on day one. Rails has been optimizing for that engineer for twenty years. Shared conventions. Declarative syntax. Predictable layout. Boring, obvious names.

Rails didn't set out to be AI-friendly. It set out to make humans productive by reducing the number of arbitrary decisions they had to make. LLMs turn out to want the exact same thing, for the exact same reason: both are pattern-matching engines whose cost of understanding drops when the code looks like every other codebase of its kind.

Rails optimized for the human version. It gets the LLM version for free.

Where this is going

What I've described here is the substrate — the reason any of this was tractable to begin with. The actual system on top of it — what the agent does with this legibility, the invariants it defends, the specific places it still fails, what we had to build around it — is a longer story.

I'm submitting the full writeup as a talk at Rails World 2026. There's no public version of the system yet, and there won't be before the talk. If it gets accepted, come find me in Austin. If it doesn't, I'll write it up afterwards.

In the meantime, if you're experimenting with agents against your own Rails app, I'd genuinely love to hear what you've found — especially the things that broke in ways you didn't expect. That's where the interesting material lives, and it's what doesn't make it into blog posts until someone shares it first.

Raj Kumar is a Toronto-based tech lead. He writes Rails for a living and is currently most interested in the seam between Rails applications and LLM agents.