This came up in Chinese AI developer circles recently: IBM has been promoting something called Chunkless RAG, where instead of the standard document-to-chunk-to-embedding pipeline, an AI agent navigates document structure the way a human reader would. Tools like Docling parse PDFs into structured representations, preserving headings, tables, and section hierarchies, and then the agent walks that structure to find relevant content rather than doing cosine similarity on chopped-up text.
The pitch is appealing. Anyone who has watched a RAG pipeline confidently return three fragments from the middle of a dense technical spec, all of which are semantically close to the query but structurally meaningless without their surrounding sections, knows that fixed-size chunking is not a solved problem. So on the surface, this sounds like progress.
But I think the core claim, that document structure is the real problem chunking destroys, is significantly oversold.
The structure argument assumes your documents have structure
Real production corpora are messier than IBM's demos suggest. Legal contracts with inconsistent heading hierarchies. Scanned PDFs where Docling's layout parser confidently returns garbage. Internal wikis where someone decided to put the entire product spec in one enormous table. Support ticket exports. Slack export dumps. Email threads.
For these, the Chunkless approach does not obviously win. If the parser cannot reliably extract a meaningful document tree, the agent navigating that tree is just traversing noise with extra latency. You have replaced one failure mode (lost context from arbitrary splits) with another (hallucinated structure from a bad parser). The latter can be harder to detect because it looks like the system understood the document.
Fixed-size chunking with overlap is dumb, yes. But its failure modes are predictable. You can measure them. You can tune chunk size and overlap against a retrieval benchmark and watch the numbers move. The agent navigation approach has failure modes that are much harder to surface in offline evaluation.
The actual bottleneck is usually not where the cuts happen
In most pipelines I have seen discussed or documented, the retrieval precision problem is not primarily that chunks lose structural context. It is that the query and the relevant passage do not share enough lexical or semantic overlap for retrieval to work at all, regardless of how you chunked. This shows up especially for multi-hop questions where the answer requires synthesizing information from sections that are semantically distant from the query.
Chunkless RAG with agent navigation does not obviously fix this. If anything, it may make it worse: the agent still has to decide which section to navigate to first, and that decision is still a retrieval or classification problem under the hood. You have just moved the hard part one layer up and added agent call overhead on top of it.
The approaches that actually seem to move the needle on this are things like HyDE (hypothetical document embeddings), query expansion through prompt mutation, or hybrid retrieval that blends BM25 with dense vectors. One commenter on the original post mentioned a project called Knowhere that combines MinerU parsing with BM25 and prompt mutation and gets good results. That combination is not about preserving document structure as a graph; it is about improving the match between query intent and retrieved content.
What Chunkless RAG is probably good for
None of this means the idea is wrong everywhere. There is a real use case where it probably wins: long, well-structured technical documents where the user's question maps clearly to a section of the document. API reference docs. Regulatory filings with consistent section numbering. Academic papers with standard IMRaD structure. In these cases, knowing that the answer to "what is the data preparation methodology" lives in section 2.2 and navigating there directly is genuinely better than hoping your embedding space put the question and that section close enough together.
The legaltech commenter on the original post noted their team is building exactly this kind of system, and for structured legal documents, the structured navigation argument is more plausible.
But that is a narrower claim than "chunking is the wrong primitive for RAG." It is closer to "for well-structured documents, structure-aware retrieval outperforms flat chunking," which is much less surprising and much less revolutionary.
The framing bothers me more than the technique
The bigger issue is that "Chunkless RAG" as a brand encourages people to think of their chunking strategy as the core failure in their pipeline, when for most production systems it is not. Developers who are struggling with recall on heterogeneous corpora will read this, spend two weeks integrating Docling and building an agent navigation layer, and find that their benchmark numbers barely moved because the problem was never the chunk boundaries.
The useful question is not "should I chunk or not chunk?" It is "where exactly is my pipeline losing information, and what is the cheapest fix for that specific failure mode?" Sometimes that is better chunking. Sometimes it is a reranker. Sometimes it is query rewriting. Sometimes it is accepting that your source documents are too poorly structured for any retrieval strategy to work well without a preprocessing investment.
Chunkless RAG is a real direction worth watching, especially as document parsers improve. But treating document structure navigation as the successor to chunking, rather than as one useful technique among several, seems like it is going to send a lot of engineers down the wrong rabbit hole.
If you have run a direct comparison between structure-aware agent retrieval and well-tuned chunked retrieval on a messy real-world corpus, what did you actually see in the numbers?
Top comments (0)