One thing kept bothering me about retrieval systems: they often destroy the document before trying to understand it.
Imagine taking a technical specification and cutting it with scissors every five hundred words.
No attention to headings.
No respect for sections.
No care for where a thought begins and ends.
Just cut.
Stack.
Index.
That sounds absurd.
Yet this is remarkably close to what many retrieval pipelines do before a document ever reaches a vector database.
We're asking the wrong question
One of the most common discussions around RAG sounds like this:
What's the optimal chunk size?
256 tokens?
512?
Should chunks overlap?
After watching enough retrieval failures, I started thinking this isn't the right question.
The real question isn't where to split a document.
It's whether every document should be split the same way at all.
Imagine this document
Suppose you have a technical specification called "Rate Limiting Strategy."
It contains four sections:
- Why rate limiting exists
- How the algorithm works
- A code example
- Exceptions and edge cases
Now index it using fixed-size chunks.
The heading lands in one chunk.
The code example ends up in another.
The explanation of edge cases becomes a third.
Each chunk receives its own embedding.
Each becomes an independent point in vector space.
The text survives.
The document, as a coherent unit of meaning, does not.
Retrieval works perfectly
Now someone asks:
"What happens when a user hits the rate limit?"
The retriever returns the chunk containing the code.
From its perspective, everything worked.
The identifiers match.
The embeddings are close.
The similarity score is high.
But something important has disappeared.
The heading explaining what this code belongs to is somewhere else.
The paragraph explaining why the algorithm behaves this way is somewhere else.
The exceptions are somewhere else.
The model receives valid code—but stripped of the reasoning that made the code meaningful.
It can explain what the function does.
It cannot explain why the system behaves that way.
Structure is semantics
It's easy to think of document structure as presentation.
I don't.
A heading tells the reader:
Everything below belongs to the same idea.
A section boundary says:
This thought ends here.
A list says:
These items should be understood together.
Those aren't visual decorations.
They're part of the information itself.
When those signals disappear during indexing, retrieval loses something that humans almost never notice consciously—but rely on constantly while reading.
Meaning doesn't have a fixed size
Eventually I stopped asking:
How should I split this document?
Instead I started asking:
What is the smallest unit that still makes sense on its own?
Sometimes that's a paragraph.
Sometimes it's an entire section.
Sometimes it's a code block together with the explanation above it.
Sometimes it's a diagram and its caption.
An architecture decision record often makes sense only when the decision, its rationale, and its consequences stay together.
An API reference is easier to understand when an endpoint remains connected to its parameters and examples.
Meeting notes become much more useful when decisions stay attached to owners and action items.
The right boundary isn't a character count.
It's a logical boundary.
Retrieval found something. Understanding failed.
One pattern kept repeating itself.
The retrieval engine successfully found text related to the question.
The model confidently explained that text.
And yet the actual answer was still missing.
Not because retrieval failed.
Because the document had already been dismantled before retrieval even began.
A document is an argument
I've gradually stopped thinking of documents as long strings of text.
They're closer to arguments.
Someone deliberately grouped ideas together.
Separated topics.
Added headings.
Built a flow of reasoning.
That structure carries meaning every bit as much as the words themselves.
If I break that structure into arbitrary pieces before indexing it, I shouldn't expect a retrieval engine—or an LLM—to reconstruct it perfectly later.
A document isn't a sequence of tokens.
It's an argument made visible.
Next time I want to explore a different question: why finding a document isn't the same thing as discovering knowledge.
Top comments (35)
Agreed that we argue about chunk size when the question that matters is whether every document should be split the same way at all. What moved retrieval quality for me was letting the document's own structure set the boundaries, headings, tables, list items, instead of a fixed token count that cuts a table in half. Do you choose a split strategy per document type up front, or detect structure at ingestion so one pipeline adapts?
That's much closer to the question I've been exploring.
Rather than choosing a fixed splitting strategy for each document type, I'm leaning toward detecting the document's structure during ingestion and letting that define the boundaries.
The reason is exactly what you pointed out: real documents rarely fit cleanly into a single template. An ADR may contain code, meeting notes may include a decision table, and a technical spec may mix prose, diagrams, and examples. Relying only on a document type can still miss where the logical units actually begin and end.
My working hypothesis is that the document itself should tell us where those boundaries are. Headings, lists, tables, code blocks, and other structural elements aren't just formatting—they're part of how the author organized the argument.
I haven't compared this systematically against document-type-specific chunking yet, so I wouldn't claim it's universally better. But it feels like a more natural way to preserve the meaning that was already present in the document before retrieval begins.
"Structure is semantics" is the part that stuck with me. I'd been treating headings and tables as things to strip away before handing content to a model, and hadn't really seen how much meaning left with them. This gave me the words for it — and a nudge to go fix a couple of places where I was quietly dropping that structure. Grateful for the piece.
Thank you—that's really encouraging to hear.
One thing I was hoping the article would do was make people question the assumption that structure is just formatting. I hadn't really thought about it that way myself until I started looking at retrieval failures.
I like how you put it: "quietly dropping" structure. That's exactly the kind of failure that's easy to miss because nothing obviously breaks. The text is still there, but some of the meaning has disappeared along with the relationships the author originally created.
I'm glad the article gave you a different way to look at it, and I'd be curious to hear what you end up changing.
Since you asked what I ended up changing — here's the concrete version.
My project is an MCP server that fetches web pages for LLM agents (Markdown conversion via a Readability-based pipeline). After your article I went back and audited what that pipeline was throwing away, and found Readability was silently removing many
elements as clutter. Nothing obviously broke — the prose was all still there — but a statistics page came back with the narrative intact and the actual 243-row data table gone. Exactly the failure you described: the text survives, the meaning leaves with the structure.The fix I shipped: rescue the dropped tables, reinsert them at their original position in the document flow (a table separated from its section heading loses its subject), and merge multi-row headers so the table still reads as data.
The "different way to look at it" also reframed a feature I already had. The tool's outline mode returns the heading tree with per-section token estimates, and the agent fetches only the sections it needs. I used to think of that as a token-saving trick; your framing makes it clearer — the heading hierarchy is the author's own segmentation of the argument, so it's also the natural retrieval boundary. Structure-detected-at-ingestion, just happening at fetch time.
Project, for context: github.com/Rererr/amenbo
Thank you for coming back with a concrete example—that's much more valuable than simply agreeing with the article.
The table example really stood out to me. Nothing "looked" broken because the narrative was still there, yet the evidence had disappeared. That's exactly the kind of silent failure I was trying to describe.
I also like the way you connected outline mode to retrieval. Thinking of the heading hierarchy as the author's own segmentation of the argument is a much better way to describe it than "saving tokens." The efficiency is almost a side effect; the more important part is that you're preserving the structure the author used to organize the information.
And I hadn't considered the point about reinserting rescued tables at their original position. That's a good example of why preserving structure isn't just about keeping elements—it's also about preserving their relationships.
Thanks for sharing the implementation details. I'll take a look at Amenbo.
This is the retrieval mistake I see most often. Chunking makes documents searchable, but it can erase the argument structure. A paragraph is not just text; it has a role in the document. If retrieval loses hierarchy, references, section intent, and sequence, the model gets facts without the shape that made them meaningful.
I like the way you put it: "a paragraph has a role in the document."
That's a better way of expressing something I was trying to get at in the article.
A paragraph doesn't just contain information—it also sits within an argument. Move it somewhere else, or separate it from the surrounding structure, and it may still be factually correct while becoming much harder to interpret in the way the author intended.
I think that's why preserving hierarchy and relationships matters. It's not just additional metadata around the text; it helps explain how the different pieces are meant to be understood together.
That's really the distinction I was trying to make: making a document searchable isn't necessarily the same as preserving its meaning.
Yes, hierarchy is not decoration around the content; it is part of the meaning. A heading, section order, and nearby paragraph can change whether a sentence is a rule, an exception, an example, or a warning. Retrieval that flattens that away makes the model do archaeology later.
I really like that analogy.
"The model has to do archaeology" captures the problem better than I had been describing it.
Once the original structure is gone, the model is left inferring relationships that the author had already made explicit. Sometimes it succeeds, sometimes it doesn't—but either way it's reconstructing rather than reading.
I'd much rather preserve those relationships during ingestion than ask the model to rediscover them later. That feels like a more reliable use of the information the document already provides.
Exactly. Preserving hierarchy lowers the amount of archaeology the model has to perform. The more structure you strip away before retrieval, the more you ask the model to reconstruct intent from scattered fragments.
Exactly. Preserving structure is not about making the prompt prettier; it is about not destroying information the author already gave you. The model should spend its reasoning budget on the user question, not on reconstructing the document.
I like that way of putting it.
The document already contains a lot of work the author has done: grouping ideas, separating topics, showing what belongs together. If that survives ingestion, the model can focus on answering the question instead of reconstructing the document first.
That's probably the perspective I've gradually arrived at while thinking about retrieval. Structure isn't there to make prompts look cleaner—it's part of the information the author was communicating in the first place.
The less reconstruction the model has to do, the more of its reasoning can be spent on the actual problem.
Exactly. Structure is author intent, not decoration. When retrieval strips it away, the model has to spend context reconstructing relationships the document already gave us.
That is the part many retrieval pipelines flatten away. Structure is not decoration around the content; it is part of the content. If ingestion destroys the author's grouping and emphasis, the model has to spend reasoning budget rebuilding what the document already told us.
I think "author intent" is the key phrase there.
The document isn't just a container of facts—it's the author's attempt to organize those facts into an argument. If that organization survives ingestion, the model can spend its effort answering the user's question instead of reconstructing the author's thinking.
That's probably the perspective I've arrived at while writing this article. Preserving structure isn't about making retrieval more elegant—it's about avoiding work that has already been done.
The framing shift from "where to cut" to "what is the document trying to communicate" is the right one — chunk boundaries should respect the semantic unit, not a token budget. In practice this means the optimal chunker is document-type-aware: a legal clause, a markdown section, and a table row have different natural boundaries, and a fixed-size window imposed across all three will break meaning in all three differently. The other piece that gets lost with naive chunking is cross-sentence coreference: split a paragraph mid-thought and the retriever pulls a fragment whose subject is unresolvable without the prior sentence. Header breadcrumbs help but they address the symptom at retrieval time rather than the cause at chunking time, and you end up papering over a structural problem with metadata.
I like the distinction you're making.
Document type is definitely a useful signal, but I suspect the document's actual structure is even more informative. Two documents of the same type can still organize their ideas very differently.
Your point about coreference is also important. Once a paragraph is split mid-thought, retrieval may return a fragment that is technically relevant but no longer self-contained. The model then has to reconstruct context that originally existed in the document.
That's one of the reasons I've been questioning fixed-size chunking. My intuition is that the smallest retrieval unit should still preserve a coherent idea, even if that means its size varies from one part of the document to another.
strong framing, and the rate-limit example nails it. one thing i'd add from doing this in production: the reason people ship bags-of-chunks anyway usually isn't that they think it's correct, it's that structure extraction from arbitrary real documents is the actual hard part.
clean markdown, sure, respect the headings. but a real corpus is never all clean markdown, it's also scanned pdfs, exported confluence, half-broken html, and the moment it's heterogeneous "respect the structure" turns into "maintain a parser per document type." what held up for us was degrading gracefully: attach the heading path as metadata where you can parse it, fall back to overlap where you can't, instead of an all-or-nothing pipeline that breaks on the first ugly file.
That's a very fair point.
The article is really about what I think the retrieval unit should be, not about claiming that structure extraction is easy. In practice, heterogeneous corpora are exactly where things get difficult.
I also like your point about degrading gracefully. A pipeline that preserves structure when it can, but falls back to simpler strategies when it can't, is much more realistic than assuming every document can be parsed perfectly.
I don't think it has to be an all-or-nothing choice. Even recovering part of the document structure—headings, tables, section boundaries—can preserve information that would otherwise be lost. And when that information isn't available, it's better to acknowledge the limitation than pretend the document has more structure than we were able to recover.
Your point is a good reminder that preserving structure is one problem; reliably extracting it across messy real-world documents is another.
exactly, and i'd push your last line one step further: make "how much we recovered" a first-class field, not just a caveat you hold in your head. tag each chunk with how it was produced, clean structural parse versus fallback overlap, and carry that downstream.
then retrieval can weight it, and when the system hedges or abstains it can point at a real reason instead of a vibe. the limitation stops being something you apologize for and becomes a signal the rest of the pipeline can actually use. good piece, it's the right reframe.
That's a really interesting extension.
I hadn't been thinking about "how much structure was recovered" as something that should travel with the document, but I think you're right. Treating it as metadata rather than an implementation detail would let downstream stages reason about the quality of the retrieved evidence, not just its relevance.
I especially like your point that it gives the system a concrete reason to hedge or abstain. That's much more informative than treating uncertainty as a generic confidence score.
It also fits the broader theme of the article: preserving structure isn't just about improving retrieval—it also gives the system more context about what it may have lost along the way.
Thanks—that's a useful idea, and it definitely pushes the argument a bit further.
Your structural boundary rule is solid. I've hit the exact problem from the other direction -- merging adjacent document chunks because the embedding model thinks they're "semantically close." You get a chunk that reads coherently but has erased every structural decision the author made.
The versioning question keeps me up. Take a policy doc that gets revised every quarter. Same structure, same section headers, different meaning. If you're building that cross-document graph, do the edges get versioned? Or is each snapshot its own node with a "superseded_by" pointer?
I keep bouncing between the two. Versioned edges are cleaner in theory but a mess when you're doing retrieval across 12 revisions. Separate nodes mean you can query "give me the Q2 version" directly, but your graph doubles in size every quarter. What's your instinct on this? Does Retineo treat versions as first-class objects or is that out of scope for now?
That's a great question, and I don't think I have a confident answer yet.
My instinct is to treat versions as first-class objects rather than trying to mutate the graph underneath them. Even if most of the structure stays the same, the meaning of a section can change enough that I'd rather preserve each revision as its own document.
The trade-off, as you point out, is graph growth. But that feels like an engineering cost, whereas collapsing revisions risks losing historical context or mixing evidence from different points in time.
I haven't explored this deeply enough to say it's the right approach, though. Versioned documents are probably one of those cases where preserving structure isn't enough—you also have to preserve time.
Your "Rate Limiting Strategy" example is perfect â the fixed chunk splits destroy the state machine that ties sections together. But I think there's an even harder case: cross-document relationships.
When you have a spec that references other specs, or an API doc that imports a shared schema, no chunk of any individual document captures the relational structure. You're not just losing intra-document coherence â you're losing the graph between documents.
The same problem surfaces in agent memory. If episodic data is stored as independent chunks, temporal queries across sessions become essentially impossible. What if you need to retrieve "everything that happened between the robot's second inspection and the firmware update"? Flattened chunks don't encode that.
I've been building moteDB (a Rust embedded multimodal database) â the core insight for structured data is: keep the document as a tree in storage, with structural IDs, and build the chunk index as a separate layer. The chunk index is your retrieval surface; the document tree is your query surface.
How do you handle chunks that cross section boundaries? Do you let them span nodes, or do you let structural join keys carry the relationship instead?
That's a really interesting extension of the idea.
The article focuses on preserving the structure within a document, but you're pointing at the next level: preserving the relationships between documents. I think that's a different problem, and probably an equally important one.
For the question at the end, my intuition is to avoid creating chunks that span structural boundaries whenever possible. Once a retrieval unit crosses those boundaries, it becomes harder to tell what the original relationships actually were.
I'd rather preserve the structural units and let the relationships be explicit. That feels closer to how the author organized the information in the first place, and it gives the retrieval pipeline more options than if those relationships are baked into a larger chunk.
I haven't explored the cross-document graph nearly as much as the document structure itself, so I don't have a strong conclusion yet. But I do think your distinction between a retrieval surface and a query surface is a useful one. It captures something I was trying to get at in the article without quite having the language for it.
The section boundary problem is actually worse than it looks: once a retrieval query crosses section lines — say, a question about edge cases that needs both the algorithm section and the exceptions section — flat chunk retrieval can't reconstruct that relationship even if both chunks are retrieved, because there's no way to signal to the model that these two sections are from the same document and meant to be read together. One approach that sidesteps the scissors problem entirely is treating the document as a graph where sections are nodes and the heading hierarchy defines edges — retrieval then becomes a subgraph walk rather than a chunk lookup, and you naturally get surrounding context for any node without inflating chunk size. The fixed-chunk debate is ultimately a symptom of using a data structure (flat vector index) that can't represent the relationships that make a document coherent in the first place.
That's a really interesting way to frame it.
I agree that the fixed-chunk discussion may be a symptom of a deeper assumption—that we're representing a document as a flat collection of independent vectors rather than as a structured object.
The graph analogy captures something I've been thinking about as well: retrieval isn't always about finding a single location. Sometimes it's about preserving the relationships between pieces of evidence that were intentionally grouped by the author.
I'm still exploring what the right representation looks like in practice, so I wouldn't claim a document graph is necessarily the answer. But I do think we're missing something important if the retrieval unit has no notion of hierarchy or relationships beyond vector similarity.
I hadn't thought about describing it as a subgraph walk—that's a useful way to think about it.
"Structure is semantics" is the line I kept coming back to. Fixed-size chunking survives as a default mostly because it is operationally easy, but your examples show why it quietly destroys the argument the document was making long before retrieval has a chance to fail in a visible way. Keeping the code block with the reasoning above it, or the decision with its consequences, is often the difference between retrieving relevant text and retrieving usable context. Curious whether your next step is hierarchical retrieval, parent-child reconstruction, or some form of document-native segmentation that preserves section intent before embedding ever happens.
Thank you. I'm glad that line resonated—that's probably the central idea of the article.
As for the next step, I don't think of those as competing directions. My intuition is that preserving the document's own structure has to come first. Once that structure is gone, hierarchical retrieval or parent-child reconstruction are trying to recover something that was already discarded.
So the direction I'm exploring starts with document-native segmentation: letting the document define its own logical units before retrieval ever begins.
From there, hierarchical retrieval becomes much more interesting because you're navigating relationships that were already present in the document, rather than reconstructing them afterwards.
I don't know yet what the best representation is, but I increasingly think the retrieval unit should be defined by the document's structure rather than by a fixed token budget.
Imo (and I have a fair bit of experience in this, having written a custom OCR and SIFT system for my autonomous accounting suite), the most important part of a document, is the structure. If you understand the structure, you can vastly improve your contextual knowledge of the page. If you further go on to identify the fonts, bold/italics/strikethrough/overlined, then you can fit the font to get way more accurate OCR results. Then you run the model once to tag it, so in the future, it knows the structure and it can extract the data accurately every single time. From my experience, running my spline-based OCR alone got around 96% accuracy across 1k pages, whereas with the structural supports, it got 99.92%, to bridge the gap to 99.99%, using a cropped structural snippet with the structural context, allows a LLM to 'guess' the right answer pretty accurately, given it knows what it's looking at, what it's matching against and it's isolated to just what's necessary. Then I created the NDA format, so the document makes coherent sense and can be accurately reconstructed and formulaically validated when maths.
That's a really interesting perspective.
What I find interesting is that you arrived at the same conclusion from a completely different problem. My article is about retrieval, while you're talking about OCR and document understanding, but the underlying idea is the same: structure carries information.
Your point about doing the structural pass once also resonates with me. It seems much more effective to preserve that understanding early than to ask the model to reconstruct it every time the document is used.
I hadn't thought much about typography as part of that process, but your example makes a good case that visual structure can be just as informative as logical structure, depending on the task.
It's interesting how often "understand the structure first" turns out to be the common pattern, even when the end goal is completely different.
You cant butter bread properly, if you dont know the size and shape of the slice
I like that analogy.
Understanding the shape before acting on it is exactly the point. A document already has a structure. If we ignore that upfront, we're asking the model to rediscover it later.