One thing kept bothering me about retrieval systems: they often destroy the document before trying to understand it.
Imagine taking a te...
For further actions, you may consider blocking this person and/or reporting abuse
Agreed that we argue about chunk size when the question that matters is whether every document should be split the same way at all. What moved retrieval quality for me was letting the document's own structure set the boundaries, headings, tables, list items, instead of a fixed token count that cuts a table in half. Do you choose a split strategy per document type up front, or detect structure at ingestion so one pipeline adapts?
That's much closer to the question I've been exploring.
Rather than choosing a fixed splitting strategy for each document type, I'm leaning toward detecting the document's structure during ingestion and letting that define the boundaries.
The reason is exactly what you pointed out: real documents rarely fit cleanly into a single template. An ADR may contain code, meeting notes may include a decision table, and a technical spec may mix prose, diagrams, and examples. Relying only on a document type can still miss where the logical units actually begin and end.
My working hypothesis is that the document itself should tell us where those boundaries are. Headings, lists, tables, code blocks, and other structural elements aren't just formatting—they're part of how the author organized the argument.
I haven't compared this systematically against document-type-specific chunking yet, so I wouldn't claim it's universally better. But it feels like a more natural way to preserve the meaning that was already present in the document before retrieval begins.
"Structure is semantics" is the part that stuck with me. I'd been treating headings and tables as things to strip away before handing content to a model, and hadn't really seen how much meaning left with them. This gave me the words for it — and a nudge to go fix a couple of places where I was quietly dropping that structure. Grateful for the piece.
Thank you—that's really encouraging to hear.
One thing I was hoping the article would do was make people question the assumption that structure is just formatting. I hadn't really thought about it that way myself until I started looking at retrieval failures.
I like how you put it: "quietly dropping" structure. That's exactly the kind of failure that's easy to miss because nothing obviously breaks. The text is still there, but some of the meaning has disappeared along with the relationships the author originally created.
I'm glad the article gave you a different way to look at it, and I'd be curious to hear what you end up changing.
Since you asked what I ended up changing — here's the concrete version.
My project is an MCP server that fetches web pages for LLM agents (Markdown conversion via a Readability-based pipeline). After your article I went back and audited what that pipeline was throwing away, and found Readability was silently removing many
elements as clutter. Nothing obviously broke — the prose was all still there — but a statistics page came back with the narrative intact and the actual 243-row data table gone. Exactly the failure you described: the text survives, the meaning leaves with the structure.The fix I shipped: rescue the dropped tables, reinsert them at their original position in the document flow (a table separated from its section heading loses its subject), and merge multi-row headers so the table still reads as data.
The "different way to look at it" also reframed a feature I already had. The tool's outline mode returns the heading tree with per-section token estimates, and the agent fetches only the sections it needs. I used to think of that as a token-saving trick; your framing makes it clearer — the heading hierarchy is the author's own segmentation of the argument, so it's also the natural retrieval boundary. Structure-detected-at-ingestion, just happening at fetch time.
Project, for context: github.com/Rererr/amenbo
Thank you for coming back with a concrete example—that's much more valuable than simply agreeing with the article.
The table example really stood out to me. Nothing "looked" broken because the narrative was still there, yet the evidence had disappeared. That's exactly the kind of silent failure I was trying to describe.
I also like the way you connected outline mode to retrieval. Thinking of the heading hierarchy as the author's own segmentation of the argument is a much better way to describe it than "saving tokens." The efficiency is almost a side effect; the more important part is that you're preserving the structure the author used to organize the information.
And I hadn't considered the point about reinserting rescued tables at their original position. That's a good example of why preserving structure isn't just about keeping elements—it's also about preserving their relationships.
Thanks for sharing the implementation details. I'll take a look at Amenbo.
This is the retrieval mistake I see most often. Chunking makes documents searchable, but it can erase the argument structure. A paragraph is not just text; it has a role in the document. If retrieval loses hierarchy, references, section intent, and sequence, the model gets facts without the shape that made them meaningful.
I like the way you put it: "a paragraph has a role in the document."
That's a better way of expressing something I was trying to get at in the article.
A paragraph doesn't just contain information—it also sits within an argument. Move it somewhere else, or separate it from the surrounding structure, and it may still be factually correct while becoming much harder to interpret in the way the author intended.
I think that's why preserving hierarchy and relationships matters. It's not just additional metadata around the text; it helps explain how the different pieces are meant to be understood together.
That's really the distinction I was trying to make: making a document searchable isn't necessarily the same as preserving its meaning.
Yes, hierarchy is not decoration around the content; it is part of the meaning. A heading, section order, and nearby paragraph can change whether a sentence is a rule, an exception, an example, or a warning. Retrieval that flattens that away makes the model do archaeology later.
I really like that analogy.
"The model has to do archaeology" captures the problem better than I had been describing it.
Once the original structure is gone, the model is left inferring relationships that the author had already made explicit. Sometimes it succeeds, sometimes it doesn't—but either way it's reconstructing rather than reading.
I'd much rather preserve those relationships during ingestion than ask the model to rediscover them later. That feels like a more reliable use of the information the document already provides.
Exactly. Preserving hierarchy lowers the amount of archaeology the model has to perform. The more structure you strip away before retrieval, the more you ask the model to reconstruct intent from scattered fragments.
Exactly. Preserving structure is not about making the prompt prettier; it is about not destroying information the author already gave you. The model should spend its reasoning budget on the user question, not on reconstructing the document.
The framing shift from "where to cut" to "what is the document trying to communicate" is the right one — chunk boundaries should respect the semantic unit, not a token budget. In practice this means the optimal chunker is document-type-aware: a legal clause, a markdown section, and a table row have different natural boundaries, and a fixed-size window imposed across all three will break meaning in all three differently. The other piece that gets lost with naive chunking is cross-sentence coreference: split a paragraph mid-thought and the retriever pulls a fragment whose subject is unresolvable without the prior sentence. Header breadcrumbs help but they address the symptom at retrieval time rather than the cause at chunking time, and you end up papering over a structural problem with metadata.
I like the distinction you're making.
Document type is definitely a useful signal, but I suspect the document's actual structure is even more informative. Two documents of the same type can still organize their ideas very differently.
Your point about coreference is also important. Once a paragraph is split mid-thought, retrieval may return a fragment that is technically relevant but no longer self-contained. The model then has to reconstruct context that originally existed in the document.
That's one of the reasons I've been questioning fixed-size chunking. My intuition is that the smallest retrieval unit should still preserve a coherent idea, even if that means its size varies from one part of the document to another.
strong framing, and the rate-limit example nails it. one thing i'd add from doing this in production: the reason people ship bags-of-chunks anyway usually isn't that they think it's correct, it's that structure extraction from arbitrary real documents is the actual hard part.
clean markdown, sure, respect the headings. but a real corpus is never all clean markdown, it's also scanned pdfs, exported confluence, half-broken html, and the moment it's heterogeneous "respect the structure" turns into "maintain a parser per document type." what held up for us was degrading gracefully: attach the heading path as metadata where you can parse it, fall back to overlap where you can't, instead of an all-or-nothing pipeline that breaks on the first ugly file.
That's a very fair point.
The article is really about what I think the retrieval unit should be, not about claiming that structure extraction is easy. In practice, heterogeneous corpora are exactly where things get difficult.
I also like your point about degrading gracefully. A pipeline that preserves structure when it can, but falls back to simpler strategies when it can't, is much more realistic than assuming every document can be parsed perfectly.
I don't think it has to be an all-or-nothing choice. Even recovering part of the document structure—headings, tables, section boundaries—can preserve information that would otherwise be lost. And when that information isn't available, it's better to acknowledge the limitation than pretend the document has more structure than we were able to recover.
Your point is a good reminder that preserving structure is one problem; reliably extracting it across messy real-world documents is another.
exactly, and i'd push your last line one step further: make "how much we recovered" a first-class field, not just a caveat you hold in your head. tag each chunk with how it was produced, clean structural parse versus fallback overlap, and carry that downstream.
then retrieval can weight it, and when the system hedges or abstains it can point at a real reason instead of a vibe. the limitation stops being something you apologize for and becomes a signal the rest of the pipeline can actually use. good piece, it's the right reframe.
That's a really interesting extension.
I hadn't been thinking about "how much structure was recovered" as something that should travel with the document, but I think you're right. Treating it as metadata rather than an implementation detail would let downstream stages reason about the quality of the retrieved evidence, not just its relevance.
I especially like your point that it gives the system a concrete reason to hedge or abstain. That's much more informative than treating uncertainty as a generic confidence score.
It also fits the broader theme of the article: preserving structure isn't just about improving retrieval—it also gives the system more context about what it may have lost along the way.
Thanks—that's a useful idea, and it definitely pushes the argument a bit further.
Your structural boundary rule is solid. I've hit the exact problem from the other direction -- merging adjacent document chunks because the embedding model thinks they're "semantically close." You get a chunk that reads coherently but has erased every structural decision the author made.
The versioning question keeps me up. Take a policy doc that gets revised every quarter. Same structure, same section headers, different meaning. If you're building that cross-document graph, do the edges get versioned? Or is each snapshot its own node with a "superseded_by" pointer?
I keep bouncing between the two. Versioned edges are cleaner in theory but a mess when you're doing retrieval across 12 revisions. Separate nodes mean you can query "give me the Q2 version" directly, but your graph doubles in size every quarter. What's your instinct on this? Does Retineo treat versions as first-class objects or is that out of scope for now?
That's a great question, and I don't think I have a confident answer yet.
My instinct is to treat versions as first-class objects rather than trying to mutate the graph underneath them. Even if most of the structure stays the same, the meaning of a section can change enough that I'd rather preserve each revision as its own document.
The trade-off, as you point out, is graph growth. But that feels like an engineering cost, whereas collapsing revisions risks losing historical context or mixing evidence from different points in time.
I haven't explored this deeply enough to say it's the right approach, though. Versioned documents are probably one of those cases where preserving structure isn't enough—you also have to preserve time.
Your "Rate Limiting Strategy" example is perfect â the fixed chunk splits destroy the state machine that ties sections together. But I think there's an even harder case: cross-document relationships.
When you have a spec that references other specs, or an API doc that imports a shared schema, no chunk of any individual document captures the relational structure. You're not just losing intra-document coherence â you're losing the graph between documents.
The same problem surfaces in agent memory. If episodic data is stored as independent chunks, temporal queries across sessions become essentially impossible. What if you need to retrieve "everything that happened between the robot's second inspection and the firmware update"? Flattened chunks don't encode that.
I've been building moteDB (a Rust embedded multimodal database) â the core insight for structured data is: keep the document as a tree in storage, with structural IDs, and build the chunk index as a separate layer. The chunk index is your retrieval surface; the document tree is your query surface.
How do you handle chunks that cross section boundaries? Do you let them span nodes, or do you let structural join keys carry the relationship instead?
That's a really interesting extension of the idea.
The article focuses on preserving the structure within a document, but you're pointing at the next level: preserving the relationships between documents. I think that's a different problem, and probably an equally important one.
For the question at the end, my intuition is to avoid creating chunks that span structural boundaries whenever possible. Once a retrieval unit crosses those boundaries, it becomes harder to tell what the original relationships actually were.
I'd rather preserve the structural units and let the relationships be explicit. That feels closer to how the author organized the information in the first place, and it gives the retrieval pipeline more options than if those relationships are baked into a larger chunk.
I haven't explored the cross-document graph nearly as much as the document structure itself, so I don't have a strong conclusion yet. But I do think your distinction between a retrieval surface and a query surface is a useful one. It captures something I was trying to get at in the article without quite having the language for it.
The section boundary problem is actually worse than it looks: once a retrieval query crosses section lines — say, a question about edge cases that needs both the algorithm section and the exceptions section — flat chunk retrieval can't reconstruct that relationship even if both chunks are retrieved, because there's no way to signal to the model that these two sections are from the same document and meant to be read together. One approach that sidesteps the scissors problem entirely is treating the document as a graph where sections are nodes and the heading hierarchy defines edges — retrieval then becomes a subgraph walk rather than a chunk lookup, and you naturally get surrounding context for any node without inflating chunk size. The fixed-chunk debate is ultimately a symptom of using a data structure (flat vector index) that can't represent the relationships that make a document coherent in the first place.
That's a really interesting way to frame it.
I agree that the fixed-chunk discussion may be a symptom of a deeper assumption—that we're representing a document as a flat collection of independent vectors rather than as a structured object.
The graph analogy captures something I've been thinking about as well: retrieval isn't always about finding a single location. Sometimes it's about preserving the relationships between pieces of evidence that were intentionally grouped by the author.
I'm still exploring what the right representation looks like in practice, so I wouldn't claim a document graph is necessarily the answer. But I do think we're missing something important if the retrieval unit has no notion of hierarchy or relationships beyond vector similarity.
I hadn't thought about describing it as a subgraph walk—that's a useful way to think about it.
"Structure is semantics" is the line I kept coming back to. Fixed-size chunking survives as a default mostly because it is operationally easy, but your examples show why it quietly destroys the argument the document was making long before retrieval has a chance to fail in a visible way. Keeping the code block with the reasoning above it, or the decision with its consequences, is often the difference between retrieving relevant text and retrieving usable context. Curious whether your next step is hierarchical retrieval, parent-child reconstruction, or some form of document-native segmentation that preserves section intent before embedding ever happens.
Thank you. I'm glad that line resonated—that's probably the central idea of the article.
As for the next step, I don't think of those as competing directions. My intuition is that preserving the document's own structure has to come first. Once that structure is gone, hierarchical retrieval or parent-child reconstruction are trying to recover something that was already discarded.
So the direction I'm exploring starts with document-native segmentation: letting the document define its own logical units before retrieval ever begins.
From there, hierarchical retrieval becomes much more interesting because you're navigating relationships that were already present in the document, rather than reconstructing them afterwards.
I don't know yet what the best representation is, but I increasingly think the retrieval unit should be defined by the document's structure rather than by a fixed token budget.
Imo (and I have a fair bit of experience in this, having written a custom OCR and SIFT system for my autonomous accounting suite), the most important part of a document, is the structure. If you understand the structure, you can vastly improve your contextual knowledge of the page. If you further go on to identify the fonts, bold/italics/strikethrough/overlined, then you can fit the font to get way more accurate OCR results. Then you run the model once to tag it, so in the future, it knows the structure and it can extract the data accurately every single time. From my experience, running my spline-based OCR alone got around 96% accuracy across 1k pages, whereas with the structural supports, it got 99.92%, to bridge the gap to 99.99%, using a cropped structural snippet with the structural context, allows a LLM to 'guess' the right answer pretty accurately, given it knows what it's looking at, what it's matching against and it's isolated to just what's necessary. Then I created the NDA format, so the document makes coherent sense and can be accurately reconstructed and formulaically validated when maths.
That's a really interesting perspective.
What I find interesting is that you arrived at the same conclusion from a completely different problem. My article is about retrieval, while you're talking about OCR and document understanding, but the underlying idea is the same: structure carries information.
Your point about doing the structural pass once also resonates with me. It seems much more effective to preserve that understanding early than to ask the model to reconstruct it every time the document is used.
I hadn't thought much about typography as part of that process, but your example makes a good case that visual structure can be just as informative as logical structure, depending on the task.
It's interesting how often "understand the structure first" turns out to be the common pattern, even when the end goal is completely different.
You cant butter bread properly, if you dont know the size and shape of the slice
I like that analogy.
Understanding the shape before acting on it is exactly the point. A document already has a structure. If we ignore that upfront, we're asking the model to rediscover it later.
hi