DEV Community: ValeryKot

Does RAG Need Better Retrieval — or Better Relationships?

ValeryKot — Thu, 17 Sep 2026 19:17:28 +0000

I started this experiment for a slightly different reason.

I was playing with a system that could reconstruct context across documents.

Not summarize documents. Not retrieve the most similar chunks.

Reconstruct context.

For example, suppose I have two documents:

In 2022, the team chose PostgreSQL.

And somewhere else:

After the migration problems, the team moved to DynamoDB in 2024.

Neither document contains the answer to:

Why did the database decision change?

The answer lives in the relationship between them.

That got me thinking about something else.

If a system can reliably recover these relationships, could it also use them to create structured training data?

Before going there, I wanted to know whether the relationships actually mattered.

So I built a small experiment.

And the first result was much stronger than I expected.

I wanted to remove retrieval from the equation

The obvious problem with testing relationships in RAG is retrieval.

If the experiment gets better answers after adding a graph, what actually improved?

Maybe the graph found documents that retrieval missed.

Maybe it simply retrieved more useful evidence.

Maybe the model got more context.

Maybe the graph structure itself helped.

Those are very different explanations.

I wanted to isolate one of them.

So I made the retrieved documents identical in both conditions.

No re-ranking.

No additional documents.

No better retrieval.

The model gets the same documents.

The only difference is whether it also receives explicit relationships between those documents.

That became the experiment:

Does explicitly stated relational context improve reasoning when retrieval is held constant?

The dataset

I built a synthetic corpus of 254 business documents.

The documents cover things like billing, DevOps and HR.

Then I created 60 multi-hop cases with manually defined ground truth:

42 supersession cases
6 contradiction cases
4 negative cases
8 link cases

Some cases require reasoning across several documents. Six of the supersession cases involve four to six documents with unrelated distractors.

I deliberately made some of the relationships hard to recover from individual documents.

For example, two documents can both be perfectly valid while representing different states of the same decision.

The important information isn't inside either document.

It's in:

A → superseded by → B

or:

A ↔ contradicts ↔ B

or:

A → belongs to → cluster B

There is also a meaningful negative case:

A → no known relation → B

That one turned out to matter more than I expected.

The baseline was deliberately weak

I used a lexical BM25-style top-5 retrieval over the full corpus.

It isn't a production BM25 implementation. It's a deliberately simple lexical baseline.

Mean recall of the oracle documents in the top five was 0.67.

27 of the 60 cases had full recall.

17 had partial recall.

10 had low recall.

Six had zero recall, all of them negative cases.

This is a real limitation.

If retrieval misses evidence, a relational hint can potentially compensate for that weakness.

So I don't want to pretend this experiment answers what happens with a very strong production retrieval system.

But it gives us something useful:

C and D see byte-identical retrieved documents.

That part is controlled.

Then I added the relationships

The baseline looks roughly like this:

retrieved documents
        ↓
       LLM
        ↓
      answer

The experimental condition adds a small structured block:

retrieved documents
        +
explicit relationships
        ↓
       LLM
        ↓
      answer

For example:

kind: superseded
from: database-choice-2022.md
to: database-choice-2024.md
reason: migration failure

The model was told that the knowledge-base engine had detected the relation and that it should use the hint if it was consistent with the documents.

Nothing else changed.

The result

I ran the main experiment with two models.

Model	No relation	Typed relations
DeepSeek	77%	94%
GPT-4o-mini	77%	98%

The gain was:

+17 percentage points for DeepSeek.

+21 points for GPT-4o-mini.

That was enough to make me stop and ask a more annoying question:

What exactly did I measure?

Because “adding a graph improves RAG” would be a very easy conclusion.

And probably a premature one.

Maybe the model just likes extra tokens

The relation hints were only around 17 words.

Perhaps the model simply benefited from getting more information.

So I added a control with approximately the same amount of unrelated text.

Parking.

Cafeteria.

Meeting rooms.

No relation labels.

No document names.

No similarity scores.

That condition scored 53%.

The baseline was 77%.

The relational condition was 98%.

So “more tokens” doesn't explain the result.

At least not by itself.

Maybe structure is the trick

The relational information was presented as fields:

kind:
from:
to:
reason:

Maybe that formatting alone makes the model perform better.

So I took the same semantic information and put it into ordinary prose.

Something like:

Note: B reflects the current state of this decision. A is the previous version.

Same information.

No structured relation fields.

That scored 53%.

The structured version scored 98%.

This result bothered me.

Because now I had learned something I hadn't originally set out to test:

how the relationship is presented may matter almost as much as the relationship itself.

And I had a new confound.

Maybe the model doesn't actually reason over the relationship.

Maybe it just trusts the block because I told it that a “knowledge base engine” generated it.

I'll come back to that.

What if the relations are wrong?

This was the test I cared about most.

If relationships really carry useful information, incorrect relationships should matter.

I corrupted the identifying fields of the relation hints.

At 20% corruption, accuracy dropped from 98% to 53%.

At 40%, it was 52%.

That is a dramatic failure.

But there is an important detail here.

I did not test every corruption level.

I don't know what happens at 5%.

I don't know what happens at 10%.

I don't know whether another corruption mechanism would behave differently.

So I can't say:

“80% correct relations are useless.”

The experiment doesn't show that.

It shows something narrower:

Under this particular corruption mechanism, introducing 20% incorrect relations almost completely destroyed the observed benefit.

That distinction matters.

Then I tried similarity instead of typed relationships

This was probably the most interesting part of the experiment.

If relationships help because they connect documents, perhaps we don't need explicit relation types.

Maybe a similarity graph is enough.

So I created another condition where the model received document-to-document similarity edges instead of typed relations.

The result improved substantially:

Model	No hint	Similarity	Typed relation
DeepSeek	76%	88%	96%
GPT-4o-mini	76%	93%	98%

Similarity clearly helped.

But it didn't behave like typed relations.

There was a particularly interesting case.

For DeepSeek, similarity actually beat typed relations on the link cases:

79% vs 71%.

The likely explanation is still only a hypothesis. The typed hint contained a reason field that may have distracted the model.

The negative cases were different.

Similarity hurt them.

The model saw related-looking documents and tended to assume that a relationship existed.

The typed relation could explicitly say:

kind: none

That distinction matters.

Similarity says:

These things look related.

A typed relation can say:

These things are related in this specific way.

Or:

These things are not related.

Those are different pieces of information.

Then I tried to break the whole thing

I added three more controls:

Shuffled relations

Give each case a relation from another case of the same kind.

Token-matched text

Give the model unrelated text with approximately the same length as the relation hint.

In-document prose

Put the same semantic relation into ordinary document text.

All three landed around 52–53%.

The baseline was 77%.

The structured, correct relation condition was 98%.

The shuffled condition also passed an anti-leakage check: none of the 60 shuffled hints happened to be correct.

At this point I was reasonably confident that the result wasn't just:

“Give the LLM some extra words that look vaguely useful.”

But I still wasn't confident about why the structured relation worked.

The uncomfortable explanation

There is one explanation my controls don't eliminate.

Maybe the model isn't using the relational structure as relational structure.

Maybe it sees:

Retineo relation: kind=superseded ...

and interprets that as privileged metadata.

In other words:

“The system says this is the answer, so I'll trust it.”

That would produce a large improvement without requiring the model to reconstruct anything from the relationship itself.

The in-document prose control actually makes this possibility more interesting.

The two conditions contain the same semantic information.

Yet the structured version was dramatically more effective.

So I can't currently separate:

relational reasoning

from

authority/salience effects caused by the representation.

That's the next control I want to run.

There is another problem: retrieval

My retrieval baseline is weak.

That's not an incidental detail.

Mean top-5 recall was only 0.67, so there is plenty of room for a relational hint to compensate for retrieval failures.

Imagine that the model needs documents A, B and C.

The retriever returns A and B.

Then I give it a relation saying:

A → related_to → C

I haven't really solved the retrieval problem.

I've given the model a shortcut around it.

So the next experiment is becoming pretty obvious.

I want to vary retrieval depth while keeping the relational condition intact.

Something like:

shallow retrieval
       ×
no relation / relation

then:

deeper retrieval
       ×
no relation / relation

The question is not whether deeper retrieval improves the baseline.

Of course it can.

The interesting question is what happens to the gap between the two conditions.

If:

+21 pp
  ↓
+10 pp
  ↓
 +2 pp

then the relational hint may mostly be compensating for weak retrieval.

If the gap remains large as retrieval quality improves, the story gets more interesting.

But this experiment needs some care. Simply dumping 100 documents into the context changes the amount of text the model has to process, so that would introduce another variable.

I want to separate more available evidence from more text in the prompt.

What I think the experiment says so far

Here is the version I'm comfortable with.

On this synthetic corpus, with this retrieval setup and these two models:

Correct typed cross-document relations improved accuracy when the retrieved documents were identical.

The effect survived controls for extra tokens, shuffled relations, and expressing the same semantic information as ordinary document prose.

Similarity structure also helped, but typed relations provided additional information in several cases, particularly where relation type or explicit absence mattered.

That's the observation.

The explanation is still open.

I don't know yet how much of the effect comes from:

information about the relationship itself;
making that information easier for the model to parse;
the model treating structured metadata as authoritative;
compensation for weak retrieval.

And I definitely don't know whether the effect survives real organizational data.

The corpus is synthetic.

The retrieval is simplified.

There are only two models.

Those are real boundaries on the result, not footnotes to hide at the end.

This started as a data-labeling idea

There was a practical reason I ran the experiment in the first place.

I was thinking about whether a context-reconstruction system could help create structured datasets.

Take a collection of documents.

Find candidate relationships.

Classify them:

supersedes
contradicts
belongs_to
references
none

Then use those relationships as structured training data.

That sounds useful.

But it also creates a nasty circular problem.

If I use the system to generate relational labels, and then use those labels to demonstrate that relational context helps models, I may just be measuring my own assumptions.

So I stopped.

Before thinking about dataset generation, I wanted to know whether the representation itself had a measurable effect.

That led to this experiment.

And now I have a better question than the one I started with.

The next experiment

I don't want to build a bigger graph yet.

I want to try to make this result disappear.

Better retrieval.

More evidence.

Interactive retrieval.

Different models.

Different relational representations.

And especially an experiment that removes the “trusted metadata” explanation.

If better retrieval makes the relational advantage disappear, that's useful.

If it doesn't, that's useful too.

Either result tells me where to look next.

The interesting question is no longer:

Does GraphRAG work?

That's too broad to be useful.

The question I'm interested in is much smaller:

When an LLM already has the relevant documents, does explicitly representing the relationships between those documents give it information it cannot reliably reconstruct from the documents themselves?

I don't have the final answer yet.

That's exactly why I'm running the next experiment.

Documentation Doesn't Accumulate Knowledge

ValeryKot — Sat, 01 Aug 2026 18:39:49 +0000

It accumulates evidence.

I used to believe something that now feels obvious.

If we documented enough, understanding would take care of itself.

Write the ADR.

Keep the runbook up to date.

Record the incident.

Update the architecture diagram.

Future engineers would have everything they needed.

I don't think that's true anymore.

The day I read everything

I was trying to understand a service I hadn't worked on before.

The documentation looked excellent.

There was an ADR explaining the original design.

A migration guide from the last major release.

An incident review.

An API reference.

A runbook.

An architecture diagram.

Everything was current.

Everything was internally consistent.

I spent almost two hours reading.

When I closed the last page, I realized something uncomfortable.

I knew what had happened.

I still didn't understand the system.

Every document was telling the truth

The ADR explained why the original architecture was chosen.

The migration guide explained why it changed.

The incident review explained why it failed.

The runbook explained how to recover it.

None of those documents contradicted each other.

They simply weren't trying to answer the same question.

Each one captured a moment.

I was looking for a story.

That's when I noticed the difference

Documentation preserves observations.

Understanding comes from connecting observations.

The documents never said:

"This incident changed the team's priorities."

"That migration only happened because of the outage three months earlier."

"This design decision stopped making sense after customer X arrived."

Nobody wrote those sentences because nobody needed to.

At the time, everyone already knew them.

Months later, those invisible connections turned out to be the only thing I actually needed.

Evidence isn't explanation

A detective doesn't solve a case because there are enough fingerprints.

The fingerprints matter.

But only after someone connects them.

Documentation works the same way.

Each page is evidence that something happened.

Knowledge appears only when someone reconstructs the relationships between those pieces of evidence.

The documents weren't incomplete.

My understanding was.

What changed for me

I stopped asking:

Do we have documentation?

Now I ask:

Could someone who wasn't here reconstruct the story?

Those are different questions.

The first measures documentation.

The second measures understanding.

I've become much less convinced they're the same thing.

Documentation preserves evidence.

Knowledge is reconstructed.

The more systems I study, the more I think confusing those two ideas is the reason so many organizations feel well documented—and still spend hours asking each other "Why does this work like this?"

Next: why I stopped treating duplicate documents as clutter—and started treating them as evidence that an organization disagrees with itself.

We Back Up Files. We Rarely Back Up Why They Exist.

ValeryKot — Tue, 28 Jul 2026 16:32:33 +0000

Everything survived. Except the explanation.

A while ago I was trying to understand why one of our services used Redis.

Finding the artifacts wasn't difficult.

There was a Jira ticket.

A pull request.

An architecture document.

A Slack discussion.

A commit.

Nothing had been deleted.

Everything was searchable.

And yet I still couldn't answer a simple question:

Why Redis?

Every file answered a different question

The Jira ticket answered:

What changed?

The pull request answered:

How was it implemented?

The architecture document answered:

Where does it fit?

The Slack thread answered:

Who was involved?

The commit answered:

When did it happen?

But none of them answered the question I actually had.

Why.

The answer wasn't missing

It just wasn't stored in one place.

Part of it lived in a security review.

Part in a design discussion.

Part in a Slack thread.

Part in the implementation.

Individually, every document made sense.

Collectively, they explained the decision.

The answer wasn't inside any single document.

It emerged from the relationships between documents.

That's when something started bothering me

We spend enormous effort preserving documents.

Git preserves commits.

Confluence preserves pages.

Jira preserves tickets.

Slack preserves messages.

Backups preserve all of them.

But none of these systems preserve the fact that one artifact exists because of another.

That relationship usually lives only in people's heads.

Once those people leave, every document survives.

The explanation doesn't.

I realized I had been asking the wrong question

I used to ask:

Did we document this?

Now I ask:

If someone finds this document a year from now, will they know what other documents they need to understand it?

Those are different questions.

The first is about documentation.

The second is about memory.

Documents aren't the whole memory

A document can survive forever.

That doesn't mean its meaning survives.

Meaning is often distributed across decisions, conversations, reviews, commits, and documentation.

I know how to preserve each of those individually.

I'm no longer convinced that's enough.

Documents preserve what happened.

Relationships preserve why it happened.

Next time: why I eventually stopped thinking of documents as the primary thing to index — and started thinking about decisions instead.

I Was Optimizing Ranking While the Real Problem Was Selection

ValeryKot — Fri, 24 Jul 2026 08:00:08 +0000

I mistook movement for improvement.

For three months, I changed our ranking algorithm every two weeks.

BM25.
Hybrid BM25 + vector search.
A cross-encoder reranker.
A different embedding model.
Yet another embedding model.

Every change moved a few documents up or down the list.

None of them made the answers better.

The Same Wrong Answer, Wearing a Different Tie

The user asked:

"Why does the gateway reject service X?"

With BM25, the top result was a glossary page defining gateway. Relevant by keyword. Useless by intent.

With hybrid retrieval, the glossary page dropped to third. A Jira ticket mentioning service X moved to second. It described the symptom, not the cause.

With a cross-encoder reranker, the Jira ticket became the top result. The glossary page fell to fourth. The architecture document that actually explained the exception remained at seventh.

With a different embedding model, that architecture document finally moved to fifth.

I remember thinking we'd finally made progress.

We hadn't.

Every new algorithm felt like progress because something moved.

The problem was that nothing important moved.

The Document That Mattered Never Became a Candidate

The architecture document existed.

I knew it existed.

I had even written part of it.

It never appeared in the top results.

Not because it was ranked poorly.

Because it was never selected.

The ranking algorithm could only sort the candidates it received.

The document that actually explained the answer never entered that candidate set.

No ranking algorithm can promote a document that was excluded before ranking even began.

The Moment It Clicked

I had five search results on my screen.

BM25 returned them in one order.

The cross-encoder returned them in another.

A different embedding model shuffled them again.

They were still the same documents.

Not one of them answered the question.

Only then did I ask myself:

Why am I ranking these documents at all?

The Question That Changed Everything

I stopped asking:

How do I rank these documents better?

I started asking:

Why are these documents candidates in the first place?

That question led me backward.

Not to another ranking algorithm.

But to every previous question I had been trying to answer.

Why didn't a larger context window help?

Why did chunking make retrieval worse?

Why did search keep finding documents but miss the explanation?

I realized those weren't separate problems.

They were different symptoms of the same one.

I had been optimizing the last step of retrieval while the earlier ones were quietly determining the outcome.

The Hierarchy of Retrieval

I now think about retrieval as three different layers.

Selection — What becomes a candidate at all?
Assembly — How is information preserved, structured, and connected?
Ranking — In what order should the candidates appear?

For months, I focused almost entirely on the third layer.

The first layer was already deciding whether the answer had any chance of being found.

Perfect ranking can only reorder what has already been selected.

It cannot retrieve what never became a candidate.

I kept polishing the order of documents that should never have been candidates.

I had mistaken ranking for retrieval.

In the next post, I'll look at a different question: why organizations rarely lose information—they lose the connections that once made that information meaningful.

Why Organizations Forget Even When Nothing Is Deleted

ValeryKot — Tue, 21 Jul 2026 22:34:58 +0000

The file survived. The explanation didn't.

A few months ago I needed to understand why one of our services was using Redis.

The engineer who made that decision had already left the company.

Finding the implementation wasn't difficult.

I found the Jira ticket.

Implement Redis cache.

I found the pull request.

The code was there.

I found the architecture document.

It mentioned caching.

I even found the Slack discussion where someone asked:

Why not Memcached?

The reply was one sentence.

Redis should be fine.

That was all.

The implementation survived.

The explanation didn't.

Nothing was missing

At first that felt strange.

Every artifact still existed.

Nothing had been deleted.

Git history was intact.

Slack history was searchable.

The documentation hadn't disappeared.

And yet the organization had clearly forgotten something.

Not what had been built.

Why it had been built that way.

We preserve artifacts better than reasoning

Most engineering organizations are very good at keeping records.

We keep tickets.

We keep code.

We keep documents.

We keep backups.

What we rarely preserve is the reasoning that connected them.

The ticket tells you what changed.

The pull request shows how it changed.

The architecture document describes what the system looks like.

None of them necessarily explains why one option was chosen over another.

That explanation often existed only during a conversation.

Context has a half-life

I've started thinking of context as something that slowly decays.

When a decision is made, everyone involved shares the same mental model.

The constraints are obvious.

The trade-offs are fresh.

Rejected alternatives are still remembered.

Six months later the code remains exactly the same.

The shared understanding doesn't.

A year later someone asks:

Why didn't we use Memcached?

Nobody knows anymore.

Not because the answer was lost.

Because it was never stored anywhere in a form that could survive the people who understood it.

This is how organizational memory disappears

It rarely happens all at once.

Nobody deletes a document.

Nobody wipes a database.

Instead, every small decision loses a little of its surrounding context.

A meeting ends.

A Slack thread sinks into history.

Someone changes teams.

Someone leaves.

Months later all the pieces still exist.

The explanation doesn't.

I stopped asking a different question

I used to ask:

Did we document this?

Now I ask:

Could someone explain this decision after everyone involved has left?

Those aren't the same question.

Documentation preserves artifacts.

Organizational memory preserves reasoning.

One can exist without the other.

Forgetting is usually invisible

Organizations don't usually notice they're forgetting.

Everything still looks healthy.

Repositories are there.

Documentation is there.

Search still works.

The problem only appears when someone asks a simple question:

Why?

And nobody can answer with confidence.

I've become less convinced that organizations lose knowledge because they delete information.

More often, they lose the relationships, assumptions, and reasoning that gave that information meaning.

The files survive.

The memory quietly fades.

Next time I'd like to explore something that surprised me while building retrieval systems: why the LLM often turned out to be the least interesting part of the pipeline.

The Difference Between Search and Discovery

ValeryKot — Thu, 16 Jul 2026 08:30:54 +0000

Sometimes the answer isn't missing. It's just scattered.

A few weeks ago I caught myself doing something I suspect every engineer has done.

I had five browser tabs open.

An ADR.

A Jira ticket.

A Slack thread.

A runbook.

A configuration file.

All of them were about the same problem.

None of them answered my question.

Only after reading all five did the picture become clear.

That was the moment I realized I wasn't searching anymore.

I was reconstructing.

Search wasn't failing

The question itself was simple.

Why does service X bypass the API gateway checks?

Search behaved exactly as it should.

It found the ADR describing the gateway.

It found the runbook explaining how to restart it.

It found a Jira issue mentioning service X.

It found a Slack discussion where someone wrote:

"We'll keep the old behavior for now."

Every result was relevant.

None of them explained why.

The explanation wasn't inside a document

Eventually the story emerged.

Months earlier, a security review introduced a new authentication header for internal services.

Most services were updated.

One legacy service wasn't.

A temporary exception was added so production wouldn't break.

Nobody ever documented that exception as a decision.

It survived as:

a commit message,
a Slack discussion,
one line of gateway configuration,
and a Jira ticket that treated the symptom instead of the reason.

Nothing was hidden.

Nothing was lost.

The explanation simply never existed in one place.

That's the difference

I've started thinking about search and discovery as two different activities.

Search retrieves artifacts.

Discovery reconstructs explanations.

Search answers questions like:

Where is the API documentation?

Discovery answers questions like:

Why does the system behave this way?

Those look similar.

In practice they're completely different.

Documents aren't the unit of knowledge

One assumption quietly sits underneath most retrieval systems.

The answer is expected to exist inside one document.

Similarity is computed.

Documents are ranked.

Top-k is returned.

This works remarkably well for factual lookup.

It works much less well for organizational memory.

Because organizations rarely write important decisions in one place.

A design review produces one document.

The implementation produces another.

Operations write a runbook.

Someone explains an exception in Slack.

Months later, all four are technically correct.

None tells the whole story.

Relationships carry meaning

What connects those documents is often more important than the documents themselves.

The Slack thread explains the motivation.

The ADR explains the architecture.

The commit explains the implementation.

The runbook explains the consequence.

Individually they're incomplete.

Together they form an explanation.

That's why I've become increasingly skeptical of measuring retrieval quality only by whether it returned the "right document."

Sometimes there isn't one.

Sometimes the answer is the relationship between several documents.

What changed for me

I stopped asking:

Which document contains the answer?

I started asking:

Which documents explain the answer together?

That small shift changed how I think about retrieval.

Not because search became less useful.

But because I realized search is often only the first step.

Discovery begins after the search results appear.

Organizations rarely lose documents.

More often, they lose the connections between them.

And once those connections disappear, every document can still be found—while the explanation quietly becomes impossible to retrieve.

Next time I'd like to explore another pattern I've seen repeatedly: why organizations can forget something even when nothing has been deleted.

Documents Aren't Bags of Chunks

ValeryKot — Sun, 12 Jul 2026 20:30:41 +0000

One thing kept bothering me about retrieval systems: they often destroy the document before trying to understand it.

Imagine taking a technical specification and cutting it with scissors every five hundred words.

No attention to headings.

No respect for sections.

No care for where a thought begins and ends.

Just cut.

Stack.

Index.

That sounds absurd.

Yet this is remarkably close to what many retrieval pipelines do before a document ever reaches a vector database.

We're asking the wrong question

One of the most common discussions around RAG sounds like this:

What's the optimal chunk size?

256 tokens?

512?

Should chunks overlap?

After watching enough retrieval failures, I started thinking this isn't the right question.

The real question isn't where to split a document.

It's whether every document should be split the same way at all.

Imagine this document

Suppose you have a technical specification called "Rate Limiting Strategy."

It contains four sections:

Why rate limiting exists
How the algorithm works
A code example
Exceptions and edge cases

Now index it using fixed-size chunks.

The heading lands in one chunk.

The code example ends up in another.

The explanation of edge cases becomes a third.

Each chunk receives its own embedding.

Each becomes an independent point in vector space.

The text survives.

The document, as a coherent unit of meaning, does not.

Retrieval works perfectly

Now someone asks:

"What happens when a user hits the rate limit?"

The retriever returns the chunk containing the code.

From its perspective, everything worked.

The identifiers match.

The embeddings are close.

The similarity score is high.

But something important has disappeared.

The heading explaining what this code belongs to is somewhere else.

The paragraph explaining why the algorithm behaves this way is somewhere else.

The exceptions are somewhere else.

The model receives valid code—but stripped of the reasoning that made the code meaningful.

It can explain what the function does.

It cannot explain why the system behaves that way.

Structure is semantics

It's easy to think of document structure as presentation.

I don't.

A heading tells the reader:

Everything below belongs to the same idea.

A section boundary says:

This thought ends here.

A list says:

These items should be understood together.

Those aren't visual decorations.

They're part of the information itself.

When those signals disappear during indexing, retrieval loses something that humans almost never notice consciously—but rely on constantly while reading.

Meaning doesn't have a fixed size

Eventually I stopped asking:

How should I split this document?

Instead I started asking:

What is the smallest unit that still makes sense on its own?

Sometimes that's a paragraph.

Sometimes it's an entire section.

Sometimes it's a code block together with the explanation above it.

Sometimes it's a diagram and its caption.

An architecture decision record often makes sense only when the decision, its rationale, and its consequences stay together.

An API reference is easier to understand when an endpoint remains connected to its parameters and examples.

Meeting notes become much more useful when decisions stay attached to owners and action items.

The right boundary isn't a character count.

It's a logical boundary.

Retrieval found something. Understanding failed.

One pattern kept repeating itself.

The retrieval engine successfully found text related to the question.

The model confidently explained that text.

And yet the actual answer was still missing.

Not because retrieval failed.

Because the document had already been dismantled before retrieval even began.

A document is an argument

I've gradually stopped thinking of documents as long strings of text.

They're closer to arguments.

Someone deliberately grouped ideas together.

Separated topics.

Added headings.

Built a flow of reasoning.

That structure carries meaning every bit as much as the words themselves.

If I break that structure into arbitrary pieces before indexing it, I shouldn't expect a retrieval engine—or an LLM—to reconstruct it perfectly later.

A document isn't a sequence of tokens.

It's an argument made visible.

Next time I want to explore a different question: why finding a document isn't the same thing as discovering knowledge.

Bigger Context Windows Didn't Make Our RAG Smarter

ValeryKot — Wed, 08 Jul 2026 10:23:41 +0000

We stopped measuring retrieval quality by how many tokens we could fit into the prompt.

When long-context models became available, many of us made the same assumption.

If an LLM can read 128K tokens, retrieval suddenly feels less important. Why spend time carefully selecting documents if the model can simply read everything?

It sounds reasonable.

In practice, it wasn't.

More context, worse answers

Imagine asking your internal assistant:

Why did we abandon microservices?

Retrieval returns thirty documents.

an architecture decision record
a few Jira tickets
Slack discussions
meeting notes
a glossary page

Everything is related.

Almost nothing answers the question.

The actual decision lives in a single ADR written months earlier. It explains the trade-offs: team size, latency, deployment complexity, operational cost.

But that document isn't especially similar to the query. It doesn't repeat the same vocabulary. It doesn't even mention "microservices" very often.

So it gets buried.

The model now receives thirty relevant documents and does what language models are very good at: it produces a coherent explanation.

The problem is that coherence is not the same thing as faithfulness.

Instead of recovering the original decision, it often synthesizes one from recurring themes across the retrieved documents.

The answer sounds plausible.

It just isn't the answer that was originally made.

Bigger windows don't fix retrieval

Research has already shown that models struggle with information buried inside very long contexts. The Lost in the Middle paper is probably the best-known example.

Our experience suggested something slightly different.

Sometimes the answer isn't lost because the context is long.

It's lost because the retrieval stage couldn't distinguish the document that contains the decision from documents that merely discuss the same topic.

Adding more context doesn't necessarily solve that problem.

Sometimes it simply gives the model more material to average together.

We were optimizing the wrong thing

For a while we treated retrieval as a packing exercise.

How many useful chunks can we fit into the prompt?

Over time the question changed.

Why is this document here?

Should it be here at all?

Does it explain the decision, or does it merely mention the same technology?

Those questions turned out to matter much more than the size of the context window.

Retrieval is a selection problem

The biggest shift wasn't moving from 8K to 128K tokens.

It was realizing that retrieval isn't about fitting more information into a prompt.

It's about selecting the few pieces of information that actually explain the answer.

Large context windows are incredibly useful.

They just don't compensate for weak retrieval.

If anything, they make weak retrieval look convincing.

Next time I'll look at another assumption I no longer believe: that documents should be treated as bags of chunks.

Why RAG Finds Documents but Still Misses the Answer

ValeryKot — Mon, 06 Jul 2026 10:22:20 +0000

After building and evaluating dozens of retrieval systems, I stopped thinking retrieval was the real problem.

Over the last couple of years I've spent a lot of time building and evaluating RAG systems.

Some were quick prototypes.

Some powered internal knowledge bases.

Some eventually became parts of products.

And I kept seeing the same pattern.

A user would ask a perfectly reasonable question.

The search would work.

The LLM would answer confidently.

And the answer would still be wrong.

Not because the model hallucinated.

Not because embeddings were bad.

Not because vector search failed.

The system had simply retrieved the wrong kind of information.

A question every RAG system should answer

Imagine a company with a thousand engineering documents.

Architecture decisions.

Meeting notes.

RFCs.

Runbooks.

ADRs.

Internal documentation.

Now someone asks:

Why did we choose Kafka over RabbitMQ?

A typical RAG pipeline looks like this:

Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Top-K Retrieval
↓
LLM

Everything seems reasonable.

The question becomes an embedding.

The nearest chunks are retrieved.

The chunks are added to the prompt.

The LLM generates an answer.

Simple.

Except this is often where things start going wrong.

Semantic similarity isn't the same as relevance

Suppose your knowledge base contains fifteen documents about Kafka.

Architecture guides.

Broker configuration.

Monitoring.

Deployment.

Developer documentation.

Vector search will happily retrieve them.

And technically, it's correct.

They are semantically close to the question.

But none of them explain why Kafka was chosen.

The actual answer might live inside a two-page Architecture Decision Record written eighteen months earlier.

That document mostly discusses RabbitMQ, latency, throughput, operational complexity, and load testing.

The word Kafka barely appears.

From the perspective of embeddings, it isn't particularly similar.

From the perspective of an engineer, it's exactly the document you needed.

That's the disconnect.

Search found similar documents.

It didn't find the decision.

Then we start optimizing

Once teams notice retrieval isn't great, the usual optimization cycle begins.

First we reduce Top-K.

Instead of retrieving 100 chunks, we retrieve 10.

Then we shrink chunk sizes.

500 tokens become 250.

250 become 150.

Then we add rerankers.

Cross-encoders.

Hybrid retrieval.

Bigger context windows.

I tried all of these.

Most of them help.

None of them completely solve the problem.

Because they're still optimizing retrieval inside the same mental model.

The document slowly disappears

One thing surprised me more than anything else.

The smaller our chunks became, the less context survived.

Eventually the system remembered paragraphs.

But forgot the document.

Headings disappeared.

Structure disappeared.

Arguments disappeared.

What remained was a collection of isolated text fragments.

I started calling this chunk amnesia.

It's not a scientific term.

Just a name for something I kept observing.

Humans don't search that way

This is where my thinking changed.

When I need information, I don't start by looking for a paragraph.

I usually ask myself:

Which document was this?

Then:

What decision did we make?

Only after that do I look for implementation details.

It's almost hierarchical.

Document
↓
Decision
↓
Evidence
↓
Implementation

Most RAG systems reverse that process.

They search for paragraphs first and hope context reconstructs itself.

Sometimes it does.

Often it doesn't.

Bigger context windows don't fix this

A common reaction is:

"Let's just send more context."

I don't think that's the answer either.

A 64K context window can hold far more text.

It doesn't magically restore structure.

The model still receives disconnected fragments.

More fragments don't automatically become a coherent narrative.

Large prompts reduce one bottleneck.

They don't eliminate the retrieval problem.

What changed my thinking

Eventually I stopped thinking of retrieval as a single search problem.

Instead, I started thinking about navigation across multiple levels of abstraction.

Before asking:

Which paragraph matters?

I first wanted to answer:

Which document matters?

Then:

Which decisions inside that document matter?

Only then should retrieval dive into individual chunks.

That's also how people naturally explore documentation.

We rarely jump directly into paragraph number seventeen.

We skim.

We orient ourselves.

Then we zoom in.

This isn't just an LLM problem

After a while I realized this discussion isn't really about language models.

It's about organizational memory.

Companies rarely lose information because documents disappear.

They lose the relationships between documents.

The ADR exists.

The meeting notes exist.

The design proposal exists.

The Slack discussion exists.

But the connections slowly fade.

Search retrieves text.

People are trying to recover context.

Those aren't always the same problem.

I'm still exploring this

I don't have a universal solution.

In fact, this realization is one of the reasons I started building Retineo.

Not because I thought I had the answer.

Because I couldn't find a system that treated organizational knowledge as something richer than independent chunks inside a vector database.

Maybe the answer isn't hierarchical retrieval.

Maybe it's something else entirely.

I'm still experimenting.

I'd love to hear your experience

If you've built RAG systems at scale, what's the failure mode you've seen most often?

For me, it isn't hallucination.

It's this:

The documents aren't missing.
The relationships between them are.

Discussion

Have you encountered similar problems?

Did hierarchical retrieval, knowledge graphs, rerankers, or another approach help?

I'm genuinely interested in how others are tackling this.