DEV Community: ValeryKot

We Back Up Files. We Rarely Back Up Why They Exist.

ValeryKot — Tue, 28 Jul 2026 16:32:33 +0000

Everything survived. Except the explanation.

A while ago I was trying to understand why one of our services used Redis.

Finding the artifacts wasn't difficult.

There was a Jira ticket.

A pull request.

An architecture document.

A Slack discussion.

A commit.

Nothing had been deleted.

Everything was searchable.

And yet I still couldn't answer a simple question:

Why Redis?

Every file answered a different question

The Jira ticket answered:

What changed?

The pull request answered:

How was it implemented?

The architecture document answered:

Where does it fit?

The Slack thread answered:

Who was involved?

The commit answered:

When did it happen?

But none of them answered the question I actually had.

Why.

The answer wasn't missing

It just wasn't stored in one place.

Part of it lived in a security review.

Part in a design discussion.

Part in a Slack thread.

Part in the implementation.

Individually, every document made sense.

Collectively, they explained the decision.

The answer wasn't inside any single document.

It emerged from the relationships between documents.

That's when something started bothering me

We spend enormous effort preserving documents.

Git preserves commits.

Confluence preserves pages.

Jira preserves tickets.

Slack preserves messages.

Backups preserve all of them.

But none of these systems preserve the fact that one artifact exists because of another.

That relationship usually lives only in people's heads.

Once those people leave, every document survives.

The explanation doesn't.

I realized I had been asking the wrong question

I used to ask:

Did we document this?

Now I ask:

If someone finds this document a year from now, will they know what other documents they need to understand it?

Those are different questions.

The first is about documentation.

The second is about memory.

Documents aren't the whole memory

A document can survive forever.

That doesn't mean its meaning survives.

Meaning is often distributed across decisions, conversations, reviews, commits, and documentation.

I know how to preserve each of those individually.

I'm no longer convinced that's enough.

Documents preserve what happened.

Relationships preserve why it happened.

Next time: why I eventually stopped thinking of documents as the primary thing to index — and started thinking about decisions instead.

I Was Optimizing Ranking While the Real Problem Was Selection

ValeryKot — Fri, 24 Jul 2026 08:00:08 +0000

I mistook movement for improvement.

For three months, I changed our ranking algorithm every two weeks.

BM25.
Hybrid BM25 + vector search.
A cross-encoder reranker.
A different embedding model.
Yet another embedding model.

Every change moved a few documents up or down the list.

None of them made the answers better.

The Same Wrong Answer, Wearing a Different Tie

The user asked:

"Why does the gateway reject service X?"

With BM25, the top result was a glossary page defining gateway. Relevant by keyword. Useless by intent.

With hybrid retrieval, the glossary page dropped to third. A Jira ticket mentioning service X moved to second. It described the symptom, not the cause.

With a cross-encoder reranker, the Jira ticket became the top result. The glossary page fell to fourth. The architecture document that actually explained the exception remained at seventh.

With a different embedding model, that architecture document finally moved to fifth.

I remember thinking we'd finally made progress.

We hadn't.

Every new algorithm felt like progress because something moved.

The problem was that nothing important moved.

The Document That Mattered Never Became a Candidate

The architecture document existed.

I knew it existed.

I had even written part of it.

It never appeared in the top results.

Not because it was ranked poorly.

Because it was never selected.

The ranking algorithm could only sort the candidates it received.

The document that actually explained the answer never entered that candidate set.

No ranking algorithm can promote a document that was excluded before ranking even began.

The Moment It Clicked

I had five search results on my screen.

BM25 returned them in one order.

The cross-encoder returned them in another.

A different embedding model shuffled them again.

They were still the same documents.

Not one of them answered the question.

Only then did I ask myself:

Why am I ranking these documents at all?

The Question That Changed Everything

I stopped asking:

How do I rank these documents better?

I started asking:

Why are these documents candidates in the first place?

That question led me backward.

Not to another ranking algorithm.

But to every previous question I had been trying to answer.

Why didn't a larger context window help?

Why did chunking make retrieval worse?

Why did search keep finding documents but miss the explanation?

I realized those weren't separate problems.

They were different symptoms of the same one.

I had been optimizing the last step of retrieval while the earlier ones were quietly determining the outcome.

The Hierarchy of Retrieval

I now think about retrieval as three different layers.

Selection — What becomes a candidate at all?
Assembly — How is information preserved, structured, and connected?
Ranking — In what order should the candidates appear?

For months, I focused almost entirely on the third layer.

The first layer was already deciding whether the answer had any chance of being found.

Perfect ranking can only reorder what has already been selected.

It cannot retrieve what never became a candidate.

I kept polishing the order of documents that should never have been candidates.

I had mistaken ranking for retrieval.

In the next post, I'll look at a different question: why organizations rarely lose information—they lose the connections that once made that information meaningful.

Why Organizations Forget Even When Nothing Is Deleted

ValeryKot — Tue, 21 Jul 2026 22:34:58 +0000

The file survived. The explanation didn't.

A few months ago I needed to understand why one of our services was using Redis.

The engineer who made that decision had already left the company.

Finding the implementation wasn't difficult.

I found the Jira ticket.

Implement Redis cache.

I found the pull request.

The code was there.

I found the architecture document.

It mentioned caching.

I even found the Slack discussion where someone asked:

Why not Memcached?

The reply was one sentence.

Redis should be fine.

That was all.

The implementation survived.

The explanation didn't.

Nothing was missing

At first that felt strange.

Every artifact still existed.

Nothing had been deleted.

Git history was intact.

Slack history was searchable.

The documentation hadn't disappeared.

And yet the organization had clearly forgotten something.

Not what had been built.

Why it had been built that way.

We preserve artifacts better than reasoning

Most engineering organizations are very good at keeping records.

We keep tickets.

We keep code.

We keep documents.

We keep backups.

What we rarely preserve is the reasoning that connected them.

The ticket tells you what changed.

The pull request shows how it changed.

The architecture document describes what the system looks like.

None of them necessarily explains why one option was chosen over another.

That explanation often existed only during a conversation.

Context has a half-life

I've started thinking of context as something that slowly decays.

When a decision is made, everyone involved shares the same mental model.

The constraints are obvious.

The trade-offs are fresh.

Rejected alternatives are still remembered.

Six months later the code remains exactly the same.

The shared understanding doesn't.

A year later someone asks:

Why didn't we use Memcached?

Nobody knows anymore.

Not because the answer was lost.

Because it was never stored anywhere in a form that could survive the people who understood it.

This is how organizational memory disappears

It rarely happens all at once.

Nobody deletes a document.

Nobody wipes a database.

Instead, every small decision loses a little of its surrounding context.

A meeting ends.

A Slack thread sinks into history.

Someone changes teams.

Someone leaves.

Months later all the pieces still exist.

The explanation doesn't.

I stopped asking a different question

I used to ask:

Did we document this?

Now I ask:

Could someone explain this decision after everyone involved has left?

Those aren't the same question.

Documentation preserves artifacts.

Organizational memory preserves reasoning.

One can exist without the other.

Forgetting is usually invisible

Organizations don't usually notice they're forgetting.

Everything still looks healthy.

Repositories are there.

Documentation is there.

Search still works.

The problem only appears when someone asks a simple question:

Why?

And nobody can answer with confidence.

I've become less convinced that organizations lose knowledge because they delete information.

More often, they lose the relationships, assumptions, and reasoning that gave that information meaning.

The files survive.

The memory quietly fades.

Next time I'd like to explore something that surprised me while building retrieval systems: why the LLM often turned out to be the least interesting part of the pipeline.

The Difference Between Search and Discovery

ValeryKot — Thu, 16 Jul 2026 08:30:54 +0000

Sometimes the answer isn't missing. It's just scattered.

A few weeks ago I caught myself doing something I suspect every engineer has done.

I had five browser tabs open.

An ADR.

A Jira ticket.

A Slack thread.

A runbook.

A configuration file.

All of them were about the same problem.

None of them answered my question.

Only after reading all five did the picture become clear.

That was the moment I realized I wasn't searching anymore.

I was reconstructing.

Search wasn't failing

The question itself was simple.

Why does service X bypass the API gateway checks?

Search behaved exactly as it should.

It found the ADR describing the gateway.

It found the runbook explaining how to restart it.

It found a Jira issue mentioning service X.

It found a Slack discussion where someone wrote:

"We'll keep the old behavior for now."

Every result was relevant.

None of them explained why.

The explanation wasn't inside a document

Eventually the story emerged.

Months earlier, a security review introduced a new authentication header for internal services.

Most services were updated.

One legacy service wasn't.

A temporary exception was added so production wouldn't break.

Nobody ever documented that exception as a decision.

It survived as:

a commit message,
a Slack discussion,
one line of gateway configuration,
and a Jira ticket that treated the symptom instead of the reason.

Nothing was hidden.

Nothing was lost.

The explanation simply never existed in one place.

That's the difference

I've started thinking about search and discovery as two different activities.

Search retrieves artifacts.

Discovery reconstructs explanations.

Search answers questions like:

Where is the API documentation?

Discovery answers questions like:

Why does the system behave this way?

Those look similar.

In practice they're completely different.

Documents aren't the unit of knowledge

One assumption quietly sits underneath most retrieval systems.

The answer is expected to exist inside one document.

Similarity is computed.

Documents are ranked.

Top-k is returned.

This works remarkably well for factual lookup.

It works much less well for organizational memory.

Because organizations rarely write important decisions in one place.

A design review produces one document.

The implementation produces another.

Operations write a runbook.

Someone explains an exception in Slack.

Months later, all four are technically correct.

None tells the whole story.

Relationships carry meaning

What connects those documents is often more important than the documents themselves.

The Slack thread explains the motivation.

The ADR explains the architecture.

The commit explains the implementation.

The runbook explains the consequence.

Individually they're incomplete.

Together they form an explanation.

That's why I've become increasingly skeptical of measuring retrieval quality only by whether it returned the "right document."

Sometimes there isn't one.

Sometimes the answer is the relationship between several documents.

What changed for me

I stopped asking:

Which document contains the answer?

I started asking:

Which documents explain the answer together?

That small shift changed how I think about retrieval.

Not because search became less useful.

But because I realized search is often only the first step.

Discovery begins after the search results appear.

Organizations rarely lose documents.

More often, they lose the connections between them.

And once those connections disappear, every document can still be found—while the explanation quietly becomes impossible to retrieve.

Next time I'd like to explore another pattern I've seen repeatedly: why organizations can forget something even when nothing has been deleted.

Documents Aren't Bags of Chunks

ValeryKot — Sun, 12 Jul 2026 20:30:41 +0000

One thing kept bothering me about retrieval systems: they often destroy the document before trying to understand it.

Imagine taking a technical specification and cutting it with scissors every five hundred words.

No attention to headings.

No respect for sections.

No care for where a thought begins and ends.

Just cut.

Stack.

Index.

That sounds absurd.

Yet this is remarkably close to what many retrieval pipelines do before a document ever reaches a vector database.

We're asking the wrong question

One of the most common discussions around RAG sounds like this:

What's the optimal chunk size?

256 tokens?

512?

Should chunks overlap?

After watching enough retrieval failures, I started thinking this isn't the right question.

The real question isn't where to split a document.

It's whether every document should be split the same way at all.

Imagine this document

Suppose you have a technical specification called "Rate Limiting Strategy."

It contains four sections:

Why rate limiting exists
How the algorithm works
A code example
Exceptions and edge cases

Now index it using fixed-size chunks.

The heading lands in one chunk.

The code example ends up in another.

The explanation of edge cases becomes a third.

Each chunk receives its own embedding.

Each becomes an independent point in vector space.

The text survives.

The document, as a coherent unit of meaning, does not.

Retrieval works perfectly

Now someone asks:

"What happens when a user hits the rate limit?"

The retriever returns the chunk containing the code.

From its perspective, everything worked.

The identifiers match.

The embeddings are close.

The similarity score is high.

But something important has disappeared.

The heading explaining what this code belongs to is somewhere else.

The paragraph explaining why the algorithm behaves this way is somewhere else.

The exceptions are somewhere else.

The model receives valid code—but stripped of the reasoning that made the code meaningful.

It can explain what the function does.

It cannot explain why the system behaves that way.

Structure is semantics

It's easy to think of document structure as presentation.

I don't.

A heading tells the reader:

Everything below belongs to the same idea.

A section boundary says:

This thought ends here.

A list says:

These items should be understood together.

Those aren't visual decorations.

They're part of the information itself.

When those signals disappear during indexing, retrieval loses something that humans almost never notice consciously—but rely on constantly while reading.

Meaning doesn't have a fixed size

Eventually I stopped asking:

How should I split this document?

Instead I started asking:

What is the smallest unit that still makes sense on its own?

Sometimes that's a paragraph.

Sometimes it's an entire section.

Sometimes it's a code block together with the explanation above it.

Sometimes it's a diagram and its caption.

An architecture decision record often makes sense only when the decision, its rationale, and its consequences stay together.

An API reference is easier to understand when an endpoint remains connected to its parameters and examples.

Meeting notes become much more useful when decisions stay attached to owners and action items.

The right boundary isn't a character count.

It's a logical boundary.

Retrieval found something. Understanding failed.

One pattern kept repeating itself.

The retrieval engine successfully found text related to the question.

The model confidently explained that text.

And yet the actual answer was still missing.

Not because retrieval failed.

Because the document had already been dismantled before retrieval even began.

A document is an argument

I've gradually stopped thinking of documents as long strings of text.

They're closer to arguments.

Someone deliberately grouped ideas together.

Separated topics.

Added headings.

Built a flow of reasoning.

That structure carries meaning every bit as much as the words themselves.

If I break that structure into arbitrary pieces before indexing it, I shouldn't expect a retrieval engine—or an LLM—to reconstruct it perfectly later.

A document isn't a sequence of tokens.

It's an argument made visible.

Next time I want to explore a different question: why finding a document isn't the same thing as discovering knowledge.

Bigger Context Windows Didn't Make Our RAG Smarter

ValeryKot — Wed, 08 Jul 2026 10:23:41 +0000

We stopped measuring retrieval quality by how many tokens we could fit into the prompt.

When long-context models became available, many of us made the same assumption.

If an LLM can read 128K tokens, retrieval suddenly feels less important. Why spend time carefully selecting documents if the model can simply read everything?

It sounds reasonable.

In practice, it wasn't.

More context, worse answers

Imagine asking your internal assistant:

Why did we abandon microservices?

Retrieval returns thirty documents.

an architecture decision record
a few Jira tickets
Slack discussions
meeting notes
a glossary page

Everything is related.

Almost nothing answers the question.

The actual decision lives in a single ADR written months earlier. It explains the trade-offs: team size, latency, deployment complexity, operational cost.

But that document isn't especially similar to the query. It doesn't repeat the same vocabulary. It doesn't even mention "microservices" very often.

So it gets buried.

The model now receives thirty relevant documents and does what language models are very good at: it produces a coherent explanation.

The problem is that coherence is not the same thing as faithfulness.

Instead of recovering the original decision, it often synthesizes one from recurring themes across the retrieved documents.

The answer sounds plausible.

It just isn't the answer that was originally made.

Bigger windows don't fix retrieval

Research has already shown that models struggle with information buried inside very long contexts. The Lost in the Middle paper is probably the best-known example.

Our experience suggested something slightly different.

Sometimes the answer isn't lost because the context is long.

It's lost because the retrieval stage couldn't distinguish the document that contains the decision from documents that merely discuss the same topic.

Adding more context doesn't necessarily solve that problem.

Sometimes it simply gives the model more material to average together.

We were optimizing the wrong thing

For a while we treated retrieval as a packing exercise.

How many useful chunks can we fit into the prompt?

Over time the question changed.

Why is this document here?

Should it be here at all?

Does it explain the decision, or does it merely mention the same technology?

Those questions turned out to matter much more than the size of the context window.

Retrieval is a selection problem

The biggest shift wasn't moving from 8K to 128K tokens.

It was realizing that retrieval isn't about fitting more information into a prompt.

It's about selecting the few pieces of information that actually explain the answer.

Large context windows are incredibly useful.

They just don't compensate for weak retrieval.

If anything, they make weak retrieval look convincing.

Next time I'll look at another assumption I no longer believe: that documents should be treated as bags of chunks.

Why RAG Finds Documents but Still Misses the Answer

ValeryKot — Mon, 06 Jul 2026 10:22:20 +0000

After building and evaluating dozens of retrieval systems, I stopped thinking retrieval was the real problem.

Over the last couple of years I've spent a lot of time building and evaluating RAG systems.

Some were quick prototypes.

Some powered internal knowledge bases.

Some eventually became parts of products.

And I kept seeing the same pattern.

A user would ask a perfectly reasonable question.

The search would work.

The LLM would answer confidently.

And the answer would still be wrong.

Not because the model hallucinated.

Not because embeddings were bad.

Not because vector search failed.

The system had simply retrieved the wrong kind of information.

A question every RAG system should answer

Imagine a company with a thousand engineering documents.

Architecture decisions.

Meeting notes.

RFCs.

Runbooks.

ADRs.

Internal documentation.

Now someone asks:

Why did we choose Kafka over RabbitMQ?

A typical RAG pipeline looks like this:

Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Top-K Retrieval
↓
LLM

Everything seems reasonable.

The question becomes an embedding.

The nearest chunks are retrieved.

The chunks are added to the prompt.

The LLM generates an answer.

Simple.

Except this is often where things start going wrong.

Semantic similarity isn't the same as relevance

Suppose your knowledge base contains fifteen documents about Kafka.

Architecture guides.

Broker configuration.

Monitoring.

Deployment.

Developer documentation.

Vector search will happily retrieve them.

And technically, it's correct.

They are semantically close to the question.

But none of them explain why Kafka was chosen.

The actual answer might live inside a two-page Architecture Decision Record written eighteen months earlier.

That document mostly discusses RabbitMQ, latency, throughput, operational complexity, and load testing.

The word Kafka barely appears.

From the perspective of embeddings, it isn't particularly similar.

From the perspective of an engineer, it's exactly the document you needed.

That's the disconnect.

Search found similar documents.

It didn't find the decision.

Then we start optimizing

Once teams notice retrieval isn't great, the usual optimization cycle begins.

First we reduce Top-K.

Instead of retrieving 100 chunks, we retrieve 10.

Then we shrink chunk sizes.

500 tokens become 250.

250 become 150.

Then we add rerankers.

Cross-encoders.

Hybrid retrieval.

Bigger context windows.

I tried all of these.

Most of them help.

None of them completely solve the problem.

Because they're still optimizing retrieval inside the same mental model.

The document slowly disappears

One thing surprised me more than anything else.

The smaller our chunks became, the less context survived.

Eventually the system remembered paragraphs.

But forgot the document.

Headings disappeared.

Structure disappeared.

Arguments disappeared.

What remained was a collection of isolated text fragments.

I started calling this chunk amnesia.

It's not a scientific term.

Just a name for something I kept observing.

Humans don't search that way

This is where my thinking changed.

When I need information, I don't start by looking for a paragraph.

I usually ask myself:

Which document was this?

Then:

What decision did we make?

Only after that do I look for implementation details.

It's almost hierarchical.

Document
↓
Decision
↓
Evidence
↓
Implementation

Most RAG systems reverse that process.

They search for paragraphs first and hope context reconstructs itself.

Sometimes it does.

Often it doesn't.

Bigger context windows don't fix this

A common reaction is:

"Let's just send more context."

I don't think that's the answer either.

A 64K context window can hold far more text.

It doesn't magically restore structure.

The model still receives disconnected fragments.

More fragments don't automatically become a coherent narrative.

Large prompts reduce one bottleneck.

They don't eliminate the retrieval problem.

What changed my thinking

Eventually I stopped thinking of retrieval as a single search problem.

Instead, I started thinking about navigation across multiple levels of abstraction.

Before asking:

Which paragraph matters?

I first wanted to answer:

Which document matters?

Then:

Which decisions inside that document matter?

Only then should retrieval dive into individual chunks.

That's also how people naturally explore documentation.

We rarely jump directly into paragraph number seventeen.

We skim.

We orient ourselves.

Then we zoom in.

This isn't just an LLM problem

After a while I realized this discussion isn't really about language models.

It's about organizational memory.

Companies rarely lose information because documents disappear.

They lose the relationships between documents.

The ADR exists.

The meeting notes exist.

The design proposal exists.

The Slack discussion exists.

But the connections slowly fade.

Search retrieves text.

People are trying to recover context.

Those aren't always the same problem.

I'm still exploring this

I don't have a universal solution.

In fact, this realization is one of the reasons I started building Retineo.

Not because I thought I had the answer.

Because I couldn't find a system that treated organizational knowledge as something richer than independent chunks inside a vector database.

Maybe the answer isn't hierarchical retrieval.

Maybe it's something else entirely.

I'm still experimenting.

I'd love to hear your experience

If you've built RAG systems at scale, what's the failure mode you've seen most often?

For me, it isn't hallucination.

It's this:

The documents aren't missing.
The relationships between them are.

Discussion

Have you encountered similar problems?

Did hierarchical retrieval, knowledge graphs, rerankers, or another approach help?

I'm genuinely interested in how others are tackling this.