DEV Community

ValeryKot
ValeryKot

Posted on

Documentation Doesn't Accumulate Knowledge

It accumulates evidence.


I used to believe something that now feels obvious.

If we documented enough, understanding would take care of itself.

Write the ADR.

Keep the runbook up to date.

Record the incident.

Update the architecture diagram.

Future engineers would have everything they needed.

I don't think that's true anymore.


The day I read everything

I was trying to understand a service I hadn't worked on before.

The documentation looked excellent.

There was an ADR explaining the original design.

A migration guide from the last major release.

An incident review.

An API reference.

A runbook.

An architecture diagram.

Everything was current.

Everything was internally consistent.

I spent almost two hours reading.

When I closed the last page, I realized something uncomfortable.

I knew what had happened.

I still didn't understand the system.


Every document was telling the truth

The ADR explained why the original architecture was chosen.

The migration guide explained why it changed.

The incident review explained why it failed.

The runbook explained how to recover it.

None of those documents contradicted each other.

They simply weren't trying to answer the same question.

Each one captured a moment.

I was looking for a story.


That's when I noticed the difference

Documentation preserves observations.

Understanding comes from connecting observations.

The documents never said:

"This incident changed the team's priorities."

"That migration only happened because of the outage three months earlier."

"This design decision stopped making sense after customer X arrived."

Nobody wrote those sentences because nobody needed to.

At the time, everyone already knew them.

Months later, those invisible connections turned out to be the only thing I actually needed.


Evidence isn't explanation

A detective doesn't solve a case because there are enough fingerprints.

The fingerprints matter.

But only after someone connects them.

Documentation works the same way.

Each page is evidence that something happened.

Knowledge appears only when someone reconstructs the relationships between those pieces of evidence.

The documents weren't incomplete.

My understanding was.


What changed for me

I stopped asking:

Do we have documentation?

Now I ask:

Could someone who wasn't here reconstruct the story?

Those are different questions.

The first measures documentation.

The second measures understanding.

I've become much less convinced they're the same thing.


Documentation preserves evidence.

Knowledge is reconstructed.

The more systems I study, the more I think confusing those two ideas is the reason so many organizations feel well documented—and still spend hours asking each other "Why does this work like this?"


Next: why I stopped treating duplicate documents as clutter—and started treating them as evidence that an organization disagrees with itself.

Top comments (2)

Collapse
 
komo profile image
Reid Marlow

This matches the failure mode I see with runbooks. They preserve the decision, but not the pressure around it. The useful artifact is often a small trail of rejected options and constraints, because that is what lets the next person tell whether the old answer is stale or just unfashionable.

Collapse
 
hannune profile image
Tae Kim

The documentation-as-evidence vs knowledge-as-reconstruction gap maps to exactly why RAG over documentation tends to fail for onboarding tasks: it retrieves the ADR, the incident review, and the runbook as separate chunks, but the relationship between them ("the incident changed priorities, which is why the next ADR reversed the earlier decision") exists between documents, not inside any of them. Retrieval precision can be high while answer fidelity is low — every chunk is accurate, but the reasoning chain the user actually needs was only ever implicit in the collective history. One partial fix is explicit relation extraction at index time: rather than chunking documents independently, build a graph where the incident review links to the ADR it caused the reversal of, and retrieval follows those typed edges rather than treating each document as an isolated embedding. The diagnostic question you've named — not "is everything documented" but "can someone reconstruct why" — is also the right evaluation criterion for a RAG corpus: can a retriever return not just the facts but the causal chain that connects them?