DEV Community

shimo4228
shimo4228

Posted on

I Deleted the LLM-Facing Architecture Docs I Had Committed 159 Times in 3 Months: Structure Goes to LSP, Reasons to ADRs, Diagrams to Humans

#claudecode #aiagents #documentation #architecture

It started with one diagram.

Architecture diagram of Contemplative Agent generated by Archify. One-directional imports from cli to adapters to core, external APIs, and evals / testing outside production, drawn as 9 nodes

On the morning of September 5, 2026, I tried Archify, a Claude Code skill (a packaged, reusable capability) that was making the rounds (it generates an architecture-diagram HTML from a JSON spec and validates the geometry), and drew one diagram of my own repository. 9 nodes, 3 cards. All 9 validation checks passed. It looked clean.

So I typed this: "This is great. Couldn't it replace the codemap?"

The codemap is docs/CODEMAPS/, a directory I have kept in the repository since March 2026. Hand-written architecture documents, written for the next session's LLM to read.

The answer was "no, it can't." A diagram is an outline; the codemap holds thresholds and reasons. That much I expected.

What I did not expect came two hours later. Instead of turning the codemap into a graph, I deleted all 6 files, 205,239 bytes, and deleted the machinery that generated them too. This article is about why the outcome was "delete" rather than "improve," and the questions you can use to make the same call on your own documents.

I replaced "should this become a graph" with "does this need to exist"

In the conversation right after drawing the diagram, I was thinking: if I described the codemap as a graph, couldn't I raise readability for both the human layer and the LLM layer? As a direction, it is natural. Apart from Archify, the family of tools that build a knowledge graph from code and hand it to an LLM keeps growing as of September 2026. Graphify, for example, shipped 8 releases between August 19 and September 5.

But I stopped myself from deciding on the spot. The same session had just finished praising Archify, and its judgment was leaning toward "build."

So I wrote a prompt that handed over only the facts and no conclusion, and asked the question again from zero in a new session. The opening line was this:

"Don't build" is an acceptable answer. Do not put the conclusion first. Verify the premises yourself before judging.

The reason for starting with the premises was simple: the numbers in my request might be stale.

Two premises were off

The request said "5 Markdown files, architecture.md is about 15,600 tokens." The first thing the new session did was measure that.

$ wc -c docs/CODEMAPS/*.md
  118891 docs/CODEMAPS/architecture.md
  ...
  205239 total
$ ls docs/CODEMAPS | wc -l
6
Enter fullscreen mode Exit fullscreen mode

Six files. adapters-moltbook.md had been left out.

architecture.md was 118,891 bytes, about 30k tokens. The "15,600" was an estimate written in the file's own header, and it was still the value from August 1, 2026. Never updated.

In other words, the header that existed to protect freshness was itself stale. At this point I cooled off a little.

I measured update frequency too.

$ git log --since=2026-06-01 --format=%h -- src | wc -l
197
$ git log --since=2026-06-01 --format=%h -- docs/CODEMAPS | wc -l
159
Enter fullscreen mode Exit fullscreen mode

In three months: 197 commits to source, 159 commits to the codemap. Nearly every time I touched source, I fixed the codemap once.

A hook, an automated check that fired after each change, prompted this sync, so I never felt any pain. And because there was no pain, the cost was invisible.

Nothing was blocked

When you re-ask a question from zero, the first question is: whose work is blocked right now, and on what?

The answer was "nobody's." Of the 68 path references in the codemap body, 2 did not exist, and both were tombstones the text itself described as "retired."

Not broken. No record of an LLM session reading the codemap and getting stuck.

The only candidate for real harm was bloat. Re-reading the Data Flow section of architecture.md, the dated parentheticals had become a changelog inlined into the body, and the re-scan paragraph in INDEX.md ran to 9,000 characters. I had been transcribing git log into prose.

Coming back to the graph idea: a graph does not solve bloat. Nodes and edges hold relationships; what had bloated was the prose about reasons. The question "should this become a graph" was placing a means where there was no problem to solve.

I split "LLM understanding" into structure and reasons

So what was the codemap for? "So the next session's LLM understands the repository." I split that "understanding" in two.

Structure. Which file calls which, which layer may import which layer.
Reasons. Why that guard exists, why the import constraint is written in that shape.

For structure, I actually ran Claude Code's LSP tool. The language server is pyright, which was already in the dev group of pyproject.toml; no extra configuration needed.

LSP incomingCalls  src/contemplative_agent/core/distill.py:104:5

Found 17 incoming calls:

src/contemplative_agent/cli/memory_cmds.py:
  _handle_distill (Function) - Line 39 [calls at: 64:18]

tests/benchmark_distill.py:
  run_benchmark (Function) - Line 166 [calls at: 203:9]

tests/test_distill.py:
  test_basic_distillation (Function) - Line 71 [calls at: 125:18]
  ...(14 more omitted)
Enter fullscreen mode Exit fullscreen mode

The 17 call sites of distill() come back with line numbers. The distill.py row in the codemap's core-modules.md, and the "who calls distill" prose in architecture.md, were exactly this answer, copied out by hand.

What I had spent 159 commits keeping up to date comes out of a single query, always current. Import direction is already enforced as a contract by import-linter.

Structure never needed to be stored. Derive it per question.

For reasons, I audited before deleting. I pulled every "why this guard exists" statement out of the Data Flow and Untrusted Boundary sections of architecture.md and grepped the ADRs (Architecture Decision Records: one file per design decision), the docstrings, and the tests.

Result: 23 of 25 items already lived somewhere else. Most were in their owning ADR, verbatim or at finer granularity. 70 source files cite ADR numbers directly, so the path from code to reason exists too.

The remaining 2 (a record of an instrument discontinuity, and the rationale for the 512-byte threshold in the watchdog script that monitors for missing output) I moved into an ADR and a script header.

The codemap was a mirror. Structure was a copy of the code; reasons were a copy of the ADRs. A copy needs updating every time the original moves, and that is what the 159 commits were.

The shrink option was the same trap at smaller scale

Even at this point, I had not chosen full deletion. My first pick was a shrink: keep only the Data Flow section and delete the rest.

The prose about reasons looked valuable.

I handed this option to a separate agent with no conversation context (its role is to judge build-or-not independently). The verdict came back as a rejection.

the Data Flow is the accretion (its dated brackets are the changelog inlined). Shrinking the file leaves the growth vector intact, and the hook will re-grow it within the month

The point was that the source of bloat was neither the header nor INDEX, but the Data Flow section itself. The dated brackets are the changelog inlined, and as long as the hook that makes me write them survives, shrinking it just grows back within a month.

I could not have seen this on my own. I was looking at "which section has value" and not at "which section grows." The shrink option kept the valuable section, and at the same time kept the growing one.

I went with delete.

What broke when I deleted it

The delete commit touched 40 files, +98 / −3,623 lines. Only 3 things broke mechanically:

  • 2 tests that asserted the codemap existed
  • Relative links from ADRs and the CHANGELOG

After fixing the links, the test suite came back 3,763 passed / 81 skipped. No functional regression.

The dangerous part was what did not break. The scan that reads documentation consistency held codemap freshness as two readings.

When the target directory disappears, those readings go empty without raising an error. Indistinguishable from "nothing wrong."

Code review caught this, and I changed it to emit FILE_MISSING when the one remaining freshness target is absent.

In a deletion, the thing to actually watch is not the test that fails but the instrument that silently goes empty.

I deleted the generating side too: the skill that writes codemaps, the script that checks their freshness, the hook that detects staleness and routes to regeneration. These lived not in one repository but in the Claude Code configuration shared across all my repositories (under ~/.claude/, which I will call the harness from here on).

The ADR I had adopted just 4 days earlier, "script the freshness gate," was superseded by a new ADR the same day and lost its force.

The independent-judgment agent recommended "watch one repository first, and remove from the whole only on a second matching conclusion." I removed the whole mechanism from the harness. As long as the generating side remains, other repositories keep being asked to regenerate, and new repositories grow one again.

The human-facing diagram stays, but it gets stamped with its origin

Back to Archify from the opening. If the codemap is gone, is the diagram unnecessary too?

It is necessary. The reader is different.

The codemap's reader was the next session's LLM. The LLM can now pull the symbol index itself, so the stored structure document is no longer needed.

A human opening the README, on the other hand, does not call LSP. Humans need an outline diagram. My policy is to spend the "write so a human can read it" effort only on the README and on output text, so that is where the Archify diagram goes.

With one condition. The diagram is an explanation derived from the source of truth, and it persists as a file. An explanation that vanishes inside a conversation (like an eli5) has no way to drift, but a 716KB HTML file squats in the repository and, six months later, confidently shows an old diagram even after a module has been removed.

Archify's architecture schema had a slot ready for exactly this.

{
  "meta": {
    "repository": { "url": "…", "revision": "<40-character commit SHA>" }
  },
  "components": [
    { "id": "core", "sources": [ { "path": "src/contemplative_agent/core/", "label": "…" } ] }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Stamp the commit at drawing time into meta.repository.revision, and the corresponding path into each node's sources. Then "which commit, and what in it, was this diagram drawn from" stays on the artifact side, and a machine can read how many commits it has drifted from HEAD. Even the camp that advocates stored graphs says "a stale graph is worse than no graph, attach a commit hash and provenance." The condition is the same.

The diagram at the top of this article does not fill in that slot. When I redraw it for the README, that is the first thing I fill in.

It pointed the same way as the official guidance

I looked this up only after finishing the draft: the Claude Code official docs say that since the July 9, 2026 version (v2.1.206), the /doctor command trims from checked-in CLAUDE.md "content Claude can derive from the code (directory layout, dependency lists, architecture overviews)" and keeps only pitfalls, reasons, and conventions. Auto memory does not store architecture or file paths either.

The contents of my codemap were precisely this trim target. I thought I was going against the grain; the accurate framing is that I had merely confirmed the official call by measuring it in my own repository.

The means of derivation, on the other hand, are not uniform across agents. As of September 4, 2026, Codex CLI has no built-in LSP, and even in Claude Code, language servers do not start in cloud sessions. Issues where the clangd and gopls plugins never register the LSP tool are also open.

"Delete what can be derived" only holds in an environment that can derive.

Running the call on your own documents

Few people keep a dedicated docs/CODEMAPS/ directory. More likely you have a single ARCHITECTURE.md, or a "Project structure" section inside CLAUDE.md. The questions are the same.

  1. Is the reader of this document a human or an LLM? Human-facing: keep it. LLM-facing: go to the next question
  2. Whose work is blocked right now, and on what? If nobody's, the "improvement" has no problem to solve
  3. For the structural part, can a tool answer each question on demand? Actually run LSP's incomingCalls and workspaceSymbol, import-linter, or grimp. If the environment cannot, the fix is installing a language server, not reviving the document
  4. For the reasons part, what fraction already has another owner (ADR, docstring, test)? Count it with grep. Mine was 23/25
  5. Does the shrink-and-keep option keep the sections that grow? A section full of dated parentheticals and phrases like "since ..." and "added ..." is a changelog inlined, and as long as the hook that makes you write it survives, it bloats again
  6. Can the diagram or explanation you keep be stamped with the commit it was drawn from? If the format cannot carry it, keep the lifetime of that explanation inside the conversation

Where this decision does not hold

  • Environments without LSP. Codex CLI, cloud sessions, languages whose language server plugin is not ready. With no derivation layer for structure, the grounds for deleting the document disappear. The fix is building the derivation layer, not reviving the document
  • Repositories where the reasons are in no ADR, docstring, or test. If the audit mostly says "nowhere else," that document is not a mirror but the source of truth. You need to build a destination before deleting
  • I changed a global setting on a reading from one repository. As the independent-judgment agent pointed out, this is a single measurement. When I delete the codemaps in the remaining 9 repositories, I will leave one line per commit message recording whether structural questions were answerable without the codemap. If a repository turns up where they were not, I will install a language server there rather than restore the document

Sources and references

Related links

Top comments (0)