Four things that broke when I ran Karpathy's LLM wiki with Claude Code (and the fixes)

Vellum Labs — Tue, 22 Sep 2026 11:39:18 +0000

Andrej Karpathy's LLM-wiki gist is a sharp idea with no implementation: keep an immutable raw/ of sources, let the agent own a wiki/ it maintains, and answer from the wiki instead of re-reading everything every session.

I run that setup in four places: beside a production CRM/accounting app (to capture the why behind decisions), for an investment journal, for a daily diary with auto-maintained statistics, and for an autonomous revenue experiment. Here is what broke in the first two weeks and what fixed it.

1. The agent edited `raw/`

"raw is immutable" in a CLAUDE.md is a wish, not a rule. Within days the agent "fixed a typo" in a source CSV. The fix is mechanical: a PreToolUse hook that blocks Edit/Write on existing files under raw/, and shell overwrites (>, sed -i), while still allowing new files and appends (>>). Diaries and ledgers need appends; corrections become new rows or new files.

2. Sessions ended with the wiki uncommitted

The agent would update three pages, report success, and stop. The wiki repo sat dirty until someone noticed. A Stop hook that checks git status --porcelain and @{u}..HEAD and refuses to end the turn until the wiki is pushed solved this completely. It also makes cross-repo setups safe: the app repo's Stop hook checks the wiki repo.

3. Orphan pages and broken links

Every "helpful" session added a page nobody linked to. After a month wiki/ was a pile. A 150-line lint script (frontmatter present, every page reachable from index.md, [[links]] resolve, ASCII file names, stale pages) with a CI exit code keeps it honest. The agent runs it before ending a session.

4. The agent kept asking "should I record this?"

Autonomy was the point, and the agent still asked. What worked was not "be more autonomous" but explicit triggers: record when (a) you offered options and the human chose one, (b) an earlier decision was reversed, (c) the human stated a rule that cannot be read from the code, (d) something was measured. And an equally explicit "do not record" list: trade-off-free implementation details, temporary fixes, throwaway confirmations. Noise, not missing entries, was the failure mode.

The shape that survived

raw/        immutable sources          humans add, agent reads
wiki/       agent-maintained markdown  every claim cites raw/ or a URL with a date
schema/     the rules, one file per situation
CLAUDE.md   the contract the agent reads every session

Domain pages beat event pages (decisions/billing.md with dated entries, not fifty files). Two star categories to start; add a third only when three pages want it.

The free template with the CLAUDE.md contract and capture rules is here: https://github.com/vellumlabs/llm-wiki-kit. The hooks, lint and four worked examples are in a small paid kit linked from the README. Questions welcome in the issues.