DEV Community

Cover image for I built Karpathy's LLM Wiki as a product — the hard part was stopping the LLM from writing
frank chu
frank chu

Posted on

I built Karpathy's LLM Wiki as a product — the hard part was stopping the LLM from writing

#showdev #llm #tools #ai

Every knowledge tool I tried failed in one of two directions.

Notion, Obsidian, Roam: passive containers. They store beautifully and maintain nothing — you are the janitor, forever. NotebookLM and the RAG chatbots fail the other way: nothing accumulates. Ask a question and the answer is re-derived from raw chunks on the spot. Ask the same question next month and you pay the same derivation cost and get a slightly different answer.

Andrej Karpathy's LLM Wiki pattern flips this. Instead of retrieving at query time, the LLM maintains a persistent markdown wiki: it ingests your sources once, pre-bakes the cross-references, pre-flags the contradictions, and writes the synthesis down. Questions get answered from the wiki, not re-derived from the pile.

I turned that pattern into a product: MindBase — open source (MIT, 135 stars), plain markdown on your own disk, free local models by default. This post is not really about the wiki, though. It's about the part nobody puts in the demo: a wiki an LLM writes into is only worth having if you can trust what it wrote. So most of the engineering went into stopping the LLM from writing.

What MindBase is, in 30 seconds

Three layers on disk, exactly as Karpathy specified:

~/mindbase-data/projects/my-research/
├── sources/contributors/<you>/   # yours — append-only, the AI never rewrites it
├── sources/research/             # AI-authored wiki pages, every one approved
├── context.md                    # the synthesized wiki — LLM writes, you approve
├── index.yaml                    # auto-generated catalog
├── logs/                         # every operation, append-only
└── state/ · artifacts/           # derived — snapshots, lint findings, rebuildable
Enter fullscreen mode Exit fullscreen mode

Three operations run the loop:

  • ingest — the AI reads a source, discusses takeaways with you, proposes a plan; only what you approve gets written.
  • build — regenerate context.md from everything not yet folded in.
  • lint — the AI audits its own wiki for contradictions, stale claims, and orphans.

It runs three ways: a web UI (npx mindbase-app, Node 20+, localhost:4321), a Claude Code plugin with sub-agents and slash commands, and a 50-tool MCP server for Cursor / Windsurf / Cline / Continue.

Now the trust problem, and the four mechanisms that solve it.

Rule 1: the model proposes JSON. It never writes prose to disk.

Every server-side operation runs through one engine (apps/server/src/ops/runner.ts). Its header comment is the whole architecture:

gather → one constrained LLM completion → (checkpoint) → validate →
apply via executors → append to logs/<date>.md
Enter fullscreen mode Exit fullscreen mode

The model's entire output is a single JSON plan, validated against a zod schema. The plan's action vocabulary, in full, is four verbs:

// apps/server/src/ops/types.ts
export const actionSchema = z.discriminatedUnion('kind', [
  z.object({ kind: z.literal('create_research_page'), slug, markdown: z.string().min(1).max(100_000) }),
  z.object({ kind: z.literal('update_context'),       markdown: z.string().min(1).max(100_000) }),
  z.object({ kind: z.literal('append_context_section'), section: z.string().min(1).max(80), markdown: z.string().min(1).max(20_000) }),
  z.object({ kind: z.literal('add_wikilinks'), path: linkTargetPath, links: z.array(slug).min(1).max(10) }),
]);
Enter fullscreen mode Exit fullscreen mode

That's it. Create a research page, replace context.md, append a section, add wikilinks (which may only point at research pages or context.md). Max 20 actions per plan. And notice what's not in the union — the comment in the file says it plainly:

Deliberately contains NO action that can write under sources/contributors or sources/raw — the append-only layer is unreachable by schema.

A prompt that says "please never modify the user's sources" is a suggestion. A missing enum variant is a wall. The model can hallucinate a fifth action kind if it wants; the schema parse fails, nothing executes, and the op errors out. The executors that apply validated plans then re-check every path against the project root anyway — belt and braces, because the input to all of this is, at the end of the day, text a language model produced.

Rule 2: nothing applies without a human checkbox

Ingest doesn't write and then tell you. The completion produces takeaways plus the plan, the UI renders the plan as checkboxes, and the apply step takes the plan id and the list of indices you actually checked:

// apps/server/src/ops/runner.ts
export async function applyContributePlan(planId: string, selected: number[], ...) {
  const actions = pending.actions.filter((_, i) => selected.includes(i));
  ...
}
Enter fullscreen mode Exit fullscreen mode

Unchecked actions simply never run. Plans live in memory and expire after 10 minutes, so a stale approval card can't fire tomorrow. Pending plans also reserve the slugs they intend to create, so two contributes running at once can't both propose the same new page and collide at write time.

Rule 3: the builder agent has no hands

The Claude Code plugin ships 5 sub-agents, each with a strict tool allowlist. Here's the one that rebuilds context.md (apps/plugin/agents/builder.md):

name: builder
description: Sub-agent dispatched by /mb:build to synthesize context.md from
  sources/. Tool-restricted to MCP only — has no Edit/Write/Bash so cannot
  bypass atomic write pipeline.
tools: Read, mcp__mb__mindbase_validate_structure, mcp__mb__mindbase_gather_sources,
  mcp__mb__mindbase_atomic_write_context, mcp__mb__mindbase_rebuild_index,
  mcp__mb__mindbase_append_log, mcp__mb__mindbase_load_project, mcp__mb__mindbase_search_wiki
Enter fullscreen mode Exit fullscreen mode

The agent that writes the most important file in the system has no file-write tool. No Edit, no Write, no Bash. Its only path to disk is mindbase_atomic_write_context, and that tool is paranoid on its behalf:

  1. Snapshot the current context.md to state/builder/snapshots/<timestamp>.md — every build is rollback-able.
  2. Write to a temp file, then rename() — atomic, no torn writes.
  3. Enforce a 400-line cap on context.md. Overflow isn't silently truncated and doesn't balloon the file either; it spills into sources/research/context-overflow-<stamp>.md with a pointer left behind.
  4. Append the log entry.

The other agents follow the same pattern: the lint curator is read-only (Read plus the lint tools), and the researcher can read the whole web but can only persist through an append-only research_save call. A prompt injection hiding in a source can make the model want to write anywhere it likes. The allowlist decides where it can.

Rule 4: lint findings must quote — and the quotes get checked

My favorite mechanism. When lint claims two pages contradict each other, the finding must include evidence: a verbatim quote from each page, 8–300 characters, copied exactly. Then the runner verifies it:

// apps/server/src/ops/runner.ts
return { page: e.page, quote: e.quote, verified: haystack !== null && haystack.includes(normalizeQuote(e.quote)) };
Enter fullscreen mode Exit fullscreen mode

It re-reads the page from disk and checks the quote is actually in there (modulo case/whitespace/markdown normalization). If the model invented the sentence — and models do — the finding comes back verified: false instead of landing in your inbox as fact. An auditor that can fabricate its own evidence is worse than no auditor, so the auditor doesn't get to.

Why one completion instead of an agent loop

This is the design decision people push back on, so here's the reasoning, straight from apps/server/src/ops/llm.ts:

One constrained JSON completion per call — the ops engine never runs multi-step tool loops (unreliable on local models).

MindBase is free and local by default, which means the default brain might be a 3B model on an 8 GB laptop. A 3B model cannot be trusted with a 12-step tool loop, but it can fill in one JSON document. So: one shot, temperature 0.2, schema-checked, and if the JSON doesn't parse, exactly one retry with the parse error appended ("Respond again with ONLY the corrected JSON"). Boring on purpose. The wiki is a long-lived artifact; every write into it should be as close to deterministic as I can make it.

The hardware wizard installs the biggest Ollama model your RAM can hold:

RAM Model Role
8 GB llama3.2:3b runs anywhere, keeps up
24 GB+ qwen3:14b interactive work, ~30s a completion
32 GB+ Apple Silicon Muse Glimmer 30B background lint/build — slower, but its contradiction findings quote the exact conflicting sentences

Cloud keys are supported, but they're the option, not the default.

I counted the tools, because of course I did

The README says the MCP server has 49 tools. While writing this I counted the register calls in apps/mcp/src/tools/index.ts and the definitions they push: 50. Somewhere along the way a fiftieth tool snuck in and the README didn't get the memo. (Half the tools are named mindbase_* and half aren't, which tells you exactly how organically this grew. It's on my list.)

I ran the tests

Fresh clone, pnpm install, build @mindbase/core first (the server tests resolve the built package — the repo's own CLAUDE.md warns about this and I ignored it exactly once), then the ops-engine suite:

 FAIL  src/ops/runner.test.ts > research > bad brave key degrades to wiki-only instead of failing
Error: Test timed out in 5000ms.

 Test Files  1 failed | 2 passed (3)
      Tests  1 failed | 57 passed (58)
   Duration  6.74s
Enter fullscreen mode Exit fullscreen mode

57 of 58. The one red test expects Brave's search API to reject a bad key quickly so the research op can degrade to wiki-only results; in my sandbox the request hangs instead of erroring, so it eats the 5-second timeout. Environment, not logic — but I'd rather show the red test than crop it out of the screenshot.

Try it 

npx mindbase-app        # web UI at localhost:4321, picks a local model for your RAM
Enter fullscreen mode Exit fullscreen mode

Claude Code (the flagship — full 8-step ingest with the sub-agents above):

/plugin marketplace add frankchu91/mindbase-llm-wiki
/plugin install mb@mindbase
Enter fullscreen mode Exit fullscreen mode

Any other MCP client (Cursor, Windsurf, Cline, Continue, Zed…):

{ "mcpServers": { "mindbase": { "command": "npx", "args": ["-y", "mindbase-mcp"] } } }
Enter fullscreen mode Exit fullscreen mode

Everything lives in ~/mindbase-data/ as plain markdown. Grep it, git it, open it in Obsidian, rm -rf it to leave. That exit door is a feature.

Honest limits

It's early access — 0.4.6, beta through 2026-Q4, and it moves fast enough to outrun its own README by one tool. The full conversational ingest (the mandatory takeaways discussion, Karpathy's step 4) lives in the Claude Code plugin; MCP-only editors get the same 50 tools but you drive the conversation yourself, and the web UI uses approval cards. On the roadmap: one-click browser capture, Whisper audio input, team projects with human review. And if you try it and bounce off, the README begs for the reason — I reply to issues same-day during beta, because the blockers are the roadmap.

Karpathy's pitch for the pattern is that a wiki is a compounding artifact. True — but compounding works for garbage too. The most important code in MindBase isn't the code that writes the wiki. It's the code that says no.

Repo: github.com/frankchu91/mindbase-llm-wiki

Top comments (0)