I tried Karpathy's LLM Wiki pattern on my AWS consulting work — here's what changed

Alex — Tue, 14 Apr 2026 00:22:22 +0000

I tried Karpathy's LLM Wiki principle on my own AWS consulting work a few weeks ago, and I can't go back.

If you missed it, Andrej Karpathy published a short gist called LLM Wiki: A Pattern for Persistent Knowledge Bases. His thesis: instead of using RAG to re-derive answers from raw sources on every query, let the LLM maintain a persistent, compounding wiki. Humans curate sources and ask questions. The model handles the bookkeeping — summarizing, cross-linking, keeping pages consistent over time.

So I tried it. I rebuilt my entire AWS consulting knowledge base around Karpathy's three-layer pattern. This post is what I actually did, what the folder looks like now, the frontmatter schema that makes it queryable by Claude Code skills, and the three rules I had to enforce to stop it from rotting.

At the bottom there's a copy-pasteable Claude Code prompt that scaffolds the same structure in an empty directory — if you want to try it yourself.

Why I tried this on AWS consulting specifically

Every engagement I run has the same four information streams, and all four used to decay fast:

Client-specific context — their account structure, compliance posture, team, kickoff constraints.
Cross-project AWS patterns — cross-account alarm routing, IAM boundaries, cost allocation.
Decisions — why I picked Aurora over RDS for this client, what tradeoff I accepted.
Gotchas — what broke, in what AWS service, under what conditions.

Before Karpathy's pattern, all four lived, well not in the same place let's say. Every new engagement brought the same conclusion that I was the bottleneck :).

Now the LLM compiles it once and keeps it current for me. I'm not the bottleneck anymore.

The three layers, as I actually set them up

Layer 1 — Raw sources (immutable)

./raw/
├── assets/                      # Downloaded images, screenshots
├── aws-reinvent-2024-obs.md     # Clipped article
├── client-kickoff-2026-03.md    # Meeting transcript
├── datadog-quote-client-a.pdf   # Vendor docs
└── aws-bill-client-b-q1.csv     # Cost exports

Raw never changes. If the wiki layer gets corrupted or I change conventions, I can re-derive from raw. That immutability is what gave me the confidence to let Claude aggressively refactor the wiki.

Layer 2 — Wiki (LLM-maintained markdown)

./wiki/
├── projects/
│   ├── client-a/
│   │   ├── _status.md             # Dashboard — I start here every session
│   │   ├── client-a-overview.md   # Architecture + key decisions
│   │   ├── decisions/             # ADR-style, one file per decision
│   │   ├── workflows/             # Automation-ready procedures
│   │   └── gotchas/               # What broke and why
│   └── client-b/
├── sa/                            # Solutions architecture patterns (cross-client)
├── aws/                           # AWS service patterns + gotchas
└── standards/                     # CDK conventions, tagging, account standards

One vault, per-client folders. Cross-client patterns live in sa/, aws/, standards/ — my opinion library. Client folders stay isolated so I never accidentally leak Client A's IAM choices into a Client B audit.

Layer 3 — `CLAUDE.md` (the schema)

Karpathy calls this the "config for how the LLM operates on the wiki." For me, it's a single file at the vault root that defines the directory structure, the frontmatter schema, what belongs in the wiki and what doesn't, and the workflows I want Claude to follow (ingest a source, update _status.md, lint for orphans).

Change CLAUDE.md and you change how every future Claude Code session operates on the vault. It's the load-bearing file.

The frontmatter schema I landed on

Every page has YAML frontmatter. Not for me to read — for skills to query:

---
title: "Cross-account CloudWatch alarm routing"
type: pattern
tags: [aws, cloudwatch, cross-account, observability]
sources: [raw/aws-reinvent-2024-obs.md, project: client-a]
created: 2026-03-02
updated: 2026-04-11
---

sources is non-negotiable for me. It keeps the agent honest when it synthesizes — every claim in the wiki traces back to something in raw.

The first skill I wrote that made me a believer

Here's the one that convinced me this was worth the effort: an architecture-review skill I can run when a client proposes a change.

---
name: architecture-review
description: Reviews a proposed architecture change against client constraints and cross-client patterns
---

Inputs:
1. Client name (must match a folder in wiki/projects/)
2. Proposed change (free text or linked diagram)

Steps:
1. Read wiki/projects/<client>/_status.md for current phase and constraints
2. Read wiki/projects/<client>/<client>-overview.md for architecture baseline
3. Query wiki/sa/ and wiki/aws/ for pages where tags intersect with the change
4. Query wiki/projects/<client>/gotchas/ for anything relevant
5. Query wiki/projects/<client>/decisions/ for prior decisions that constrain this change
6. Produce a review document with:
   - Compatibility with existing architecture
   - Relevant cross-client patterns that apply
   - Prior decisions that would be affected
   - Gotchas that could bite
   - Recommended next steps

Output: wiki/projects/<client>/decisions/<YYYY-MM-DD>-<slug>.md (draft for human review)

There's no magic prompt here. The whole point is that the skill is running against structured, maintained context — not a blank LLM. The quality of the output is about the quality of the wiki, not the prompt.

The three rules I had to enforce

Karpathy warns that maintenance is what kills human wikis. Even with an LLM doing the work, I found three rules that actually matter in practice:

Frontmatter or it didn't happen. No frontmatter = skills can't index it. I make Claude add frontmatter before content on every new page.
_status.md updates at session end, not start. I set up a Claude Code Stop hook that blocks me from ending a session until the dashboard moves forward. Without the hook, dashboards went stale inside a week.
One page, one concept. Gotchas get their own pages, not subheadings under a shared "troubleshooting.md". The retrieval unit is a page — keep it small enough that multiple fit in context without bleed.

What actually changed in my work

Three concrete things shifted after a few weeks:

Architecture designs are regenerated, not rewritten. A client constraint changes → I update the relevant page → rerun the design skill → new design doc.
Audits are reproducible. Same vault + same skill on the same day = same findings. I couldn't get that property out of ad-hoc Claude chats no matter how careful I was with prompts.
Handover is the deliverable. At the end of an engagement, the client gets the CDK and their vault folder. The vault is the documentation.

The output layer

Everything Claude generates in my setup ends up as CDK. I open-sourced the constructs at up-cdk — compliance-aligned building blocks for S3, VPC, pipelines, so every stack is compliant by default. Wiki + skills + opinionated output library is the combination that finally made the setup reproducible across clients.

If you want to try it — the bootstrap prompt

The fastest way to see if this pattern works for you is to try it. Here's the Claude Code prompt I use to scaffold the whole structure in an empty directory. No scripts, no deps, just markdown and conventions.

You are setting up a new LLM Wiki following Andrej Karpathy's pattern
(https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f),
adapted for AWS consulting / platform engineering work.

The wiki has three layers:
- `raw/` — immutable source material (meeting notes, clipped articles, bills, screenshots)
- `wiki/` — LLM-maintained markdown synthesized from raw + conversation
- `CLAUDE.md` at the root — the schema document that tells you how to operate on the wiki

Create the following structure in the current directory. For every file below,
create it with the frontmatter and seed content described. Do not skip any
file. Do not invent additional files.

## Directory tree

./
├── CLAUDE.md
├── index.md
├── log.md
├── raw/
│   ├── assets/.gitkeep
│   └── README.md
├── templates/
│   ├── project-status.md
│   ├── page.md
│   └── workflow.md
└── wiki/
    ├── projects/
    │   └── _example-client/
    │       ├── _status.md
    │       ├── _example-client-overview.md
    │       ├── decisions/.gitkeep
    │       ├── workflows/.gitkeep
    │       ├── gotchas/.gitkeep
    │       └── diagrams/.gitkeep
    ├── sa/.gitkeep
    ├── aws/.gitkeep
    └── standards/.gitkeep

## CLAUDE.md (the schema document)

Write a root CLAUDE.md that defines:
- Purpose: personal AWS knowledge base following Karpathy's LLM Wiki pattern.
- The three layers: raw/ (immutable), wiki/ (LLM-generated), CLAUDE.md (schema).
- What belongs in the wiki: durable knowledge (gotchas, decisions, patterns,
  strategy, workflows, status dashboards). What does NOT belong: raw doc
  summaries, git state, ephemeral task context.
- Page format — every wiki page must have YAML frontmatter with fields:
  title, type (pattern | gotcha | decision | concept | process | status |
  workflow), tags, sources, created, updated.
- File naming — kebab-case. Overview files: <project>-overview.md. Status
  files: _status.md (underscore prefix sorts to top).
- Cross-references — [[filename]] Obsidian-style wiki links.
- Every page ends with a ## Related section.
- Workflows (document all four):
  1. Quick context recovery at session start — read
     wiki/projects/<project>/_status.md first.
  2. Update _status.md at session end — refresh Last/Next session lines,
     append one row to Session History (keep last 5), update Recent Decisions
     and Open Questions, set updated: to today.
  3. Ingest a raw source — read thoroughly, write/update pages in the right
     wiki/ subdirectory, cross-link, update index.md, append to log.md.
  4. Lint — find orphan pages, stale content, missing back-links.
- Common tags taxonomy — starter list for AWS, SA, meta tags.
- Persistence layer rules — wiki is the single source of truth for durable
  knowledge.

## index.md
A one-page catalog. Headers for each top-level wiki directory (## Projects,
## Solutions Architecture, ## AWS, ## Standards), each with "No pages yet."

## log.md
Append-only activity log. Seed with today's date and "Initialized wiki
following Karpathy LLM Wiki pattern."

## templates/project-status.md
Template for _status.md files. type: status. Sections: Phase, Last session,
Next session, Current State, Recent Decisions, Open Questions, Session
History (Date | Summary table), Quick Links.

## templates/page.md
Generic page template. Frontmatter + ## Key Points / ## Details / ## Related
/ ## Sources.

## templates/workflow.md
Automation-ready procedure template. type: workflow. Sections: Goal,
Prerequisites, Steps (numbered, imperative), Verification, Rollback (if
applicable), Related.

## wiki/projects/_example-client/_status.md
Realistic example status dashboard. Phase: active. Plausible placeholder
content. Marked as example.

## wiki/projects/_example-client/_example-client-overview.md
Example architecture overview. Frontmatter + 3-sentence overview + Key
Points + Architecture + Recent Decisions + Related.

## raw/README.md
One-page explanation of what goes in raw/ and the immutability rule: never
edit raw files after adding them.

---

After creating everything, print a summary (file count + tree) and next
steps: (1) add your first raw source to raw/, (2) create your first real
project folder under wiki/projects/, (3) commit to git.

Do not add files beyond what's specified. Do not add a root README.md.
Do not add a LICENSE.

Run it from an empty directory, then git init, then add your first raw source. I had a working LLM Wiki in under two minutes.

The original gist is short and worth reading: Karpathy — LLM Wiki.

If you try it, drop a comment with how you ended up shaping it. I'd genuinely like to compare notes.

DEV Community: Alex