DEV Community

vadim albarov
vadim albarov

Posted on

A Design Flaw in Claude Code's Documentation Skill: One Question, 265k–355k Tokens

I typed a one-line question into Claude Code - does fable use api billing? - and then ran /context out of habit.

30% of a 1,000,000-token context window was gone. One question, one answer, 295k tokens used.

Then I ran the same prompt on a second laptop: 40%. Same question, same answer, ~100k tokens more.

Same CLI version on both machines. I'll call them laptop 1 (30%) and laptop 2 (40%). This is the story of finding those 100k tokens. Spoiler: every theory I had was wrong, and the root cause turned out to be a design flaw in a single bundled skill - one you can partially work around by adding files to a folder.

The baseline

Here's what /context showed on laptop 1 after that single exchange:

Category Tokens
System prompt 5.3k
System tools 23.7k
MCP tools (59 tools, deferred) 0
Memory files 318
Skills (16 skill descriptions) 2.2k
Messages 264.6k
Free space 703.9k

Two things jumped out.

First, the fixed overhead everyone worries about - MCP servers, skills, memory - is nearly free. 59 MCP tools sat at 0 tokens because Claude Code defers their schemas until they're actually used. All 16 skill descriptions together cost 2.2k tokens.

Second, the Messages category held 264.6k tokens after a one-line question. The conversation itself was maybe 2k tokens. The rest arrived because my question mentioned a Claude model name, which triggered the built-in claude-api skill - and a skill trigger doesn't just load instructions. This one injected its entire documentation payload into the conversation as a single message.

Worth pausing on that: my question was a billing lookup that a single Google search answers in five seconds. And before you conclude "well, agent sessions are just expensive" - they aren't. As a control, I asked two other lookup questions in fresh sessions: "what's the latest Node LTS version?" and "MIT vs Apache 2.0?". Both together cost 7.5k tokens. Ordinary questions are cheap.

The 265k burn has one specific trigger: mentioning a Claude model name. That summons the built-in claude-api skill, which has no notion of question weight - a casual pricing question gets the exact same multi-hundred-KB documentation payload as "implement a streaming tool-use loop." I paid a quarter of a million tokens for what one web search would have told me.

On laptop 2, that same category showed 354k tokens. The 90k-token mystery lived entirely inside one message.

Wrong theory #1: local CLAUDE.md files

My first guess: laptop 2 has more project instruction files - CLAUDE.md, memory, rules - quietly injected into context.

Dead on arrival. Memory files accounted for 318 tokens on laptop 1, and the gap was ~90k tokens ≈ 360 KB of text. A CLAUDE.md would have to be a small book. More importantly, the gap persisted when I ran the prompt from an empty folder on laptop 2 - no project files at all, still ~355k.

Wrong theory #2: skill version

Bundled skills live in a content-addressed cache:

%LOCALAPPDATA%\Temp\claude\bundled-skills\<cli-version>\<hash>\claude-api\
Enter fullscreen mode Exit fullscreen mode

Both machines ran CLI 2.1.226 with skill version 2.1.226. Same version... but different hashes. Promising! Except when I compared the actual files, every doc folder was byte-identical - 847 KB of assets on both machines. The hash difference was real but, as it turned out later, a symptom rather than a cause.

Wrong theory #3: the tokenizer

This one almost fooled me, because the arithmetic was beautiful. I had tried Sonnet 5 at some point, and it ships a new tokenizer that produces roughly 30% more tokens for the same text.

264.6k × 1.3 ≈ 344k. Almost exactly laptop 2's number. Same bytes, different ruler!

Then I ran the control: same model, same effort, both machines. The gap survived. Tokenizer eliminated. (Keep this failure mode in mind though - token-budget intuitions genuinely don't transfer across model families.)

Transcript forensics

Same CLI, same skill assets, same model, same prompt - and different token counts. At this point the only honest move was to stop theorizing and diff the actual bytes.

Claude Code writes every session to a JSONL transcript. Finding the heavy message takes one loop:

$proj = Get-ChildItem "$env:USERPROFILE\.claude\projects" -Directory |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$t = Get-ChildItem $proj.FullName -Filter *.jsonl |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$i = 0
Get-Content $t.FullName | ForEach-Object { $i++
  if ($_.Length -gt 50KB) { "Line $i : $([math]::Round($_.Length/1KB)) KB" } }
Enter fullscreen mode Exit fullscreen mode

Laptop 1 transcript: one message of 719 KB. Laptop 2 transcript: one message of 957 KB. There's the gap - 238 KB of text, ~90k tokens at ~2.6 characters per token.

The payload is the skill's documentation, embedded as <doc path="..."> blocks. Extracting the doc lists:

[regex]::Matches($line, '<doc path=\\"([^\\"]+)\\"') |
  ForEach-Object { $_.Groups[1].Value }
Enter fullscreen mode Exit fullscreen mode
  • Laptop 1: 32 docs - shared API docs + the python/ folder.
  • Laptop 2: 65 docs - shared API docs + all eight language folders: Python, TypeScript, Go, Java, C#, PHP, Ruby, and cURL.

All 32 docs the payloads had in common were byte-identical. The laptop 2 payload simply contained 33 extra language docs totaling 237 KB.

The reveal

Diffing the instruction text at the top of the two payloads (60 KB each, otherwise identical) surfaced exactly one difference:

Laptop 1 payload:

→ Refer to python/claude-api/README.md

Laptop 2 payload:

→ Refer to unknown/claude-api/README.md
No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

That's the whole mechanism. When the claude-api skill triggers, Claude Code detects your project's language from the working directory and injects only that language's documentation. When it detects nothing - an empty folder, a docs-only repo - the fallback is to inject documentation for every supported language, because the model might need any of them.

And why did laptop 1 detect Python? The folder I was in had a subdirectory containing a Python project with a .venv - thousands of .py files. A stray virtualenv saved me 90k tokens.

The different cache hashes made sense now too: the skill bundle appears to be cached per rendered variant - the assets are identical, but the instruction text differs by detection outcome, so each outcome gets its own hash directory.

The irony, and the side effect worth knowing

Here's my favorite part. When I got serious about controlling variables, I ran the "clean" experiment: empty folder, fresh session, same prompt. That methodologically pure setup is precisely what maximizes the payload. The controlled experiment created the condition it was measuring.

The flip side is a genuinely useful, if odd, side effect: having language context in your working folder reduces token usage. Any file that lets Claude Code detect a language - a .py file, a package.json, a pyproject.toml, even a leftover .venv - pins the skill payload to one language's docs and cuts ~90k tokens (about 25% of the payload) off every claude-api skill trigger. The final scoreboard, reproduced on both machines:

Working directory Docs injected Payload Messages after one question
Any folder with language markers 32 (shared + one language) 719 KB ~265k tokens
Empty / language-less folder 65 (shared + all 8 languages) 957 KB ~355-360k tokens

So if you're about to ask Claude Code API questions from some scratch directory: don't. Run it from a real project - or drop a single pyproject.toml (or the equivalent for your language) into the scratch folder first. It reads as a joke, but it's a measurable 90k-token difference per session, and on smaller context windows it's not funny at all: the all-languages payload alone wouldn't fit in a 200k-token context window. This entire question is only answerable on a 1M-window model.

Why this is a flaw, not a feature

Skills in Claude Code are designed around progressive disclosure: a one-line description sits in context (all 16 bundled skills together cost 2.2k tokens), and the full instructions load only when triggered. The claude-api skill follows that pattern for its trigger - and then abandons it entirely for its content: instead of letting the model read the docs it needs on demand, it eagerly injects the whole documentation set as a single message. It's the only bundled skill big enough to need a disk cache at all (847 KB; the other 15 are trivially small).

The no-language fallback makes it worse in a way that's almost comic. The injected instruction text literally says:

No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

...but by the time the model can ask, all eight languages' documentation has already been paid for. The question comes after the purchase. Any of the obvious designs - ask first and inject one language; inject the shared docs and let the model read language files on demand; scale the payload to the question - would cap the cost at the detected-language level or below.

To be fair about scope: this is one skill, in one CLI version (2.1.226), and the payload is genuinely useful when you're writing code against the Claude API in a detected-language project. The flaw is the eager all-languages fallback and the trigger's insensitivity to question weight - both fixable upstream without losing what the skill is for.

Takeaways

  1. Skill payloads dominate context cost. Everything people usually blame - MCP servers, memory files, system prompts - added up to ~31k tokens for me. One skill trigger added 265-360k. Audit accordingly.
  2. The flaw is specific, not general agent overhead. Control questions in fresh sessions ("latest Node LTS?", "MIT vs Apache 2.0?") cost 7.5k tokens combined. The expensive trigger is mentioning a Claude model or API name. For quick Claude pricing/docs lookups, use a web search - inside Claude Code, the skill will fire.
  3. Deferred loading works. 59 MCP tools at 0 tokens until used. If your setup loads MCP schemas eagerly, that's worth fixing, but it wasn't my problem.
  4. /context tells you the category; the transcript tells you the culprit. The JSONL line-length trick above takes 30 seconds and points at the exact message.
  5. Same CLI version ≠ same context cost. The cost depends on runtime conditions - in this case, what's sitting in your working directory.
  6. Language markers in the folder are a token optimization. Unintuitive, but reproducible: give the language detector something to find, and the skill injects one language's docs instead of eight.
  7. Token intuitions don't transfer across model families. My tokenizer theory was wrong this time, but the ~30% Sonnet 5 difference is real - re-baseline when you switch models.

Everything above was measured on Claude Code 2.1.226 (bundled claude-api skill 2.1.226) with Fable 5 and Sonnet 5, on two Windows machines. The behavior may well change in future releases - arguably the fallback should ask before injecting 237 KB of polyglot documentation - but the audit method will keep working regardless.

Reproduce it yourself: ask Claude Code any Claude-API question from an empty folder, run /context, then do the same from inside a Python or TypeScript repo and compare the Messages category.

Top comments (2)

Collapse
 
alexshev profile image
Alex Shev

This is a good reminder that a skill can look helpful while quietly becoming a context budget attack. I would treat documentation retrieval like dependency loading: it needs a small manifest, a reason for each included file, and a hard budget that fails visibly when the skill tries to pull the whole world in.

Collapse
 
reidmarlow profile image
Reid Marlow

This feels like a regression test hiding in plain sight. A skill loader needs a cheap preflight step and a token budget it can only cross explicitly. If an empty folder costs more than a real project, the fallback path needs the most tests.