How We Achieved a 12.2x Token Reduction on Codebase Context

When feeding codebase context into LLMs, bigger context windows have created sloppy habits. Handing 150k tokens of raw source files to Claude 3.5 Sonnet or GPT-4o degrades recall accuracy and costs serious money when run in automated agent loops.

We built TokenCap to solve this locally. In recent benchmark evaluations against representative multi-file repositories, TokenCap compressed prompt payloads from 152,000 tokens down to 12,450 tokens while maintaining complete call-chain resolution.

Benchmark Methodology

We evaluated TokenCap on standard multi-file refactoring and bug-localization tasks:

1. Baseline: Full-file concatenation of all modules in the task directory tree.
2. TokenCap: Running tokencap make --profile balanced with tree-sitter AST extraction and retrieval-gated budgeting.
3. Metric: Token volume measured via js-tiktoken (cl100k_base tokenizer) and recall accuracy on target symbol call sites.

The result was an average 12.2x reduction in token volume. More importantly, hallucinated import calls dropped because low-relevance code was pruned before context generation.

How the Budget Engine Works

Instead of naive file truncation, TokenCap parses the codebase into an AST graph using WebAssembly tree-sitter grammars. It runs Louvain community clustering to identify functional boundaries and ranks symbols using call frequency and proximity:

tokencap make --profile balanced

CLI output:

Indexed 84 files in 1.1s
Call graph: 412 symbols, 891 call edges resolved (88% precision)
Token budget: 12,450 / 150,000 max (12.2x compression)
Wrote .tokencap/snapshot.md

The resulting snapshot provides the model with full function signatures, exact export shapes, and high-priority implementation spans while folding repetitive structures.

If you are developing LLM workflows or MCP servers, stop feeding raw file dumps to your agents.

Read more at tokencap.vansharora.app