DEV Community

VANSH ARORA
VANSH ARORA

Posted on

Budget-Exact Context Packing: Render First, Then Charge

A task pack has a token budget. TokenCap's v3.1 packer is built around one rule: a file costs what it renders, not what we estimated it would cost. This post walks through how that works in src/pack/fitSections.js.

The problem with estimates

The older allocator assigned each file a compression level before any content existed. A file reserved as "whole" but later demoted, or spanned down to 10% of its size, left its reservation unused. Budget was lost to bookkeeping.

The new packer renders files in rank order and subtracts the real token count of each rendered section from the remaining budget.

Pass 1: code, with a per-file allowance

Each file gets an allowance that scales with its relevance score. The best match may take up to 18% of the budget. Weaker matches get proportionally less, with a floor of 500 tokens. This stops one large file from taking the budget that five medium ones needed.

If the whole file fits its allowance, it ships whole. If not, it ships spans.

Spans around the rarest terms

Every line that contains a query term gets a score. The score is not how many query terms it contains. It is how rare those terms are inside this file:

rarity(term) = log(1 + totalLines / linesContainingTerm)
Enter fullscreen mode Exit fullscreen mode

A term on every other line of a file is noise for that file. A term on one line is a signal. Lines are taken best-first until the allowance is used.

Around each chosen line, the packer shows a window of 6 lines before and 10 after. The file's import header is kept, up to a quarter of the line allowance.

Whole-function spans

Windows cut functions in half, and a half function is hard to edit correctly. So when the matching line is inside a function of 60 lines or fewer, the whole function is shown instead of the window.

Function boundaries come from the Tree-sitter parser when the grammar is loaded, with a pattern-based fallback otherwise. A function longer than 60 lines gets its signature line kept, so the window still shows which function it belongs to.

v3.1 also fixed grammar loading on Node 20. Grammars were loaded all at once, and that left only Rust parseable. JavaScript, TypeScript, Python, Go and Java silently fell back to text heuristics for boundaries. They now load one at a time, and packs are identical across Node 18 and 20.

Pass 2 and pass 3

Whatever budget is left after pass 1 is spent showing spanned files whole, best match first.

Files that did not fit as code get, in order of preference, a structural outline, a summary, or a single reference line. You still know they exist and can ask for them.

Reference-only files

Files that score under 15% of the best match skip code entirely. They go straight to a reference line under "Referenced Files": path, score and size. A file that weakly matches is context, not an answer, and showing its code costs budget the strong matches need.

Protected task budget

The Chrome companion's task packs carry two extra sections: a project primer for a cold chat, and a focus brief listing the selected files and why they were chosen. Both are bounded separately. The primer gets at most 1,400 tokens or 20% of the budget, the focus brief at most 600 tokens or 16%, and source code gets the rest. On small model budgets, orientation text cannot displace the code you asked about.

Running it

tokencap ask "cookie fallback" --budget 8000
Enter fullscreen mode Exit fullscreen mode

The default budget is 12,000 tokens. Spanned files are labeled in their header, for example span: 48/612 lines.

What the benchmark shows

On the benchmarks/fair/ set of 100 real commits across nine repositories, the task pack held the needed files as code 87% of the time, at a median 18.9x fewer tokens than reading every matching file. Reading every matching file finds more, 96%. Spans also miss edits: on average 73% of the lines a commit changed were in the pack, and 56-58% on the repositories with very large files. No model is called in this benchmark, so it measures retrieval and packing, not answer quality. The full limits are in docs/HONEST-NUMBERS.md.

More: tokencap.vansharora.app

Top comments (0)