DEV Community

VANSH ARORA
VANSH ARORA

Posted on

Measuring Context Packs Against Real Commits: Recall, Tokens, and the Limits

#ai #devtools #vscode #productivity

Most "fewer tokens" claims compare a small number to a big number and stop there. That rewards a tool for leaving things out. The TokenCap benchmark in benchmarks/fair/ scores two things for every task: the token count, and whether the files the change actually needed are in the context.

Where the tasks come from

Each task is a real commit from one of nine repositories:

  • fastify-example-todo (JavaScript)
  • full-stack-fastapi-template (Python + TypeScript)
  • hoppscotch (TypeScript + Vue)
  • flask and httpx (Python)
  • gin and cobra (Go)
  • ripgrep (Rust)
  • gson (Java)

The task text is the commit subject. The ground truth is the set of non-test source files the commit changed. The repository is evaluated at the commit's parent, so the change does not exist yet.

The selection rule is recorded in tasks.json. Per repository: the most recent non-merge commits that change 1 to 5 non-test source files (at most 8 files in total), are not dependency, release, docs, lint or translation commits, and have a subject of at least three words. Commits were listed by script before any measurement, and none was dropped after seeing results.

50 tasks were used while changing retrieval and packing code. 50 were held out. Two more selected commits are skipped because none of their files exists at the parent.

The three arms

  • TokenCap task pack: what tokencap ask "<task>" builds from the commit subject. The companion bridge builds the same pack.
  • Naive, all matching files: keyword grep over the repository, every matching file read in full.
  • Naive, 10 best-matching files: the top ten files from that grep.

Results (all 100 tasks)

Measure TokenCap task pack All matching files 10 best-matching files
Needed files present 87% 96% 76%
Average tokens per task 9,997 382,258 87,761
Median saving vs. that baseline n/a 18.9× 7.5×

On average, 73% of the exact lines a commit edited are in the pack.

Split by set: the held-out half scored 88% needed files, 74% edit-line coverage, and an 18.4× median saving. The tuning half scored 86%, 72%, and 20.4×. One caveat on the held-out side: the held-out tasks of the first three repositories were also run during an earlier round of work, before the other six repositories were added.

Running it 

node benchmarks/fair/fetch-repos.js ../bench-repos
node benchmarks/fair/run.js --repos ../bench-repos --set holdout
Enter fullscreen mode Exit fullscreen mode

CI runs the full set with --set all --gate. The gate fails the job when needed-file recall, edit-line coverage, or the median saving drops below the thresholds stored in tasks.json.

The limits

These are published next to the table in docs/HONEST-NUMBERS.md, not in a footnote.

  • It is not an answer-quality measurement. No model is called. A pack can contain the right file and still lead to a wrong answer. The harness for that check, benchmarks/fair/answer-eval.js, exists but has not been run, and there is no published model result.
  • Grepping everything finds more. 96% against 87%. The pack wins on tokens (38× fewer on average) and beats the 10-file baseline on both counts.
  • Spans can miss the edit. Large files ship as selected functions and lines. Edit-line coverage is lowest on repositories with very large files: 56-58% on cobra, ripgrep, and gson.
  • Small repositories gain little. The median saving on the small repository is 1.6×, and in 3 of its 9 tasks the pack was larger than the naive read.
  • Median, not mean. The mean saving is 34.2×, pulled up by the large repository. The median, 18.9×, is the representative figure.
  • Task text is terse. Commit subjects are shorter than what a developer usually types.
  • Scope. This measures the task pack built by tokencap ask and the companion bridge. It does not measure the tokencap make capsule.

Why it was worth building

Before the v3.1 retrieval changes, the first three repositories scored 25% needed-file recall on this benchmark. Those result sets are still in benchmarks/fair/results/; the 100-task results are in benchmarks/fair/results/v2/.

A benchmark that only counts tokens would have rated that 25% version as a success. This one did not, which is the whole point of having ground truth.

Full table, method, and limits at tokencap.vansharora.app

Top comments (0)