The previous experiment found that a count table over the current document overtakes a small transformer at a few hundred tokens of document, and that the transformer's marginal contribution collapses past that point. There is one obvious objection and it deserves a measurement rather than an argument:
Just train more. 8M tokens is nothing. Of course a 2.45M-parameter transformer loses; train it on a real corpus and the crossover disappears.
The objection is directionally right and quantitatively weak, and the useful part is the rate. The research repo is not public, so the figures come from its own harness.
Why this needed no new training
An earlier experiment had saved checkpoints at 500K, 2M and 8M tokens of code with identical architecture, so the sweep is a re-evaluation rather than a training run. The cache uses no training data at all, which makes its accuracy a constant across the sweep: whatever the transformer gains from 16x more data is exactly what moves the crossover.
There is one wrinkle worth stating because it could have quietly broken the comparison. Each checkpoint built its own 16K vocabulary from its own subset, and only 39.8% of the 500K and 8M vocabularies overlap. So every cell is encoded with the vocabulary of the checkpoint it tests, and the corpus count model is built on the same tokens the transformer was trained on. Comparisons are always within one data scale; what gets compared across scales is only the crossover position.
The table
In-distribution held-out code. The value is the paired delta, cache minus transformer, so positive means the zero-parameter cache wins.
train 250 400 600 800 1000 4000
500,000 0.073* 0.123* 0.124* 0.193* 0.196* 0.268*
2,000,000 0.022 0.077* 0.078* 0.151* 0.135* 0.226*
8,000,000 -0.057* 0.006 -0.009 0.055* 0.044* 0.098*
(* = paired-significant)
The cache's own top-1 is 0.311, 0.312, 0.313 at L=250 and 0.487, 0.488, 0.487 at L=4000 across the three rows. It is supposed to be constant, because it never sees the training corpus, and it is, to within a vocabulary difference. That invariance is the experiment's internal control: every bit of movement in the table is the trained component improving, and if the cache row had drifted, nothing else here would be readable.
The exchange rate
A log-linear fit of each row's delta against document length:
| training tokens | crossover |
|---|---|
| 500K | ~74 tokens of document |
| 2M | ~155 |
| 8M | ~492 |
Sixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling of training data. Extrapolating that rate, which is illustrative rather than measured, since this project's own earlier work is a warning about trusting a slope past its fitted range:
| to be competitive at | needs |
|---|---|
| 1000 tokens of open file | ~23M tokens (3x) |
| 4000 tokens | ~174M tokens (22x) |
| 16000 tokens | ~1.3B tokens (167x) |
Using only the 2M to 8M segment, where the model is less undertrained, the rate is 1.78 per doubling and L=4000 needs about 97M tokens, so 12x. The honest range for matching a zero-parameter cache on a 4000-token file is therefore roughly 10 to 20 times more in-domain code than CPython's standard library and PyTorch combined.
That is what makes the objection weak rather than wrong. The crossover does move. It moves at a price, and per-repo completion is precisely the setting where the code to pay it does not exist.
Why the rate is so poor: two unrelated slopes
Training data lifts the transformer's flat line. It does not give the model access to the document. A 64-token window cannot see a 4000-token file no matter how much text it was trained on, so the transformer's gain from data is roughly the same at every document length: +0.132 at L=250, +0.148 at L=1000, +0.170 at L=4000 across the full 16x.
The crossover moves only because the flat line rises, which makes the exchange rate a race between two slopes that have nothing to do with each other. They turn out to be the same size:
- 16x more training data moves the transformer from 0.219 to 0.389 at L=4000, so +0.170.
- The cache seeing 250 rather than 4000 tokens of the file it is already inside moves it from 0.313 to 0.487, so +0.174.
Sixteen times the training corpus buys about as much as letting a zero-parameter mechanism read the rest of the file it is already sitting in. One of those is a data-collection project; the other is already in the editor's buffer.
A side result that surprised me more than the headline
At 8M tokens and L=250 the corpus count table scores 0.415 against the transformer's 0.370. That reproduces an earlier finding, that counting beats the small transformer on code, on a different harness, which is the sort of accidental replication worth keeping.
Combined with the corpus-plus-cache numbers from a neighbouring experiment, 0.461 rising to 0.551 across these lengths, the transformer does not beat the free components at any document length in this range. It only ever earns its place as a mixture component, at a weight of about 0.2.
What this does not settle, and it is the more serious half
The objection had two halves and this addresses one. The other one is that parameters and context window are confounded in every experiment in this line. Everything here varies training data at a fixed 64-token window, and the cache's advantage is entirely about seeing text the transformer cannot see. A model with a 4096-token window would read the document itself.
That is the experiment this line of work still owes, and until it is run, the correct summary is narrow: at a fixed short context, buying document access is far cheaper than buying training data, and the exchange rate is measured rather than argued.
Three limits on the numbers themselves. The middle checkpoint is the weak link, because the transformer was nearly flat from 500K to 2M and jumped at 8M, so the global fit averages a slow segment with a fast one; both segment rates are given above for that reason. Everything is in-distribution, and cross-project the cache wins earlier, so this is the conservative split for the cache. And the chunking takes text from the start of each file, so larger L also means positions deeper in a file: the within-cell comparisons that make up the whole table are clean, but a single system's trend across L is not.
Top comments (0)