Every experiment in this series had charged each byte of a count table as resident memory. That is a real assumption and nobody had examined it, which I noticed only after reading someone else's project: a third-party runtime that runs 35B and 80B mixture-of-experts models on Apple hardware by keeping a dense core in memory and streaming experts from SSD, reporting 43 to 70% cache hit rates.
A count table is a random-access lookup structure, so in principle it pages. Only the contexts a document actually touches need to be in memory, and a dense network cannot do this because every forward pass reads every weight, which is why mixture-of-experts had to exist before streaming made sense. Code is also extraordinarily self-repetitive, with a measured repeat rate of 0.88 in an earlier experiment here, so the hot set should be small.
That reframes the question from "how much accuracy fits in B bytes" to "how many resident bytes does a given accuracy cost". The hypothesis is false. The more interesting part is what happened when I checked the number the conclusion depended on. The research repo is not public, so these come from its own harnesses.
The correctness gate, before anything else
The tables were rebuilt as fixed-stride, memory-mappable arrays: an open-addressed hash table per order at 20 bytes a slot, plus token and count regions. No auxiliary in-memory index, because an offset dictionary would be exactly the resident cost under measurement. Each record stores its own context tokens and they are compared on every probe, so a 64-bit hash collision cannot silently return the wrong distribution.
Then the gate. 13,500 sampled lookups, hits and misses, matched the in-memory tables exactly, and every evaluation position asserts the flat probability vector equals the in-memory one: zero mismatches across all 7,200 positions. Accuracy is identical by construction; only cost is at issue.
That step is not ceremony. A pageable rewrite of a lookup structure is precisely the kind of change that can be subtly wrong in a way that improves your numbers. The accuracy check also reproduces an earlier published figure, 0.421 against 0.413 for corpus-only at L=250, which is the second reason to believe the harness.
The working set does not saturate
The decisive measurement is to complete roughly 2,700 consecutive positions inside one real file and watch which table pages get touched.
| split | file tokens | positions | final working set | of 142.7 MB | last quarter added |
|---|---|---|---|---|---|
| in-dist | 186,057 | 2,702 | 3.9 MB | 2.7% | +0.31 MB |
| in-dist | 70,272 | 2,801 | 21.7 MB | 15.2% | +1.02 MB |
| in-dist | 55,809 | 2,772 | 25.2 MB | 17.6% | +2.71 MB |
| in-dist | 53,453 | 2,686 | 22.3 MB | 15.6% | +2.76 MB |
| cross-project | 107,204 | 2,715 | 20.0 MB | 14.0% | +1.58 MB |
| cross-project | 67,586 | 2,739 | 21.2 MB | 14.9% | +3.76 MB |
| cross-project | 62,998 | 2,622 | 18.1 MB | 12.7% | +2.85 MB |
| cross-project | 58,509 | 2,862 | 23.3 MB | 16.3% | +3.10 MB |
Seven of eight files are still climbing at 2,700 completions, with the last quarter of the session adding between 1 and 3.8 MB. There is no small plateau to hold resident. A bounded cache confirms it from the other direction: the hit rate plateaus at 0.744 even with 16 MB resident, against 0.656 at 4 MB and 0.473 at 1 MB.
So "a 143 MB table for 2 MB resident" is not on offer. At 2 MB, 43% of page reads miss.
Why, and it was already in the notes
The locality argument confused two things this project had already separated. Code's 0.88 repeat rate is about tokens recurring; the table is indexed by context. Two earlier experiments had measured exactly that distinction: within one document the cache saturates at order 4, and verbatim 5-grams never recur at any length tested. Novel order-3 contexts keep arriving for as long as you keep typing, so each position is a fresh key into a 1.93M-context table.
The runtime that prompted this works because a router picks a handful of experts per layer and the same experts serve many tokens, which is a reuse structure. An n-gram table has no router and nothing like that structure. This is the fourth time a locality or retrieval idea has lost in this project, and one sentence explains three of them: the predictor is whether the context recurs, not whether the token does.
One file did plateau, at 3.9 MB, six times smaller than its siblings. Locality varies enormously between files; any single hot-set figure is a fiction, and a tool cannot rely on the good case. If I had run one file instead of eight, there was a one-in-eight chance of publishing the opposite result.
The number the verdict rested on
Paging is still defensible as an engineering trade. About 15 page reads per position, with 26% missing at 16 MB resident, is roughly 4 faults per position. At 100 µs per read that is 0.4 ms against the code completer's 10 ms budget.
Except that 100 µs was a nominal figure. Every array in the experiment was in RAM, the page counts are analytic, and the latency was an assumption rather than an observation. A published verdict resting on a number nobody measured is the one thing this project does not tolerate elsewhere, so it got measured.
A sibling experiment had this machine's SSD at 2.5 to 7 GB/s for blocks of 1 MB and up, which does not answer the question at all: at 1 MB, 2.5 GB/s is 420 µs of pure bandwidth, so per-read latency is buried. A page fault reads a page, where latency dominates and bandwidth is irrelevant.
The method uses F_NOCACHE so reads reach the device, with random page-aligned offsets over a 12 GB blob; every read is content-verified against an offset-derived pattern. Two gates, because "I bypassed the cache" is exactly the claim that fails silently: a second pass over the same offsets under F_NOCACHE must not speed up, and a cached control must. Without the second gate the first one proves nothing.
| random read | 1 thread | 8 threads |
|---|---|---|
| 4 KB | 178.7 µs | 18.2 µs |
| 16 KB | 139.4 µs | 24.1 µs |
| 64 KB | 175.1 µs | 27.0 µs |
The gates came back at 1.40 for the F_NOCACHE two-pass ratio and 62.5x for the cached control at 2.1 µs per read, so these are device reads.
The assumption was optimistic by 1.8x. The real cost is 0.71 ms per position rather than 0.40; the qualitative verdict survives only because it had an order of magnitude of headroom, and it could easily not have.
Two things fell out that I would not have predicted. A 16 KB page is cheaper per read than a 4 KB page, 139 against 179 µs, so fewer faults and lower latency per fault point the same way and a pager should use larger pages. And eight threads buys about a 10x improvement in effective latency, which means read cost is not the binding constraint for a prefetching pager at all.
That last one is the part worth keeping. This experiment's negative stands on the working set never saturating, not on reads being slow, and the careful latency measurement I ran to defend the verdict showed that the risk I had priced was never the problem. Measuring an assumption is worth doing even when it does not change the answer. Finding out which of your worries was the real one is most of what a measurement is for.
Limits
8M tokens, order 4, code only. A 60M-token table would have a larger absolute working set, and whether the fraction falls is untested. Sessions are 2,700 sequential positions in one file, where a real editing session revisits regions and would show somewhat better locality. The cache policy is LRU at 4 KB granularity, and the table layout is insertion-ordered, which is the worst case for locality; a policy with frequency as well as recency, or a layout clustering co-occurring contexts onto the same page, is the one remaining lever if this direction is ever reopened. The latency figures are a Mac SSD at low queue depths, not phone storage, and the 1.40 residual ratio means some device-level caching may survive F_NOCACHE.
Worth noting too that making a table pageable is not free: the flat format costs 142.7 MB against 63 MB pickled, from open addressing at 50% load plus storing context tokens for collision safety.
Top comments (1)
The repeat-rate trap is well spotted: 0.88 token recurrence reads like locality, but the table is keyed by context, and those are different quantities. I've hit the same confusion in another shape — assuming a hit rate measured on tokens transfers to a structure indexed by sequences. The MoE comparison makes the diagnosis sharp: streaming experts works because a router creates heavy reuse of the same experts per layer, and an n-gram table has no router, so every new position can arrive as a fresh key into the context space.
The bounded-cache result is the part I'd trust most, since it measures the thing directly instead of arguing from a repeat statistic: a 0.744 plateau at 16 MB against 143 MB of table means there is no hot set to pin resident. Did the one file that plateaued at 3.9 MB share anything detectable up front — heavy boilerplate, repeated imports? Predicting from cheap file statistics when a working set stays small would be the useful takeaway for anyone deciding whether a paged variant is worth building.