Private AI has a hardware story nobody measures honestly. The pitch is that your data never leaves the building. The unstated cost is that the building contains a 6GB laptop GPU, and the corpus is 440,000 tokens that a frontier model would swallow in one call.
We measured the whole thing. One RTX 3060 Laptop, 6144 MiB. One 2,625-record driving-QA dataset rendered to 957,493 characters. One question with a single correct answer: which category is least common. Ground truth computed in Python, so neither model gets to define success.
The 4B won. It took four failed attempts to understand why.
The context problem was never the hard part
Direct context on that card, measured with flash attention and a q8_0 KV cache:
| num_ctx | resident | fits 5.5GB usable |
|---|---|---|
| 8,192 | 3.2 GB | yes |
| 16,384 | 3.3 GB | yes |
| 32,768 | 3.3 GB | yes, the ceiling |
| 65,536 | 10.4 GB | no, 31 percent on GPU |
| 131,072 | 10.9 GB | no |
KV is nearly free to 32k, then the allocator falls off a cliff. Switching the KV cache to q4_0 changed those numbers not at all, so the cliff is not the KV cache and you cannot quantize your way past it. Direct ceiling: 32,768 tokens.
Now stream a 261,226-token corpus through the same card in chunks, keeping the text in CPU RAM and only ever showing the model a fragment. Peak GPU across four runs: 4.23, 4.24, 4.25, 4.54 GB. Every one fits.
That is an 8x context multiple at constant VRAM. Constant is the load-bearing word: the corpus never enters the KV cache, so the limit stops being memory and becomes wall-clock.
Which is a solved problem. The unsolved one was that the answers were wrong.
Four ways to fail the same question
| attempt | sub-calls | answer |
|---|---|---|
| 1 | 7 | prose describing the data, having read about a third |
| 2 | 11 |
Spatial — right arithmetic, truncated label |
| 3 | 65 |
Counterfactual — swept everything, well-formed, wrong |
| 4 | 72 | Status: beta, Status: delta, Status: gamma, Status: alpha |
Attempt 4 is the one that gives it away. Asked which label was least common, the model listed all four candidates instead of selecting one. Attempt 3 had read all 425,054 input tokens and still combined the partial counts wrongly.
We had been calling this a capability gap. It was not. A 4B can count rows in a fragment. What it cannot reliably do is plan a traversal and then perform arithmetic across 65 partial results — and we had been asking it to do both, inside a loop, while also formatting an answer.
The fix: treat the corpus like a video stream
Nothing about aggregation requires a language model. So we stopped asking one.
| video streaming | the port |
|---|---|
| manifest / playlist | segment plan, computed in code before any model call |
| buffer | N segments in flight, latency hidden behind compute |
| decoder | model sees ONE segment, emits key<TAB>number, never prose |
| playback | reduce phase — aggregation in code, strategy chosen explicitly |
The model's entire job becomes extraction from a window it comfortably fits. Planning is deterministic. Summation is a loop. Selecting the minimum is one comparison.
Result on the same corpus, same 4B, same 6GB card:
17 segments · failed=0 · records=611 · unparsed_lines=3 · keys=15 · 641s
Category: Spatial Relationship
Correct. 15 keys recovered, matching the 15 real categories. Of 611+ emitted lines, 3 failed the output contract and were counted as failures rather than silently dropped.
The frontier comparison, which is not flattering to the frontier
We used Claude Opus over the full 439,742-token context as the reference. It answered the same question two different ways on byte-identical input — correct once, wrong once — and landed 2 out of 3 across three samples, at $4.79 per call.
| answer | correct | cost | |
|---|---|---|---|
| Opus 4.8, one call, 439,742 tok | varies by run | 2/3 | $4.79 |
| 4B + streaming, 6GB laptop | Spatial Relationship |
yes | $0.00 |
A reference that disagrees with itself is not an oracle. Ground truth has to come from code, with the frontier model scored as just another candidate.
For private AI this cuts two ways. The bar is lower than the marketing implies, because context rot is real and frontier models are not deterministic at 400K tokens. But "matches the frontier model" is also the wrong success criterion. Match the ground truth, and measure both.
What we would tell anyone building this
Recursion is not free. On a benchmark where the context already fit the model's window, wrapping the same model in a recursive harness scored 0.269 against 0.428 for a plain direct read. Recursion only pays when the context genuinely does not fit. Reach for it as a last resort, not a default. That comparison is from our OOLONG reproduction, and the paired run data is public: huggingface.co/datasets/Rickesh/rlm-oolong-reproduction.
Silent truncation is the dangerous failure. Our serving layer auto-sized context to the prompt up to a ceiling, then quietly fell back: 30,021 tokens passed intact, 50,000 and 70,000 both clipped to exactly 16,387, with no error and a confident answer from the fragment. Compare processed-token counts against what you sent, and make the mismatch fatal.
Substring scoring manufactures success. One run reported correct because the model dumped raw corpus rows and the dump happened to contain the gold label. Require the declared answer form and reject anything that looks like regurgitated input.
Report what you could not parse. unparsed_lines=3 is the number that makes the rest of the output trustworthy. A harness that silently drops what it cannot read will happily report a clean answer over half the data.
A partial sweep must refuse to answer. If any segment fails, every key is undercounted. Ours exits non-zero rather than printing a number that looks fine.
The engine is C++17 with zero third-party dependencies, and the planning and reduction stages are covered by 61 tests that need no GPU, no network and no tokens — because once the model is only doing extraction, everything else is ordinary code you can actually test.
Top comments (0)