A thread on r/LocalLLaMA ran Qwen 3.8 27B with a 524,000-token context window on two RTX 3090s this week, holding 60 to 88 tokens per second. Two consumer GPUs, half a million tokens of context, one box. Five years ago that sentence would have been science fiction.
Every milestone like this revives the same argument: long context kills RAG. If the model can read everything, why build an index? The "RAG Is Simpler Than You Think" discussion on Hacker News (515 points, 218 comments, August 26) pulled the debate back to earth. The interesting question was never retrieval versus context. It is which layer should carry which cost.
What half a million tokens cost per query
Transformer serving is linear in context. Prefill reads the whole prompt, the KV cache keeps keys and values for every token resident in VRAM, and each generated token re-reads all of it. Double the window and you double the memory and the per-token work, on every query, even when the answer needed three paragraphs.
That is the deal the r/LocalLLaMA setup signed up for. Dual 3090s, one query stream, half a million tokens resident. It works, and it is heavy by design.
Retrieval moves the work off the hot path
A retrieval pipeline does the opposite. Build the index once. Per query, fetch the few chunks that matter and read only those. Cost per query scales with what you retrieve, not with what you own.
That is also why the "simpler than you think" argument landed. For lookup-shaped questions, and most agent workloads are lookup-shaped, a small index plus a small read beats carrying half a million tokens through every turn.
The layer nobody budgets for: embedding storage
If retrieval is the plan, the index becomes the artifact you ship, mirror, and back up. The storage math is unforgiving.
A 768-dimension f32 embedding is 3,072 bytes. A million chunks of corpus is about 3.07 GB before you write a single replica. On a server you shrug. On a phone, a robot, or an offline box, you are suddenly choosing between your index and your photos.
vecq is our answer to that layer: training-free scalar quantization (Lloyd-Max centroids, configurable 4/5/6-bit width, default 5-bit) with search built in. Numbers from docs/BENCHMARK.md, measured on real EmbeddingGemma vectors on an aarch64 dev box:
| mode | bytes/vector | recall@10 | ms/query |
|---|---|---|---|
| 5-bit (default) | 642 | 0.979 | 3.21 |
| 4-bit | 514 | 0.958 | 0.89 |
| 4-bit + residual | 1,028 | 0.984 | 1.76 |
At the default width, that million-chunk index drops from 3.07 GB to about 642 MB, 4.78x smaller, with recall 1.6 points from full f32. When you need near-perfect first hits, residual mode holds recall@1 at 0.990 and remains the fastest high-recall scan. There is a third lever as well: Matryoshka-trained models such as EmbeddingGemma degrade gracefully when truncated, so you can quantize only the leading 256 dimensions of a 768-dim model before quantization ever starts.
Build is fast too. A 2k-vector index constructs in 75 ms against 893 ms for HNSW in our benchmark, because there is no graph to build. You quantize, persist a single file, and mmap it.
Trade-offs, honestly
vecq is brute force. There is no graph and no navigational approximation, so scan time grows with corpus size while HNSW stays roughly flat. At 2k vectors the 5-bit scan costs 3.21 ms per query against HNSW's 0.23. That gap is architectural, and it is why vecq targets edge, mobile, and single-box indexes instead of billion-vector server fleets. If you are serving a data center, use a server engine.
Long context genuinely wins some rounds too. One controlled test from a European bank, reported in a March 2026 comparison, put long context 34% more accurate on simple single-document queries. When every query really does need every page of one document, stuffing wins. The failure mode is treating a 524k window as the default container for everything. Cost per query scales with the window you carry, and most workloads are not shaped like that.
Reproduce the numbers
git clone https://github.com/codecoradev/vecq
cd vecq && git checkout develop
cargo run --release -p vecq-bench --bin widths # full 4/5/6-bit + residual matrix
cargo run --release -p vecq-bench --bin real # recall and latency vs exact cosine
Or skip the benchmark and use the crate:
cargo add vecq-core
If this is your first pass on vecq, start with Meet vecq, the lean vector layer. Repo: github.com/codecoradev/vecq. Crate: vecq-core on crates.io.
This article is crossposted with blog.codecora.dev as the canonical source.
Top comments (0)