The paper: Sleep-time Compute: Beyond Inference Scaling at Test-time
There's a class of LLM cost that is almost pure waste: the reasoning you do again, live, on every query. Ask a chatbot the same question twice and it burns tokens and latency recomputing the same chain of thought both times. For a fixed document — a product page, a codebase, a knowledge base — users keep asking the same predictable questions, and each one triggers a fresh, expensive round-trip.
The "Sleep-time Compute" paper (arXiv 2504.13171) names the fix: flip when you reason. While the system is idle, anticipate the questions a user will actually ask and pre-compute the reasoning chains and answers. At query time, match the live question to the closest pre-computed chain and answer from cache. Their headline: roughly 5x fewer live tokens/latency at equal accuracy, because reasoning is compressible exactly when questions are predictable.
I built context-warmer — a tiny Python CLI that implements this in about 300 lines. The design is two phases.
Sleep (offline). Give it a document and it chunks it, asks DeepSeek to anticipate the N most likely diverse questions a reader would ask, then pre-computes a concise reasoning chain + answer for each. It embeds every (question + answer) pair with local Ollama embeddings and indexes them into a flat cache/ (JSONL + a numpy vector file).
Wake (online). Embed the live question, retrieve the top-k pairs by cosine similarity, and if the best match clears the threshold, answer from cache — zero DeepSeek tokens, ~milliseconds. If the question is genuinely novel, fall back to a normal DeepSeek call and tag it [fresh]. Every ask feeds cumulative stats: hit rate, tokens saved, latency saved.
The measured win (real run, deepseek-v4-flash + nomic-embed-text on a product doc):
| fresh (query-time) | warm (sleep-time) | |
|---|---|---|
| avg latency / question | 1904 ms | 37 ms |
| live LLM tokens (5 Qs) | 123 | 0 |
| cache hit rate | — | 5/5 (100%) |
The detail that makes it work is the judge. To prove accuracy holds rather than hope it does, the benchmark answers the same five questions fresh and from cache, then has DeepSeek score each cached answer against its fresh counterpart. Cached answers won or tied 4/5 and averaged 7.2/10 vs 6.4/10 for the fresh baseline — accuracy holds while latency collapses ~50x on the live path.
The lesson generalizes: any system where the query distribution is narrower than the model's full capability is a candidate. Docs, support, onboarding — pre-compute the common path, keep the model for the tail. It's a focused primitive: one doc, one cache, two commands. The repo is small on purpose — the idea is the interesting part, not the framework.
Try it: pip install-able, one demo doc included, ./demo/run_demo.sh warms it and asks five questions so you can watch the before/after yourself.
Code & more: https://www.dailybuild.xyz/project/232-context-warmer
Top comments (0)