DEV Community

Shridhar Shah
Shridhar Shah

Posted on

The Agent That Answers Before You Ask

#ai #llm #python #architecture

Sleep-time compute: split the budget so a background worker does the predictable thinking while idle — and the user waits far less when they finally ask.

TL;DR: Most agents only think when a request arrives — the user waits and you pay full latency every time. But between sessions there's idle capacity, and many queries are predictable variants of past ones over context that barely changes. Sleep-time compute (Letta; Lin et al., 2025) splits the budget: a background worker pre-answers likely queries while idle, so the foreground serves warm answers instantly and only falls back to a live call on a miss. A freshness check makes sure a changed document never yields a stale answer. In a tiny demo, foreground latency dropped 57% — with novel queries still handled live and stale pre-answers correctly rejected.

Mental model: a prep cook who chops the vegetables before the dinner rush. When orders come in, plating is fast because the prep is already done — and anything that spoiled gets thrown out and re-prepped fresh, never served stale.

The problem: you pay full price at the worst possible moment

Interactive inference spends compute at test time — the exact moment the user is waiting. For one-shot questions over fresh context that's unavoidable. But a huge share of real traffic isn't one-shot: users query the same codebase, the same document set, the same dashboard repeatedly, and the underlying corpus doesn't change between most of those queries. You're re-deriving the same expensive answers on the critical path, over and over.

Meanwhile the infrastructure sits idle between sessions, with nobody waiting.

The pattern: move predictable work to idle time

Sleep-time compute separates compute while serving a query from compute between queries. A background worker runs during idle periods and does two things: distill the standing context into dense summaries, and speculatively pre-answer the queries most likely to be asked next.

Prediction can be simple — the queries you've seen most are the ones you'll likely see again — and each pre-answer is tagged with the source version it was computed against, so freshness can be checked later:

def prepare(self, history, top_n=8):
    for query, _ in Counter(history).most_common(top_n):     # predict likely next queries
        ver = self.corpus.version[query_topic(query)]
        if self.cache.get(query, (None, None))[1] != ver:    # (re)compute if missing or stale
            self.cache[query] = (f"answer[{query} @v{ver}]", ver)
            self.bg_cost += LIVE_COST                        # paid in the background
Enter fullscreen mode Exit fullscreen mode

At request time the foreground checks the cache — and serves a pre-answer only if it's still fresh. A miss or a stale entry falls back to a live call:

def serve(self, query):
    hit = self.cache.get(query)
    if hit and hit[1] == self.corpus.version[query_topic(query)]:  # present AND fresh
        return hit[0], WARM_MS, WARM_COST, "warm"
    return None   # miss or stale -> go live
Enter fullscreen mode Exit fullscreen mode

The result 

Sleep-Time Compute — precompute while idle so the user waits far less
  400 queries, 70% predictable / 30% novel, one source update mid-stream.

                           foreground latency   foreground cost
   answer on demand                  320.0s             8.00$
   with sleep-time                   137.2s             3.40$

   warm hits served instantly : 230/400  (58%)
   stale pre-answers rejected  : 1  (freshness check forced a live call — no wrong answer)
   background cost (amortized) : $0.26  spent while idle, off the critical path
Enter fullscreen mode Exit fullscreen mode

The predictable 70% mostly got answered before the user asked. The novel 30% still went live — as they should. And when a source document changed mid-stream, the stale pre-answers were rejected and recomputed rather than served wrong.

Reality check: the 57% is from the toy model above — directional, not a benchmark. The paper behind it reports ~5× less test-time compute for equal accuracy and ~2.5× lower cost per query when a context is shared across queries — and, crucially, the win tracks how predictable your traffic is (it's pure overhead for genuinely one-shot asks).

Why this is where 2026 is heading

The proven part: the associated research (Letta's write-up; Lin et al.) shows that shifting reasoning to before a query arrives measurably cuts test-time latency and cost when the work is reused. The transferable engineering idea is the split budget: reserve interactive compute for latency-sensitive work, and spend cheap background compute where it can be amortized.

Where it's heading: as agents accumulate persistent memory, idle time stops being wasted. Expect a foreground agent and a cheaper background agent to share memory — the background one consolidating summaries, reflecting over past sessions, and pre-answering likely follow-ups. The same design shows up as "context distillation" and memory consolidation across 2026 agent stacks.

How faithful is this demo?

It's a minimal model of the control flow, not the reasoning: "inference" is a fixed-latency stub, prediction is a frequency count, and freshness is an integer version bump. Real systems predict from richer signals and must guard hard against two failure modes this only gestures at — serving stale answers (freshness must gate every hit) and privacy leakage during background consolidation. And the honest caveat stands: sleep-time compute pays off only when future queries reuse the work — it's pure overhead for one-shot asks.

When not to use this

  • One-shot or unpredictable traffic. If future queries rarely reuse the work, precompute is pure wasted spend.
  • Fast-changing context. If the underlying data churns constantly, pre-answers go stale faster than you can serve them — you'll pay to recompute anyway.
  • No cheap idle capacity. The whole trick assumes background compute is cheaper than blocking a user; if idle time costs the same, there's no arbitrage.

Try it 

python3 demo.py   # standard library only
Enter fullscreen mode Exit fullscreen mode

Sources & further reading

Paper

Engineering blogs

Top comments (0)