Streaming returned 0 tokens and llama_decode died: asyncio.wait_for was killing my generator

Short prompts streamed fine. Anything document-sized came back empty: the route
logged done (~0 chars streamed), the UI showed a canned fallback line, and if
anything retried on the same instance the whole backend went down with

llama_decode: failed to decode, ret = -3
GGML_ASSERT: tensor buffer not set

I spent two days on the runtime. The runtime was fine.

What everyone tells you to look at

Search that symptom and you get one answer, from GitHub issues, from forum
threads, and — I checked while writing this — from the search engine's own
summary: your KV cache is too small, or your quantized KV cache needs flash
attention, or llama-cpp-python is broken again. Increase n_ctx. Drop the
batch. Turn off the quantized cache and go back to f16, the safe side.

It is a good story. It fits the evidence: only big prompts die, big prompts use
more KV, therefore KV. I believed it for a day and a half.

The measurements

Eventually I stopped reasoning and started booting. One configuration per boot,
same 17k-token prompt, same model (Gemma-4-12B-Q4), M4 Pro with 24 GB:

flash attention	KV type	n_ctx	result
on	q8_0	32k	streams fine, 259 chunks
on	f16	32k	llama_decode -3, no tokens
off	f16	32k	llama_decode -3, no tokens

Read that table twice, because it says the opposite of the advice.

The "risky" configuration — flash attention plus a quantized key/value cache,
the one every thread warns you about — is the only one that worked. The "safe
side" I was being told to retreat to is the broken one. f16 KV simply does not
fit a 32k sliding-attention window on a 24 GB box, so retreating there swaps a
bug you can fix for an out-of-memory you cannot.

That table killed the KV theory. It also meant I had been tuning the wrong
component for a day and a half, which is a specific kind of annoying.

What it actually was

The route streams tokens to the client and sends a heartbeat while it waits, so
the client's idle timer stays fed. The waiting looked like this:

chunk = await asyncio.wait_for(stream_aiter.__anext__(), timeout=6)

asyncio.wait_for does not merely stop waiting when the timeout expires. It
cancels the thing it was waiting on. The coroutine here is __anext__() of an
async generator, so cancelling it does not cancel one step — it kills the
generator. The next __anext__() on a dead generator raises
StopAsyncIteration, which to the async for above it is indistinguishable
from a model that finished with nothing to say.

So the route did exactly what it was written to do: the stream ended, zero
characters had arrived, it logged that honestly and served the fallback.

Meanwhile llama.cpp was still inside llama_decode, holding a context that now
belonged to nobody. Any later decode on that instance walked into the torn state
and hit the GGML_ASSERT, which does not raise — it aborts the process. That is
why the crash looked like a runtime crash: by the time it happened, my bug was
several seconds in the past.

And the reason only long prompts died is the least mysterious part of the whole
story. Prefill on a document-sized prompt takes longer than six seconds.
Heartbeat fires, wait_for cancels, generator dies — before the model has
produced its first token. Short prompts finish prefill inside one heartbeat and
never meet the bug.

A keep-alive that kills the thing it is keeping alive. I have written better
code.

The fix

Keep one task alive across heartbeats and poll it with asyncio.wait, which
returns on timeout and leaves the task running:

pending = asyncio.create_task(stream_aiter.__anext__())
while True:
    done, _ = await asyncio.wait({pending}, timeout=6)
    if not done:
        yield heartbeat()          # the task is still alive, still prefilling
        continue
    try:
        chunk = pending.result()
    except StopAsyncIteration:
        break                      # a real end, not a cancelled one
    yield chunk
    pending = asyncio.create_task(stream_aiter.__anext__())

That is the whole difference. wait_for cancels; wait does not. One of them
is a timeout, the other is a kill switch with a timeout-shaped name.

How to tell whether it is you or the runtime

If you are staring at an empty stream right now, this ordering would have saved
me most of two days:

1. Run the same prompt without streaming. If non-stream produces text, the model, the weights, the KV config and the context size are all fine. You have a plumbing bug. I had this evidence on day one and explained it away.
2. Grep your own code for wait_for anywhere near an async generator. Also async_timeout, also any framework middleware with a request timeout. All of them cancel.
3. Correlate with prompt length, not prompt content. "Only big prompts" says something takes too long, which points at a timer, not at a tensor.
4. Change one thing per boot. Two of my configurations differed by two variables and told me nothing; the table above is boring precisely because each row moved one.
5. Distrust the safe-sounding fallback. f16 KV was the retreat everyone recommended, and on this machine it is strictly worse than the configuration it was supposed to rescue me from. Measure your own box.

What I would say to the version of me on day one

The symptom appeared in the model layer, so I searched in the model layer, and
the internet had a confident, popular, wrong answer waiting there. Nothing about
llama_decode -3 points at an await in a web route thirty files away.

The thing that finally broke it open was the least clever step available: stop
theorising, boot once per configuration, write down what happened. The table
took an afternoon and ended the argument. The two days before it were spent
being smart.

---

Config, for anyone matching symptoms: llama-cpp-python 0.3.33, Metal, M4 Pro
24 GB, Gemma-4-12B-Q4, 32k context, K and V both q8_0 — keep them symmetric,
a q8/q4 mix with flash attention crashes on Metal for real, and that one is not
a heartbeat.