DEV Community: Jasur Yuldoshev

Streaming returned 0 tokens and llama_decode died: asyncio.wait_for was killing my generator

Jasur Yuldoshev — Tue, 11 Aug 2026 07:30:09 +0000

Short prompts streamed fine. Anything document-sized came back empty: the route
logged done (~0 chars streamed), the UI showed a canned fallback line, and if
anything retried on the same instance the whole backend went down with

llama_decode: failed to decode, ret = -3
GGML_ASSERT: tensor buffer not set

I spent two days on the runtime. The runtime was fine.

What everyone tells you to look at

Search that symptom and you get one answer, from GitHub issues, from forum
threads, and — I checked while writing this — from the search engine's own
summary: your KV cache is too small, or your quantized KV cache needs flash
attention, or llama-cpp-python is broken again. Increase n_ctx. Drop the
batch. Turn off the quantized cache and go back to f16, the safe side.

It is a good story. It fits the evidence: only big prompts die, big prompts use
more KV, therefore KV. I believed it for a day and a half.

The measurements

Eventually I stopped reasoning and started booting. One configuration per boot,
same 17k-token prompt, same model (Gemma-4-12B-Q4), M4 Pro with 24 GB:

flash attention	KV type	n_ctx	result
on	`q8_0`	32k	streams fine, 259 chunks
on	`f16`	32k	`llama_decode -3`, no tokens
off	`f16`	32k	`llama_decode -3`, no tokens

Read that table twice, because it says the opposite of the advice.

The "risky" configuration — flash attention plus a quantized key/value cache,
the one every thread warns you about — is the only one that worked. The "safe
side" I was being told to retreat to is the broken one. f16 KV simply does not
fit a 32k sliding-attention window on a 24 GB box, so retreating there swaps a
bug you can fix for an out-of-memory you cannot.

That table killed the KV theory. It also meant I had been tuning the wrong
component for a day and a half, which is a specific kind of annoying.

What it actually was

The route streams tokens to the client and sends a heartbeat while it waits, so
the client's idle timer stays fed. The waiting looked like this:

chunk = await asyncio.wait_for(stream_aiter.__anext__(), timeout=6)

asyncio.wait_for does not merely stop waiting when the timeout expires. It
cancels the thing it was waiting on. The coroutine here is __anext__() of an
async generator, so cancelling it does not cancel one step — it kills the
generator. The next __anext__() on a dead generator raises
StopAsyncIteration, which to the async for above it is indistinguishable
from a model that finished with nothing to say.

So the route did exactly what it was written to do: the stream ended, zero
characters had arrived, it logged that honestly and served the fallback.

Meanwhile llama.cpp was still inside llama_decode, holding a context that now
belonged to nobody. Any later decode on that instance walked into the torn state
and hit the GGML_ASSERT, which does not raise — it aborts the process. That is
why the crash looked like a runtime crash: by the time it happened, my bug was
several seconds in the past.

And the reason only long prompts died is the least mysterious part of the whole
story. Prefill on a document-sized prompt takes longer than six seconds.
Heartbeat fires, wait_for cancels, generator dies — before the model has
produced its first token. Short prompts finish prefill inside one heartbeat and
never meet the bug.

A keep-alive that kills the thing it is keeping alive. I have written better
code.

The fix

Keep one task alive across heartbeats and poll it with asyncio.wait, which
returns on timeout and leaves the task running:

pending = asyncio.create_task(stream_aiter.__anext__())
while True:
    done, _ = await asyncio.wait({pending}, timeout=6)
    if not done:
        yield heartbeat()          # the task is still alive, still prefilling
        continue
    try:
        chunk = pending.result()
    except StopAsyncIteration:
        break                      # a real end, not a cancelled one
    yield chunk
    pending = asyncio.create_task(stream_aiter.__anext__())

That is the whole difference. wait_for cancels; wait does not. One of them
is a timeout, the other is a kill switch with a timeout-shaped name.

How to tell whether it is you or the runtime

If you are staring at an empty stream right now, this ordering would have saved
me most of two days:

Run the same prompt without streaming. If non-stream produces text, the model, the weights, the KV config and the context size are all fine. You have a plumbing bug. I had this evidence on day one and explained it away.
Grep your own code for wait_for anywhere near an async generator. Also async_timeout, also any framework middleware with a request timeout. All of them cancel.
Correlate with prompt length, not prompt content. "Only big prompts" says something takes too long, which points at a timer, not at a tensor.
Change one thing per boot. Two of my configurations differed by two variables and told me nothing; the table above is boring precisely because each row moved one.
Distrust the safe-sounding fallback. f16 KV was the retreat everyone recommended, and on this machine it is strictly worse than the configuration it was supposed to rescue me from. Measure your own box.

What I would say to the version of me on day one

The symptom appeared in the model layer, so I searched in the model layer, and
the internet had a confident, popular, wrong answer waiting there. Nothing about
llama_decode -3 points at an await in a web route thirty files away.

The thing that finally broke it open was the least clever step available: stop
theorising, boot once per configuration, write down what happened. The table
took an afternoon and ended the argument. The two days before it were spent
being smart.

Config, for anyone matching symptoms: llama-cpp-python 0.3.33, Metal, M4 Pro
24 GB, Gemma-4-12B-Q4, 32k context, K and V both q8_0 — keep them symmetric,
a q8/q4 mix with flash attention crashes on Metal for real, and that one is not
a heartbeat.

Hot-swapping GGUF models kernel-panicked my M4 Mac: wired memory, llama.cpp, and why we restart the server instead

Jasur Yuldoshev — Sat, 18 Jul 2026 13:43:17 +0000

I shipped a model switcher last week. A settings screen, two local GGUF models on disk, a "Make active" button. I clicked it, watched a spinner for about ninety seconds, and then my Mac rebooted.

Not the app. The Mac.

panic(cpu 0 caller 0xfffffe004ac0433c): watchdog timeout: no checkins
from watchdogd in 93 seconds (299 total checkins since monitoring last enabled)
...
Compressor Info: 12% of compressed pages limit (OK) and 8% of segments
limit (OK) with 7 swapfiles and OK swap space

A kernel panic is the operating system filing a formal complaint. This particular flavor — a watchdog timeout — is not about my code crashing. It means userspace as a whole stopped responding, and after 93 seconds of silence the kernel concluded the machine was beyond saving and pulled the plug itself.

My settings button did that.

What the watchdog actually saw

watchdogd is a tiny daemon with one job: telling the kernel "userspace is still alive" every few seconds. It doesn't do anything heavy. For it to miss checkins for a minute and a half, the system has to be starved so badly that a trivial process can't get scheduled or can't allocate a page.

The panic log has the tell: 7 swapfiles, and a memory compressor reporting "OK" with plenty of headroom. The machine spent its final minutes frantically paging out everything that could be paged. It wasn't enough — because the memory that mattered couldn't be paged at all.

Two models, one process

The arithmetic is embarrassingly simple in hindsight. My app runs llama.cpp in a Python sidecar (llama-cpp-python 0.3.33), all layers on Metal, on a 24 GB Apple Silicon machine. The active model was a 12B at Q4_K_M — 7.4 GB of weights, call it ~9 GB resident with the KV cache. The user switches to a 14B at Q5_K_M — 9.8 GB of weights, ~11 GB resident.

The switch endpoint did the obvious thing:

self._llama = None                      # release the old model
self._llama = Llama(new_path, ...)      # load the new one

Drop the reference, load the replacement. It works in every tutorial, because every tutorial has one model and enough RAM.

Here's what actually happened: 9 GB of "released" old model + 11 GB of new model loading + macOS + my app + a browser, on a 24 GB machine. And the crucial detail: model weights on Metal are wired memory — physical pages pinned so the GPU can address them. Wired pages don't swap. They don't compress. When wired allocations eat the machine, the kernel has nothing left to steal, userspace grinds to a halt, and the watchdog does what watchdogs do.

"Released" is not "returned"

That self._llama = None line does less than it appears to. On our stack — llama-cpp-python 0.3.33, Metal backend — we had already observed this once in a different corner: our idle unloader drops the model object after a few minutes of inactivity, and the process footprint... doesn't come down. The Python object dies; the wired pages stay attached to the process. The only event we've found that reliably returns that memory to the OS is process exit.

I won't claim this is a law of nature. Maybe an explicit close() at exactly the right moment behaves better in your version, maybe a future release fixes it. But we measured ours, twice, on the machine that matters: memory comes back when the process dies, and not before. If your swap design depends on the old model's memory being available for the new one, you're betting your users' machines on a deallocation you don't control.

The fix is boring, and that's the point

The switch button no longer touches the loaded model at all. It writes the chosen model path to a small state file and returns {"restart_required": true}. The app then restarts the sidecar process: the old process dies (taking every wired byte with it, guaranteed, by the only mechanism that guarantees it), the new process reads the state file and loads the chosen model.

Cost to the user: 10–20 seconds of "Restarting the server…" instead of occasionally costing them their entire machine. I'll take that trade every day of the week.

Three traps between me and the boring fix

1. terminate() doesn't wait. Signalling the old process and immediately launching the new one reintroduces the exact bug through a different door — for a few seconds both processes hold their models. You have to wait for actual death: SIGTERM, poll until the process is gone, SIGKILL after a deadline (a load stuck inside C code never reaches a signal handler). Only then launch.

2. The port lies after death. Our supervisor probes the sidecar's port before launching, with a plain bind(). The connections a just-killed process leaves in FIN_WAIT_2 make that probe fail — port "busy" — so the supervisor helpfully relocated to the next port on every single swap. The server itself binds with SO_REUSEADDR and retakes the port without complaint. The probe now does the same, plus a connect() check — because SO_REUSEADDR alone will happily bind 127.0.0.1 right over someone else's wildcard listener, and shadowing a stranger's server is worse than moving.

3. Gate by total RAM, not available. We added a pre-flight check: refuse to load a model that can't fit. First version used available memory. Wrong metric — at swap time the old model is still resident, so "available" is tiny and the gate refused every legitimate swap. Whether a model fits this machine is a property of total RAM (the weights are wired; the OS will evict everything else to make room). "Available" is only good for a soft warning: close some apps, this will be tight.

If you're building a model switcher

Assume wired model memory returns on process exit and at no other time. Design the swap as a restart.
Wait for the old process to actually die before starting the new one. Poll, then SIGKILL. No overlap, ever.
Probe ports the way your server binds them (SO_REUSEADDR), and verify with connect(), or enjoy your app quietly migrating ports.
Hard-refuse models by total RAM. Warn by available. Never block on available.
If the refusal message says "needs ≥N GB", compute N from the actual numbers. Users on a 24 GB machine reading "requires ≥24 GB" will have questions.

The repro script and the full panic log are here: https://github.com/JackYU96/swap-models-restart-process

The Mac survived. The hot-swap didn't.

Your Hugging Face download isn't stuck — you're being rate-limited

Jasur Yuldoshev — Mon, 13 Jul 2026 10:39:24 +0000

I killed a perfectly healthy 2 GB model download three times before I understood what was happening. Each time the same picture: the first 100–200 MB fly by in about a minute, then the progress bar freezes at 0 B/s. Not slow — zero. A minute passes. Five. Ten. Any reasonable person concludes the download is dead, kills the process, and tries again. And again the first 200 MB arrive instantly, and again everything stops.

The download was fine. I was being rate-limited, and everything about how that presents itself is designed — unintentionally, I assume — to convince you it's a hang.

The part where I blamed the VPN

My setup at the time went through a VPN, so naturally the VPN got blamed first. It's the obvious suspect: flaky route, dropped connection, MTU weirdness, pick your favorite.

But the evidence didn't fit, and it took me embarrassingly long to notice. The first burst always came through at full speed. A broken network path doesn't hand you 200 MB in a minute and then die at exactly the same point every retry. Broken networks are random. This was punctual. Whatever was stopping the download lived on the other end and had a policy about it.

That's the diagnostic worth remembering, because it applies to a lot more than Hugging Face: a fast start followed by a consistent stall is a limiter, not a failure. Random is hardware; punctual is policy.

The warning everyone scrolls past

Once I stopped blaming infrastructure and read my own logs from the top, the answer was sitting in plain sight, printed once at the very beginning:

You are sending unauthenticated requests to the HF Hub.
Please set a HF_TOKEN to enable higher rate limits and faster downloads.

It scrolls by in the first second, right before hundreds of progress-bar updates bury it. It reads like boilerplate — every tool prints some variation of "log in for a better experience", and we've all learned to ignore that sentence shape. But this one is literal.

Since then Hugging Face has actually published official numbers, and they explain the shape of what I saw. All quotas run in fixed 5-minute windows — which is precisely why the pattern is burst, wall, burst. File downloads ("resolver" requests) get 3,000 requests per window for anonymous users, and a big model is nowhere near one request: multiple files, ranged chunks, retries and redirects all count. Blow through the window and the Hub answers 429 with a RateLimit header saying exactly how long until reset.

And here's the detail that turns a rate limit into a "hang": recent huggingface_hub (1.2+) reads that header and silently sleeps until the window resets, then retries. Older versions sit in exponential backoff. Either way, what you see is a progress bar frozen at 0 B/s while the client obediently waits out its penalty. Nothing is printed. It's the correct behavior, and it looks exactly like a dead download. For the model I was pulling — about 2.2 GB, stored as both safetensors and pytorch bins, so effectively downloaded twice — that turned an expected 3–5 minutes into 15–25.

One more trap hiding in there: the anonymous quota is shared per IP address. Behind an office NAT, a university network or a busy CI runner, you're splitting those 3,000 requests with everyone else on the same address. That's how you get "works from home, stalls at work" — and, to be fair, the reverse mystery too: some anonymous users pull terabytes and never see a stall. Whether you hit the wall depends on who you share an IP with. It can even get sillier than a slow download: there's a llama.cpp issue where hitting the limit made the client unable to use a model it had already downloaded, because it checked freshness against the Hub before loading the cache.

How to check whether it's alive

Before killing anything, look at the cache. Hugging Face downloads land in content-addressed blobs:

ls -lh ~/.cache/huggingface/hub/models--<org>--<name>/blobs/

There'll be a file ending in .incomplete. Note its size, wait a few minutes, look again. Flat, then a jump of a few hundred megabytes, then flat again — that's the window cycle, and the process (sitting quietly in sleep/IO-wait, not spinning CPU) will eventually get there. And because blob names are content hashes, partial files survive a restart — resume actually resumes.

The fixes

The token. A free account moves you from the shared anonymous pool to a per-user quota (and bumps resolvers to 5,000 per window). That per-user part is the real win — nobody else's CI can eat your budget anymore:

hf auth login          # huggingface_hub 1.0+
huggingface-cli login  # older installs
# or just: export HF_TOKEN=hf_...

If this had been the first line of that warning message in bold red, this article wouldn't exist.

The mirror. For public open-weight models there's a fix that needs no account at all:

HF_ENDPOINT=https://hf-mirror.com python your_script.py

hf-mirror.com is a long-running public mirror (still alive and growing as of 2026). It speaks the same protocol, so resume picks up your existing partial blobs. Since you're downloading tensors, not executable code, and integrity is checked against hashes, the trust story is manageable — a judgment call I'd only make for public weights, though, never for anything sensitive.

The speed flag — but check your version. This one changed under everyone's feet. The classic advice — pip install hf_transfer + HF_HUB_ENABLE_HF_TRANSFER=1 — is dead in huggingface_hub 1.0+: the Xet backend replaced it, and the old env var is now silently ignored, which is its own little gotcha-inside-a-gotcha. On current installs the equivalent knob is HF_XET_HIGH_PERFORMANCE=1 (meant for fat pipes and machines with RAM to spare). On pre-1.0 installs, the old hf_transfer advice still applies. Neither raises your rate limit — they just make the bytes flow faster between penalties.

The product-grade conclusion

I hit all of this building a desktop app whose backend pulls models on first run — which means my users would hit it too, on their machines, on their network routes, sharing IPs with strangers, with no idea what an HF_TOKEN is.

So the durable lesson for anything you ship to end users: don't make your users talk to Hugging Face at all. Host the models you depend on yourself — object storage behind a CDN, a manifest with SHA-256 checksums, resumable downloads. First-run should depend on your infrastructure, not on the rate-limit policy of a third party toward an anonymous user you'll never get to debug.

For your own machine, though, the whole fix is one line in your shell profile. Set the token, or set the mirror — and the next time a download sits at 0 B/s, check the .incomplete file before you reach for Ctrl-C. It's probably not dead. It's waiting for a five-minute window to roll over, and unlike you, it knows exactly how long that takes.

My local RAG silently returned nothing: a missing Ollama daemon, and moving embeddings in-process with llama.cpp

Jasur Yuldoshev — Sat, 11 Jul 2026 10:41:08 +0000

I'm building a desktop AI app for non-technical users — the kind of person who double-clicks an icon and expects search to work, and who will never open a terminal in their life. Under the hood it does local RAG: ingest documents, embed them, retrieve on each question. For a while, the embedding step went through Ollama.

The bug that didn't look like a bug

On my machine everything was fine, because on my machine Ollama is always running. Then I ran the flow a real user would hit — fresh login, Ollama not started — and watched the app quietly fall apart:

/ask returned an answer with no retrieved context.
/search returned nothing at all.
Ingestion accepted documents and indexed none of them.

No stack trace. No red banner. The app looked healthy and did nothing useful.

To be clear, this wasn't Ollama being sneaky. Ollama fails loudly — you get connection refused the instant you hit localhost:11434 with nothing behind it. The failure was mine: my error handling caught that exception and turned it into an empty result, and every caller downstream treated "empty" as "no matches." A user asking a perfectly good question got a confident, sourceless answer.

Why it happens: the daemon is a dependency you can't see

The swallowed exception is the easy part to fix. The architectural lesson is the one worth keeping: by routing embeddings through Ollama, I had made a separate background process's liveness a hard requirement for my app to function correctly. That's a fine trade when you are the operator. It's a terrible trade when your user doesn't know a daemon exists, can't tell whether it's running, and certainly won't restart it after a reboot.

For a desktop app aimed at non-technical people, "keep this background service alive or your search silently breaks" is not a requirement you get to impose.

The fix

Load the same model in-process. bge-m3, 1024-dim, normalized — identical to what I was pulling from Ollama, just running inside my own Python backend via sentence-transformers.

Before:

import requests

def embed(texts: list[str]) -> list[list[float]]:
    resp = requests.post(
        "http://localhost:11434/api/embed",
        json={"model": "bge-m3", "input": texts},
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]

After:

from sentence_transformers import SentenceTransformer

_model: SentenceTransformer | None = None

def _get_model() -> SentenceTransformer:
    global _model
    if _model is None:                       # lazy singleton
        _model = SentenceTransformer("BAAI/bge-m3")
    return _model

def embed(texts: list[str]) -> list[list[float]]:
    model = _get_model()
    return model.encode(texts, normalize_embeddings=True).tolist()

encode is blocking and CPU/GPU-heavy, so in a FastAPI backend you don't call it on the event loop. Push it to a worker thread:

import anyio

async def embed_async(texts: list[str]) -> list[list[float]]:
    return await anyio.to_thread.run_sync(embed, texts)

And make health deterministic instead of guessing. A tiny status enum beats a boolean, because "loading a 2 GB model" is a real state that lasts several seconds:

from enum import Enum

class ModelStatus(str, Enum):
    NOT_LOADED = "NOT_LOADED"
    LOADING = "LOADING"
    READY = "READY"
    FAILED = "FAILED"

Now /health can tell the UI exactly where it stands, and the UI can disable search until the answer is READY instead of returning empty nonsense.

Why not a hybrid fallback

The obvious next thought is: keep both. Try Ollama, fall back to in-process if the daemon is down. For embeddings, that is a bug — not a robustness feature.

Ollama serves a quantized GGUF. sentence-transformers runs fp32. Same model name, different numerics, and therefore a different vector space. The two backends do not produce interchangeable vectors.

That matters because your index is written once and queried many times. If some vectors were written by the GGUF path and you query with the fp32 path — or you re-ingest under a different backend than you started with — the cosine distances between them are quietly meaningless. Nothing throws. Retrieval just gets subtly, unpredictably worse, and you'll waste days blaming your chunking or your reranker.

So the rule is one backend per index. If you want to switch backends, you re-embed the whole corpus once, deliberately. You never mix them and hope.

Trade-offs

I'm not going to pretend this is free.

PyTorch is heavy. sentence-transformers pulls in PyTorch — gigabytes on disk. You're trading a daemon dependency for a large Python dependency. This only makes sense if your app already ships a Python backend or sidecar, which mine does.
The first run downloads ~2.2 GB. The bge-m3 weights come from Hugging Face on first use. For real users you bundle them with the app or self-host them, because anonymous HF downloads get throttled once you're past a few hundred megabytes, and "the app hangs on first launch" is a terrible first impression.
fp32 eats RAM. In-process fp32 uses more memory than Ollama's quantized GGUF, and it competes with the LLM for that memory on the same machine. On an 8 GB laptop, that competition is real.

When Ollama is still the right call

If your users are developers running their own stack — people who already have Ollama up, who want bring-your-own-model, who treat the daemon as infrastructure they control — then routing through it is the right design. The daemon stops being a hidden liability and becomes a feature. My users aren't developers, so for me it wasn't.

The code and a longer write-up are here: https://github.com/JackYU96/embed-without-daemon

The app itself isn't launched yet — this is a pattern I settled on while building it, not a shipped product. If you've solved the desktop-embeddings problem a different way, I'd genuinely like to hear it.