DEV Community: Jasur Yuldoshev

Your local RAG isn't slow — it re-reads every document on every question

Jasur Yuldoshev — Wed, 26 Aug 2026 10:18:51 +0000

A user opens a project with nine files in it, types the most obvious question
anyone types at a document app — "what are these documents about?" — and waits.

291 seconds.

Then they ask a second question, about one of those documents, and wait again.
Minutes, not seconds. At that point the app has told them something about
itself, and what it has told them is: this model is slow and probably stupid.

The model was neither. It was reading. It read the entire retrieved corpus,
from token zero, for question one. Then it read it again for question two.

Where the 291 seconds went: prefill, not generation

Setup, for symptom matching: an offline desktop app on llama.cpp, M4 Pro with
24 GB, a 14B at Q5, nine files in the project.

Two throughput numbers, both from my own logs:

generation:  ~8 tokens/second     <- normal for a 14B at Q5 on this box
prefill:   ~100 tokens/second     <- also normal

Neither is a bug. The bug is the ratio, and what got multiplied by it. That one
question assembled a 13,773-token prompt, and the turn as a whole pushed
17,373 prompt tokens through the model once you count the service calls it
makes on the side. At a hundred tokens a second, reading dominated writing by
roughly five to one; the profile came out north of 85% prefill.

Divide those numbers yourself and you'll land a few dozen seconds off. Prefill
throughput sags as the context grows, and one "question" is more than one model
call. The shape is the point, not the arithmetic: the machine spent its evening
reading, and the part the user was waiting for — the answer — was cheap.

I want to be precise about the thing I had gotten wrong for a long time,
because I don't think I'm alone in it. I had indexing. Chunks, embeddings, a
vector store, the whole ritual, run once when files are added. What I assumed
that ritual bought me was that the documents were, in some sense, already
read.

It buys nothing of the kind. Retrieval is a table of contents, not a memory. It
finds the right pieces quickly; it does not make the model read them faster,
and it does not make the model remember having read them. Every question ships
fresh text into the context window and the model chews it from the first token,
at prefill speed, every time. In classical RAG, indexing time and reading time
are separate budgets, and only one of them is ever spent in advance.

Three things came out of the logs before I got to the interesting part.

Finding 1: two retrievers, one prompt, ~9,000 duplicated tokens

I run lexical search by default and vector search behind a flag. On the
machines where both are on, both contribute passages to the prompt. They union
their results, deduplicated by chunk id.

Chunk id. Not content. The two engines index with different chunk boundaries,
so the same paragraph arrives as two different ids with substantially the same
text, and the union happily keeps both.

Roughly 9,000 tokens per question were the same passages, twice. Not once
in a while — on every question, for as long as both engines had been on. Nobody
caught it by reading the prompt, because nobody reads the prompt; it is a wall
of text that scrolls past in a debug log and looks exactly like a wall of text
is supposed to look.

Deduplicating on normalized content instead of id took the question from 291
to 150 seconds. That is the least interesting bug here and it was
worth more seconds than anything I did on purpose that week.

Finding 2: the prefix cache hit 3 tokens out of 17,373

llama.cpp keeps a per-slot prompt cache. Send a prompt that shares a prefix
with the last one that slot saw, and it skips prefill for the shared part. In a
chat app that should be most of the system prompt, most of the time.

Mine, trimmed:

slot update_slots: id  0 | task 412 | n_past = 3, cache_tokens = 3, n_prompt_tokens = 13773

Three tokens. Beginning-of-sequence and a bit of chat template. The cache had
never once helped, on any turn, since the feature existed.

The cause was a helper I'd been pleased with. Before retrieval, a small service
call rewrites the user's question into a better search query — resolving "it"
and "that contract" against the conversation. It's a good feature. It ran with
its own system prompt, and it ran into the same slot as the conversation.

Slot caches key on the longest common prefix. Two different system prompts
diverge at token three. So the service call evicted the conversation's cached
prefix, the conversation's next turn evicted the service call's, and the two
took turns doing this forever. The cache was working exactly as designed. It
was caching a conversation that alternated, every single turn, with a
completely different conversation, in the same chair.

Splitting the slots — a pool for conversations, a pool for service calls, never
shared — moved the hit rate from 0% to 45-48%.

Finding 3: the study queue lived in RAM and died with the process

There was already a background "study" pass in the codebase, meant to
pre-summarize documents after import. It kept its queue in process memory.

Close the app, and every unfinished item is gone. Not retried — gone, with no
record that it had been scheduled. And it only ran while the app was open and
otherwise idle, which on a desktop app is a narrow and unreliable window.

Net effect: summaries were essentially never present when a question arrived,
so every question fell back to reading document bodies. A background job that
doesn't survive a restart is a background job that never finishes, because the
user closes the window constantly and does not consider this an unusual thing
to do.

Study at indexing: pay the minutes once per file, not once per question

Here is the reframe the whole thing turned on. Prefill is not a cost you can
optimize away — a 14B reading N tokens has to read N tokens. It is a cost you
can move. The question is whether the model reads a document at question
time, while a human watches a spinner, or at indexing time, when nobody is
waiting.

Three pieces, in order of how cheap they are.

The document passport, ~200 tokens, no model involved

For each file, a small structured record: document type, parties, dates,
amounts, page count, and a table of contents extracted from heading patterns
with a regular expression. Not with the model — with a regex, at import, in
milliseconds.

That's about 200 tokens per document. For a nine-file project, "what are these
documents about?" now has a 1,800-token answer surface where it used to
have 13,773.

The honest limit: heading extraction works on documents that have formatting
and fails flat on an unstructured wall of text. Those fall through to the pass
below.

Map-reduce summaries, in the background

Per file: chunk it, summarize each chunk, summarize the summaries. This costs
exactly the minutes you were paying before — the model still reads the whole
document — except it costs them once, on import, and never again.

The queue rules matter more than the summarization prompt:

it yields to live questions, always; a user typing outranks it instantly
it goes silent on battery
it checkpoints per chunk and resumes after a restart, because see Finding 3
it stores state in the database, not in a variable

The user does pay for this. They pay in fan noise, once per file, at a moment
when they are not staring at a progress bar.

A cascading router that picks the cheapest lane that can answer

Broad question ("what are these documents about", "summarize the project") — answer from passports plus summaries. 1-2k tokens.
Specific question ("what's the termination clause in the lease") — vector retrieval, narrow set of chunks, as before but smaller.
Explicit deep read — full bodies, no shortcuts, and the user knows they asked for the slow path because they had to say so.

Routing is the risky part and I'd rather name the risk than sell around it.
Misroute a specific question into the digest lane and you answer from a summary
that dropped the exact number the user wanted, confidently and wrongly. Two
mitigations, both boring: the router is biased to escalate — if a question
mentions a term that appears in a passport's table of contents but not in the
summary, it goes to chunks — and the app shows which lane answered, so a thin
answer has an obvious "go read it properly" next to it.

What the numbers look like now

	before	after
"what are these documents about?", 9 files	291 s	102 s	2.9x
prompt for that question	13,773 tok	5,709 tok	2.4x
the same question asked verbatim again	minutes	~1 s	exact-match cache
prefix cache hit rate, specific questions	0%	45-48%	slot split

102 seconds is not a good number. It is a much better number, and it is an
honest one: a 14B on a laptop reading five thousand tokens has to read five
thousand tokens, and no amount of architecture argues with that.

The exact-match cache is the cheapest line in the table and it exists because
of a behavior I did not predict. Users re-ask the identical question. They
close the app, come back, and type the same words to see whether it's still
right. Hashing the normalized question plus the resolved context set and
keeping the answer turns that into a second.

Why 100% prefix-cache hits are not on the table

The reusable part of a prompt is the part that doesn't change. System prompt
and passports are stable, so they cache. Retrieved excerpts change with the
question — by construction, since changing them is the entire job of retrieval.

You could force them to cache. Retrieve once per conversation, freeze the
context, and every subsequent turn shares a long identical prefix. I sat with
that for a while and turned it down. The second question in a conversation is
usually about something the first question didn't retrieve; freezing the
context buys cache hits and pays for them in wrong answers. 45-48% is what the
stable prefix is genuinely worth in this layout, and I'd rather report that
number than a better one I bought with accuracy.

One thing that is free: order the prompt by volatility. Stable first
(system prompt, passports), volatile last (excerpts, then the question). Get
that backwards and your hit rate is zero no matter how much of the prompt is
technically stable.

I went and checked what everyone else does

Before claiming any of this was novel I looked at what the neighbors ship: LM
Studio, AnythingLLM, Jan, GPT4All, Open WebUI with Ollama.

All of them chunk and embed at index time. Not one of them precomputes
per-document digests or summaries. The best of them keep a prompt cache, which
helps with the system prompt and does nothing for the retrieved half.

Which means that on "what are these documents about?" — the single most common
opening question a human asks a document app, the one they type before they
type anything else — every one of these re-reads the corpus at full prefill
cost, every time.

I'm supposed to call that a gap in the market. It's really a gap in the default
architecture: the reference RAG design does retrieval at question time and
nothing at index time except embeddings, everybody copied it faithfully, and
the copy is correct. It's just that "correct" and "the user waited five
minutes" are compatible states.

If you're building a local RAG

Measure prefill against generation before you optimize anything. If 85% of
your wall clock is reading, a faster sampler and a smaller quant are noise. The
engine prints both numbers; find the line.

Deduplicate context sources by content, not by id. Two retrievers with
different chunk boundaries will hand you the same paragraph twice and neither
will look wrong in isolation. This was 9,000 tokens a question in my app.

Give service LLM calls their own KV slot. Query rewriting, classification,
title generation — anything with its own system prompt sharing a slot with the
conversation will zero your prefix cache and the logs will still say the cache
is enabled.

Precompute passports and summaries at index time. A ~200-token structured
passport per document, with headings pulled by regex rather than by the model,
answers most broad questions on its own and costs no GPU at all.

Put the study queue in the database. In-memory queues on a desktop app do
not survive contact with users, who close windows. Checkpoint per chunk, resume
on launch, yield to live traffic, stop on battery.

Cache the literal repeats. People re-ask the same question verbatim more
than you'd think. Hash the normalized question plus the resolved context set.

If you want to know whether any of this is worth your afternoon, the arithmetic
is short enough to run before you commit to it: file size, prefill speed,
number of questions, and it tells you how many seconds of pure reading you have
already signed up for. That's how_long_will_my_rag_wait.py — https://github.com/JackYU96/rag-rereads-every-question — in the repo next
to this post — no dependencies, one file, bring your own numbers.

The last time I chased a number like this, the KV cache itself turned out to be
eating it: "V cache quantization requires flash_attn" — the llama.cpp error
that quietly halves your context
window.
Different layer, same lesson. The engine will tell you where your seconds went.
It tells you in a log line that scrolls past at startup, in a field you have
never once looked at.

"V cache quantization requires flash_attn" — the llama.cpp error that quietly halves your context window

Jasur Yuldoshev — Fri, 21 Aug 2026 05:23:22 +0000

I did not meet this error while debugging a crash. I met it while writing a
calculator.

llama_context: quantized V cache requires flash_attn to be enabled

There is a second wording, thrown as an exception a little later in startup and
surfacing as failed to initialize the context:

quantized V cache was requested, but this requires Flash Attention

and a third, older one — V cache quantization requires flash_attn — which is
no longer in the tree but is what most of the search results still show you,
because most of the world runs llama.cpp through something that vendors a build
from six months ago.

All three read like a configuration nag: you asked for one thing, turn on the
other thing, move along. That framing is why almost nobody asks the interesting
question, which is why those two settings are welded together. The answer is a
memory-layout decision several levels below the flag you typed, and it is worth
knowing, because it tells you precisely which half of the cache you can still
quantize when flash attention isn't available to you.

But first the calculator, because that is how I got here and it is the part that
cost me real time.

The number I actually needed

I ship an offline desktop app that runs llama.cpp locally. Users have whatever
machine they have. Before the app picks a context window it has to answer one
question honestly:

window = (RAM - model weights - reserve) / cost_per_token_of_KV

Three of those four terms are easy. RAM you ask the OS. Weights you take from
the file. The reserve is a policy number you choose — mine is deliberately fat,
because on macOS unified memory, overshooting what the GPU can wire does not
politely hand you an allocation failure. It panics the kernel. I have the scars
and the commit history.

The fourth term is where I went wrong. cost_per_token_of_KV looks like
something you compute from model metadata: layers, KV heads, head dimension, two
tensors, two bytes each. Multiply, done. Every context-size calculator on the
internet does exactly this.

On the model I care about it was wrong by a factor of four.

That model is a Gemma-family 12B, and Gemma interleaves its attention: a
minority of layers attend over the full context, the rest run a short sliding
window that does not grow with n_ctx at all. Metadata math doesn't know that.
It multiplies one per-layer cost by every layer and confidently describes a
model that does not exist. On a 24 GB box a 4x overestimate is not a rounding
error — it is the difference between offering the user 16k of context and
telling them their machine can manage 4k.

Metadata describes the model. I needed a number that describes the allocation.
Those are different things, and only one of them gets printed at runtime.

So I stopped computing and started booting

llama.cpp already knows the answer. It says it at startup, in a line most people
scroll past on the way to the prompt:

llama_kv_cache: size =  160.00 MiB (  4096 cells,   8 layers,  1 seqs), K (f16): ...

Bytes, cells, layers. No metadata, no architecture assumptions — this is the
allocator reporting what it actually took. Note the layer count: eight, on a
model with far more layers than that. The interleaved model gets more than one
of these lines, one per cache, and you want the sum.

So the probe is dumb and reliable: boot the engine with a small context, parse
its own log, divide, kill it. Two seconds, no inference, nothing downloaded. My
app does this once per model on first run and caches the result.

One trap, and it's why the probe takes a context argument instead of using the
smallest number that loads. Until late 2025, llama.cpp padded the cache size
itself, and the multiple depended on flash attention:

// the FA kernels require padding to avoid extra runtime boundary checks
return cparams.flash_attn ? 256u : 32u;

That's gone — PR #16812
removed KV cache size padding in October 2025, and the only rounding left is on
the per-graph n_kv view, a flat 256 whether or not flash attention is on. Good
news you should not rely on, because the llama.cpp inside your LM Studio or your
ollama is quite possibly older than that commit. Probe well above the padding
floor regardless. It costs nothing, and it's the difference between measuring a
model and measuring a rounding rule with beautiful precision.

The probe was lying too

First real run, the probe reported:

368,640 bytes per token.

Meanwhile the production config, same machine, same model, was demonstrably
holding a window that this number says is impossible. So I read the production
allocation directly:

182,784 bytes per token.

Ratio: 2.02x. My careful runtime measurement was off by more than the metadata
error I had built it to fix — same direction, same machine, same model.

The reason is embarrassing and took ten minutes to find. The probe booted the
engine with default flags: f16 K, f16 V, flash attention off. The app boots it
with q8_0 K, q8_0 V, flash attention on. I had measured, very rigorously, a
configuration I do not ship.

The fix is one line of "pass the same flags." The lesson outlived the fix,
because the arithmetic doesn't land where you'd guess. f16 is 2 bytes per value;
q8_0 is 34 bytes per 32 values, or 1.0625. That predicts a 1.88x gap. I measured
2.02x. The remainder comes from layout and padding differences that ride along
with flash attention, which appear nowhere in the dtype arithmetic and which I
would never have thought to include.

Which is the argument for measuring, made better by how nearly I missed it: had
the gap come out at exactly 1.88x, I'd have hardcoded the ratio and shipped a
formula that drifts silently every time llama.cpp changes its padding.

The confirmation was that with the honest number, the formula reproduces the
16,384-token ceiling my app had been running for months — a figure originally
arrived at by hand, by trial, by someone getting tired of crashes. The
measurement agreed with the scar tissue. That's when I believed it.

	probe under defaults	probe under production flags
K cache	`f16`	`q8_0`
V cache	`f16`	`q8_0`
flash attention	off	on
measured cost	368,640 B/token	182,784 B/token
window from the same ~2.8 GiB KV budget	8,123 tokens	16,384 tokens

Same 24 GB, same model, same afternoon. One of those rows is a product decision
and the other is a support ticket with a head start.

Why quantized V needs flash attention at all

Now the part that sent me into the source, and the part I couldn't find written
down anywhere.

Last time I wrote about llama.cpp, the internet's confident answer to my problem
was "it's the quantized KV cache," and it wasn't. So it seems only fair
that I now explain what the quantized KV cache is legitimately guilty of.

The classic, non-flash attention path ends like this:

ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq);

ggml_mul_mat reduces over ne[0], the first dimension. So V has to arrive
with the KV-position axis as its row axis — that is, V transposed. llama.cpp
could transpose on the fly, and the code comments explain why it doesn't: that
means a ggml_cont(ggml_transpose(...)) over the whole cache every single step.
So V is stored pre-transposed instead, behind a flag declared exactly like this:

bool v_trans = true; // the value tensor is transposed

and set, at every single cache construction site, to literally
!cparams.flash_attn. That flag is the entire story. Flash attention on, V is
stored naturally, because ggml_flash_attn_ext wants it the other way round.
Flash attention off, V is stored transposed.

Now consider what transposed storage does to a write. Appending one token in the
natural layout means writing one contiguous row of n_embd_v_gqa values. In the
transposed layout those same values scatter: one element into each of
n_embd_v_gqa different rows, striding by kv_size.

llama.cpp expresses that scatter with ggml_set_rows, and the transposed branch
does something that looks unhinged until you see why — it reshapes the
destination so that every row is exactly one element long:

// in this branch the v_idxs are constructed in such a way that each row is a single head element
ggml_tensor * v_view = ggml_reshape_2d(ctx, v, 1, ggml_nelements(v));
v_cur = ggml_reshape_2d(ctx, v_cur, 1, ggml_nelements(v_cur));
return ggml_set_rows(ctx, v_view, v_cur, v_idxs);

And ggml_set_rows quantizes one whole row at a time — it calls the type's
from_float(src, dst, nc) with nc equal to the row length. With nc == 1 and
q8_0's 32-element blocks there is simply nothing to quantize:
quantize_row_q8_0 asserts that the count is a multiple of 32. ggml_set_rows
also hard-asserts that its source is F32 or F16.

The operation you'd need instead is read the 32-element block, dequantize it,
replace one value, recompute the shared scale, requantize. ggml does not have
that operation, and you would not want it in the hot path anyway — every new
token would rewrite a block whose scale then shifts underneath values written
several tokens ago.

So it isn't a policy or an unfinished feature. There is no quantized write path
in ggml with sub-block granularity, and the non-flash-attention V layout offers
nothing but sub-block writes.

K is a different tensor with a different fate. K is never transposed. Its
update writes whole rows of n_embd_k_gqa values, one row per token, contiguous
and block-aligned by construction — the same code with or without flash
attention. And the non-FA path consumes it as ggml_mul_mat(ctx0, k, q), where
a quantized first operand is the ordinary, thoroughly supported case.

There is no guard anywhere in llama.cpp rejecting a quantized type_k without
flash attention. The only type_k check is gated on flash attention not being
disabled, and all it verifies is that the head dimension divides evenly by the
block size — a constraint that exists because the FA path views K per-head,
while the non-FA path only ever needs whole rows.

Which means the workaround people trade in the issue threads — drop -ctv q8_0,
keep -ctk q8_0 — isn't folklore. It falls straight out of the layout.

What changed in 2026, and who this actually bites

Flash attention is no longer a boolean. Since
PR #15434 (merged 30 August
2025) it's a tri-state, and the default is AUTO:

enum llama_flash_attn_type {
    LLAMA_FLASH_ATTN_TYPE_AUTO     = -1,
    LLAMA_FLASH_ATTN_TYPE_DISABLED = 0,
    LLAMA_FLASH_ATTN_TYPE_ENABLED  = 1,
};

On the command line that's -fa on|off|auto. And in AUTO, the engine resolves
the conflict for you rather than complaining about it:

if (ggml_is_quantized(params.type_v) && params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_ENABLED) {
    if (params.flash_attn_type == LLAMA_FLASH_ATTN_TYPE_AUTO) {
        LLAMA_LOG_INFO("%s: enabling flash_attn since it is required for quantized V cache\n", __func__);
        params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
    }
    if (params.flash_attn_type == LLAMA_FLASH_ATTN_TYPE_DISABLED) {
        LLAMA_LOG_ERROR("%s: quantized V cache requires flash_attn to be enabled\n", __func__);
        return nullptr;
    }
}

So on a current build, typing -ctv q8_0 and touching nothing else never
produces the error. You get the info line and a working model.

Which means, in 2026, essentially everyone who does hit this had flash
attention turned off by something. There are three somethings.

You turned it off. Someone told you flash attention was unstable on your
backend, you set -fa off, and the quantized V flag stayed in your config from
an earlier experiment. Own goal, thirty-second fix, and honestly the nicest
version of this problem to have.

The model forced it off. Grok is hardcoded to disable flash attention
(flash_attn is not compatible with Grok - forcing off) and that happens
before the quantized-V check. So the state arriving at the check is DISABLED,
not AUTO, and you get the hard error rather than the friendly promotion. That is
exactly ollama#15043 — "when
flash attention is not supported, quantized KV cache should be disregarded
instead of aborting the model run," which is a reasonable request phrased with
impressive restraint.

Your backend turned it off. This is the big one.
LM Studio bug tracker #1943:
the Vulkan runtime 2.15.0 silently force-disables flash attention while the
quantized KV config stays exactly as it was, so a setup that loaded yesterday
fails today, and rolling back to 2.14.4 fixes it. llama.cpp itself can also drop
flash attention late, during graph resolution, when the FA node ends up on a
device that can't take it — it logs ... not supported, set to disabled and
then the exception fires after the fact, which is why one of the two error
strings arrives suspiciously late in startup.

On Apple Silicon the same family shows up as
ggml-org/llama.cpp#21450:
Metal fails on mixed quantized KV when flash attention is unavailable, while
uniform q4_0/q4_0 and f16/f16 load fine — which is a good reminder to
keep K and V symmetric. That code is moving, too: as recently as 20 August 2026,
Metal gained a pass that dequantizes quantized KV to F16 before flash attention
(#27390). Pin your build if
you're measuring.

The common thread is that the error names neither the model nor the backend.
It's why those threads are full of people insisting they never disabled flash
attention. They didn't.

What to do about it

Don't disable flash attention while asking for a quantized V cache — and if
you didn't disable it yourself, find out what did. That's now the most common
route to this error, and the message really ought to say "your Vulkan runtime
made this decision for you."

If flash attention genuinely isn't available, quantize K only.
--cache-type-k q8_0 with V left at f16 works without flash attention,
because K lives in the layout that quantizes cleanly. That keeps about half the
saving — both halves quantized puts the cache at roughly 53% of f16, K alone at
about 77% — and, more valuable than the bytes, the engine starts. A partial win
that boots beats a total win that aborts.

Measure the per-token cost under the flags you ship. Not from metadata,
which describes a model rather than an allocation and overshot mine by 4x. Not
under default flags, which cost me a clean 2.02x. Boot the engine the way your
users will boot it, read the line it prints, divide.

The probe is about forty lines of bash — two boots, one awk, no models
bundled, bring your own GGUF. It's in the repo: https://github.com/JackYU96/v-cache-requires-flash-attn.

Streaming returned 0 tokens and llama_decode died: asyncio.wait_for was killing my generator

Jasur Yuldoshev — Tue, 11 Aug 2026 07:30:09 +0000

Short prompts streamed fine. Anything document-sized came back empty: the route
logged done (~0 chars streamed), the UI showed a canned fallback line, and if
anything retried on the same instance the whole backend went down with

llama_decode: failed to decode, ret = -3
GGML_ASSERT: tensor buffer not set

I spent two days on the runtime. The runtime was fine.

What everyone tells you to look at

Search that symptom and you get one answer, from GitHub issues, from forum
threads, and — I checked while writing this — from the search engine's own
summary: your KV cache is too small, or your quantized KV cache needs flash
attention, or llama-cpp-python is broken again. Increase n_ctx. Drop the
batch. Turn off the quantized cache and go back to f16, the safe side.

It is a good story. It fits the evidence: only big prompts die, big prompts use
more KV, therefore KV. I believed it for a day and a half.

The measurements

Eventually I stopped reasoning and started booting. One configuration per boot,
same 17k-token prompt, same model (Gemma-4-12B-Q4), M4 Pro with 24 GB:

flash attention	KV type	n_ctx	result
on	`q8_0`	32k	streams fine, 259 chunks
on	`f16`	32k	`llama_decode -3`, no tokens
off	`f16`	32k	`llama_decode -3`, no tokens

Read that table twice, because it says the opposite of the advice.

The "risky" configuration — flash attention plus a quantized key/value cache,
the one every thread warns you about — is the only one that worked. The "safe
side" I was being told to retreat to is the broken one. f16 KV simply does not
fit a 32k sliding-attention window on a 24 GB box, so retreating there swaps a
bug you can fix for an out-of-memory you cannot.

That table killed the KV theory. It also meant I had been tuning the wrong
component for a day and a half, which is a specific kind of annoying.

What it actually was

The route streams tokens to the client and sends a heartbeat while it waits, so
the client's idle timer stays fed. The waiting looked like this:

chunk = await asyncio.wait_for(stream_aiter.__anext__(), timeout=6)

asyncio.wait_for does not merely stop waiting when the timeout expires. It
cancels the thing it was waiting on. The coroutine here is __anext__() of an
async generator, so cancelling it does not cancel one step — it kills the
generator. The next __anext__() on a dead generator raises
StopAsyncIteration, which to the async for above it is indistinguishable
from a model that finished with nothing to say.

So the route did exactly what it was written to do: the stream ended, zero
characters had arrived, it logged that honestly and served the fallback.

Meanwhile llama.cpp was still inside llama_decode, holding a context that now
belonged to nobody. Any later decode on that instance walked into the torn state
and hit the GGML_ASSERT, which does not raise — it aborts the process. That is
why the crash looked like a runtime crash: by the time it happened, my bug was
several seconds in the past.

And the reason only long prompts died is the least mysterious part of the whole
story. Prefill on a document-sized prompt takes longer than six seconds.
Heartbeat fires, wait_for cancels, generator dies — before the model has
produced its first token. Short prompts finish prefill inside one heartbeat and
never meet the bug.

A keep-alive that kills the thing it is keeping alive. I have written better
code.

The fix

Keep one task alive across heartbeats and poll it with asyncio.wait, which
returns on timeout and leaves the task running:

pending = asyncio.create_task(stream_aiter.__anext__())
while True:
    done, _ = await asyncio.wait({pending}, timeout=6)
    if not done:
        yield heartbeat()          # the task is still alive, still prefilling
        continue
    try:
        chunk = pending.result()
    except StopAsyncIteration:
        break                      # a real end, not a cancelled one
    yield chunk
    pending = asyncio.create_task(stream_aiter.__anext__())

That is the whole difference. wait_for cancels; wait does not. One of them
is a timeout, the other is a kill switch with a timeout-shaped name.

How to tell whether it is you or the runtime

If you are staring at an empty stream right now, this ordering would have saved
me most of two days:

Run the same prompt without streaming. If non-stream produces text, the model, the weights, the KV config and the context size are all fine. You have a plumbing bug. I had this evidence on day one and explained it away.
Grep your own code for wait_for anywhere near an async generator. Also async_timeout, also any framework middleware with a request timeout. All of them cancel.
Correlate with prompt length, not prompt content. "Only big prompts" says something takes too long, which points at a timer, not at a tensor.
Change one thing per boot. Two of my configurations differed by two variables and told me nothing; the table above is boring precisely because each row moved one.
Distrust the safe-sounding fallback. f16 KV was the retreat everyone recommended, and on this machine it is strictly worse than the configuration it was supposed to rescue me from. Measure your own box.

What I would say to the version of me on day one

The symptom appeared in the model layer, so I searched in the model layer, and
the internet had a confident, popular, wrong answer waiting there. Nothing about
llama_decode -3 points at an await in a web route thirty files away.

The thing that finally broke it open was the least clever step available: stop
theorising, boot once per configuration, write down what happened. The table
took an afternoon and ended the argument. The two days before it were spent
being smart.

Config, for anyone matching symptoms: llama-cpp-python 0.3.33, Metal, M4 Pro
24 GB, Gemma-4-12B-Q4, 32k context, K and V both q8_0 — keep them symmetric,
a q8/q4 mix with flash attention crashes on Metal for real, and that one is not
a heartbeat.

Hot-swapping GGUF models kernel-panicked my M4 Mac: wired memory, llama.cpp, and why we restart the server instead

Jasur Yuldoshev — Sat, 18 Jul 2026 13:43:17 +0000

I shipped a model switcher last week. A settings screen, two local GGUF models on disk, a "Make active" button. I clicked it, watched a spinner for about ninety seconds, and then my Mac rebooted.

Not the app. The Mac.

panic(cpu 0 caller 0xfffffe004ac0433c): watchdog timeout: no checkins
from watchdogd in 93 seconds (299 total checkins since monitoring last enabled)
...
Compressor Info: 12% of compressed pages limit (OK) and 8% of segments
limit (OK) with 7 swapfiles and OK swap space

A kernel panic is the operating system filing a formal complaint. This particular flavor — a watchdog timeout — is not about my code crashing. It means userspace as a whole stopped responding, and after 93 seconds of silence the kernel concluded the machine was beyond saving and pulled the plug itself.

My settings button did that.

What the watchdog actually saw

watchdogd is a tiny daemon with one job: telling the kernel "userspace is still alive" every few seconds. It doesn't do anything heavy. For it to miss checkins for a minute and a half, the system has to be starved so badly that a trivial process can't get scheduled or can't allocate a page.

The panic log has the tell: 7 swapfiles, and a memory compressor reporting "OK" with plenty of headroom. The machine spent its final minutes frantically paging out everything that could be paged. It wasn't enough — because the memory that mattered couldn't be paged at all.

Two models, one process

The arithmetic is embarrassingly simple in hindsight. My app runs llama.cpp in a Python sidecar (llama-cpp-python 0.3.33), all layers on Metal, on a 24 GB Apple Silicon machine. The active model was a 12B at Q4_K_M — 7.4 GB of weights, call it ~9 GB resident with the KV cache. The user switches to a 14B at Q5_K_M — 9.8 GB of weights, ~11 GB resident.

The switch endpoint did the obvious thing:

self._llama = None                      # release the old model
self._llama = Llama(new_path, ...)      # load the new one

Drop the reference, load the replacement. It works in every tutorial, because every tutorial has one model and enough RAM.

Here's what actually happened: 9 GB of "released" old model + 11 GB of new model loading + macOS + my app + a browser, on a 24 GB machine. And the crucial detail: model weights on Metal are wired memory — physical pages pinned so the GPU can address them. Wired pages don't swap. They don't compress. When wired allocations eat the machine, the kernel has nothing left to steal, userspace grinds to a halt, and the watchdog does what watchdogs do.

"Released" is not "returned"

That self._llama = None line does less than it appears to. On our stack — llama-cpp-python 0.3.33, Metal backend — we had already observed this once in a different corner: our idle unloader drops the model object after a few minutes of inactivity, and the process footprint... doesn't come down. The Python object dies; the wired pages stay attached to the process. The only event we've found that reliably returns that memory to the OS is process exit.

I won't claim this is a law of nature. Maybe an explicit close() at exactly the right moment behaves better in your version, maybe a future release fixes it. But we measured ours, twice, on the machine that matters: memory comes back when the process dies, and not before. If your swap design depends on the old model's memory being available for the new one, you're betting your users' machines on a deallocation you don't control.

The fix is boring, and that's the point

The switch button no longer touches the loaded model at all. It writes the chosen model path to a small state file and returns {"restart_required": true}. The app then restarts the sidecar process: the old process dies (taking every wired byte with it, guaranteed, by the only mechanism that guarantees it), the new process reads the state file and loads the chosen model.

Cost to the user: 10–20 seconds of "Restarting the server…" instead of occasionally costing them their entire machine. I'll take that trade every day of the week.

Three traps between me and the boring fix

1. terminate() doesn't wait. Signalling the old process and immediately launching the new one reintroduces the exact bug through a different door — for a few seconds both processes hold their models. You have to wait for actual death: SIGTERM, poll until the process is gone, SIGKILL after a deadline (a load stuck inside C code never reaches a signal handler). Only then launch.

2. The port lies after death. Our supervisor probes the sidecar's port before launching, with a plain bind(). The connections a just-killed process leaves in FIN_WAIT_2 make that probe fail — port "busy" — so the supervisor helpfully relocated to the next port on every single swap. The server itself binds with SO_REUSEADDR and retakes the port without complaint. The probe now does the same, plus a connect() check — because SO_REUSEADDR alone will happily bind 127.0.0.1 right over someone else's wildcard listener, and shadowing a stranger's server is worse than moving.

3. Gate by total RAM, not available. We added a pre-flight check: refuse to load a model that can't fit. First version used available memory. Wrong metric — at swap time the old model is still resident, so "available" is tiny and the gate refused every legitimate swap. Whether a model fits this machine is a property of total RAM (the weights are wired; the OS will evict everything else to make room). "Available" is only good for a soft warning: close some apps, this will be tight.

If you're building a model switcher

Assume wired model memory returns on process exit and at no other time. Design the swap as a restart.
Wait for the old process to actually die before starting the new one. Poll, then SIGKILL. No overlap, ever.
Probe ports the way your server binds them (SO_REUSEADDR), and verify with connect(), or enjoy your app quietly migrating ports.
Hard-refuse models by total RAM. Warn by available. Never block on available.
If the refusal message says "needs ≥N GB", compute N from the actual numbers. Users on a 24 GB machine reading "requires ≥24 GB" will have questions.

The repro script and the full panic log are here: https://github.com/JackYU96/swap-models-restart-process

The Mac survived. The hot-swap didn't.

Your Hugging Face download isn't stuck — you're being rate-limited

Jasur Yuldoshev — Mon, 13 Jul 2026 10:39:24 +0000

I killed a perfectly healthy 2 GB model download three times before I understood what was happening. Each time the same picture: the first 100–200 MB fly by in about a minute, then the progress bar freezes at 0 B/s. Not slow — zero. A minute passes. Five. Ten. Any reasonable person concludes the download is dead, kills the process, and tries again. And again the first 200 MB arrive instantly, and again everything stops.

The download was fine. I was being rate-limited, and everything about how that presents itself is designed — unintentionally, I assume — to convince you it's a hang.

The part where I blamed the VPN

My setup at the time went through a VPN, so naturally the VPN got blamed first. It's the obvious suspect: flaky route, dropped connection, MTU weirdness, pick your favorite.

But the evidence didn't fit, and it took me embarrassingly long to notice. The first burst always came through at full speed. A broken network path doesn't hand you 200 MB in a minute and then die at exactly the same point every retry. Broken networks are random. This was punctual. Whatever was stopping the download lived on the other end and had a policy about it.

That's the diagnostic worth remembering, because it applies to a lot more than Hugging Face: a fast start followed by a consistent stall is a limiter, not a failure. Random is hardware; punctual is policy.

The warning everyone scrolls past

Once I stopped blaming infrastructure and read my own logs from the top, the answer was sitting in plain sight, printed once at the very beginning:

You are sending unauthenticated requests to the HF Hub.
Please set a HF_TOKEN to enable higher rate limits and faster downloads.

It scrolls by in the first second, right before hundreds of progress-bar updates bury it. It reads like boilerplate — every tool prints some variation of "log in for a better experience", and we've all learned to ignore that sentence shape. But this one is literal.

Since then Hugging Face has actually published official numbers, and they explain the shape of what I saw. All quotas run in fixed 5-minute windows — which is precisely why the pattern is burst, wall, burst. File downloads ("resolver" requests) get 3,000 requests per window for anonymous users, and a big model is nowhere near one request: multiple files, ranged chunks, retries and redirects all count. Blow through the window and the Hub answers 429 with a RateLimit header saying exactly how long until reset.

And here's the detail that turns a rate limit into a "hang": recent huggingface_hub (1.2+) reads that header and silently sleeps until the window resets, then retries. Older versions sit in exponential backoff. Either way, what you see is a progress bar frozen at 0 B/s while the client obediently waits out its penalty. Nothing is printed. It's the correct behavior, and it looks exactly like a dead download. For the model I was pulling — about 2.2 GB, stored as both safetensors and pytorch bins, so effectively downloaded twice — that turned an expected 3–5 minutes into 15–25.

One more trap hiding in there: the anonymous quota is shared per IP address. Behind an office NAT, a university network or a busy CI runner, you're splitting those 3,000 requests with everyone else on the same address. That's how you get "works from home, stalls at work" — and, to be fair, the reverse mystery too: some anonymous users pull terabytes and never see a stall. Whether you hit the wall depends on who you share an IP with. It can even get sillier than a slow download: there's a llama.cpp issue where hitting the limit made the client unable to use a model it had already downloaded, because it checked freshness against the Hub before loading the cache.

How to check whether it's alive

Before killing anything, look at the cache. Hugging Face downloads land in content-addressed blobs:

ls -lh ~/.cache/huggingface/hub/models--<org>--<name>/blobs/

There'll be a file ending in .incomplete. Note its size, wait a few minutes, look again. Flat, then a jump of a few hundred megabytes, then flat again — that's the window cycle, and the process (sitting quietly in sleep/IO-wait, not spinning CPU) will eventually get there. And because blob names are content hashes, partial files survive a restart — resume actually resumes.

The fixes

The token. A free account moves you from the shared anonymous pool to a per-user quota (and bumps resolvers to 5,000 per window). That per-user part is the real win — nobody else's CI can eat your budget anymore:

hf auth login          # huggingface_hub 1.0+
huggingface-cli login  # older installs
# or just: export HF_TOKEN=hf_...

If this had been the first line of that warning message in bold red, this article wouldn't exist.

The mirror. For public open-weight models there's a fix that needs no account at all:

HF_ENDPOINT=https://hf-mirror.com python your_script.py

hf-mirror.com is a long-running public mirror (still alive and growing as of 2026). It speaks the same protocol, so resume picks up your existing partial blobs. Since you're downloading tensors, not executable code, and integrity is checked against hashes, the trust story is manageable — a judgment call I'd only make for public weights, though, never for anything sensitive.

The speed flag — but check your version. This one changed under everyone's feet. The classic advice — pip install hf_transfer + HF_HUB_ENABLE_HF_TRANSFER=1 — is dead in huggingface_hub 1.0+: the Xet backend replaced it, and the old env var is now silently ignored, which is its own little gotcha-inside-a-gotcha. On current installs the equivalent knob is HF_XET_HIGH_PERFORMANCE=1 (meant for fat pipes and machines with RAM to spare). On pre-1.0 installs, the old hf_transfer advice still applies. Neither raises your rate limit — they just make the bytes flow faster between penalties.

The product-grade conclusion

I hit all of this building a desktop app whose backend pulls models on first run — which means my users would hit it too, on their machines, on their network routes, sharing IPs with strangers, with no idea what an HF_TOKEN is.

So the durable lesson for anything you ship to end users: don't make your users talk to Hugging Face at all. Host the models you depend on yourself — object storage behind a CDN, a manifest with SHA-256 checksums, resumable downloads. First-run should depend on your infrastructure, not on the rate-limit policy of a third party toward an anonymous user you'll never get to debug.

For your own machine, though, the whole fix is one line in your shell profile. Set the token, or set the mirror — and the next time a download sits at 0 B/s, check the .incomplete file before you reach for Ctrl-C. It's probably not dead. It's waiting for a five-minute window to roll over, and unlike you, it knows exactly how long that takes.

My local RAG silently returned nothing: a missing Ollama daemon, and moving embeddings in-process with llama.cpp

Jasur Yuldoshev — Sat, 11 Jul 2026 10:41:08 +0000

I'm building a desktop AI app for non-technical users — the kind of person who double-clicks an icon and expects search to work, and who will never open a terminal in their life. Under the hood it does local RAG: ingest documents, embed them, retrieve on each question. For a while, the embedding step went through Ollama.

The bug that didn't look like a bug

On my machine everything was fine, because on my machine Ollama is always running. Then I ran the flow a real user would hit — fresh login, Ollama not started — and watched the app quietly fall apart:

/ask returned an answer with no retrieved context.
/search returned nothing at all.
Ingestion accepted documents and indexed none of them.

No stack trace. No red banner. The app looked healthy and did nothing useful.

To be clear, this wasn't Ollama being sneaky. Ollama fails loudly — you get connection refused the instant you hit localhost:11434 with nothing behind it. The failure was mine: my error handling caught that exception and turned it into an empty result, and every caller downstream treated "empty" as "no matches." A user asking a perfectly good question got a confident, sourceless answer.

Why it happens: the daemon is a dependency you can't see

The swallowed exception is the easy part to fix. The architectural lesson is the one worth keeping: by routing embeddings through Ollama, I had made a separate background process's liveness a hard requirement for my app to function correctly. That's a fine trade when you are the operator. It's a terrible trade when your user doesn't know a daemon exists, can't tell whether it's running, and certainly won't restart it after a reboot.

For a desktop app aimed at non-technical people, "keep this background service alive or your search silently breaks" is not a requirement you get to impose.

The fix

Load the same model in-process. bge-m3, 1024-dim, normalized — identical to what I was pulling from Ollama, just running inside my own Python backend via sentence-transformers.

Before:

import requests

def embed(texts: list[str]) -> list[list[float]]:
    resp = requests.post(
        "http://localhost:11434/api/embed",
        json={"model": "bge-m3", "input": texts},
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]

After:

from sentence_transformers import SentenceTransformer

_model: SentenceTransformer | None = None

def _get_model() -> SentenceTransformer:
    global _model
    if _model is None:                       # lazy singleton
        _model = SentenceTransformer("BAAI/bge-m3")
    return _model

def embed(texts: list[str]) -> list[list[float]]:
    model = _get_model()
    return model.encode(texts, normalize_embeddings=True).tolist()

encode is blocking and CPU/GPU-heavy, so in a FastAPI backend you don't call it on the event loop. Push it to a worker thread:

import anyio

async def embed_async(texts: list[str]) -> list[list[float]]:
    return await anyio.to_thread.run_sync(embed, texts)

And make health deterministic instead of guessing. A tiny status enum beats a boolean, because "loading a 2 GB model" is a real state that lasts several seconds:

from enum import Enum

class ModelStatus(str, Enum):
    NOT_LOADED = "NOT_LOADED"
    LOADING = "LOADING"
    READY = "READY"
    FAILED = "FAILED"

Now /health can tell the UI exactly where it stands, and the UI can disable search until the answer is READY instead of returning empty nonsense.

Why not a hybrid fallback

The obvious next thought is: keep both. Try Ollama, fall back to in-process if the daemon is down. For embeddings, that is a bug — not a robustness feature.

Ollama serves a quantized GGUF. sentence-transformers runs fp32. Same model name, different numerics, and therefore a different vector space. The two backends do not produce interchangeable vectors.

That matters because your index is written once and queried many times. If some vectors were written by the GGUF path and you query with the fp32 path — or you re-ingest under a different backend than you started with — the cosine distances between them are quietly meaningless. Nothing throws. Retrieval just gets subtly, unpredictably worse, and you'll waste days blaming your chunking or your reranker.

So the rule is one backend per index. If you want to switch backends, you re-embed the whole corpus once, deliberately. You never mix them and hope.

Trade-offs

I'm not going to pretend this is free.

PyTorch is heavy. sentence-transformers pulls in PyTorch — gigabytes on disk. You're trading a daemon dependency for a large Python dependency. This only makes sense if your app already ships a Python backend or sidecar, which mine does.
The first run downloads ~2.2 GB. The bge-m3 weights come from Hugging Face on first use. For real users you bundle them with the app or self-host them, because anonymous HF downloads get throttled once you're past a few hundred megabytes, and "the app hangs on first launch" is a terrible first impression.
fp32 eats RAM. In-process fp32 uses more memory than Ollama's quantized GGUF, and it competes with the LLM for that memory on the same machine. On an 8 GB laptop, that competition is real.

When Ollama is still the right call

If your users are developers running their own stack — people who already have Ollama up, who want bring-your-own-model, who treat the daemon as infrastructure they control — then routing through it is the right design. The daemon stops being a hidden liability and becomes a feature. My users aren't developers, so for me it wasn't.

The code and a longer write-up are here: https://github.com/JackYU96/embed-without-daemon

The app itself isn't launched yet — this is a pattern I settled on while building it, not a shipped product. If you've solved the desktop-embeddings problem a different way, I'd genuinely like to hear it.