"V cache quantization requires flash_attn" — the llama.cpp error that quietly halves your context window

I did not meet this error while debugging a crash. I met it while writing a
calculator.

llama_context: quantized V cache requires flash_attn to be enabled

There is a second wording, thrown as an exception a little later in startup and
surfacing as failed to initialize the context:

quantized V cache was requested, but this requires Flash Attention

and a third, older one — V cache quantization requires flash_attn — which is
no longer in the tree but is what most of the search results still show you,
because most of the world runs llama.cpp through something that vendors a build
from six months ago.

All three read like a configuration nag: you asked for one thing, turn on the
other thing, move along. That framing is why almost nobody asks the interesting
question, which is why those two settings are welded together. The answer is a
memory-layout decision several levels below the flag you typed, and it is worth
knowing, because it tells you precisely which half of the cache you can still
quantize when flash attention isn't available to you.

But first the calculator, because that is how I got here and it is the part that
cost me real time.

The number I actually needed

I ship an offline desktop app that runs llama.cpp locally. Users have whatever
machine they have. Before the app picks a context window it has to answer one
question honestly:

window = (RAM - model weights - reserve) / cost_per_token_of_KV

Three of those four terms are easy. RAM you ask the OS. Weights you take from
the file. The reserve is a policy number you choose — mine is deliberately fat,
because on macOS unified memory, overshooting what the GPU can wire does not
politely hand you an allocation failure. It panics the kernel. I have the scars
and the commit history.

The fourth term is where I went wrong. cost_per_token_of_KV looks like
something you compute from model metadata: layers, KV heads, head dimension, two
tensors, two bytes each. Multiply, done. Every context-size calculator on the
internet does exactly this.

On the model I care about it was wrong by a factor of four.

That model is a Gemma-family 12B, and Gemma interleaves its attention: a
minority of layers attend over the full context, the rest run a short sliding
window that does not grow with n_ctx at all. Metadata math doesn't know that.
It multiplies one per-layer cost by every layer and confidently describes a
model that does not exist. On a 24 GB box a 4x overestimate is not a rounding
error — it is the difference between offering the user 16k of context and
telling them their machine can manage 4k.

Metadata describes the model. I needed a number that describes the allocation.
Those are different things, and only one of them gets printed at runtime.

So I stopped computing and started booting

llama.cpp already knows the answer. It says it at startup, in a line most people
scroll past on the way to the prompt:

llama_kv_cache: size =  160.00 MiB (  4096 cells,   8 layers,  1 seqs), K (f16): ...

Bytes, cells, layers. No metadata, no architecture assumptions — this is the
allocator reporting what it actually took. Note the layer count: eight, on a
model with far more layers than that. The interleaved model gets more than one
of these lines, one per cache, and you want the sum.

So the probe is dumb and reliable: boot the engine with a small context, parse
its own log, divide, kill it. Two seconds, no inference, nothing downloaded. My
app does this once per model on first run and caches the result.

One trap, and it's why the probe takes a context argument instead of using the
smallest number that loads. Until late 2025, llama.cpp padded the cache size
itself, and the multiple depended on flash attention:

// the FA kernels require padding to avoid extra runtime boundary checks
return cparams.flash_attn ? 256u : 32u;

That's gone — PR #16812
removed KV cache size padding in October 2025, and the only rounding left is on
the per-graph n_kv view, a flat 256 whether or not flash attention is on. Good
news you should not rely on, because the llama.cpp inside your LM Studio or your
ollama is quite possibly older than that commit. Probe well above the padding
floor regardless. It costs nothing, and it's the difference between measuring a
model and measuring a rounding rule with beautiful precision.

The probe was lying too

First real run, the probe reported:

368,640 bytes per token.

Meanwhile the production config, same machine, same model, was demonstrably
holding a window that this number says is impossible. So I read the production
allocation directly:

182,784 bytes per token.

Ratio: 2.02x. My careful runtime measurement was off by more than the metadata
error I had built it to fix — same direction, same machine, same model.

The reason is embarrassing and took ten minutes to find. The probe booted the
engine with default flags: f16 K, f16 V, flash attention off. The app boots it
with q8_0 K, q8_0 V, flash attention on. I had measured, very rigorously, a
configuration I do not ship.

The fix is one line of "pass the same flags." The lesson outlived the fix,
because the arithmetic doesn't land where you'd guess. f16 is 2 bytes per value;
q8_0 is 34 bytes per 32 values, or 1.0625. That predicts a 1.88x gap. I measured
2.02x. The remainder comes from layout and padding differences that ride along
with flash attention, which appear nowhere in the dtype arithmetic and which I
would never have thought to include.

Which is the argument for measuring, made better by how nearly I missed it: had
the gap come out at exactly 1.88x, I'd have hardcoded the ratio and shipped a
formula that drifts silently every time llama.cpp changes its padding.

The confirmation was that with the honest number, the formula reproduces the
16,384-token ceiling my app had been running for months — a figure originally
arrived at by hand, by trial, by someone getting tired of crashes. The
measurement agreed with the scar tissue. That's when I believed it.

	probe under defaults	probe under production flags
K cache	f16	q8_0
V cache	f16	q8_0
flash attention	off	on
measured cost	368,640 B/token	182,784 B/token
window from the same ~2.8 GiB KV budget	8,123 tokens	16,384 tokens

Same 24 GB, same model, same afternoon. One of those rows is a product decision
and the other is a support ticket with a head start.

Why quantized V needs flash attention at all

Now the part that sent me into the source, and the part I couldn't find written
down anywhere.

Last time I wrote about llama.cpp, the internet's confident answer to my problem
was "it's the quantized KV cache," and it wasn't. So it seems only fair
that I now explain what the quantized KV cache is legitimately guilty of.

The classic, non-flash attention path ends like this:

ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq);

ggml_mul_mat reduces over ne[0], the first dimension. So V has to arrive
with the KV-position axis as its row axis — that is, V transposed. llama.cpp
could transpose on the fly, and the code comments explain why it doesn't: that
means a ggml_cont(ggml_transpose(...)) over the whole cache every single step.
So V is stored pre-transposed instead, behind a flag declared exactly like this:

bool v_trans = true; // the value tensor is transposed

and set, at every single cache construction site, to literally
!cparams.flash_attn. That flag is the entire story. Flash attention on, V is
stored naturally, because ggml_flash_attn_ext wants it the other way round.
Flash attention off, V is stored transposed.

Now consider what transposed storage does to a write. Appending one token in the
natural layout means writing one contiguous row of n_embd_v_gqa values. In the
transposed layout those same values scatter: one element into each of
n_embd_v_gqa different rows, striding by kv_size.

llama.cpp expresses that scatter with ggml_set_rows, and the transposed branch
does something that looks unhinged until you see why — it reshapes the
destination so that every row is exactly one element long:

// in this branch the v_idxs are constructed in such a way that each row is a single head element
ggml_tensor * v_view = ggml_reshape_2d(ctx, v, 1, ggml_nelements(v));
v_cur = ggml_reshape_2d(ctx, v_cur, 1, ggml_nelements(v_cur));
return ggml_set_rows(ctx, v_view, v_cur, v_idxs);

And ggml_set_rows quantizes one whole row at a time — it calls the type's
from_float(src, dst, nc) with nc equal to the row length. With nc == 1 and
q8_0's 32-element blocks there is simply nothing to quantize:
quantize_row_q8_0 asserts that the count is a multiple of 32. ggml_set_rows
also hard-asserts that its source is F32 or F16.

The operation you'd need instead is read the 32-element block, dequantize it,
replace one value, recompute the shared scale, requantize. ggml does not have
that operation, and you would not want it in the hot path anyway — every new
token would rewrite a block whose scale then shifts underneath values written
several tokens ago.

So it isn't a policy or an unfinished feature. There is no quantized write path
in ggml with sub-block granularity, and the non-flash-attention V layout offers
nothing but sub-block writes.

K is a different tensor with a different fate. K is never transposed. Its
update writes whole rows of n_embd_k_gqa values, one row per token, contiguous
and block-aligned by construction — the same code with or without flash
attention. And the non-FA path consumes it as ggml_mul_mat(ctx0, k, q), where
a quantized first operand is the ordinary, thoroughly supported case.

There is no guard anywhere in llama.cpp rejecting a quantized type_k without
flash attention. The only type_k check is gated on flash attention not being
disabled, and all it verifies is that the head dimension divides evenly by the
block size — a constraint that exists because the FA path views K per-head,
while the non-FA path only ever needs whole rows.

Which means the workaround people trade in the issue threads — drop -ctv q8_0,
keep -ctk q8_0 — isn't folklore. It falls straight out of the layout.

What changed in 2026, and who this actually bites

Flash attention is no longer a boolean. Since
PR #15434 (merged 30 August
2025) it's a tri-state, and the default is AUTO:

enum llama_flash_attn_type {
    LLAMA_FLASH_ATTN_TYPE_AUTO     = -1,
    LLAMA_FLASH_ATTN_TYPE_DISABLED = 0,
    LLAMA_FLASH_ATTN_TYPE_ENABLED  = 1,
};

On the command line that's -fa on|off|auto. And in AUTO, the engine resolves
the conflict for you rather than complaining about it:

if (ggml_is_quantized(params.type_v) && params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_ENABLED) {
    if (params.flash_attn_type == LLAMA_FLASH_ATTN_TYPE_AUTO) {
        LLAMA_LOG_INFO("%s: enabling flash_attn since it is required for quantized V cache\n", __func__);
        params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
    }
    if (params.flash_attn_type == LLAMA_FLASH_ATTN_TYPE_DISABLED) {
        LLAMA_LOG_ERROR("%s: quantized V cache requires flash_attn to be enabled\n", __func__);
        return nullptr;
    }
}

So on a current build, typing -ctv q8_0 and touching nothing else never
produces the error. You get the info line and a working model.

Which means, in 2026, essentially everyone who does hit this had flash
attention turned off by something. There are three somethings.

You turned it off. Someone told you flash attention was unstable on your
backend, you set -fa off, and the quantized V flag stayed in your config from
an earlier experiment. Own goal, thirty-second fix, and honestly the nicest
version of this problem to have.

The model forced it off. Grok is hardcoded to disable flash attention
(flash_attn is not compatible with Grok - forcing off) and that happens
before the quantized-V check. So the state arriving at the check is DISABLED,
not AUTO, and you get the hard error rather than the friendly promotion. That is
exactly ollama#15043 — "when
flash attention is not supported, quantized KV cache should be disregarded
instead of aborting the model run," which is a reasonable request phrased with
impressive restraint.

Your backend turned it off. This is the big one.
LM Studio bug tracker #1943:
the Vulkan runtime 2.15.0 silently force-disables flash attention while the
quantized KV config stays exactly as it was, so a setup that loaded yesterday
fails today, and rolling back to 2.14.4 fixes it. llama.cpp itself can also drop
flash attention late, during graph resolution, when the FA node ends up on a
device that can't take it — it logs ... not supported, set to disabled and
then the exception fires after the fact, which is why one of the two error
strings arrives suspiciously late in startup.

On Apple Silicon the same family shows up as
ggml-org/llama.cpp#21450:
Metal fails on mixed quantized KV when flash attention is unavailable, while
uniform q4_0/q4_0 and f16/f16 load fine — which is a good reminder to
keep K and V symmetric. That code is moving, too: as recently as 20 August 2026,
Metal gained a pass that dequantizes quantized KV to F16 before flash attention
(#27390). Pin your build if
you're measuring.

The common thread is that the error names neither the model nor the backend.
It's why those threads are full of people insisting they never disabled flash
attention. They didn't.

What to do about it

Don't disable flash attention while asking for a quantized V cache — and if
you didn't disable it yourself, find out what did. That's now the most common
route to this error, and the message really ought to say "your Vulkan runtime
made this decision for you."

If flash attention genuinely isn't available, quantize K only.
--cache-type-k q8_0 with V left at f16 works without flash attention,
because K lives in the layout that quantizes cleanly. That keeps about half the
saving — both halves quantized puts the cache at roughly 53% of f16, K alone at
about 77% — and, more valuable than the bytes, the engine starts. A partial win
that boots beats a total win that aborts.

Measure the per-token cost under the flags you ship. Not from metadata,
which describes a model rather than an allocation and overshot mine by 4x. Not
under default flags, which cost me a clean 2.02x. Boot the engine the way your
users will boot it, read the line it prints, divide.

The probe is about forty lines of bash — two boots, one awk, no models
bundled, bring your own GGUF. It's in the repo: https://github.com/JackYU96/v-cache-requires-flash-attn.