For most of this project's life, one rule was untouchable: Flash Attention off. Non-negotiable. Pascal GPUs have no Tensor Cores, and the measured result on my Tesla P40s matched the community's: enabling FA ran about 50% slower. The flag -fa off sat in every start script, and it belonged there.
Today, several of my most-used stacks run with Flash Attention on. On the same GPUs. And they're right to.
What changed
Not the hardware. The reason.
The stacks in question run quantized KV cache (-ctv q8_0), which halves the memory cost of every token of context. llama.cpp requires Flash Attention for quantized V cache; that's a hard constraint in the code, not a preference. So on those stacks, FA on is the admission price for q8_0 KV.
And q8_0 KV is what buys the headline capability of my daily-driver stack: a 262k-token context window on a 27B dense model, in about 23 GB of VRAM, on a GPU that was current when Obama was president. Without quantized KV, that context would need roughly twice the memory and would not fit.
So the deployed matrix today looks like this:
- Dense SSM-hybrid stacks: FA on, because quantized KV requires it, and long context is their job.
- One MoE hybrid: FA on with f16 KV, deployed as measured-best on that family (recorded honestly in my notes as observed-deployed, not a controlled A/B).
- The older families, Gemma 4 and the pre-hybrid Qwens and the 80B MoEs: FA off, because for them the classic rule is still correct.
Same silicon. Three different answers. The deciding variables are model family and KV quantization, not GPU generation.
The trade underneath
Nothing here is free, and naming the cost is the point. Quantized KV costs about 16% decode speed, a known measured penalty. My long-context stack pays 16% of its speed for double the context per byte of VRAM. For a latency-sensitive stack, the same trade would be wrong.
"FA off on Pascal" was a measured fact about one kernel generation and one use case. I mistook it for a law of the hardware. The measurement was never wrong; my generalization of it was.
The pattern
This is the third post in a row that lands on the same shape of lesson, and that's not an accident, it's the thesis of the series:
- The deleted flag (post 2): a true rule expired when upstream changed.
- The dead env var (post 3): a true conclusion rode on a false mechanism.
- Flash Attention (this post): a true measurement got over-generalized into a false rule.
A verdict without its mechanism expires. "FA off" was a verdict. "FA is slower on Pascal's kernel generation, unless a feature you need requires it, in which case pay the price knowingly" is a mechanism with its conditions attached. The second form survives upstream changes, new model families, and new requirements. The first form silently goes stale and costs you capabilities you didn't know you'd given up.
When someone hands you a hardware rule, ask what it's conditioned on. If the answer is "nothing, it's just true," it's probably a verdict that hasn't met its exception yet.
Top comments (0)