Eggs, Cholesterol, and GPU Flags

For decades, nutrition science flip-flopped on eggs. Bad for you: dietary cholesterol. Then fine, then good in some contexts, then it depends on the person and the rest of the diet. People read this as science failing. It's the opposite. It's what knowledge looks like while it's maturing: early evidence produces a verdict, later evidence produces conditions, and the mature answer names the deciding variable instead of picking a side.

Six months of running LLM inference on old hardware took me through exactly that arc, on about half of everything I thought I knew.

The scorecard

Claims I held in the spring, audited in the fall:

Overturned or narrowed:

• "Flash Attention must be off on Pascal" became per-family: off where the old measurement still applies, required where quantized KV cache demands it.
• "Force the MMQ kernel env var" turned out to be dead code. The kernels were always on; the variable was never read.
• "Row split is the P40 answer" was true, then expired twice: a fork replaced it with a mode that crashes Pascal, then upstream deleted it. The recovery came from parallel slots and speculative decoding instead.
• "Graph split is 40% faster" crashed on my hardware on the first real run.
• "Recompile to enable AVX2" was solving a problem the running binary didn't have.
• "48 GB of VRAM" is 45 usable once the driver takes its cut. My earliest hard lesson is literally titled "a 47GB model does not fit in 48GB."
• "Concurrency caps are static numbers" fell when the same endpoint admitted different loads at different hours. A cap is a worst-observed defense, not a promise.

Survived unchanged:

• CUDA as the only viable backend on this hardware, and the compute-capability facts underneath that.
• vLLM's non-viability on Pascal, confirmed by experiment rather than docs.
• Sparse MoE as the biggest throughput lever available.
• Q6_K as the fleet default quant.
• The single-variable-change rule.

Look at what separates the lists. The survivors are hardware facts and rules with a mechanism attached. Everything that expired was a verdict about a moving target: upstream code, kernel generations, a vendor's capacity pools. A verdict without its mechanism expires. The mechanism survives the verdict.

The evidence ladder

The deeper takeaway isn't any single reversal. It's learning to rank evidence by how it fails. Mine, in the order I learned to trust it:

1. Community claims. Reddit, GitHub issues, vendor blogs. Cheap, often right, occasionally catastrophic (the "40% faster" mode that crashes Pascal came from here).
2. Ad-hoc measurements. Your own numbers, one config, no controls. Better; this made a 5x regression visible but couldn't say which of four changes caused it.
3. Controlled single-variable A/B on your own hardware. The first rung where a number becomes trustworthy.
4. Source verification. Reading the code settles what a flag even does. This is the rung that exposed the dead env var.
5. Runtime artifact inspection. The running binary's own startup banner beats the build directory's story about it.
6. Production evidence over time. Months of deployment. The only rung that catches things like a vendor silently rerouting a model id to a different capacity pool.

Each rung catches a failure mode the rungs below can't see. The expensive mistakes in this series all came from acting on rung 1 or 2 evidence as if it were rung 5 or 6.

Why bother, on $2,000 of used parts

Because the discipline is the product. The server is nice; the habits are transferable to any system whose ground truth moves: date-stamp claims, name the binary, one variable at a time, verify the premise before optimizing it, let gates outrank judgment, and treat every reversal as content rather than embarrassment.

Nutrition science didn't fail when the egg advice changed. It was doing the only thing evidence-based work can do: hold the best current answer with its conditions attached, and revise when better evidence lands. Performance engineering on a fast-moving stack deserves the same posture. The point was never to be right in March. The point is for September's answer to be better, and to know exactly why it changed.