The single most important llama.cpp flag for my dual Tesla P40 setup was -sm row. It split every layer's tensors across both GPUs and it was worth nearly double the throughput of the alternative: 12-14 tokens/sec against about 7 for layer split. Every stack I built was tuned around it.
In July 2026, upstream llama.cpp deleted it. Not deprecated. Deleted.
This is the story of a performance rule that died twice, and what replaced the throughput it took with it. It's the longest arc in my notes, and it runs in five acts.
Act 1: Row wins
February 2026. My first serious model, a 72B, started at 2.7 to 3.6 tokens/sec in what my notes politely call "poor configuration." Working up the offload ladder to full GPU residency and switching to row split produced the first real daily driver: about 10.3 tokens/sec generation, 60 tokens/sec prompt processing.
The standing rule crystallized: row split, 12-14 tokens/sec. Layer split, about 7. And a rumor from a vendor blog said a newer "graph" split mode was worth another 30-40%.
Act 2: The regression incident
March 2026. I rebuilt the same model on a "modernized" fork, and changed four variables at once: split mode, SIMD compile flags, kernel selection method, and compression settings. Prompt processing collapsed from 153 tokens/sec to 29. Five times slower, and with four simultaneous changes, nothing was attributable.
The cleanup A/B on March 4 isolated everything. Same model, same prompt, one variable at a time:
- Original binary, row split: 60 tokens/sec prompt, 10.3 generation. Works.
- New fork, layer split: half the speed.
- New fork, graph split: CUDA crash.
ROPE failed: an illegal memory access.
The "40% faster" graph mode does not run on Pascal at all. That's the difference between a community claim and a measurement on your own hardware: one of them can crash.
The verdict written that day: the original binary is optimal for this hardware, do not switch. And a rule was born that I now treat as non-negotiable: one variable per change. That rule was paid for in lost throughput and a wasted week.
Act 3: The fast mode becomes the wrong mode
May 2026. Gemma 4 arrived, and its architecture uses shared KV layers implemented as tensor views. Those crash row split on multi-GPU, a hard assert deep in the CUDA backend, known upstream issue. Every Gemma stack I run is layer split by necessity, knowingly paying the throughput cost. The Qwen architectures have no such bug and kept row.
So split mode is not a performance dial. It's also a correctness knob, and the answer is per-architecture. "Row is faster" was true and useless without the condition attached.
Act 4: Upstream deletes row
July 6, 2026. llama.cpp removed -sm row entirely. Any stack built from a newer clone has layer split as its only multi-GPU option. The flag I had organized my fleet around no longer exists in the binaries I build.
Row died twice, in two lineages: first the fork replaced it with a mode that crashes Pascal, then upstream removed it outright.
Act 5: The win comes back from somewhere else
Here's the part that justified the whole ordeal. On the new stack, layer split alone measured 8.46 tokens/sec single-stream, right where the old "layer is about 7" rule predicted. But two features that didn't exist in my February binaries changed the math:
-
--parallel 4(four concurrent slots): 8.46 / 12.8 / 15.0 tokens/sec at 1, 2, and 4 slots. Aggregate throughput nearly doubles before per-slot latency degrades. - MTP speculative decoding: 8.46 to about 13.3 tokens/sec single-stream, a 57% lift, acceptance rates 0.38 to 0.63, output correctness verified.
Net result: the fleet ended up faster than the row-split era without row split. The recovery didn't come from finding a replacement flag. It came from features orthogonal to the one I lost.
What this taught me
Date-stamp every performance claim and name the binary it was measured on. A tuning rule is a fact about a specific artifact at a specific moment, not a law of the hardware.
And when the flag you tuned around disappears, re-measure before assuming regression. The replacement win may live somewhere you weren't looking. Mine did, twice over.
Top comments (0)