It Fit in Memory and Was Still Unusable — Do the Bandwidth Arithmetic First

Originally published on hexisteme notes.

"Will it fit on our hardware?" is the wrong first question. It's the one everyone asks, because
it's free to answer — the thing either loads or it doesn't.

Throughput costs you a measurement. So the capacity gate passes, and it feels like the
decision is made.

The measurement

Mac Mini M4, 24GB unified memory, ~120GB/s memory bandwidth. A 27B model, IQ4_XS quantized,
15GB on disk.

Capacity gate: pass. Metal's recommendedMaxWorkingSet is 17.76GB, the model is 15GB,
ollama ps reports 100% GPU resident. No swap, no spillover. By every "does it fit" criterion
this is a clean win.

Generation: 5.6 tokens/second.

That's not a usable interactive worker. It's barely a usable batch worker. And nothing about
the capacity check hinted at it.

The arithmetic that would have told me in advance

Autoregressive generation reads the entire model's weights once per token. So:

ceiling ≈ memory bandwidth ÷ bytes touched per operation
        = 120 GB/s ÷ 15 GB
        = 8 tokens/second

Measured 5.6 against a ceiling of 8. Ratio 0.70.

That ratio is the whole verdict. When measured throughput is a large fraction of the arithmetic
ceiling, you are bandwidth-bound, and you now know something concrete: the bottleneck is
not your configuration, not memory pressure, not thermal throttling. It's how fast bytes move.

Rule of thumb I now use: ratio ≥ 0.5 → bandwidth-bound, and size-reduction fixes are dead.

Why "just quantize harder" doesn't work

The natural move when capacity is tight is to shrink. Lower quantization, smaller batch,
heavier compression. It's the reflex, and in a bandwidth-bound regime it's close to useless.

I was considering Q3_K_M at 13.8GB. Run the same division:

120 ÷ 13.8 = 8.7 tokens/second      (up from 8)

Under 9% more throughput. For a real drop in output quality, because quantization error
doesn't scale linearly with size the way bandwidth does — you give up more than you get, every
time, in this regime.

I killed that plan without downloading anything. That's the saving this rule buys: an
arithmetic rejection instead of an afternoon of benchmarking a model that couldn't have won.

The deeper reason is that capacity and throughput are governed by different resources.
Capacity is bytes of memory. Throughput is bytes per second across a bus. Pulling the capacity
lever moves the capacity number. It touches the throughput ceiling only through the incidental
fact that a smaller model has fewer bytes to stream — a weak, strictly linear coupling, and you
pay for it non-linearly.

Naming the right bottleneck tells you which lever works

This is the part that makes the arithmetic worth doing. Once you know it's bandwidth, the same
model on different hardware is a division away:

Machine	Bandwidth	Ceiling for a 15GB model
M4	~120 GB/s	8 tok/s
M4 Pro	~273 GB/s	18 tok/s
M4 Max	~546 GB/s	36 tok/s

The verdict flips on hardware, not on model size. "This model is too slow" was never true —
"this model is too slow on 120GB/s" was. Those lead to completely different purchase
decisions, and only one of them is correct.

When you correctly name the constrained resource, the set of interventions that can possibly
work falls out of it. Naming the wrong one sends you tuning things that were never the
problem.

Anti-patterns, all of which I've done

• "It loaded, so we can use it." Loading is the capacity gate. Passing it leaves throughput entirely unknown.
• "It's slow, so quantize lower." In a bandwidth-bound regime, size and speed are linearly coupled and quality degrades faster. This is the classic symptom of conflating the two gates.
• "It must be swap / memory pressure." Measure warm, at least once, with the model already resident. Otherwise you're mixing the ceiling with transient congestion — and if you go clean up applications before separating them, you'll never find the actual cause. Here there was no pressure at all and the ceiling was exactly where the arithmetic put it.
• "Theoretical bandwidth isn't effective bandwidth, so the calculation is meaningless." Effective is typically 60–80% of theoretical. Include that and the estimate is still good to an order of magnitude — and order of magnitude is the entire decision. 8 vs 5.6 is the same answer. 8 vs 36 is a different one.

The procedure

1. Before checking capacity, compute the throughput ceiling: ceiling ≈ bandwidth ÷ bytes touched per operation.
2. Measure at least once warm, so cold-load and swap pressure don't contaminate the number.
3. If measured / ceiling ≥ 0.5, declare bandwidth-bound and reject size-reduction fixes.
4. Choose from what's actually left: (a) restrict to latency-tolerant uses, (b) change hardware, (c) move the work off this machine.
5. Write the falsifier as a bandwidth number. If you've named the constrained resource, the point at which that resource changes is the condition that overturns your verdict.

That last step is why this generalizes past local models. The same shape applies to any
streaming bottleneck — disk-bound ETL, network-bound sync, cache-line-bound inner loops.
"It fits" and "it's fast enough" are separate gates, and the first one is free to check, which
is exactly why it gets mistaken for the second.

What would change my mind

Bandwidth is the named bottleneck, so the falsifier is a bandwidth number: on a 273GB/s
machine this model clears 18 tok/s and the "not usable as a live worker" verdict is void. Any
architecture change that stops reading all weights per token — heavy MoE sparsity, aggressive
speculative decoding — also breaks the bytes touched = model size assumption the ceiling is
built on, and the division has to be redone with the real figure.

---

The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.