A machine with a 24 GB consumer GPU is not a hundred times worse than a datacentre. It is running a fundamentally different workload, and the difference is visible in one ratio that has nothing to do with how good the hardware is.
The comparison is not the one you think
The instinctive explanation is that hosted providers have better silicon. They do, but that is not what you are feeling. The disqualifying difference is that a hosted endpoint is serving your request alongside many others through the same weights, and your machine is serving exactly one. That single fact changes the class of problem the hardware is being asked to solve, and it changes it in the direction that wastes the most expensive thing a GPU has.
This is worth arguing carefully, because the wrong diagnosis leads to the wrong purchase. If you believe the problem is raw speed, you buy a faster card. If the problem is that batch-size-one decode cannot use a fast card, a faster card buys you less than the spec sheet implies.
Arithmetic intensity at batch one
Arithmetic intensity is operations performed per byte of memory traffic, and it is the number that decides whether a kernel is limited by compute or by bandwidth. For the decode step of a dense transformer the accounting is simple. Each weight is loaded once and used in roughly one multiply-accumulate — two floating-point operations — per sequence in the batch. So:
arithmetic intensity ~= 2 x batch_size FLOP per byte of weights
At batch size one that is about 2 FLOP per byte. Now compare it with what the hardware wants. NVIDIA’s published specification for the GeForce RTX 4090 lists 24 GB of GDDR6X on a 384-bit interface with a 450 W total graphics power, and its tensor throughput is measured in hundreds of TFLOP/s. Whatever the exact bandwidth figure, the ratio of compute to bandwidth on any modern accelerator is in the hundreds of FLOP per byte — that is what the tensor cores were added for.
A workload offering 2 FLOP per byte to hardware that wants several hundred is not slightly inefficient. It is using a low single-digit percentage of the arithmetic the card can do, and spending the rest of every microsecond waiting for weights to arrive. Your GPU utilisation reading will still say something high, because that metric counts whether kernels are resident, not whether they are computing — which is why GPU monitoring during inference is so misleading.
What a serving fleet does instead
The one variable in that expression is batch size, and it is the only lever a provider has that does not require better hardware. Load the weights once, run sixty-four sequences through them in the same pass, and each of those sequences got its token for a sixty-fourth of the memory traffic. Arithmetic intensity rises with the batch, the tensor cores start to be the constraint instead of the bus, and the cost per token collapses.
Modern serving stacks push this further with continuous batching: rather than waiting for a batch to fill and finish together, a request joins the running batch at the next step and leaves when it emits a stop token. llama.cpp implements the same idea for local use — its server advertises parallel decoding and continuous batching, and -np, --parallel sets how many slots it keeps. That is the machinery a provider is running at a scale you are not.
Notice what this implies, because it is the useful part. The provider is not giving each request more hardware than you have. It is giving each request less, and getting more out of it, by never spending a full weight read on a single token. Your local model, alone on a card, is the least efficient possible way to use that card — and it is also, for one user, the fastest that card can answer, because batching improves throughput per watt and per dollar, not latency for the first sequence in the batch.
The part that is not throughput at all
Some of the perceived gap is not generation speed. Three things contribute, and they are worth separating because their fixes are unrelated:
- Load time. A hosted model is always resident. Yours may be reading gigabytes off an SSD. Ollama keeps a model in memory for five minutes after the last request by default and unloads it after that, so the first message after a coffee break pays the full load again. Setting
OLLAMA_KEEP_ALIVE, orkeep_alive: -1on a request, holds it. - Prefill. Time to first token grows with prompt length, and it grows faster on a machine without a fast prefill path. A long system prompt costs you before the first character appears.
- Model size. You are frequently comparing an 8B at four bits with a frontier model. Some of what feels like slowness is the smaller model needing more turns to get somewhere, which is a capability difference wearing a latency costume.
What actually helps locally
Given the diagnosis, the interventions that follow are narrow but real. Reduce bytes moved per token: a smaller model, or a lower quantization level where the task tolerates it. Keep the model resident so load time stops appearing in your latency. Stream, so the reader starts at time to first token rather than at the end. And if you genuinely have concurrent work — a batch of documents to classify rather than a conversation — run it through the parallel slots instead of sequentially, because that is the one case where your machine gets to use the same trick the fleet does.
What does not help is buying compute. The card you have is already idle most of every decode step. Bandwidth, model size and residency are the three terms in the problem, and only those three.
The common resolution is not to pick a side: a local model for the cheap, private or offline path and a hosted one for the requests it cannot serve. That means one call site talking to two backends whose streaming shapes, error codes and context limits differ, and whose failure modes are nothing alike — a local server that is simply not running versus a rate limit. A gateway such as Multigrid exists to make that one interface with a declared fallback order, so the switch is configuration rather than a branch in your application.
Top comments (0)