Running models at home rewires the question you ask about them. In a hosted console the only thing you ever feel is quality, because a stranger is paying for the silicon. On machines you own, that silicon is the invoice, so every model has to answer something blunter than "is it good," which is "are you worth the card you are sitting on." This is what happened when I put that question to one specific model, Zhipu's GLM 4.7, on a pair of Nvidia RTX 3090s, and got back an answer that flipped on me halfway through.
What a home lab actually asks of a model
I keep a small stack of GPUs busy instead of renting them, so a model does not get to win on vibes. It has to be cheaper per useful token than the alternative already loaded next to it. GLM 4.7 is the interesting kind of candidate: it is a mixture-of-experts build, roughly thirty billion parameters on paper but only about three billion doing work per token, wrapped around a memory trick called Multi-head Latent Attention. On paper that is a model that should be both fast and light. The last time I tried it, on a much weaker box, it was neither, and I wrongly blamed the model before realizing the runtime simply was not doing the latent-attention math. So this round I gave it fair hardware and a fair opponent.
The two contenders, and a slow wire between the cards
The opponent was deliberately boring: an ordinary dense, grouped-query model in the same size class, the sort of thing that never trends but never surprises you either. Both ran under the same server so nothing about the plumbing favored one side. The hardware was two Nvidia RTX 3090s, twenty-four gigabytes apiece, forty-eight together, on a mid-range desktop. No fancy interconnect. The two cards gossip over a link I clocked at about 1.4 gigabytes a second, which is glacial by GPU standards, and that single unglamorous number ended up steering half the results.
Then I stopped theorizing and just clocked both models at every context length from a few thousand tokens out to a hundred and twenty-eight thousand.
Generation speed has a tipping point you can predict
Out of the gate GLM 4.7 was thrilling, close to ninety tokens a second on short prompts, well over double the dense model. And then it leaked speed as the context filled: down through the sixties, forties, twenties, and into the mid-teens at the deep end. The dense model, meanwhile, did almost nothing interesting, holding its lane the entire way with the stubbornness of a machine that has nothing clever to lose. The two curves met somewhere around sixty-four thousand tokens. Below that line the exciting model runs away with it. Above that line, the model nobody writes about is quietly faster.
The tax that never shows up in the demo
Everyone benchmarks generation speed. Almost nobody benchmarks the wait before the first token, the time the machine spends swallowing your prompt, which on long inputs is most of the wait you actually feel.
| context | Zhipu GLM 4.7 | dense model |
|---|---|---|
| 4K | 2626 tok/s | 979 tok/s |
| 16K | 1849 | 947 |
| 32K | 1308 | 899 |
| 64K | 805 | 780 |
| 128K | 450 | 631 |
Same silhouette as before. GLM 4.7 devours a short prompt roughly three times quicker, then bleeds that advantage as the prompt grows, and by the deepest context the plain model is ingesting text faster than it is. The clever model is a sprinter on both halves of the job, and sprinters are not who you want at the back of a long race.
Memory runs the story backwards
Here is the twist that keeps GLM 4.7 in the conversation. The same latent-attention trick that costs it speed deep in a long context also keeps its running memory unusually small, so on a single twenty-four gigabyte card it fits noticeably more context before it has to spill onto the second GPU.
So the two stories point in opposite directions. On speed the dense model wins the long game. On memory the clever model does, packing a bigger context onto one card. If your problem is "hold an enormous document on a single 3090," GLM 4.7 has a real, physical answer for you, it just charges you generation speed to give it. I even tried shrinking the memory further with a compressed cache and it backfired, slower at every depth for no real headroom, so I left that idea on the floor.
A second card can make things slower, not faster
The most counterintuitive stretch came from trying to use both RTX 3090s at once. Instinct says two cards beat one. For a single conversation at short context, one card actually won outright, because splitting the model across the pair makes every token pay a toll crossing that 1.4 gigabyte wire, and on short prompts the toll costs more than the extra card saves.
The math only tips once the context grows long enough that the per-token work dwarfs the toll, at which point splitting across both cards roughly doubles the memory bandwidth the model is starved for, and the pair pulls ahead. And the moment you stop chatting and start serving a crowd, it stops being a contest: batched across many streams the two cards together pushed toward five hundred tokens a second in aggregate, several times what one card manages alone. So even "use all my hardware" turned out to be another it-depends.
The decision tree I actually kept
I walked away with a verdict so unexciting it is almost rude:
- Short, snappy, one person at a keyboard: run Zhipu's GLM 4.7 on a single card, it is genuinely quicker.
- Long single-user context: run the boring dense model, still on one card, it is faster and simpler.
- Serving real traffic: split GLM 4.7 across both RTX 3090s and take the throughput.
There is no overall champion, only a right answer per shape of job, and the skill is being honest about which job you are actually doing. I keep a quick model and a dependable model side by side and reach for whichever the task rewards, which is the thing the leaderboards structurally cannot tell you.
None of this benchmarking is academic for me. The same two cards run the small tools I build on ModelDirectory, and the heaviest of them are exactly the short-to-medium GPU jobs where a fast model on one card wins: an STL upscaler that adds real detail to a printable model, and an STL texture generator that paints one from a prompt. The lighter utilities barely warm the GPU, like the print orientation and printability checker and the print cost calculator, and being able to run all of them on hardware I already paid for is the whole reason the run-or-rent math keeps landing on run.
The next thing I want to know is whether that sixty-four thousand token crossover survives contact with real long workloads instead of tidy synthetic ones. That needs a better test rig, and it is where I am headed next.
Top comments (0)