We all know the default answer when someone asks about running local LLMs: "Get a GPU." Preferably an NVIDIA one. And honestly, for fine-tuning or heavy multimodal work, that's true.
But what if you just want to run inference? A chatbot for your internal tools, a quick summarizer, or a private assistant that doesn't send your data to the cloud? Do you really need a discrete graphics card for that?
I decided to find out. This is a write-up of my benchmark running four different models on a CPU-only server — no GPU, just an AMD EPYC and 32 GB of RAM. Spoiler: it works, and some results were genuinely surprising.
The Testbed
Here's the hardware and software setup:
CPU: AMD EPYC-Genoa Processor (8 cores, 16 threads via SMT)
RAM: 32 GB
OS: Ubuntu 24.04
Runtime: Ollama (for easy model switching and built-in quantization support)
The goal was simple: measure generation speed (tokens/sec) and subjective quality of Russian-language responses. I used similar prompts across models to keep the comparison fair. Note that on CPU, prompt evaluation time can differ significantly from generation time — I focused on the latter, since that's what affects the "feel" of chatting with a model in real time.
The Contenders
I picked four models with different sizes and quantization levels:
Qwen3.5:9b — the heavyweight, 9B parameters.
SparkLLM/Spark-X2.6-1.7B:q8_0 — a lightweight 1.7B model with 8-bit quantization.
Gemma3:4b-it-q4_K_M — 4B parameters, 4-bit quantization (the "golden middle").
MiniCPM-V 4.7 1B — a tiny 1B multimodal model.
Results: Speed
Here's the raw data. Generation speed is measured in tokens per second (tok/s), and total time is for the full response.
Model Generation (tok/s) Input tokens Output tokens Total time
Qwen3.5:9b 6.71 19 467 70.46 s
Spark-X2.6-1.7B:q8_0 16.72 88 4,008 240.47 s
Gemma3:4b-it-q4_K_M 13.29 27 296 22.73 s
MiniCPM-V 4.7 1B 52.63 806 277 5.94 s
A few observations:
Qwen3.5:9b is painfully slow. 6.71 tok/s means you're waiting over a minute for a medium-length answer. It works, but it's not something you'd want to use interactively.
Spark is decent at 16.72 tok/s, but it generated a massive 4,008-token response, which took 4 minutes total.
Gemma3:4b hits a sweet spot at 13.29 tok/s. Fast enough to feel responsive, slow enough that you notice it's "thinking."
MiniCPM-V is blazing fast at 52.63 tok/s — over 7x faster than Qwen and 4x faster than Gemma.
A note on fairness
I have to be honest here: MiniCPM's prompt evaluation rate (345 tok/s) looks impressive, but 577 of its 806 input tokens were already cached. So that number isn't directly comparable to models that processed prompts from scratch. However, the generation speed of 52.63 tok/s is real — it produced 277 tokens in under 6 seconds.
Results: Quality (Russian Language)
Speed is great, but if the model outputs gibberish, it's useless. This is where things got interesting.
I ran a few diagnostic prompts in Russian and evaluated the responses subjectively. Here's what I found:
MiniCPM-V 4.7 1B — despite its speed, it was the weakest in Russian. It's primarily designed for multimodal tasks (images, video) and English. For complex text tasks, 1B parameters just aren't enough.
Gemma3:4b — the clear winner for Russian. It officially supports 140+ languages and was clearly trained with multilingual text in mind. Responses felt natural, without the "machine accent" or hallucinated facts.
Qwen3.5:9b — the heavy artillery. Slow, but potentially better for complex reasoning tasks. I'd keep it around for deep analysis, not quick chats.
Practical Takeaways: What Actually Works on CPU?
After all the tests, I settled on three use cases:
Everyday tasks and chat
Gemma3:4b is my daily driver. ~13 tok/s, good Russian quality, and it fits comfortably in 32 GB of RAM.Ultra-fast responses and multimodal tasks
MiniCPM-V 4.7 1B is my go-to when I need a quick answer or need to process an image. Yes, its Russian is weaker, but 52 tok/s on CPU is genuinely impressive.Complex reasoning
Qwen3.5:9b stays as a "heavy alternative." I wouldn't use it for simple questions, but for deep analysis, it might be worth the wait.
The Gotchas
It wasn't all smooth sailing. Here are the pain points I ran into:
Thermal throttling. During long generations, the CPU heats up and clock speeds can drop. This affects consistency.
32 GB RAM is not infinite. If you run multiple models simultaneously, the system can start swapping to disk, and performance tanks.
Context window matters. The more text you feed into the model, the slower it gets. On CPU, this is especially noticeable.
Final Verdict
Can you live without a GPU? Absolutely. If you're not fine-tuning models or generating hundreds of images per minute, modern CPUs handle inference just fine.
For my AMD EPYC with 32 GB RAM on Ubuntu 24.04, Gemma3 4B is my primary model. It offers the best balance of speed and Russian-language quality. MiniCPM is a great tool for fast experiments and multimodal tasks.
Have you tried running LLMs on CPU? Which model worked best for you? Share your experience in the comments!
Top comments (1)
The interesting thing in your table is that three of the four numbers imply the same memory bandwidth, which means your box is the bottleneck and the model name is almost decorative.
Generation on CPU reads the whole quantized model once per token, so tok/s is bytes divided by effective bandwidth. Take the two you can check against file size: Gemma3 4B at q4_K_M is about 2.5 GB, and 13.29 tok/s against 2.5 GB is 33 GB/s. Qwen3.5 9B at roughly 5.5 to 5.8 GB, 6.71 tok/s, is 37 to 39 GB/s. Two models 2.2x apart in size landing within 15% on the same bandwidth is the signature of a bandwidth wall, not a model-quality difference.
That turns your benchmark into a predictor: on that server, tok/s ≈ 36 / model size in GB. A 14B at Q4 (9 GB) will do about 4 tok/s, and no runtime setting will change it. The 32 GB of RAM is not the constraint; the eight cores' share of the memory controller is.
The one result that breaks the pattern is MiniCPM at 52.63 tok/s, which implies 45 to 60 GB/s for a 1B. That is either a much smaller file than its parameter count suggests or a shorter read per step, and it is worth knowing which before you use it as the speed reference.
The next measurement I would take: tok/s at 512, 2k and 4k context on Gemma3 4B. KV read grows with context and does not amortize, so the drop you see there is the real cost of long chats on that box.