Ten rounds. One verdict. No survivors.
You've got a GPU. You've got a model. You need something to run it.
Two contenders step into the ring: Ollama, the scrappy underdog that just works, and vLLM, the production-grade inference engine that arrived with a PhD and a dependency tree the size of a small country.
Both are good. Neither is free to operate. And a fast benchmark is not the same thing as an affordable service.
So, only one can be your daily driver. Let's find out who will win a Death Battle.
Spoiler: the battlefield matters.
The Contenders
🟦 Ollama — Powered by the famous llama.cpp (among other backends). Easy to install, easy to pull models, and equipped with a built-in API. Your grandmother could probably deploy it. (Don't test this, or if you do, share the results with us in the comments!)
🟥 vLLM — Born in a Berkeley research lab. Continuous batching, PagedAttention, quantization backends, and production metrics. Reads like a GPU conference talk. Runs like one too.
The rules: Today's battlefield is self-hosted inference for coding assistants and development teams, including rented AWS GPUs. We'll judge deployment, compatibility, performance, maintenance, and the final cloud bill.
This isn't a controlled, identical-model benchmark. We're using practical deployment observations, and configurations can change the outcome.
The contenders are set. Let the Death Battle begin!
Round 1: Deployment — First Blood
Installation, configuration, and time to first inference
Ollama:
curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:30b
That's the basic Linux path. Download the model, start chatting, and move on with your day. You still need the right drivers and enough memory, but the initial experience is deliberately simple.
vLLM:
pip install vllm
vllm serve Qwen/Qwen3-32B
That can work too. The trouble often starts when your GPU, CUDA stack, Python packages, model architecture, and chosen vLLM release disagree. Then the installation becomes less of a command and more of a diplomatic negotiation. For example,
pip install vllm
# Error: torch version incompatible
pip install vllm==0.9.1
# Error: transformers 4.52.0 conflicts with required 4.51.3
pip install transformers==4.51.3
# Error: FlashInfer not found
# ... (three more errors)
# Model loads. You weep quietly.
Not my best negotiation ever. Maybe I should upgrade my negotiation skills.
It might not be slower, but it's certainly another hurdle before the fight even begins.
🥊 Bonus Hit: Windows Users, Good Luck!
Ollama runs natively on Windows, Linux, and macOS. vLLM? Linux is its natural habitat. Windows users need WSL2 or community alternatives.
🏆 Round 1: 🟦 Ollama. Faster to get started for most individuals and small teams.
Round 2: Model Compatibility — The New Kid in Town
Model architectures, formats, and the Nemotron problem
Ollama makes supported models easy to download and run. Its GGUF ecosystem offers a huge selection of quantized checkpoints, which is particularly attractive on constrained GPUs.
But GGUF is a format, not magic. A new architecture still needs runtime support, and not every model or feature works in every backend.
vLLM supports numerous architectures and quantization formats, including GGUF in supported configurations (still experimental). Its compatibility matrix evolves quickly, but that also means version selection matters. Depending on your model, quantization format, and configuration, you might need a bigger VM than you expected. Your cloud bill (and your boss) will be delighted!
During the evaluation behind this article, a Nemotron model using the nemotron_h architecture failed on an older vLLM release with an unsupported-architecture error. Moving to a newer release meant revisiting the dependency stack. That is a real example of version-specific friction, not proof that vLLM cannot run Nemotron.
NVIDIA's Nemotron 3 Super 120B is a hybrid Mamba, attention, and mixture-of-experts model. New architectures like this are precisely why checking the exact model, runtime version, and checkpoint format is essential.
🏆 Round 2: 🟦 Ollama, narrowly. Model discovery and deployment are often simpler, but neither runtime supports everything automatically.
Round 3: Quantization Warfare
FP16, BF16, FP8, and four-bit models
Quantization matters because model weights are often the biggest item on your GPU memory bill.
| Weight format | Approximate bytes per parameter | Approximate weights for 32B parameters |
|---|---|---|
| FP32 | 4 | 128 GB |
| FP16 / BF16 | 2 | 64 GB |
| FP8 / INT8 | 1 | 32 GB |
| 4-bit | 0.5, plus metadata | 16 GB, plus overhead |
These are weight-only estimates, not total VRAM requirements. Add the KV cache, runtime buffers, temporary allocations, and the practical cost of distributing a model across GPUs.
vLLM has broad options including FP8, AWQ, GPTQ, and others. Native FP8 weight-and-activation acceleration is available on GPUs such as NVIDIA L4, while older architectures can use certain weight-only FP8 implementations. So it is incorrect to say that an A10G is limited to FP16 in vLLM.
Ollama's GGUF-based quantizations make four-bit deployments accessible on a wide range of machines. That can make a larger model viable where full precision would be impossible.
Neither engine owns quantization. The model, format, accuracy target, and GPU determine the winner.
🏆 Round 3: Draw. vLLM offers powerful optimized kernels; Ollama makes compact quantized deployments easy.
Round 4 — The VRAM Wall: Mathematics Doesn't Negotiate
GPU memory requirements, KV cache, large models, and multiple loaded models
VRAM is the arena. The model must fit inside it or the fight doesn't happen.
A 120B model can require hundreds of gigabytes at full precision, or considerably less when quantized. The same applies to smaller models.
The problem isn't simply how many parameters you have. It's how those parameters are stored, how much memory the runtime consumes, and how much space remains for inference.
The KV cache twist: Every active conversation requires additional memory. Context length, architecture, concurrency, and cache precision all influence that requirement.
Hybrid architectures such as NVIDIA's Nemotron 3 Super combine Mamba and attention layers, allowing different memory characteristics from conventional transformers.
vLLM's PagedAttention manages KV cache memory efficiently, particularly under concurrency. However, its memory allocation strategy can consume a large portion of available VRAM depending on configuration.
On a G6.12xlarge running Qwen3-32B, one evaluation showed approximately 22,551 MB occupied per GPU at startup, leaving just 16 MB available.
That's a lot of reserved space when you want to run additional models.
The allocation is configurable, though. It is not a permanent architectural limitation.
Ollama takes a different approach.
It can keep multiple models loaded simultaneously when sufficient GPU memory is available.
In one G5.48xlarge deployment, the following models were loaded together:
120B model: ~101 GB
27B model: ~17 GB
Two models. One server. Both answering questions.
And here's where things get interesting.
Add a smart router, and you can direct straightforward requests to the smaller 27B model while reserving the 120B model for complex tasks.
One GPU VM. Two different levels of intelligence. No additional GPU instance required.
Of course, the router itself needs configuration, and the actual savings depend on how effectively it distributes requests.
vLLM can implement a similar architecture using separate serving instances and appropriate GPU allocations. But that requires additional planning around model placement, memory reservation, and routing.
🏆 Round 4: 🟦 Ollama, for the simplicity of operating multiple models on shared hardware.
PagedAttention is clever. But sometimes what you need isn't a more sophisticated memory manager. It's the ability to run several models without turning deployment into a second job.
Round 5: The Speed Demon
Throughput, latency, and tokens per second
Here's where vLLM earns its reputation.
Its scheduling and batching are designed to keep accelerators busy across concurrent requests. With the right model, hardware, and quantization, vLLM can deliver substantially higher aggregate throughput than a simpler serving setup.
But be careful with the word faster. There are at least three different questions:
- How quickly does the first token arrive?
- How fast does one response generate?
- How many useful responses can the whole system complete under load?
Those are not interchangeable metrics. vLLM does not automatically win every single-user tokens-per-second test, and a different model or quantization invalidates a direct comparison.
For a production inference service with enough traffic to keep its scheduler busy, its architecture provides a compelling advantage.
🏆 Round 5: 🟥 vLLM. Especially for aggregate throughput under a measured concurrent workload.
Round 6: Concurrency Apocalypse
Simultaneous developers and sustained request pressure
vLLM's continuous batching is one of its defining features. It can schedule work from multiple requests efficiently instead of treating each request as a completely separate job.
Ollama also supports concurrent requests. The OLLAMA_NUM_PARALLEL setting controls parallel processing per model, subject to available memory. Raising it increases memory pressure, especially at long context lengths. It is not a free “more users” switch.
Now consider twenty developers using a coding assistant. Are twenty prompts being generated at the same time? Or are people reading, editing, testing, and thinking between requests?
Those are very different load patterns. Headcount alone does not tell you how much concurrency you need.
Before buying a monster GPU VM, measure peak simultaneous generations, request lengths, queueing, time to first token, and completed requests per minute.
🏆 Round 6: 🟥 vLLM. When sustained concurrency is real, efficient batching matters. The winning margin must still be measured.
Round 7: The Scaling Wars
One large server or several independent ones?
vLLM can scale across GPUs within a deployment and horizontally through multiple replicas and a routing layer. Ollama can likewise run multiple independent servers behind a load balancer.
Neither platform has a monopoly on horizontal scaling.
Ollama's appeal is operational simplicity: each server can be an independent unit, with limited coordination. vLLM's appeal is extracting more throughput from each appropriately configured unit.
There is a catch with the usual architectural diagrams: three servers cost three times the hourly compute of one server of the same size. They may offer more capacity and better fault isolation, but they are not a fair price comparison against a single server.
Scaling is a trade-off between utilization, resilience, performance, and operating complexity. A single powerful VM is not automatically economical; several smaller ones are not automatically cheaper.
🏆 Round 7: Draw. Measure the capacity you need and price equivalent service levels.
Round 8: The Cloud Tax
GPU selection shapes the economics of inference. In AWS US East (N. Virginia), G5 and G6 offer different combinations of accelerator generation, GPU memory, and hourly price. Both inference engines can use compatible NVIDIA GPU instances.
| AWS instance | GPU configuration | Advertised aggregate GPU memory | Approx. USD/hour | Approx. USD/month, 220 hours |
|---|---|---|---|---|
g5.48xlarge |
8 × NVIDIA A10G | 192 GB | $16.29 | $3,583 |
g6.12xlarge |
4 × NVIDIA L4 | 96 GB | $4.60 | $1,012 |
g6.48xlarge |
8 × NVIDIA L4 | 192 GB | $13.35 | $2,937 |
Illustrative Linux on-demand compute prices for us-east-1, checked in November 2026. Monthly cost assumes 10 hours per day × 22 days. Figures are rounded; storage, data transfer, taxes, and discounts are excluded. Verify the rate and regional availability before purchasing.
That makes the G6.48xlarge about 18% cheaper per hour than the G5.48xlarge in this example, with the same advertised aggregate GPU memory.
Both Ollama and vLLM run on compatible NVIDIA CUDA GPUs, including the G6 family. Choosing a newer GPU generation does not require choosing a particular inference engine.
An equivalent-instance comparison isolates the differences between the serving stacks:
| Setup | GPU compute cost for 220 hours | Main operational difference |
|---|---|---|
Ollama on g6.48xlarge
|
$2,937 | Easier model management and multiple loaded models when memory permits |
vLLM on g6.48xlarge
|
$2,937 | Advanced scheduling and batching, with more configuration options |
Same instance, same hourly infrastructure price. Not necessarily the same throughput or cost per completed request. That is what a benchmark must determine.
The g6.12xlarge provides 96 GB of aggregate GPU memory across four GPUs. The models it can serve depend on checkpoint size, memory overhead, context requirements, and how weights are distributed across those GPUs.
🏆 Round 8: Draw. With the same instance and usage hours, GPU rental costs are equal. Throughput, utilization, and operating effort determine which setup provides better value.
Round 9: Maintenance Nightmare
Model updates, dependencies, and operational surprises
Adding a supported model in Ollama:
ollama pull <model-name>
That's usually the pleasant part. Pull it, test it, and use it. Downloading a model does not automatically guarantee it remains loaded forever, but the lifecycle is straightforward.
Adding a model in vLLM:
vllm serve <model-or-checkpoint>
Also straightforward when the model, release, drivers, and environment align.
When they don't, the checklist can get longer: architecture support, compatible versions of the inference stack, quantization kernels, GPU compute capability, context configuration, and regression tests. One upgrade may uncover another issue. At that point, consider a career change. (Or a container image. Try that first.)
In the evaluation informing this article, making a Nemotron checkpoint work with the selected vLLM version required additional troubleshooting. That experience illustrates the maintenance risk; it does not mean every vLLM upgrade is an incident.
Both systems require patching, monitoring, model validation, and capacity planning. vLLM generally exposes more knobs, which is an advantage if you have people to tune them and a burden if you do not.
🏆 Round 9: 🟦 Ollama. For teams without dedicated inference operations expertise, the simpler lifecycle is valuable.
Round 10: The Price of Survival
Total cost of ownership, engineering time, and useful output
AWS is only one line of the invoice. Someone still has to deploy, update, monitor, recover, and troubleshoot the service.
Here is a hypothetical planning scenario, not a universal benchmark of either product:
| Monthly operational effort | Ollama scenario | vLLM scenario |
|---|---|---|
| Hours | 4–6 | 15–25 |
| Cost at an illustrative $150/hour | $600–900 | $2,250–3,750 |
This difference may be smaller, larger, or reversed in your environment. An experienced vLLM team with mature deployment automation may spend less time than an inexperienced team maintaining any runtime.
Likewise, the cheapest hourly engineer is not necessarily the cheapest person to operate a service. If a fix takes three times longer, the lower rate can become expensive quickly.
And vLLM's additional maintenance may be entirely justified if its throughput allows you to serve the same workload with fewer GPUs or fewer instances.
The correct calculation is not simply:
VM price + maintenance hours
It's closer to:
Total cost of ownership = compute + storage + networking + engineering + monitoring + downtime + support, evaluated against the number and quality of useful requests completed.
A $13.35/hour server that completes twice as many useful requests may be a better deal than a less productive server at the same rate. Or it may be an expensive machine sitting idle between prompts.
🏆 Round 10: 🟦 Ollama for the low-operations scenario. For high-utilization services, the outcome is genuinely workload-dependent.
Scoreboard
| Round | Decision | Why |
|---|---|---|
| 1. Deployment | 🟦 Ollama | Easier initial experience |
| 2. Model compatibility | 🟦 Ollama, narrowly | Simpler discovery and common deployment flows |
| 3. Quantization | Draw | Different strengths, broad support on both sides |
| 4. VRAM wall | 🟦 Ollama | Simpler multi-model management and efficient use of shared GPU resources |
| 5. Speed | 🟥 vLLM | Stronger throughput-oriented design |
| 6. Concurrency | 🟥 vLLM | Efficient sustained batching |
| 7. Scaling | Draw | Both can scale horizontally |
| 8. Cloud tax | Draw | Both can use the same, cheaper G6 hardware |
| 9. Maintenance | 🟦 Ollama | Less day-to-day operational friction for small teams |
| 10. TCO | 🟦 Ollama, conditionally | Best fit for the low-operations scenario, not every workload |
A scorecard is fun. An actual cost model is better. No number of imaginary knockouts can substitute for measuring your own workload.
The Final Boss: Choose Your Battlefield
🖥️ Battlefield A: Personal Workstation
One user, limited VRAM, occasional inference
You want a local model for coding assistance, experimentation, or occasional tasks. You prefer to spend your time using the model rather than maintaining its serving stack.
Winner: 🟦 Ollama. The deployment experience and compact quantized model ecosystem are more useful here than maximum aggregate throughput.
👥 Battlefield B: Small Development Team
Five to twenty developers, bursty coding requests, no dedicated MLOps team
You need reasonably good response times and predictable operations. Some developers send prompts simultaneously; most of the day, they don't.
Winner: 🟦 Ollama, probably. Start with the simplest G6 configuration that fits the chosen model and observed peak load. Tune parallelism, measure queueing, and add capacity only when needed. Don't assume twenty employees need twenty simultaneous generations.
vLLM can still win if benchmarking shows enough efficiency to offset its operating overhead. That's exactly why you should test it.
🏭 Battlefield C: High Throughput Inference Service
Consistent traffic, tight service targets, dedicated engineering ownership
You're providing an inference API, handling large numbers of overlapping requests, and watching latency, throughput, availability, and cost per request.
Winner: 🟥 vLLM, probably. Its scheduling, batching, and production features are designed for this workload. The extra operational control starts paying for itself when utilization is consistently high.
Measure it with identical checkpoints, comparable quantization, matching context windows, and a representative request mix.
🧠 Battlefield D — The Hybrid Intelligence Strategy
Small local models, powerful remote models, intelligent routing
Why bring a battleship to a knife fight?
Imagine a development team using AI coding assistants. Not every request requires a 120B model.
Some developers want explanations. Others need simple code generation, documentation, or boilerplate. Occasionally, someone needs serious architectural reasoning or complex debugging.
Instead of throwing your most expensive model at everything, introduce smart routing.
- Small tasks: Local Ollama with a 4B or 8B model.
- Medium tasks: Ollama with a larger quantized model, when available.
- Complex tasks: A powerful model hosted remotely, potentially using vLLM or a commercial inference API.
- Failures or insufficient answers: Escalate to a stronger model.
Your routing layer decides which model receives the request based on complexity, capabilities, cost, or other policies.
The economic advantage is simple: not every request needs to consume your most expensive computing resources.
If your local hardware is already available, smaller models can handle a portion of the workload without requiring dedicated cloud GPU instances for every interaction.
Of course, smart routing isn't magic. Someone must design it, evaluate the quality of its decisions, and ensure that complex requests aren't accidentally assigned to incapable models.
And here's the important distinction: vLLM can participate in exactly the same architecture. The intelligent routing happens outside the inference engine. Ollama's advantage is how conveniently it can serve smaller local models.
🏆 Battlefield D: 🟦 Ollama has an edge as the lightweight local inference layer. But the real winner is the hybrid architecture.
Sometimes the smartest AI isn't the biggest model. It's knowing when you don't need it.
The Verdict
What happens when the expected winner discovers that performance alone isn't enough to win the battle?
vLLM is a serious inference engine. Its architecture can deliver excellent throughput and concurrency, especially when you have the right hardware, high utilization, and people who know how to run it.
Ollama's achievement is different. It lowers the effort required to get useful local and self-hosted inference working, and it can make multi-model setups surprisingly straightforward.
For most teams — small, cloud-hosted, without dedicated MLOps — Ollama on
G6.48xlarge wins: cheaper than G5, simpler than vLLM, runs two models, and
requires no dependency archaeology to add a new one. The best inference engine is the one your team can actually maintain at 2am.
The fight isn't over. If vLLM delivers meaningful memory optimizations that allow a G6.48xlarge to comfortably run a 120B model alongside a second model, the economics could shift significantly toward vLLM. For now, it can barely accommodate one, leaving little room for anything else.
And yes, newer GPU generations will bring more memory and processing power. But that benefits both contenders. More expensive hardware alone doesn't decide the winner. What matters is how efficiently each engine uses the resources available. Because trust me, nobody wants to explain to their boss why the cloud bill suddenly doubled the following month.
Check back in mid-2027.
Until then: 🟦 Ollama is the Winner 🏆.
Pricing checked in November 2026 for AWS us-east-1. Cost scenarios are illustrative, not quotations or guaranteed performance. Model memory and deployment observations are configuration-specific. Verify your region, instance availability, exact model variant, and performance before making a purchasing decision.
References
- Ollama
- Ollama FAQ: concurrency, memory and multiple loaded models
- vLLM
- vLLM quantization support
- vLLM configuration and GPU memory controls
- NVIDIA Nemotron 3 Super technical overview
- AWS G5 instances
- AWS G6 instances
- AWS EC2 on-demand pricing
- G5 pricing cross-check, us-east-1
- G6 pricing cross-check, us-east-1
Top comments (4)
Round 4 lists what drives the KV cache but not its size, so I ran the number for the model you named there. Qwen3-32B's own config.json, at revision 9216db57, says 64 layers, 8 key/value heads and head_dim 128. That is 2 x 64 x 8 x 128 x 2 bytes = 256 KiB of cache per token at bf16, and the same file declares max_position_embeddings 40960, so one sequence that fills the window costs 10 GiB of cache. Store the cache in FP8 and it halves to 5 GiB.
On the g6.12xlarge that is the half of the wall your 22,551 MB reading is pointing at. With tensor parallelism over four L4s the 8 KV heads split four ways, so a full-context sequence is 2.5 GiB on each GPU, and the 16 MB left over is a question of how many such sequences fit rather than a question about the weights.
I took those numbers from the checkpoint's config and did the arithmetic. I did not run vLLM on a G6, so the cache figures are computed, not measured on hardware. Anyone checking against a different model needs four fields from its config and two multiplications: layers x key/value heads x head_dim x 2, for K and V, times the bytes per element.
That's an interesting calculation! 😅
However, that's exactly why I would be curious to see the results of an actual deployment. The theoretical KV cache requirements are useful, but they don't necessarily reflect the complete memory consumption or runtime behavior of vLLM.
My comparison was based on the configurations I actually tested, including their practical hardware requirements. If you manage to reproduce those numbers on a G6 instance with vLLM, I'd be very interested in seeing the results, especially for agentic software development workloads.
After all, the purpose of the article is to compare practical deployment costs, not just theoretical memory efficiency. 🙂
You are right that those two numbers are different animals, and I should have said so in the comment instead of leaving it implied.
I cannot give you the G6 + vLLM number. I have no GPU here, so anything I wrote about resident memory on a real deployment would be a guess dressed as a measurement, which is exactly what you are pushing back on.
What I can separate, and measured this afternoon, is why a cross-model VRAM comparison at the same parameter scale is not comparing the same thing. I read the config of six checkpoints and computed the KV term from
num_hidden_layers,num_key_value_heads,head_dimand the dtype:So the 3.8B model costs 1.5x the cache of the 32B one per token, because it keeps 32 KV heads where Qwen3-32B keeps 8. If a benchmark table lists parameters and VRAM but not KV heads, the cache column can be off by that factor in either direction - and that is before any engine overhead.
Two things I would need to reproduce your numbers, and neither is on the model card: the
--max-model-lenyou served with, and how much of the GPU vLLM was allowed to claim for cache. If you still have those two flags, the floor is arithmetic and I will happily compute it for each of your configurations - the part I will not do is claim a G6 figure I have not seen.Thanks for clarifying! 😅
And that's precisely the distinction I wanted to highlight. The calculations are useful, but the actual deployment can introduce a completely different set of challenges.
I encountered quite a few surprises with vLLM, from model compatibility and memory requirements to simply getting certain configurations running.
I don't have those two flags available right now, but I appreciate the offer. My main interest remains the practical results, especially whether a configuration can be deployed reliably and economically for software development.
If you ever get access to suitable hardware and test it, I'd be genuinely interested in the results!