Aeza

Posted on Sep 3

Ollama vs vLLM for Concurrent Inference: How to Choose the Right Server

#devops #performance #machinelearning #ai

Choosing between Ollama and vLLM is not simply a matter of asking:

Which inference engine is faster?

The answer depends heavily on the workload.

A personal chat interface with one active user has very different requirements from a multi-user API handling dozens of concurrent generations.

For one workload, simple model management may matter more than maximum throughput.

For another, the deciding factors may be:

continuous batching
queue behavior
KV-cache utilization
TTFT
p99 latency
GPU saturation
overload handling

That means a useful Ollama vs vLLM comparison cannot be reduced to a single tokens/s number.

In this article, we'll build a practical benchmark methodology for comparing the two under real concurrent load.

By the end, you'll know:

when vLLM's batching can provide a measurable advantage
why Ollama may still be the better choice for smaller workloads
how to build a fair concurrency sweep
which latency and throughput metrics actually matter
how to detect queueing and saturation
how to choose the server that stays inside your SLO

The Short Answer

There is no universal concurrency level at which vLLM suddenly becomes better than Ollama.

vLLM begins to gain an advantage when enough requests are active or waiting for its scheduler to form useful batches while the GPU still has sufficient compute and memory capacity.

Conceptually:

Low concurrency
      ↓
Little opportunity for batching
      ↓
Ollama may be completely sufficient

Higher concurrency
      ↓
More simultaneous requests
      ↓
Continuous batching becomes useful
      ↓
vLLM can improve aggregate throughput

Too much concurrency
      ↓
Queue grows
      ↓
TTFT and p99 increase
      ↓
OOM / timeout / rejected requests

The real question is therefore not:

Which engine reaches the highest tokens per second?

It is:

Which engine delivers the required throughput while keeping latency, errors, memory usage, and operational complexity inside acceptable limits?

Start with the Workload, Not the Inference Engine

Before testing either server, define what the service actually needs to handle.

Three common scenarios illustrate why this matters.

Personal Chat

A personal chat usually has:

one active user
low concurrency
occasional model loading
relatively tolerant startup latency
no large server-side request queue

In this environment, maximum aggregate GPU throughput may not matter very much.

Operational simplicity often matters more.

Small Internal API

An internal API may receive:

several simultaneous users
short request bursts
varying context lengths
occasional concurrency spikes
moderate latency requirements

Both Ollama and vLLM may fit this workload.

The correct choice depends on the actual traffic curve.

High-Concurrency Inference Service

A public or heavily used internal service may have:

many simultaneous requests
sustained queues
strict TTFT targets
p95/p99 latency requirements
high GPU utilization targets
overload conditions that must be handled predictably

This is where continuous batching and queue-aware scheduling become much more important.

Record the Workload Before Benchmarking

At minimum, record:

Input tokens — determine prefill cost.
Output tokens — determine how long a request occupies generation resources.
Average concurrency — represents normal load.
Peak concurrency — reveals saturation behavior.
Streaming ratio — affects user-perceived latency.
Context length — directly affects KV-cache consumption.
Request bursts — determine queue behavior.
Target SLO — defines when performance becomes unacceptable.

Long contexts consume more KV-cache.

Long generations keep execution resources occupied for longer.

Average concurrency can also hide dangerous workload patterns.

For example, imagine a service that usually has four active requests.

That sounds moderate.

But if several long conversations overlap at the same moment, the system may suddenly experience:

KV-cache pressure
+
longer queue
+
higher TTFT
+
higher p99

even though average concurrency still looks reasonable.

Important: Higher server throughput does not automatically mean a better user experience.

A configuration can produce more total tokens per second while making individual requests noticeably slower.

Why Ollama Is Attractive for Smaller Workloads

Ollama combines several parts of the local model lifecycle into one workflow:

model downloading
model storage
Modelfiles
configuration
local HTTP API
model loading
model unloading

For a developer or small team, this can make deployment and maintenance significantly easier.

You can replace a model quickly, modify the system prompt, and control how long a model remains loaded without building a large serving stack.

Cold-start performance should still be evaluated separately from warm inference.

Important Ollama Parameters

In a warmed-up environment, Ollama behavior depends on settings such as:

OLLAMA_NUM_PARALLEL
OLLAMA_MAX_QUEUE
OLLAMA_MAX_LOADED_MODELS
keep_alive
context size

Parallel requests increase memory consumption because each active request may require additional context state.

A configuration that works perfectly with short prompts may behave very differently with longer contexts.

For example:

Short context
+
4 parallel requests
=
works normally

may become:

Long context
+
4 parallel requests
=
larger memory usage
+
queue growth
+
possible failure

So do not treat parallelism as an isolated setting.

Context length matters too.

Ollama's Main Advantage

For many workloads, Ollama's strongest advantage is not peak benchmark performance.

It is operational simplicity.

That can be extremely valuable for:

personal assistants
development environments
prototypes
small internal APIs
teams that frequently change models

If the workload does not create enough concurrency for continuous batching to matter, a more complicated serving stack may provide little practical benefit.

Why vLLM Benefits from Concurrency

vLLM is designed around server-side inference workloads.

Its scheduler can continuously adjust active batches as requests arrive and complete.

Instead of waiting for one static batch to fully finish, the engine can use newly available capacity for other requests.

This is commonly referred to as continuous batching.

Conceptually:

Request A ────────────────>
Request B      ────────────────>
Request C           ────────────────>

Scheduler continuously updates
the active batch as requests
arrive and finish.

This becomes useful when:

several requests are active
more requests are waiting
the GPU supports the required backend
enough memory remains available
batching improves accelerator utilization

With one short request, there may simply be nothing useful to batch.

Important vLLM Parameters

Record the server configuration with every benchmark.

Important settings include:

--max-num-seqs
--max-model-len
--gpu-memory-utilization
--dtype

Also record whether chunked prefill is enabled and how it is configured.

For multi-GPU deployments, record:

--tensor-parallel-size

Tensor parallelism can increase the effective memory available for larger models, but it also introduces communication between GPUs.

That communication is not free.

So:

More GPUs

does not automatically mean:

Linear performance scaling

Monitor vLLM's Server-Side Behavior

In addition to normal application metrics, monitor:

request queue length
KV-cache utilization
GPU memory utilization
preemption
rejected requests
OOM events

Both Ollama and vLLM should also be monitored for:

queue growth
memory pressure
timeouts
failed requests

The important difference is that vLLM exposes more of the machinery involved in high-concurrency serving.

Which Server Is Easier for One User?

For a single user, Ollama will usually be easier to operate.

The model lifecycle and API are integrated into one workflow.

vLLM becomes more attractive when you specifically require:

high concurrent throughput
continuous batching
detailed production metrics
specific GPU backends
larger request queues
a workload expected to grow rapidly

The decision should follow the workload rather than the popularity of either tool.

Make the Benchmark Fair

A benchmark only makes sense if both systems are processing equivalent work.

This sounds obvious.

In practice, it is easy to get wrong.

Try to keep the following equivalent:

model weights
model architecture
precision
quantization
tokenizer
chat template
generation parameters

Different weight formats may require different inference paths.

For example:

GGUF

and:

safetensors

do not automatically represent identical runtime behavior.

If exactly the same model build cannot be used by both engines, document that clearly.

The benchmark conclusion then applies only to the tested configurations.

Record Model Sources and Hashes

For reproducibility, save:

Model name
Model version
Weight source
File hash
Quantization
dtype

Otherwise, a future benchmark may use slightly different weights and produce results that appear inconsistent.

Keep Generation Parameters Identical

Use the same values for parameters such as:

max_tokens
temperature
top_p
stop
stream

For deterministic benchmark workloads, you may use:

temperature: 0

But matching generation settings is still not enough.

Verify Tokenization and Chat Templates

The same chat payload can produce different token sequences if the two servers use different tokenizers or templates.

For example:

{
  "messages": [
    {
      "role": "user",
      "content": "Explain TCP slow start."
    }
  ]
}

may be transformed differently before reaching the model.

That means two systems may appear to be benchmarking the same request while actually processing different input lengths.

Compare input-token counts before trusting the results.

If one server sees:

312 input tokens

and another sees:

356 input tokens

the workload is not identical.

Record the Benchmark Environment

Every run should include:

Inference server version
Model name
Model hash
GPU model
VRAM
CPU
RAM
Operating system
Driver version
Context length
Generation settings

A result such as:

8,200 output tokens/s

has little value without the environment that produced it.

Separate Cold and Warm Performance

Cold-start performance and steady-state inference answer different questions.

Measure them separately.

Cold Test

A cold test includes:

model loading
initialization
first request latency

This matters for workloads where models are frequently unloaded.

Warm Test

A warm test measures:

steady-state throughput
TTFT
token generation
latency
concurrency scaling

Do not combine cold and warm measurements into one average.

They describe different behavior.

Handle Prompt Caching Consistently

Prompt caching can significantly change benchmark results.

Either:

disable prompt caching on both systems

or:

use it consistently on both systems

Do not allow one server to reuse cached prompt state while the other processes every prompt from scratch.

Make Sure the Load Generator Is Not the Bottleneck

Your client must have enough:

CPU
network bandwidth
connection capacity

to generate the intended load.

Otherwise, you may accidentally benchmark the load generator rather than the inference server.

Monitor the client machine during heavy tests too.

Build a Concurrency Sweep

A simple benchmark matrix could look like this:

input_tokens:
  - 256
  - 2048

output_tokens: 128

concurrency:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32

repeats: 5
stream: true
temperature: 0

The exact values should match your real workload.

If production requests usually have:

8k input tokens
+
1k output tokens

then a benchmark using:

256 input tokens
+
64 output tokens

may tell you very little about production behavior.

Increase Concurrency Until Saturation

Start at:

concurrency = 1

Then increase gradually.

For every level:

warm up the server
execute several repeated runs
collect latency
collect throughput
collect errors
collect queue metrics
record GPU utilization
record GPU memory usage

Continue until you reach the first unacceptable condition.

Examples include:

OOM
Timeout
HTTP 503
Rejected request
p99 > SLO
GPU memory exhaustion
Queue growth without recovery

That point is much more useful than the absolute maximum throughput.

Closed-Loop vs Open-Loop Testing

There are two common ways to generate concurrent load.

They measure different things.

Closed-Loop Load

A closed-loop generator maintains a fixed number of active requests.

For example:

Concurrency = 16

Whenever one request finishes, another starts.

This answers questions such as:

How does the server behave with 16 continuously active clients?

Open-Loop Load

An open-loop generator sends requests at a defined arrival rate.

For example:

20 requests/second

Requests continue arriving regardless of how quickly previous requests finish.

This is useful for exposing queueing and overload behavior.

If the server can process:

15 requests/second

while receiving:

20 requests/second

then approximately:

5 requests/second

are being added to the queue.

The queue may continue growing until:

latency becomes unacceptable
requests time out
requests are rejected
memory is exhausted

Do not combine closed-loop and open-loop measurements into a single curve.

They answer different questions.

Throughput Alone Is Not Enough

Aggregate tokens per second describes server capacity.

It does not fully describe the user experience.

For every concurrency level, collect:

requests per second
output tokens per second
TTFT
ITL
TPOT
end-to-end latency
successful-request rate
queue length
memory usage

You need both throughput and latency.

TTFT: Time to First Token

TTFT measures:

Request sent
      ↓
First actual generated token

Do not automatically treat the first streaming chunk as the first token.

Some APIs may send metadata or other fields before actual generated text appears.

Measure the first real generated output.

TTFT strongly affects perceived responsiveness in chat applications.

A user may tolerate a long generation if text begins appearing quickly.

A long silent wait before the first token often feels much worse.

ITL: Inter-Token Latency

ITL describes the delay between generated tokens.

Conceptually:

Token 1
   ↓ 40 ms
Token 2
   ↓ 43 ms
Token 3

This describes how smoothly the response streams after generation begins.

If the client measures network chunks rather than actual tokens, call the measurement:

inter-chunk latency

instead.

TPOT: Time per Output Token

TPOT can be represented conceptually as:

generation time after first token
---------------------------------
remaining output tokens

It helps separate initial responsiveness from steady-state generation speed.

Measure End-to-End Latency

End-to-end latency measures:

Request sent
      ↓
Queueing
      ↓
Prefill
      ↓
Generation
      ↓
Final response

This is often the metric most directly connected to the total user wait time.

Always Measure Tail Latency

Do not report only the average.

At minimum, report:

p50
p95
p99

Why?

Because batching and queueing may affect a minority of requests much more severely than the median request.

Suppose:

p50 = 1.8 s
p95 = 4.2 s
p99 = 12.7 s

An average of:

2.4 s

would hide an important part of the user experience.

For production inference APIs, p99 often matters much more than the mean.

Understand the Throughput-Latency Trade-Off

Continuous batching can improve aggregate throughput.

But that does not mean every request becomes faster.

Consider:

Concurrency 4
Throughput: 3,000 tok/s
p99 TTFT: 400 ms

and:

Concurrency 32
Throughput: 7,500 tok/s
p99 TTFT: 4.8 s

The second configuration produces much more aggregate throughput.

But it may violate the application's SLO.

If your requirement is:

p99 TTFT < 2 seconds

then the higher-throughput operating point is not acceptable.

The best benchmark result is therefore not necessarily the highest point on the throughput curve.

It is the highest useful operating point inside the SLO.

Measure Queueing Explicitly

Queueing is one of the most important parts of concurrent inference.

Imagine requests arriving faster than they can be processed:

Arrival rate
     ↓
[Request]
[Request]
[Request]
[Request]
     ↓
Server

If the server cannot keep up:

Queue length ↑
TTFT ↑
p99 ↑
timeouts ↑

This is why a large or unlimited queue does not solve overload.

It only changes:

rejected request

into:

very slow request

Backpressure and overload behavior should therefore be part of the benchmark.

Test Fairness Between Short and Long Requests

A realistic workload rarely contains identical prompts.

Create a mixed workload with:

short inputs
long inputs
short outputs
long outputs

Then check whether short requests are excessively delayed by long ones.

For example:

Request A:
256 input tokens
64 output tokens

Request B:
8,000 input tokens
2,000 output tokens

A scheduler should not allow a large request to make every small request unreasonably slow.

Also observe whether large prefill operations create visible pauses for requests already generating tokens.

Store Raw Benchmark Data

A professional benchmark should produce reproducible artifacts.

For example:

benchmark/
├── requests.jsonl
├── run.json
├── results.csv
├── server.log
└── gpu.csv

A minimal result schema could look like:

server,run,concurrency,input_tokens,output_tokens,ttft_ms,itl_ms,e2e_ms,status,error

You may also want:

timestamp,gpu_utilization,gpu_memory_used,queue_length

Screenshots alone are not sufficient.

Raw data allows you to:

recalculate percentiles
change aggregation windows
inspect outliers
compare future server versions
reproduce the analysis

Graph Throughput and Latency Together

A useful benchmark graph uses concurrency on the X-axis.

Then plot metrics such as:

Requests/s
Output tokens/s
TTFT p50
TTFT p95
TTFT p99
End-to-end p99
Queue length

The important shape often looks like:

Throughput
    /
   /
  /
 /_____
       \
        saturation

Latency
        /
       /
______/

At low concurrency:

throughput increases
latency remains stable

Near saturation:

throughput growth slows
queueing increases
p99 rises rapidly

That knee in the curve is often more important than the absolute maximum.

Compare API Compatibility Before Migrating

Both Ollama and vLLM may expose OpenAI-compatible interfaces.

But:

OpenAI-compatible

does not mean:

behaviorally identical

Verify the features your application actually uses.

For example:

Chat Completions
streaming
structured output
embeddings
tool calling
generation parameters
stop sequences
error responses
token limits

Do not discover incompatibilities after production traffic has already been switched.

Production Readiness Is More Than the Inference Engine

A working HTTP endpoint is not automatically a production-ready inference service.

The surrounding platform may need to provide:

TLS
Authentication
Rate limiting
Logging
Monitoring
Health checks

A health check should also distinguish between:

Server process is running

and:

Model is loaded and ready to serve

A listening TCP port does not guarantee that the next request will meet the latency SLO.

Compare Observability

For concurrent production workloads, observability becomes part of the product decision.

Useful vLLM signals may include:

queue state
KV-cache utilization
TTFT
request latency
memory pressure
preemption

Ollama also exposes timing information around:

model loading
prompt evaluation
generation

Regardless of the server, collect the same client-side metrics.

Client-side instrumentation provides the most direct view of what users actually experience.

Test Upgrades Before Production

Inference-server updates can change:

memory consumption
scheduler behavior
inference backends
configuration defaults
chat templates
model compatibility

Before upgrading production, replay a realistic workload against the new version.

Keep the previous:

Container image or package
Configuration
Model hash

available for rollback.

A benchmark that is valid for one version should not automatically be assumed valid for the next.

Use a Shared API Layer to Keep Migration Easy

If possible, avoid coupling application logic directly to one inference engine.

Keep:

model names
generation parameters
timeout logic
error handling

behind an internal adapter.

Then make the server endpoint configurable.

Conceptually:

Application
     ↓
Internal inference adapter
     ↓
Ollama OR vLLM

This makes it much easier to:

benchmark alternatives
canary a new server
roll back
migrate later

Canary the Migration

Do not send 100% of production traffic to the new server immediately.

Start with a small percentage.

For example:

5% new server
95% old server

Measure:

latency
errors
throughput
queue growth
memory usage

Then increase gradually if the results remain inside the SLO.

Keep the old server available for rapid rollback.

A Practical Benchmark Workflow

Here is a reusable process.

Step 1: Define the Real Workload

Record:

Input length
Output length
Average concurrency
Peak concurrency
Request rate
Streaming ratio
SLO

Step 2: Normalize the Models

Match:

Weights
Precision
Quantization
Tokenizer
Chat template

Step 3: Match Generation Settings

Use identical:

max_tokens
temperature
top_p
stop
stream

Step 4: Record the Environment

Save:

Server version
Model hash
GPU
VRAM
CPU
RAM
OS
Driver

Step 5: Warm Up the Server

Separate cold-start measurements from steady-state results.

Step 6: Run the Concurrency Sweep

For example:

Step 7: Measure Latency and Throughput

Collect:

Requests/s
Tokens/s
TTFT
ITL
TPOT
E2E latency

Step 8: Measure Tail Latency

Calculate:

p50
p95
p99

Step 9: Watch Queue and GPU State

Collect:

Queue length
GPU utilization
GPU memory
KV-cache
OOM events

Step 10: Stop at the SLO Boundary

The benchmark should stop being considered "better" once:

p99 > SLO

even if tokens per second continue increasing.

Starting Recommendation by Scenario

Single User

Start with:

Ollama

Confirm:

the model is supported
cold start is acceptable
memory consumption is acceptable
latency is acceptable

The simplicity may be worth more than extra concurrent throughput.

Small Internal API

Start with:

Ollama or vLLM

Benchmark:

request bursts
moderate concurrency
queue behavior
p95/p99
overload handling

Either system may be the better fit.

High-Concurrency GPU Service

Start with:

vLLM

Then confirm that:

throughput improves with concurrency
p99 remains inside the SLO
TTFT remains acceptable
KV-cache remains healthy
queue growth remains controlled
OOM does not occur

Do not assume the answer before measuring it.

Practical Checklist

Before declaring one server faster than the other:

[ ] Use comparable model weights
[ ] Match precision and quantization
[ ] Match tokenizer behavior
[ ] Match chat templates
[ ] Match generation parameters
[ ] Record software versions
[ ] Record GPU and VRAM
[ ] Separate cold and warm tests
[ ] Use realistic input lengths
[ ] Use realistic output lengths
[ ] Sweep concurrency
[ ] Test open-loop and closed-loop load separately
[ ] Measure requests per second
[ ] Measure output tokens per second
[ ] Measure TTFT
[ ] Measure ITL or TPOT
[ ] Measure end-to-end latency
[ ] Calculate p50/p95/p99
[ ] Monitor queue length
[ ] Monitor GPU utilization
[ ] Monitor GPU memory
[ ] Record OOM and rejected requests
[ ] Preserve raw benchmark data
[ ] Evaluate results against the SLO

Final Takeaway

Ollama and vLLM solve overlapping problems, but they optimize for different priorities.

Ollama is attractive when you value:

fast setup
simple model management
a straightforward local lifecycle
moderate concurrency
lower operational complexity

vLLM becomes more attractive when you need:

continuous batching
higher concurrent throughput
efficient KV-cache management
high GPU utilization
larger request queues
detailed production observability

The correct comparison is not:

Which server produces the most tokens per second?

It is:

Which server delivers the throughput you need while keeping TTFT, p99 latency, queueing, errors, memory usage, and operational complexity inside acceptable limits?

If vLLM produces higher aggregate throughput but pushes p99 beyond your SLO, that is not a win.

If Ollama already handles your real traffic comfortably, migrating to a more complex serving stack may provide little practical benefit.

Benchmark the workload you actually have.

Find the saturation point.

Measure the queue.

Measure the tails.

Then choose the simplest system that satisfies the SLO.

Top comments (1)

Max Quimby • Sep 3

The framing of "start with the workload, not the engine" is the part most benchmarks skip, and it's the part that actually decides the outcome. One thing I'd add from running both in production: the crossover point isn't just about concurrency count, it's about arrival-time distribution. vLLM's continuous batching shines when requests arrive close enough together that the scheduler can keep partial batches full, but a bursty workload with idle gaps can leave it looking no better than Ollama because the batches never fill. We ended up plotting p99 against sustained RPS rather than raw concurrency, because "50 concurrent" from a load tester and "50 concurrent" from real traffic behave very differently once you factor in generation-length variance. The other thing that bit us: KV-cache pressure from long-context requests silently caps your effective batch size well before you hit GPU compute limits, so an OOM at "low" concurrency is usually a context-length problem in disguise. Curious whether your sweep held output length fixed or sampled it — that variable alone can flip the conclusion.