DEV Community: Aeza

Ollama vs vLLM for Concurrent Inference: How to Choose the Right Server

Aeza — Thu, 03 Sep 2026 14:13:06 +0000

Choosing between Ollama and vLLM is not simply a matter of asking:

Which inference engine is faster?

The answer depends heavily on the workload.

A personal chat interface with one active user has very different requirements from a multi-user API handling dozens of concurrent generations.

For one workload, simple model management may matter more than maximum throughput.

For another, the deciding factors may be:

continuous batching
queue behavior
KV-cache utilization
TTFT
p99 latency
GPU saturation
overload handling

That means a useful Ollama vs vLLM comparison cannot be reduced to a single tokens/s number.

In this article, we'll build a practical benchmark methodology for comparing the two under real concurrent load.

By the end, you'll know:

when vLLM's batching can provide a measurable advantage
why Ollama may still be the better choice for smaller workloads
how to build a fair concurrency sweep
which latency and throughput metrics actually matter
how to detect queueing and saturation
how to choose the server that stays inside your SLO

The Short Answer

There is no universal concurrency level at which vLLM suddenly becomes better than Ollama.

vLLM begins to gain an advantage when enough requests are active or waiting for its scheduler to form useful batches while the GPU still has sufficient compute and memory capacity.

Conceptually:

Low concurrency
      ↓
Little opportunity for batching
      ↓
Ollama may be completely sufficient

Higher concurrency
      ↓
More simultaneous requests
      ↓
Continuous batching becomes useful
      ↓
vLLM can improve aggregate throughput

Too much concurrency
      ↓
Queue grows
      ↓
TTFT and p99 increase
      ↓
OOM / timeout / rejected requests

The real question is therefore not:

Which engine reaches the highest tokens per second?

It is:

Which engine delivers the required throughput while keeping latency, errors, memory usage, and operational complexity inside acceptable limits?

Start with the Workload, Not the Inference Engine

Before testing either server, define what the service actually needs to handle.

Three common scenarios illustrate why this matters.

Personal Chat

A personal chat usually has:

one active user
low concurrency
occasional model loading
relatively tolerant startup latency
no large server-side request queue

In this environment, maximum aggregate GPU throughput may not matter very much.

Operational simplicity often matters more.

Small Internal API

An internal API may receive:

several simultaneous users
short request bursts
varying context lengths
occasional concurrency spikes
moderate latency requirements

Both Ollama and vLLM may fit this workload.

The correct choice depends on the actual traffic curve.

High-Concurrency Inference Service

A public or heavily used internal service may have:

many simultaneous requests
sustained queues
strict TTFT targets
p95/p99 latency requirements
high GPU utilization targets
overload conditions that must be handled predictably

This is where continuous batching and queue-aware scheduling become much more important.

Record the Workload Before Benchmarking

At minimum, record:

Input tokens — determine prefill cost.
Output tokens — determine how long a request occupies generation resources.
Average concurrency — represents normal load.
Peak concurrency — reveals saturation behavior.
Streaming ratio — affects user-perceived latency.
Context length — directly affects KV-cache consumption.
Request bursts — determine queue behavior.
Target SLO — defines when performance becomes unacceptable.

Long contexts consume more KV-cache.

Long generations keep execution resources occupied for longer.

Average concurrency can also hide dangerous workload patterns.

For example, imagine a service that usually has four active requests.

That sounds moderate.

But if several long conversations overlap at the same moment, the system may suddenly experience:

KV-cache pressure
+
longer queue
+
higher TTFT
+
higher p99

even though average concurrency still looks reasonable.

Important: Higher server throughput does not automatically mean a better user experience.

A configuration can produce more total tokens per second while making individual requests noticeably slower.

Why Ollama Is Attractive for Smaller Workloads

Ollama combines several parts of the local model lifecycle into one workflow:

model downloading
model storage
Modelfiles
configuration
local HTTP API
model loading
model unloading

For a developer or small team, this can make deployment and maintenance significantly easier.

You can replace a model quickly, modify the system prompt, and control how long a model remains loaded without building a large serving stack.

Cold-start performance should still be evaluated separately from warm inference.

Important Ollama Parameters

In a warmed-up environment, Ollama behavior depends on settings such as:

OLLAMA_NUM_PARALLEL
OLLAMA_MAX_QUEUE
OLLAMA_MAX_LOADED_MODELS
keep_alive
context size

Parallel requests increase memory consumption because each active request may require additional context state.

A configuration that works perfectly with short prompts may behave very differently with longer contexts.

For example:

Short context
+
4 parallel requests
=
works normally

may become:

Long context
+
4 parallel requests
=
larger memory usage
+
queue growth
+
possible failure

So do not treat parallelism as an isolated setting.

Context length matters too.

Ollama's Main Advantage

For many workloads, Ollama's strongest advantage is not peak benchmark performance.

It is operational simplicity.

That can be extremely valuable for:

personal assistants
development environments
prototypes
small internal APIs
teams that frequently change models

If the workload does not create enough concurrency for continuous batching to matter, a more complicated serving stack may provide little practical benefit.

Why vLLM Benefits from Concurrency

vLLM is designed around server-side inference workloads.

Its scheduler can continuously adjust active batches as requests arrive and complete.

Instead of waiting for one static batch to fully finish, the engine can use newly available capacity for other requests.

This is commonly referred to as continuous batching.

Conceptually:

Request A ────────────────>
Request B      ────────────────>
Request C           ────────────────>

Scheduler continuously updates
the active batch as requests
arrive and finish.

This becomes useful when:

several requests are active
more requests are waiting
the GPU supports the required backend
enough memory remains available
batching improves accelerator utilization

With one short request, there may simply be nothing useful to batch.

Important vLLM Parameters

Record the server configuration with every benchmark.

Important settings include:

--max-num-seqs
--max-model-len
--gpu-memory-utilization
--dtype

Also record whether chunked prefill is enabled and how it is configured.

For multi-GPU deployments, record:

--tensor-parallel-size

Tensor parallelism can increase the effective memory available for larger models, but it also introduces communication between GPUs.

That communication is not free.

So:

More GPUs

does not automatically mean:

Linear performance scaling

Monitor vLLM's Server-Side Behavior

In addition to normal application metrics, monitor:

request queue length
KV-cache utilization
GPU memory utilization
preemption
rejected requests
OOM events

Both Ollama and vLLM should also be monitored for:

queue growth
memory pressure
timeouts
failed requests

The important difference is that vLLM exposes more of the machinery involved in high-concurrency serving.

Which Server Is Easier for One User?

For a single user, Ollama will usually be easier to operate.

The model lifecycle and API are integrated into one workflow.

vLLM becomes more attractive when you specifically require:

high concurrent throughput
continuous batching
detailed production metrics
specific GPU backends
larger request queues
a workload expected to grow rapidly

The decision should follow the workload rather than the popularity of either tool.

Make the Benchmark Fair

A benchmark only makes sense if both systems are processing equivalent work.

This sounds obvious.

In practice, it is easy to get wrong.

Try to keep the following equivalent:

model weights
model architecture
precision
quantization
tokenizer
chat template
generation parameters

Different weight formats may require different inference paths.

For example:

GGUF

and:

safetensors

do not automatically represent identical runtime behavior.

If exactly the same model build cannot be used by both engines, document that clearly.

The benchmark conclusion then applies only to the tested configurations.

Record Model Sources and Hashes

For reproducibility, save:

Model name
Model version
Weight source
File hash
Quantization
dtype

Otherwise, a future benchmark may use slightly different weights and produce results that appear inconsistent.

Keep Generation Parameters Identical

Use the same values for parameters such as:

max_tokens
temperature
top_p
stop
stream

For deterministic benchmark workloads, you may use:

temperature: 0

But matching generation settings is still not enough.

Verify Tokenization and Chat Templates

The same chat payload can produce different token sequences if the two servers use different tokenizers or templates.

For example:

{
  "messages": [
    {
      "role": "user",
      "content": "Explain TCP slow start."
    }
  ]
}

may be transformed differently before reaching the model.

That means two systems may appear to be benchmarking the same request while actually processing different input lengths.

Compare input-token counts before trusting the results.

If one server sees:

312 input tokens

and another sees:

356 input tokens

the workload is not identical.

Record the Benchmark Environment

Every run should include:

Inference server version
Model name
Model hash
GPU model
VRAM
CPU
RAM
Operating system
Driver version
Context length
Generation settings

A result such as:

8,200 output tokens/s

has little value without the environment that produced it.

Separate Cold and Warm Performance

Cold-start performance and steady-state inference answer different questions.

Measure them separately.

Cold Test

A cold test includes:

model loading
initialization
first request latency

This matters for workloads where models are frequently unloaded.

Warm Test

A warm test measures:

steady-state throughput
TTFT
token generation
latency
concurrency scaling

Do not combine cold and warm measurements into one average.

They describe different behavior.

Handle Prompt Caching Consistently

Prompt caching can significantly change benchmark results.

Either:

disable prompt caching on both systems

or:

use it consistently on both systems

Do not allow one server to reuse cached prompt state while the other processes every prompt from scratch.

Make Sure the Load Generator Is Not the Bottleneck

Your client must have enough:

CPU
network bandwidth
connection capacity

to generate the intended load.

Otherwise, you may accidentally benchmark the load generator rather than the inference server.

Monitor the client machine during heavy tests too.

Build a Concurrency Sweep

A simple benchmark matrix could look like this:

input_tokens:
  - 256
  - 2048

output_tokens: 128

concurrency:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32

repeats: 5
stream: true
temperature: 0

The exact values should match your real workload.

If production requests usually have:

8k input tokens
+
1k output tokens

then a benchmark using:

256 input tokens
+
64 output tokens

may tell you very little about production behavior.

Increase Concurrency Until Saturation

Start at:

concurrency = 1

Then increase gradually.

For every level:

warm up the server
execute several repeated runs
collect latency
collect throughput
collect errors
collect queue metrics
record GPU utilization
record GPU memory usage

Continue until you reach the first unacceptable condition.

Examples include:

OOM
Timeout
HTTP 503
Rejected request
p99 > SLO
GPU memory exhaustion
Queue growth without recovery

That point is much more useful than the absolute maximum throughput.

Closed-Loop vs Open-Loop Testing

There are two common ways to generate concurrent load.

They measure different things.

Closed-Loop Load

A closed-loop generator maintains a fixed number of active requests.

For example:

Concurrency = 16

Whenever one request finishes, another starts.

This answers questions such as:

How does the server behave with 16 continuously active clients?

Open-Loop Load

An open-loop generator sends requests at a defined arrival rate.

For example:

20 requests/second

Requests continue arriving regardless of how quickly previous requests finish.

This is useful for exposing queueing and overload behavior.

If the server can process:

15 requests/second

while receiving:

20 requests/second

then approximately:

5 requests/second

are being added to the queue.

The queue may continue growing until:

latency becomes unacceptable
requests time out
requests are rejected
memory is exhausted

Do not combine closed-loop and open-loop measurements into a single curve.

They answer different questions.

Throughput Alone Is Not Enough

Aggregate tokens per second describes server capacity.

It does not fully describe the user experience.

For every concurrency level, collect:

requests per second
output tokens per second
TTFT
ITL
TPOT
end-to-end latency
successful-request rate
queue length
memory usage

You need both throughput and latency.

TTFT: Time to First Token

TTFT measures:

Request sent
      ↓
First actual generated token

Do not automatically treat the first streaming chunk as the first token.

Some APIs may send metadata or other fields before actual generated text appears.

Measure the first real generated output.

TTFT strongly affects perceived responsiveness in chat applications.

A user may tolerate a long generation if text begins appearing quickly.

A long silent wait before the first token often feels much worse.

ITL: Inter-Token Latency

ITL describes the delay between generated tokens.

Conceptually:

Token 1
   ↓ 40 ms
Token 2
   ↓ 43 ms
Token 3

This describes how smoothly the response streams after generation begins.

If the client measures network chunks rather than actual tokens, call the measurement:

inter-chunk latency

instead.

TPOT: Time per Output Token

TPOT can be represented conceptually as:

generation time after first token
---------------------------------
remaining output tokens

It helps separate initial responsiveness from steady-state generation speed.

Measure End-to-End Latency

End-to-end latency measures:

Request sent
      ↓
Queueing
      ↓
Prefill
      ↓
Generation
      ↓
Final response

This is often the metric most directly connected to the total user wait time.

Always Measure Tail Latency

Do not report only the average.

At minimum, report:

p50
p95
p99

Why?

Because batching and queueing may affect a minority of requests much more severely than the median request.

Suppose:

p50 = 1.8 s
p95 = 4.2 s
p99 = 12.7 s

An average of:

2.4 s

would hide an important part of the user experience.

For production inference APIs, p99 often matters much more than the mean.

Understand the Throughput-Latency Trade-Off

Continuous batching can improve aggregate throughput.

But that does not mean every request becomes faster.

Consider:

Concurrency 4
Throughput: 3,000 tok/s
p99 TTFT: 400 ms

and:

Concurrency 32
Throughput: 7,500 tok/s
p99 TTFT: 4.8 s

The second configuration produces much more aggregate throughput.

But it may violate the application's SLO.

If your requirement is:

p99 TTFT < 2 seconds

then the higher-throughput operating point is not acceptable.

The best benchmark result is therefore not necessarily the highest point on the throughput curve.

It is the highest useful operating point inside the SLO.

Measure Queueing Explicitly

Queueing is one of the most important parts of concurrent inference.

Imagine requests arriving faster than they can be processed:

Arrival rate
     ↓
[Request]
[Request]
[Request]
[Request]
     ↓
Server

If the server cannot keep up:

Queue length ↑
TTFT ↑
p99 ↑
timeouts ↑

This is why a large or unlimited queue does not solve overload.

It only changes:

rejected request

into:

very slow request

Backpressure and overload behavior should therefore be part of the benchmark.

Test Fairness Between Short and Long Requests

A realistic workload rarely contains identical prompts.

Create a mixed workload with:

short inputs
long inputs
short outputs
long outputs

Then check whether short requests are excessively delayed by long ones.

For example:

Request A:
256 input tokens
64 output tokens

Request B:
8,000 input tokens
2,000 output tokens

A scheduler should not allow a large request to make every small request unreasonably slow.

Also observe whether large prefill operations create visible pauses for requests already generating tokens.

Store Raw Benchmark Data

A professional benchmark should produce reproducible artifacts.

For example:

benchmark/
├── requests.jsonl
├── run.json
├── results.csv
├── server.log
└── gpu.csv

A minimal result schema could look like:

server,run,concurrency,input_tokens,output_tokens,ttft_ms,itl_ms,e2e_ms,status,error

You may also want:

timestamp,gpu_utilization,gpu_memory_used,queue_length

Screenshots alone are not sufficient.

Raw data allows you to:

recalculate percentiles
change aggregation windows
inspect outliers
compare future server versions
reproduce the analysis

Graph Throughput and Latency Together

A useful benchmark graph uses concurrency on the X-axis.

Then plot metrics such as:

Requests/s
Output tokens/s
TTFT p50
TTFT p95
TTFT p99
End-to-end p99
Queue length

The important shape often looks like:

Throughput
    /
   /
  /
 /_____
       \
        saturation

Latency
        /
       /
______/

At low concurrency:

throughput increases
latency remains stable

Near saturation:

throughput growth slows
queueing increases
p99 rises rapidly

That knee in the curve is often more important than the absolute maximum.

Compare API Compatibility Before Migrating

Both Ollama and vLLM may expose OpenAI-compatible interfaces.

But:

OpenAI-compatible

does not mean:

behaviorally identical

Verify the features your application actually uses.

For example:

Chat Completions
streaming
structured output
embeddings
tool calling
generation parameters
stop sequences
error responses
token limits

Do not discover incompatibilities after production traffic has already been switched.

Production Readiness Is More Than the Inference Engine

A working HTTP endpoint is not automatically a production-ready inference service.

The surrounding platform may need to provide:

TLS
Authentication
Rate limiting
Logging
Monitoring
Health checks

A health check should also distinguish between:

Server process is running

and:

Model is loaded and ready to serve

A listening TCP port does not guarantee that the next request will meet the latency SLO.

Compare Observability

For concurrent production workloads, observability becomes part of the product decision.

Useful vLLM signals may include:

queue state
KV-cache utilization
TTFT
request latency
memory pressure
preemption

Ollama also exposes timing information around:

model loading
prompt evaluation
generation

Regardless of the server, collect the same client-side metrics.

Client-side instrumentation provides the most direct view of what users actually experience.

Test Upgrades Before Production

Inference-server updates can change:

memory consumption
scheduler behavior
inference backends
configuration defaults
chat templates
model compatibility

Before upgrading production, replay a realistic workload against the new version.

Keep the previous:

Container image or package
Configuration
Model hash

available for rollback.

A benchmark that is valid for one version should not automatically be assumed valid for the next.

Use a Shared API Layer to Keep Migration Easy

If possible, avoid coupling application logic directly to one inference engine.

Keep:

model names
generation parameters
timeout logic
error handling

behind an internal adapter.

Then make the server endpoint configurable.

Conceptually:

Application
     ↓
Internal inference adapter
     ↓
Ollama OR vLLM

This makes it much easier to:

benchmark alternatives
canary a new server
roll back
migrate later

Canary the Migration

Do not send 100% of production traffic to the new server immediately.

Start with a small percentage.

For example:

5% new server
95% old server

Measure:

latency
errors
throughput
queue growth
memory usage

Then increase gradually if the results remain inside the SLO.

Keep the old server available for rapid rollback.

A Practical Benchmark Workflow

Here is a reusable process.

Step 1: Define the Real Workload

Record:

Input length
Output length
Average concurrency
Peak concurrency
Request rate
Streaming ratio
SLO

Step 2: Normalize the Models

Match:

Weights
Precision
Quantization
Tokenizer
Chat template

Step 3: Match Generation Settings

Use identical:

max_tokens
temperature
top_p
stop
stream

Step 4: Record the Environment

Save:

Server version
Model hash
GPU
VRAM
CPU
RAM
OS
Driver

Step 5: Warm Up the Server

Separate cold-start measurements from steady-state results.

Step 6: Run the Concurrency Sweep

For example:

Step 7: Measure Latency and Throughput

Collect:

Requests/s
Tokens/s
TTFT
ITL
TPOT
E2E latency

Step 8: Measure Tail Latency

Calculate:

p50
p95
p99

Step 9: Watch Queue and GPU State

Collect:

Queue length
GPU utilization
GPU memory
KV-cache
OOM events

Step 10: Stop at the SLO Boundary

The benchmark should stop being considered "better" once:

p99 > SLO

even if tokens per second continue increasing.

Starting Recommendation by Scenario

Single User

Start with:

Ollama

Confirm:

the model is supported
cold start is acceptable
memory consumption is acceptable
latency is acceptable

The simplicity may be worth more than extra concurrent throughput.

Small Internal API

Start with:

Ollama or vLLM

Benchmark:

request bursts
moderate concurrency
queue behavior
p95/p99
overload handling

Either system may be the better fit.

High-Concurrency GPU Service

Start with:

vLLM

Then confirm that:

throughput improves with concurrency
p99 remains inside the SLO
TTFT remains acceptable
KV-cache remains healthy
queue growth remains controlled
OOM does not occur

Do not assume the answer before measuring it.

Practical Checklist

Before declaring one server faster than the other:

[ ] Use comparable model weights
[ ] Match precision and quantization
[ ] Match tokenizer behavior
[ ] Match chat templates
[ ] Match generation parameters
[ ] Record software versions
[ ] Record GPU and VRAM
[ ] Separate cold and warm tests
[ ] Use realistic input lengths
[ ] Use realistic output lengths
[ ] Sweep concurrency
[ ] Test open-loop and closed-loop load separately
[ ] Measure requests per second
[ ] Measure output tokens per second
[ ] Measure TTFT
[ ] Measure ITL or TPOT
[ ] Measure end-to-end latency
[ ] Calculate p50/p95/p99
[ ] Monitor queue length
[ ] Monitor GPU utilization
[ ] Monitor GPU memory
[ ] Record OOM and rejected requests
[ ] Preserve raw benchmark data
[ ] Evaluate results against the SLO

Final Takeaway

Ollama and vLLM solve overlapping problems, but they optimize for different priorities.

Ollama is attractive when you value:

fast setup
simple model management
a straightforward local lifecycle
moderate concurrency
lower operational complexity

vLLM becomes more attractive when you need:

continuous batching
higher concurrent throughput
efficient KV-cache management
high GPU utilization
larger request queues
detailed production observability

The correct comparison is not:

Which server produces the most tokens per second?

It is:

Which server delivers the throughput you need while keeping TTFT, p99 latency, queueing, errors, memory usage, and operational complexity inside acceptable limits?

If vLLM produces higher aggregate throughput but pushes p99 beyond your SLO, that is not a win.

If Ollama already handles your real traffic comfortably, migrating to a more complex serving stack may provide little practical benefit.

Benchmark the workload you actually have.

Find the saturation point.

Measure the queue.

Measure the tails.

Then choose the simplest system that satisfies the SLO.

CPU Steal Time Explained: What %st Really Means on a Virtual Machine

Aeza — Thu, 03 Sep 2026 14:11:28 +0000

CPU steal time is one of those Linux metrics that looks simple until you actually need to diagnose a performance problem.

You open top, notice that %st is above zero, and the obvious conclusion seems to be:

The host is overloaded.

Or perhaps:

Another VM on the server is stealing my CPU.

Both explanations are possible.

Neither can be proven from %st alone.

A useful diagnosis requires at least three things:

the steal-time measurement itself
CPU pressure inside the VM
measurable impact on the application

In practice, that means correlating %st with metrics such as the run queue, p95/p99 latency, throughput, per-vCPU utilization, and cgroup throttling.

In this article, we'll build a practical workflow for doing exactly that.

By the end, you'll know:

what %st actually measures
how to collect steal-time data correctly
when high steal time is worth investigating
how to separate host contention from guest CPU saturation
what evidence to collect before contacting your hosting provider
how to monitor steal time continuously

The Short Answer

%st represents time during which a virtual CPU was ready to run but did not receive physical CPU time from the host.

But a high value only becomes operationally meaningful when it is:

abnormal for that specific VM
sustained across multiple measurement intervals
correlated with CPU pressure or application degradation

A single spike is not enough.

A much stronger signal looks like this:

Elevated %st
+
Growing CPU run queue
+
Increasing p99 latency
+
Falling throughput

That combination is worth investigating.

What Does `%st` Mean in `top` and `vmstat`?

In a virtual machine, %st — or steal time — represents the percentage of a measurement interval during which a vCPU was ready to execute but did not receive physical CPU time.

You may see it in top:

%Cpu(s): 12.7 us, 6.4 sy, 0.0 ni, 72.2 id,
          3.1 wa, 0.0 hi, 0.0 si, 5.6 st

In this example:

5.6 st

means that approximately 5.6% of the measured CPU interval was accounted as steal time.

The important part is understanding why this can happen.

A VM does not control the physical CPU directly.

There is another scheduling layer underneath it.

A Virtual Machine Has Two CPU Schedulers

To understand steal time, separate the guest scheduler from the host scheduler.

The guest scheduler

The operating system inside the VM decides:

Which process should run on this vCPU?

For example:

nginx
postgres
python
node

may all compete for CPU time inside the guest.

The host scheduler

The physical host or hypervisor decides something different:

When does this vCPU get access to an actual physical CPU?

That gives us two scheduling layers:

Application process
        ↓
Guest OS scheduler
        ↓
       vCPU
        ↓
Host / hypervisor scheduler
        ↓
Physical CPU

Suppose the guest has a runnable process.

The guest scheduler wants to execute it.

The vCPU is therefore ready.

But the host scheduler does not immediately schedule that vCPU onto a physical core.

The guest waits.

That waiting time can be recorded as steal time.

Conceptually:

Guest has runnable work
        ↓
vCPU is ready
        ↓
Physical CPU is not assigned
        ↓
Guest waits
        ↓
Time is recorded as steal

This is why %st applies to the vCPU as a whole rather than to one particular process.

Do Not Confuse `%st` with Other CPU Metrics

Several CPU metrics can indicate performance pressure, but they describe different things.

`%usr`

Time spent executing user-space code.

Examples:

application logic
JavaScript
Python
database execution
compression
model inference

`%sys`

Time spent executing kernel code.

`%idle`

CPU time during which the guest had no runnable work.

`%iowait`

Time associated with waiting while I/O is pending.

`%st`

Time during which a runnable virtual CPU did not receive physical CPU time.

A useful mental model is:

%usr / %sys → guest is actively using CPU

%iowait     → work is waiting around I/O

%st         → guest wanted CPU but the host did not provide it

%iowait is therefore not another form of steal time.

CPU Throttling Is Also Not the Same as Steal Time

A process can be CPU-limited inside the VM even when %st is zero.

One common cause is a cgroup CPU quota.

With cgroup v2, inspect the configured quota:

cat /sys/fs/cgroup/cpu.max

Then inspect CPU statistics:

cat /sys/fs/cgroup/cpu.stat

Useful counters can include:

nr_throttled
throttled_usec

If these counters increase while the workload is running, the process or cgroup may be hitting a CPU quota.

That means the CPU restriction originates inside the guest environment, not necessarily from hypervisor scheduling.

So this situation is possible:

%st = 0

while application latency still increases because the process is being throttled.

This distinction matters when trying to determine whether the VM itself is overloaded or whether the underlying platform is contributing to the problem.

A Zero `%st` Does Not Prove the Host Is Healthy

Another common mistake is assuming:

%st = 0

means:

No host-side scheduling problems exist

That conclusion is too strong.

A zero value only tells you that the guest kernel did not account for steal time during that particular interval.

The availability of steal-time accounting depends on factors such as:

hypervisor
guest kernel
architecture
platform configuration

KVM can expose steal time on supported systems, but the guest still needs access to the relevant accounting information.

Other forms of host-side delay may also exist without appearing as %st.

So think of zero as:

No steal time was recorded here.

Not:

The physical host is definitely healthy.

Measure Steal Time over Intervals

One of the easiest ways to misread steal time is to use the wrong measurement window.

For example, the first row printed by vmstat can represent averages since system boot unless you suppress it.

A since-boot average is nearly useless when investigating a short incident.

Imagine that your VM has been running for 30 days.

A two-minute scheduling problem may almost disappear inside the long-term average.

For incident analysis, collect interval-based measurements instead.

Collect One Minute of `vmstat`

A useful starting point is:

TZ=UTC LC_ALL=C vmstat -y -t 1 60 | tee vmstat.txt

This gives you:

1 sample per second
×
60 samples

Using -y skips the misleading initial since-boot row.

The output allows you to observe steal time together with metrics such as:

r
us
sy
id
wa
st

during exactly the same intervals.

Collect Per-vCPU Data with `mpstat`

A system-wide CPU average can hide an important detail.

Suppose a four-vCPU machine looks like this:

CPU 0 → heavily delayed
CPU 1 → mostly idle
CPU 2 → mostly idle
CPU 3 → mostly idle

The average across all CPUs may look relatively normal.

To inspect individual virtual CPUs, use:

TZ=UTC LC_ALL=C mpstat -P ALL 1 60 | tee mpstat.txt

This lets you compare each vCPU separately.

That becomes especially important for workloads that rely heavily on a small number of threads.

What Does the `r` Column in `vmstat` Mean?

The r field represents runnable tasks.

Conceptually:

r =
tasks currently executing
+
tasks ready and waiting for CPU

It is not an exact queue size for each individual CPU core.

But it is useful as a CPU-pressure signal.

Consider these two situations.

Situation A

%st: brief spike
r: normal
p99: normal
throughput: normal

There may be nothing operationally important happening.

Situation B

%st: elevated
r: increasing
p99: increasing
throughput: falling

Now the evidence is much stronger.

The second case tells us that:

the VM is waiting for CPU
runnable work is accumulating
users are experiencing measurable degradation

That is the kind of correlation worth investigating.

Record the Environment with Every Incident

Raw performance numbers are much less useful without environmental context.

At minimum, collect the timestamp:

date -u --iso-8601=seconds

Kernel version:

uname -r

Number of processors visible to the guest:

nproc

Virtualization type:

systemd-detect-virt

And hypervisor information where available:

LC_ALL=C lscpu | sed -n '/Hypervisor vendor/p;/Virtualization type/p'

Store at least:

UTC timestamp
Kernel version
Number of vCPUs
Virtualization type
VM plan
CPU limits
Region

This matters because two results such as:

%st = 8%

may describe completely different environments.

An 8% reading on a one-vCPU burstable instance is not automatically comparable to 8% on an eight-vCPU VM with a different CPU policy.

Establish a Baseline Before Calling `%st` High

There is no universal steal-time threshold.

You will sometimes see rules such as:

%st > 5% = bad

or:

%st > 10% = overloaded host

These rules are convenient.

They are also too simplistic.

A better question is:

Is this value abnormal for this VM under a comparable workload?

Build a baseline using measurements collected during:

low-load periods
normal-load periods
comparable hours
comparable weekdays
similar request volumes

Then calculate values such as:

median
p95
p99

for the steal-time series.

The goal is to understand the normal distribution for the machine.

Why the Maximum Is Often a Bad Metric

Suppose you observe:

Maximum %st this month: 35%

That sounds alarming.

But what if the 35% value lasted for one second and nothing happened to the application?

Now compare that with:

%st: 7–10%
Duration: 12 minutes
p99 latency: +70%
Throughput: -20%

The second event may be much more important even though its maximum value is lower.

For performance incidents, persistence and application impact often matter more than the highest isolated value.

Rebuild the Baseline After Major VM Changes

The old baseline may stop being meaningful after changing:

number of vCPUs
kernel version
VM class
service plan
CPU limits

For example, moving from:

2 vCPU

to:

8 vCPU

changes thread distribution and the way aggregate CPU statistics should be interpreted.

Treat significant configuration changes as the beginning of a new baseline period.

When Should You Investigate High Steal Time?

A useful rule is to investigate when all three conditions begin to appear.

1. `%st` is above normal

Not simply above an arbitrary internet threshold.

It should be above the historical level for that VM.

2. The condition persists

Several consecutive elevated intervals are more important than a single isolated sample.

3. Something measurable becomes worse

For example:

CPU run queue increases
p95 latency increases
p99 latency increases
throughput decreases
errors increase

A weak signal might look like:

%st: 12%
Duration: 1 second
p99: unchanged
Throughput: unchanged

A stronger signal might look like:

%st: elevated for 8 minutes
r: increased
p99: +80%
Throughput: -25%

The second case is much more actionable.

Separate Host Contention from Guest CPU Saturation

Steal time does not prevent the guest from also being CPU-bound.

Both can happen at the same time.

Start by examining:

%usr
%sys
%idle
r
%st

Suppose you observe:

High %usr
High %sys
Low %idle
r > number of vCPUs

The guest itself may already be CPU saturated.

That does not mean %st is irrelevant.

It means you may have multiple sources of delay.

Inspect Individual vCPUs

Run:

TZ=UTC LC_ALL=C mpstat -P ALL 1 60

This can expose per-vCPU imbalance that disappears from the average.

For example:

CPU 0 → 100% busy
CPU 1 → 20%
CPU 2 → 15%
CPU 3 → 10%

A single CPU-bound thread may saturate one virtual CPU while the rest remain mostly free.

Inspect Application Threads

Use pidstat to inspect thread-level CPU usage:

TZ=UTC LC_ALL=C pidstat -u -t -p PID 1 60

This helps answer questions such as:

Is one thread consuming nearly all available CPU?
Is CPU usage distributed evenly?
Is the application itself creating the bottleneck?

Only after checking these guest-side causes does it become safer to focus on the host.

Do Not Ignore I/O, Locks, or External Dependencies

High latency does not automatically mean CPU contention.

Other possibilities include:

storage latency
database locking
application locks
garbage collection
network latency
external APIs
DNS
remote databases

So remember:

High latency ≠ automatically CPU contention

And:

High %st ≠ automatically a noisy neighbor

A useful diagnosis should eliminate alternative explanations before assigning a cause.

Correlate `%st` with Application Metrics

The most useful analysis begins when system metrics and application metrics share the same timeline.

Use UTC for everything.

For each interval, compare:

%st
r
request rate
throughput
error rate
p50 latency
p95 latency
p99 latency

This allows you to answer a much better question than:

Was steal time high?

You can ask:

Did application performance become worse during exactly the same period?

Do Not Compare Different Aggregation Windows

Imagine this comparison:

%st → one-second samples

p99 → five-minute window

A CPU scheduling event lasting 30 seconds may disappear almost completely inside a five-minute application aggregate.

The metrics then appear unrelated even when they describe the same incident.

Try to align:

timestamps
collection intervals
aggregation windows

as closely as possible.

A "Noisy Neighbor" Is a Hypothesis, Not a Measurement

One of the most common explanations for steal time is another VM consuming physical CPU resources on the same host.

That is the classic "noisy neighbor" scenario.

It may be correct.

But guest metrics alone cannot tell you:

which VM caused the delay
whether another tenant caused it
whether the host itself was busy
whether a scheduler policy was involved

So instead of saying:

A noisy neighbor caused our outage.

A technically defensible statement would be:

During this interval, the VM experienced increased CPU steal time while the run queue and application latency also increased.

That statement describes what you actually measured.

Host telemetry is needed for stronger attribution.

When Does the Host-Contention Hypothesis Become Stronger?

The hypothesis becomes more convincing when degradation:

repeats under similar traffic
coincides with elevated %st
is not explained by a deployment
is not explained by garbage collection
is not explained by I/O
is not explained by cgroup throttling

Host-side scheduling metrics provide the strongest confirmation.

Store Raw Data, Not Just Screenshots

A useful incident dataset might include:

incident/
├── metadata.txt
├── vmstat.txt
├── mpstat.txt
├── application.csv
└── README.md

A minimal application/system CSV could use:

timestamp_utc,st,r,usr,sys,p95_ms,p99_ms,rps,error_rate

Raw data lets you:

recalculate percentiles
change aggregation windows
inspect individual events
compare multiple incidents
reproduce the analysis later

A screenshot cannot do that.

Once metrics have been compressed into an image, much of the original information is gone.

Test the Hypothesis with Controlled CPU Load

A controlled test can help you understand how the VM behaves under a known CPU workload.

First record the stress-ng version:

stress-ng --version | tee stress-ng-version.txt

Then run a CPU-bound workload pinned to one vCPU:

taskset -c 0 stress-ng \
  --cpu 1 \
  --cpu-method matrixprod \
  --timeout 120s \
  --metrics-brief \
  2>&1 | tee stress-ng.txt

At the same time, collect system metrics:

TZ=UTC LC_ALL=C vmstat -y -t 1 120 | tee vmstat.txt

And:

TZ=UTC LC_ALL=C mpstat -P ALL 1 120 | tee mpstat.txt

Production warning: Do not deliberately saturate CPU on a production VM unless you have an approved test or maintenance window.

A controlled test is useful because one known variable is introduced:

Predictable CPU workload

You can then observe how the VM and its metrics respond.

Compare Like with Like

Do not compare stress-ng numbers from unrelated systems as if they were equivalent.

Try to keep the following constant:

stress-ng version
CPU method
operating-system image
CPU class
vCPU count
CPU limits

Otherwise, differences may come from the test environment rather than from the platform behavior you are trying to measure.

What Can the Hosting Provider See That You Cannot?

Inside the VM, you only see guest-level evidence.

The infrastructure team may have access to:

vCPU thread wait time
host oversubscription
scheduler statistics
physical CPU utilization
vCPU pinning
host events
hypervisor-level contention

Those measurements are much closer to the actual scheduling layer.

If possible, ask the provider to compare its host-side data with the exact same UTC interval from your guest logs.

A Practical Diagnostic Workflow

Here is a simple sequence you can reuse during an incident.

Step 1: Measure `%st`

Collect interval-based vmstat data:

TZ=UTC LC_ALL=C vmstat -y -t 1 60

Step 2: Check the run queue

Look at:

Ask whether runnable work is accumulating.

Step 3: Inspect each vCPU

TZ=UTC LC_ALL=C mpstat -P ALL 1 60

Look for imbalance.

Step 4: Inspect application threads

TZ=UTC LC_ALL=C pidstat -u -t -p PID 1 60

Check whether the guest workload itself is saturating CPU.

Step 5: Check cgroup throttling

cat /sys/fs/cgroup/cpu.stat

Look for increasing throttling counters.

Step 6: Align application metrics

Compare the same interval for:

p95
p99
throughput
error rate
request rate

Step 7: Eliminate alternative causes

Check:

I/O
garbage collection
locks
deployments
network dependencies

Step 8: Ask for host telemetry

If the symptoms still point toward host CPU scheduling, ask the platform team to inspect the corresponding host interval.

How to Write a Useful Support Ticket

Do not start with an accusation.

This is weak:

Your server is overloaded because another customer is stealing our CPU.

You cannot prove that from guest metrics.

Instead, write something measurable.

For example:

Between 10:14 and 10:22 UTC, %st increased significantly relative to the normal baseline for this VM. During the same interval, the CPU run queue increased and application p99 latency exceeded the SLO.

That gives the provider a concrete period and measurable symptoms to investigate.

Include:

VM identifier
region
number of vCPUs
kernel version
virtualization type
exact UTC interval
normal baseline
observed %st
run queue
p95/p99
throughput

Attach your raw data:

metadata.txt
vmstat.txt
mpstat.txt
application.csv
README.md

Before sending anything, remove:

passwords
API keys
access tokens
user data
sensitive request contents

Repeat the Test After Migration or a Plan Change

Suppose the provider migrates the VM to another host.

Or you change:

the VM plan
CPU limits
vCPU count

Do not simply look at the next random workload and declare the problem fixed.

Repeat a comparable test.

Keep the following as consistent as possible:

Duration
Input workload
Number of processes
Traffic profile
Measurement interval

One run is weak evidence.

Several repeated runs are much stronger.

For example:

Before:
%st elevated
p99 elevated
throughput reduced

After migration:
%st lower
p99 restored
throughput restored

If that pattern repeats across several comparable tests, the case for a platform-related effect becomes much stronger.

Monitor Steal Time with Prometheus

If you use node_exporter, CPU steal time can be observed through:

node_cpu_seconds_total{mode="steal"}

A five-minute percentage can be calculated with PromQL:

100 * avg by (instance) (
  rate(node_cpu_seconds_total{mode="steal"}[5m])
)

This converts the cumulative steal-time counter into an approximate percentage over the selected window.

Do Not Alert on a Single Spike

A simplistic alert might look conceptually like:

%st > X

But this ignores:

the VM's normal baseline
duration
user impact

A more useful alerting model is:

%st above expected baseline
AND
condition persists
AND
application impact exists

Application impact might include:

p99 > SLO

or:

run queue increased

or:

throughput decreased

or:

error rate increased

With Prometheus, the for clause can prevent a single temporary spike from creating an incident.

For example:

for: 10m

means the condition must remain true for ten minutes before the alert fires.

The exact duration should depend on the workload and SLO.

Monitor the Monitoring System Too

One subtle problem remains.

Suppose your dashboard shows:

%st = 0

That only means something if the metric is actually being collected.

If the exporter stops reporting the time series entirely, you should not interpret the missing metric as zero.

A reliable monitoring system should therefore distinguish between:

Steal time is zero

and:

Steal-time telemetry disappeared

Monitoring the monitoring pipeline is part of production observability.

The Three-Signal Model

A useful way to think about steal-time incidents is to use three layers.

1. Scheduling signal
        ↓
       %st

2. CPU pressure
        ↓
        r

3. User-visible impact
        ↓
 p95 / p99 / throughput

The strongest diagnosis appears when all three move together.

For example:

%st ↑
r ↑
p99 ↑
throughput ↓

is strong evidence that CPU scheduling pressure is having a measurable effect.

By contrast:

%st spike
r unchanged
p99 unchanged
throughput unchanged

may not justify any operational response at all.

Practical Checklist

When you notice elevated CPU steal time, work through this list.

[ ] Collect %st using interval-based measurements
[ ] Record UTC timestamps
[ ] Check the vmstat run queue
[ ] Inspect individual vCPUs with mpstat
[ ] Inspect application threads with pidstat
[ ] Check cgroup CPU throttling
[ ] Compare p95 and p99 latency
[ ] Compare throughput
[ ] Compare error rate
[ ] Match aggregation windows
[ ] Compare against the VM's baseline
[ ] Check I/O and application bottlenecks
[ ] Preserve raw data
[ ] Request host telemetry if needed
[ ] Repeat the test after migration or configuration changes

Final Takeaway

CPU steal time %st tells you that a runnable vCPU did not receive physical CPU time during part of a measurement interval.

That information is useful.

But it is not a complete diagnosis.

Do not ask only:

Is %st high?

Ask instead:

Is %st persistently abnormal for this VM, and does it coincide with CPU pressure and measurable application degradation?

A solid investigation combines:

interval-based steal-time measurements
CPU run queue
per-vCPU utilization
cgroup throttling
application latency
throughput
historical baseline
host telemetry when available

Without the run queue and application impact, %st remains an observation.

With synchronized system and application metrics, it becomes evidence you can actually use.

DEV Community: Aeza

Ollama vs vLLM for Concurrent Inference: How to Choose the Right Server

The Short Answer

Start with the Workload, Not the Inference Engine

Personal Chat

Small Internal API

High-Concurrency Inference Service

Record the Workload Before Benchmarking

Why Ollama Is Attractive for Smaller Workloads

Important Ollama Parameters

Ollama's Main Advantage

Why vLLM Benefits from Concurrency

Important vLLM Parameters

Monitor vLLM's Server-Side Behavior

Which Server Is Easier for One User?

Make the Benchmark Fair

Record Model Sources and Hashes

Keep Generation Parameters Identical

Verify Tokenization and Chat Templates

Record the Benchmark Environment

Separate Cold and Warm Performance

Cold Test

Warm Test

Handle Prompt Caching Consistently

Make Sure the Load Generator Is Not the Bottleneck

Build a Concurrency Sweep

Increase Concurrency Until Saturation

Closed-Loop vs Open-Loop Testing

Closed-Loop Load

Open-Loop Load

Throughput Alone Is Not Enough

TTFT: Time to First Token

ITL: Inter-Token Latency

TPOT: Time per Output Token

Measure End-to-End Latency

Always Measure Tail Latency

Understand the Throughput-Latency Trade-Off

Measure Queueing Explicitly

Test Fairness Between Short and Long Requests

Store Raw Benchmark Data

Graph Throughput and Latency Together

Compare API Compatibility Before Migrating

Production Readiness Is More Than the Inference Engine

Compare Observability

Test Upgrades Before Production

Use a Shared API Layer to Keep Migration Easy

Canary the Migration

A Practical Benchmark Workflow

Step 1: Define the Real Workload

Step 2: Normalize the Models

Step 3: Match Generation Settings

Step 4: Record the Environment

Step 5: Warm Up the Server

Step 6: Run the Concurrency Sweep

Step 7: Measure Latency and Throughput

Step 8: Measure Tail Latency

Step 9: Watch Queue and GPU State

Step 10: Stop at the SLO Boundary

Starting Recommendation by Scenario

Single User

Small Internal API

High-Concurrency GPU Service

Practical Checklist

Final Takeaway

CPU Steal Time Explained: What %st Really Means on a Virtual Machine

The Short Answer

What Does %st Mean in top and vmstat?

A Virtual Machine Has Two CPU Schedulers

The guest scheduler

The host scheduler

Do Not Confuse %st with Other CPU Metrics

%usr

%sys

%idle

%iowait

%st

CPU Throttling Is Also Not the Same as Steal Time

A Zero %st Does Not Prove the Host Is Healthy

Measure Steal Time over Intervals

Collect One Minute of vmstat

What Does `%st` Mean in `top` and `vmstat`?

Do Not Confuse `%st` with Other CPU Metrics

`%usr`

`%sys`

`%idle`

`%iowait`

`%st`

A Zero `%st` Does Not Prove the Host Is Healthy

Collect One Minute of `vmstat`

Collect Per-vCPU Data with `mpstat`

What Does the `r` Column in `vmstat` Mean?

Establish a Baseline Before Calling `%st` High

1. `%st` is above normal

Correlate `%st` with Application Metrics

Step 1: Measure `%st`