CPU Steal Time Explained: What %st Really Means on a Virtual Machine

CPU steal time is one of those Linux metrics that looks simple until you actually need to diagnose a performance problem.

You open top, notice that %st is above zero, and the obvious conclusion seems to be:

The host is overloaded.

Or perhaps:

Another VM on the server is stealing my CPU.

Both explanations are possible.

Neither can be proven from %st alone.

A useful diagnosis requires at least three things:

• the steal-time measurement itself
• CPU pressure inside the VM
• measurable impact on the application

In practice, that means correlating %st with metrics such as the run queue, p95/p99 latency, throughput, per-vCPU utilization, and cgroup throttling.

In this article, we'll build a practical workflow for doing exactly that.

By the end, you'll know:

• what %st actually measures
• how to collect steal-time data correctly
• when high steal time is worth investigating
• how to separate host contention from guest CPU saturation
• what evidence to collect before contacting your hosting provider
• how to monitor steal time continuously

---

The Short Answer

%st represents time during which a virtual CPU was ready to run but did not receive physical CPU time from the host.

But a high value only becomes operationally meaningful when it is:

1. abnormal for that specific VM
2. sustained across multiple measurement intervals
3. correlated with CPU pressure or application degradation

A single spike is not enough.

A much stronger signal looks like this:

Elevated %st
+
Growing CPU run queue
+
Increasing p99 latency
+
Falling throughput

That combination is worth investigating.

---

What Does %st Mean in top and vmstat?

In a virtual machine, %st — or steal time — represents the percentage of a measurement interval during which a vCPU was ready to execute but did not receive physical CPU time.

You may see it in top:

%Cpu(s): 12.7 us, 6.4 sy, 0.0 ni, 72.2 id,
          3.1 wa, 0.0 hi, 0.0 si, 5.6 st

In this example:

5.6 st

means that approximately 5.6% of the measured CPU interval was accounted as steal time.

The important part is understanding why this can happen.

A VM does not control the physical CPU directly.

There is another scheduling layer underneath it.

---

A Virtual Machine Has Two CPU Schedulers

To understand steal time, separate the guest scheduler from the host scheduler.

The guest scheduler

The operating system inside the VM decides:

Which process should run on this vCPU?

For example:

nginx
postgres
python
node

may all compete for CPU time inside the guest.

The host scheduler

The physical host or hypervisor decides something different:

When does this vCPU get access to an actual physical CPU?

That gives us two scheduling layers:

Application process
        ↓
Guest OS scheduler
        ↓
       vCPU
        ↓
Host / hypervisor scheduler
        ↓
Physical CPU

Suppose the guest has a runnable process.

The guest scheduler wants to execute it.

The vCPU is therefore ready.

But the host scheduler does not immediately schedule that vCPU onto a physical core.

The guest waits.

That waiting time can be recorded as steal time.

Conceptually:

Guest has runnable work
        ↓
vCPU is ready
        ↓
Physical CPU is not assigned
        ↓
Guest waits
        ↓
Time is recorded as steal

This is why %st applies to the vCPU as a whole rather than to one particular process.

---

Do Not Confuse %st with Other CPU Metrics

Several CPU metrics can indicate performance pressure, but they describe different things.

%usr

Time spent executing user-space code.

Examples:

• application logic
• JavaScript
• Python
• database execution
• compression
• model inference

%sys

Time spent executing kernel code.

%idle

CPU time during which the guest had no runnable work.

%iowait

Time associated with waiting while I/O is pending.

%st

Time during which a runnable virtual CPU did not receive physical CPU time.

A useful mental model is:

%usr / %sys → guest is actively using CPU

%iowait     → work is waiting around I/O

%st         → guest wanted CPU but the host did not provide it

%iowait is therefore not another form of steal time.

---

CPU Throttling Is Also Not the Same as Steal Time

A process can be CPU-limited inside the VM even when %st is zero.

One common cause is a cgroup CPU quota.

With cgroup v2, inspect the configured quota:

cat /sys/fs/cgroup/cpu.max

Then inspect CPU statistics:

cat /sys/fs/cgroup/cpu.stat

Useful counters can include:

nr_throttled
throttled_usec

If these counters increase while the workload is running, the process or cgroup may be hitting a CPU quota.

That means the CPU restriction originates inside the guest environment, not necessarily from hypervisor scheduling.

So this situation is possible:

%st = 0

while application latency still increases because the process is being throttled.

This distinction matters when trying to determine whether the VM itself is overloaded or whether the underlying platform is contributing to the problem.

---

A Zero %st Does Not Prove the Host Is Healthy

Another common mistake is assuming:

%st = 0

means:

No host-side scheduling problems exist

That conclusion is too strong.

A zero value only tells you that the guest kernel did not account for steal time during that particular interval.

The availability of steal-time accounting depends on factors such as:

• hypervisor
• guest kernel
• architecture
• platform configuration

KVM can expose steal time on supported systems, but the guest still needs access to the relevant accounting information.

Other forms of host-side delay may also exist without appearing as %st.

So think of zero as:

No steal time was recorded here.

Not:

The physical host is definitely healthy.

---

Measure Steal Time over Intervals

One of the easiest ways to misread steal time is to use the wrong measurement window.

For example, the first row printed by vmstat can represent averages since system boot unless you suppress it.

A since-boot average is nearly useless when investigating a short incident.

Imagine that your VM has been running for 30 days.

A two-minute scheduling problem may almost disappear inside the long-term average.

For incident analysis, collect interval-based measurements instead.

---

Collect One Minute of vmstat

A useful starting point is:

TZ=UTC LC_ALL=C vmstat -y -t 1 60 | tee vmstat.txt

This gives you:

1 sample per second
×
60 samples

Using -y skips the misleading initial since-boot row.

The output allows you to observe steal time together with metrics such as:

• r
• us
• sy
• id
• wa
• st

during exactly the same intervals.

---

Collect Per-vCPU Data with mpstat

A system-wide CPU average can hide an important detail.

Suppose a four-vCPU machine looks like this:

CPU 0 → heavily delayed
CPU 1 → mostly idle
CPU 2 → mostly idle
CPU 3 → mostly idle

The average across all CPUs may look relatively normal.

To inspect individual virtual CPUs, use:

TZ=UTC LC_ALL=C mpstat -P ALL 1 60 | tee mpstat.txt

This lets you compare each vCPU separately.

That becomes especially important for workloads that rely heavily on a small number of threads.

---

What Does the r Column in vmstat Mean?

The r field represents runnable tasks.

Conceptually:

r =
tasks currently executing
+
tasks ready and waiting for CPU

It is not an exact queue size for each individual CPU core.

But it is useful as a CPU-pressure signal.

Consider these two situations.

Situation A

%st: brief spike
r: normal
p99: normal
throughput: normal

There may be nothing operationally important happening.

Situation B

%st: elevated
r: increasing
p99: increasing
throughput: falling

Now the evidence is much stronger.

The second case tells us that:

1. the VM is waiting for CPU
2. runnable work is accumulating
3. users are experiencing measurable degradation

That is the kind of correlation worth investigating.

---

Record the Environment with Every Incident

Raw performance numbers are much less useful without environmental context.

At minimum, collect the timestamp:

date -u --iso-8601=seconds

Kernel version:

uname -r

Number of processors visible to the guest:

nproc

Virtualization type:

systemd-detect-virt

And hypervisor information where available:

LC_ALL=C lscpu | sed -n '/Hypervisor vendor/p;/Virtualization type/p'

Store at least:

UTC timestamp
Kernel version
Number of vCPUs
Virtualization type
VM plan
CPU limits
Region

This matters because two results such as:

%st = 8%

may describe completely different environments.

An 8% reading on a one-vCPU burstable instance is not automatically comparable to 8% on an eight-vCPU VM with a different CPU policy.

---

Establish a Baseline Before Calling %st High

There is no universal steal-time threshold.

You will sometimes see rules such as:

%st > 5% = bad

or:

%st > 10% = overloaded host

These rules are convenient.

They are also too simplistic.

A better question is:

Is this value abnormal for this VM under a comparable workload?

Build a baseline using measurements collected during:

• low-load periods
• normal-load periods
• comparable hours
• comparable weekdays
• similar request volumes

Then calculate values such as:

median
p95
p99

for the steal-time series.

The goal is to understand the normal distribution for the machine.

---

Why the Maximum Is Often a Bad Metric

Suppose you observe:

Maximum %st this month: 35%

That sounds alarming.

But what if the 35% value lasted for one second and nothing happened to the application?

Now compare that with:

%st: 7–10%
Duration: 12 minutes
p99 latency: +70%
Throughput: -20%

The second event may be much more important even though its maximum value is lower.

For performance incidents, persistence and application impact often matter more than the highest isolated value.

---

Rebuild the Baseline After Major VM Changes

The old baseline may stop being meaningful after changing:

• number of vCPUs
• kernel version
• VM class
• service plan
• CPU limits

For example, moving from:

2 vCPU

to:

8 vCPU

changes thread distribution and the way aggregate CPU statistics should be interpreted.

Treat significant configuration changes as the beginning of a new baseline period.

---

When Should You Investigate High Steal Time?

A useful rule is to investigate when all three conditions begin to appear.

1. %st is above normal

Not simply above an arbitrary internet threshold.

It should be above the historical level for that VM.

2. The condition persists

Several consecutive elevated intervals are more important than a single isolated sample.

3. Something measurable becomes worse

For example:

• CPU run queue increases
• p95 latency increases
• p99 latency increases
• throughput decreases
• errors increase

A weak signal might look like:

%st: 12%
Duration: 1 second
p99: unchanged
Throughput: unchanged

A stronger signal might look like:

%st: elevated for 8 minutes
r: increased
p99: +80%
Throughput: -25%

The second case is much more actionable.

---

Separate Host Contention from Guest CPU Saturation

Steal time does not prevent the guest from also being CPU-bound.

Both can happen at the same time.

Start by examining:

%usr
%sys
%idle
r
%st

Suppose you observe:

High %usr
High %sys
Low %idle
r > number of vCPUs

The guest itself may already be CPU saturated.

That does not mean %st is irrelevant.

It means you may have multiple sources of delay.

---

Inspect Individual vCPUs

Run:

TZ=UTC LC_ALL=C mpstat -P ALL 1 60

This can expose per-vCPU imbalance that disappears from the average.

For example:

CPU 0 → 100% busy
CPU 1 → 20%
CPU 2 → 15%
CPU 3 → 10%

A single CPU-bound thread may saturate one virtual CPU while the rest remain mostly free.

---

Inspect Application Threads

Use pidstat to inspect thread-level CPU usage:

TZ=UTC LC_ALL=C pidstat -u -t -p PID 1 60

This helps answer questions such as:

• Is one thread consuming nearly all available CPU?
• Is CPU usage distributed evenly?
• Is the application itself creating the bottleneck?

Only after checking these guest-side causes does it become safer to focus on the host.

---

Do Not Ignore I/O, Locks, or External Dependencies

High latency does not automatically mean CPU contention.

Other possibilities include:

• storage latency
• database locking
• application locks
• garbage collection
• network latency
• external APIs
• DNS
• remote databases

So remember:

High latency ≠ automatically CPU contention

And:

High %st ≠ automatically a noisy neighbor

A useful diagnosis should eliminate alternative explanations before assigning a cause.

---

Correlate %st with Application Metrics

The most useful analysis begins when system metrics and application metrics share the same timeline.

Use UTC for everything.

For each interval, compare:

%st
r
request rate
throughput
error rate
p50 latency
p95 latency
p99 latency

This allows you to answer a much better question than:

Was steal time high?

You can ask:

Did application performance become worse during exactly the same period?

---

Do Not Compare Different Aggregation Windows

Imagine this comparison:

%st → one-second samples

p99 → five-minute window

A CPU scheduling event lasting 30 seconds may disappear almost completely inside a five-minute application aggregate.

The metrics then appear unrelated even when they describe the same incident.

Try to align:

• timestamps
• collection intervals
• aggregation windows

as closely as possible.

---

A "Noisy Neighbor" Is a Hypothesis, Not a Measurement

One of the most common explanations for steal time is another VM consuming physical CPU resources on the same host.

That is the classic "noisy neighbor" scenario.

It may be correct.

But guest metrics alone cannot tell you:

• which VM caused the delay
• whether another tenant caused it
• whether the host itself was busy
• whether a scheduler policy was involved

So instead of saying:

A noisy neighbor caused our outage.

A technically defensible statement would be:

During this interval, the VM experienced increased CPU steal time while the run queue and application latency also increased.

That statement describes what you actually measured.

Host telemetry is needed for stronger attribution.

---

When Does the Host-Contention Hypothesis Become Stronger?

The hypothesis becomes more convincing when degradation:

• repeats under similar traffic
• coincides with elevated %st
• is not explained by a deployment
• is not explained by garbage collection
• is not explained by I/O
• is not explained by cgroup throttling

Host-side scheduling metrics provide the strongest confirmation.

---

Store Raw Data, Not Just Screenshots

A useful incident dataset might include:

incident/
├── metadata.txt
├── vmstat.txt
├── mpstat.txt
├── application.csv
└── README.md

A minimal application/system CSV could use:

timestamp_utc,st,r,usr,sys,p95_ms,p99_ms,rps,error_rate

Raw data lets you:

• recalculate percentiles
• change aggregation windows
• inspect individual events
• compare multiple incidents
• reproduce the analysis later

A screenshot cannot do that.

Once metrics have been compressed into an image, much of the original information is gone.

---

Test the Hypothesis with Controlled CPU Load

A controlled test can help you understand how the VM behaves under a known CPU workload.

First record the stress-ng version:

stress-ng --version | tee stress-ng-version.txt

Then run a CPU-bound workload pinned to one vCPU:

taskset -c 0 stress-ng \
  --cpu 1 \
  --cpu-method matrixprod \
  --timeout 120s \
  --metrics-brief \
  2>&1 | tee stress-ng.txt

At the same time, collect system metrics:

TZ=UTC LC_ALL=C vmstat -y -t 1 120 | tee vmstat.txt

And:

TZ=UTC LC_ALL=C mpstat -P ALL 1 120 | tee mpstat.txt

Production warning: Do not deliberately saturate CPU on a production VM unless you have an approved test or maintenance window.

A controlled test is useful because one known variable is introduced:

Predictable CPU workload

You can then observe how the VM and its metrics respond.

---

Compare Like with Like

Do not compare stress-ng numbers from unrelated systems as if they were equivalent.

Try to keep the following constant:

• stress-ng version
• CPU method
• operating-system image
• CPU class
• vCPU count
• CPU limits

Otherwise, differences may come from the test environment rather than from the platform behavior you are trying to measure.

---

What Can the Hosting Provider See That You Cannot?

Inside the VM, you only see guest-level evidence.

The infrastructure team may have access to:

• vCPU thread wait time
• host oversubscription
• scheduler statistics
• physical CPU utilization
• vCPU pinning
• host events
• hypervisor-level contention

Those measurements are much closer to the actual scheduling layer.

If possible, ask the provider to compare its host-side data with the exact same UTC interval from your guest logs.

---

A Practical Diagnostic Workflow

Here is a simple sequence you can reuse during an incident.

Step 1: Measure %st

Collect interval-based vmstat data:

TZ=UTC LC_ALL=C vmstat -y -t 1 60

Step 2: Check the run queue

Look at:

r

Ask whether runnable work is accumulating.

Step 3: Inspect each vCPU

TZ=UTC LC_ALL=C mpstat -P ALL 1 60

Look for imbalance.

Step 4: Inspect application threads

TZ=UTC LC_ALL=C pidstat -u -t -p PID 1 60

Check whether the guest workload itself is saturating CPU.

Step 5: Check cgroup throttling

cat /sys/fs/cgroup/cpu.stat

Look for increasing throttling counters.

Step 6: Align application metrics

Compare the same interval for:

p95
p99
throughput
error rate
request rate

Step 7: Eliminate alternative causes

Check:

• I/O
• garbage collection
• locks
• deployments
• network dependencies

Step 8: Ask for host telemetry

If the symptoms still point toward host CPU scheduling, ask the platform team to inspect the corresponding host interval.

---

How to Write a Useful Support Ticket

Do not start with an accusation.

This is weak:

Your server is overloaded because another customer is stealing our CPU.

You cannot prove that from guest metrics.

Instead, write something measurable.

For example:

Between 10:14 and 10:22 UTC, %st increased significantly relative to the normal baseline for this VM. During the same interval, the CPU run queue increased and application p99 latency exceeded the SLO.

That gives the provider a concrete period and measurable symptoms to investigate.

Include:

• VM identifier
• region
• number of vCPUs
• kernel version
• virtualization type
• exact UTC interval
• normal baseline
• observed %st
• run queue
• p95/p99
• throughput

Attach your raw data:

metadata.txt
vmstat.txt
mpstat.txt
application.csv
README.md

Before sending anything, remove:

• passwords
• API keys
• access tokens
• user data
• sensitive request contents

---

Repeat the Test After Migration or a Plan Change

Suppose the provider migrates the VM to another host.

Or you change:

• the VM plan
• CPU limits
• vCPU count

Do not simply look at the next random workload and declare the problem fixed.

Repeat a comparable test.

Keep the following as consistent as possible:

Duration
Input workload
Number of processes
Traffic profile
Measurement interval

One run is weak evidence.

Several repeated runs are much stronger.

For example:

Before:
%st elevated
p99 elevated
throughput reduced

After migration:
%st lower
p99 restored
throughput restored

If that pattern repeats across several comparable tests, the case for a platform-related effect becomes much stronger.

---

Monitor Steal Time with Prometheus

If you use node_exporter, CPU steal time can be observed through:

node_cpu_seconds_total{mode="steal"}

A five-minute percentage can be calculated with PromQL:

100 * avg by (instance) (
  rate(node_cpu_seconds_total{mode="steal"}[5m])
)

This converts the cumulative steal-time counter into an approximate percentage over the selected window.

---

Do Not Alert on a Single Spike

A simplistic alert might look conceptually like:

%st > X

But this ignores:

• the VM's normal baseline
• duration
• user impact

A more useful alerting model is:

%st above expected baseline
AND
condition persists
AND
application impact exists

Application impact might include:

p99 > SLO

or:

run queue increased

or:

throughput decreased

or:

error rate increased

With Prometheus, the for clause can prevent a single temporary spike from creating an incident.

For example:

for: 10m

means the condition must remain true for ten minutes before the alert fires.

The exact duration should depend on the workload and SLO.

---

Monitor the Monitoring System Too

One subtle problem remains.

Suppose your dashboard shows:

%st = 0

That only means something if the metric is actually being collected.

If the exporter stops reporting the time series entirely, you should not interpret the missing metric as zero.

A reliable monitoring system should therefore distinguish between:

Steal time is zero

and:

Steal-time telemetry disappeared

Monitoring the monitoring pipeline is part of production observability.

---

The Three-Signal Model

A useful way to think about steal-time incidents is to use three layers.

1. Scheduling signal
        ↓
       %st

2. CPU pressure
        ↓
        r

3. User-visible impact
        ↓
 p95 / p99 / throughput

The strongest diagnosis appears when all three move together.

For example:

%st ↑
r ↑
p99 ↑
throughput ↓

is strong evidence that CPU scheduling pressure is having a measurable effect.

By contrast:

%st spike
r unchanged
p99 unchanged
throughput unchanged

may not justify any operational response at all.

---

Practical Checklist

When you notice elevated CPU steal time, work through this list.

• [ ] Collect %st using interval-based measurements
• [ ] Record UTC timestamps
• [ ] Check the vmstat run queue
• [ ] Inspect individual vCPUs with mpstat
• [ ] Inspect application threads with pidstat
• [ ] Check cgroup CPU throttling
• [ ] Compare p95 and p99 latency
• [ ] Compare throughput
• [ ] Compare error rate
• [ ] Match aggregation windows
• [ ] Compare against the VM's baseline
• [ ] Check I/O and application bottlenecks
• [ ] Preserve raw data
• [ ] Request host telemetry if needed
• [ ] Repeat the test after migration or configuration changes

---

Final Takeaway

CPU steal time %st tells you that a runnable vCPU did not receive physical CPU time during part of a measurement interval.

That information is useful.

But it is not a complete diagnosis.

Do not ask only:

Is %st high?

Ask instead:

Is %st persistently abnormal for this VM, and does it coincide with CPU pressure and measurable application degradation?

A solid investigation combines:

• interval-based steal-time measurements
• CPU run queue
• per-vCPU utilization
• cgroup throttling
• application latency
• throughput
• historical baseline
• host telemetry when available

Without the run queue and application impact, %st remains an observation.

With synchronized system and application metrics, it becomes evidence you can actually use.