Is Your Cloud Server Slow Because of CPU Steal? How to Check

Your cloud server feels slower than it did last month. Pages take longer, a cron job that used to finish in two minutes now needs five, and yet nothing changed in your code. Before you start tuning PHP or buying a bigger plan, check one number that a lot of people never look at: CPU steal time.

On a cloud server or VPS, your vCPUs are threads on a physical host that you share with other machines. Steal is the time your vCPU was ready to run but had to wait because the hypervisor gave the physical core to someone else. Your server can't use that time, and nothing inside it can get it back. The good news is that Linux counts it, so you can see it with tools that are already installed.

Where to see CPU steal time

Run top and look at the %Cpu(s) line. The last field, st, is steal. vmstat shows the same thing in its st column, and it's easier to watch over a few seconds:

top -bn1 | head -n 5
vmstat 2 5

Ignore the first vmstat line. It's an average since boot, not a current reading. The lines after it are 2-second samples, and in that output steal sits between 7 and 11 percent while the server is busy. Put differently, for roughly every ten seconds of CPU the server wanted, close to one went to another tenant.

If you want it per vCPU, mpstat from the sysstat package splits it out:

sudo apt install sysstat
mpstat -P ALL 2 5

The %steal column there is the same counter. All of these tools read it from the eighth number on the cpu line of /proc/stat, so they agree with each other, give or take when each one takes its sample.

When the number means something

CPU steal time only appears when your server asks for CPU. A machine that sits at 95% idle can show zero steal on a host that is completely overloaded, simply because it never asks for more than a sliver. So a quick look at 3 a.m. tells you very little. Measure while the site is busy, during a deploy, or while a heavy job runs.

As a rough guide, this is how we read it during busy periods:

• Under 5%: normal on shared virtual hardware. Leave it alone.
• 5 to 15%: worth watching. Collect data at different hours before drawing conclusions.
• Above 15% for long stretches: you'll feel it in response times, and it's worth raising with your provider.

One detail that confuses people: high steal also pushes up the load average. Your processes are still runnable while they wait for the host, so they count toward load even though the server isn't doing more work. A load of 3 on a 2 vCPU server with 12% steal is a different problem from a load of 3 with zero steal.

Also check what kind of virtualization you're on with systemd-detect-virt. On KVM and Xen the steal counter is filled in. Inside an LXC or OpenVZ container it often stays at zero no matter what the host is doing, so a zero there doesn't prove much.

Reading the pattern

A single reading tells you steal exists. The shape over a day tells you why.

Steal follows your own load. It climbs when your app gets busy and stays high after a long burst. That usually points to a CPU limit on the plan, or to a burstable plan that runs on CPU credits. When the credits run out, the throttling often shows up as steal. Check your plan's CPU terms and the credit balance in the provider panel, if there is one.

Steal rises at the same hours whatever you do. Your traffic is flat, but steal jumps every evening or every night at 2 a.m. That's the physical host being busy with other tenants. It's not something you can fix from inside the server.

Steal stays near zero, but the server is slow. Then the slowness is yours. Look at wa (waiting on disk), memory and swap, or the app itself. The disk full guide and the RAM sizing guide are good places to continue.

Keep a record with sar

The second pattern is hard to prove with one screenshot. sar, also from sysstat, saves CPU stats every ten minutes so you can look back. On Debian and Ubuntu the collection is off by default:

sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat

After a day or two you can pull the history:

sar -u                              # today, one line per 10 minutes
sar -u -f /var/log/sysstat/sa24     # the 24th of this month

The %steal column, lined up against %user, shows which of the patterns you're dealing with. On RHEL-based systems the files live in /var/log/sa/ instead.

It also helps to have a simple benchmark you can repeat. Run the same fixed workload at a quiet hour and at a busy one and compare:

openssl speed -seconds 10 sha256     # about a minute, six block sizes

The numbers depend on the CPU model, so don't compare them across providers. Compare them on the same server over time. If the throughput drops at the same hours that sar shows steal going up, you have your answer in two places.

Doing it in one command

We put the sampling into a small script. It reads /proc/stat at a fixed interval, prints user, system, iowait, steal and idle for each sample, and ends with the average and peak steal plus the share since boot. Because steal only shows under load, --busy keeps every vCPU at 100% for the length of the sample. It asks before it starts and stops the load when it's done:

./performance/cpu-steal-check.sh --busy -n 10

With -q it prints a single line and returns a non-zero exit code on a warning, so it fits in cron and builds up a log you can send to support. You can get cpu-steal-check.sh from our server toolkit on GitLab; it's read-only apart from that optional load.

What to do about it

Once you have the data, the options are fairly short:

1. Send it to your provider. Times, sar output and the script log are far more useful than "the server feels slow". They can often move the VM to a quieter host.
2. Check the plan type. If steal follows your own load, you may be on a shared or burstable CPU plan that is doing exactly what it was sold to do. A plan with dedicated vCPUs, or a step up in size, fixes that. The guide to scaling a cloud server covers how to do it one step at a time.
3. Smooth out your own peaks. Move heavy cron jobs, backups and image processing away from your busiest hours, so you ask for less CPU at the moments it's hardest to get.
4. Consider dedicated hardware. If steal keeps coming back after a move and the workload is steady and heavy, a dedicated server takes the neighbors out of the picture entirely. The cloud vs dedicated comparison goes through the trade-offs.

If you're testing a new home for a workload anyway, the same checks work on a fresh Pro Gineous VPS: run the script under load at a few different hours before you move anything.

CPU steal time is one of the few performance problems you can't tune away from inside the server. What you can do is measure it properly, keep the numbers, and make the decision with data instead of a hunch.

Related reading

• Disk Full on a Linux VPS? How to Find What Is Using the Space
• How to Scale a Cloud Server One Step at a Time
• Cloud Server vs Dedicated Server

Originally published on the Pro Gineous blog.