DEV Community

Cover image for My REGXA VPS Had 94% CPU Steal With Zero Traffic
Sergei Solod
Sergei Solod

Posted on

My REGXA VPS Had 94% CPU Steal With Zero Traffic

I am not writing this as a negative review of REGXA.

I am not trying to tell anyone whether they should or should not use the provider.

This is simply one of my days as a developer.

I moved a normal workload to a KVM VPS with:

2 vCPU
2 GB RAM
60 GB NVMe
Enter fullscreen mode Exit fullscreen mode

Nginx was running.

The backend was running.

The machine was reachable.

But it behaved as if it were completely overloaded.

Requests accumulated. TLS operations became slow. Connections stayed open for far too long. Some requests eventually returned HTTP 504 after more than two minutes.

My first assumption was simple:

something inside my VPS was consuming all the CPU.

Then I ran mpstat.

Average CPU steal:   92.58%

CPU 0 steal:         90.62%
CPU 1 steal:         94.57%

CPU user:             4.02%
CPU system:           1.47%
CPU iowait:           0.41%
CPU idle:             0.53%
Enter fullscreen mode Exit fullscreen mode

That changed the investigation completely.

My application was not consuming 90% of the processor.

It was barely getting CPU time at all.

CPU usage and CPU steal are very different things

If an application is genuinely CPU-bound, I expect something like this:

user:    80%
system:  10%
steal:    0%
idle:    10%
Enter fullscreen mode Exit fullscreen mode

The VM is receiving CPU time and spending it on application or kernel work.

My VPS looked roughly like this:

user:     4%
system:   1%
steal:   93%
idle:     1%
Enter fullscreen mode Exit fullscreen mode

%steal is time during which a virtual CPU is ready to run but the hypervisor does not schedule it.

So when I say that CPU time was being taken away from the VPS, I mean it in that specific virtualization sense.

I cannot prove that REGXA deliberately took physical cores assigned exclusively to me and gave them to another customer.

Guest metrics cannot prove that.

They also cannot prove intent.

What they can show is much simpler:

the VM had runnable work, but its vCPUs were repeatedly not being scheduled.

Individual samples kept landing in the same range:

89.85% steal
92.16% steal
96.48% steal
94.87% steal
95.69% steal
95.95% steal
94.57% steal
93.75% steal
Enter fullscreen mode Exit fullscreen mode

CPU pressure pointed in the same direction:

CPU PSI some avg10:   79.47
CPU PSI some avg60:   75.32
CPU PSI some avg300:  76.31

Load average:
5.85 / 5.75 / 5.73
Enter fullscreen mode Exit fullscreen mode

This was a two-vCPU machine.

There was clearly runnable work waiting for processor time.

But I still wanted to rule out the most obvious explanation:

my own workload.

I removed all production traffic

I moved the active workload to another server.

Then I stopped all production traffic to the affected VPS.

The request queues cleared.

I repeated the CPU test.

The result:

Average CPU steal:   94.17%

CPU 0 steal:         95.56%
CPU 1 steal:         92.83%

CPU user:             1.95%
CPU system:           0.59%
CPU iowait:           0.59%
CPU idle:             2.18%
Enter fullscreen mode Exit fullscreen mode

CPU steal actually became worse.

The important combination was:

user:    1.95%
system:  0.59%
steal:  94.17%
Enter fullscreen mode Exit fullscreen mode

My applications were almost idle.

Yet the VPS was still spending roughly 94% of measured CPU time waiting for the hypervisor.

vmstat samples were still showing:

91% steal
98% steal
98% steal
97% steal
97% steal
95% steal
96% steal
95% steal
Enter fullscreen mode Exit fullscreen mode

At some points, as many as 15 runnable processes were waiting for CPU.

That was the strongest experiment in the entire incident.

Once the workload was gone, application CPU consumption could no longer explain what I was seeing.

localhost was taking seconds

Next I wanted to remove the external network from the equation.

So I tested HTTPS through:

127.0.0.1
Enter fullscreen mode Exit fullscreen mode

That removes:

  • public DNS;
  • my ISP;
  • geographic distance;
  • internet routing;
  • upstream transit;
  • the public network path to the server.

The client and server were inside the same VM.

Under production traffic, I ran ten localhost HTTPS requests.

Four failed during the TLS handshake.

The successful ones took:

29.30 s
22.77 s
12.25 s
11.87 s
11.12 s
 9.40 s
Enter fullscreen mode Exit fullscreen mode

Some TLS handshakes alone took around nine seconds.

For localhost.

After removing production traffic, localhost improved, but remained extremely inconsistent:

0.061 s
0.745 s
0.830 s
0.873 s
1.010 s
1.117 s
1.121 s
2.188 s
3.355 s
Enter fullscreen mode Exit fullscreen mode

The same local operation could take:

61 ms
Enter fullscreen mode Exit fullscreen mode

and then more than:

3.3 seconds
Enter fullscreen mode Exit fullscreen mode

on another attempt.

That is not the kind of result I would explain with public networking.

The rest of the stack started backing up

The application-level symptoms were also visible in socket state.

At one point I had approximately:

450 established connections
122 orphaned connections
110 FIN-WAIT-1
33 CLOSE-WAIT
Enter fullscreen mode Exit fullscreen mode

The backend listen queue on localhost reached around:

14–15
Enter fullscreen mode Exit fullscreen mode

The HTTPS listen queue reached around:

40
Enter fullscreen mode Exit fullscreen mode

Nginx logged requests like:

142.857 s  -> HTTP 504
138.902 s  -> HTTP 504
135.064 s  -> HTTP 504
129.819 s  -> HTTP 504
128.657 s  -> HTTP 504
Enter fullscreen mode Exit fullscreen mode

Other requests remained open for roughly 67–130 seconds.

I also saw:

secure-connect timeout: ~35 s
secure-connect timeout: ~41 s
database network timeouts
delayed TLS operations
Enter fullscreen mode Exit fullscreen mode

Looking at these individually could have sent me in completely different directions.

A 504?

Maybe Nginx.

Database timeout?

Maybe MongoDB.

Slow TLS?

Maybe networking.

Growing queues?

Maybe backend concurrency.

But all of those systems depend on the process being scheduled.

Increasing a timeout does not create CPU time.

It only allows the request to wait longer.

RAM and disk were not the bottleneck

Memory looked fine:

Total RAM:       ~1.9 GiB
Available RAM:   ~1.0–1.1 GiB
Swap used:       ~1–2 MiB

OOM events:      none
OOM killer:      none
Enter fullscreen mode Exit fullscreen mode

Storage looked fine too:

Filesystem:      ~58 GB
Used:            ~11 GB
Available:       ~44 GB
Usage:           ~20%
Enter fullscreen mode Exit fullscreen mode

During the important CPU measurements, I/O wait remained below 1%.

So I did not have:

  • exhausted memory;
  • heavy swapping;
  • OOM activity;
  • a full filesystem;
  • enough disk wait to explain the behavior.

The number that kept dominating the system was still:

%steal
Enter fullscreen mode Exit fullscreen mode

I compared it with another KVM VPS

A useful diagnostic needs a control.

So I ran the same style of test on another KVM VPS that was actively handling normal traffic.

It reported:

Average CPU steal:   0.02%
CPU idle:           87.86%

CPU PSI avg10:       0.29
CPU PSI avg60:       0.63
CPU PSI avg300:      0.49

Load average:
0.47 / 0.33 / 0.14
Enter fullscreen mode Exit fullscreen mode

Ten localhost HTTPS requests completed between roughly:

37 ms
and
69 ms
Enter fullscreen mode Exit fullscreen mode

Later, I tested another production VPS that genuinely was CPU-busy:

CPU user:     61.71%
CPU system:    5.08%
CPU idle:     24.09%
CPU steal:     0.13%
Enter fullscreen mode Exit fullscreen mode

That comparison made the difference very clear.

Busy because my application is running

user:   high
system: some
steal:  low
Enter fullscreen mode Exit fullscreen mode

Busy because the guest is not getting scheduled

user:   low
system: low
steal:  extremely high
Enter fullscreen mode Exit fullscreen mode

Both can result in a slow website.

But they require completely different debugging.

REGXA later confirmed resource contention

Until this point, I only had guest-side evidence.

I could see what Linux was reporting.

I could not see:

  • the physical host;
  • total host CPU load;
  • CPU quotas;
  • scheduler weights;
  • neighbouring VMs;
  • the provider's oversubscription level.

REGXA could.

Support eventually told me the VPS was running on:

shared CPU infrastructure

They explained that CPU resources were shared across multiple virtual machines and that performance could vary with physical-node load.

More importantly, they explicitly said that the elevated CPU steal I was observing resulted from:

resource contention on the underlying infrastructure

They also said the Frankfurt infrastructure was experiencing particularly high demand.

According to support, they could not change the CPU quotas or scheduling policy or provide additional dedicated CPU resources for that shared VPS.

The proposed solution was to move the VPS to a less-loaded location.

At that point, resource contention was no longer only my interpretation of mpstat.

It was also the provider's explanation.

The CPU wording was difficult to reconcile

This part bothered me.

The REGXA VPS material I had seen used wording around:

Dedicated CPU Cores

and guaranteed resources.

But support described my VPS as using shared CPU infrastructure whose performance depended on physical-node load.

I do not know the exact host configuration.

I cannot tell whether the immediate mechanism was:

  • CPU overcommitment;
  • CPU quotas;
  • scheduler weighting;
  • throttling;
  • another hypervisor configuration;
  • or some combination.

And again, I cannot prove intent.

What I can say is:

Under workload:
CPU steal = 92.58%

With production traffic removed:
CPU steal = 94.17%
Enter fullscreen mode Exit fullscreen mode

And REGXA itself later attributed the elevated steal to resource contention on its shared infrastructure.

That is enough evidence for me to document what happened without pretending I know more than I do.

The refund became another part of the incident

Once the infrastructure issue was clear, I did not want to keep moving the VPS around and testing different nodes.

I wanted to cancel it.

Initially, I was offered only a partial refund.

And the proposed refund was not back to the card I had used.

It was REGXA account credit.

For me, provider credit is not the same thing as a refund if I am trying to stop using the provider.

So I kept replying.

I asked for:

100% of the payment
Enter fullscreen mode Exit fullscreen mode

to be returned to:

the original payment method
Enter fullscreen mode Exit fullscreen mode

It took additional back-and-forth.

Eventually, REGXA agreed and refunded the full amount to the original payment method, describing the full refund as an exception.

I appreciate that they eventually returned everything.

But having to keep pushing for the financial resolution after the infrastructure problem had already been acknowledged is also part of the experience.

The debugging lesson was more useful than the hosting story

The easiest mistake here would have been to keep optimizing my application.

I could have:

  • increased Nginx timeouts;
  • changed keepalive settings;
  • reduced concurrency;
  • added retries;
  • tuned connection pools;
  • investigated MongoDB;
  • rewritten backend code.

Some of those changes might have altered individual symptoms.

None would have answered the real question:

Why is the guest spending more than 90% of its CPU time waiting for the hypervisor?

That problem lives below the application layer.

My VPS smoke test now includes %steal

I no longer consider a new VPS healthy just because:

SSH works
Nginx starts
/health returns 200
Enter fullscreen mode Exit fullscreen mode

I also check:

mpstat -P ALL 1 20
Enter fullscreen mode Exit fullscreen mode
vmstat 1 20
Enter fullscreen mode Exit fullscreen mode
cat /proc/pressure/cpu
Enter fullscreen mode Exit fullscreen mode

And for a web server, I test localhost latency.

The metrics I care about are:

%user
%system
%iowait
%idle
%steal
CPU PSI
run queue
localhost latency
Enter fullscreen mode Exit fullscreen mode

And when the result looks suspicious, I do one more test:

remove the workload and measure again.

That was the most useful step in this incident.

The final numbers were hard to misread:

CPU user:     1.95%
CPU system:   0.59%
CPU steal:   94.17%
Enter fullscreen mode Exit fullscreen mode

I am not claiming that every REGXA VPS behaves like this.

I tested one VPS and documented what happened on that machine.

But in this case, the evidence was unusually clear:

  • my applications were barely using CPU;
  • Linux reported 92–94% steal;
  • localhost HTTPS became multi-second and sometimes failed;
  • the condition remained after production traffic was removed;
  • another KVM VPS behaved normally under comparable testing;
  • REGXA later attributed the elevated steal to resource contention on its shared infrastructure.

That is why I am writing this.

Not as a rating.

Not as a recommendation.

Just as one of my days as a developer — and one Linux metric I will never ignore on a VPS again.

Top comments (0)