CPU Looks Fine but the VDS Is Slow: A Linux Incident Runbook

EniyiSunucum — Thu, 17 Sep 2026 00:17:29 +0000

A user reports that an application is slow or intermittently unreachable. The monitoring dashboard shows 20% CPU, memory still available, and no obvious outage. It is tempting to answer, “the server looks healthy.”

That answer is usually premature.

A single CPU graph cannot tell you whether a Linux guest is waiting on storage, reclaiming memory, losing packets, stalling inside a cgroup, contending for hypervisor time, or following a degraded network path. The useful question is not “is CPU high?” but where did the request spend its time?

This runbook provides a repeatable way to answer that question on a Linux virtual server. It starts with low-risk evidence, separates guest-level faults from upstream faults, and produces a compact incident package you can hand to an application team or infrastructure provider.

Safety note: The first stages below are read-only. Run load generators such as fio or iperf3 only during an approved window, against endpoints you control, and with explicit rate limits. Never benchmark a production filesystem or a third-party host without permission.

1. Freeze the incident boundary

Before collecting metrics, write down four facts:

The exact start and end time in UTC.
The affected hostname, IP, port, and protocol.
Whether all clients were affected or only one location/ISP.
One concrete symptom: timeout, connection reset, slow TTFB, packet loss, or high application latency.

Capture a small identity bundle:

date -u -Is
hostnamectl
uname -a
uptime
who -b
journalctl --list-boots | head

This prevents a common failure: comparing a user report from 14:05 with metrics from 14:40, after the system has already recovered.

If the problem is HTTP, record timings from both the server and an external client:

time curl -sS -o /dev/null https://example.com/
curl -sS -D - -o /dev/null https://example.com/

Repeat the same request against localhost or the private service address when possible. Fast locally but slow externally points away from the application process and toward the proxy, firewall, network, or client path.

2. Read CPU as a scheduler, not a percentage

Start with a short time series rather than a single snapshot:

vmstat 1 10
mpstat -P ALL 1 10
pidstat -u -w 1 10

These commands are provided by procps and sysstat on most distributions.

Pay attention to:

r in vmstat: runnable tasks waiting for CPU.
b: tasks blocked, often on I/O.
wa: time waiting for I/O.
st: steal time, when the virtual CPU was ready but the hypervisor scheduled something else.
Context switches and migrations in pidstat.

Low aggregate CPU does not rule out saturation. One single-threaded worker can max one vCPU while the average across eight vCPUs remains near 12.5%. Per-CPU output from mpstat exposes that pattern.

Sustained steal time that correlates with latency is strong evidence of host-side contention, but one sample is not proof. Capture several intervals and correlate them with application timing.

Linux Pressure Stall Information adds another useful view:

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

The “some” and “full” lines describe how much time tasks were delayed because a resource was unavailable. PSI is often more informative than utilization because it measures waiting experienced by workloads.

3. Distinguish available memory from memory pressure

Linux deliberately uses free RAM for cache, so the “free” column alone is not an alarm. Look for reclaim, swap activity, and allocation failures:

free -h
vmstat 1 10
cat /proc/pressure/memory
ps -eo pid,ppid,comm,%mem,rss,vsz --sort=-rss | head -20
journalctl -k --since '-30 min' | grep -Ei 'oom|out of memory|memory cgroup'

Interpret the evidence together:

Continuous si/so in vmstat indicates active swapping.
Rising memory PSI means tasks are stalling during reclaim.
OOM or memory-cgroup messages explain sudden process restarts.
A large page cache with low swap activity is usually normal.

Inside containers, host memory may look comfortable while a service is hitting its cgroup limit. On cgroup v2 systems, inspect the relevant scope under /sys/fs/cgroup and compare memory.current with memory.max. Do not assume the VM total is the service limit.

4. Check storage latency, not only disk usage

A filesystem can have plenty of free space and still respond slowly. First rule out capacity and inode exhaustion:

df -hT
df -i
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS

Then sample the block layer:

iostat -xz 1 10
pidstat -d 1 10
cat /proc/pressure/io
journalctl -k --since '-30 min' | grep -Ei 'I/O error|timeout|reset|nvme|blk_update'

Useful indicators include:

await: average time for an I/O request to complete.
Queue size: sustained queues suggest the device cannot keep up.
I/O PSI: application-visible time lost waiting on storage.
Per-process reads and writes from pidstat.

Treat %util carefully on modern virtual and parallel storage. A high value can be meaningful, but a low value does not guarantee low latency. The combination of await, queueing, PSI, and application timing is stronger than any one number.

Avoid immediately running fio. A benchmark can turn a partial incident into a full outage and can contaminate the very evidence you are trying to preserve. If a controlled test is necessary, use a disposable file, direct I/O where appropriate, a fixed size, a runtime limit, and an agreed bandwidth or IOPS cap.

5. Inspect sockets and the network stack

Begin at the guest boundary:

ss -s
ss -lntup
ss -tin
ip -s link
nstat -az
sar -n DEV,TCP,ETCP 1 10

Look for:

Retransmissions increasing during the incident.
Receive or transmit drops on the interface.
A growing listen or SYN backlog.
Many sockets stuck in SYN-SENT, SYN-RECV, or CLOSE-WAIT.
TCP RTT and retransmission fields in ss -tin.

If the interface counters are clean but the application times out, inspect each layer of the request path: resolver, firewall, reverse proxy, application listener, and upstream dependency.

For DNS, compare authoritative and recursive answers:

dig example.com A +noall +answer +stats
dig @1.1.1.1 example.com A +noall +answer +stats
resolvectl query example.com

A slow resolver can make an otherwise healthy service feel randomly slow, especially when caches expire.

6. Test the path without over-reading ICMP

From a client that actually experienced the problem, collect a bounded path sample:

mtr -rwzc 50 server.example.com
tracepath server.example.com

Run the reverse direction when you control both endpoints. Internet paths are often asymmetric.

Do not diagnose packet loss from one intermediate hop alone. Routers may rate-limit or deprioritize ICMP while continuing to forward application traffic normally. Loss becomes meaningful when it begins at a hop and persists through later hops, especially to the destination, and when it correlates with TCP or application symptoms.

Use iperf3 only between systems you control. Start the server on one endpoint, choose a modest bandwidth cap for UDP tests, and stop if the link or production workload degrades. Throughput is not a substitute for application timing; it is one controlled data point.

7. Add external routing evidence

When the guest looks healthy but multiple remote networks fail, check the route from outside your infrastructure:

Is the prefix visible from several public route collectors?
Did the origin ASN change?
Is the announcement covered by a valid RPKI Route Origin Authorization?
Did the AS path change near the incident time?
Is the problem limited to one region or upstream?

A route visible from one collector is not proof of global reachability. Compare more than one vantage point and record timestamps. This is especially important for regional hosting, where an upstream or peering issue may affect one country while local monitoring stays green.

8. Decide which boundary owns the next action

Use the evidence to route the incident:

Evidence	Most likely next owner
One process or one vCPU saturated	Application/service team
Swap, memory PSI, or cgroup limit	Guest configuration/application team
High await, queueing, and I/O PSI	Storage or infrastructure team
Sustained steal time with normal guest load	Virtualization provider
Interface drops or retransmits from the guest	Guest networking or provider edge
Clean guest, path degradation across several clients	Network/upstream provider
Local service fast, public hostname slow	DNS, proxy, firewall, or network path

This table is not a verdict. It tells you where the next discriminating test belongs.

9. Build an evidence package a provider can act on

A useful escalation is short and reproducible. Include:

VM identifier and affected public IP.
UTC incident window.
Source locations or ISPs affected.
Exact destination IP, port, and protocol.
Ten to sixty seconds of vmstat, mpstat, iostat, PSI, and interface counters.
Application timing from inside and outside.
MTR from both directions when available.
Public routing observations and timestamps.
A clear request, such as “please check host scheduling for this VM between 14:02 and 14:09 UTC.”

Avoid screenshots without axes, “the network is slow,” or a five-megabyte log dump with no timeline. The goal is to let another engineer test a specific hypothesis.

10. Keep the runbook lightweight

The best incident kit is the one already installed and practiced. A small baseline of sysstat, curl, dig, mtr, and journal access covers a large share of Linux performance incidents. Add persistent monitoring for PSI, steal time, disk latency, retransmissions, and application percentiles so the next investigation begins with history instead of guesswork.

A VDS plan should make the CPU allocation, memory limit, storage type, network policy, and support boundary explicit. Those details determine which signals you can observe and which evidence your provider must supply.

Final principle

When CPU looks fine, do not jump straight to a benchmark or a provider ticket. Follow the request across boundaries:

application → scheduler → memory → storage → socket → guest interface → network path → routing

At each boundary, collect one time-stamped signal that can falsify a hypothesis. That turns “the server is slow” into an incident another engineer can reproduce and resolve.

Disclosure: I work on Linux-based virtual server infrastructure and AS215068 at EniyiSunucum. This runbook is vendor-neutral and the commands apply to standard Linux environments.

DEV Community: EniyiSunucum