DEV Community

Cover image for My FDCServers VPS Was Online. Disk Reads Were Taking 18.7 Seconds
Sergei Solod
Sergei Solod

Posted on

My FDCServers VPS Was Online. Disk Reads Were Taking 18.7 Seconds

I actually liked FDCServers.

That is probably the strangest way to start an article about a VPS that eventually became unusable enough for me to cancel it.

For quite a while, the server worked normally.

It handled real production traffic. Nginx ran normally. The application ran normally. I had no reason to think I was about to spend days chasing one of the most interesting Linux failures I have seen on a VPS.

Then requests started slowing down.

Sometimes they timed out completely.

And the most confusing part was that the machine often still looked alive.

The VPS was online.

Nginx was:

active (running)
Enter fullscreen mode Exit fullscreen mode

The backend process was alive.

Ping worked.

Sometimes I would connect to the server and see nothing obviously catastrophic.

Then I finally captured the VPS while the failure was actually happening.

This is what I saw:

CPU iowait:       97-100%
I/O PSI full:     ~95-98%
read latency:     up to 18.7 seconds
flush latency:    up to 53.6 seconds
I/O queue depth:  128+
Enter fullscreen mode Exit fullscreen mode

Yes.

Seconds.

Not milliseconds.

At one point, ordinary reads were taking up to 18.7 seconds.

Flush latency reached 53.6 seconds.

The VPS was technically running.

The workload was not.


TL;DR

During intermittent failures on one FDCServers VPS, I captured:

read latency:       up to 18.7 s
flush latency:      up to 53.6 s
I/O PSI full:       almost 100%
CPU iowait:         almost 100%
I/O queue depth:    128+
Nginx:              blocked in filesystem reads
EXT4/jbd2:          blocked waiting for I/O
direct backend:     ~3 ms
requests via Nginx: timing out
Enter fullscreen mode Exit fullscreen mode

FDCServers investigated the issue.

One severe incident overlapped with a scheduled backup.

But the same class of storage stall later appeared outside that window.

The VPS was eventually migrated to another node.

After the migration, I captured another severe guest-side storage stall.

I eventually cancelled the VPS.

FDCServers refunded me.

This article is not a claim that every FDCServers server has this problem.

It is a debugging story about one VPS, one intermittent infrastructure problem, and how Linux made the failure visible.


active (running) turned out to mean almost nothing

My first checks were boring:

top
free -h
df -h
systemctl status nginx
Enter fullscreen mode Exit fullscreen mode

Nothing unusual there.

And during healthy periods, the VPS really did look healthy.

For example:

D-state processes: 0
CPU iowait:        ~0%
disk latency:      ~2-4 ms

I/O PSI some:      0.04
I/O PSI full:      0.04
Enter fullscreen mode Exit fullscreen mode

If I had checked the server only during one of those periods, I could honestly have concluded:

Everything looks fine.

And I would have been completely wrong about the incident.

That became the first important lesson:

A healthy snapshot tells you very little about an intermittent failure.

If the problem comes and goes, you need to capture the system while it is failing.

Not five minutes later.

Not after a reboot.

Not after the workload disappears.

During the failure.


The iostat sample that changed the investigation

Then I caught this:

r_await = 18744 ms
f_await = 53561 ms
aqu-sz  = 128.54
util    ≈ 100%
read    = 88 KB/s
Enter fullscreen mode Exit fullscreen mode

That immediately changed the investigation.

Roughly translated:

read latency    ≈ 18.7 seconds
flush latency   ≈ 53.6 seconds
queue depth     > 128
utilization     ≈ 100%
read throughput ≈ 88 KB/s
Enter fullscreen mode Exit fullscreen mode

A disk showing 100% utilization is not automatically broken.

A busy device can be completely healthy.

But this combination was different:

  • enormous latency
  • enormous queue depth
  • almost no useful throughput
  • blocked processes
  • requests timing out

The virtual block path was saturated while accomplishing very little.

That is very different from a disk simply being busy.


I stopped treating iowait as the diagnosis

At the worst moments:

blocked processes: 4-9
CPU iowait:        97-100%
CPU idle:          0%
Enter fullscreen mode Exit fullscreen mode

It is tempting to summarize that as:

The CPU spent 100% of its time waiting for disk.

But I do not think that is a useful way to reason about Linux iowait.

iowait is CPU accounting.

It is a symptom.

It is not a direct hardware latency measurement.

So I did not want my whole conclusion to depend on one number.

I started looking for independent signals.

That is where PSI became extremely useful.


PSI showed that useful work was actually stopping

Linux exposes Pressure Stall Information under /proc/pressure.

For I/O:

cat /proc/pressure/io
Enter fullscreen mode Exit fullscreen mode

During one of the severe periods:

some avg10=99.14
full avg10=95.55
Enter fullscreen mode Exit fullscreen mode

Later, full approached 98%.

That is a much stronger signal than simply saying:

The disk is slow.

In simplified terms:

  • some means at least some non-idle work is stalled on I/O
  • full means all non-idle tasks are stalled simultaneously

When full is near 100%, almost every opportunity for the workload to make progress is being consumed by I/O stalls.

The machine can still be online.

Processes can still exist.

But useful work can almost stop.

Which is exactly what I was seeing.


D-state told me to stop blaming one application

Next I checked processes stuck in uninterruptible sleep:

ps -eo state,pid,ppid,etime,wchan:50,comm,args \
  | awk 'NR==1 || $1 ~ /^D/'
Enter fullscreen mode Exit fullscreen mode

During the failures I saw unrelated components entering D-state:

jbd2
systemd-journald
Nginx workers
Nginx cache processes
other filesystem activity
Enter fullscreen mode Exit fullscreen mode

A single process briefly entering D-state is not enough to diagnose anything.

But when unrelated processes start blocking together, the picture changes.

At that point, blaming one buggy Nginx worker or one application process made less sense.

Multiple components were waiting on a shared filesystem or storage path.


The kernel stacks showed me the next layer

The wait paths were even more interesting.

For jbd2:

wait_on_buffer
jbd2_log_wait_commit
jbd2_journal_commit_transaction
Enter fullscreen mode Exit fullscreen mode

For Nginx:

folio_wait_bit_common
filemap_read
generic_file_read_iter
ext4_file_read_iter
vfs_read
pread64
Enter fullscreen mode Exit fullscreen mode

And then Linux printed this:

INFO: task nginx blocked for more than 122 seconds.
Enter fullscreen mode Exit fullscreen mode

Meanwhile:

systemctl status nginx
Enter fullscreen mode Exit fullscreen mode

still reported:

active (running)
Enter fullscreen mode Exit fullscreen mode

Both statements were true.

That is the important part.

active (running) answers:

Does the process exist?

It does not answer:

Can this process complete useful work?

That distinction matters enormously when debugging production systems.


My cleanest experiment took about three milliseconds

This was probably the most useful experiment in the entire investigation.

I tested the normal path through Nginx.

It failed:

HTTP=000
SSL connection timeout
Enter fullscreen mode Exit fullscreen mode

Then I bypassed Nginx and called the backend directly.

Result:

connect = 0.000423 s
TTFB    = 0.003063 s
total   = 0.003139 s
Enter fullscreen mode Exit fullscreen mode

About 3 milliseconds.

The HTTP status itself was not important.

What mattered was that the backend:

  • accepted the connection
  • executed the request
  • returned almost immediately

At the same time, Nginx workers were blocked inside EXT4 reads.

That gave me a much cleaner failure boundary:

application execution       -> progressing normally
filesystem-backed web path  -> not progressing normally
Enter fullscreen mode Exit fullscreen mode

That did not prove the physical cause.

But it made an application-level explanation much less convincing.


The VPS could collapse in about 30 seconds

One of the reasons this problem was so annoying was how quickly the machine could change state.

I captured this healthy period:

HTTP:          200
D-state:       0
CPU iowait:    3%
r_await:       ~1.18 ms
I/O PSI full:  ~2.95%
Enter fullscreen mode Exit fullscreen mode

That looks normal.

Then roughly 30 seconds later:

D-state:       4
CPU iowait:    91%
CPU idle:      0%
I/O PSI some:  86.11%
I/O PSI full:  78.02%
r_await:       236.50 ms
HTTP:          000
Enter fullscreen mode Exit fullscreen mode

And later:

CPU iowait:    96-100%
I/O PSI full:  ~98%
HTTPS queue:   512
HTTP:          000
Enter fullscreen mode Exit fullscreen mode

Then the workload was removed.

The system recovered:

D-state:      0
CPU iowait:   6%
r_await:      ~0.98 ms
queue depth:  ~0.07
HTTP:         200
Enter fullscreen mode Exit fullscreen mode

That is what makes intermittent infrastructure failures difficult.

Someone can inspect the server ten minutes later and tell you:

Storage latency is under 1 ms. Everything looks good.

And they may be completely correct.

They are just looking at a different state.


A zero latency value can be surprisingly useless

At one point I saw:

r_await = 0
Enter fullscreen mode Exit fullscreen mode

while the machine was clearly unhealthy.

High iowait.

Processes in D-state.

Outstanding I/O.

Requests failing.

Almost no read throughput.

So was storage suddenly perfect?

No.

Almost nothing was completing.

And metrics based on completed operations become less informative when almost no operations complete.

That is why I stopped reading storage metrics individually.

I looked at them together:

latency
IOPS
throughput
queue depth
in-flight I/O
D-state tasks
PSI
request completion
Enter fullscreen mode Exit fullscreen mode

No single value solved the incident.

The agreement between them did.


The first severe incident overlapped with a backup

This is where the provider side of the story became interesting.

One of the first severe events happened while a scheduled backup was running.

FDCServers confirmed the backup was running at that time.

That was a reasonable explanation candidate.

If the story had been:

backup starts
storage stalls
backup ends
problem never returns
Enter fullscreen mode Exit fullscreen mode

I would have been satisfied.

But that was not what happened.

The same class of storage stall later returned outside the original backup window.

And the problem returned across multiple days.

So the backup may have explained one event.

It did not explain the entire incident.


One outage lasted more than four hours

During one incident, my service logs later showed the VPS unavailable for:

4 hours, 41 minutes and 15 seconds
Enter fullscreen mode Exit fullscreen mode

I want to be very precise here.

I cannot prove that the storage stall caused that exact VM state.

That would require visibility I did not have.

Which brings us to one of the most important limitations of debugging a VPS.

From inside the guest, I could see this:

application
    ↓
Linux VFS
    ↓
EXT4
    ↓
virtual block device
    ↓
?
Enter fullscreen mode Exit fullscreen mode

The question mark matters.

Behind it could be:

  • the hypervisor
  • host queues
  • storage networking
  • distributed storage
  • physical media
  • host-level scheduling
  • other infrastructure invisible to the guest

Linux could show me where progress stopped from inside the VM.

It could not show me the provider's complete physical storage path.

So I am not going to claim:

SSD X failed.

or:

storage node Y was broken.

I did not have that evidence.


FDCServers moved the VPS to another node

Support investigated the issue.

Eventually, FDCServers migrated the VPS to another node.

That was a reasonable intervention.

At that point I hoped the incident was over.

If the failure had been isolated to the original host, migration could have solved it.

But after the migration, I captured another severe guest-side storage stall.

That does not prove that every FDCServers node had the same problem.

It does not prove the provider's entire platform was unhealthy.

It does not even prove that the post-migration event had the exact same physical root cause.

What it proved from my perspective was much narrower:

The production problem I cared about had not been eliminated.

And eventually that was enough.


At some point debugging becomes a business decision

This is where the fun technical investigation stopped mattering.

I like debugging infrastructure problems.

Actually, I loved this investigation.

Linux gave me enough evidence to turn this:

My website sometimes stops responding.

into this:

The application backend is healthy, but filesystem-dependent processes are simultaneously blocked below VFS/EXT4 while the virtual block device shows severe queueing and the workload spends almost all of its time stalled on I/O.

That is a very satisfying technical answer.

But I still did not control:

  • the hypervisor
  • the storage backend
  • the host
  • the virtual block path
  • anything behind it

And production is not supposed to be an endless forensic experiment.

After the problem returned even after migration, I no longer wanted to depend on that VPS.

I asked FDCServers to cancel it.

They refunded me.

That deserves to be part of the story too.


So, is this a negative FDCServers review?

For this VPS?

Yes.

Absolutely.

The storage behavior became unacceptable for production.

I eventually lost confidence in the server and cancelled it.

But I do not think one incident gives me enough evidence to write:

FDCServers is bad.

That would be a much broader claim than my data supports.

There are parts of the experience I appreciated:

  • the VPS worked normally before the incident
  • support investigated
  • they attempted a node migration
  • they refunded me when I decided to leave

There were also things I could not accept:

  • severe intermittent I/O stalls
  • requests timing out
  • 18.7-second read latency
  • 53.6-second flush latency
  • the problem returning on multiple days
  • another severe stall after migration

So my actual conclusion is simple:

One FDCServers VPS became unreliable enough for my production workload that I cancelled it.

I can defend that statement.

I cannot defend a fleet-wide conclusion.

I also have not retested their current infrastructure, so I cannot tell you what FDCServers VPS performance looks like today.

Infrastructure changes.

Hardware gets replaced.

Storage gets rebalanced.

Workloads move.

Bugs disappear.

New ones appear.

A past incident is not a permanent benchmark.


What I could prove

From inside the VPS, I had strong evidence for this:

read latency:       up to 18.7 s
flush latency:      up to 53.6 s
I/O PSI full:       almost 100%
CPU iowait:         almost 100%
I/O queue:          128+
Nginx:              blocked in filesystem reads
EXT4/jbd2:          blocked waiting for I/O
direct backend:     ~3 ms
HTTP through Nginx: timing out
Enter fullscreen mode Exit fullscreen mode

I could show that:

  1. the application backend could still respond quickly
  2. filesystem-dependent processes were blocking
  3. multiple unrelated processes were affected
  4. Linux reported severe I/O pressure
  5. the virtual block path showed enormous latency and queueing
  6. useful workload progress nearly stopped

That was enough to locate the failure boundary.


What I could not prove

I could not prove:

  • which physical disk was involved
  • whether a physical disk failed
  • whether the cause was host contention
  • whether the cause was distributed storage
  • whether the cause was storage networking
  • which host-side component was responsible
  • whether every severe event had exactly the same physical root cause

And I think that distinction is important.

A good incident report should separate:

what I observed
Enter fullscreen mode Exit fullscreen mode

from:

what I suspect
Enter fullscreen mode Exit fullscreen mode

and from:

what I cannot know from this layer
Enter fullscreen mode Exit fullscreen mode

The Linux workflow I use now

This incident changed the way I investigate mysterious slowdowns.

I still start with:

top
free -h
df -h
Enter fullscreen mode Exit fullscreen mode

Then:

date -u
uptime

cat /proc/pressure/io
cat /proc/pressure/memory
cat /proc/pressure/cpu

vmstat 1 10
iostat -x 1 10

ps -eo state,pid,ppid,etime,wchan:50,comm,args \
  | awk 'NR==1 || $1 ~ /^D/'

ss -lntp

journalctl -k --since "30 min ago" --no-pager
Enter fullscreen mode Exit fullscreen mode

Then I isolate the request path.

For example:

public request
      ↓
reverse proxy
      ↓
direct backend
      ↓
filesystem
      ↓
block device
Enter fullscreen mode Exit fullscreen mode

And instead of asking:

Why is the server slow?

I ask:

At which layer does useful work stop completing?

That question is much easier to investigate.


One final rule: collect evidence before rebooting

Sometimes the correct production response is:

reboot
Enter fullscreen mode Exit fullscreen mode

That is fine.

Availability matters more than curiosity.

But a reboot also destroys evidence.

Before reboot:

D-state:     high
I/O PSI:     ~97%
iowait:      ~100%
queues:      large
requests:    failing
Enter fullscreen mode Exit fullscreen mode

After reboot:

D-state:     0
latency:     milliseconds
requests:    healthy
Enter fullscreen mode Exit fullscreen mode

Now the problem is gone.

And so is your best evidence.

If the business impact allows it, capture at least:

UTC timestamp
PSI
vmstat
iostat
D-state + wchan
kernel messages
socket queues
request timings
Enter fullscreen mode Exit fullscreen mode

Then recover the machine.

Five minutes of evidence can save hours of arguing later.


The server was running. The workload wasn't.

That is the part of this incident I keep coming back to.

The VPS was online.

Nginx was running.

The backend was alive.

The service manager was happy.

And yet, during the worst periods, Linux showed:

read latency:       18.7 seconds
flush latency:      53.6 seconds
I/O PSI full:       ~98%
CPU iowait:         ~100%
queue depth:        128+
Enter fullscreen mode Exit fullscreen mode

Useful work had almost stopped.

Eventually:

FDCServers investigated
        ↓
VPS migrated
        ↓
severe stall reproduced again
        ↓
I cancelled
        ↓
FDCServers refunded me
Enter fullscreen mode Exit fullscreen mode

I never learned exactly what physical component behind the virtual block device caused the incident.

And I do not need to invent an answer.

The guest-side evidence was enough to:

  • stop blaming the application
  • identify the failing layer
  • communicate the problem clearly
  • decide that production should move elsewhere

That is what good observability is for.

A server can be online.

A process can be running.

A service can be active.

Ping can work.

And the system can still be accomplishing almost nothing.

Linux will usually tell you.

You just have to catch it while it is happening.

Top comments (0)