I actually liked FDCServers.
That is probably the strangest way to start an article about a VPS that eventually became unusable enough for me to cancel it.
For quite a while, the server worked normally.
It handled real production traffic. Nginx ran normally. The application ran normally. I had no reason to think I was about to spend days chasing one of the most interesting Linux failures I have seen on a VPS.
Then requests started slowing down.
Sometimes they timed out completely.
And the most confusing part was that the machine often still looked alive.
The VPS was online.
Nginx was:
active (running)
The backend process was alive.
Ping worked.
Sometimes I would connect to the server and see nothing obviously catastrophic.
Then I finally captured the VPS while the failure was actually happening.
This is what I saw:
CPU iowait: 97-100%
I/O PSI full: ~95-98%
read latency: up to 18.7 seconds
flush latency: up to 53.6 seconds
I/O queue depth: 128+
Yes.
Seconds.
Not milliseconds.
At one point, ordinary reads were taking up to 18.7 seconds.
Flush latency reached 53.6 seconds.
The VPS was technically running.
The workload was not.
TL;DR
During intermittent failures on one FDCServers VPS, I captured:
read latency: up to 18.7 s
flush latency: up to 53.6 s
I/O PSI full: almost 100%
CPU iowait: almost 100%
I/O queue depth: 128+
Nginx: blocked in filesystem reads
EXT4/jbd2: blocked waiting for I/O
direct backend: ~3 ms
requests via Nginx: timing out
FDCServers investigated the issue.
One severe incident overlapped with a scheduled backup.
But the same class of storage stall later appeared outside that window.
The VPS was eventually migrated to another node.
After the migration, I captured another severe guest-side storage stall.
I eventually cancelled the VPS.
FDCServers refunded me.
This article is not a claim that every FDCServers server has this problem.
It is a debugging story about one VPS, one intermittent infrastructure problem, and how Linux made the failure visible.
active (running) turned out to mean almost nothing
My first checks were boring:
top
free -h
df -h
systemctl status nginx
Nothing unusual there.
And during healthy periods, the VPS really did look healthy.
For example:
D-state processes: 0
CPU iowait: ~0%
disk latency: ~2-4 ms
I/O PSI some: 0.04
I/O PSI full: 0.04
If I had checked the server only during one of those periods, I could honestly have concluded:
Everything looks fine.
And I would have been completely wrong about the incident.
That became the first important lesson:
A healthy snapshot tells you very little about an intermittent failure.
If the problem comes and goes, you need to capture the system while it is failing.
Not five minutes later.
Not after a reboot.
Not after the workload disappears.
During the failure.
The iostat sample that changed the investigation
Then I caught this:
r_await = 18744 ms
f_await = 53561 ms
aqu-sz = 128.54
util ≈ 100%
read = 88 KB/s
That immediately changed the investigation.
Roughly translated:
read latency ≈ 18.7 seconds
flush latency ≈ 53.6 seconds
queue depth > 128
utilization ≈ 100%
read throughput ≈ 88 KB/s
A disk showing 100% utilization is not automatically broken.
A busy device can be completely healthy.
But this combination was different:
- enormous latency
- enormous queue depth
- almost no useful throughput
- blocked processes
- requests timing out
The virtual block path was saturated while accomplishing very little.
That is very different from a disk simply being busy.
I stopped treating iowait as the diagnosis
At the worst moments:
blocked processes: 4-9
CPU iowait: 97-100%
CPU idle: 0%
It is tempting to summarize that as:
The CPU spent 100% of its time waiting for disk.
But I do not think that is a useful way to reason about Linux iowait.
iowait is CPU accounting.
It is a symptom.
It is not a direct hardware latency measurement.
So I did not want my whole conclusion to depend on one number.
I started looking for independent signals.
That is where PSI became extremely useful.
PSI showed that useful work was actually stopping
Linux exposes Pressure Stall Information under /proc/pressure.
For I/O:
cat /proc/pressure/io
During one of the severe periods:
some avg10=99.14
full avg10=95.55
Later, full approached 98%.
That is a much stronger signal than simply saying:
The disk is slow.
In simplified terms:
-
somemeans at least some non-idle work is stalled on I/O -
fullmeans all non-idle tasks are stalled simultaneously
When full is near 100%, almost every opportunity for the workload to make progress is being consumed by I/O stalls.
The machine can still be online.
Processes can still exist.
But useful work can almost stop.
Which is exactly what I was seeing.
D-state told me to stop blaming one application
Next I checked processes stuck in uninterruptible sleep:
ps -eo state,pid,ppid,etime,wchan:50,comm,args \
| awk 'NR==1 || $1 ~ /^D/'
During the failures I saw unrelated components entering D-state:
jbd2
systemd-journald
Nginx workers
Nginx cache processes
other filesystem activity
A single process briefly entering D-state is not enough to diagnose anything.
But when unrelated processes start blocking together, the picture changes.
At that point, blaming one buggy Nginx worker or one application process made less sense.
Multiple components were waiting on a shared filesystem or storage path.
The kernel stacks showed me the next layer
The wait paths were even more interesting.
For jbd2:
wait_on_buffer
jbd2_log_wait_commit
jbd2_journal_commit_transaction
For Nginx:
folio_wait_bit_common
filemap_read
generic_file_read_iter
ext4_file_read_iter
vfs_read
pread64
And then Linux printed this:
INFO: task nginx blocked for more than 122 seconds.
Meanwhile:
systemctl status nginx
still reported:
active (running)
Both statements were true.
That is the important part.
active (running) answers:
Does the process exist?
It does not answer:
Can this process complete useful work?
That distinction matters enormously when debugging production systems.
My cleanest experiment took about three milliseconds
This was probably the most useful experiment in the entire investigation.
I tested the normal path through Nginx.
It failed:
HTTP=000
SSL connection timeout
Then I bypassed Nginx and called the backend directly.
Result:
connect = 0.000423 s
TTFB = 0.003063 s
total = 0.003139 s
About 3 milliseconds.
The HTTP status itself was not important.
What mattered was that the backend:
- accepted the connection
- executed the request
- returned almost immediately
At the same time, Nginx workers were blocked inside EXT4 reads.
That gave me a much cleaner failure boundary:
application execution -> progressing normally
filesystem-backed web path -> not progressing normally
That did not prove the physical cause.
But it made an application-level explanation much less convincing.
The VPS could collapse in about 30 seconds
One of the reasons this problem was so annoying was how quickly the machine could change state.
I captured this healthy period:
HTTP: 200
D-state: 0
CPU iowait: 3%
r_await: ~1.18 ms
I/O PSI full: ~2.95%
That looks normal.
Then roughly 30 seconds later:
D-state: 4
CPU iowait: 91%
CPU idle: 0%
I/O PSI some: 86.11%
I/O PSI full: 78.02%
r_await: 236.50 ms
HTTP: 000
And later:
CPU iowait: 96-100%
I/O PSI full: ~98%
HTTPS queue: 512
HTTP: 000
Then the workload was removed.
The system recovered:
D-state: 0
CPU iowait: 6%
r_await: ~0.98 ms
queue depth: ~0.07
HTTP: 200
That is what makes intermittent infrastructure failures difficult.
Someone can inspect the server ten minutes later and tell you:
Storage latency is under 1 ms. Everything looks good.
And they may be completely correct.
They are just looking at a different state.
A zero latency value can be surprisingly useless
At one point I saw:
r_await = 0
while the machine was clearly unhealthy.
High iowait.
Processes in D-state.
Outstanding I/O.
Requests failing.
Almost no read throughput.
So was storage suddenly perfect?
No.
Almost nothing was completing.
And metrics based on completed operations become less informative when almost no operations complete.
That is why I stopped reading storage metrics individually.
I looked at them together:
latency
IOPS
throughput
queue depth
in-flight I/O
D-state tasks
PSI
request completion
No single value solved the incident.
The agreement between them did.
The first severe incident overlapped with a backup
This is where the provider side of the story became interesting.
One of the first severe events happened while a scheduled backup was running.
FDCServers confirmed the backup was running at that time.
That was a reasonable explanation candidate.
If the story had been:
backup starts
storage stalls
backup ends
problem never returns
I would have been satisfied.
But that was not what happened.
The same class of storage stall later returned outside the original backup window.
And the problem returned across multiple days.
So the backup may have explained one event.
It did not explain the entire incident.
One outage lasted more than four hours
During one incident, my service logs later showed the VPS unavailable for:
4 hours, 41 minutes and 15 seconds
I want to be very precise here.
I cannot prove that the storage stall caused that exact VM state.
That would require visibility I did not have.
Which brings us to one of the most important limitations of debugging a VPS.
From inside the guest, I could see this:
application
↓
Linux VFS
↓
EXT4
↓
virtual block device
↓
?
The question mark matters.
Behind it could be:
- the hypervisor
- host queues
- storage networking
- distributed storage
- physical media
- host-level scheduling
- other infrastructure invisible to the guest
Linux could show me where progress stopped from inside the VM.
It could not show me the provider's complete physical storage path.
So I am not going to claim:
SSD X failed.
or:
storage node Y was broken.
I did not have that evidence.
FDCServers moved the VPS to another node
Support investigated the issue.
Eventually, FDCServers migrated the VPS to another node.
That was a reasonable intervention.
At that point I hoped the incident was over.
If the failure had been isolated to the original host, migration could have solved it.
But after the migration, I captured another severe guest-side storage stall.
That does not prove that every FDCServers node had the same problem.
It does not prove the provider's entire platform was unhealthy.
It does not even prove that the post-migration event had the exact same physical root cause.
What it proved from my perspective was much narrower:
The production problem I cared about had not been eliminated.
And eventually that was enough.
At some point debugging becomes a business decision
This is where the fun technical investigation stopped mattering.
I like debugging infrastructure problems.
Actually, I loved this investigation.
Linux gave me enough evidence to turn this:
My website sometimes stops responding.
into this:
The application backend is healthy, but filesystem-dependent processes are simultaneously blocked below VFS/EXT4 while the virtual block device shows severe queueing and the workload spends almost all of its time stalled on I/O.
That is a very satisfying technical answer.
But I still did not control:
- the hypervisor
- the storage backend
- the host
- the virtual block path
- anything behind it
And production is not supposed to be an endless forensic experiment.
After the problem returned even after migration, I no longer wanted to depend on that VPS.
I asked FDCServers to cancel it.
They refunded me.
That deserves to be part of the story too.
So, is this a negative FDCServers review?
For this VPS?
Yes.
Absolutely.
The storage behavior became unacceptable for production.
I eventually lost confidence in the server and cancelled it.
But I do not think one incident gives me enough evidence to write:
FDCServers is bad.
That would be a much broader claim than my data supports.
There are parts of the experience I appreciated:
- the VPS worked normally before the incident
- support investigated
- they attempted a node migration
- they refunded me when I decided to leave
There were also things I could not accept:
- severe intermittent I/O stalls
- requests timing out
- 18.7-second read latency
- 53.6-second flush latency
- the problem returning on multiple days
- another severe stall after migration
So my actual conclusion is simple:
One FDCServers VPS became unreliable enough for my production workload that I cancelled it.
I can defend that statement.
I cannot defend a fleet-wide conclusion.
I also have not retested their current infrastructure, so I cannot tell you what FDCServers VPS performance looks like today.
Infrastructure changes.
Hardware gets replaced.
Storage gets rebalanced.
Workloads move.
Bugs disappear.
New ones appear.
A past incident is not a permanent benchmark.
What I could prove
From inside the VPS, I had strong evidence for this:
read latency: up to 18.7 s
flush latency: up to 53.6 s
I/O PSI full: almost 100%
CPU iowait: almost 100%
I/O queue: 128+
Nginx: blocked in filesystem reads
EXT4/jbd2: blocked waiting for I/O
direct backend: ~3 ms
HTTP through Nginx: timing out
I could show that:
- the application backend could still respond quickly
- filesystem-dependent processes were blocking
- multiple unrelated processes were affected
- Linux reported severe I/O pressure
- the virtual block path showed enormous latency and queueing
- useful workload progress nearly stopped
That was enough to locate the failure boundary.
What I could not prove
I could not prove:
- which physical disk was involved
- whether a physical disk failed
- whether the cause was host contention
- whether the cause was distributed storage
- whether the cause was storage networking
- which host-side component was responsible
- whether every severe event had exactly the same physical root cause
And I think that distinction is important.
A good incident report should separate:
what I observed
from:
what I suspect
and from:
what I cannot know from this layer
The Linux workflow I use now
This incident changed the way I investigate mysterious slowdowns.
I still start with:
top
free -h
df -h
Then:
date -u
uptime
cat /proc/pressure/io
cat /proc/pressure/memory
cat /proc/pressure/cpu
vmstat 1 10
iostat -x 1 10
ps -eo state,pid,ppid,etime,wchan:50,comm,args \
| awk 'NR==1 || $1 ~ /^D/'
ss -lntp
journalctl -k --since "30 min ago" --no-pager
Then I isolate the request path.
For example:
public request
↓
reverse proxy
↓
direct backend
↓
filesystem
↓
block device
And instead of asking:
Why is the server slow?
I ask:
At which layer does useful work stop completing?
That question is much easier to investigate.
One final rule: collect evidence before rebooting
Sometimes the correct production response is:
reboot
That is fine.
Availability matters more than curiosity.
But a reboot also destroys evidence.
Before reboot:
D-state: high
I/O PSI: ~97%
iowait: ~100%
queues: large
requests: failing
After reboot:
D-state: 0
latency: milliseconds
requests: healthy
Now the problem is gone.
And so is your best evidence.
If the business impact allows it, capture at least:
UTC timestamp
PSI
vmstat
iostat
D-state + wchan
kernel messages
socket queues
request timings
Then recover the machine.
Five minutes of evidence can save hours of arguing later.
The server was running. The workload wasn't.
That is the part of this incident I keep coming back to.
The VPS was online.
Nginx was running.
The backend was alive.
The service manager was happy.
And yet, during the worst periods, Linux showed:
read latency: 18.7 seconds
flush latency: 53.6 seconds
I/O PSI full: ~98%
CPU iowait: ~100%
queue depth: 128+
Useful work had almost stopped.
Eventually:
FDCServers investigated
↓
VPS migrated
↓
severe stall reproduced again
↓
I cancelled
↓
FDCServers refunded me
I never learned exactly what physical component behind the virtual block device caused the incident.
And I do not need to invent an answer.
The guest-side evidence was enough to:
- stop blaming the application
- identify the failing layer
- communicate the problem clearly
- decide that production should move elsewhere
That is what good observability is for.
A server can be online.
A process can be running.
A service can be active.
Ping can work.
And the system can still be accomplishing almost nothing.
Linux will usually tell you.
You just have to catch it while it is happening.
Top comments (0)