Your app feels sluggish, queries are timing out, and top shows CPU is fine. Nine times out of ten the culprit is disk I/O — but knowing that doesn't tell you which disk, which process, or whether it's a hardware problem or just one runaway job hammering the filesystem. This guide walks through exactly how to measure I/O latency on a Linux server and narrow it down to the real bottleneck.
Start With iostat — Your First View Into Disk Performance
The fastest way to get a snapshot of disk activity is iostat, part of the sysstat package. If it's not installed, run: sudo apt install sysstat (Debian/Ubuntu) or sudo yum install sysstat (RHEL/CentOS).
Run it with extended stats and a refresh interval: iostat -xz 2 10 — that gives you extended metrics, hides idle devices, refreshes every 2 seconds, and runs 10 times. The columns that matter most for latency are:
- await — average time (ms) for I/O requests to complete, including queue wait. This is your headline latency number.
- r_await / w_await — separate read and write latency, so you can tell if it's reads or writes causing pain.
- svctm — deprecated, ignore it.
- %util — how busy the device is. Near 100% means it's saturated, but on SSDs and NVMe drives this can be misleading because they handle parallel I/O.
A healthy NVMe SSD should show await below 1 ms. A SATA SSD might run 1–5 ms. A spinning disk is acceptable around 5–20 ms but anything above that consistently signals a problem. If you see await spiking to 100 ms or more, something is seriously wrong.
Use iotop to Find the Offending Process
Once you know which disk is struggling, the next question is: who is hammering it? iotop shows per-process I/O in real time, similar to top for CPU. Install it with sudo apt install iotop, then run: sudo iotop -o — the -o flag shows only processes that are actively doing I/O, keeping the output clean.
Look for the DISK READ and DISK WRITE columns. A process reading or writing hundreds of megabytes per second when it shouldn't be is your bottleneck. Note the PID, then cross-reference with ps aux or lsof -p to see what files it's accessing.
Pro tip: if iotop shows I/O is spread evenly across many processes rather than one dominant culprit, the problem is likely at the storage layer itself — a failing drive, a full disk cache, or a RAID controller under pressure.
Go Deeper With blktrace and biolatency
If iostat and iotop point to a disk but not a clear cause, you need to look at the I/O stack more granularly. Two tools are especially useful here.
blktrace captures every block I/O event on a device. Run sudo blktrace -d /dev/sda -o trace for 10–15 seconds, then stop it with Ctrl+C and analyse with blkparse: blkparse -i trace.blktrace.0 | tail -50. You'll see queue times vs. service times broken down by I/O type. This is verbose but reveals whether latency is happening in the kernel queue or at the hardware level.
If you're running a modern kernel (4.7+) with BPF tools installed, biolatency from the BCC toolkit is far more ergonomic: sudo biolatency -d /dev/sda 1 10. It prints a histogram of I/O completion times, showing you the distribution rather than just averages. Averages hide bimodal latency — where most I/Os complete fast but a small percentage take seconds and drag down your p99 response times.
Check for Filesystem and Mount-Level Issues
Sometimes the disk hardware is fine but the filesystem is the bottleneck. A few things to check:
- Disk fullness — run df -h. Linux filesystems degrade badly above ~90% capacity. Inodes can also be exhausted separately: df -i.
- Dirty page writeback — cat /proc/meminfo | grep -i dirty. If dirty pages are consistently high (gigabytes), the kernel is accumulating writes faster than it can flush them. Tune vm.dirty_ratio and vm.dirty_background_ratio in /etc/sysctl.conf.
- Mount options — check /proc/mounts. If a database volume is mounted without noatime, every read triggers a metadata write. Add noatime to the mount options in /etc/fstab and remount.
- Filesystem errors — sudo dmesg | grep -i 'error|EXT4|XFS|I/O error'. Hardware errors show up here before drives fully fail.
Build a Baseline and Set Alerts Before It Gets Bad
Reactive troubleshooting is fine, but the real goal is catching latency before it becomes an outage. Run iostat -xz 1 60 > baseline-$(date +%F).txt during normal operating hours and save the output. Now you have a reference for what normal looks like — which makes future spikes obvious rather than ambiguous.
You can wire iostat into basic alerting by parsing await in a cron job or a shell script that pushes to Slack when it crosses a threshold. For something more automatic, Opservo continuously tracks I/O metrics across your servers, surfaces latency anomalies in plain English, and can correlate a slow-disk event with the specific process responsible — without you having to remember which command to run at 2 AM.
Whether you choose to script your own monitoring or use a tool, the key habit is the same: don't wait for users to report slowness. Disk latency tends to creep up gradually — a drive entering the early stages of failure, a cron job that gets bigger every week, a log directory nobody ever cleans up. Measure it regularly and you'll almost always see the problem coming.
Invite the reader to connect their server to Opservo's free 2-server plan so I/O latency and process-level disk activity are tracked automatically without keeping a cron script alive. — one command installs a read-only agent and your first findings arrive in five minutes: start free. Related: guides why is my server slow.
Originally published on the Opservo blog.
Top comments (0)