A dedicated server with two disks in RAID 1 can lose one of them and keep running as if nothing happened. That's the point of RAID, and it's also the trap. The site stays up, nobody gets an alert, and months later the second disk starts throwing errors while the first one has been dead the whole time.
Disks rarely die without warning. They remap bad sectors, fail self-tests and log errors for weeks before they give up, and Linux can read all of it. Here's how to check disk health on Linux with smartctl and mdadm, what the numbers mean, and what to do when one of them looks bad.
The quick check
If you only have a minute, run these three as root:
cat /proc/mdstat
smartctl -H /dev/sda
smartctl -A /dev/sda | grep -E 'Realloc|Pending|Offline_Unc|CRC'
The first shows whether every RAID array still has all its members, the second gives the drive's own verdict, and the third shows the four counters that matter most on a SATA drive. Repeat the last two for each disk.
Find the disks and install smartmontools
SMART data comes from the drive's firmware, and smartctl from the smartmontools package reads it:
sudo apt install smartmontools
lsblk -d -o NAME,MODEL,SIZE,ROTA,TRAN
ROTA is 1 for spinning disks and 0 for SSDs, and TRAN shows whether a disk is SATA, SAS or NVMe. If you see vda or xvda instead, you're on a virtual machine. Virtual disks have no SMART data, and the physical drives are the host's job. That's one of the real differences when you compare a cloud server with a dedicated one.
One big disk on a server that should have several usually means a hardware RAID controller. smartctl can still reach the drives behind most of them, for example smartctl -a -d megaraid,0 /dev/sda for the first disk on a MegaRAID card.
Why PASSED doesn't mean healthy
smartctl -H prints SMART overall-health self-assessment test result: PASSED on most drives, including plenty that are on their way out. The verdict only flips to FAILED when a normalized attribute value drops below the threshold the manufacturer set, and those thresholds are generous. Treat FAILED as an emergency and PASSED as "keep reading".
What you want is the raw values in smartctl -A, the last column of the attribute table. On a SATA drive, four of them do most of the work:
- 5 Reallocated_Sector_Ct: sectors the drive gave up on and replaced from its spare pool. A handful that never changes can live for years. A number that grows every week is a drive that's failing.
- 197 Current_Pending_Sector: sectors the drive couldn't read and hasn't remapped yet. These are reads that already failed. On a RAID member, mdadm usually rewrites them from the other disk, and they either get remapped or clear.
- 198 Offline_Uncorrectable: sectors that failed during the drive's own background scan. Read it together with 197.
- 199 UDMA_CRC_Error_Count: errors on the cable between disk and controller, not on the platters. If this climbs, reseat or replace the cable or backplane slot before you blame the drive.
Ideally all four are zero. Write down anything above zero with the date, so you can see next week whether it moved.
NVMe drives report different numbers
smartctl -a /dev/nvme0n1 shows a short health section instead of the attribute table. Watch Critical Warning (anything other than 0x00 is bad), Available Spare against Available Spare Threshold, Percentage Used (the share of the rated write endurance already used, and it can go past 100), and Media and Data Integrity Errors, which should stay at zero.
Ask the drive to test itself
Counters only change when the drive touches a bad area. A self-test makes it look on purpose:
smartctl -t short /dev/sda # about two minutes
smartctl -t long /dev/sda # hours on a large disk, runs in the background
smartctl -l selftest /dev/sda
Tests run inside the drive and leave your data alone, though a long test adds read load. The newest result is # 1. You want Completed without error. A line like Completed: read failure with an LBA next to it means the drive found a sector it can't read, and that disk should go on the replacement list.
Read the RAID state
/proc/mdstat is the fastest view of software RAID:
md2 : active raid1 sdb3[1] sda3[0]
3902418944 blocks super 1.2 [2/2] [UU]
[2/2] [UU] means two members expected, two working. A degraded array shows [2/1] [U_], and a member marked (F) has been kicked out as failed. For the full picture, including which partition is missing, run mdadm --detail /dev/md2 and check the State line. clean is good, while degraded or recovering needs attention.
One script for all of it
Checking every disk and array by hand gets old fast, so we wrote a small read-only script for it. It runs smartctl -a on each disk, pulls out the counters above, adds the last self-test and the RAID state from /proc/mdstat, and ends with a verdict and an exit code you can use from cron.
sudo ./disk/disk-health-check.sh
sudo ./disk/disk-health-check.sh -q # one line, for cron
In that run both arrays are still [UU], so nothing looks wrong from the outside. But sdb has 8 reallocated and 16 pending sectors and failed its last short test. That's the early warning RAID hides, and the moment to check your backups (here's how to back up a server and prove the restore works) and book a disk swap.
Replacing a failing disk in mdadm RAID 1
Before anything else, confirm the backup is recent. Then note the serial number of the bad disk with smartctl -i /dev/sdb, because that's how the data center tech finds the right drive in the chassis. Pulling the healthy one by mistake takes the server down.
Mark the old disk's partitions as failed and remove them from each array:
mdadm --manage /dev/md2 --fail /dev/sdb3 --remove /dev/sdb3
mdadm --manage /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2
Once the new disk is in, check its name with lsblk, since it won't always come back as sdb. Copy the partition layout from the healthy disk to the new one. Get the direction right, because this overwrites the target's partition table:
sgdisk /dev/sda -R /dev/sdb # copy sda's GPT layout onto sdb
sgdisk -G /dev/sdb # give the copy new unique GUIDs
mdadm --manage /dev/md1 --add /dev/sdb2
mdadm --manage /dev/md2 --add /dev/sdb3
watch cat /proc/mdstat
On an MBR disk, use sfdisk -d /dev/sda | sfdisk /dev/sdb instead of sgdisk. Install the boot loader on the new disk too (grub-install /dev/sdb on BIOS systems), or the server won't boot if the other disk dies next. On UEFI, copy the EFI partition over as well. The rebuild can take hours, and the array stays usable meanwhile.
Make the server tell you
Two daemons that ship with these packages can email you instead of waiting for you to look. In /etc/smartd.conf, this line watches every disk and schedules a short test every night and a long one on Saturdays:
DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m root
For RAID, set MAILADDR you@example.com in /etc/mdadm/mdadm.conf and send a test alert with mdadm --monitor --scan --test --oneshot. Both need working outgoing mail on the server. If your disks are filling up rather than wearing out, that's a different problem, and our guide on what's using the space on a full disk covers it.
The script is free on GitLab: disk-health-check.sh in our server toolkit, and a daily cron line for it is in its header. If you'd rather leave physical disks to someone else entirely, Pro Gineous has VPS plans from $6.96/mo.
FAQ
How often should you check disk health on Linux servers?
Let smartd and mdadm watch all the time, and look at the counters yourself about once a week. The trend matters most, so compare with the last reading.
Is a disk with a few reallocated sectors safe to keep?
A small number that stays flat for months is common on older drives. If it rises between checks, or pending sectors show up at the same time, plan the replacement.
Can smartctl check disks on a VPS?
No. A VPS sees virtual disks without SMART data, and the provider watches the physical drives.
Originally published on the Pro Gineous blog.
Top comments (0)