DEV Community

Cover image for Stop Silent Software RAID Failures: Practical mdadm Monitoring and Scrubbing on Linux
Lyra
Lyra

Posted on

Stop Silent Software RAID Failures: Practical mdadm Monitoring and Scrubbing on Linux

#storage #devops #linux #opensource

Software RAID is great right up until a member fails quietly and you only notice when the second disk dies.

Linux md will keep serving data from a degraded RAID1/5/6/10 array. That is the point of redundancy. It is also how people lose arrays: the first failure is survivable, the second is not, and nothing loudly interrupted dinner.

This post is the operational layer around Linux software RAID:

  • How to read array health without guessing
  • How to make mdadm --monitor / mdmonitor.service actually alert you
  • How to schedule consistency checks (check / repair)
  • How to replace a failed member cleanly

It is not a RAID-level selection guide, not Btrfs/ZFS scrub, and not hardware RAID or multipath-tools.

What “healthy” means for md

A redundant md array is healthy when:

  1. All expected active members are present and in_sync
  2. No component is faulty / blocked
  3. No long-running recovery/reshape is stuck
  4. Periodic scrubbing is not finding unexplained mismatches

Useful built-in surfaces:

Surface What it tells you
/proc/mdstat Live arrays, recovery progress, missing devices
mdadm --detail /dev/mdX State, UUID, member roles, failed/spare slots
mdadm --examine /dev/sdX Superblock metadata on a component disk
/sys/block/mdX/md/* Kernel knobs: array_state, sync_action, mismatch_cnt, per-device state

Quick baseline:

cat /proc/mdstat
sudo mdadm --detail --scan
sudo mdadm --detail /dev/md0
Enter fullscreen mode Exit fullscreen mode

If you have no arrays yet, create a lab RAID1 on throwaway loop devices or spare disks before you practice fail/remove/add. Do not experiment on production root arrays.

Install and identify the monitor path

Debian / Ubuntu 

sudo apt update
sudo apt install mdadm
Enter fullscreen mode Exit fullscreen mode

Config commonly lives at:

  • /etc/mdadm/mdadm.conf
  • and/or /etc/mdadm/mdadm.conf.d/*.conf

Fedora / RHEL-family 

sudo dnf install mdadm
Enter fullscreen mode Exit fullscreen mode

Config commonly lives at:

  • /etc/mdadm.conf
  • and/or /etc/mdadm.conf.d/*.conf

Confirm the monitor unit exists:

systemctl status mdmonitor.service
systemctl cat mdmonitor.service
Enter fullscreen mode Exit fullscreen mode

On current mdadm packaging, mdmonitor.service is the preferred system-wide monitor. The man page notes it is designed to stay alive while a redundant RAID array is active, and that you should customize MAILADDR (and optionally PROGRAM, MAILFROM, MONITORDELAY) in mdadm.conf.

If the unit is inactive/dead with no arrays, that can be normal. The important part is: once you have a redundant array and a configured alert destination, the monitor should be running.

Part 1 — Make mdadm able to scream

1. Put arrays in the config file

Generate ARRAY lines from currently assembled devices:

# Debian/Ubuntu path example
sudo mdadm --detail --scan
sudo tee -a /etc/mdadm/mdadm.conf >/dev/null <<'EOF'
# review before trusting auto-generated lines in production
EOF

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
Enter fullscreen mode Exit fullscreen mode

On Fedora/RHEL-family hosts, append to /etc/mdadm.conf instead.

Review the result. You want stable identity keys such as UUID=..., not fragile by-path accidents you did not mean to pin.

Example shape:

DEVICE partitions containers

ARRAY /dev/md/data metadata=1.2 UUID=aaaaaaaa:bbbbbbbb:cccccccc:dddddddd
MAILADDR ops@example.com
MAILFROM mdadm@$(hostname -f)
PROGRAM /usr/local/sbin/mdadm-alert
MONITORDELAY 60
Enter fullscreen mode Exit fullscreen mode

From mdadm.conf(5):

  • MAILADDR — one address used by mdadm --monitor --scan
  • MAILFROM — optional From: override (config-only; not a CLI flag)
  • PROGRAM — optional helper run on events
  • MONITORDELAY — poll interval in seconds (CLI -d/--delay wins if both set)
  • ARRAY ... spares=N — if set, monitor can emit SparesMissing when fewer spares are present at first sight

Update initramfs after ARRAY changes if the array is needed early in boot (distro-specific: update-initramfs -u / dracut -f). That is an assembly concern more than a monitoring concern, but stale initramfs configs are a classic footgun.

2. Understand which events actually email

mdadm monitor mode knows many events. Important ones from mdadm(8):

Event Meaning Syslog priority
Fail Active member marked faulty Critical
FailSpare Spare failed while rebuilding Critical
DegradedArray Array already degraded when first noticed Critical
SparesMissing Fewer spares than config expects Warning
TestMessage Produced by --test Info
RebuildStarted / RebuildNN / RebuildFinished Recovery/resync/check progress Warning
SpareActive Rebuild finished; spare became active Info
DeviceDisappeared Previously seen array vanished Critical
NewArray New array appeared in /proc/mdstat Info
MoveSpare Spare migrated via spare-group/domain Info

Critical operational detail from the man page:

Only Fail, FailSpare, DegradedArray, SparesMissing, and TestMessage cause Email to be sent.

All events cause PROGRAM to be run.

That means rebuild progress mails will not flood your inbox by default. If you want rebuild start/finish alerts, use PROGRAM (or syslog + journal shipping), not MAILADDR alone.

3. Optional alert program

PROGRAM is invoked with two or three arguments:

  1. event name
  2. array device (for example /dev/md0)
  3. related component device when applicable (Fail, FailSpare, SpareActive, MoveSpare) 
sudo tee /usr/local/sbin/mdadm-alert >/dev/null <<'EOF'
#!/bin/bash
set -euo pipefail
event="${1:-unknown}"
array="${2:-}"
device="${3:-}"
ts="$(date -Is)"
msg="$ts mdadm event=$event array=$array device=$device host=$(hostname -f)"

logger -t mdadm-alert -p daemon.warning "$msg"

# Example: also append a local breadcrumb file
mkdir -p /var/log/mdadm
printf '%s\n' "$msg" >> /var/log/mdadm/events.log

# Hook your notifier here (ntfy, Matrix, ChatOps, ticket queue, etc.)
# curl -fsS -d "$msg" https://ntfy.example.com/raid-alerts || true
EOF
sudo chmod 755 /usr/local/sbin/mdadm-alert
Enter fullscreen mode Exit fullscreen mode

Keep the helper boring and fast. mdadm is not your notification platform; it is the sensor.

4. Enable and verify the monitor 

# Ensure a local MTA or outbound mail path exists if you rely on MAILADDR.
# Many hosts need postfix/msmtp/nullmailer configured first.

sudo systemctl restart mdmonitor.service
systemctl status mdmonitor.service --no-pager
Enter fullscreen mode Exit fullscreen mode

Manual foreground test (useful before trusting the unit):

sudo mdadm --monitor --scan --oneshot --test
Enter fullscreen mode Exit fullscreen mode

What those flags mean:

  • --monitor / -F — monitor mode
  • --scan — discover arrays / alert settings from config and /proc/mdstat
  • --oneshot / -1 — check once; especially useful for DegradedArray / SparesMissing / NewArray
  • --test / -t — generate a TestMessage for each array found at startup so you can verify mail/program delivery

You should see:

  • a TestMessage handled by PROGRAM
  • an email if MAILADDR is set and your MTA path works
  • syslog lines if you also pass --syslog / -y on a manual run

Daemon form from the man page examples:

sudo mdadm --monitor --scan --daemonise
Enter fullscreen mode Exit fullscreen mode

Prefer the distro mdmonitor.service over hand-rolled daemons when the unit is packaged.

Part 2 — Daily/weekly oneshot checks (belt and suspenders)

The long-running monitor catches transitions. A oneshot still helps after reboots, maintenance windows, or “the service was stopped and nobody noticed.”

systemd timer example:

sudo tee /etc/systemd/system/mdadm-oneshot.service >/dev/null <<'EOF'
[Unit]
Description=One-shot mdadm array health poll
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/sbin/mdadm --monitor --scan --oneshot
Nice=10
IOSchedulingClass=idle
EOF

sudo tee /etc/systemd/system/mdadm-oneshot.timer >/dev/null <<'EOF'
[Unit]
Description=Daily mdadm oneshot health poll

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=30m

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mdadm-oneshot.timer
systemctl list-timers mdadm-oneshot.timer --no-pager
Enter fullscreen mode Exit fullscreen mode

If mdadm lives at /usr/sbin/mdadm on your distro, point ExecStart there. Check with command -v mdadm.

Part 3 — Scrub the array on a schedule

Monitoring answers “did a disk fall out?”

Scrubbing answers “are the mirrors/parity still consistent?”

From md(4) SCRUBBING AND MISMATCHES:

  • Write check or repair to md/sync_action
  • md reads all blocks and verifies consistency
  • RAID1/10: copies should match
  • RAID4/5/6: parity should match data
  • check records mismatches; repair rewrites redundancy like resync
  • mismatch count lands in md/mismatch_cnt
  • counts are in md IO units, not necessarily exact single-sector events

Manual scrub 

# Preferred modern helper
sudo mdadm --action=check /dev/md0

# Equivalent sysfs form on systems where you manage it directly
# echo check | sudo tee /sys/block/md0/md/sync_action
Enter fullscreen mode Exit fullscreen mode

Watch progress:

cat /proc/mdstat
cat /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cnt
# optional:
# cat /sys/block/md0/md/sync_completed
Enter fullscreen mode Exit fullscreen mode

Abort or freeze if needed:

sudo mdadm --action=idle /dev/md0     # abort current action
sudo mdadm --action=frozen /dev/md0   # abort and prevent auto-start
Enter fullscreen mode Exit fullscreen mode

Interpreting mismatch_cnt without panicking

From the kernel docs/man page:

  • On a clean RAID5/6, mismatches usually mean a real storage/path problem
  • On RAID1/10, some mismatches can be benign, especially if swap lives on the array
  • mdadm --create ... --assume-clean can leave an array that later check will flag
  • mismatch counts are coarse; a value like 128 may mean one 64 KiB unit, not 128 independent disasters

Practical policy:

  1. Run check regularly
  2. If mismatch_cnt is non-zero on RAID5/6 data arrays, investigate disks/cables/controllers and consider a careful repair during a maintenance window
  3. If RAID1/10 holds swap, expect occasional noise; keep data arrays separate when you can
  4. Never treat scrub success as a backup

Monthly scrub timer 

sudo tee /etc/systemd/system/mdadm-scrub.service >/dev/null <<'EOF'
[Unit]
Description=Start mdadm consistency check on all active arrays
After=local-fs.target

[Service]
Type=oneshot
# Idle I/O so daytime interactive work hurts less if the timer drifts
Nice=19
IOSchedulingClass=idle
ExecStart=/bin/bash -c 'set -euo pipefail; mapfile -t arrs < <(awk "/^md/{print \"/dev/\"\$1}" /proc/mdstat); for a in "${arrs[@]:-}"; do /sbin/mdadm --detail "$a" >/dev/null 2>&1 || continue; echo "Checking $a"; /sbin/mdadm --action=check "$a" || true; done'
EOF

sudo tee /etc/systemd/system/mdadm-scrub.timer >/dev/null <<'EOF'
[Unit]
Description=Monthly mdadm scrub

[Timer]
OnCalendar=*-*-01 03:30:00
Persistent=true
RandomizedDelaySec=45m

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mdadm-scrub.timer
Enter fullscreen mode Exit fullscreen mode

Notes:

  • RAID0/linear have nothing meaningful to monitor for failed members; scrubbing also does not buy redundancy they never had
  • Overlapping scrubs on many large arrays can saturate disks — stagger hosts or arrays if needed
  • Some distros ship their own cron/timer for RAID checks; prefer one scheduler, not two fighting owners

After scrub:

for f in /sys/block/md*/md/mismatch_cnt; do
  printf '%s %s\n' "$f" "$(cat "$f")"
done
Enter fullscreen mode Exit fullscreen mode

Alert if any data array stays non-zero after you understand the workload.

Part 4 — Replace a failed disk without folklore

Assume /dev/md0 is RAID1/5/6/10 and /dev/sdb failed.

1. Confirm state 

cat /proc/mdstat
sudo mdadm --detail /dev/md0
lsblk -o NAME,SIZE,TYPE,MODEL,SERIAL,STATE
Enter fullscreen mode Exit fullscreen mode

Identify the failed member by name and serial. Hot-swap bays lie less often than humans under stress if you recorded slot maps earlier.

2. Fail and remove (if not already gone) 

# If the kernel has not already marked it faulty:
sudo mdadm /dev/md0 --fail /dev/sdb

# Then detach it from the array
sudo mdadm /dev/md0 --remove /dev/sdb
Enter fullscreen mode Exit fullscreen mode

If the device already vanished from the kernel, mdadm supports failing/removing detached components:

sudo mdadm /dev/md0 --fail detached --remove detached
Enter fullscreen mode Exit fullscreen mode

Physically replace the drive (or allocate a new virtual disk).

3. Add the replacement 

# Prefer stable by-id paths when scripting
sudo mdadm /dev/md0 --add /dev/disk/by-id/wwn-0x...
Enter fullscreen mode Exit fullscreen mode

From mdadm(8) manage mode: --add re-adds a recent member when possible; otherwise it adds a hot-spare. On a degraded array, recovery onto that spare starts immediately.

Watch rebuild:

watch -n2 cat /proc/mdstat
sudo mdadm --detail /dev/md0
Enter fullscreen mode Exit fullscreen mode

Optional: wait until idle:

sudo mdadm --wait /dev/md0
Enter fullscreen mode Exit fullscreen mode

4. Rebuild the map / config if your process requires it 

sudo mdadm --detail --scan
# update ARRAY lines if membership metadata in conf is hand-managed
# refresh initramfs if boot depends on this array
Enter fullscreen mode Exit fullscreen mode

5. Keep a spare if downtime is expensive

Create arrays with --spare-devices= or add a standing spare later with --add. For multiple arrays, spare-group= in mdadm.conf lets the monitor move a spare from a healthy array to a degraded peer in the same group. That is powerful and easy to get wrong — only share spares across arrays that are allowed to borrow disks from each other.

Part 5 — Operator dashboard commands worth muscle memory 

# Whole-host summary
cat /proc/mdstat
sudo mdadm --detail --scan --verbose

# One array, human detail
sudo mdadm --detail /dev/md0

# Component superblock
sudo mdadm --examine /dev/sdb

# Export machine-readable detail
sudo mdadm --detail --export /dev/md0

# Sysfs truth
grep -H . /sys/block/md0/md/array_state \
         /sys/block/md0/md/degraded \
         /sys/block/md0/md/sync_action \
         /sys/block/md0/md/mismatch_cnt 2>/dev/null

# Per-member state
grep -H . /sys/block/md0/md/dev-*/state 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

If degraded is 1, you are already on borrowed time.

Boundaries and common mistakes

This workflow is for Linux md software RAID.

It is not a substitute for:

  • SMART / NVMe health — dying disks often warn before they fail out of the array
  • Filesystem scrub — Btrfs/ZFS checksums catch different classes of silent corruption
  • Backups — RAID is availability, not a restore plan
  • Hardware RAID / proprietary controllers — different tools, different failure domains
  • Device-mapper multipath — path redundancy, not md RAID member redundancy

Common mistakes:

  1. Assuming “the server still boots” means the array is clean
  2. Relying on MAILADDR alone and missing rebuild events (PROGRAM/syslog needed)
  3. Never testing alerts with --test
  4. Running scrub only after the first failure
  5. Misreading RAID1/10 mismatch_cnt when swap shares the array
  6. Adding the wrong physical disk because serials were never recorded
  7. Forgetting that degraded dirty RAID5/6 root arrays may refuse to assemble without deliberate force / md-mod.start_dirty_degraded=1 recovery procedures

Minimal acceptance checklist

You are done when all of these are true:

  • [ ] mdadm --detail --scan lists every production array by UUID
  • [ ] MAILADDR and/or PROGRAM is configured
  • [ ] mdmonitor.service is active while redundant arrays are up
  • [ ] mdadm --monitor --scan --oneshot --test delivers a real alert
  • [ ] A daily/weekly oneshot timer exists as backup polling
  • [ ] A monthly (or better) check scrub runs and mismatch_cnt is reviewed
  • [ ] You have practiced fail/remove/add on a lab array
  • [ ] Disk serial ↔ slot inventory exists before the emergency

References

  • mdadm(8) — monitor mode, manage mode, --action=, event list, packaging note for mdmonitor.service
  • mdadm.conf(5) — MAILADDR, MAILFROM, PROGRAM, MONITORDELAY, ARRAY, spare-group, POLICY
  • md(4) — scrubbing and mismatches, sync_action, mismatch_cnt, array states
  • Kernel admin guide: RAID arrays (md)
  • Project docs/index: Linux RAID wiki
  • Upstream releases: kernel.org mdadm tree

Redundancy only helps if somebody hears the first failure. Wire the monitor, test the alert path, scrub on a calendar, and treat a degraded array like a page — because the next disk does not care that you were going to check it later.

Top comments (0)