DEV Community

Cover image for Stop Losing Access When One Storage Path Dies: Practical Device Mapper Multipath on Linux
Lyra
Lyra

Posted on

Stop Losing Access When One Storage Path Dies: Practical Device Mapper Multipath on Linux

#linux #opensource #storage #devops

Stop Losing Access When One Storage Path Dies: Practical Device Mapper Multipath on Linux

Software RAID protects you when a disk dies. Multipath protects you when a path dies.

If a host reaches the same LUN over two iSCSI NICs, two Fibre Channel HBAs, or two NVMe-oF fabrics, Linux will happily show you two (or four) block devices for one volume. Without multipath, a switch blip or cable pull turns into I/O errors — even though the data is still online on another path.

Device Mapper Multipath (multipath-tools / multipathd) collapses those paths into one stable device under /dev/mapper/, fails over automatically, and can load-balance when the array allows it.

This is a practical operator guide: install, configure safely, verify topology, test failover, and avoid the classic "local disk got swallowed by multipath" footgun.

What multipath is (and is not)

Layer Problem it solves
mdadm RAID Disk/device redundancy and striping on the host
LVM Volume composition, snapshots, thin pools
DM Multipath Multiple I/O paths to the same remote LUN
Btrfs/ZFS scrub Checksum integrity inside the filesystem/pool

Use multipath when the storage presents one logical unit through multiple SCSI/NVMe paths. Do not use it as a substitute for RAID, backups, or array-side replication.

RHEL's storage docs describe DM Multipath exactly this way: multiple I/O paths between server and array become a single device, spanning separate cables, switches, and controllers.

Packages and service

Debian / Ubuntu:

sudo apt update
sudo apt install multipath-tools multipath-tools-boot
Enter fullscreen mode Exit fullscreen mode

RHEL / Fedora / Alma / Rocky:

sudo dnf install device-mapper-multipath
sudo mpathconf --enable
Enter fullscreen mode Exit fullscreen mode

Enable and start the daemon:

sudo systemctl enable --now multipathd.service
systemctl status multipathd.service --no-pager
Enter fullscreen mode Exit fullscreen mode

On systemd builds, multipathd.socket can start the daemon on first CLI contact, and the service can integrate with WatchdogSec= (which overrides polling_interval / max_polling_interval when set). Prefer the enabled service so path monitoring is always running.

Discover what the host already sees

Before writing config, inventory raw paths:

# SCSI / iSCSI / FC disks
lsblk -o NAME,SIZE,TYPE,TRAN,WWN,MODEL,SERIAL

# Persistent IDs (prefer these over /dev/sdX)
ls -l /dev/disk/by-id/ | head

# iSCSI sessions (if applicable)
sudo iscsiadm -m session -P 3 2>/dev/null | sed -n '1,80p'

# FC remote ports (if applicable)
ls /sys/class/fc_remote_ports 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

You are looking for two or more block devices that share the same WWID/WWN but differ by path (by-path, HBA, or session target portal).

Dry-run multipath discovery without changing maps:

sudo multipath -d -v3
Enter fullscreen mode Exit fullscreen mode

Show the live effective config (defaults + built-in hardware table + your file):

sudo multipath -t | less
# or
sudo multipathd show config | less
Enter fullscreen mode Exit fullscreen mode

For a template limited to devices actually present:

sudo multipath -T
Enter fullscreen mode Exit fullscreen mode

A safe baseline /etc/multipath.conf

Create or edit /etc/multipath.conf. Start conservative: friendly names, sensible path checks, no infinite hang-on-total-path-loss, and a blacklist that keeps local boot disks out of multipath.

sudo tee /etc/multipath.conf >/dev/null <<'EOF'
defaults {
    user_friendly_names yes
    find_multipaths     yes
    path_selector       "service-time 0"
    path_grouping_policy group_by_prio
    path_checker        tur
    prio                const
    detect_prio         yes
    failback            immediate
    no_path_retry       12
    queue_without_daemon no
    verbosity           2
}

blacklist {
    # Keep common local disks out unless you know you need them multipathed.
    # Property/devnode defaults already exclude many non-disk types.
    device {
        vendor  "ATA"
        product ".*"
    }
    device {
        vendor  "NVMe"
        product ".*"
    }
}

blacklist_exceptions {
    # Explicitly allow SAN/iSCSI vendors you trust (examples — match your array).
    # device {
    #     vendor  "NETAPP"
    #     product "LUN"
    # }
}

# Optional: pin a stable alias for a known LUN WWID
# multipaths {
#     multipath {
#         wwid  36001405xxxxxxxxxxxxxxxxxxxxxxx
#         alias data01
#     }
# }
EOF
Enter fullscreen mode Exit fullscreen mode

Reload configuration into the running daemon:

sudo multipathd reconfigure
# or
sudo systemctl reload multipathd.service
Enter fullscreen mode Exit fullscreen mode

Why these knobs

From multipath.conf(5) (Debian multipath-tools / upstream):

  • user_friendly_names yes — aliases like mpatha via /etc/multipath/bindings instead of raw WWIDs. Per-map alias still wins.
  • find_multipaths yes — create a map when the WWID was seen before or at least two non-blacklisted paths share a WWID. Safer than greedy (everything becomes multipath) and more automatic than default strict (WWIDs file only).
  • path_checker tur — async TEST UNIT READY; default checker and a good general choice. detect_checker yes (default behavior path) will prefer tur on ALUA devices.
  • path_selector "service-time 0" — default selector; picks paths based on outstanding I/O and relative throughput. Alternatives: round-robin 0, queue-length 0, historical-service-time 0 (kernel 5.8+).
  • path_grouping_policy group_by_prio — groups by priority (works well with ALUA via detect_prio yes → sysfs/alua prioritizer). multibus puts all paths in one group; failover is one path per group.
  • failback immediate — return to the best path group when it recovers. Use manual or followover in some active/passive dual-host designs.
  • no_path_retry 12 — queue briefly while paths recover, then fail I/O. Prefer this over deprecated features "1 queue_if_no_path", which can leave unkillable D state processes if all paths stay down (KNOWN ISSUES in the man page).
  • queue_without_daemon no — if multipathd stops, disable queuing so shutdown cannot hang forever waiting for dead paths.

Built-in hardware tables already know 100+ arrays. Only add a devices { device { ... } } override when vendor docs require it.

Blacklist strategy that will not eat your root disk

Evaluation order for blacklist criteria: property → devnode → device → protocol → wwid. Whitelist (blacklist_exceptions) wins per criterion.

Defaults already:

  • Blacklist non-sd / non-dasd / non-nvme devnodes via !^(sd[a-z]|dasd[a-z]|nvme[0-9])
  • Require a sensible udev property match (SCSI_IDENT_|ID_WWN) via the default property exception

Still blacklist local SATA/NVMe explicitly in homelabs where a USB bridge or odd WWN might look "SAN-like". Confirm with:

sudo multipathd show devices
sudo multipathd show blacklist
Enter fullscreen mode Exit fullscreen mode

Protocol filter example (only multipath iSCSI + FC, not local ATA):

blacklist {
    protocol "scsi:ata"
    protocol "nvme:pcie"
}
Enter fullscreen mode Exit fullscreen mode

Recognized protocol strings include scsi:fcp, scsi:iscsi, scsi:sas, nvme:tcp, nvme:rdma, nvme:fc, and others listed in multipath.conf(5). Inspect live paths with:

sudo multipathd show paths format "%d %P %w %t %T %i %o"
Enter fullscreen mode Exit fullscreen mode

Build and inspect maps 

# Create/update maps from current paths
sudo multipath -v2

# Topology from sysfs + device-mapper
sudo multipath -ll

# Richer view (includes checker state)
sudo multipath -ll -v3
Enter fullscreen mode Exit fullscreen mode

Typical healthy topology looks like:

mpatha (36001405aabbccddeeff001122334455) dm-3 NETAPP,LUN
size=500G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 8:0:0:1 sdc 8:32 active ready running
| `- 8:0:1:1 sde 8:64 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 9:0:0:1 sdd 8:48 active ready running
  `- 9:0:1:1 sdf 8:80 active ready running
Enter fullscreen mode Exit fullscreen mode

Read this as: one multipath device mpatha, two path groups (often ALUA active/optimized vs non-optimized), four underlying paths.

Daemon-side checks:

sudo multipathd show maps status
sudo multipathd show maps topology
sudo multipathd show paths
sudo multipathd show status
sudo multipathd show daemon
Enter fullscreen mode Exit fullscreen mode

Device nodes you should use in fstab/LVM/crypttab:

ls -l /dev/mapper/mpath*
readlink -f /dev/mapper/mpatha
ls -l /dev/disk/by-id/dm-name-mpatha /dev/disk/by-id/dm-uuid-* 2>/dev/null | head
Enter fullscreen mode Exit fullscreen mode

Always mount /dev/mapper/mpatha (or a by-id dm link), never /dev/sdc. Path device names reshuffle; the multipath map does not.

Partitions on multipath devices

kpartx maps partition tables on top of multipath (and is invoked from hotplug when maps appear):

sudo kpartx -av /dev/mapper/mpatha
ls -l /dev/mapper/mpatha*

# list only
sudo kpartx -l /dev/mapper/mpatha
Enter fullscreen mode Exit fullscreen mode

You should see nodes like /dev/mapper/mpatha1. Put filesystems or PVs there.

Skip automatic partition maps only if you intentionally manage whole-LUN devices:

defaults {
    skip_kpartx yes
}
Enter fullscreen mode Exit fullscreen mode

LVM and multipath coexistence

Tell LVM to prefer multipath nodes and ignore underlying path devices so it does not activate the same PV four times.

/etc/lvm/lvm.conf (snippet):

devices {
    # Obtain with: sudo multipath -l | ...
    # Modern LVM ships multipath-aware filters; verify with:
    #   sudo lvmdevices
    #   sudo pvs -a -o+devices
    obtain_device_list_from_udev = 1
    multipath_component_detection = 1
}
Enter fullscreen mode Exit fullscreen mode

After maps exist:

sudo pvcreate /dev/mapper/mpatha
sudo vgcreate vg_data /dev/mapper/mpatha
sudo lvcreate -n lv_data -l 100%FREE vg_data
sudo mkfs.xfs /dev/vg_data/lv_data
Enter fullscreen mode Exit fullscreen mode

If LVM still sees duplicate PVs on sd* paths, fix filters or ensure multipath claimed the devices before LVM (multipath -u during udev; multipath-tools-boot on Debian helps early boot).

Controlled failover test

Do this on a non-production LUN first, or during a maintenance window.

1. Start a write workload 

# Example: sustained write to a test filesystem on the multipath LV
sudo mkdir -p /mnt/mpath-test
sudo mount /dev/vg_data/lv_data /mnt/mpath-test
sudo dd if=/dev/zero of=/mnt/mpath-test/probe.bin bs=1M count=2048 oflag=direct &
echo $!
Enter fullscreen mode Exit fullscreen mode

2. Fail one path administratively 

# Pick a path device from multipath -ll, e.g. sdc
sudo multipathd fail path sdc
sudo multipath -ll
Enter fullscreen mode Exit fullscreen mode

I/O should continue on remaining paths. dd should not die.

3. Reinstate the path 

sudo multipathd reinstate path sdc
sudo multipath -ll
Enter fullscreen mode Exit fullscreen mode

With failback immediate and healthy priorities, the preferred path group should become active again.

4. Real link pull (best test)

  • Unplug one FC cable, or
  • sudo ip link set ethX down on one iSCSI interface, or
  • Log out one iSCSI session: sudo iscsiadm -m node -T <iqn> -p <portal> --logout

Watch:

sudo journalctl -u multipathd -f
sudo multipathd show paths format "%d %t %T %o %P"
Enter fullscreen mode Exit fullscreen mode

5. Force path-group switch (optional) 

sudo multipathd switch multipath mpatha group 2
sudo multipath -ll
Enter fullscreen mode Exit fullscreen mode

Operational commands cheat sheet

Goal Command
List topology multipath -ll
Reload all maps multipath -r or multipathd reconfigure
Effective config multipath -t
Add WWID to wwids file multipath -a /dev/sdc
Remove WWID multipath -w /dev/sdc
Flush unused map multipath -f mpatha
Flush all unused multipath -F
Fail/reinstate path multipathd fail path sdc / reinstate path sdc
Disable queuing (emergency) multipathd disablequeueing maps
Resize after array grow multipathd resize map mpatha then grow FS/LVM

Interactive shell: sudo multipathd -k then show topology, show paths, etc.

Boot-time notes

  • Install the distro boot integration package (multipath-tools-boot on Debian/Ubuntu, mpathconf --enable on RHEL) so initramfs can assemble maps before mounting multipathed root or critical volumes.
  • Rebuild initramfs after first working config: sudo update-initramfs -u (Debian/Ubuntu) or sudo dracut -f (Fedora/RHEL).
  • In crypttab / fstab, reference /dev/mapper/<alias> or /dev/disk/by-id/dm-name-<alias>, never ephemeral /dev/sd*.
  • find_multipaths smart can delay udev for single-path devices; prefer yes plus explicit blacklists in mixed local+SAN hosts.

Monitoring that catches path loss

A tiny daily oneshot is enough for many labs:

sudo tee /usr/local/sbin/check-multipath-health >/dev/null <<'EOF'
#!/bin/bash
set -euo pipefail
out="$(multipathd show paths format '%d %t %T %o' 2>/dev/null || true)"
if [[ -z "$out" ]]; then
  echo "multipath: no paths reported (daemon down or no maps)"
  exit 0
fi
# Flag anything not ready/running
bad="$(printf '%s\n' "$out" | awk '($2 != "active" && $2 != "ghost") || ($3 != "ready") {print}' || true)"
if [[ -n "${bad}" ]]; then
  echo "multipath path problems detected:"
  echo "$bad"
  multipath -ll || true
  exit 1
fi
echo "multipath paths OK"
exit 0
EOF
sudo chmod 755 /usr/local/sbin/check-multipath-health
Enter fullscreen mode Exit fullscreen mode 
sudo tee /etc/systemd/system/multipath-health.service >/dev/null <<'EOF'
[Unit]
Description=Check DM-Multipath path health
After=multipathd.service
Requires=multipathd.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/check-multipath-health
EOF

sudo tee /etc/systemd/system/multipath-health.timer >/dev/null <<'EOF'
[Unit]
Description=Daily multipath path health check

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=15m

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now multipath-health.timer
sudo systemctl start multipath-health.service
journalctl -u multipath-health.service -n 20 --no-pager
Enter fullscreen mode Exit fullscreen mode

Wire failures into email/ChatOps however you already alert on systemd unit failures.

Common failure modes

  1. Mounted /dev/sdd instead of /dev/mapper/mpatha

    Path loss = filesystem death. Fix mounts/LVM to the dm device.

  2. Local disk multipathed

    Tighten blacklist; multipath -f unused maps; scrub bindings/wwids if needed (multipath -W resets wwids to current maps — understand it before running).

  3. Infinite I/O hang when array disappears

    You used perpetual queueing. Set no_path_retry to a finite value; keep queue_without_daemon no.

  4. Flapping marginal paths

    Look at san_path_err_* / marginal_pathgroups in multipath.conf(5) ("Shaky paths detection") before blaming the HBA.

  5. Wrong active controller forever

    Check ALUA: detect_prio yes, hardware handler alua, and array host mode set to a Linux DM-Multipath profile per vendor docs.

  6. Duplicate LVM PVs

    Enable multipath component detection; never pvcreate on raw path devices once multipath owns the LUN.

Cleanup / rollback 

# Stop using maps (unmount/vgchange -an first!)
sudo umount /mnt/mpath-test || true
sudo vgchange -an vg_data 2>/dev/null || true

sudo multipath -F
sudo systemctl disable --now multipathd.service
# Remove or rename /etc/multipath.conf if you are abandoning multipath
Enter fullscreen mode Exit fullscreen mode

Boundary check

Use multipath Use something else
Dual-HBA FC to one SAN LUN mdadm for local disk RAID
Two NIC iSCSI sessions to one target LUN Bond/LACP alone (helps link, not SCSI session failover the same way)
NVMe-oF multipathing via DM-MP or native NVMe multipath Btrfs/ZFS scrub for bitrot
Transparent path failover under LVM/XFS backups (restic, snapshots) for data loss

Native NVMe multipathing in the kernel is a related but separate mode (nvme_core.multipath); multipath-tools also ships foreign library support for NVMe. Pick one approach per stack — do not double-multipath blindly.

Quick bring-up checklist

  1. Install multipath-tools / device-mapper-multipath and enable multipathd.
  2. Confirm multiple paths share one WWID (lsblk, by-id, multipath -d -v3).
  3. Drop a minimal multipath.conf with friendly names, find_multipaths yes, finite no_path_retry, and local-disk blacklist.
  4. multipathd reconfigure && multipath -ll — one map, multiple ready paths.
  5. Create partitions with kpartx or whole-LUN LVM on /dev/mapper/mpathX only.
  6. Fail a path with multipathd fail path … under load; confirm I/O survives.
  7. Point fstab/crypttab/LVM at mapper nodes; rebuild initramfs if needed at boot.
  8. Add a daily health oneshot so silent path loss is not discovered during the next outage.

Redundant cables only help if the OS treats them as one volume. Multipath is how Linux does that — configure it once, test failover on purpose, and stop treating /dev/sdX as stable storage names.

Sources and references

Top comments (0)