DEV Community

hosni1982
hosni1982

Posted on

RHEL: /data Disappeared After Reboot? Troubleshoot LVM and Multipath Before Touching pvcreate

#linux #devops #sysadmin #debugging

A filesystem disappearing after a reboot is one of those incidents where a quick "fix" can make the situation much worse.

Imagine this:

/data is missing
Application cannot start
systemd reports a mount failure
Server may even enter emergency mode
Enter fullscreen mode Exit fullscreen mode

The first reaction might be:

pvcreate
vgcreate
lvcreate
mkfs.xfs
Enter fullscreen mode Exit fullscreen mode

On an existing production LUN, that can be exactly the wrong thing to do.

The problem may not be LVM at all.

Start with the failure layers

In an enterprise RHEL environment, /data can depend on several layers:

/data
  ↓
Filesystem
  ↓
Logical Volume
  ↓
Volume Group
  ↓
Physical Volume
  ↓
Multipath
  ↓
SAN LUN
  ↓
Storage paths
Enter fullscreen mode Exit fullscreen mode

The goal is simple:

Find the first broken layer before changing anything.

1. Is /data actually expected to be mounted?

Start with:

findmnt /data
grep -w '/data' /etc/fstab
lsblk -f
Enter fullscreen mode Exit fullscreen mode

Then inspect the boot errors:

journalctl -b -p err..alert --no-pager
Enter fullscreen mode Exit fullscreen mode

You may see something like:

Failed to mount /data
Dependency failed for Local File Systems
Enter fullscreen mode Exit fullscreen mode

That only proves the mount failed.

It does not prove that the filesystem is corrupted.

2. Does the Logical Volume exist?

Check LVM:

lvs -a -o lv_name,vg_name,lv_attr,devices
Enter fullscreen mode Exit fullscreen mode

A healthy configuration might show:

LV       VG       Attr       Devices
lv_data  vg_data  -wi-a----- /dev/mapper/mpatha(0)
Enter fullscreen mode Exit fullscreen mode

If the LV is missing, move down one layer.

Do not recreate it yet.

3. Does the Volume Group exist? 

vgs
Enter fullscreen mode Exit fullscreen mode

Expected example:

VG       #PV #LV #SN Attr   VSize    VFree
vg_data    1   1   0 wz--n- 500.00g  20.00g
Enter fullscreen mode Exit fullscreen mode

If vg_data is missing, resist the temptation to run vgcreate.

Check the PV first.

4. Is the Physical Volume visible? 

pvs -o pv_name,vg_name,pv_size,pv_free
Enter fullscreen mode Exit fullscreen mode

Expected:

PV                  VG       PSize    PFree
/dev/mapper/mpatha  vg_data  500.00g  20.00g
Enter fullscreen mode Exit fullscreen mode

If /dev/mapper/mpatha is missing, the problem is probably below LVM.

Now the investigation changes direction.

5. Check Multipath 

multipath -ll
Enter fullscreen mode Exit fullscreen mode

Example of a healthy map:

mpatha (3600508b400105df70000e00000ac0000) dm-3
size=500G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:1 sdb 8:16 active ready running

`-+- policy='service-time 0' prio=10 status=enabled
  `- 3:0:0:1 sdc 8:32 active ready running
Enter fullscreen mode Exit fullscreen mode

You can inspect the individual paths with:

multipathd show paths
Enter fullscreen mode Exit fullscreen mode

and:

lsscsi
Enter fullscreen mode Exit fullscreen mode

If the expected WWID is absent completely, recreating the VG will not fix the root cause.

The SAN LUN may simply not be visible to the server.

A typical failure chain

Suppose your checks give this result:

findmnt /data
    → not mounted

lvs
    → lv_data missing

vgs
    → vg_data missing

pvs
    → expected PV missing

multipath -ll
    → expected WWID missing
Enter fullscreen mode Exit fullscreen mode

Now the picture is very different:

SAN LUN not visible
        ↓
Multipath map missing
        ↓
PV missing
        ↓
VG missing
        ↓
LV missing
        ↓
Filesystem unavailable
        ↓
/data not mounted
Enter fullscreen mode Exit fullscreen mode

The missing filesystem was only the visible symptom.

The real failure occurred much lower in the stack.

Commands I would avoid at this stage

Unless you have proven that the storage is new and intentionally being initialized, avoid commands such as:

pvcreate /dev/mapper/mpatha
vgcreate vg_data /dev/mapper/mpatha
lvcreate ...
mkfs.xfs ...
Enter fullscreen mode Exit fullscreen mode

These are creation commands, not troubleshooting commands.

On an existing production disk, they may overwrite metadata you are trying to recover.

Evidence to collect before escalation

Before calling the storage, SAN, VMware or infrastructure team, collect evidence:

date
hostnamectl
uname -r
lsblk -f
findmnt
pvs
vgs
lvs -a -o +devices
multipath -ll
multipathd show paths
journalctl -b -p err..alert --no-pager
Enter fullscreen mode Exit fullscreen mode

In complex cases, and if permitted by your environment:

sos report
Enter fullscreen mode Exit fullscreen mode

This gives the next support level something concrete to work with.

The troubleshooting habit that matters

When /data disappears, don't immediately ask:

Which command can recreate /data?

Ask:

Which command proves why /data disappeared?

That small change in mindset can prevent a recoverable infrastructure incident from becoming a data-loss incident.

Free Linux troubleshooting resources

I maintain a small public project called Linux Ops Rescue Kit with free RHEL troubleshooting examples and read-only diagnostic tools.

The GitHub repository currently includes:

  • High load but low CPU troubleshooting
  • LVM / Multipath missing filesystem troubleshooting
  • A read-only Linux snapshot script
  • An evidence-first troubleshooting methodology

GitHub:

https://github.com/hosni1982/linux-ops-rescue-kit

The free release is also available here:

https://github.com/hosni1982/linux-ops-rescue-kit/releases/tag/v1.0-free

For engineers who need a larger on-call runbook, the complete Linux Ops Rescue Kit contains 30 production incident workflows, verification, rollback and Stop & Escalate guidance.

If you work on RHEL production systems, I'd be interested to know:

What is the most dangerous "quick fix" you've seen during a storage incident?

Top comments (0)