DEV Community

Cover image for Longhorn CSI pvc attachment issues fixing with multipath
Kader Khan
Kader Khan

Posted on

Longhorn CSI pvc attachment issues fixing with multipath

#webdev #tutorial #opensource #devops

🚀 Longhorn CSI Mount Issue Fix

❗ Issue

Pods using Longhorn volumes may fail to start due to errors in longhorn-csi-plugin, specifically related to mount failures caused by multipathd.

🔍 Error Message 

Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-xxxx /var/lib/kubelet/pods/xxx/mount
Output: mount: /var/lib/kubelet/pods/xxx/mount: /dev/longhorn/pvc-xxxx already mounted or mount point busy.
Enter fullscreen mode Exit fullscreen mode

🎯 Root Cause

The multipath daemon (multipathd) automatically creates multipath devices for block devices, including Longhorn volumes. This results in conflicts when mounting Longhorn volumes, preventing pods from starting.

✅ Solution

1️⃣ Check Longhorn Devices

Run the following command to list devices created by Longhorn:

lsblk
Enter fullscreen mode Exit fullscreen mode

🔹 Longhorn devices typically have names like /dev/sd[x].

2️⃣ Modify multipath.conf

  1. Create the configuration file (if it doesn’t exist): 
   sudo touch /etc/multipath.conf
Enter fullscreen mode Exit fullscreen mode
  1. Add the following blacklist rule: 
   blacklist {
       devnode "^sd[a-z0-9]+"
   }
Enter fullscreen mode Exit fullscreen mode

3️⃣ Restart Multipath Service

Apply the changes by restarting the multipath daemon:

sudo systemctl restart multipathd.service
Enter fullscreen mode Exit fullscreen mode

4️⃣ Verify Configuration

Check if the new configuration is applied:

multipath -t
Enter fullscreen mode Exit fullscreen mode

🎉 Your pods should now be able to mount Longhorn volumes correctly!

📌 Additional Tips

  • Ensure that longhorn-csi-plugin logs are clear of mount errors.
  • If the issue persists, consider rebooting the node after applying the fix.
  • Check the status of multipath with: 
  systemctl status multipathd.service
Enter fullscreen mode Exit fullscreen mode

🛠️ Need More Help?

🔹 Visit the Longhorn Documentation

🔹 Join the Longhorn Community

🚀 Happy Deploying!
Enter fullscreen mode Exit fullscreen mode

Top comments (0)