DEV Community

urgensherpa
urgensherpa

Posted on

slow boot disk initialization on linux

#linux #initrd

Scenario: the boot disk is slow SAN multipath disk where the root is installed, due to this the boot process fails

The Boot Process and The "Wait"

  1. Kernel and initramfs Start: After GRUB, the Linux kernel is loaded into memory along with the initramfs image. The kernel then unpacks the initramfs and executes the /init
    script within it.

  2. Parsing Kernel Parameters: The /init script (the main script you see in the file list) begins by parsing the kernel command line (/proc/cmdline). It looks for several parameters,
    but the most important ones for this scenario are:

    • root=...: This specifies the root device. For your SAN disk, this would likely be a UUID (e.g., root=UUID=...) or a device mapper path (e.g., root=/dev/mapper/mpath-a).
    • rootdelay=...: This is a crucial parameter. The init script explicitly parses this value and stores it in a shell variable ROOTDELAY.

  3. The Wait for the Root Device: The core logic for mounting a local filesystem is in /scripts/local. This script is called by the main /init script. The critical part of this
    process is the call to the wait-for-root program.

    • The source code for this is in src/wait-for-root.c. It's a small C program that takes two arguments: the DEVICE path and a TIMEOUT.
    • The init script will launch wait-for-root $ROOT $ROOTDELAY.
      1. The Race Condition: Your 50-Second Delay

This is where your scenario becomes interesting. The wait-for-root program does not just sleep; it actively listens for udev events. udev is the kernel's device manager, and it creates
device nodes (like /dev/sda, /dev/disk/by-uuid/...) as hardware is detected.

  • The Goal: wait-for-root is waiting for a udev event that announces the creation of the block device matching the root= parameter.
  • The Timeout: The program sets an alarm() for the number of seconds specified by ROOTDELAY. The default value for ROOTDELAY if not specified on the kernel command line is 30 seconds.

Here’s what happens:

  • Scenario A: rootdelay is too short (e.g., default 30s)

    1. The init script calls wait-for-root /dev/disk/by-uuid/... 30.
    2. Your SAN multipath disk is still initializing and hasn't been detected by the kernel yet. No udev events for it have been sent.
    3. After 30 seconds, the alarm() in wait-for-root goes off. The program exits with an error code.
    4. The init script sees the failure and cannot mount the root filesystem.
    5. Outcome: The boot process halts and drops you into the initramfs emergency shell (usually busybox). You will see a message like "Gave up waiting for root file system device."

  • Scenario B: rootdelay is long enough (e.g., rootdelay=60)

    1. The init script calls wait-for-root /dev/disk/by-uuid/... 60.
    2. The script waits.
    3. At around the 50-second mark, the multipath driver finishes its setup, and the kernel recognizes the final SAN device. udev creates the device node and its symlinks.
    4. wait-for-root receives the udev event, sees that it matches the device it's looking for, prints the filesystem type, and exits successfully.
    5. Outcome: The init script proceeds to mount the device, and the boot continues normally. You will experience a ~50 second pause during the early boot phase before the screen changes or Plymouth (the boot splash) appears.

Solution

In this scenario, the boot process will fail if the default rootdelay is used. The system will drop to an initramfs prompt because it gives up waiting before the SAN disk is ready.

To fix this, you must add or edit the rootdelay parameter on the kernel command line in your bootloader (GRUB).

  1. Edit /etc/default/grub.
  2. Find the GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX line.
  3. Add rootdelay=60 (or a higher value like 90 for safety) to the string. For example: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootdelay=60"
  4. Run sudo update-grub to apply the changes.
  5. Reboot.

Method 2:
Create a script that manually scans for new LUNs or waits for a specific device file to appear before allowing the boot to continue.

Example Scenario: A script that waits for your specific multipath device to appear.

  1. Create a hook script file:
    sudo touch /etc/initramfs-tools/hooks/wait_for_my_san
    sudo chmod +x /etc/initramfs-tools/hooks/wait_for_my_san

  2. Edit the script:
    The hook needs to copy a script into the initramfs that will run during boot.

    File: /etc/initramfs-tools/hooks/wait_for_my_san

    1     #!/bin/sh
    2     PREREQ=""
    3     prereqs()
    4     {
    5         echo "$PREREQ"
    6     }
    7 
    8     case $1 in
    9     prereqs)
   10         prereqs
   11         exit 0
   12         ;;
   13     esac
   14 
   15     . /usr/share/initramfs-tools/hook-functions
   16     # Copy a script into the initramfs to be run at boot time
   17     copy_exec /usr/local/sbin/wait-for-san-device /scripts/local-premount/
Enter fullscreen mode Exit fullscreen mode

  1. Create the boot-time script:
    sudo touch /usr/local/sbin/wait-for-san-device
    sudo chmod +x /usr/local/sbin/wait-for-san-device

  2. Edit the boot-time script:
    This script will contain the logic that polls for the device.

    File: /usr/local/sbin/wait-for-san-device


    1     #!/bin/sh
    2     # This script runs in the initramfs just before the root device is mounted.
    3 
    4     # The expected multipath device for our root filesystem
    5     # Replace with your actual device name from /dev/mapper/
    6     ROOT_MPATH_DEVICE="/dev/mapper/mpath_root"
    7 
    8     echo "Waiting for SAN device ${ROOT_MPATH_DEVICE} to appear..."
    9 
   10     # Wait for up to 90 seconds
   11     for i in $(seq 1 90); do
   12         if [ -b "${ROOT_MPATH_DEVICE}" ]; then
   13             echo "Found ${ROOT_MPATH_DEVICE}."
   14             exit 0
   15         fi
   16         sleep 1
   17     done
   18 
   19     echo "Gave up waiting for ${ROOT_MPATH_DEVICE}."
   20     # Returning a non-zero exit code might drop you to a shell
   21     exit 1
Enter fullscreen mode Exit fullscreen mode
  1. Rebuild the initramfs: sudo update-initramfs -u

Method 3

Create initramfs local premount script
e.g /etc/initramfs-tools/scripts/local-premount/scriptName

#!/bin/sh

PREREQ=""

prereqs()
{
  echo "$PREREQ"
}

case $1 in
prereqs)

  prereqs

exit 0

;;

esac

echo "Sleeping 60 seconds for san to be initialized"

/usr/bin/sleep 60

exit 0
Enter fullscreen mode Exit fullscreen mode

Now update-initramfs -u

Direct Copy (/etc/initramfs-tools/scripts/):
* Use Case: Best for adding simple, self-contained shell scripts that do not require any extra binaries or libraries beyond what's already included in the initramfs by default (like sleep, echo, mount, etc.).
* Advantage: Very simple and quick. No need to write a separate hook file.
* Disadvantage: It's a "dumb" copy. If your script needs a specific binary (e.g., multipath, lsscsi), this method will not automatically find and add that binary or its library dependencies. The script would fail during boot.

  • Hooks (/etc/initramfs-tools/hooks/):
    • Use Case: The standard, most robust method. It's necessary whenever your script has dependencies.
    • Advantage: Gives you full programmatic control. You can use helper functions like copy_exec to intelligently copy a binary and all of its required libraries into the initramfs. This is essential for any non-trivial task.
    • Disadvantage: Requires creating a separate hook file, which is slightly more work.

Top comments (0)