DEV Community

DaShaun
DaShaun

Posted on • Originally published at dashaun.com on

Raspberry Pi 3B+ ISCSI boot with RaspiOS ARM64

#raspberrypi

Diskless Raspberry Pi 3B+

Raspios 64-bit recently went GA and I recently found new joy in my Raspberry Pi 3B+ and Raspberry Pi Zero 2 W devices.

Now feels like a great time to revisit diskless booting.

This is what I'm using:

  • UniFi for DHCP
  • QNAP NAS for TFTP and ISCSI
  • Raspberry Pi 3B+

Install RaspiOS 64-bit

Use rpi-imager and install the GA version of 64-bit RaspiOS!

Enable USB Boot Mode

The devices that I have are 4-6 years old. I'm not sure if new ones will have this set already or not.

Run this command, the "OTP dump", to see if usb_boot_mode is enabled:

vcgencmd otp_dump | grep 17:
Enter fullscreen mode Exit fullscreen mode

This is what my OTP dump looked like before usb_boot_mode enabled.

17:1020000a
Enter fullscreen mode Exit fullscreen mode

If the output looks like this, get the latest firmware:

sudo rpi-update
Enter fullscreen mode Exit fullscreen mode

I was a little nervous about the rpi-update, but I'm also following my own notes.

 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
 *** Backing up modules 5.10.63-v8+
#############################################################
WARNING: This update bumps to rpi-5.15.y linux tree
See: https://forums.raspberrypi.com/viewtopic.php?t=322879

'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.
##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   173  100   173    0     0    678      0 --:--:-- --:--:-- --:--:--   678
100  123M    0  123M    0     0  4052k      0 --:--:--  0:00:31 --:--:-- 3014k
 *** Updating firmware
 *** Updating kernel modules
 *** depmod 5.15.21-v8+
 *** depmod 5.15.21-v7+
 *** depmod 5.15.21-v7l+
 *** depmod 5.15.21+
 *** Updating VideoCore libraries
 *** Using SoftFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to fbcab73d5a20f592aa9f6b0e364757ef131dae27
 *** A reboot is needed to activate the new firmware
Enter fullscreen mode Exit fullscreen mode

After rebooting from the rpi-update we still need to enable usb_boot_mode, just needed the latest firmware to do so.

Add this line to /boot/config.txt it doesn't matter where, but needs its own line. I put it at the top.

program_usb_boot_mode=1
Enter fullscreen mode Exit fullscreen mode

Then reboot the device again. Now the OTP dump should look like this:

17:3020000a
Enter fullscreen mode Exit fullscreen mode

Now that the OTP dump is set, it is safe to remove the program_usb_boot_mode=1 line from /boot/config.txt file. There is no need to reboot, but if you remove the line, and reboot, you will see that the OTP dump still shows usb_boot_mode enabled.

Enable ISCSI module with initramfs tool

These are the steps I took:

# Update repositories
sudo apt update
# Install available upgrades
sudo apt upgrade -y
# Install open-iscsi
sudo apt install open-iscsi
# Start open-iscsi to create the configs that we will use later
sudo systemctl start open-iscsi
# Create initramfs flag file for the iscsi module
sudo touch /etc/iscsi/iscsi.initramfs
# Create a new initramfs image that includes the iscsi module from the current kernel
sudo update-initramfs -v -k `uname -r` -c
Enter fullscreen mode Exit fullscreen mode

You should now have an initrd.img in your /boot directory.

ls -latr /boot/initrd.img*
Enter fullscreen mode Exit fullscreen mode

The output should look like this:

-rwxr-xr-x 1 root root 10487750 Feb 11 10:59 /boot/initrd.img-5.15.21-v8+
Enter fullscreen mode Exit fullscreen mode

We will refer to this new initrd.img file later!

Add TFTP options to DHCP server

At this point, the bootloader on our device will accept a network boot, similar to PXE, but not exactly the same.

Setup your DHCP server to point to your TFTP server. The DHCP protocol has Option 66 for TFTP Server. Basically, you want to tell your DHCP client (the Raspberry Pi 3B+) which server to connect to next, to get further instructions.

Setup TFTP Server

The network boot process for the Raspberry Pi 3B+ is PXE-like. So it requires a slightly custom setup. It requires the serial number of the device, to be known in advance.

Grab the serial number of the device:

grep Serial /proc/cpuinfo
Enter fullscreen mode Exit fullscreen mode

In my case it looks like this:

Serial      : 000000002edd5668
Enter fullscreen mode Exit fullscreen mode

On the TFTP server, create a directory for the device. In the case of my QNAP server, the TFTP root is at /share/tftp but I'll use the variable TFTP_ROOT.

# Set the implementation specific TFTP_ROOT
export TFTP_ROOT=/share/tftp
# Create directory using the device serial number, without leading 0s
sudo mkdir $TFTP_ROOT/2edd5668
Enter fullscreen mode Exit fullscreen mode

Now, from the TFTP server, copy over the /boot partition, of the Raspberry Pi, that includes our new kernel.

sudo scp -r pi@2edd5668:/boot/* $TFTP_ROOT/2edd5668/
Enter fullscreen mode Exit fullscreen mode

If you want to check your work, the directory should look something like this:

COPYING.linux*            bcm2708-rpi-cm.dtb*       bcm2710-rpi-3-b-plus.dtb* bcm2711-rpi-4-b.dtb*      cmdline.txt*              fixup4db.dat*             initrd.img-5.15.21-v8+*   kernel8.img*              start4db.elf*
LICENCE.broadcom*         bcm2708-rpi-zero-w.dtb*   bcm2710-rpi-3-b.dtb*      bcm2711-rpi-400.dtb*      config.txt*               fixup4x.dat*              issue.txt*                overlays/                 start4x.elf*
bcm2708-rpi-b-plus.dtb*   bcm2708-rpi-zero.dtb*     bcm2710-rpi-cm3.dtb*      bcm2711-rpi-cm4.dtb*      fixup.dat*                fixup_cd.dat*             kernel.img*               start.elf*                start_cd.elf*
bcm2708-rpi-b-rev1.dtb*   bcm2709-rpi-2-b.dtb*      bcm2710-rpi-zero-2-w.dtb* bcm2711-rpi-cm4s.dtb*     fixup4.dat*               fixup_db.dat*             kernel7.img*              start4.elf*               start_db.elf*
bcm2708-rpi-b.dtb*        bcm2710-rpi-2-b.dtb*      bcm2710-rpi-zero-2.dtb*   bootcode.bin*             fixup4cd.dat*             fixup_x.dat*              kernel7l.img*             start4cd.elf*             start_x.elf*
Enter fullscreen mode Exit fullscreen mode

Now, because this is PXE-like, we need to copy the bootcode.bin to the TFTP_ROOT

sudo cp $TFTP_ROOT/2edd5668/bootcode.bin $TFTP_ROOT/
Enter fullscreen mode Exit fullscreen mode

At this point, without an SD Card installed, the Raspberry Pi 3B+ will:

  • Ask the DHCP server for an IP address
  • Our DHCP server will provide an IP address AND tell the device to visit the TFTP server for more info
  • Our device will ask the TFTP server for $TFTP_ROOT/bootcode.bin
  • Our device will try to boot from $TFTP_ROOT/2edd5668/

At this point though, our $TFTP_ROOT/2edd5668/ exactly matches what is on the SD Card in our device. We need to tell the device to use our new ISCSI enabled kernel.

I added the following line to the bottom of $TFTP_ROOT/2edd5668/config.txt, but you should use your specific initrd.img version.

initramfs initrd.img-5.15.21-v8+ followkernel
Enter fullscreen mode Exit fullscreen mode

Create ISCSI LUN for the device

For simplicity, I'm creating a 32GB LUN, with "thick" provisioning on my NAS, and mapping that to an ISCSI Target. I chose 32GB simply because that is the size of SDcard that I have been using. I also used the serial number, of the Raspberry Pi, in the naming of the LUN and ISCI target.

To verify that your new ISCSI LUN is accessible we will connect to it from the Raspberry Pi 3B+.

# set the IP address of the ISCSI server
export ISCSI_IP=192.168.1.248
# 
sudo iscsiadm -m discovery -t sendtargets -p $ISCSI_IP | grep 2edd5668
Enter fullscreen mode Exit fullscreen mode

The output should contain the new ISCSI target.

192.168.1.248:3260,1 iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
Enter fullscreen mode Exit fullscreen mode

From above the TARGET_IQN is iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c and I create an enviroment varialbe to store it.

export TARGET_IQN=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
Enter fullscreen mode Exit fullscreen mode

Mount the ISCSI LUN to the Raspberry Pi

Login to the ISCSI LUN, this example is not using CHAP authentication.

sudo iscsiadm -m node -l -T $TARGET_IQN -p $ISCSI_IP
Enter fullscreen mode Exit fullscreen mode

Success should look similar to this:

Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260]
Login to [iface: default, target: iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c, portal: 192.168.1.248,3260] successful.
Enter fullscreen mode Exit fullscreen mode

I validate the new ISCSI connection using sudo fdisk -l from the Raspberry Pi 3B+ and look for something like this, showing the 32GB LUN attached via ISCSI

Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: iSCSI Storage   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 1048576 bytes / 1048576 bytes
Enter fullscreen mode Exit fullscreen mode

The LUN is showing up as /dev/sda in this case.

# Format the LUN attached as /dev/sda to ext4
sudo mkfs.ext4 /dev/sda
Enter fullscreen mode Exit fullscreen mode 
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: f581fa52-f7ea-4c87-91cc-e236d64f63a6
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done
Enter fullscreen mode Exit fullscreen mode 
# Get the UUID of the LUN
sudo blkid /dev/sda
Enter fullscreen mode Exit fullscreen mode 
/dev/sda: UUID="f581fa52-f7ea-4c87-91cc-e236d64f63a6" BLOCK_SIZE="4096" TYPE="ext4"
Enter fullscreen mode Exit fullscreen mode 
# Mount the new disk
sudo mount /dev/sda /mnt
Enter fullscreen mode Exit fullscreen mode

Copy the existing installation to the mounted LUN

This step takes some time. We are copying all of the files from the / partition of the SD card on the Raspberry Pi 3B+ to the network attached LUN. We are excluding the directories that are mounted during boot.

# Copy / partition to /mnt
sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
# Create directories that are mounted during boot
sudo mkdir /mnt/{dev,proc,sys,boot,mnt}
Enter fullscreen mode Exit fullscreen mode

Remove SD Card details from /etc/fstab 

cat /mnt/etc/fstab
Enter fullscreen mode Exit fullscreen mode

Before:

proc            /proc           proc    defaults          0       0
PARTUUID=9633a3a2-01  /boot           vfat    defaults          0       2
PARTUUID=9633a3a2-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
Enter fullscreen mode Exit fullscreen mode

Remove the lines containing PARTUUID which are referring to the SDCard.

After:

proc            /proc           proc    defaults          0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
Enter fullscreen mode Exit fullscreen mode

Update the kernel configs on TFTP server

Now that we have updated our filesystem. We need to revisit kernel that is getting loaded over the network. We need to tell it how to mount the / filesystem via ISCSI.

The ISCSI Initiator, which identifies the Raspberry Pi 3B+ device, can be found at /etc/iscsi/initiatorname.iscsi

sudo cat /etc/iscsi/initiatorname.iscsi
Enter fullscreen mode Exit fullscreen mode 
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:2ccb20a316
Enter fullscreen mode Exit fullscreen mode

On the TFTP server, I'm modifying the $TFTP_ROOT/2edd5668/cmdline.txt file.

I'm adding/updating these details for mounting the ISCSI drive:

  • ip=::::2edd5668:eth0:dhcp
    • hostname of the device
  • root=UUID=f581fa52-f7ea-4c87-91cc-e236d64f63a6
    • From the blkid command against the LUN
  • ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316
  • ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c
  • ISCSI_TARGET_IP=192.168.1.248
  • ISCSI_TARGET_PORT=3260
  • rw

Before:

console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0
Enter fullscreen mode Exit fullscreen mode

After:

console=serial0,115200 console=tty1 root=PARTUUID=9633a3a2-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_enable=pids cgroup_memory=1 systemd.unified_cgroup_hierarchy=0 ISCSI_INITIATOR=iqn.1993-08.org.debian:01:2ccb20a316 ISCSI_TARGET_NAME=iqn.2004-04.com.qnap:ts-231p:iscsi.2edd5668.21250c ISCSI_TARGET_IP=192.168.1.248 ISCSI_TARGET_PORT=3260 rw
Enter fullscreen mode Exit fullscreen mode

This all needs to be on a single line.

Ready to reboot!

From the Raspberry Pi 3B+ device:

# Unmount the drive
sudo umount /mnt
# Disconnect from ISCSI
sudo iscsiadm -m node --logout all
# Shutdown
sudo shutdown now
Enter fullscreen mode Exit fullscreen mode

Remove the SD card and restart the device!

Summary

If you were paying close attention, you may have notice that the cmdline.txt also had some non-default settings in it. Those are there to allow k3s to run as well.

This article is very specific to the Raspberry Pi 3B+ and using ISCSI. Look for a follow-up example using the Raspberry Pi 4.

Soon, I'll document a diskless-pipeline for setting up Raspberry Pi 3B+ and Raspberry Pi 4, that includes using arkade and k3sup to configure the kubernetes abstraction layer across all of these homelab devices.

Acknowledgements and References

Top comments (0)

Read next

yhshanto profile image

Build Cheapest High-Performance Serverless WebSocket Solution

Yousuf Hossain -

aaattt profile image

８章５と６

TANIAOKA, Akihiro -

sitebid profile image

How to Analyze Data on Your Utility Infrastructure Map

SiteBid -

chuckcarpenter profile image

Introducing Shepherd Pro: Adding User Journey Analytics

Chuck Carpenter -