# Booting up Linux Kernel - Linux for Engineers #1

Ever wondered what happens when your Linux machine starts? There is a
fascinating song and dance between different components which brings a machine
to life. Lets dive into the world of booting Linux Kernel.

Linux kernel is a monumental achievement of collaboration in the world of
open-source software.

It has resulted in a powerful, versatile operating system base that powers an
estimated 80-90% of the modern tech world

From smartphones in our pockets to the vast cloud infrastructure, Linux is
the backbone of many technologies we rely on daily.

This series chronicles different aspects of modern day Linux system.

Each article aims to be a tldr; While linking to amazing
talks/presentations/articles by some great minds.

Treat this as a crash course for the lazy and a jump-off point for the
curious.

Happy reading!

Have feedback or questions, or want to be notified about more such articles?
Follow me on Twitter @wiresurfer

⚡ And then there was light! ⚡

Ever wondered what happens when you press the power button on your machines.

The fan's come to life, your RGB lights light up, there is a bell sound, and the
screen shows a sea of gibberish [or if you started using laptops, you probably
see a fancy logo and a spinner. How boring is that!]

By the end of this article, I want to unwrap the song and dance that goes on
between hardware and different types of software to finally bring to you a
usable PC.

I won't hold back on some technical details and gloss over some in the interest
of salvaging some brevity.

This post in particular will have a buildup and posturing in the beginning about
hardware, bios and grub.

While it isn't absolutely necessary for a Linux Kernel bootup saga, I do feel
its important for a well rounded understanding.

For the impatient, feel free to jump to
Linux Kernel Bootup Sequence

---

🖥️ Modern PC Architecture

A modern day PC hasn't changed its core design philosophy for over 30 years.
Yes, there has been miniaturization and our electronics design has improved by
leaps and bounds, but the basic building blocks have remained the same.

Here is a picture of a motherboard annotated to show different subsystems.

Even this can be simplified down to the following block diagram.

If you pay attention, most of the peripherals used with modern machines connect
over the PCI express bus.

• Most demanding high throughput devices are the graphics cards and memory
  modules. They attach directly to the CPU over dedicated lanes often referred
  to as the Northbridge.

• Southbridge on the other hand is a separate bus controller which manages
  other peripherals and connects to the CPU with a dedicated link.
  [Direct Media Interface or DMI3.0 is an Intel specific link]

One thing omitted in this block diagram is a plethora of
i2c, uart, serial ,
PWM and GPIO¹
which help the motherboard maintain its function.

Notable peripherals in a motherboard include:

• CMOS Clock [helps with timekeeping specially in embedded devices. Modern OS's often use NTP to sync time]
• Temperature Sensors ²
• PWM controllers for fan control

The first step in bringing the computer to working order is performed by a
special program called the BIOS which is programmed onto a ROM chip on the
motherboard, right from the factory. Lets see what happens in the BIOS.

🧩 BIOS and Power-On Self Test (POST)

BIOS is an embedded program responsible for starting the computer and performing
a POST (Power-On Self Test). UEFI is the new modern reincarnation on the BIOS
and the new kid on the block. From the Linux kernel bootup perspective, BIOS vs
UEFI has limited implications in the startup process³

1. Power-On Self Test (POST): During POST, the BIOS initializes the CPU and memory subsystem. It checks if these components are functioning correctly. If they start properly, the computer is ready to boot. However, this depends on the rest of the hardware functioning correctly as well.
2. Hardware Initialization: The BIOS initializes and lists all other peripheral hardware attached to the system. These peripherals are generally connected via buses like I2C, PCI Express, and SATA, which connect them to the CPU
3. CMOS Battery Check: Keeping correct time is important. Not so critical these days, but back in the day, a depleted CMOS battery could lead to a lot of mess. Often a motherboard would refuse to boot automatically if it detected a fault in the CMOS Clock.
4. RAM check: Verify RAM speed and make sure the bus speed of various subsystems are compatible. Overclockers tinker with the voltage levels of the RAM and the CPU to force it to run at higher than prescribed clock speeds. BIOS/UEFI these days protects such folks from frying up their threadripper/i9 cpu
5. Boot a Storage Device: Try to find bootable storage devices. Either a disk with Master Boot Record entries, or a GPT UUID partitioned disk.

Handing over to a Bootloader

As we learned BIOS acts as the first sanity check, initializing the system
and performing POST. It hands over control to a Bootloader, a somewhat more
advanced program that is tasked with loading operating systems

---

💽 Bootloader and GRUB

After POST, BIOS looks for special I/O devices to provide the next program to
run, typically the bootloader.

Locating the bootloader is done by following a few conventions which have been
in the PC world for decades. In short, BIOS has a configured set of boot
devices. This preference is stored in the CMOS.

After POST, BIOS will start going through each Boot Device and verify if it has
a Master Boot Record. This
is where GRUB enters the picture. GRUB
is a program residing in the Master Boot Record of a bootable device.

Here is a quick anatomy of the Master Boot Record. Its the first sector of a
disk. A disk sector is traditionally 512bytes. Modern disks have larger sectors
up to 4096 bytes [Advanced Format Disks] but even these disks access the
physical media in 512byte emulated mode [512e]

512bytes of the MBR are broken up as follows

• 1-446bytes - Bootloader code. Grub boot.img
• 64 bytes - Master Partition Entries, Up to 4 entries, 16 bytes each.
• Last 2bytes - A special magic number 0xAA55. This indicates to the BIOS, there is a bootloader on this disk device.

With that out of the way, lets look at GRUB as a bootloader.

Now. lets be honest. 512bytes is a tight space, and adding things like splash
image, nice graphics and menus to dual boot an OS will take way more than
512bytes. For eg. EasyBMP a tiny library to
display images on screen when statically linked is 20kb!

To work around this limitation, Grub does a multi-stage boot.

Stage 1: Boot.img, Fits in 440byte MBR. Its a simple sled program
which loads Stage 1.5 using a
LBA address jump

Stage 1.5: Core.img, Fits in 32kb. It has just enough file system modules
to load \/boot\/grub.

Stage 2: \/boot\/grub is a special folder in /boot partition or a path on
the root / partition. This is where all grub modules are available. It contains
module drives for a large set of IO devices including networking booting. It
also brings some nice things like a GUI, selection menus and splash pages!

Now all that theory out of the way, how do we boot the kernel itself? We usually
just select a menu and boom! Linux starts loading up?

Well, there is a bunch of configuration files and menu entries which makes that
happen. Just peek into /boot/grub/grub.cfg. But we aren't looking at GRUB and
its magic. Lets try to boot a system by first principles.

Here is the minimal set of commands you can use on a GRUB prompt to start a
Linux system.

insmod linux
set root=(hd0,1)
linux /boot/vmlinuz-3.13.0-29-generic root=/dev/sda1
initrd /boot/initrd.img-3.13.0-29-generic
boot

A quick description of what's happening.

• Load the linux grub module. This module tells grub how to start a Linux like operating system.
• hd0,1 - This is the tricky bit! This is grub's way of selecting a hard drive partition. varies on each machine, depending on how it was partitioned. This stackexchange discussion should come in handy
• /boot/vmlinuz-3.13.0-29-generic is the compressed compiled linux kernel. Tab completion is your friend. Version number 3.13.0 would likely change in your case.
• /dev/sda1 is the root file system device.
• /boot/initrd.img-3.13.0-29-generic is the initial ram disk
• boot kicks off the boot process and runs vmlinux.

Once we write boot, GRUB is officially trying to hand over control to the
kernel. Phew!

Future Topics

If you're interested, I can cover BIOS and bootloader in more detail in future
posts. However, for practicality, most of us won't be dealing directly with
building BIOS or bootloaders. This high-level overview should provide a clear
understanding of the initial steps in the boot sequence.

---

🐧 Linux Kernel Bootup Sequence

Many seasoned developers have observed a Linux system boot up, witnessing a
stream of text scrolling by on the screen. This output primarily consists of
driver initializations and service startups. Despite its apparent complexity,
the Linux kernel boot process is relatively straightforward. Let's delve into
the most prevalent Linux kernel boot sequence

Bare Minimum Linux Kernel?

As we saw in the grub commands, achieving a minimal boot for a practical PC
experience requires,

• vmlinuz : Compressed compiled linux kernel binary
• initramfs or initrd : compressed archive which can be expanded by vmlinuz or grub and placed into memory
• (additionally) root filesystem

Linux follows a two stage booting process. The bootloader first loads the stage
1 kernel.

Stage 1 kernel's aim is simple and can be listed as follows

• Load just enough kernel modules to mount a proper filesystem.
• Mount said root file system
• Hand over control to an init executable in the root filesystem. (/init, /sbin/init or configured paths in linux/init/main.c at torvalds/linux · GitHub)

Because storage hardware comes in various formats (curse of linux's ubiquity),
kernel developers have chosen to break the booting process into an initial
in-memory file system load (Initramfs or initrd) which then mounts and
loads the root filesystem

Stage 2 kernel boot is where the init process starts configuring hardware and
running services, usually running programs initiated in userspace but invoking
kernel space syscalls. This ensures the boot process can be configured without
having to recompile the kernel for every small customizations. This design
choice is the reason why we all aren't kernel developers (yet)

Note: About minimal linux

For a truly minimal boot experience we could have a stripped down vmlinuz
which runs a statically compiled /sbin/init program to print hello world!
all in < 10mbs of RAM, if you configure your kernel boot correctly!

Initramfs and initrd (initial ramdisk) serve a similar purpose.
initramfs is a modern take on initrd;

Write to me if you think I should write about the internals of
initramfs/initrd.

PS: A quick search would point to some great resources.

I do find
What’s the Difference Between initrd and initramfs? | Baeldung on Linux
a good reference.

🌴 Root File System : Linux Kernel Directory Structure

Upon starting, the Linux kernel prepares and virtually presents some special
file system in a specific way. The root file system and the initial RAM file
system (initramfs) are crucial parts of this structure.

Initramfs is the first file system that the kernel mounts and operates in
memory, rather than on disk.

Root filesystem is then loaded and presents more kernel modules, libraries,
binary utilities and daemons.

The root filesystem also starts various daemons which could further mount other
storage devices and filesystems.

We need to be aware of /proc and /sys directories, which contain information
about the running kernel and allow certain kernel parameters to be modified.

We also need to learn how all these file systems are combined together to
provide a singular working view of the running system using overlayfs

Init Process and the Role of PID 1: systemd and upstart in modern linux distros

In the world of Linux, PID 1 holds a special place. PID 1, or Process ID 1, is
the first process started by the Linux kernel during the boot sequence and is
the ancestor of all other processes. Understanding PID 1 and its role is crucial
for grasping how a Linux system initializes and manages services.

When the Linux kernel finishes its initial setup, it launches the first
user-space process, which is assigned PID 1. Traditionally, this process was the
init system, responsible for starting system processes, handling system
initialization, and managing services. The init system follows a predefined
sequence to bring the system to a usable state.

PID 1 is critical because it remains running as long as the system is up and
serves as the parent for all other processes. If PID 1 terminates, the kernel
will panic, causing the system to halt or reboot, as there would be no process
to adopt orphaned child processes.

systemd as init

In modern Linux distributions, systemd has largely replaced the traditional
init system as the default system and service manager. systemd is designed
to provide a more efficient and feature-rich way of managing system processes
and services. It still occupies the PID 1 slot and takes on the responsibilities
of its predecessor but with enhanced capabilities.

How systemd Works?

When the kernel passes control to systemd as PID 1, systemd begins its
initialization process by mounting the initial file systems and starting
essential services. It reads its configuration from unit files located in
/etc/systemd/system/ and /usr/lib/systemd/system/. These unit files describe
how to manage services, sockets, devices, and other system components.

systemd also sets up the cgroups (control groups) to manage resource
allocation and limits for processes

Kernel Privilege rings and security. PID 1's security implications

In the architecture of modern computer systems, privilege rings play a crucial
role in maintaining security and stability. The Linux kernel operates in these
rings to control access to resources and enforce security policies.
Understanding the concept of privilege rings and the security implications of
PID 1 helps in comprehending how Linux ensures a secure environment.

Privilege Rings Explained

Privilege rings are hierarchical levels of privileges that a system's processes
can have. They range from Ring 0, the highest level of privilege, to Ring 3, the
lowest.

• Ring 0 (Kernel Mode): The most privileged level, where the operating system kernel operates. It has unrestricted access to all system resources and hardware.
• Ring 3 (User Mode): The least privileged level, where user applications run. It has restricted access, requiring sudo to elevate to kernelmode

The kernel mode (Ring 0) allows the operating system to execute critical tasks
that require direct access to hardware and memory. User mode (Ring 3) provides a
restricted environment for applications, preventing them from directly accessing
hardware and system memory, thus protecting the system from malicious software
and user errors.

Ironically Ring 1 and Ring 2 have felt out of favor.

As a hobbyist Linux nerd, I discovered that the benefits of rings 1 and 2 in
the modern protection model are greatly diminished due to paging only
distinguishing between privileged (ring 0, 1, 2) and unprivileged levels.

Anecdotally Intel designed rings 1 and 2 to house device drivers, providing
them with some privileges while keeping them somewhat separate from kernel
code. Although rings 1 and 2 can access supervisor pages, they still trigger a
General Protection Fault (GPF) if they use a privileged instruction, similar
to ring 3. Despite this, rings 1 and 2 are useful in certain designs.

For instance, VirtualBox places guest kernel code in ring 1, and some
operating systems do utilize them, though it is not a widely popular design
choice at present.

🏁 Conclusion

Dear reader, I hope this gives you a glimpse into the fascinating engineering
that powers your PC.

We followed a popular path through the woods of bootloaders and linux kernels.

In reality, there are many ways to boot a machine. We've got three components to
mix and match. The BIOS, Bootloader and the OS/Kernel.

In modern devices with a single powerful System on Chip design, we sometimes
encounter BIOS and Bootloader becoming very slim and having features to offer
OEM locking and security. In most systems there is a proprietary "BIOS".⁴

Keen readers would noticed how Grub was a two stage bootloader, and Linux kernel
was a two stage OS boot system. There have been efforts like Direct Kernel
Boot⁵ to merge these two components + four steps into a streamlined two
step process. This speeds up the boot process and is usually seen as an option
in Virtualization solutions and hypervisors.

In no particular order or importance, here are some interesting projects
powering the modern device bootup space.

• U-Boot - Usually used in embedded devices. Gives fine grained control of where the kernel gets loaded into device memory. As seen in:
  • SpaceX Dragon/falcon/
  • Ubiquiti/TP-Link network devices
• Coreboot
  • Chrome OS devices, Lenovo ThinkPad's and
• Android Devices :
  • While android uses a variant of the Linux kernel, there is no standard boot loader prescribed for running android os.
  • OEMs implement their own boot loader depending on the storage options available and the SoC used.
  • Qualcomm chipsets use Little Kernel and try to be UEFI compatible.
  • MediaTek chipsets use a variant of U-Boot.
  • Major players like Samsung have their own variants of the bootloader.
• Cloud VMs : AWS/Azure/GCP support custom virtualized BIOS/UEFI for their VMs.
  • Cloud BIOS supports emulating keyboard/serial console during the boot process, even before the bootloader or OS has loaded. This makes supporting any standard bootloader possible on cloud infrastructure. Critical for disaster recovery scenarios where your server, 7000kms away stops booting!
  • Virtio and OS Images: Cloud Boot disks are virtualized and attached on demand. You could explore virtio locally on qemu kvm. Virtualized storage enables using pre-built images for different operating systems. A superpower for running repeatable consistent infrastructure at scale!
  • Cloud-Init : They also have additional initialization steps after the kernel and operating system start called cloud-init. This helps setup networking, user accounts, passwords and ssh keys, networking and services. It also helps platform operators to further extend the boot process.

---

Have feedback or questions, or want to be notified about more such articles?
Follow me on Twitter @wiresurfer

---

Footnotes

---

1. GPIOs are common in embedded boards like raspberry pi or industrial PCs.
   Here is an insightful stackoverflow discussion about hacks to
   use GPIOs on desktop motherboards ↩

2. PWM Controllers and Temperature Sensors together play a critical role in
   thermal management. Most modern UEFI motherboards offer fine grained control
   over the sensor readings and the Fan control curve. ↩

3. This is a loose statement and I accept its far from true. For all you
   advanced practitioners out there, remember we are writing this guide down to
   be approachable. I am shying away from a lot of complexity and keeping
   things simple.
   Adam Williamson from Redhat, who has written about Linux, has a great post about this ↩

4. "BIOS" is a symbolic name here. Each SoC needs to start the hardware and do
   some rudimentary POST. Most Mobile devices also need to initialize a special
   telephony subsystem which runs its own Baseband OS. A traditional BIOS/UEFI
   introduced here would mean pushing a square peg through a round hole. ↩

5. 20.2.3. Direct kernel boot Red Hat Enterprise Linux 6 | Red Hat Customer Portal ↩