DEV Community: Shaishav Kumar | @wiresurfer

Manage Sensitive API Keys in Public Dotfiles Using PGP and SOPS

Shaishav Kumar | @wiresurfer — Wed, 11 Sep 2024 09:58:38 +0000

In modern development environment, it’s common to host dotfiles
publicly—especially for sharing across machines or with the broader developer
community. However, this convenience introduces security risks, such as
accidentally exposing sensitive information like API keys, tokens, and
credentials in public repositories. API keys for services like OpenAI,
Anthropic, Google Cloud, and more, if exposed, can lead to security breaches and
unauthorized access.

Imagine pushing your meticulously crafted dotfiles to GitHub, only to realize
hours later that you've just exposed your OpenAI API key to the entire internet.
Sounds like a nightmare? Unfortunately, it's an all-too-common reality for many
developers.

The recent explosion of interest in AI development has brought a wave of
excitement - and a flood of new developers eager to experiment with powerful AI
APIs. From ChatGPT, anthropic to midjourney, flux, these tools are reshaping how
we approach software development.

But with great power comes great responsibility, and I've noticed an alarming
trend: more and more developers, especially those new to development, are
accidentally exposing their API keys and other sensitive credentials in public
repositories.

This isn't just a rookie mistake - even seasoned developers can fall into this
trap when rushing to share their latest AI project or dotfiles setup. The
consequences can be severe: unauthorized API usage, compromised accounts, and in
some cases, significant financial losses. It's this recurring issue that
compelled me to write this guide. We need a solution that's both secure and
accessible, especially for those just starting their journey in AI development.

In today's interconnected development landscape, we often find ourselves walking
a tightrope between convenience and security. We want to share our development
setups across multiple machines, collaborate with team members, and contribute
to the open-source community. But how do we do this without compromising our
sensitive data? This post delves into a bare bones solution for this pervasive
problem. We'll explore how to leverage the power of PGP encryption and Mozilla's
SOPS to create a system where your API keys, tokens, and other secrets remain
hidden, even in public repositories. By the end of this guide, you'll have a
Unix-native, cost-effective method to:

Securely store sensitive data in your dotfiles
Easily share your development setup across multiple machines
Maintain the convenience of public repositories without sacrificing security

Whether you're a solo developer juggling multiple AI projects or part of a team
managing shared configurations, this guide will provide you with the tools to
keep your secrets... well, secret. Let's dive in and transform those vulnerable
dotfiles into a fortress of security!

Functional Requirements
Solutions Considered
Solution Blueprint
PGP
SOPS
Setting up SOPS with PGP
Conclusion
TLDR

Functional Requirements

Before diving into the solution, let’s outline the functional and non-functional
requirements:

Must work as early as .profile or .bashrc.
Should enable easy sharing of the same setup across multiple machines.
Must remain as Unix-native as possible (i.e., avoid desktop password managers or proprietary key-sharing solutions).
Should not introduce ongoing costs for key management.

Solutions Considered

Here are the primary options we explored for managing sensitive data in
dotfiles:

Enterprise Tools: Solutions like HashiCorp Vault or Azure KMS offer robust key management but come with added complexity and cost.
- Pros: Secure, widely used, highly customizable.
- Cons: Requires maintenance, has a learning curve, and may incur costs for smaller teams.
Password Managers or Secret Managers: Storing secrets in managers like LastPass or Bitwarden is an option.
- Pros: Easy to use, available on multiple platforms.
- Cons: Not ideal for dotfiles or Unix-native environments; overkill for this use case.
PGP + SOPS: This is the Unix-native method we will focus on. It’s simple, secure, and there’s no ongoing cost. PGP encrypts your secrets, and SOPS manages them effortlessly.
- Pros: Cost-effective, simple, Unix-friendly.
- Cons: Requires knowledge of PGP and some configuration.

Solution Blueprint

To manage your secrets in public dotfiles, follow these steps:

[ ] Install PGP (gpg).
[ ] Install Mozilla SOPS.
[ ] Generate a PGP key and store it securely.
[ ] Create a secrets file (~/secrets.env) outside of your git repository.
[ ] Use PGP and SOPS to encrypt the secrets file and commit the encrypted version to your dotfiles (~/dot-files/secrets.gpg.env).
[ ] Modify your .profile or .bashrc to decrypt and source the secrets directly into memory on startup.

Here's a diagram to visualize the solution flow:

PGP

PGP is a powerful encryption system that uses public-key cryptography. In
our solution, the client (your machine) uses a public key to encrypt the
data, and the server (or another machine) uses a private key to decrypt
it.

Here’s how it works:

You encrypt your secrets file using your public PGP key.
On the machine where you need the secrets (such as a server), you decrypt the file using the corresponding private key.

If you’re unfamiliar with PGP’s encryption mechanism, read this simplified
explanation of
Web of Trust.

SOPS

SOPS (Secrets OPerationS), created by Mozilla, is a powerful tool that makes
working with encrypted files easy. It supports PGP, AWS KMS, Google Cloud KMS,
and other encryption systems.

For our solution, we’ll use PGP with SOPS. SOPS ensures the secrets are
encrypted in place and can easily be decrypted when needed. You can either use a
single PGP key for all your environments or assign a unique key for each
service.

In more complex setups, you might want to look into using AWS KMS to manage
your keys, as it decouples key management from access credentials, which offers
a higher level of security for larger teams.

Setting up SOPS with PGP

Let’s walk through the step-by-step process of setting up SOPS with PGP
to secure your secrets:

Install PGP and SOPS.
Generate a PGP key by running gpg --gen-key and configuring as follows:
- Key type: RSA and RSA
- Key size: 2048
- Expiration: 0 (never expires)
- Real name: "{repo} PGP"
- Email: "{email}"
- Comment: "PGP credentials for {repo} secrets"

Create an encrypted secrets file using SOPS:

sops --pgp '{full_fingerprint}' secret.enc.yml

Store your private key securely, for instance, in a cloud vault or
outside the git repository:
```
gpg --export-secret-keys --armor {fingerprint} > private.rsa
```

Modify your .profile to decrypt the secrets on startup:

export $(sops --decrypt ~/dot-files/secrets.gpg.env | xargs)

Conclusion

As we've explored in this guide, managing sensitive information in public
dotfiles is not just a best practice—it's a critical component of modern secure
development, especially in the rapidly evolving world of AI and machine
learning.

By leveraging the power of PGP and SOPS, we've unlocked a robust, Unix-native
solution that allows us to:

Encrypt our secrets securely, ensuring they're only accessible to intended recipients
Share our development setups across multiple machines without compromising security
Contribute to open-source projects and share our dotfiles publicly with confidence

This approach scales well from individual developers to small and medium-sized
teams, integrating seamlessly with existing Unix-like environments. It's
particularly valuable for those working with AI APIs, where exposed credentials
can lead to significant security breaches and unexpected costs.

Remember, as you continue your journey in AI development or any field requiring
API keys and other secrets:

Regularly audit your public repositories for any accidentally exposed credentials
Consider implementing this PGP + SOPS solution as part of your standard development workflow
Educate your team or community about the importance of secure credential management

For those looking to take their security posture to the next level, consider
exploring more advanced setups supported by SOPS, such as integrating with AWS
KMS for larger-scale key management, or incorporating these practices into your
CI/CD pipelines.

Securing your secrets doesn't have to come at the cost of collaboration or
convenience. With the approach outlined in this guide, you can embrace the open,
sharing culture of the developer community while keeping your sensitive
information locked down tight.

Stay curious, keep experimenting with AI, and above all, keep your secrets
secret. Happy (and secure) coding!

Here's the steps in a single "not thoroghly tested" bash script. It should be
easy to follow and even easier to fix

TLDR

⚠️ Greybeard Territory: Proceed with Caution ⚠️

For the Impatient Greybeards: The All-in-One Setup Script

The following section contains a bash script that automates the entire setup
process. It's for those who prefer to dive in headfirst and tweak things
later. If you're new to this, we strongly recommend following the step-by-step
guide above instead.

Remember: With great power comes great responsibility. This script will make
changes to your system. Review it carefully before running, and maybe pour
yourself a strong cup of coffee first.

For those of you who scoffed at the detailed explanations and just want to get
things running, here's a bash script that sets up the entire PGP + SOPS
environment in one go. It's not thoroughly tested, so consider this a starting
point for your own customized setup.

To use this script:

Save it as setup_secrets.sh
Make it executable: chmod +x setup_secrets.sh
Run it: ./setup_secrets.sh

Remember, this script is a starting point. You might need to adjust it based on
your specific environment or requirements. And as always, review any script
carefully before running it on your system.

Now, for those of you who skipped straight to this section: go back and read the
rest of the post. There's valuable context and explanations up there that will
help you understand what this script is actually doing. Don't say we didn't warn
you!

#!/bin/bash

# Function to confirm actions from the user
confirm() {
    read -r -p "${1:-Are you sure? [y/N]} " response
    case "$response" in
        [yY][eE][sS]|[yY])
            true
            ;;
        *)
            false
            ;;
    esac
}

# Install GPG (GNU Privacy Guard)
if confirm "Do you want to install GPG (GNU Privacy Guard)? [y/N]"; then
    echo "Installing GPG..."
    sudo apt-get update
    sudo apt-get install -y gnupg
else
    echo "Skipping GPG installation."
fi

# Install SOPS (Secrets OPerationS)
if confirm "Do you want to install Mozilla SOPS? [y/N]"; then
    echo "Installing SOPS..."
    wget https://github.com/mozilla/sops/releases/download/v3.7.3/sops-v3.7.3.linux -O sops
    chmod +x sops
    sudo mv sops /usr/local/bin/
else
    echo "Skipping SOPS installation."
fi

# Remove SOPS default PGP keys
if confirm "Do you want to remove any default SOPS PGP keys? [y/N]"; then
    echo "Listing GPG keys..."
    gpg --list-keys
    echo "Enter the fingerprint of the key you want to delete (or press Enter to skip):"
    read -r fingerprint
    if [ -n "$fingerprint" ]; then
        echo "Deleting key with fingerprint $fingerprint..."
        gpg --delete-keys "$fingerprint"
        gpg --delete-secret-keys "$fingerprint"
    else
        echo "No keys deleted."
    fi
else
    echo "Skipping sops default key deletion."
fi
# Generate a GPG key
if confirm "Do you want to generate a new GPG key for encrypting secrets? [y/N]"; then
    echo "Generating GPG key..."
    gpg --gen-key
else
    echo "Skipping GPG key generation."
fi

# Create the secrets.env file
if confirm "Do you want to create a new secrets.env file at ~/secrets.env? [y/N]"; then
    echo "Creating a new secrets.env file at ~/secrets.env..."
    cat <<EOL > ~/secrets.env
# Add your secrets here in KEY=VALUE format
API_KEY=your_api_key_here
SECRET_KEY=your_secret_key_here
EOL
    echo "Created secrets.env. Please add your sensitive data."
else
    echo "Skipping secrets.env creation."
fi

# List available GPG keys and their fingerprints
echo "Encrypting your ~/secrets.env"
echo "Select Available GPG keys:"
gpg --list-keys --fingerprint

# Ask the user to select a GPG key fingerprint
echo "Enter the fingerprint of the GPG key you want to use for encryption:"
read -r fingerprint

# Check if the user input is empty
if [ -z "$fingerprint" ]; then
    echo "No fingerprint provided. Aborting encryption."
    exit 1
fi

# Encrypt the secrets.env file using the selected fingerprint and save it as secrets.gpg.env
if [ -f ~/secrets.env ]; then
    echo "Encrypting secrets.env into secrets.gpg.env..."
    sops --encrypt --pgp "$fingerprint" ~/secrets.env > ~/secrets.gpg.env
    echo "Encryption complete. Encrypted file saved as ~/secrets.gpg.env"
else
    echo "secrets.env not found in home directory. Please create it first."
    exit 1
fi

echo "Setup complete!"

Footnotes

PGP vs. SSH Encryption: Link to Wikipedia
SOPS GitHub Repo: SOPS on GitHub
Here's a great info-graphic of all the ways sops can be used by the good folks at GitGuardian
1. Infographic
2. Blog post

Unleash the Forbidden - Enabling eBPF/XDP for Kernel Tinkering on WSL2

Shaishav Kumar | @wiresurfer — Fri, 23 Aug 2024 07:39:00 +0000

Running eBPF on WSl2 running on Windows 10 feels forbidden. And yet, that's the entire premise of this blog.

Why the Heck Would We Do That?

As proud nerds, we find joy in understanding the deepest workings of our systems. Enabling eBPF in WSL2 is one of those pursuits that opens up a world of debugging possibilities. It allows us to use eBPF tools, provides full tracing support via DebugFS, and gives us insights into WSL2's networking stack. This article will guide you through enabling eBPF in WSL2. Future posts will delve deeper into specific use cases and the fascinating world of eBPF.

What's the Plan?

We'll focus on enabling eBPF on Ubuntu 20.04 running on WSL2. You can try it on 18.04, but your mileage may vary. Here’s a high-level overview of what we’ll do:

Recompile the WSL2 kernel with specific eBPF-related flags enabled.
Change the WSL2 configuration in Windows to point to our newly compiled kernel.
Restart WSL2.
Verify the new kernel is running and eBPF is enabled.

Show Me the Code?!

Let's get our hands dirty with some kernel compiling. hold tight.

1. Install Dependencies

First, make sure you have the necessary tools installed:



sudo apt update
sudo apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev

2. Clone the WSL2 Kernel Source

Download the source code of the WSL2 kernel:



git clone https://github.com/microsoft/WSL2-Linux-Kernel.git
cd WSL2-Linux-Kernel

3. Configure the Kernel

Copy the default WSL2 kernel configuration:



cp Microsoft/config-wsl .config

Linux kernel devs have made it intuitive to toggle various features in the linux kernel using a handy TUI.

To edit the build configuration of the kernel we have three options.

We use the TUI configurator, and then carefully toggle the features. Its gonna take a little more effort, but will let you see how you can change kernel characteristics. Great Learning, Would Recommend, at least once. :)



make menuconfig

In the menu, navigate to:

General setup
- Enable Debug Filesystem
- BPF subsystem
  - Enable BPF syscall support
  - Enable Enable bpf() system call
  - Enable Enable extended BPF (eBPF) JIT
Networking support
- Enable Networking options
- Enable Packet socket
- Enable Network packet filtering framework (Netfilter)
- Enable BPF-based packet filtering framework
- (optionally) Enable XDP sockets
  - (optionally) Enable XDP sockets: Monitoring interface
  - (optionally) enable BPF_STREAM_PARSES
  - (optionally) go into Network Testing
    - Enable as Module (M) Packet Generator (USE WITH CAUTION)

Alternatively, you can manually edit the .config file and ensure the following options are set:



CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m

Third and probably the most uninspired way would be, well here is a prepared config file. Download this, and place it in the WLS2-kernel folder.

4. Compile the Kernel

Compile the kernel with the new configuration:



make -j$(nproc)

5. Set WSL2 to Use the Custom Kernel

Copy the compiled kernel to a suitable location:



cp arch/x86/boot/bzImage /mnt/c/wsl_custom_kernel

Edit your .wslconfig file (create it if it doesn’t exist) in your Windows home directory (C:\Users\<YourUsername>\.wslconfig) and add the following lines:



[wsl2]
kernel=C:\\wsl_custom_kernel

Restart the WSL2 instance to apply the new kernel:



wsl --shutdown
wsl

6. Verify the New Kernel and eBPF

Check if your new kernel is running:



uname -r

You should see a version number corresponding to your custom kernel build.

Verify eBPF support:



ls /sys/kernel/debug

You should see bpf listed among the directories.

Happy Kernel Hacking!

Congratulations! You've successfully enabled eBPF in WSL2. This setup will serve as a solid foundation for further kernel debugging and customization. Stay tuned for future posts where we'll dive deeper into utilizing these capabilities for various debugging tasks.

](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n60s5eaj03ldp8rup8el.png)

# Booting up Linux Kernel - Linux for Engineers #1

Shaishav Kumar | @wiresurfer — Thu, 22 Aug 2024 18:18:05 +0000

Ever wondered what happens when your Linux machine starts? There is a
fascinating song and dance between different components which brings a machine
to life. Lets dive into the world of booting Linux Kernel.

Linux kernel is a monumental achievement of collaboration in the world of
open-source software.

It has resulted in a powerful, versatile operating system base that powers an
estimated 80-90% of the modern tech world

From smartphones in our pockets to the vast cloud infrastructure, Linux is
the backbone of many technologies we rely on daily.

This series chronicles different aspects of modern day Linux system.

Each article aims to be a tldr; While linking to amazing
talks/presentations/articles by some great minds.

Treat this as a crash course for the lazy and a jump-off point for the
curious.

Happy reading!

Have feedback or questions, or want to be notified about more such articles?
Follow me on Twitter @wiresurfer

⚡ And then there was light! ⚡

Ever wondered what happens when you press the power button on your machines.

The fan's come to life, your RGB lights light up, there is a bell sound, and the
screen shows a sea of gibberish [or if you started using laptops, you probably
see a fancy logo and a spinner. How boring is that!]

By the end of this article, I want to unwrap the song and dance that goes on
between hardware and different types of software to finally bring to you a
usable PC.

I won't hold back on some technical details and gloss over some in the interest
of salvaging some brevity.

This post in particular will have a buildup and posturing in the beginning about
hardware, bios and grub.

While it isn't absolutely necessary for a Linux Kernel bootup saga, I do feel
its important for a well rounded understanding.

For the impatient, feel free to jump to
Linux Kernel Bootup Sequence

🖥️ Modern PC Architecture

A modern day PC hasn't changed its core design philosophy for over 30 years.
Yes, there has been miniaturization and our electronics design has improved by
leaps and bounds, but the basic building blocks have remained the same.

Here is a picture of a motherboard annotated to show different subsystems.

Even this can be simplified down to the following block diagram.

If you pay attention, most of the peripherals used with modern machines connect
over the PCI express bus.

Most demanding high throughput devices are the graphics cards and memory
modules. They attach directly to the CPU over dedicated lanes often referred
to as the Northbridge.
Southbridge on the other hand is a separate bus controller which manages
other peripherals and connects to the CPU with a dedicated link.
[Direct Media Interface or DMI3.0 is an Intel specific link]

One thing omitted in this block diagram is a plethora of
i2c, uart, serial ,
PWM and GPIO¹
which help the motherboard maintain its function.

Notable peripherals in a motherboard include:

CMOS Clock [helps with timekeeping specially in embedded devices. Modern OS's often use NTP to sync time]
Temperature Sensors ²
PWM controllers for fan control

The first step in bringing the computer to working order is performed by a
special program called the BIOS which is programmed onto a ROM chip on the
motherboard, right from the factory. Lets see what happens in the BIOS.

🧩 BIOS and Power-On Self Test (POST)

BIOS is an embedded program responsible for starting the computer and performing
a POST (Power-On Self Test). UEFI is the new modern reincarnation on the BIOS
and the new kid on the block. From the Linux kernel bootup perspective, BIOS vs
UEFI has limited implications in the startup process³

Power-On Self Test (POST): During POST, the BIOS initializes the CPU and memory subsystem. It checks if these components are functioning correctly. If they start properly, the computer is ready to boot. However, this depends on the rest of the hardware functioning correctly as well.
Hardware Initialization: The BIOS initializes and lists all other peripheral hardware attached to the system. These peripherals are generally connected via buses like I2C, PCI Express, and SATA, which connect them to the CPU
CMOS Battery Check: Keeping correct time is important. Not so critical these days, but back in the day, a depleted CMOS battery could lead to a lot of mess. Often a motherboard would refuse to boot automatically if it detected a fault in the CMOS Clock.
RAM check: Verify RAM speed and make sure the bus speed of various subsystems are compatible. Overclockers tinker with the voltage levels of the RAM and the CPU to force it to run at higher than prescribed clock speeds. BIOS/UEFI these days protects such folks from frying up their threadripper/i9 cpu
Boot a Storage Device: Try to find bootable storage devices. Either a disk with Master Boot Record entries, or a GPT UUID partitioned disk.

Handing over to a Bootloader

As we learned BIOS acts as the first sanity check, initializing the system
and performing POST. It hands over control to a Bootloader, a somewhat more
advanced program that is tasked with loading operating systems

💽 Bootloader and GRUB

After POST, BIOS looks for special I/O devices to provide the next program to
run, typically the bootloader.

Locating the bootloader is done by following a few conventions which have been
in the PC world for decades. In short, BIOS has a configured set of boot
devices. This preference is stored in the CMOS.

After POST, BIOS will start going through each Boot Device and verify if it has
a Master Boot Record. This
is where GRUB enters the picture. GRUB
is a program residing in the Master Boot Record of a bootable device.

Here is a quick anatomy of the Master Boot Record. Its the first sector of a
disk. A disk sector is traditionally 512bytes. Modern disks have larger sectors
up to 4096 bytes [Advanced Format Disks] but even these disks access the
physical media in 512byte emulated mode [512e]

512bytes of the MBR are broken up as follows

1-446bytes - Bootloader code. Grub boot.img
64 bytes - Master Partition Entries, Up to 4 entries, 16 bytes each.
Last 2bytes - A special magic number 0xAA55. This indicates to the BIOS, there is a bootloader on this disk device.

With that out of the way, lets look at GRUB as a bootloader.

Now. lets be honest. 512bytes is a tight space, and adding things like splash
image, nice graphics and menus to dual boot an OS will take way more than
512bytes. For eg. EasyBMP a tiny library to
display images on screen when statically linked is 20kb!

To work around this limitation, Grub does a multi-stage boot.

Stage 1: Boot.img, Fits in 440byte MBR. Its a simple sled program
which loads Stage 1.5 using a
LBA address jump

Stage 1.5: Core.img, Fits in 32kb. It has just enough file system modules
to load \/boot\/grub.

Stage 2: \/boot\/grub is a special folder in /boot partition or a path on
the root / partition. This is where all grub modules are available. It contains
module drives for a large set of IO devices including networking booting. It
also brings some nice things like a GUI, selection menus and splash pages!

Now all that theory out of the way, how do we boot the kernel itself? We usually
just select a menu and boom! Linux starts loading up?

Well, there is a bunch of configuration files and menu entries which makes that
happen. Just peek into /boot/grub/grub.cfg. But we aren't looking at GRUB and
its magic. Lets try to boot a system by first principles.

Here is the minimal set of commands you can use on a GRUB prompt to start a
Linux system.

insmod linux
set root=(hd0,1)
linux /boot/vmlinuz-3.13.0-29-generic root=/dev/sda1
initrd /boot/initrd.img-3.13.0-29-generic
boot

A quick description of what's happening.

Load the linux grub module. This module tells grub how to start a Linux like operating system.
hd0,1 - This is the tricky bit! This is grub's way of selecting a hard drive partition. varies on each machine, depending on how it was partitioned. This stackexchange discussion should come in handy
/boot/vmlinuz-3.13.0-29-generic is the compressed compiled linux kernel. Tab completion is your friend. Version number 3.13.0 would likely change in your case.
/dev/sda1 is the root file system device.
/boot/initrd.img-3.13.0-29-generic is the initial ram disk
boot kicks off the boot process and runs vmlinux.

Once we write boot, GRUB is officially trying to hand over control to the
kernel. Phew!

Future Topics

If you're interested, I can cover BIOS and bootloader in more detail in future
posts. However, for practicality, most of us won't be dealing directly with
building BIOS or bootloaders. This high-level overview should provide a clear
understanding of the initial steps in the boot sequence.

🐧 Linux Kernel Bootup Sequence

Many seasoned developers have observed a Linux system boot up, witnessing a
stream of text scrolling by on the screen. This output primarily consists of
driver initializations and service startups. Despite its apparent complexity,
the Linux kernel boot process is relatively straightforward. Let's delve into
the most prevalent Linux kernel boot sequence

Bare Minimum Linux Kernel?

As we saw in the grub commands, achieving a minimal boot for a practical PC
experience requires,

vmlinuz : Compressed compiled linux kernel binary
initramfs or initrd : compressed archive which can be expanded by vmlinuz or grub and placed into memory
(additionally) root filesystem

Linux follows a two stage booting process. The bootloader first loads the stage
1 kernel.

Stage 1 kernel's aim is simple and can be listed as follows

Load just enough kernel modules to mount a proper filesystem.
Mount said root file system
Hand over control to an init executable in the root filesystem. (/init, /sbin/init or configured paths in linux/init/main.c at torvalds/linux · GitHub)

Because storage hardware comes in various formats (curse of linux's ubiquity),
kernel developers have chosen to break the booting process into an initial
in-memory file system load (Initramfs or initrd) which then mounts and
loads the root filesystem

Stage 2 kernel boot is where the init process starts configuring hardware and
running services, usually running programs initiated in userspace but invoking
kernel space syscalls. This ensures the boot process can be configured without
having to recompile the kernel for every small customizations. This design
choice is the reason why we all aren't kernel developers (yet)

Note: About minimal linux

For a truly minimal boot experience we could have a stripped down vmlinuz
which runs a statically compiled /sbin/init program to print hello world!
all in < 10mbs of RAM, if you configure your kernel boot correctly!

Initramfs and initrd (initial ramdisk) serve a similar purpose.
initramfs is a modern take on initrd;

Write to me if you think I should write about the internals of
initramfs/initrd.

PS: A quick search would point to some great resources.

I do find
What’s the Difference Between initrd and initramfs? | Baeldung on Linux
a good reference.

🌴 Root File System : Linux Kernel Directory Structure

Upon starting, the Linux kernel prepares and virtually presents some special
file system in a specific way. The root file system and the initial RAM file
system (initramfs) are crucial parts of this structure.

Initramfs is the first file system that the kernel mounts and operates in
memory, rather than on disk.

Root filesystem is then loaded and presents more kernel modules, libraries,
binary utilities and daemons.

The root filesystem also starts various daemons which could further mount other
storage devices and filesystems.

We need to be aware of /proc and /sys directories, which contain information
about the running kernel and allow certain kernel parameters to be modified.

We also need to learn how all these file systems are combined together to
provide a singular working view of the running system using overlayfs

Init Process and the Role of PID 1: systemd and upstart in modern linux distros

In the world of Linux, PID 1 holds a special place. PID 1, or Process ID 1, is
the first process started by the Linux kernel during the boot sequence and is
the ancestor of all other processes. Understanding PID 1 and its role is crucial
for grasping how a Linux system initializes and manages services.

When the Linux kernel finishes its initial setup, it launches the first
user-space process, which is assigned PID 1. Traditionally, this process was the
init system, responsible for starting system processes, handling system
initialization, and managing services. The init system follows a predefined
sequence to bring the system to a usable state.

PID 1 is critical because it remains running as long as the system is up and
serves as the parent for all other processes. If PID 1 terminates, the kernel
will panic, causing the system to halt or reboot, as there would be no process
to adopt orphaned child processes.

systemd as init

In modern Linux distributions, systemd has largely replaced the traditional
init system as the default system and service manager. systemd is designed
to provide a more efficient and feature-rich way of managing system processes
and services. It still occupies the PID 1 slot and takes on the responsibilities
of its predecessor but with enhanced capabilities.

How systemd Works?

When the kernel passes control to systemd as PID 1, systemd begins its
initialization process by mounting the initial file systems and starting
essential services. It reads its configuration from unit files located in
/etc/systemd/system/ and /usr/lib/systemd/system/. These unit files describe
how to manage services, sockets, devices, and other system components.

systemd also sets up the cgroups (control groups) to manage resource
allocation and limits for processes

Kernel Privilege rings and security. PID 1's security implications

In the architecture of modern computer systems, privilege rings play a crucial
role in maintaining security and stability. The Linux kernel operates in these
rings to control access to resources and enforce security policies.
Understanding the concept of privilege rings and the security implications of
PID 1 helps in comprehending how Linux ensures a secure environment.

Privilege Rings Explained

Privilege rings are hierarchical levels of privileges that a system's processes
can have. They range from Ring 0, the highest level of privilege, to Ring 3, the
lowest.

Ring 0 (Kernel Mode): The most privileged level, where the operating system kernel operates. It has unrestricted access to all system resources and hardware.
Ring 3 (User Mode): The least privileged level, where user applications run. It has restricted access, requiring sudo to elevate to kernelmode

The kernel mode (Ring 0) allows the operating system to execute critical tasks
that require direct access to hardware and memory. User mode (Ring 3) provides a
restricted environment for applications, preventing them from directly accessing
hardware and system memory, thus protecting the system from malicious software
and user errors.

Ironically Ring 1 and Ring 2 have felt out of favor.

As a hobbyist Linux nerd, I discovered that the benefits of rings 1 and 2 in
the modern protection model are greatly diminished due to paging only
distinguishing between privileged (ring 0, 1, 2) and unprivileged levels.

Anecdotally Intel designed rings 1 and 2 to house device drivers, providing
them with some privileges while keeping them somewhat separate from kernel
code. Although rings 1 and 2 can access supervisor pages, they still trigger a
General Protection Fault (GPF) if they use a privileged instruction, similar
to ring 3. Despite this, rings 1 and 2 are useful in certain designs.

For instance, VirtualBox places guest kernel code in ring 1, and some
operating systems do utilize them, though it is not a widely popular design
choice at present.

🏁 Conclusion

Dear reader, I hope this gives you a glimpse into the fascinating engineering
that powers your PC.

We followed a popular path through the woods of bootloaders and linux kernels.

In reality, there are many ways to boot a machine. We've got three components to
mix and match. The BIOS, Bootloader and the OS/Kernel.

In modern devices with a single powerful System on Chip design, we sometimes
encounter BIOS and Bootloader becoming very slim and having features to offer
OEM locking and security. In most systems there is a proprietary "BIOS".⁴

Keen readers would noticed how Grub was a two stage bootloader, and Linux kernel
was a two stage OS boot system. There have been efforts like Direct Kernel
Boot⁵ to merge these two components + four steps into a streamlined two
step process. This speeds up the boot process and is usually seen as an option
in Virtualization solutions and hypervisors.

In no particular order or importance, here are some interesting projects
powering the modern device bootup space.

U-Boot - Usually used in embedded devices. Gives fine grained control of where the kernel gets loaded into device memory. As seen in:
- SpaceX Dragon/falcon/
- Ubiquiti/TP-Link network devices
Coreboot
- Chrome OS devices, Lenovo ThinkPad's and
Android Devices :
- While android uses a variant of the Linux kernel, there is no standard boot loader prescribed for running android os.
- OEMs implement their own boot loader depending on the storage options available and the SoC used.
- Qualcomm chipsets use Little Kernel and try to be UEFI compatible.
- MediaTek chipsets use a variant of U-Boot.
- Major players like Samsung have their own variants of the bootloader.
Cloud VMs : AWS/Azure/GCP support custom virtualized BIOS/UEFI for their VMs.
- Cloud BIOS supports emulating keyboard/serial console during the boot process, even before the bootloader or OS has loaded. This makes supporting any standard bootloader possible on cloud infrastructure. Critical for disaster recovery scenarios where your server, 7000kms away stops booting!
- Virtio and OS Images: Cloud Boot disks are virtualized and attached on demand. You could explore virtio locally on qemu kvm. Virtualized storage enables using pre-built images for different operating systems. A superpower for running repeatable consistent infrastructure at scale!
- Cloud-Init : They also have additional initialization steps after the kernel and operating system start called cloud-init. This helps setup networking, user accounts, passwords and ssh keys, networking and services. It also helps platform operators to further extend the boot process.

Have feedback or questions, or want to be notified about more such articles?
Follow me on Twitter @wiresurfer

Footnotes

GPIOs are common in embedded boards like raspberry pi or industrial PCs.
Here is an insightful stackoverflow discussion about hacks to
use GPIOs on desktop motherboards ↩
PWM Controllers and Temperature Sensors together play a critical role in
thermal management. Most modern UEFI motherboards offer fine grained control
over the sensor readings and the Fan control curve. ↩
This is a loose statement and I accept its far from true. For all you
advanced practitioners out there, remember we are writing this guide down to
be approachable. I am shying away from a lot of complexity and keeping
things simple.
Adam Williamson from Redhat, who has written about Linux, has a great post about this ↩
"BIOS" is a symbolic name here. Each SoC needs to start the hardware and do
some rudimentary POST. Most Mobile devices also need to initialize a special
telephony subsystem which runs its own Baseband OS. A traditional BIOS/UEFI
introduced here would mean pushing a square peg through a round hole. ↩
20.2.3. Direct kernel boot Red Hat Enterprise Linux 6 | Red Hat Customer Portal ↩

DEV Community: Shaishav Kumar | @wiresurfer

Manage Sensitive API Keys in Public Dotfiles Using PGP and SOPS

Table of Contents

Functional Requirements

Solutions Considered

Solution Blueprint

PGP

SOPS

Setting up SOPS with PGP

Conclusion

TLDR

Footnotes

Unleash the Forbidden - Enabling eBPF/XDP for Kernel Tinkering on WSL2

Why the Heck Would We Do That?

What's the Plan?

Show Me the Code?!

1. Install Dependencies

2. Clone the WSL2 Kernel Source

3. Configure the Kernel

4. Compile the Kernel

5. Set WSL2 to Use the Custom Kernel

6. Verify the New Kernel and eBPF

Happy Kernel Hacking!

Further Reading

# Booting up Linux Kernel - Linux for Engineers #1

⚡ And then there was light! ⚡

🖥️ Modern PC Architecture

🧩 BIOS and Power-On Self Test (POST)

Handing over to a Bootloader

💽 Bootloader and GRUB

🐧 Linux Kernel Bootup Sequence

Bare Minimum Linux Kernel?

🌴 Root File System : Linux Kernel Directory Structure

Init Process and the Role of PID 1: systemd and upstart in modern linux distros

systemd as init

How systemd Works?

Kernel Privilege rings and security. PID 1's security implications

Privilege Rings Explained

🏁 Conclusion

Footnotes