DEV Community

Cover image for Architecting an Ultra-Minimal Linux VM with Buildroot | Part 1: Build, Break, Fix
Niklas
Niklas

Posted on • Originally published at niklas-heringer.com

Architecting an Ultra-Minimal Linux VM with Buildroot | Part 1: Build, Break, Fix

#linux #buildroot #embedded #devops

This is Part 1 of a multi-part series. We go from zero to a working VM that passes every audit check. Part 2 will cover packaging it into a bootable ISO and squeezing under the 20MB target.

My new semester just started, and one of the first challenges from my OS module was this:

"Build the smallest bootable Linux VM you can. It must only pass the tests in this bash-testing-script, nothing more. Personally, I'd love to see <20MB."

Sounds simple. It's not. Everything has gotten bigger. Software that would've been lean 15 years ago is now bloated by default. But let's try to ace it anyway.

The Audit Script: What We're Actually Building For

Before touching a config file, we need to know what success looks like. The professor gave us audit.sh. If it doesn't return all-OK, the VM fails; no matter how small it is.

Here's what the script checks:

Shell & basic commands

command -v sh >/dev/null 2>&1 && ok "POSIX-Shell"
for cmd in cp mv cat rm ls; do
  command -v "$cmd" >/dev/null 2>&1 && ok "$cmd present"
done
Enter fullscreen mode Exit fullscreen mode

Privilege & process inspection

command -v sudo >/dev/null 2>&1 && ok "sudo present"
if ps -e 2>/dev/null | head -n1 | grep -q 'PID'; then
  ok "ps works correctly"
fi
Enter fullscreen mode Exit fullscreen mode

Networking — needs a valid RFC1918 address, the ip command, and a working lo interface.

Connectivity tools — OpenSSH (both client and sshd running), arp-scan, curl with HTTPS, and nc for raw TCP.

if pgrep -x sshd >/dev/null 2>&1 || pgrep -x dropbear >/dev/null 2>&1; then
    ok "sshd/dropbear running"
fi
command -v arp-scan >/dev/null 2>&1 && ok "arp-scan present"
Enter fullscreen mode Exit fullscreen mode

Data handling — tar, awk, HTTP banner grabbing via netcat.

The challenge: packages like openssh, curl, and arp-scan aren't tiny. Fitting everything under 20MB is going to be a fight.

Full post published on niklas-heringer.com. I write about offensive security, embedded Linux, and building things from scratch. If you want, feel free to follow me there or on GitHub.

Why Buildroot?

A friend in my OS class pointed me toward Buildroot, a set of Makefiles and patches that automates building a complete Linux system from scratch. You specify exactly which packages you want, it builds a cross-compilation toolchain, the kernel, and the root filesystem.

Why not just use Alpine? You could. But Buildroot gives exact control over what ends up in the image. If audit.sh doesn't check for it, it doesn't ship.

The four layers

Layer What it is Our output
Kernel Bridge between software and hardware bzImage
Root Filesystem /bin, /etc, /usr — everything the running system uses rootfs.ext2
BusyBox Single ~1MB binary replacing hundreds of GNU tools built-in
Buildroot Automates compiling all of the above the whole pipeline

A note on the cross-compiler

Even though our build machine and target VM are both x86_64, Buildroot builds its own isolated toolchain first. This guarantees reproducible, minimal output with controlled compiler flags. It's also why the first build takes 30–60 minutes, it's literally compiling a compiler.

Buildroot
├── builds cross-compiler (host-gcc, ~20 min)
├── compiles kernel → bzImage
├── compiles packages (openssh, curl, arp-scan ...)
└── assembles rootfs → rootfs.ext2
Enter fullscreen mode Exit fullscreen mode

Setting Up the Build Environment

We use a Debian Bookworm Docker container. It gives a known-good environment that won't interfere with the host (I had a Kali VM lying around).

sudo apt install -y docker.io
systemctl enable --now docker

sudo docker run --privileged --dns 8.8.8.8 \
  -it --name minlinux debian:bookworm bash
Enter fullscreen mode Exit fullscreen mode

Two flags worth noting:

  • --privileged: needed later when we mount the rootfs image as a loop device
  • --dns 8.8.8.8: without this, apt inside the container silently fails to resolve hostnames

Everything from here runs inside the container unless stated otherwise.

The Build Script: Step by Step

Step 1: Install build dependencies 

apt-get update
apt-get install -y \
    wget make gcc g++ unzip bc \
    libncurses-dev rsync cpio xz-utils \
    bzip2 file perl patch python3 git qemu-system-x86
Enter fullscreen mode Exit fullscreen mode

libncurses-dev is for menuconfig. git is required even without cloning anything, some Buildroot package scripts call it internally.

Step 2: Download Buildroot 

BUILDROOT_VERSION="2026.02"
wget "https://buildroot.org/downloads/buildroot-${BUILDROOT_VERSION}.tar.xz"
tar -xf buildroot-${BUILDROOT_VERSION}.tar.xz -C /opt/buildroot --strip-components=1
Enter fullscreen mode Exit fullscreen mode

Step 3: Load the base config 

cd /opt/buildroot
make qemu_x86_64_defconfig
Enter fullscreen mode Exit fullscreen mode

qemu_x86_64_defconfig gives a working 64-bit QEMU baseline. We use it as a starting point to layer on top of.

Step 4: Apply our configuration 

cat >> .config << 'EOF'

# --- System ---
BR2_TARGET_GENERIC_HOSTNAME="minlinux"
BR2_TARGET_GENERIC_ROOT_PASSWD="aeb"
BR2_SYSTEM_DHCP="eth0"

# --- Minimize size ---
BR2_STRIP_strip=y

# --- Filesystem ---
BR2_TARGET_ROOTFS_EXT2=y
BR2_TARGET_ROOTFS_EXT2_SIZE="64M"
BR2_TARGET_ROOTFS_CPIO=y
BR2_TARGET_ROOTFS_CPIO_GZIP=y

# --- SSH ---
BR2_PACKAGE_DROPBEAR=n
BR2_PACKAGE_OPENSSH=y
BR2_PACKAGE_OPENSSH_SERVER=y
BR2_PACKAGE_OPENSSH_CLIENT=y
BR2_PACKAGE_OPENSSH_KEY_UTILS=y

# --- Network ---
BR2_PACKAGE_IPROUTE2=y
BR2_PACKAGE_IPUTILS=y
BR2_PACKAGE_ARP_SCAN=y

# --- curl + TLS ---
BR2_PACKAGE_LIBCURL=y
BR2_PACKAGE_LIBCURL_CURL=y
BR2_PACKAGE_CA_CERTIFICATES=y

# --- sudo ---
BR2_PACKAGE_SUDO=y

# --- pgrep ---
BR2_PACKAGE_PROCPS_NG=y

# --- netcat-openbsd ---
BR2_PACKAGE_NETCAT_OPENBSD=y

# --- BusyBox: sh, ls, cp, mv, cat, rm, ps, tar, awk, ping ---
BR2_PACKAGE_BUSYBOX=y

EOF

make olddefconfig
Enter fullscreen mode Exit fullscreen mode

make olddefconfig resolves the full dependency tree. Any option that requires another package gets pulled in automatically, but it can also silently override your settings. More on that in a minute.

Step 5: The actual build 

export FORCE_UNSAFE_CONFIGURE=1
make -j$(nproc) 2>&1 | tee /tmp/build.log
Enter fullscreen mode Exit fullscreen mode
  • FORCE_UNSAFE_CONFIGURE=1: bypasses the check that refuses to run as root (we're in Docker, we're always root)
  • -j$(nproc): parallel compile jobs; if you have <16GB RAM, drop to -j4 to avoid OOM kills
  • tee /tmp/build.log: saves output so you can scroll back if the build crashes at minute 45

Wait 30–60 minutes. When done:

bzImage        ~6.4M
rootfs.ext2    ~60M
rootfs.cpio.gz ~11M
Enter fullscreen mode Exit fullscreen mode

Three Gotchas That Cost Me Hours

1. The Dropbear trap

My first build used Dropbear instead of OpenSSH. It's a fraction of the size and does SSH perfectly well. Then the audit script hit me:

if ! pgrep -x sshd >/dev/null; then
    warn "sshd inactive"
fi
Enter fullscreen mode Exit fullscreen mode

pgrep -x matches exact process names. Dropbear runs as dropbear in the process table, not sshd. No symlink or alias changes what the kernel reports in /proc. The audit would fail every time.

Lesson: Read the audit script before choosing packages. The system follows the test, never the other way around.

2. HTTPS ≠ HTTP + TLS library

curl was compiled, TLS support was there, but HTTPS failed:

curl -sL https://www.google.com/ >/dev/null 2>&1  # FAIL
Enter fullscreen mode Exit fullscreen mode

Root cause: no CA certificates. Without BR2_PACKAGE_CA_CERTIFICATES=y, curl has no way to verify the certificate chain for any HTTPS connection. One line in the config, hours of confusion.

3. No single netcat does everything

The audit uses nc in two incompatible ways:

# Raw TCP: uses -q (quit after EOF delay)
nc -l -p 12345 -q 1 >/dev/null 2>&1 &

# Port scan: uses -z (zero-I/O mode)
nc -z -w1 "$CLIENT_IP" "$PORT"
Enter fullscreen mode Exit fullscreen mode

BusyBox nc has -q but not -z. netcat-openbsd has -z but not -q. No single implementation satisfies both. Solution: install netcat-openbsd. It handles -z correctly and silently ignores -q rather than crashing, and the raw TCP test still passes because the pipe closes stdin naturally anyway.

First Boot: 4 Failures

After 45 minutes of building, boot the VM, run the audit:

[OK]  POSIX-Shell
[OK]  cp, mv, cat, rm, ls
[OK]  ps
[OK]  lo interface
[OK]  ssh client
[NOK] sshd couldn't be started     ← pgrep missing
[OK]  SSH key setup
[OK]  Ping / DNS
[OK]  arp-scan
[NOK] nc missing                   ← never built
[OK]  curl / HTTP
[NOK] HTTPS failed                 ← no CA certs
[NOK] HTTP banner not found        ← nc cascade
Enter fullscreen mode Exit fullscreen mode

make olddefconfig had silently dropped BR2_PACKAGE_PROCPS_NG and BR2_PACKAGE_NETCAT_OPENBSD. The defconfig baseline had its own opinions, and dependency resolution overwrote ours without a warning.

The Fix: No Full Rebuild

The toolchain, kernel, and everything that built correctly is already cached. We only need the missing packages:

cd /opt/buildroot
export FORCE_UNSAFE_CONFIGURE=1

make procps-ng-rebuild
make netcat-openbsd-rebuild
Enter fullscreen mode Exit fullscreen mode

~30 seconds each. Verify the binaries landed:

find output/target -name "pgrep"  # → output/target/bin/pgrep ✓
find output/target -name "nc"     # → output/target/usr/bin/nc ✓
Enter fullscreen mode Exit fullscreen mode

Then rebuild just the filesystem:

make rootfs-ext2
Enter fullscreen mode Exit fullscreen mode

30 seconds. Boot. Run the audit.

Second Boot: All Green 

✓ POSIX Shell
✓ ls, cp, mv, cat, rm
✓ ps
✓ lo interface
✓ ssh / key setup
✓ Ping 8.8.8.8 / DNS
Network: 10.0.2.15/24
-- Live Hosts (ARP) --
10.0.2.2  52:55:0a:00:02:02
10.0.2.3  52:55:0a:00:02:03
✓ All tests complete
Enter fullscreen mode Exit fullscreen mode

The only sections not passing are the SSH Client Analysis and OS Fingerprinting; those require connecting via SSH from an external machine. When you SSH in through port-forwarded 2222, the script detects your client IP and scans you back. That's working as designed. The OS Fingerprinting section is a TODO the professor intentionally left for each team.

Key Takeaways

  1. Read the audit script before choosing packages. 15 minutes upfront saves hours of debugging.
  2. make olddefconfig can silently drop your config options. Always verify what ended up in the image, not what you put in the config.
  3. HTTPS requires CA certificates, not just TLS support. BR2_PACKAGE_CA_CERTIFICATES=y is not optional.
  4. No single netcat handles all flag combinations. netcat-openbsd is the pragmatic choice here.
  5. Incremental rebuilds are fast. make <package>-rebuild + make rootfs-ext2 takes seconds. No need to start from scratch.

What's Next

In Part 2 I'll tackle:

  • Packaging into a bootable ISO
  • Squeezing under 20MB (rootfs.cpio.gz is 11MB + bzImage at 6.4MB = ~17.4MB before ISO overhead.. it's going to be tight)
  • Implementing the OS Fingerprinting TODO

Top comments (0)