Building & Debugging a Custom ARM64 Linux Kernel — Yocto, QEMU, GDB

You can rebuild an ARM64 Linux kernel, boot it, and single-step through start_kernel in GDB without owning a single piece of ARM hardware. It runs on your laptop. The recipe is the easy part and it's well-trodden; what actually eats an afternoon the first time is a quieter problem — GDB attaches, your breakpoint hits, and then it tells you it can't find the source file. This walks the whole loop and spends its time on that part.

The loop, four steps

The workflow in the video is four steps, and each maps to one tool:

1. Change the kernel config and capture it as a config fragment (not a hand-edited .config you'll lose on the next build).
2. Rebuild the kernel and root filesystem with bitbake.
3. Boot the image under QEMU, with the GDB stub enabled.
4. Attach GDB to the stub and debug the running kernel.

The reason to do it this way — Yocto for the build rather than a raw make — is reproducibility: the config fragment and the recipe are the source of truth, so the debug kernel you built today is the debug kernel you get next month.

Step 1 — a debug config, as a fragment

Open the kernel config through Yocto rather than poking the tree directly:

bitbake -c menuconfig virtual/kernel

The settings that matter for debugging aren't about features, they're about keeping the information GDB needs:

CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=n     # reduced info drops what you need for inline unwinding
CONFIG_GDB_SCRIPTS=y            # brings in the vmlinux-gdb.py helpers
CONFIG_RANDOMIZE_BASE=n         # turn KASLR off so symbol addresses are stable

That last one is the difference between a productive session and confusion: with KASLR on, the kernel's runtime addresses are randomized and won't line up with the symbols GDB reads from vmlinux. Turn it off for debugging (or pass nokaslr on the kernel command line). Save these as a fragment and wire it into the kernel recipe (SRC_URI += "file://debug.cfg") so it survives rebuilds instead of living in your shell history.

Step 2 & 3 — build, then boot with the stub open

bitbake core-image-minimal
runqemu qemuarm64 nographic qemuparams="-s -S"

The two QEMU flags are the whole trick: -s opens the GDB stub on TCP :1234, and -S freezes the machine at reset so nothing runs until you attach. Without -S the kernel is already past early boot before GDB connects, and start_kernel breakpoints never fire.

Step 4 — attach, and the part that actually trips people

Point the cross-GDB at the vmlinux with symbols (the one from the build tree, not the stripped image that boots) and connect:

aarch64-linux-gnu-gdb vmlinux
(gdb) target remote :1234
(gdb) break start_kernel
(gdb) continue

The breakpoint hits — and GDB prints something like "No such file or directory" for the source line. This is the moment the tutorials skip, and it's exactly where the video slows down. The symbols are fine; the addresses are fine. What's wrong is that the debug info records the build-time source path — some long /usr/src/kernel/... or Yocto tmp/work/... path from the build host — and that path doesn't exist where you're now running GDB. GDB is looking in the right conceptual place and the wrong literal one.

The fix is to tell GDB how to translate the build path to your actual source tree:

(gdb) set substitute-path /usr/src/kernel /home/you/yocto/.../linux-source
(gdb) list start_kernel

Now the source resolves and list, step, and inline frames all work. Because you'll do this every session, put the connect-and-substitute sequence in a .gdbinit (or a -x script) so a single gdb -x debug.gdb vmlinux gets you to a live, source-resolved breakpoint every time. That small bit of automation is what turns "I got it working once" into a debugging loop you'll actually use.

Why bother, if you ship Go and not kernels

Most backend and infra engineers never look below the syscall boundary. They see goroutine yields, container CPU throttling, and latency spikes they can't explain, reach for pprof, and when that runs out, blame "the cluster." But a lot of what shows up as p99 lives in the kernel scheduler: which core your thread runs on, how often it migrates, whether the kernel preempted you at a CFS slice boundary or you yielded. You can't reason confidently about any of that from user space alone.

Being able to break on schedule() in a running kernel and watch it decide is what lets you make a claim about where time goes instead of guessing. For AI infrastructure the same logic is sharper: every inference call is a stack of userspace→kernel→userspace round trips — file I/O, network, GPU driver entry — and the latency variance you're tempted to pin on the model is often kernel-side scheduling and syscall cost. A kernel you can stop and inspect is the instrument that settles those arguments.

You won't build a Yocto image at work. But having done it once — and knowing why GDB couldn't find the source, and how to make it — is the difference between treating the kernel as a black box and treating it as something you can open.

Related

• Video: Build & Debug a Custom ARM64 Linux Kernel with Yocto, QEMU, GDB
• Sibling walkthrough (x86): Debugging the Ubuntu 6.8 x86-64 kernel with GDB + QEMU
• GitHub: CoreTracer — separate low-level experiments (scheduling, cache, lock-free) in the same "open the box" spirit, not the Yocto build above.

Comments

[Raknaos]

"GDB attaches, the breakpoint hits, and then it can't find the source" is the exact wall I hit too, and most write-ups walk straight past it. The debug info recorded the build-time path — how do you resolve it in practice: a set directories / safe-localsources step, or do the Yocto recipes bake it so the paths line up from the start?

CONFIG_RANDOMIZE_BASE = n is the other big one people miss. nokaslr on the command line feels less invasive than rebuilding, but with a config fragment the rebuild is cheap and reproducible, so I'd land where you did. The -s -S pair being "the whole trick" is underrated — without -S the breakpoint at start_kernel is a race you usually lose.

[Harrison Guo]

I used set substitute-path to map the build-time prefix in the debug info to the local source tree:

(gdb) set substitute-path /usr/src/kernel /home/you/yocto/.../linux-source

I didn't make the recipe reproduce the build host's paths. That works until you debug an artifact built somewhere else; the substitution in .gdbinit travels better. I also prefer the config fragment over a forgotten nokaslr boot argument. And yes, -S matters—without it, breaking at start_kernel is a race.