DEV Community: Raghu Bharadwaj

Kernel & Embedded News: Linux 7.2 Final RC; FFmpeg 9.0 Released

Raghu Bharadwaj — Tue, 11 Aug 2026 06:05:03 +0000

Linux 7.2 has reached its final planned release candidate, and Linus Torvalds expects the stable release on 16 August; he now describes the volume of late fixes, many produced by AI review tools, as the new normal. FFmpeg 9.0 shipped on 4 August with an ABI break across all seven libraries and TLS certificate verification enabled by default. In the kernel, a missed check left two legacy PCI sysfs files writable under lockdown since 2019, and the removal of Qualcomm's QCE crypto driver has been proposed after benchmarks showed it far slower than the CPU. In embedded hardware, Espressif's ESP32-S31 entered mass production and Radxa announced two Qualcomm-based system-on-modules.

Two themes connect this edition: a release process adjusting to the fix volume that AI review tools generate, and the value of measuring what you assume — a lockdown mode that did not lock everything, and a crypto accelerator measured slower than the CPU it serves. Alongside them sit a major userspace release and two hardware announcements at different layers of an embedded product.

In this edition

Linux 7.2-rc7 is the final planned release candidate. Torvalds released 7.2-rc7 on 9 August and expects the stable kernel on 16 August, calling the AI-driven fix volume "the new normal".
FFmpeg 9.0 "Lei" is released. The 4 August release bumps the major version of all seven libraries, rewrites swscale, decodes animated WebP, and turns on TLS certificate verification by default.
Kernel lockdown left legacy PCI files open since 2019. A queued PCI fix adds the missing lockdown checks to the legacy_io and legacy_mem sysfs handlers.
Removal of Qualcomm's QCE crypto driver is proposed. Eric Biggers posted a patch to delete the driver after measuring it far slower than the ARMv8 Crypto Extensions; the discussion is ongoing.
ESP32-S31 enters mass production. Espressif's dual-core RISC-V SoC combines Wi-Fi 6, Bluetooth 5.4 with Classic and LE Audio, 802.15.4 and a Gigabit Ethernet MAC in one chip.
Radxa reveals two Qualcomm-based system-on-modules. The rCore-Q8280 and rCore-Q6490 pair a Snapdragon 8cx Gen 3 and a Dragonwing QCS6490 with very different product-availability commitments.

Linux 7.2 closes with a large final RC, and Torvalds calls it the new normal

Linus Torvalds released Linux 7.2-rc7 on Sunday, 9 August. By the seventh release candidate a development cycle has usually gone quiet. This one has not, and Torvalds was direct about the reason:

"I can't say that I'm exactly thrilled about the size of this all, but it is what it is: the new normal with a lot of fixes, many of them due to review by various AI tools."

He added that nothing in the RC looks particularly worrying — it is small fixes spread across drivers, filesystems, networking and architecture code, with s390 crypto, Btrfs and netfilter ipset standing out in the diffstat. He sees no value in delaying the release, so 7.2 stable is expected on 16 August unless a serious problem appears and forces an rc8.

The memory-management side closed a race condition that could produce a use-after-free and had been in the kernel for around eight years. Btrfs brought back its fixup worker infrastructure to deal with a silent data-loss case. The hardware-monitoring subsystem took a batch of fixes its maintainer described as mostly critical or high severity, many of them for defects the hwmon maintainer credits to the Sashiko AI review bot.

The practical shift is not the release date; it is that AI review has permanently raised the fix density of every late RC and every stable point release that follows. Downstream kernels will carry more patches per update, and the burden of deciding what matters moves further onto the teams consuming them.

What it means for teams planning on 7.2

If your next BSP or distribution baseline targets 7.2 — Ubuntu 26.10 is expected to ship it as the default kernel — put rc7 on your target hardware this week rather than waiting for the tag. The delta from rc7 to final is normally small, so problems found now can still be reported as release blockers. Teams using Btrfs on embedded storage should read the fixup-worker change specifically, since silent data loss is the failure mode that never shows up in a quick test. Expect the 7.2.y stable series to move fast in its first weeks, and budget regression-test time for point releases accordingly.

FFmpeg 9.0 is released: an ABI break and new defaults that affect devices

FFmpeg 9.0 "Lei" was released on 4 August, about four and a half months after 8.1, carrying more than 2,200 commits from over 160 authors. It bumps the major version of all seven libraries — libavutil 61, libavcodec 63, libavformat 63, libavdevice 63, libavfilter 12, libswscale 10, libswresample 7 — which means an ABI break across the board. Jean-Baptiste Kempf's release write-up is the best single overview.

The largest engineering effort is the multi-year rewrite of swscale: conversions are now decomposed into operation lists compiled by C, x86 SIMD, AArch64 NEON or Vulkan SPIR-V backends, though the new paths stay behind SWS_UNSTABLE for now. Elsewhere, FFmpeg finally decodes animated WebP (a ticket open since 2015), and the AAC decoder handles the 960-sample frames used by DAB+ digital radio. A large body of NEON work speeds up software conversion on Arm boards, and the DNN filters gain an ONNX Runtime backend for running inference inside a filter graph.

For device teams, the removals and default changes matter more than the features. TLS peer certificates are now verified by default, so a product that pulls streams or updates from an endpoint with a self-signed or expired certificate will start failing after the upgrade. The OpenMAX encoders are gone, which removes the last supported path for old OMX-based encode pipelines. On the command line, -vsync is removed in favour of -fps_mode, and NVENC drops SDKs older than 11.1.

What to check before you upgrade

Plan the 9.0 upgrade as a porting exercise, not a version bump. Every application linking the libav* libraries needs a rebuild against seven new sonames, so coordinate the migration with your distribution or Yocto layer rather than upgrading one recipe in isolation. Search startup scripts for removed CLI flags, and test every TLS endpoint your product contacts against the new verification default — the correct response to a failure is fixing the certificate, not disabling verification. Pipelines still using OMX encoders should schedule the move to V4L2 M2M or vendor plugins now.

A lockdown gap: legacy PCI sysfs files stayed writable since 2019

Kernel lockdown mode restricts what root can do to a running kernel — no writes to /dev/mem, no direct PCI BAR access, and so on. It is commonly enabled together with UEFI Secure Boot, and many product security arguments lean on it: even if an attacker gains root, the kernel itself stays out of reach.

It turns out two interfaces were missed. The legacy_io and legacy_mem sysfs files, which give access to legacy port I/O and legacy PCI memory space, never received lockdown checks. Krzysztof Wilczyński explains it plainly in the fix now queued in the PCI subsystem's next branch:

"As a result, root can still write arbitrary I/O ports and map the legacy I/O and memory spaces while the kernel is locked down, which is the same capability the lockdown is meant to remove."

The checks were added elsewhere in 2019 by commit eb627e17727e ("PCI: Lock down BAR access when the kernel is locked down"), covering config-space writes, resource mmap and resource I/O writes — but not the legacy handlers. The new patch adds the missing security_locked_down(LOCKDOWN_PCI_ACCESS) checks, targets the 7.3 merge window, and is marked as a fix for the 2019 commit, so backports into stable series are the likely next step.

The instructive part is how the gap survived for seven years. Lockdown is not a single global switch; it is a collection of per-interface checks, and every interface that can reach kernel memory has to opt in individually. A new or overlooked interface fails open, not closed. Security audits should therefore test observed behaviour on the shipped kernel instead of trusting the feature description.

How to check your device

First confirm what lockdown state your product actually runs in — the bracketed entry is the active mode:

raghu@techveda.org:~$ cat /sys/kernel/security/lockdown
none [integrity] confidentiality

If your threat model counts on lockdown to keep root away from the running kernel, add this patch to your tracking list and pick it up when it reaches your stable series; it is a small, self-contained backport. Where the platform exposes the legacy PCI files, a direct write attempt under lockdown makes a useful acceptance test.

Qualcomm's QCE crypto driver faces removal for being slower than the CPU

In late July, kernel crypto maintainer Eric Biggers proposed removing the Qualcomm Crypto Engine (QCE) driver — around 3,500 lines under drivers/crypto/qce — from the kernel outright. The driver had already been marked broken earlier in July over its performance and bug history; the removal patch followed new measurements.

The numbers in his benchmark mail are direct: a SHA-256 workload that takes 0.10 seconds through the ARMv8 Crypto Extensions took 10.76 seconds through QCE, using over 50 times more CPU time — much of it in hardirq and softirq context. Even with recent locking changes rolled back, the engine remained roughly 48 times slower than the CPU instructions. The offload engine costs more than it saves: per-request DMA setup, interrupts and completions dominate at typical request sizes. The discussion is ongoing — at least one developer has offered to keep maintaining the driver, and Greg Kroah-Hartman has been brought into the thread — so removal is proposed, not decided.

The general lesson applies beyond Qualcomm. Crypto offload engines were useful when CPUs lacked AES and SHA instructions; on any Armv8 core with the Crypto Extensions, that assumption needs re-testing, and BSPs frequently enable vendor crypto drivers by default without anyone measuring them.

What to do on Qualcomm-based products

If your BSP builds the QCE driver, benchmark both paths on your hardware — dm-crypt via cryptsetup benchmark, or the kernel's tcrypt tests — before trusting the offload. Should the removal go through, nothing breaks functionally: the kernel crypto API falls back to the next-priority implementation, which on these SoCs is the faster CPU path. The same measurement habit is worth applying to any vendor offload block in your tree, not only crypto.

ESP32-S31 enters mass production: one chip for Wi-Fi 6, Thread and Bluetooth

Espressif announced on 27 July that the ESP32-S31 has entered mass production and is available for purchase. The chip pairs two 32-bit RISC-V cores at up to 320 MHz — one with a 128-bit data path and SIMD instructions — with a 40 MHz low-power coprocessor, 512 KB of on-chip SRAM, support for 250 MHz DDR PSRAM, and up to 60 GPIOs.

The radio set is the notable part. One chip carries 2.4 GHz Wi-Fi 6, Bluetooth 5.4 including both LE Audio and Classic BR/EDR, and IEEE 802.15.4 for Zigbee and Thread, plus a Gigabit Ethernet MAC. Matter is supported over both Wi-Fi and Thread, and HMI support covers camera and LCD interfaces, capacitive touch, and a 2D graphics accelerator.

Be clear about what this is: a microcontroller programmed with ESP-IDF, not a Linux part. Its relevance to a Linux product is as the radio side of the design. A hub or gateway that needs Wi-Fi, Thread and Bluetooth today often carries two or three radio chips with separate antennas, drivers and certification effort; consolidating them into one part reduces board area and RF integration work. Bluetooth Classic support is increasingly rare in new silicon, which matters for products that still talk to older audio and serial-profile devices.

Where it fits in a Linux design

Espressif's ESP-Hosted and ESP-AT frameworks let the chip serve as a connectivity co-processor next to a Linux SoC, with the application processor driving it over SDIO, SPI or UART. For a Matter-capable gateway, that structure keeps the protocol radios and their certified stacks on one side and your Linux application on the other. Before committing a BOM, verify module-level regulatory certification for your target markets and check long-term availability terms — Espressif publishes a longevity commitment per product line, and the difference between chip and module sourcing affects both.

Radxa's rCore SoMs bring Snapdragon and Dragonwing to the edge

Radxa has added two Qualcomm-based system-on-modules to its rCore line, aimed at edge AI, multimedia, robotics and industrial products. The rCore-Q8280 (50 × 80 mm) is built on the Snapdragon 8cx Gen 3 (SC8280XP): eight Kryo cores at up to 3.0 GHz, an Adreno 690 GPU, a Qualcomm AI Engine rated at 29+ TOPS, up to 32 GB of LPDDR4X, and nine PCIe Gen3 lanes. The rCore-Q6490 (42 × 36 mm) uses the Dragonwing QCS6490: an eight-core Kryo 670, Adreno 643, a Hexagon NPU rated at up to 12 dense TOPS, up to 16 GB of LPDDR5, and five 4-lane MIPI CSI camera interfaces supporting sensors up to 64 MP.

The TOPS numbers are vendor ratings aggregated across CPU, GPU and NPU, so compare candidates by running your own model, not by the datasheet figure. The more decision-relevant numbers are the availability commitments: Radxa states the Q8280 will remain available until at least June 2029, while the Q6490 carries a commitment through at least June 2036. For an industrial product with a ten-year service life, that difference outweighs the performance gap.

Software support also splits along product lines. The Q8280 lists desktop-style distributions — Debian, Ubuntu, Armbian and others — while the Q6490 lists Yocto alongside Ubuntu and Windows 11 IoT Enterprise, which signals where Radxa expects each module to be designed in. Pricing and retail availability have not been announced for either module.

How to evaluate them

Treat the two modules as different products rather than two variants of the same design. The Q8280 is a laptop-class SoC repurposed for compute-heavy edge boxes; the Q6490 is the one shaped for long-life, camera-heavy embedded designs. Before shortlisting either, ask which kernel the vendor images track, what the upgrade path looks like over the module's life, and how much of the SoC support is upstream versus carried in a vendor tree — on a Qualcomm part, that answer determines your maintenance cost more than any spec-sheet line. Get it in writing, and prototype your camera pipeline early if the five CSI interfaces are the reason you are looking at the Q6490.

References

If you build products on the Linux kernel and want structured, instructor-led depth on kernel internals, device drivers and BSP engineering, TECH VEDA's training programs are at techveda.live.

Originally published at techveda.live.

— Raghu Bharadwaj

Capturing a Kernel Crash Dump with kdump on ARM64

Raghu Bharadwaj — Tue, 11 Aug 2026 05:26:01 +0000

When a panic happens on a board on your desk, you read dmesg. When it happens in the field and the register dump scrolls off a console nobody was watching, you need the memory image instead. That is what a kernel crash dump is for. Our earlier tutorial on decoding a kernel oops ended by recommending panic_on_oops with kdump; this post is the part that was left out.

What you need

An ARM64 target you can reboot and whose kernel command line you control (a board with U-Boot, or QEMU with -append).
A kernel tree you can rebuild, plus the matching vmlinux with debug information.
kexec-tools on the target, and roughly 128 MB of RAM you can set aside.
Root on the target. Every command below is run as root.

QEMU -M virt is enough to learn the mechanics. Boot it with -kernel rather than UEFI firmware, because a UEFI boot leaves no /sys/firmware/fdt for kexec to reuse, and give it at least -m 2048. It will not reproduce the failures described at the end of this post, which are all properties of real boards.

What kdump does when the kernel panics

kdump is built on kexec, which boots one kernel directly from another without going back to firmware. The variant used here is kexec -p, the panic path: it preloads a second kernel, the capture kernel, into memory the first kernel reserved at boot. When panic(), die() or the SysRq crash handler fires, the kernel calls crash_kexec(), saves CPU registers as ELF notes, and jumps to that image. The address of the ELF header describing the old memory reaches the second kernel through the elfcorehdr= parameter, and that memory then appears at /proc/vmcore.

Kernel configuration for a kernel crash dump

The system kernel needs the kexec syscall; the capture kernel needs the dump-reading side. On ARM64 the same relocatable Image can serve as both, so in practice you enable all of it in one config. A separate, smaller capture-kernel build is only worth the trouble if you need it to fit a smaller reservation:

CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y

CONFIG_KEXEC (or CONFIG_KEXEC_FILE, the file-descriptor-based variant that CONFIG_KEXEC_SIG builds signature checking on) selects CONFIG_KEXEC_CORE. CONFIG_CRASH_DUMP selects CONFIG_VMCORE_INFO and CONFIG_CRASH_RESERVE, and enables CONFIG_PROC_VMCORE by default. One point causes confusion: CONFIG_DEBUG_INFO has no menu entry of its own any more — it is selected by whichever entry you pick in the "Debug information" choice under Kernel hacking, and that choice appears only when CONFIG_DEBUG_KERNEL is set. Debug information is not needed to capture a dump, only to read it. Confirm what you built:

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'CONFIG_(KEXEC|CRASH_DUMP|PROC_VMCORE)'
CONFIG_KEXEC_CORE=y
CONFIG_KEXEC=y
# CONFIG_KEXEC_FILE is not set
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y

If /proc/config.gz does not exist, the running kernel was built without CONFIG_IKCONFIG_PROC; check the build tree's .config instead. One ARM64 trap: CONFIG_KEXEC depends on ARCH_SUPPORTS_KEXEC, which on this architecture is def_bool PM_SLEEP_SMP. On a uniprocessor board, or one built without suspend support, the "Enable kexec system call" prompt does not appear at all. CONFIG_KEXEC_FILE has no such dependency and is the way out.

Reserving memory with crashkernel=

The reservation happens at boot and cannot be done later, so it goes on the kernel command line. That timing is the point: at the moment of a panic the kernel cannot be trusted to allocate anything, so the region has to be set aside while the machine is still healthy. The simplest form gives a size and lets the kernel choose the address:

crashkernel=128M

You can pin the address with crashkernel=128M@0xb0000000. On ARM64 an explicitly given start address must be aligned to 2 MiB (0x200000), and the reservation is refused if the region is already in use. A range form, crashkernel=512M-2G:64M,2G-:128M, scales the size with the RAM present, which helps when one image ships to several board variants. For a first setup, use the plain form.

Reboot with the parameter added and confirm the reservation happened. The kernel prints it, and the region appears in /proc/iomem under the name Crash kernel:

raghu@techveda.org:~$ dmesg | grep -i crashkernel
[    0.000000] crashkernel reserved: 0x00000000b0000000 - 0x00000000b8000000 (128 MB)
raghu@techveda.org:~$ grep -i 'crash kernel' /proc/iomem
  b0000000-b7ffffff : Crash kernel

If nothing is printed, the reservation failed. The kernel says so in the same log, either with crashkernel reservation failed - memory is in use. for a fixed address, or a line beginning cannot allocate crashkernel when no suitable region was found.

Physical memory, before and after the panic

Physical address	While the system kernel runs	After `panic()` calls `crash_kexec()`
`0xb8000000 - 0xbfffffff`	System RAM	Preserved. Read through /proc/vmcore.
`0xb0000000 - 0xb7ffffff`	Crash kernel, 128 MB. Reserved by `crashkernel=128M`. The system kernel never allocates here; `kexec -p` loads the capture kernel and its initramfs into it.	The capture kernel runs here — and with a plain `crashkernel=N` reservation, only here.
`0x40000000 - 0xafffffff`	System RAM — kernel, drivers, page cache, user space	Preserved. Read through /proc/vmcore.

Addresses are from a 2 GB board whose RAM starts at 0x40000000, pinned with crashkernel=128M@0xb0000000 so the output above is reproducible. The separation is the whole design: because the two kernels never share memory, the image that has to survive the crash cannot be damaged by the kernel that is crashing.

Loading the capture kernel with kexec -p

Now preload the capture kernel into that reserved region. Check first that an uncompressed Image is actually present on the target: ARM64 kexec cannot load Image.gz, zImage, uImage or a FIT image, and most embedded root filesystems keep no kernel under /boot at all, so copy arch/arm64/boot/Image from your build tree onto the board before running the command below.

raghu@techveda.org:~$ kexec -p /boot/Image --initrd=/boot/initramfs-kdump.img --append="1 nr_cpus=1 reset_devices panic=10 console=ttyAMA0,115200"

Build that initramfs before you load the capture kernel. It needs the block or network driver for wherever you intend to write, plus a minimal user space containing makedumpfile and vmcore-dmesg. On most embedded images the quickest route is to reuse the initramfs the board already boots with, after confirming those two binaries are in it. If the storage driver is built into the kernel rather than a module, you can drop --initrd and give the capture kernel root=/dev/mmcblk0p2 rootwait rw instead, so it mounts the real root filesystem and runs your normal init.

Each argument here has a specific purpose. 1 boots to single-user mode, nr_cpus=1 keeps the capture kernel to one CPU because it has very little memory, and reset_devices asks drivers to reset their hardware rather than assume the state the crashed kernel left behind. panic=10 matters more than it looks: if the capture kernel itself panics, the board reboots after ten seconds instead of hanging until someone visits it. Keep the console setting, or a failure in the capture kernel is silent. No device tree is needed on the usual path: ARM64 kexec-tools reuses the running one from /sys/firmware/fdt unless you pass --dtb.

Verify the image is loaded before you trust it:

raghu@techveda.org:~$ cat /sys/kernel/kexec_crash_loaded
1
raghu@techveda.org:~$ cat /sys/kernel/kexec_crash_size
134217728

A 0 from the first file means no capture kernel is loaded, and a panic will simply hang or reboot. In current mainline these attributes were regrouped under /sys/kernel/kexec/ as crash_loaded and crash_size, with the old names kept as compatibility symlinks, so the paths above work on old and new kernels alike.

kexec_crash_size is writable: writing a smaller value shrinks the reservation and returns the rest to the system, but only while no capture kernel is loaded. To unload, run kexec -p -u.

Triggering a panic and saving the dump

Test the path deliberately, before you need it. SysRq gives you a crash on demand. Writing 1 first enables all SysRq functions, because many distributions ship a restricted mask that does not include the crash trigger:

raghu@techveda.org:~$ echo 1 > /proc/sys/kernel/sysrq
raghu@techveda.org:~$ echo c > /proc/sysrq-trigger

The console should show the panic, then a second kernel banner. When it settles, /proc/vmcore is the crashed kernel's memory. Filter it rather than copying it whole — a full copy is the size of your RAM, which is rarely what you want on a device with eMMC:

raghu@techveda.org:~$ makedumpfile -c --message-level 1 -d 31 /proc/vmcore /var/crash/vmcore
raghu@techveda.org:~$ vmcore-dmesg /proc/vmcore > /var/crash/dmesg.txt

-d 31 is the dump level: it discards zero, cache, user and free pages and keeps kernel data, which covers most driver bugs. -c compresses each page with zlib; -l (LZO) and -z (zstd) are usually smaller and faster, but they have to be enabled when makedumpfile is built, and the stock Yocto build has only zlib. --message-level 1 keeps the progress output quiet. vmcore-dmesg, also from kexec-tools, extracts just the kernel log ring buffer — on a constrained target that one small file is often enough to identify the fault.

Reading the kernel crash dump

Reboot into the normal kernel first, then analyse offline on your workstation. The crash utility is the usual tool, and it needs the exact vmlinux that produced the dump:

raghu@techveda.org:~$ crash vmlinux /var/crash/vmcore
      KERNEL: vmlinux
    DUMPFILE: /var/crash/vmcore  [PARTIAL DUMP]
        CPUS: 4
       ... (date, uptime, release, machine and memory lines omitted)
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
crash> bt
crash> log
crash> ps

[PARTIAL DUMP] is expected: it means the file was filtered by makedumpfile -d rather than copied whole. bt gives the backtrace of the panicking task, log the kernel ring buffer, and ps the task list at the moment of the crash.

Plain gdb can inspect registers and memory too, but only on an uncompressed ELF dump — it cannot open the kdump-compressed file that makedumpfile -c produces. Expand it first with makedumpfile -R, and expect gdb to understand far less about kernel structures than crash does. Reading a dump well is a skill in itself, and it is one of the areas covered in the TECH VEDA Linux Device Drivers course.

Where kdump fails on embedded boards

The reservation never happened. The bootloader replaced your command line, or the fixed address overlapped a carveout for a display or DSP. Check dmesg, not the bootloader environment.
kexec loading is blocked. If the kernel.kexec_load_disabled sysctl has been set to 1, kexec -p fails; that sysctl cannot be set back to 0, so only a reboot clears it. A locked-down kernel refuses the legacy kexec_load() path as well, leaving kexec_file_load() with a signed image as the sanctioned route.
The capture kernel boots but cannot write anywhere. Its initramfs must contain the storage or network driver and a userspace able to run makedumpfile. This is the most common reason a setup that appears to work produces no file.
The reservation is too small. If the capture kernel dies during boot with allocation failures, increase it rather than guessing.
The reservation cannot be placed in low memory. A plain crashkernel=128M is placed below 4 GB, which fails on boards whose low memory is already full of reserved-memory carveouts. Use crashkernel=128M,high crashkernel=64M,low so the bulk goes above 4 GB and only the DMA-capable part stays low.
Not every oops produces a dump. An oops reaches crash_kexec() only when kexec_should_crash() agrees: in interrupt context, in the idle task or init, or when panic_on_oops is set. A plain oops in ordinary process context kills the task and leaves no dump. Set panic_on_oops=1, and optionally panic_on_warn=1, to cover those.
The watchdog reboots the board mid-dump. The SoC or PMIC watchdog keeps counting while the capture kernel boots and makedumpfile runs, and nothing in that path services it. Either disable it on the capture kernel's command line, or accept vmcore-dmesg only, which finishes in seconds.
Devices left running by the crashed kernel disturb the capture kernel. In-flight DMA and IOMMU contexts survive the panic, so you can see SMMU stream-ID faults or a hang as soon as the capture kernel touches that controller. Keep reset_devices even when it looks unnecessary, and blacklist the offending driver on the capture kernel's command line.

Building kdump into a Yocto image

OpenEmbedded-Core carries the recipes, so nothing has to be written by hand. At the time of writing it packages kexec-tools 2.0.32, splits it into kexec, kdump and vmcore-dmesg, and runtime-depends on makedumpfile, which is also in OE-core. That dependency hangs off the kexec-tools meta-package rather than off kdump, so list everything you want explicitly:

IMAGE_INSTALL:append = " kexec kdump vmcore-dmesg makedumpfile"

The kdump package installs /etc/sysconfig/kdump.conf and either a sysvinit script or a kdump.service unit, depending on your distro features. Set KDUMP_KIMAGE to the capture kernel and KDUMP_CMDLINE to the arguments above, and add crashkernel=128M to the machine's command line.

Three things about that script are worth knowing before you rely on it. It never passes an initrd — there is no KDUMP_INITRD variable — so the capture kernel has to reach your real root filesystem on its own. Its save path runs when /proc/vmcore is non-empty, which is only true inside the capture kernel, so kdump and makedumpfile must be present in the capture environment and not merely in the normal image. And the shipped kdump.conf sets MAKEDUMPFILE_ARGS to --dump-dmesg, so out of the box you get a kernel-log extract rather than the filtered vmcore this post produced. Change that variable if you want the full dump.

Key takeaways

A kernel crash dump needs three things in place before the crash: a crashkernel= reservation, a capture kernel loaded with kexec -p, and somewhere for it to write.
Verify the reservation in dmesg and /proc/iomem, and the load in /sys/kernel/kexec_crash_loaded. Do not assume either worked.
Test with echo c > /proc/sysrq-trigger while you can still watch the console.
Use makedumpfile -d 31 to filter, and vmcore-dmesg when you only need the log. The result is kdump-compressed, so read it with crash, not gdb.
On ARM64 kexec loads only an uncompressed Image, an explicit crashkernel base must be 2 MiB aligned, and the capture kernel's initramfs must carry the driver it needs to save the file.

Frequently asked questions

Why does kdump need memory reserved at boot instead of allocating it when the crash happens?

At the moment of a panic the kernel cannot be trusted to allocate anything. The reserved region is set aside at boot and never used by the running kernel, so the capture kernel sitting there is intact when it is needed.

Do I need to build a separate capture kernel?

Not on ARM64. The architecture supports relocatable kernels, so the same Image can act as both. A separate, smaller build is only worth doing if you want to fit a smaller reservation.

How much memory should I give to crashkernel on an embedded board?

128 MB is a reasonable starting point. If the capture kernel fails to boot with allocation errors, raise it. You can shrink an existing reservation at runtime by writing a smaller value to /sys/kernel/kexec_crash_size, but only while no capture kernel is loaded.

Why did my board reboot on panic without producing a dump?

Either no capture kernel was loaded, which /sys/kernel/kexec_crash_loaded will tell you, or it booted but its initramfs had no driver for the storage you asked it to write to. A hardware watchdog can also reboot the board before the dump finishes. Keep a console configured in the capture kernel's command line so you can see which happened.

SoC Kernel Support: How to Tell If Yours Has Already Lapsed

Raghu Bharadwaj — Mon, 10 Aug 2026 17:35:02 +0000

SoC kernel support is not one date, it is three independent clocks: the upstream LTS end-of-life date, the last time your vendor's BSP branch actually moved, and whether the silicon carries a public availability commitment. All three are measurable in a few minutes, and most teams never measure any of them. We maintain a public BSP tracker covering seventeen SoCs and re-verify every field weekly against the real git refs and kernel.org; the most recent refresh was 9 August 2026. Running the measurement across that dataset: two vendor branches have not received a commit in more than 300 days, one shipping platform runs a kernel series that is not an LTS at all, and for only four is a specific silicon longevity date available in the public domain at all.

Most embedded teams discover that their SoC kernel support has lapsed at the worst possible moment: when a CVE lands, a customer asks for a patched image, or a certification body asks how long the product will receive updates. The uncomfortable part is that the lapse is almost always visible months or years earlier, in public data, to anyone who thinks to look.

This article gives the method. It is deliberately reproducible: every check below is something you can run yourself against public sources in an afternoon, and every number we quote is dated so you can re-run it and see whether it has moved.

We also publish the result, because the aggregate is more instructive than any single board. TECH VEDA maintains a public BSP tracker covering seventeen SoCs, and our engineering team rechecks every field once a week against the vendors' real git refs and against kernel.org — the last refresh ran on 9 August 2026. The branch ages quoted below were re-checked directly against the GitHub API on 10 August 2026, so they can be reproduced rather than taken on trust. The living data is at kernel lifecycles and on the per-SoC pages linked throughout.

The three clocks behind SoC kernel support

"Supported" is a word that hides three different guarantees, made by different parties, that expire at different times.

The upstream LTS clock. The kernel.org stable team publishes an end-of-life date for each longterm series. After it, no more stable fixes are produced by anyone, for anybody.
The vendor branch clock. Your silicon vendor maintains a BSP branch. It has a head commit, and that commit has a date. If that date stops advancing, the vendor has stopped merging fixes into the tree you are consuming, whatever the roadmap says.
The silicon longevity clock. Separately from software, the vendor may commit to keeping the part orderable until a stated year. This is the one most often assumed and least often verified.

Clock	What it answers	Evidence to collect	Risk when it expires
Upstream kernel	How long does upstream stable maintenance exist?	Kernel series, kernel.org longterm list and EOL date	Stable-fix and CVE cliff
Vendor BSP	Is the exact tree you ship still visibly maintained?	Branch head committer date, release cadence, advisories	Patch flow stops reaching your build
Silicon longevity	How long can the part still be sourced?	Published vendor lifecycle commitment	Redesign and supply-chain exposure

These are independent. A part can have a 2045 availability commitment and a BSP branch that has not moved in eight months. A vendor can be committing daily to a branch built on a kernel that reaches end-of-life this year. Your actual exposure is set by whichever clock expires first, not by the one the datasheet advertises.

Clock one: the upstream end-of-life date

Start here, because it is the cheapest to check and it caps everything else. Find your kernel series and read its projected end-of-life from kernel.org:

raghu@techveda.org:~$ uname -r
6.1.182-mydevice
raghu@techveda.org:~$ curl -s https://www.kernel.org/releases.json | \
    jq -r '.releases[] | select(.moniker=="longterm") | "\(.version)  EOL \(.eol // "n/a")"'

Verified against kernel.org on 10 August 2026, the six longterm series stand as follows. Two of them expire this year. We keep this table current at kernel lifecycles:

Series	Current	Projected EOL	Time remaining
6.18	6.18.44	31 Dec 2028	about 29 months
6.12	6.12.103	31 Dec 2028	about 29 months
6.6	6.6.151	31 Dec 2027	about 17 months
6.1	6.1.182	31 Dec 2027	about 17 months
5.15	5.15.215	31 Dec 2026	about 5 months
5.10	5.10.264	31 Dec 2026	about 5 months

There is a trap in this step that affects more products than the EOL dates do. Not every kernel a vendor ships is a longterm series at all. If your kernel series does not appear on that list, it has no upstream longterm stable-maintenance window to plan against. That is not the same as having no maintenance: continued fixes may still arrive through the vendor's own release stream, a distribution, or another explicitly documented maintenance path. What it does mean is that you cannot rely on upstream, and the burden shifts to establishing what that documented path actually is. We found exactly this case in our own data, described below.

Key insight: Check that your kernel series is on the longterm list before you check its end-of-life date. A series that is absent is not early in its upstream support window — it is outside one, and whatever maintenance it receives has to be established from the vendor rather than assumed.

Clock two: is the vendor branch still moving

This is the measurement almost nobody runs, and it is two commands. Resolve the branch head, then ask when that commit was made:

raghu@techveda.org:~$ git ls-remote https://github.com/rockchip-linux/kernel develop-6.6
1ba51b059f25533c5529b7f68186190b47d6a7b3    refs/heads/develop-6.6
raghu@techveda.org:~$ curl -s https://api.github.com/repos/rockchip-linux/kernel/branches/develop-6.6 | \
    jq -r '.commit.commit.committer.date'
2025-09-01T11:22:35Z

That is the whole check. Run it against the branch you actually build from, not the branch named in the marketing material. Record the date and re-run it quarterly; the trend matters more than any single reading.

Two cautions on method. Use the committer date rather than the author date, because a patch written months earlier can be merged today and the committer date is what tells you the tree is alive. And on repositories that are not on GitHub, the equivalent is a shallow clone: git clone --depth 1 --branch <branch> <url> followed by git log -1 --format=%cI.

Treat the resulting number as an investigation trigger rather than a support verdict. A quiet branch may still be supported through periodic SDK drops or a declared rebase cadence. What a long gap does establish is that you owe yourself a documented answer. As rough review triggers — our own, not an industry standard — we treat under 90 days as unremarkable, 90 to 180 days as worth raising at the next vendor call, and beyond a year with no published cadence as something to get in writing before design lock-in.

Clock three: the silicon longevity commitment

The third clock is not about software at all. It is the vendor's public commitment to keep the part orderable until a stated year, and it matters because it bounds how long the software question is worth asking. This is the clock most often assumed from a sales conversation and least often traced to a published document.

When we tried to pin a specific date to each of the seventeen SoCs in our tracker, requiring a public source rather than a sales claim, we could verify one for four of them. AMD states 2045 for Zynq UltraScale+, Renesas 2038 for RZ/G2L, Qualcomm July 2036 for QCS6490, and NVIDIA January 2032 for Jetson Orin.

For the other thirteen, no end date for that part is available in the public domain. That is a finding about what a buyer can obtain without a sales conversation, and it is worth being precise about how it was reached: for each part we checked the vendor's public product page, its lifecycle or longevity documentation, and any published product-change or availability notice. Where none of those carried a date for that specific part, the cell reads "no public date".

It does not mean the vendor offers nothing. Several of them operate published longevity programmes: NXP runs a Product Longevity Program offering a minimum of 10 or 15 years from launch, and ST publishes a commitment spanning 7 to 20 years, with a 10-year programme covering parts in the STM32MP13 series. The gap is between the programme and the part. Enrolment is generally documented per part number rather than per SoC family, and for these thirteen the per-part answer is not published where a buyer can find it.

The practical consequence is the same either way: a commitment you cannot obtain from public documentation is one you cannot cite in a design review or pass to a customer, so ask your vendor to confirm enrolment and the end date for your exact part number in writing. A common substitute is a longevity claim from a module or system-on-module vendor; that covers the module, depends on the module vendor's own supply of the silicon, and is not interchangeable with a commitment from the chip vendor.

What seventeen SoCs actually look like

We applied all three checks across all seventeen SoCs on 10 August 2026. The full table is not reproduced here, because it would be out of date within the week and we maintain a live version instead: SBC support status for development boards, sorted by how much cover is left, and the SoC mainline support matrix for the per-IP-block picture. The shape of the result on the day we measured:

Branch head ages spanned 6 days to 370 days. Most sat under 80 days. Two were past 300, and one SoC has no public git to measure at all.
Kernel bases spanned 5.15 to 6.18, with one shipping platform on a series that is not on the longterm list.
Four of the seventeen carried a silicon longevity date we could find in public documentation. For the rest the cell reads "no public date".

Three results are worth drawing out.

The Allwinner T507 line ships on 5.15, which reaches end-of-life in about five months. There is also no public git server with release branches to measure against; the SDK is distributed as a repo manifest and archive downloads. Both facts are properties of how that vendor distributes software, and both mean an integrator cannot answer "when does my support end" from public information.

Jetson Orin's current JetPack ships kernel 6.8, and 6.8 is not a longterm series. It does not appear on the kernel.org longterm list, so there is no upstream stable stream behind it and no projected end-of-life to plan against. Support for that platform flows entirely through NVIDIA's own releases, which is a coherent model, but it means the upstream clock gives you nothing and the vendor clock is the only one that exists.

For Rockchip's RK3588, the newest branch is staler than the default one. develop-6.6 last moved 343 days ago; develop-6.1, which is the repository default, last moved 227 days ago. An integrator who reasonably picks the highest kernel version on offer lands on the less maintained tree. Note also that no 6.12 or 6.18 branch exists in that repository at all, so the newest kernel available there is already two LTS generations behind current.

Key insight: The newest branch in a vendor repository is not necessarily the maintained one. Check the head date of every candidate branch before choosing, because the highest version number can be the least supported option on offer.

What a stale branch does not prove

This is the most important section in the article, because the measurement above is easy to over-read. A branch head date is a fact. What it means requires care, and there are at least five reasons a quiet branch is not evidence of an abandoned product.

Public git is not the only channel. Several vendors deliver production BSPs under NDA, through a support portal, or as periodic SDK releases rather than as continuous commits. A quiet public mirror can sit alongside an actively maintained internal tree that you receive as a customer. What the public branch measures is what a prospective integrator can see before signing anything, which is a real and different question.
A board vendor's fork is not the silicon vendor's tree. The 370-day figure above belongs to Khadas, not Amlogic. Amlogic publishes no public vendor tree for this part at all, so the nearest observable proxy is a board vendor's fork. That is a fair thing to report about what an integrator can inspect, and an unfair thing to characterise as Amlogic neglecting the part.
Older branches are supposed to go quiet. NXP's lf-6.6.y and lf-6.12.y have taken no commits since November 2025 while lf-6.18.y is current. That is ordinary branch retirement after a migration, not decay. The question is only whether the branch you build from is the current one.
Some industries deliberately pin. Automotive teams working with J721e typically track the branch matching a released Processor SDK rather than the newest branch, because the qualified configuration matters more than the freshest one. A pinned branch under a support contract is a different situation from a pinned branch by accident.
Cadence differs legitimately. ST rebases the STM32MP branch roughly every eighteen to twenty-four months by design. Measured on any given day that looks slow next to a vendor that merges weekly, but it is a stated cadence rather than a lapse.

The honest formulation is narrow: a stale public branch tells you that you cannot verify from public sources that fixes are still flowing into the tree you build from. That is a procurement and due-diligence problem. Turning it into a claim about a vendor's intentions goes well beyond what the data supports.

Key insight: Measure what you can see, and say only what the measurement supports. "I cannot verify that this tree is receiving fixes" is defensible and actionable. "This vendor has abandoned the part" usually is not.

Why these dates matter

None of this is bookkeeping. Each SoC kernel support clock has a consequence attached, and teams usually meet them in the same order.

A vulnerability lands and there is no fix to take. While a series is in its upstream window, a CVE in a subsystem you ship arrives as a stable patch you can pull. After end-of-life, nobody produces that patch. Your options narrow to backporting it yourself from mainline, paying someone to do it, or shipping a known-vulnerable image. The first is expensive and needs a person who can read the subsystem; the third is increasingly not a choice.

Someone asks a question you cannot answer quickly. This is the one that has changed most, and it is no longer a single regulation. Four triggers now reach the same set of facts from different directions:

EU Cyber Resilience Act. Reporting obligations for manufacturers begin on 11 September 2026: an actively exploited vulnerability in a product with digital elements must be reported within 24 hours of the manufacturer becoming aware of it. To report inside a day you must first establish whether the vulnerability is in your image, which is a question about which kernel you ship and which patches reached it.
UK PSTI. The product security regulations require a manufacturer to publish a defined support period — in the wording of the regulations, "the minimum length of time, expressed as a period of time with an end date, for which security updates will be provided". Once published it may be extended but not shortened. You cannot responsibly publish that date without knowing the three clocks behind it, because the commitment outlives your ability to revise it.
EU Radio Equipment Directive. The cybersecurity provisions of the delegated act became mandatory on 1 August 2025 for radio equipment placed on the EU market, which covers most connected embedded products with a wireless interface.
Sector frameworks. Automotive programmes work to UN Regulation No. 156 on software update management systems; industrial and medical programmes reach the same questions through their own standards and regulator guidance. A customer negotiating a support term asks it in plainer language and expects a date.

The common thread is that all four ask you to state, in advance and in writing, how long a product will keep receiving security updates. That is not a policy question. It is the minimum of your three SoC kernel support clocks.

A routine uplift turns into a migration. Moving from one LTS to the next while you still have a choice is a planned engineering task. Doing it because your series died, or because a vendor branch stopped moving, means the same work with no schedule, and on a shipped product it drags re-validation and often re-certification behind it. The engineering content is similar; the cost is not, because unplanned work displaces whatever else was in the quarter.

The part stops being orderable. The longevity clock is the one that cannot be solved with software. If the silicon goes end-of-life inside your product's support window, the answer is a board respin and a new bring-up, and the lead time on that is measured in quarters.

The reason to measure early is that every one of these is cheaper as a decision than as an emergency. A stale branch found during SoC selection changes a line in a comparison table. The same fact found when a CVE is already public changes a release plan.

Key insight: Support lapse is rarely discovered by the team that caused it. It surfaces when an outside party — an attacker, a regulator, an auditor or a customer — asks a question with a deadline attached. Measuring the three clocks is how you find out on your own schedule instead.

What to do before your next SoC selection

Measuring SoC kernel support is only useful if it happens while you still have a choice of part. Five steps, in the order that saves the most time:

Check the kernel series is on the longterm list at all, then read its end-of-life date. If the series is absent, all support flows through the vendor and you should price that accordingly.
Measure the head date of every candidate branch, not just the one you were pointed at. This is the step that surfaces the newest-is-not-maintained case.
Ask for the longevity commitment in writing, and check whether it covers the silicon or a module. If you cannot find a public statement, that is a question for procurement rather than an assumption to carry.
Compare the vendor tree against mainline for your blocks. If the peripherals you depend on are upstream, a stale vendor branch is an inconvenience. If your camera, video encode or accelerator exists only in the vendor fork, that branch is your only source of fixes and its head date is your security posture.
Re-run all of it quarterly, and record the result where the product plan lives. A single reading tells you little; four readings tell you whether a tree is slowing down. Store the evidence, the date of the next review, and the condition that would force a migration — a kernel EOL, a branch crossing your staleness threshold, or a longevity date entering your support window. A number in someone's notebook is not a lifecycle plan.

Steps four and five are where this connects to the wider shift we wrote about in upstream-first BSPs and in architecting mainline-friendly products. The reason upstream-first lowers risk is precisely that it moves your dependency from a branch whose head date you cannot control to a tree with a published support window.

Where to find the current numbers

Every figure in this article goes stale, which is the point. Rather than ask you to trust a snapshot, we keep the underlying data live and re-verify it weekly:

SBC support status — the three clocks applied to 29 development boards across these SoCs, sorted by how much cover is left, with a "fixes stop" date on each row. This is the fastest way to find your own board.
SoC mainline support matrix — all seventeen SoCs against each IP block, showing which peripherals are upstream and which still need the vendor tree. This is the check in step four above.
Kernel lifecycles — current patch level, release date and projected end-of-life for every longterm series, checked against kernel.org. This is clock one.
Per-SoC pages carrying the BSP branch, its head date and the longevity position, including RK3588, QCS6490, Jetson Orin and T507.
Yocto layer compatibility — whether your vendor BSP layer actually has a branch for the Yocto release you are targeting, read from each layer's own git history.

A field with no primary source stays empty rather than being filled in by inference, which is why some cells read "no public date" rather than carrying a number. The wording is deliberate. It records that the figure is not obtainable from public documentation, which is the thing that affects you, and it makes no claim about what a vendor will commit to privately.

If you are planning a long-lived embedded Linux product, this assessment is worth running across the whole product line rather than one board: which of your peripherals exist upstream, what a migration off a stale branch would actually cost, and where your real support cliff falls. Doing it before a CVE, a certification audit or a supply-chain event forces the question is the difference between a scheduled uplift and an unplanned migration project. That is the work our BSP and kernel engineering team does.

Key takeaways

SoC kernel support runs on three independent clocks: upstream LTS end-of-life, the vendor branch head date, and the silicon longevity commitment. Your exposure is set by whichever expires first.
Check that your kernel series appears on the kernel.org longterm list before checking its EOL date. Jetson Orin's current JetPack ships 6.8, which is not a longterm series.
Measuring a vendor branch takes two commands, and the committer date of the branch head is the number that matters.
The newest branch is not always the maintained one. Rockchip's develop-6.6 is 343 days stale while the default develop-6.1 is 227 days stale.
A specific silicon longevity date is publicly available for only four of the seventeen SoCs we track. Several other vendors run published longevity programmes, but the per-part enrolment and end date are not published where a buyer can find them. Module-level claims are not silicon-level claims.
The consequences are asymmetric: a stale branch found during SoC selection is a line in a comparison table, while the same fact found after a CVE is public is a release-plan problem. CRA reporting obligations from 11 September 2026 make the question harder to defer.
A quiet public branch means you cannot verify that fixes are flowing, not that a vendor has abandoned a part. NDA channels, SDK releases and deliberate branch retirement all look identical from outside.

Frequently asked questions

How do I check when a vendor BSP branch last moved?
Resolve the branch head with git ls-remote, then read the committer date of that commit. On GitHub-hosted trees the branches API returns it directly; elsewhere a shallow clone followed by git log -1 --format=%cI gives the same answer. Use the committer date rather than the author date, because it reflects when the change entered the tree.

My kernel version is not on the kernel.org longterm list. What does that mean?
It means the series is not receiving upstream stable maintenance and has no projected end-of-life to plan against. All fixes for that platform come from the vendor, so the vendor's release cadence becomes your only support clock.

Does a stale vendor branch mean the SoC is abandoned?
No. It means you cannot verify from public sources that fixes are still reaching the tree you build from. Vendors also ship through NDA channels and periodic SDK releases, older branches are retired deliberately after a migration, and some vendors rebase on a stated eighteen to twenty-four month cadence.

Is a ten-year longevity claim from my module supplier the same as one from the chip vendor?
No. A module vendor's commitment covers the module and depends on that vendor's own supply of the silicon. Several chip vendors run published longevity programmes, but enrolment is documented per part number and for most of the parts we track that per-part date is not in the public domain. Ask for your exact part number in writing rather than assuming family-level coverage.

How often should I re-run these SoC kernel support checks?
Quarterly is enough for a product in development, and the trend matters more than any single reading. Record the head dates each time, because a tree that is slowing down is visible in a series of measurements and invisible in one.

What fork() Actually Costs, and Why the Process-Builder RFC Does Not Fix It Yet

Raghu Bharadwaj — Mon, 10 Aug 2026 09:16:13 +0000

A process-builder API is proposed for Linux — pidfd_open(0, PIDFD_EMPTY), pidfd_config(), pidfd_spawn_run() — to construct a process incrementally instead of copying the parent with fork() and discarding the copy with exec(). We traced the cost argument through the kernel source at v7.1.7, and it has three parts: the mapping walk, the page-table copy for anonymous memory, and a descriptor-table copy sized by your highest descriptor number. posix_spawn() already removes the first two and not the third. In its current RFC form the proposed API does not improve on that: it uses CLONE_VM | CLONE_VFORK internally too, so it avoids the same two costs and still copies the descriptor table.

Status: RFC. First series 28 May 2026 (13 patches, superseded by a redesign); second series 16 July 2026 (24 patches, against linux-next). No pidfd or exec maintainer had replied to the second series at the time of writing. Nothing described here is in a released or -rc kernel; you would have to apply the series to linux-next yourself to try it.

What happened

The practical conclusion first, because it decides whether the rest of this concerns you: this interface is not in any released kernel, and in its current form it is not faster than what posix_spawn() already gives you. The useful question is not whether to wait for it, but where process creation actually costs you time and latency today. That is what most of this article is about.

In May 2026, Li Chen posted a patch series to speed up a specific pattern: runtimes that repeatedly launch the same short-lived helpers. The stated target was coding agents that start rg, git, sed and python over and over. The design cached executable metadata behind a template file descriptor and spawned from that.

Reviewers rejected the shape while accepting the goal. Mateusz Guzik argued it optimised the wrong end — "Most of this shaves off a tiny amount of work, while retaining the primary avoidable reason for bad performance: the very fact that fork is part of the picture, especially the part mucking with mm" — and Andy Lutomirski wrote that it was "a lot of complexity in the kernel for a teeny tiny gain". Christian Brauner proposed building on pidfds instead, modelled on fsconfig(), and said that "any implementation should also allow userspace to implement posix_spawn() on top of it". Kees Cook agreed.

Chen accepted the redirection and, in July, posted a 24-patch replacement following Brauner's sketch. It is a single-descriptor model:

int fd = pidfd_open(0, PIDFD_EMPTY);

/* Optional executable-path staging. */
pidfd_config(fd, PIDFD_CONFIG_SET_STRING,
             PIDFD_CONFIG_KEY_PATH, path, 0);

pidfd_spawn_run(fd, &args, sizeof(args));

The empty pidfd is "a taskless future pidfd with a stable pidfs inode, but no task, PID, or process-count charge". Live-task operations return -ESRCH until the run step creates one; afterwards the same descriptor is the child's pidfd. LWN has covered both rounds, and the mailing-list threads are linked in full at the end. The rest of this article checks the performance claim against the kernel source and works out who it applies to.

What we checked, and what it changes

The usual one-line summary is that fork() is expensive because it copies the process, including its memory. We traced the paths in the v7.1.7 tree. There are three separate costs, they scale with different things, and each one changes what you would do about it.

First, the mapping walk; second, the page-table copy. These are usually described as one cost and they are not. Copy-on-write means page contents are not duplicated. But page tables are, for some mappings, and that is the part usually left out. The walk in dup_mmap() costs one structure per virtual memory area, so it is proportional to the number of mappings. Then, per mapping, copy_page_range() decides whether to copy page tables at all. Its gate, vma_needs_copy(), returns false — skipping the work entirely — for mappings a fault can refill later; the comment there reads "Fork becomes much lighter when there are big shared or private readonly mappings." But it returns true whenever the source mapping has an anon_vma, which every private mapping acquires as soon as it is written. For those, every present page table entry is copied.

So the first cost tracks the number of mappings and the second tracks resident anonymous pages. A large read-only file mapping is close to free, and a large heap is not. A process with twelve mappings and a two-gigabyte touched heap is twelve lines in /proc/PID/maps and roughly half a million page-table entries to copy. Measure both terms: wc -l < /proc/PID/maps for the first, and AnonRSS in /proc/PID/status for the second. Note also that a library's writable data segment acquires an anon_vma at relocation time, so "read-only libraries are free" applies to their text segments and not their data.

A side effect of the walk is worth knowing if you read traces: the parent's mmap_lock is held for write throughout, and each mapping is also write-locked individually as it is walked. Under per-VMA locking it is that per-mapping lock, rather than mmap_lock alone, that stalls a concurrent fault in another thread.

The second cost has a delayed part. Setting up copy-on-write write-protects the page-table entries of both processes, not just the child's. The line in __copy_present_ptes() is commented "If it's a COW mapping, write protect it both processes", and it calls wrprotect_ptes() on the source mm. So after fork() returns, the parent's own writable private pages are write-protected, and its next write to each one takes a minor fault.

Whether that fault also copies the page depends on who else holds it. do_wp_page() reuses the page in place when the folio is exclusively owned, so once the child has exec'd or exited the parent usually pays one fault per page and no copy; it pays the copy as well while the page is genuinely still shared. Either way the cost is deferred and does not appear in a benchmark that times the fork() call, and the write-protect pass itself, including its TLB flush, is paid before fork() returns.

Third, the descriptor table, sized by your highest descriptor number rather than the count. The extent of the walk in dup_fd() comes from sane_fdtable_size(), which is built on find_last_bit() over the open-descriptor bitmap and rounds up to a multiple of BITS_PER_LONG. A process holding descriptors 0, 1, 2 and one stray at 100000 walks 100032 slots and allocates a table for them, against 64 slots and no allocation without the stray. Two processes with identical open-file counts can differ by three orders of magnitude on the descriptor-copying step. This is the cost that produces behaviour with no obvious cause, and it is also the one you can fix today.

What posix_spawn() already avoids

One further check, because it surprises people who have read the POSIX page: there is no spawn system call in Linux. We grepped every architecture syscall table at v7.1.7 to confirm it. glibc implements posix_spawn() in user space, and its own source comment says how — it "uses the clone syscall directly with CLONE_VM and CLONE_VFORK flags and an allocated stack", which today means clone3() with a fallback for older kernels.

That matters for a reason the RFC discussion does not draw out. CLONE_VM collapses both memory-side costs to a single mmget(), and CLONE_VFORK is what makes sharing the address space safe, by suspending the calling thread until the child execs or exits. But glibc does not pass CLONE_FILES, so posix_spawn() still copies the descriptor table. Of the three costs above, the C library already removes the first two and leaves the third to you.

The three shapes side by side, which is the comparison the discussion rarely shows:

/* 1. fork() + exec(): pays all three costs */
pid_t pid = fork();
if (pid == 0) {
        execve(path, argv, envp);
        _exit(127);
}

/* 2. posix_spawn(): CLONE_VM | CLONE_VFORK in the C library.
      Removes the mapping walk and the page-table copy.
      Still copies the descriptor table. */
pid_t pid;
posix_spawn(&pid, path, NULL, NULL, argv, envp);

/* 3. The proposed builder. Not in any released kernel.
      Today it is CLONE_VM | CLONE_VFORK internally, so it
      pays what posix_spawn() pays. */
int fd = pidfd_open(0, PIDFD_EMPTY);
pidfd_spawn_run(fd, &run, sizeof(run));

What this costs on an embedded target

The code above is the same on every architecture Linux runs on. dup_mmap(), copy_page_range() and dup_fd() do not vary by platform. What varies is the size of each input, and which consequence you care about — and on one point, whether the system call exists at all.

Without an MMU there is no fork(). On a CONFIG_MMU=n build, fork() returns -EINVAL; the kernel source comments it "can not support in nommu mode". vfork() carries no such guard, so on no-MMU targets vfork() and posix_spawn() are what you have. For a large class of embedded systems, then, avoiding the fork() cost is not an optimisation to consider; it is already the only path available.

On PREEMPT_RT the cost is jitter, not throughput. The question on a real-time system is not whether a process launches a few percent faster. It is whether a fork() elsewhere on the system injects latency into unrelated real-time work. It does: the fork holds the parent's mmap_lock for write, allocates, and write-protects page-table entries, which requires TLB shootdown across CPUs. The RFC's throughput numbers say nothing about this, and the shootdown cost scales with the number of pages being write-protected.

A large daemon spawning small helpers. Media and camera frameworks are the common case: a long-running process with a large heap and many buffer mappings launching short-lived codec or processing helpers. The heap, not the framework, is what makes it expensive. For the large device and dma-buf mappings these processes hold, MADV_DONTFORK is the direct remedy — it sets VM_DONTCOPY so the mapping is skipped entirely. It applies to distinct mmap() regions, so it is a remedy for buffer mappings and not for a large heap.

Descriptor tables are relatively more expensive on small systems. The high-water-mark cost is the same code everywhere, but a descriptor table sized for number 100000 is several hundred kilobytes of pointer array plus bitmaps. On a server that is noise. On a 128 MB target already balancing media buffers, DMA heaps and graphics memory, it is not.

Two things cut the other way, and they are the reason the agent-runtime framing does not transfer directly. Guzik's main objection to building the child's state in the parent is that the allocations happen on the parent's node, so a child intended to run elsewhere starts with its memory on the wrong node; on a single-node SoC that objection does not apply at all. And on a small target booting from eMMC, NAND or SPI-NOR, spawn latency is dominated by ELF loading, dynamic-linker relocation and cold-cache demand paging — the exec half. Even where fork cost is measurable, it is often not the term that decides how long a spawn takes.

The general shape, then: on a BusyBox shell, an init or supervisor, or a watchdog respawning a service, all three cost terms are short and there is nothing here to fix. The cost becomes real when the spawning process is large, when descriptor numbers have drifted high, or when the system has latency requirements rather than throughput requirements. The Android zygote shows the first case is solvable in user space. It starts early, loads the shared libraries and runtime that every application will need, and then waits. Applications are forked from that process rather than from a large parent, so each fork copies a small, known address space instead of whatever the parent has grown into.

The seccomp question nobody has answered

One item in the July series deserves separate attention from anyone shipping a sandboxed product, and the author raised it himself rather than being caught out on it. Seccomp sees pidfd_spawn_run() and cannot dereference the user pointer to inspect the path or the action records behind it. An exec-only denylist that permits unknown syscalls would therefore not block this exec. The cover letter puts it as "policy must filter the builder syscall as a unit", and ends the paragraph with an open question to reviewers rather than a decision.

The operational consequence is what matters: if a builder syscall is merged, seccomp policies will have to filter it explicitly. A policy that reasons about execve() today will not automatically extend to it, because seccomp cannot inspect the staged path and action data behind the user pointer. That inability is long-standing and documented, so this follows from how the interface is shaped rather than from a bug in the implementation. No reviewer has responded to the question.

What the process-builder API does not do yet

The most useful sentence in the July cover letter is the author's own assessment of what he has built: "The implementation still uses CLONE_VM | CLONE_VFORK plus exec internally", and it "does not yet construct a pristine target process without first inheriting source state". The elided sentence between those two makes clear this was deliberate — Guzik had suggested starting with vfork to begin the implementation, and that is what the RFC does.

Read against the cost model above, that means the process-builder API currently does what posix_spawn() already does, with a cleaner interface: it removes the two memory-side costs and not the descriptor-table copy. The pristine-process work — the part that would remove the remaining cost, argued for by Guzik and by John Ericson, who described a partially initialised process that stays unscheduled while callers install its state — is listed as follow-up. Anyone reading the headline as "process creation is about to get faster" is describing work that has not been written yet.

It is also not close to replacing posix_spawn() functionally. The series lists what is missing: "open and close file actions, resetids, signal masks/defaults, process groups, sessions, scheduler attributes, affinity, cgroup placement, PATH lookup/posix_spawnp(), and exec by fd."

What we could not verify

Three things here are reported rather than checked, and should be read that way.

We did not measure any of this. The cost model above comes from reading the code at a pinned tag, not from timing it on hardware. Where we say a cost scales with something, that is what the source does, not a curve we plotted.

The benchmark is the author's own. The first series reported between +2.00% and +4.99% higher tool-call throughput on his own harness against a Python subprocess baseline, with a later thread remark of "about +14% for printf-style work" for very short single-tool runs. We have not reproduced it. No reviewer reproduced it either, and two of them argued the gain was small regardless of the measurement.

The review status is the claim most likely to be out of date by the time you read this. At the time of writing, neither Brauner nor Kees Cook had replied to the July series. The two responses on the list were from Justin Suess, who argued the executable should be passed as a descriptor rather than a path string and questioned whether pidfd_spawn_run() should exist rather than extending execveat(), and Andy Lutomirski, replying to one patch, who flagged that checking whether a task is embryonic must happen before reading its credentials or the read is a security-relevant data race.

On authorship: the July series discloses substantial LLM assistance in its cover letter and carries Assisted-by: trailers on its patches. The May series did not disclose it up front — it came out when Guzik asked directly whether the work was "vibe-coded", meaning produced from an LLM prompt rather than from a working understanding of the subsystem, and Chen answered "Partly, yes". We note it because it is on the record and became part of the review discussion. It does not change the engineering question, which is whether the series can justify its interface, its security model and its performance value upstream.

What to do today

None of the proposed interface is available in a released kernel. The cost model is, and it points at six things that do not depend on this RFC landing.

Prefer posix_spawn() over fork() plus exec() when the child setup fits within spawn attributes and file actions. It removes the mapping walk and the page-table copy, which are the two largest costs for most processes. glibc and musl both implement it with CLONE_VM | CLONE_VFORK; check your C library if you ship uClibc-ng or an old Bionic. Two caveats: CLONE_VFORK suspends the calling thread until the child execs or exits, which moves ELF loading and relocation onto your critical path and can be the worse trade on slow flash; and you cannot run arbitrary code between spawn and exec.
Find out what your runtime actually does. strace -f -e trace=clone,clone3,vfork,execve on the spawning process settles in one run whether you are paying for a full fork() or not.
Measure both memory cost terms. wc -l < /proc/PID/maps gives the mapping count and AnonRSS in /proc/PID/status gives the anonymous footprint. A few dozen mappings and a few megabytes of anonymous memory means you have no problem. A few dozen mappings and a gigabyte heap means you do.
Keep descriptor numbers dense. The table is sized by the highest number you hold, not by how many. Close what you do not need before spawning, which close_range() does in one call, and do not raise RLIMIT_NOFILE beyond what you use. This is the one cost posix_spawn() does not remove.
Use MADV_DONTFORK for large inherited mappings that should not cross the spawn boundary. It applies to distinct mmap() regions, so it is a remedy for device and dma-buf buffers rather than for a large heap.
On PREEMPT_RT, evaluate jitter rather than average throughput. The cost that matters there is the latency a spawning process injects into unrelated real-time tasks, which no throughput benchmark will show you.

If a large daemon must launch helpers and none of the above is enough, the structural fix is the zygote pattern described earlier: fork a small helper early and let it do the launching.

What to watch upstream

The signal that this is moving is a non-RFC posting, or an ack from the pidfd or exec maintainers on the lore thread linked below. A reply from Christian Brauner in particular would settle whether the pidfd-based shape survives. The thing to watch for after that is whether the pristine-process work appears, because that, and not the interface, is what would change the numbers.

Frequently asked questions

What is the process-builder API?
It is a proposed Linux interface for constructing a new process step by step and then running it, instead of copying the parent with fork() and replacing that copy with exec(). The current RFC uses pidfd_open with the PIDFD_EMPTY flag to create an empty process handle, pidfd_config() to configure it, and pidfd_spawn_run() to start it.

Can I use the process-builder API today?
Not in a released kernel. It is an unmerged RFC patch series and the design has already changed once, so trying it means applying the patches to linux-next yourself.

Will it make process creation faster?
Not in its current form. The cover letter states the implementation still uses CLONE_VM and CLONE_VFORK plus exec internally and does not yet construct a pristine target process, so it performs about as posix_spawn() already does. The work that would remove the remaining cost is listed as follow-up.

Is fork() slow because it copies all the parent's memory?
Page contents are not copied, but page tables are, for any private mapping that has been written. So fork cost has two terms: one proportional to the number of mappings, and one proportional to the number of resident pages in anonymous mappings. A large read-only file mapping is nearly free; a large heap is not.

Which of these costs does posix_spawn() actually remove?
Two of the three. CLONE_VM removes the mapping walk and the page-table copy. It does not pass CLONE_FILES, so the descriptor table is still copied and the cost of a high descriptor number remains. Close unneeded descriptors before spawning to deal with that separately.

Does fork cost differ by architecture?
The code does not. dup_mmap(), copy_page_range() and dup_fd() are the same everywhere. What differs is the size of each input and which consequence matters. One real exception: on a CONFIG_MMU=n build fork() returns -EINVAL, so no-MMU targets use vfork() or posix_spawn() instead.

When does this actually cost me on an embedded target?
When the spawning process is large, such as a media daemon with a big heap launching codec helpers; when descriptor numbers have drifted high, since the table is sized by the highest number in use; and on PREEMPT_RT systems, where the TLB shootdown a fork requires injects latency into unrelated tasks on other cores. On a BusyBox shell or a small supervisor all three cost terms are short.

Does Linux have a posix_spawn() system call?
No. There is no spawn system call in the kernel. glibc and musl implement posix_spawn() in user space using the clone family of system calls with CLONE_VM and CLONE_VFORK, which is why it avoids the address space duplication that fork() performs.

References

RFC PATCH 00/24 — pidfd: add a minimal process spawn builder — Li Chen, 16 July 2026. Quotes from Justin Suess and Andy Lutomirski are replies on this thread.
RFC PATCH v1 00/13 — exec: add spawn templates for repeated executable startup — Li Chen, 28 May 2026. Quotes from Mateusz Guzik, Andy Lutomirski, Christian Brauner and Kees Cook are replies on this thread.
Moving beyond fork() + exec() — Jonathan Corbet, LWN.net, 5 June 2026
The beginning of a process-builder API — Jonathan Corbet, LWN.net, 4 August 2026
mm/mmap.c at v7.1.7 — dup_mmap and the mapping walk
mm/memory.c at v7.1.7 — vma_needs_copy, __copy_present_ptes and do_wp_page
fs/file.c at v7.1.7 — dup_fd, sane_fdtable_size and close_range
kernel/fork.c at v7.1.7 — copy_mm, copy_files and vfork handling
posix_spawn(3) manual page
madvise(2) manual page — MADV_DONTFORK

— Raghu Bharadwaj

Linux Kernel CVEs: What to Patch by Device (2 – 8 Aug 2026)

Raghu Bharadwaj — Sat, 08 Aug 2026 14:06:59 +0000

The Linux kernel project published 46 Linux kernel CVEs in the week of 2 to 8 August 2026. Every one is already fixed in a stable release. There is no 0-day and no public exploit code. The only action is to update your kernel to the fixed version for your branch. The minimum safe release per branch is 5.10.264, 5.15.215, 6.1.182, 6.6.150, 6.12.102, 6.18.43, 7.1.7, or mainline 7.2-rc5. The 6.19 and 7.0 series are end of life; move those systems to 7.1.7.

This is a normal-volume week, with 46 CVEs. Most are narrow driver and networking fixes. Which of these Linux kernel CVEs matter for you depends on what your product builds and how it is configured. No CVE this week has a public exploit or a CVSS score in the kernel record, so the order of work is set by how the bug is reached. A few are reachable over the air or over the network, so schedule those first where they apply. This advisory gives the version to install, then explains which CVEs matter for mobile and automotive, embedded and IoT, cloud and datacenter, and medical devices.

The one action: update to the fixed version for your branch

Run uname -r to see your current kernel version. Compare it with the target for your branch in the table below. If your version is lower than the target, update to the target release. Updating to the target release for your branch clears every CVE in this week's set.

Stable branch	Update to at least	Notes
5.10 LTS	5.10.264	Affected; the SRSO fix CVE-2026-68480 sets this point release.
5.15 LTS	5.15.215	Affected, same as 5.10.
6.1 LTS	6.1.182	Current release; carries this week's 6.1 backports.
6.6 LTS	6.6.150	Current release.
6.12 LTS	6.12.102	Current release.
6.18 LTS	6.18.43	Current release.
7.1 stable	7.1.7	Current stable release.
mainline	7.2-rc5	All fixes merged.

Branches not listed were not affected this week, or their fix predates this window. The 6.19 and 7.0 series are end of life and receive no update; move those systems to 7.1.7.

Mobile and automotive

The wireless fixes matter most for phones and car head units. CVE-2026-64571 is an out-of-bounds read in the Wi-Fi p54 driver; a crafted frame during an EEPROM readback reads past the socket buffer, so it is reachable over the air on devices that use a p54 USB or PCI adapter (CONFIG_P54_COMMON). CVE-2026-64573 is an out-of-bounds write in the Qualcomm Bluetooth NVM parser; a length underflow lets a 12-byte structure be written past a short firmware buffer, and Qualcomm Bluetooth parts are common in phones and head units (CONFIG_BT, CONFIG_BT_QCA). CVE-2026-64586 is a use-after-free in the Broadcom FullMAC Wi-Fi driver when the device is removed, and Broadcom Wi-Fi is widely used in mobile platforms (CONFIG_BRCMFMAC). The mac80211 bugs CVE-2026-64568 and CVE-2026-64570 are double frees on the beacon and FILS discovery configuration path, and CVE-2026-64574 is a use-after-free reached through the per-link debugfs files; all three need CONFIG_MAC80211 and a software access point or multi-link setup. For connected-car network stacks, CVE-2026-64577 is a buffer under-run panic in the GTP-U echo path (CONFIG_GTP) and CVE-2026-64564 is a use-after-free in SCTP (CONFIG_IP_SCTP); both are reachable from the network.

Embedded and IoT

The Broadcom Wi-Fi use-after-free CVE-2026-64586 matters here too, because many single-board computers use Broadcom FullMAC parts (CONFIG_BRCMFMAC). Gateways and network-attached storage devices that run the in-kernel SMB server should apply CVE-2026-64578, an out-of-bounds read that an SMB client can trigger against a crafted compound request (CONFIG_SMB_SERVER). Panels and kiosks that use the IMS PCU device should apply CVE-2026-64565, a heap buffer overflow driven by a malicious USB device (CONFIG_INPUT_IMS_PCU); it needs USB access. Industrial gateways with USB-CAN adapters should apply CVE-2026-64585, a use-after-free when an esd USB-CAN device is removed (CONFIG_CAN_ESD_USB). Devices that mount removable btrfs media should apply CVE-2026-64567, an out-of-bounds read from a crafted free-space cache (CONFIG_BTRFS_FS). Products still on 5.10 or 5.15 are affected this week through the SRSO fix and shared driver fixes, so plan an update to 5.10.264 or 5.15.215.

Cloud and datacenter

The KVM guest-to-host bugs are the highest priority here. CVE-2026-64562 is a use-after-free where a shadow VMCS is freed while it is still referenced and a vCPU migration races a VMCLEAR. CVE-2026-64561 lets guest activity map pages into an invalid shadow-MMU root. CVE-2026-68081 leaks pinned host pages when a nested VM-Enter fails. All three apply to hosts that run nested virtualization on Intel parts (CONFIG_KVM_INTEL with nested mode on). CVE-2026-64582 is a use-after-free in the soft-RoCE driver (CONFIG_RDMA_RXE), and CVE-2026-68082 is an out-of-bounds read in the Ceph client that a malicious or compromised OSD can trigger (CONFIG_CEPH_LIB). File servers should apply the ksmbd bug CVE-2026-64578 (CONFIG_SMB_SERVER) and clients should apply CVE-2026-64597 and CVE-2026-64598 in the SMB client (CONFIG_CIFS). On multi-tenant hosts, CVE-2026-64581 is a double free in the xfrm policy path that an unprivileged user can reach through user and network namespaces (CONFIG_XFRM_USER), and CVE-2026-68480 is the SRSO speculation fix that matters where guests from different tenants share an affected AMD CPU (CONFIG_MITIGATION_SRSO).

Medical devices

The wireless and USB bugs are the ones to review for connected clinical devices. CVE-2026-64573 in the Qualcomm Bluetooth parser matters for wearables, patient monitors, and hearing devices that use Bluetooth (CONFIG_BT). The Wi-Fi bugs CVE-2026-64571 (p54) and CVE-2026-64586 (Broadcom) matter for clinical devices on wireless networks (CONFIG_P54_COMMON, CONFIG_BRCMFMAC). CVE-2026-64565 matters for bedside units with USB input peripherals (CONFIG_INPUT_IMS_PCU). For networked clinical devices that carry DICOM or HL7 and FHIR over IP, CVE-2026-64564 in SCTP is relevant, because SCTP carries signaling in some clinical and telecom stacks (CONFIG_IP_SCTP); devices that expose an SMB share should also apply CVE-2026-64578 (CONFIG_SMB_SERVER). Medical-device patching is gated by validation and regulatory revalidation, under US FDA postmarket cybersecurity guidance, the EU Medical Device Regulation (MDR), and IEC 62304, so the action is to plan the stable kernel update through the manufacturer's change-control process rather than to apply it ad hoc.

How to check which Linux kernel CVEs apply to you

There are three questions.

First, version. Run uname -r and compare it with the table above. If your release is below the target for your branch, update to the target. This clears every CVE in this week's set for your branch.

Second, configuration. Check your kernel config for the feature behind each CVE. For example, run zcat /proc/config.gz | grep CONFIG_BT, or check CONFIG_MAC80211, CONFIG_P54_COMMON, CONFIG_SMB_SERVER, CONFIG_IP_SCTP, CONFIG_KVM_INTEL, or CONFIG_BRCMFMAC. If the feature is not built, the matching CVE cannot affect you.

Third, reachability, which sets the order of work. Over-the-air and network-reachable bugs come first: the Wi-Fi bug CVE-2026-64571, the Bluetooth bug CVE-2026-64573, and the network bugs CVE-2026-64564 (SCTP), CVE-2026-64577 (GTP-U) and CVE-2026-64578 (ksmbd). Local unprivileged bugs come next, such as the xfrm double free CVE-2026-64581. Guest-to-host bugs come after that on shared virtualization hosts, led by CVE-2026-64562. Bugs that need privilege or a specific configuration come last.

Key takeaways

Update to the target release for your branch: 5.10.264, 5.15.215, 6.1.182, 6.6.150, 6.12.102, 6.18.43, 7.1.7, or mainline 7.2-rc5.
Schedule the over-the-air and network-reachable bugs first: CVE-2026-64571 (Wi-Fi), CVE-2026-64573 (Bluetooth), CVE-2026-64564 (SCTP), CVE-2026-64577 (GTP-U), and CVE-2026-64578 (ksmbd).
On virtualization hosts, apply the KVM guest-to-host bugs CVE-2026-64562, CVE-2026-64561, and CVE-2026-68081.
The 6.19 and 7.0 series are end of life; move those systems to 7.1.7.
The SRSO fix CVE-2026-68480 sets the newest required point release on every LTS branch, including 5.10 and 5.15.
None of these are 0-days; all fixes are already in stable. Use your kernel config to remove the CVEs that do not apply, and test the update on your own hardware.

Frequently asked questions

What is the one thing I should do?
Update to the target release for your branch. Then use your kernel config and device type to decide what to test first.

Which ones should I schedule first?
The over-the-air and network-reachable bugs: CVE-2026-64571 (Wi-Fi p54), CVE-2026-64573 (Bluetooth QCA), CVE-2026-64564 (SCTP), CVE-2026-64577 (GTP-U), and CVE-2026-64578 (ksmbd SMB server). None has public exploit code.

Are any of these a 0-day?
No. All 46 CVEs are already fixed in stable, and there is no public exploit code. The action is to schedule the stable update through your normal process.

I run virtualization hosts. Which CVEs apply?
The KVM guest-to-host bugs CVE-2026-64562, CVE-2026-64561, and CVE-2026-68081. They apply to hosts running nested virtualization on Intel parts, so they need CONFIG_KVM_INTEL with nested mode on.

We do not build Wi-Fi, Bluetooth or the SMB server. Do these affect us?
Each bug is gated by a kernel config option. If CONFIG_MAC80211, CONFIG_P54_COMMON, CONFIG_BT and CONFIG_SMB_SERVER are not built, those bugs cannot affect you. Check your config to remove the ones that do not apply.

Choosing an A/B Update Layout for Your Product

Raghu Bharadwaj — Sat, 08 Aug 2026 05:13:01 +0000

Symmetric A/B — two equal root filesystem slots — is the right default for most connected products: install to the inactive slot, reboot into it, and fall back automatically if the new software fails its health check, all while the device keeps running during the write. Limited storage, slow boot, or an application that ships on its own schedule make asymmetric, split, or rescue-augmented layouts the better choice instead. The decision is not whether to use A/B, but which A/B update layout matches your constraint.

Most embedded Linux teams accept early that field updates need redundancy: keep a known-good copy so an interrupted or faulty update cannot leave the device unbootable. The usual answer is an A/B update layout — two slots the device switches between. But "A/B" is not a single design. Each layout is tuned to a different constraint, and choosing the wrong one costs storage, boot time, or availability you did not account for. This post describes the common layouts on their merits and maps each to the scenario where it is the right choice.

The context

An A/B update layout answers the interrupted-update problem, and the mechanism is common to every form — the rollback cycle shown below. What differs is how four factors are balanced, and those factors often conflict. Storage: a second full root filesystem roughly doubles the rootfs space. Availability: some layouts keep the application running during the update, while others must reboot into an update mode. Boot time and bootloader complexity: where the kernel and device tree sit affects both. Release cadence: if the base system and application ship on different schedules, updating them together wastes bandwidth.

That shared rollback cycle:

Install the bundle into the inactive slot.
Mark that slot as the next boot target.
Reboot into it.
Run early boot and application-level health checks.
Mark the slot good only after the checks pass.
Fall back automatically if a boot counter or watchdog detects a failure.

The A/B update layouts

Symmetric root-fs slots

Two root partitions of equal size, A and B. Running from A, an update installs into B, and the reverse when running from B; both slots hold equivalent software, including the main application. To keep the design simple, the kernel and device tree live inside the rootfs (usually in /boot), which requires a bootloader that understands the rootfs type. RAUC describes this as probably the most common setup.

Slot A — active	Slot B — inactive (update target)
rootfs + kernel/DTB (/boot)	rootfs + kernel/DTB (/boot)

The two slot entries in system.conf look like this, with the mandatory [system] section omitted for brevity:

raghu@techveda.org:~$ cat /etc/rauc/system.conf
[slot.rootfs.0]
device=/dev/mmcblk0p2
type=ext4
bootname=system-a

[slot.rootfs.1]
device=/dev/mmcblk0p3
type=ext4
bootname=system-b

Advantages: an update can start from either slot while the application keeps running; the bootloader fallback logic stays simple; and the process is easy for technicians and users to reason about. The new software takes effect only after a reboot into the updated slot and a passing health check. The reasons to look elsewhere are limited storage, or a need for more redundancy or update flexibility.

Asymmetric slots

When storage is very limited, replace the second full partition with a small "update" (or rescue) partition alongside the full "main" system. To update, the device reboots into the small update system, which writes the new image into the main slot and switches back. Being small, that system usually fits entirely in RAM as a kernel with an internal initramfs (a raw slot). It is a temporary update environment, not a preserved recovery image; that is a separate design, covered below. The costs, as RAUC states them: two reboots per update; a failed update leaves the main application unavailable until a later update succeeds; and any data that must survive in the main slot must be saved elsewhere first and restored afterwards, because the slot is overwritten.

main — full rootfs (active)	update — kernel + initramfs, runs in RAM

Split (multiple) slots

Separate the application from the base root filesystem so each updates on its own schedule — useful when a different team owns the application, or when several applications reuse one base. RAUC models it with an appfs slot whose parent is the rootfs slot. You can also split the boot files (kernel) into their own slot to reduce boot time and bootloader complexity. Split slots layer on top of a symmetric or asymmetric setup. One caution from RAUC: it cannot currently guarantee compatibility between a rootfs and an appfs when a bundle updates only one of them, so always build bundles that contain every required slot, or enforce compatibility in your own release process.

Base	Slot A	Slot B
rootfs	base system	base system
appfs	application	application

Symmetric A/B plus a rescue slot

A rescue slot is a deliberately preserved recovery environment, distinct from the temporary update system of an asymmetric layout; normal updates never write to it. It guards against failure modes that can disable both A and B at once: a latent software error undetected across releases; corruption from mishandled power failures when slots are mounted read-write; or a configuration error that fails both slots the same way. The bootloader starts the rescue system after repeated boot failures of the normal slots, or on user request.

rescue — never written by updates	Slot A — rootfs	Slot B — rootfs

The decision: matching the layout to the product

Match the A/B update layout to the dominant constraint. The table gives the quick mapping; the notes below cover the cases that need judgement.

Layout	Best fit	Main benefit	Main cost
Symmetric A/B	Most connected products	Simple rollback; device runs during the write	Two full rootfs slots
Asymmetric	Tight flash budgets	Lower storage requirement	Two reboots and update-mode downtime
Split slots	Separate app and base release cycles	Independent update cadence	Compatibility management
A/B plus rescue	Remote or high-consequence devices	Recovery path if both normal slots fail	More storage and boot logic

The table maps each layout to its constraint; three points do not fit in a table. Use a proven framework — RAUC, SWUpdate, or Mender — rather than scripting slot selection and rollback yourself; when you can afford the space, symmetric is easier to operate than asymmetric. With split slots, enforce rootfs/appfs compatibility in your bundle policy so a partial update cannot pair mismatched versions. And pair a rescue slot with a read-only, verified rootfs (see our note on how dm-verity keeps a read-only rootfs verified).

Slot layout is only one part of a safety-relevant update strategy: secure boot, signed bundles, watchdog validation, and failure analysis are also required. Payload size is a separate choice too — whichever A/B update layout you pick, frameworks support delta or adaptive payloads to cut download size, decided independently of the layout.

Consequences

Storage follows directly from the A/B update layout: symmetric roughly doubles the rootfs, split avoids duplicating a shared base or heavy assets across slots, and asymmetric trades storage for reboots and downtime. In every layout, keep application state and user data on a separate writable partition, not inside the slots, so it survives updates and is not duplicated — this also removes the asymmetric layout's need to save and restore that data around the overwrite. Treat that partition as its own design problem: power-fail-safe writes, a schema that still loads after a rollback to older software, and integrity or encryption where the product needs it.

Someone must own the bootloader boot-selection logic and a hardware watchdog so that a boot hang counts as a failure and triggers fallback. Symmetric keeps this simple; splitting boot files or adding a rescue slot adds selection cases to maintain in U-Boot, GRUB, or Barebox. Symmetric A/B on a standard framework is well understood and easy to hand to new engineers; asymmetric and split layouts concentrate more logic and more failure modes, so document them. The build-system layers that realize these layouts — meta-rauc or meta-swupdate — are part of the earlier Yocto versus Buildroot decision, and integrating them is covered in our Embedded Linux on Edge AI course.

Key takeaways

The question is not whether to use A/B, but which A/B update layout fits your constraint; the comparison table maps each to its scenario.
In every layout the new software is activated only after a reboot and a health check, with automatic fallback to the previous slot on failure.
Keep mutable state on a separate writable partition in every layout; slot layout is one part of a strategy that also needs secure boot, signed bundles, and a watchdog.

Frequently asked questions

Which A/B update layout should I start with?
Symmetric root-fs slots, unless storage is too limited to hold two full root filesystems. It is the simplest robust layout and lets updates run while the device keeps working.

Why would I choose asymmetric slots?
When storage cannot hold two full root filesystems. You keep one full main system and a small update or rescue system, at the cost of two reboots per update and the main application being unavailable if an update fails.

What does a rescue slot protect against?
Failures that disable both A and B together: a latent bug that spreads across releases, corruption from power loss on read-write slots, or a configuration error that fails both slots identically. Normal updates never write to it.

Where should the kernel and device tree live?
In the simplest symmetric layout they sit inside the rootfs, under /boot, which needs a bootloader that understands the rootfs type. Splitting them into a separate slot can reduce boot time and bootloader complexity. The exact implications depend on the bootloader, filesystem support, and image format, so treat boot-artifact placement as a per-platform decision.

Embedded Linux Boot Time: The Phase You Are Not Measuring

Raghu Bharadwaj — Fri, 07 Aug 2026 06:49:30 +0000

A board spends its boot time in four phases: ROM and SPL, U-Boot proper, the kernel, and user space. Each one has its own clock, and each clock starts at zero. That matters because the dmesg timestamp of 0.000000 is not power-on, so measuring the kernel alone tells you nothing about the two phases before it. This article walks through the four phases, shows how to put them on a single axis using a timestamped serial log, and covers measuring the bootloader with bootstage.

Every embedded Linux product eventually gets a boot time requirement. A backup camera has to show an image within two seconds. An industrial controller has to answer a fieldbus request before the master times out. A battery device has to finish booting before the wake budget is spent.

The requirement usually arrives late, and the response is usually the same. Someone opens dmesg, finds the slowest line, and starts working on it. Two weeks later the number has barely moved.

The engineer did not pick a bad target. The problem is that dmesg shows only one of the four phases a board goes through, and its clock starts at zero after two of the others have already finished. You can spend a long time optimizing something that was never the expensive part.

So this article is about measuring rather than tuning. Once you have a timeline that runs from power-on to the moment your application is ready, the decisions get easy and usually small. Without one, you are guessing.

Why boot time work usually goes wrong

Three things about the boot path make it hard to reason about until you instrument it.

Each phase has its own clock. The kernel log timestamp of 0.000000 is not power-on. It is the moment the kernel started running and printk timestamping became available. Everything the ROM, the SPL, and U-Boot did before that is not part of that number. On a board with slow storage and a large U-Boot, the time you cannot see is often longer than the entire kernel phase.

Time spent working and time spent waiting look identical. An initcall that takes 900 ms might be doing 900 ms of work. It might also be doing 5 ms of work and waiting 895 ms for a regulator, a clock, or a firmware file. The log line reads the same either way, and optimizing a driver that was blocked on something else saves you nothing.

In user space, a slow unit is not always a blocking unit. A service that takes 400 ms to start may not be on the critical path at all. If three other services start alongside it and the slowest takes 900 ms, removing your 400 ms service changes the ready time by zero.

Put those together and the result is a familiar pattern. A team spends two weeks removing kernel configuration options, ships an image 30 percent smaller, and finds that boot time moved by 40 ms. The real cost was a 1.2 second driver initialization sequence in U-Boot that nobody had measured.

The four phases, and the handoffs between them

Agree on where each phase starts and ends before you instrument anything. The handoffs between them are where measurements get lost.

Phase	Clock starts at	Measured with
1. ROM and SPL	power-on	scope or switched supply, no console output
2. U-Boot proper	U-Boot timer init	`bootstage`
3. Kernel	kernel entry	`initcall_debug`
4. User space	init exec	`systemd-analyze`

One external clock on the serial line joins phases 2, 3 and 4. Phase 1 needs hardware.

Phase 1: ROM and SPL

The SoC boot ROM runs from mask ROM, reads a boot mode strap or fuse, and loads a small first-stage image into on-chip SRAM. That image is the SPL, or TF-A BL2 on many Arm platforms. The SPL brings up DRAM, which is the reason it exists. Nothing bigger can run until the DDR controller is trained and calibrated.

On most platforms this phase prints almost nothing, and two things dominate its cost: how fast the ROM reads from the boot medium, and how long DDR training takes. On some DDR4 and LPDDR4 platforms training runs to several hundred milliseconds. A few vendors let you save the trained parameters so later boots skip the full sequence, which is worth asking about early.

Phase 2: U-Boot proper

U-Boot relocates itself to DRAM, initializes drivers, reads the environment, loads the kernel, device tree, and initramfs, verifies signatures if you use verified boot, and jumps to the kernel.

This is where unmeasured time builds up most often, because U-Boot sets up devices the kernel will set up again a second later. The duplication is not automatic: under the U-Boot driver model, a device that is bound but never probed costs very little. But any subsystem your boot path actually touches and your product does not need is time spent twice.

Phase 3: kernel

The kernel decompresses itself, sets up memory management and the scheduler, runs initcalls in level order, mounts the root filesystem, and executes the init process. Almost all of the variable cost is in initcalls and in waiting for the root device to appear. The kernel's own instrumentation is the initcall_debug boot parameter, which logs every initcall with a duration.

Phase 4: user space

The init system starts services until it reaches the unit or target your application depends on. On most product images that is systemd, and the number worth having is systemd-analyze critical-chain, not the total.

Notice what all four have in common. The requirement you were given is measured from power-on, and every tool in the standard set measures from the start of its own phase. Joining them up is a separate job, and it is the one most teams skip.

Building one timeline across all four phases

The problem: four clocks, no shared zero

U-Boot's bootstage timestamps start when U-Boot's timer is available. The kernel's printk timestamps start when the kernel starts. systemd-analyze reports a kernel time, an initrd time, and a userspace time, but its idea of "kernel time" begins where the bootloader handed over. None of these share an origin with the moment power was applied or the reset line was released.

Adding the phase totals together gets you close, but it leaves out the ROM time and every handoff gap along the way. Those gaps are exactly where unaccounted time usually is.

The fix: an external clock on the serial line

What works is timestamping the serial console from outside the board, so one clock covers everything the board prints, from the first SPL character to your application's ready message.

grabserial does this. It reads the serial port, adds a timestamp to each line, and can reset its base time when a line matches a pattern, so you can anchor zero on a marker you choose.

$ grabserial -v -d /dev/ttyUSB0 -b 115200 -e 30 -t -m "U-Boot SPL"

Reset the board with power rather than a warm reset, because a warm reset can skip ROM work that a cold boot performs. Capture the whole log to a file. That file becomes your timeline, and you read every later measurement against it.

Two settings make the log easier to work with:

Build the kernel with CONFIG_PRINTK_TIME=y, or pass printk.time=1, so kernel lines carry their own timestamps as well. The kernel records the timestamp internally either way and exports it through /dev/kmsg; the option controls whether it appears in console output.
The console itself costs time. A serial console at 115200 baud with 8N1 framing carries ten bits per character, so it moves about 11.5 kilobytes per second, and a verbose boot can spend hundreds of milliseconds printing. Measure with the console on to find your targets, then measure again with quiet to see the production number. Do not confuse the two.

What the serial line cannot show you

This method has one limit, and it is worth being clear about it. The serial timeline begins at the first character the board prints, which on most platforms is the first SPL message. Everything before that is invisible: the boot ROM reading the SPL off eMMC, SD, or SPI-NOR prints nothing, so none of it reaches the log.

Measure that interval separately, with hardware. Trigger the boot from something you can observe — a switched supply, a controlled reset line, or a scope probe on the reset signal and the serial TX line. What you get is the ROM read plus the SPL load, and on a slow boot medium that interval can be large.

You only have to measure it once. It rarely changes between software builds, so take it per board and per boot medium, write it down, and record it with the four phase totals.

Measuring the bootloader with bootstage

U-Boot can time its own boot, and most people never turn it on. Enable CONFIG_BOOTSTAGE in boot/Kconfig and it records timing marks at known points. There are two ways to read them back:

CONFIG_BOOTSTAGE_REPORT prints a timing summary automatically before U-Boot hands off to the OS.
CONFIG_CMD_BOOTSTAGE adds the bootstage command, so you can run bootstage report from the prompt. This is a separate symbol from the report option, and it depends on BOOTSTAGE.

The report is a table of marks in microseconds:

=> bootstage report
Timer summary in microseconds:
       Mark    Elapsed  Stage
          0          0  reset
    213,405    213,405  board_init_f
    418,772    205,367  board_init_r
    724,190    305,418  main_loop
    944,318    220,128  bootm_start
  1,204,551    260,233  start_kernel

Accumulated time:
                 41,220  dm_spl
                162,663  dm_r

The Elapsed column is the one to read, and it is easy to read it backwards. Elapsed is the time between the previous mark and this one, not the time spent inside the stage the row is named after. The 305,418 microseconds on the main_loop row is therefore the interval from board_init_r to main_loop, which is where U-Boot initializes drivers and reads its environment.

Read that way, the intervals here rank as:

305 ms, board_init_r to main_loop — driver initialization and environment work in U-Boot proper.
260 ms, bootm_start to start_kernel — image load, signature verification, and kernel decompression.
220 ms, main_loop to bootm_start — environment processing and boot command parsing.
213 ms, reset to board_init_f — SPL work and relocation, as U-Boot's own timer sees it.
205 ms, board_init_f to board_init_r — early init and relocation into DRAM.

The Accumulated section at the bottom narrows the largest item further. Of that 305 ms interval, 162,663 microseconds went on driver model probes in U-Boot proper (dm_r), against 41,220 in the SPL (dm_spl). If you need more detail than that, bootstage_mark_name() lets you add marks of your own around whichever subsystem you suspect, then rebuild.

Reducing bootloader time

Once you know where the bootloader spends its time, there are two options inside U-Boot itself and a third that belongs to the phase before it. None of them is tuning. All are decisions someone has to sign off on.

Reduce U-Boot's driver set. Anything U-Boot probes and never uses is time spent twice, because the kernel probes it again a moment later. Network, USB, and display are the usual candidates on a product that boots from local storage. They are almost always there for a factory or recovery path, which means the answer is usually two build configurations rather than one stripped-down one.

Falcon mode. CONFIG_SPL_OS_BOOT lets the SPL load and start the kernel directly, skipping U-Boot proper altogether. That removes the whole of phase 2, and on some designs it is the largest single saving available to you. It also costs you the U-Boot command line, the environment, and the normal update and recovery path, so most products keep a way back into full U-Boot. doc/develop/falcon.rst in the U-Boot tree describes how it works.

Saved DDR training parameters. This one belongs to phase 1 and depends on your SoC: some vendors let you save the trained parameters so later boots skip the full calibration. It reduces time in the phase that is hardest to instrument, which is why it is worth raising with your vendor early rather than late.

One warning if your product uses verified boot. All three of these affect signing. Two U-Boot configurations means two signing paths to maintain through the release process, and Falcon mode changes which component verifies the kernel.

A worked example

The numbers below are constructed rather than taken from one particular project. This is a common form of the problem, and the arithmetic is what makes the argument concrete, so it is worth walking through.

What the team expected

An industrial gateway on an i.MX8M Plus, booting from eMMC. The requirement is four seconds from power-on to the first camera frame on the display. Measured boot time is 11.0 seconds.

The kernel image is 9.8 MB, which feels large to everyone who looks at it. So the plan is two weeks of kernel configuration work, plus an evaluation of whether moving the image from Yocto to Buildroot would help. Nobody has measured anything except the total.

What the timeline shows

An afternoon of measurement produces this split:

Phase	Time
ROM and SPL, up to U-Boot proper (scope plus serial log)	0.4 s
U-Boot proper (`bootstage report`)	2.9 s
Kernel (`dmesg`)	3.2 s
User space (`systemd-analyze`)	4.5 s
Total	11.0 s

The first thing to notice is that 3.3 seconds of the 11 happen before the kernel starts, and none of it appears in dmesg. The team had been reading a log that begins after almost a third of the boot is already over.

Inside U-Boot, bootstage report puts 1.6 of the 2.9 seconds in USB and Ethernet initialization, and the Accumulated dm_r figure confirms most of it is driver model probing. Both subsystems exist for a TFTP recovery path. The boot script calls usb start and brings the interface up unconditionally, so the cost is paid on every boot, including the deployed units that never use that path.

What changes

Splitting U-Boot into a factory configuration with USB and Ethernet, and a production configuration without them, removes 1.6 seconds. That is an afternoon of work against the two weeks that had been planned, and the kernel configuration is never touched. The 9.8 MB image ships.

It is worth being honest about what this does not do. Removing 1.6 seconds from 11.0 leaves 9.4, against a requirement of four. The bootloader was not the whole problem, and finding the rest means measuring the kernel and user space, which is a separate investigation. What the afternoon established is that two weeks of kernel configuration work would have produced very little.

What it costs

Two U-Boot configurations now have to be maintained. Because the product uses verified boot, that also means two signing paths and two sets of keys to keep track of through the release process. There is a further risk that is easy to miss: the factory configuration and the production configuration can diverge, and a fault introduced in one may only appear on the production line, which is the worst place to find it. None of that is free. It is still smaller than two weeks spent on the wrong phase.

A bootloader measurement checklist

Here is the whole sequence for the first two phases, in order. On a board you already have running it takes about an hour.

Measure the interval from reset release to the first serial character once, with a scope or a controlled supply. This is your ROM and SPL-load cost, and it will not appear in any software log.
Connect a host to the serial console and start grabserial with an anchor pattern on the first SPL message.
Power cycle the board. Do not use a warm reset. Capture the full log to a file.
Enable CONFIG_BOOTSTAGE and CONFIG_CMD_BOOTSTAGE in U-Boot, boot, and run bootstage report.
Record the Elapsed column, reading each value as the interval before that mark rather than time spent inside the named stage.
Check the Accumulated section for dm_spl and dm_r. Those totals tell you how much of the bootloader's time is driver model probing.
Compare U-Boot's own total against the same interval in your grabserial log. If they disagree, the difference is a handoff gap, and you want to know where it is.
Repeat with quiet on the kernel command line to get the production number rather than the instrumented one.

What this means in practice

For the first day, treat boot time as a measurement problem. It becomes an engineering problem after that, and not before. Get an external clock onto the serial line so all four phases sit on one axis. Measure the pre-console ROM interval with hardware, because no amount of software will show it to you. And instrument the bootloader before you touch the kernel, because the phase you cannot see in dmesg is often the expensive one.

If the bootloader does turn out to account for most of your budget, the options above are design decisions with real costs attached, and you want to base them on a measurement rather than a guess. If it does not, the afternoon was still worth spending: you have ruled out two whole phases, and whatever remains is in the kernel or user space. That is a much smaller place to look.

Key takeaways

Boot time spans four phases with four independent clocks. dmesg timestamps start at kernel entry, not at power-on, so kernel-only measurement misses everything before it.
Join the phases with an external timestamp on the serial console. grabserial with an anchor pattern is the standard method, and a cold power cycle is required.
The interval before the first serial character cannot be measured in software. Use a scope or a controlled supply, once per board and boot medium.
U-Boot has built-in timing: CONFIG_BOOTSTAGE for the marks, CONFIG_BOOTSTAGE_REPORT to print automatically, CONFIG_CMD_BOOTSTAGE for the bootstage report command.
Elapsed is the interval before each mark, not the time spent inside the named stage. Reading it the other way inverts your conclusions.
The Accumulated section separates driver model probe time in the SPL (dm_spl) from U-Boot proper (dm_r).
The structural bootloader options are a reduced driver set, Falcon mode, and saved DDR training parameters. All trade against update, recovery, or vendor dependency, and all interact with verified boot signing.
The console costs real time: 115200 baud with 8N1 framing is about 11.5 kilobytes per second. Measure with it on to find targets, and again with quiet to report the production number.

Frequently asked questions

Why does dmesg show a boot time much shorter than what I measure with a stopwatch?

Because the kernel's printk timestamp starts at zero when the kernel starts running, not at power-on. Everything the boot ROM, the SPL, and U-Boot did before that point is outside the number. On boards with slow storage or long DDR training, that interval can exceed the whole kernel phase. To see it, timestamp the serial console from outside the board with a tool such as grabserial, enable CONFIG_BOOTSTAGE in U-Boot, and measure the pre-console ROM interval with a scope or a controlled supply.

What exactly does the Elapsed column in bootstage report mean?

It is the time between the previous mark and that mark, not the time spent inside the stage the row is named after. So an Elapsed value on the start_kernel row is the interval from bootm_start to start_kernel. Reading it as "time spent in start_kernel" will point you at the wrong stage. The Accumulated section at the bottom is different: those values are totals for driver model probing in the SPL (dm_spl) and in U-Boot proper (dm_r).

How do I measure the boot ROM, when it prints nothing?

You cannot do it in software, because there is no log until the SPL starts printing. Use hardware: a switched supply or a controlled reset line, and a scope probe on the reset signal and the serial TX line. The interval from reset release to the first serial character is the ROM read plus the SPL load. It does not usually change between software builds, so measure it once per board and per boot medium and carry it as a fixed number.

Does turning off the serial console actually change boot time?

Yes, and usually by more than people expect. A console at 115200 baud with 8N1 framing carries about 11.5 kilobytes per second, so a verbose boot can spend hundreds of milliseconds doing nothing but printing. Keep it on while you are looking for targets, since it is your measurement instrument. Then take a second measurement with quiet on the command line, and report that one, because it is the number the product will actually have.

Kernel & Embedded News: AI-Written Patch Rules; AMD Ryzen X100

Raghu Bharadwaj — Thu, 06 Aug 2026 14:52:51 +0000

This edition tracks how Linux is governed, scheduled, and where its AI now runs. The kernel documentation tree carries new guidelines for tool-generated content, which set expectations for AI-assisted patches. The sched_ext framework is close to supporting a hierarchy of sub-schedulers together with proxy execution. And two moves push inference outward: AMD's Ryzen AI Embedded X100 folds CPU, GPU, and NPU behind one memory space for robots, while Google's LiteRT.js runs on-device models straight in the browser.

The threads this edition are accountability and placement: who is responsible for a machine-generated patch, how a scheduler divides CPU time without losing correctness, and where AI inference runs — on a consolidated robotics SoC, or inside the browser itself. None of these is a finished product on its own, but each shapes how engineering teams will plan work over the next few cycles.

In this edition

The kernel documents its expectations for tool-generated patches. A new Guidelines for Tool-Generated Content page has entered the process documentation, asking contributors to be transparent about tools, including AI, and to defend everything they submit.
sched_ext gains sub-schedulers and proxy execution. Work on enqueue() for sub-schedulers lets a BPF scheduler be attached to a control group, and a long-standing incompatibility with proxy execution is being resolved.
AMD announces the Ryzen AI Embedded X100 for robots. The X100 Series combines up to 16 Zen 5 cores, an integrated GPU, and an NPU on one SoC with unified memory, sampling now with production expected in Q4 2026.
Google's LiteRT.js runs inference in the browser. LiteRT.js executes .tflite models locally through WebGPU, which is relevant to any embedded product whose interface is a web UI.

The kernel now documents how it handles tool-generated patches

The kernel's process documentation has gained a page titled Kernel Guidelines for Tool-Generated Content. It is visible in the current documentation build (7.2.0-rc6), filed under the policy guides and developer statements, next to a separate AI Coding Assistants page. This is guidance, not an enforcement mechanism, and it is deliberately framed around tools in general rather than AI alone.

The scope is drawn carefully. Routine tooling stays out of scope: reformatting with clang-format, rustfmt or Lindent, spelling and grammar fixes, identifier completion, and mechanical changes such as variable renaming. The guidelines apply when a meaningful amount of a contribution was not written by a person in the Signed-off-by chain. The examples named include checkpatch.pl --fix, Coccinelle scripts, a chatbot-generated function, a C file first produced by a coding assistant and then cleaned up by hand, and a changelog written by a generative model.

The core requirement is transparency and accountability. Contributors are asked to state, in the cover letter or changelog, which tools were used, the input given to them, a summary of prompts for longer sessions, which parts of the change were affected, and how the result was tested. The document rests on the existing Developer Certificate of Origin and states the expectation plainly: you must understand and be able to defend everything you submit, and if you cannot, you should not send it. Maintainers keep full discretion, from treating a patch normally to rejecting a machine-generated series without detailed review.

This lands against a sharper backdrop. In a mid-July mailing-list message, Linus Torvalds said the kernel is not an anti-AI project and that contributors who object to others using AI can fork the tree or step away. Separately, in the earlier discussions around AI tagging policy, he observed that a mandatory tagging rule mainly constrains people who already follow rules, while bad actors will not self-identify. Both points are worth keeping in view when reading the guidelines, and both are drawn from secondary coverage rather than primary posts.

For teams, the practical change is to your submission checklist, not your build. If your group already uses Coccinelle, checkpatch.pl --fix or a coding assistant, add a short provenance note to the changelog now, before a maintainer asks. The heavier point is cultural: the burden of understanding a patch stays with the human who signs it, so treat generated code as a draft to be reviewed line by line, not as output to be forwarded. That is the same standard good reviewers already applied; the value here is that it is written down and can be pointed to.

What it means for kernel contributors

If you send patches upstream, fold this into your workflow now. When a tool produced a meaningful part of a change, say so in the changelog or cover letter: name the tool, give the input, and describe what you did with the output. This page mandates no specific tag — plain, honest prose satisfies it — but where AI was involved, pair that prose with the Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2] tag the AI Coding Assistants page recommends, and remember that the same page says an AI agent must never add a Signed-off-by tag: only the human submitter can certify the DCO. For example, in a changelog:

This conversion was produced with the Coccinelle script below, then
reviewed and adjusted by hand. Tested with the rcutorture selftests.

  @@ expression E; @@
  - kfree(E);
  + kfree_sensitive(E);

The reviewer can then judge the change with the right context, and you have met the transparency the document expects.

sched_ext gains sub-schedulers and proxy execution

The extensible scheduler class, sched_ext, lets a custom CPU scheduler be loaded as a set of BPF programs. Recent work, described by Jonathan Corbet, brings the ability to build a hierarchy of sub-schedulers close to completion, and ends a long-standing incompatibility with proxy execution.

A sub-scheduler associates a sched_ext scheduler with a control group, so every process in that group is managed by the attached scheduler. A capability mechanism lets a parent share part of its access downward: a newly attached sub-scheduler starts with access to no CPUs, and its parent then grants it a specific set. The design work in the enqueue() path is about keeping control groups isolated from one another while avoiding needless calls up through the hierarchy, which would otherwise cost performance.

Proxy execution is the other half. It addresses priority inversion by letting a task that holds a lock run on behalf of a higher-priority task that is waiting for it. Making that cooperate with sched_ext matters for realtime and latency-sensitive workloads, which are common in embedded and industrial systems, because a scheduler that cannot handle priority inheritance correctly will produce latency spikes that are hard to diagnose.

Custom schedulers are attractive when a general-purpose policy leaves performance on a mixed workload, where latency-critical and throughput tasks share the same cores. The trade-off is that you now own a scheduler, and BPF scheduling is a skill your team either has or has to build. Sub-scheduler support makes the model more practical for multi-tenant and container hosts, where per-group policies are exactly what you want.

How to try it

You can experiment with sched_ext on a mainline kernel today. Build a kernel with the extensible scheduler class and BPF support enabled:

CONFIG_SCHED_CLASS_EXT=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_DEBUG_INFO_BTF=y

Then build and load one of the in-tree example schedulers:

raghu@techveda.org:~$ make -j -C tools/sched_ext
raghu@techveda.org:~$ sudo ./tools/sched_ext/build/bin/scx_simple

With the scheduler running, /sys/kernel/sched_ext/state reports that a BPF scheduler is active; stop the binary and the kernel reverts to its default scheduler. The scx_simple example runs a global weighted virtual-time policy, and the sched-ext/scx project carries fuller schedulers to read and adapt. Trying a sample scheduler first is the cheapest way to decide whether a custom policy is worth owning.

AMD's Ryzen AI Embedded X100 targets robots and the industrial edge

AMD used its Advancing AI 2026 event to announce the Ryzen AI Embedded X100 Series, a family of embedded processors aimed at what it calls physical AI: robotics, industrial automation, healthcare, and aerospace and defense. Each part combines up to 16 Zen 5 CPU cores, a discrete-class integrated GPU, and an NPU on a single SoC, tied together by a unified memory architecture that AMD says improves determinism and lowers latency. AMD has not published a maximum memory capacity for the series; its own benchmark configurations used 64 GB and 128 GB of LPDDR5x, so treat exact capacity as unconfirmed until the system-on-module partners publish real specifications. The parts are rated for industrial use, with a -40 to 105 C operating range and 24/7 operation for up to ten years.

The number worth attention is not the TOPS rating; it is the unified memory architecture. A typical robot today spreads its compute across a control board, a separate GPU, and an inference accelerator, and moving data between them costs latency and predictability. Placing the CPU, GPU, and NPU behind one memory space removes those cross-device copies, which is what a real-time control loop actually needs.

That raises the harder question for anyone building on it: real-time behavior. A robot has to run deterministic control alongside AI inference, and doing that on an x86 Linux SoC is a systems problem, not a benchmark. AMD's answer is to pair Linux with the Xen hypervisor for mixed-criticality partitioning, so a real-time control domain can run beside the AI and GPU workloads. Teams still have to decide where PREEMPT_RT, CPU isolation, and partitioning fit, because the chip does not remove latency on its own.

The software stack is the other decision. AMD is pushing an open stack — Linux, the ROCm stack for the integrated GPU, and frameworks such as PyTorch and ONNX — with tools to migrate CUDA code to ROCm. That answers NVIDIA's Jetson line and its CUDA ecosystem, but the real test is whether ROCm runs cleanly on this embedded part under the exact kernel you ship, worth confirming on real hardware rather than a reference image.

Two caveats keep this in proportion. AMD's headline comparisons, such as up to 2.1 times the multi-threaded CPU performance of an Intel Core Ultra Series 3 part, are its own projections measured on a proxy processor configured to represent the top X100 model, not independent numbers on final silicon. And the series is not shipping yet: sampling began in June 2026, with production expected in the fourth quarter through system-on-module partners. It is a platform to evaluate now and design around later, not one to build a shipping product on today.

What it means for engineering teams

Even before silicon ships, there is work to scope, and the early decisions are about the software platform. If you are evaluating the X100 for a real-time product, decide whether you run a PREEMPT_RT kernel or partition the machine with a hypervisor, and how you isolate the control cores from the AI and GPU load — the usual levers are boot parameters such as isolcpus and nohz_full plus IRQ affinity. Confirm early that the ROCm stack and its kernel driver build and load against the kernel you intend to ship, not just AMD's reference image. An x86 embedded SoC also moves a team used to Arm device trees toward ACPI-based enumeration, a real BSP and skills consideration to plan for and budget.

Google's LiteRT.js pushes inference into the browser

Google has released LiteRT.js, a JavaScript build of its LiteRT on-device runtime, the runtime formerly known as TensorFlow Lite, that runs .tflite models directly in the browser. It brings the LiteRT C++ runtime to the web through WebAssembly, using XNNPACK on the CPU, ML Drift over WebGPU for the GPU, and an experimental WebNN path for neural accelerators. Google reports up to three times the performance of previous web runtimes on vision and audio models, and much larger gains over its own CPU path when WebGPU is available.

For an embedded audience this matters wherever the product's interface is a browser: kiosks, industrial HMIs, medical and retail front ends, and any device that already ships a web UI. Running inference locally in that interface removes the round trip to a server, which means lower latency, no per-inference cloud cost, and data that stays on the device.

The trade-off is the sandbox. WebGPU gives the browser real GPU compute, but through a portable abstraction, not the native driver and delegate control you get from LiteRT on the device itself. Memory is bounded by the browser, the NPU path through WebNN is still experimental and limited to some browsers, and you give up the deterministic, hardware-specific tuning a native runtime allows. What you gain is portability: one build runs across whatever GPU the host exposes, with no per-target driver work.

The practical read is that browser inference is now fast enough to be a real option for on-device features in an embedded web UI, not only a demonstration. Where you need full control of the accelerator and predictable latency, the native runtime remains the right tool; where portability and privacy matter more, LiteRT.js is worth evaluating. A companion preview, LiteRT-LM.js, extends the idea to language models in the browser.

How to use it

Getting a model running in the browser is a short path. Install the runtime, load the Wasm, then compile a .tflite model against a backend:

raghu@techveda.org:~$ npm install @litertjs/core

import {loadLiteRt, loadAndCompile, Tensor} from '@litertjs/core';

await loadLiteRt('https://cdn.jsdelivr.net/npm/@litertjs/core/wasm/');
const model = await loadAndCompile('/models/vision.tflite', {
  accelerator: 'webgpu',   // 'webgpu' | 'webnn' | 'wasm'
});
const out = await model.run(new Tensor(pixels, [1, 3, 224, 224]));

Operations unsupported on the chosen backend fall back to the CPU automatically, so the same code runs where WebGPU is unavailable. An existing TensorFlow.js pipeline can keep its pre- and post-processing and swap only the model, using the @litertjs/tfjs-interop package. For an embedded HMI, that means one build serves whatever GPU the device's browser exposes.

References

Originally published at techveda.live.

— Raghu Bharadwaj

The Mindset Behind Hard Debugging

Raghu Bharadwaj — Thu, 06 Aug 2026 06:07:53 +0000

Hard debugging is rarely defeated by a lack of tools. It is defeated by three quiet habits: assuming the fault is where the symptom appears, clinging to the first explanation, and hoping a tool will do the thinking. A difficult fault is usually lost to those habits before you read a line of code. The engineers who resolve hard faults are the ones who notice these defaults and replace them with a patient, evidence-first mindset.

Most hard bugs are lost before we touch them, in the attitude we bring to the session.

When something breaks, the average person rushes in with three quiet habits: they assume the fault lives exactly where it shows up, they cling to the first explanation their mind offers, and they hope a tool or a smarter person will tell them what to do next. Those habits feel natural, but on hard faults they are exactly what keep us stuck.

Put two engineers on the same failing board. One finds a way through in an afternoon; the other is still going three days later. The difference is rarely raw intelligence or how many commands they know. It is the mental posture each brings to the work before the first step. Handling a hard debug session is less about knowing every tool and more about managing your own assumptions, reactions, and impatience.

A tough problem is usually lost in your mindset before it is lost in your methods.

Habit one: starting too narrow

The first habit is to fix on the most visible symptom and refuse to look anywhere else. Something breaks, so we stare at the last thing we changed, and we return to it because it is familiar and close at hand. When the answer is not there, we look harder in the same place instead of stepping back.

Here is what that looks like on real hardware. A device keeps dropping off the bus. You are a kernel person, so you open the driver and read it, carefully, for three days: the probe path, the error handling, the power-management callbacks. Every line is correct, and the device still fails. The fault was a layer below the entire time, in a marginal connector and a power rail that sagged under load. Reading C could not localise it, because the problem was not written in C.

The mindset shift is to start wide. A real system is a stack of layers: application, kernel, driver, board, power, and the signals on the wires. What you see on the surface is often just where a deeper fault finally shows itself. This is why David Agans opens Debugging: The 9 Indispensable Rules with the rule that comes before all the others: "Understand the System." If your model of the system stops at the software, the layer that owns the fault is not in your model, and you cannot search a layer your model does not contain. Engineers who handle hard sessions well resist the pull toward the layer they know best; they pause, sketch the whole picture, and choose where to look.

Do not confuse where it hurts with where it started.

Habit two: trusting the first explanation

The second habit is to treat your first explanation as the truth. Within seconds, the mind produces a story: it must be this driver, this flag, this commit. From that moment it collects only the facts that support the story and quietly ignores the rest. You feel as though you are analysing, when you are often defending a guess.

"The first principle is that you must not fool yourself and you are the easiest person to fool." - Richard Feynman

The mindset shift is humility. Treat the first theory as a possibility, not a verdict, and read all of the evidence the system already gave you (the full message, the log, the oops, the register that held the bad value) before you decide what you believe. Agans states this as a rule of its own, "Quit Thinking and Look": observe the actual behaviour before you explain it. And when you do change something, change one thing at a time, so each result answers exactly one question; change several at once and you can no longer tell which change mattered.

Strong debuggers care more about being right in the end than feeling right in the moment.

Habit three: outsourcing the thinking

The third habit is modern and seductive: believing a tool can do the thinking for us. Stuck, we search, paste, prompt, or ask around, and treat the reply as a substitute for understanding. An assistant answers inside whichever layer you named; it never held a model of your board, your load, or your timing, so it cannot tell you that the layer you asked about was the wrong place to look. The reply may be fluent, fast, and confident, and a fluent explanation is still not a tested one.

"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." - Brian Kernighan

Tools are useful, sometimes indispensable, but they are at their best when they test a question you have already framed. A print statement, a trace, an oscilloscope, or an AI prompt can help you inspect; it cannot decide what deserves attention, which pattern matters, or which assumption to challenge next. That part remains yours.

A good tool answers questions. A good mindset decides which questions are worth asking.

The mindset that handles hard faults

These habits are not signs of a weak engineer. They are the natural defaults of any fast mind under pressure, and hard debugging is the practice of noticing those defaults and replacing them with better ones: patience instead of haste, curiosity instead of ego, observation instead of premature certainty. On the next hard fault, run a small, repeatable routine.

Pause before acting and name the layers the failure passes through: application, kernel, driver, board, power, and signal.
Read everything the system is already telling you (the full error, the log, the oops) before you form a theory.
Name your current hypothesis about which layer owns the fault, but stay willing to discard it.
Change one thing at a time, and ask one clear question with each change.
Reach for the one instrument, or the one AI prompt, that tests that single hypothesis, and treat its answer as evidence about one layer, not a verdict on the whole system.
Let evidence, not relief, tell you the fault is solved. A disappearing symptom is not the same as a fixed cause.

Tools will keep improving, and an assistant will keep producing quicker and more plausible answers. None of that removes the work at the centre of every hard session, which is human judgement: deciding what to believe, what to doubt, and what to test next. That judgement is what turns a chaotic failure into a manageable investigation, and it is the habit we build deliberately in TECH VEDA training: understand the system first, read the evidence before you commit to a theory, and use tools only to confirm a conclusion you have already reasoned your way toward. It is also what makes an engineer trustworthy when the easy answers fail.

- Raghu Bharadwaj

The Kernel Maintainer Bottleneck and What It Means for Upstreaming

Raghu Bharadwaj — Thu, 06 Aug 2026 04:18:56 +0000

Contributions to the Linux kernel are at record highs, but the number of people who review and merge those contributions has not grown at the same rate. This is the kernel maintainer bottleneck: the constraint has moved from writing code to getting it reviewed and accepted. For embedded teams sending drivers and board support upstream, it means longer lead times and a real advantage for patches that are easy to review.

If you have sent a patch to a kernel subsystem and watched it sit on the mailing list for weeks, you have met the kernel maintainer bottleneck first hand. The problem is usually not that your code is wrong. It is that the small number of people who can review and merge it are handling more work than ever. Kernel development in 2026 has plenty of hands writing code. What it has less of is time to review that code and move it toward the mainline.

This is not a complaint about individuals. It is a structural feature of how the kernel is built, and it is now discussed openly by the people who run the project. Understanding it helps you plan upstreaming work realistically instead of being surprised by it.

More code, more contributors, every year

By raw activity, the kernel community looks healthy and is still growing. The 6.18 release, which closed the last full development cycle of 2025, brought in 13,710 commits from 2,134 developers. That developer count is the highest in the kernel's history. The release also drew 333 first-time contributors, just short of the record 335 set a year earlier in 6.12.

Across the full year covered by releases 6.13 through 6.18, the kernel absorbed 80,035 non-merge changesets from 5,275 developers working for 372 known employers. Greg Kroah-Hartman has noted that every single release now features the work of at least 200 first-time contributors. The supply of people who want to write kernel code is not the problem.

Where the kernel maintainer bottleneck actually sits

Writing a patch is the part of the process that scales with the number of contributors. Reviewing and merging that patch does not scale the same way, because it depends on a much smaller and more stable group of people.

The tags recorded on merged commits show this. In 6.18, 53.6% of commits carried a Reviewed-by tag and only 9.0% carried a Tested-by tag, both typical figures for recent releases. The most active reviewers are close to the same names release after release; as the kernel's own statistics put it, the community's most diligent reviewers are seemingly in it for the long term. On the merge side, just over half of all patches reaching the mainline pass through the hands of maintainers employed by only five companies: Meta and Google, Intel and AMD, and Red Hat. When a handful of trusted people gate most of the flow, their available hours determine how fast everyone else's work moves.

💡 Key insight: The limit on kernel throughput is not how much code gets written. It is how much code a small, relatively fixed set of reviewers and maintainers can examine and accept.

A warning from 2018 that reads differently now

This concern is not new. In 2018 Daniel Vetter, a longtime graphics maintainer, studied how often maintainers commit their own patches and used that as a proxy for maintainer capacity. He found that the number was falling relative to the kernel's growth, and that the community was not training new maintainers fast enough to keep up. His extrapolation was blunt: "around the year 2025 large numbers of kernel maintainers will do nothing else than be the bottleneck, preventing everyone else from getting their work merged and not contributing anything of their own."

He also pointed to a double standard that still exists: ordinary contributors must get their patches reviewed, while maintainers can commit their own code with no recorded review. His darker forecast, that the community would implode under its own bureaucratic weight, did not come true. Contributions kept climbing to record levels. But the underlying concern he raised has moved from a contrarian blog post to a mainstream topic that the project now plans around.

💡 Key insight: The kernel did not collapse under maintainer load. Instead, the load became a first-class process problem that the project actively manages, rather than an edge worry that gets dismissed.

Burnout, succession, and more patches than ever

The human side of this is now discussed directly at the yearly Maintainers Summit. At the 2023 summit, Ted Ts'o observed that maintainers end up absorbing every task nobody else wants to take on: patch review, release engineering, testing, and responding to security reports. Those expectations have only grown over time.

The 2025 Maintainers Summit, held in Tokyo on 10 December, devoted a full session to continuity and succession planning. There are now provisions so that no single person is a point of failure: several people can commit to Linus Torvalds's repository, and the stable tree has similar redundancy. If a smooth handover ever became impossible, the plan is for the attendees of the most recent summit to convene and decide the path forward. Torvalds, for his part, has signed a new contract with the Linux Foundation and does not plan to step back soon.

Two recent developments show why the process layer, not the code, is often the part under strain. The bcachefs filesystem was removed from the mainline kernel in 2025 after repeated disagreements over the development process, an episode one participant described as a "perfect storm." And the same 2025 summit opened a discussion toward a policy for machine-learning tools in kernel development, driven in part by a simple worry: tools that generate patches faster add review load to the same limited group of people.

What the kernel maintainer bottleneck means for your team

If your work involves sending drivers, board support, or fixes upstream, the bottleneck has direct, practical consequences. A few habits help you work with it rather than against it.

Budget real lead time for upstreaming. A non-trivial driver or board-support series can take several release cycles to land, not one. Plan product schedules around that reality instead of an optimistic single-cycle merge.
Make patches cheap to review. Small, well-split series, clear commit messages, and included tests move faster because each one costs a maintainer less time and attention.
Become a reviewer, not only a submitter. Review capacity is the scarce resource. A team that helps review other people's work in a subsystem builds the trust and goodwill that gets its own patches merged.
Prefer subsystems with active, multi-person maintainership. Where you have a design choice, a well-staffed subsystem is less likely to stall on one person's availability.
Keep an out-of-tree plan. If a series will not land before you ship, know the cost of carrying it as a downstream patch and have a concrete plan to upstream it later.

These are exactly the working habits we focus on in our Linux Kernel Infrastructure training, where writing reviewable, mainline-ready code is treated as a core skill rather than an afterthought.

💡 Key insight: For an embedded team, the useful response to the bottleneck is to lower the review cost of your own contributions and to help carry the review load, not to wait for the constraint to disappear.

Key takeaways

Kernel contributions are at record highs; review and merge capacity is the real constraint.
Only about 54% of merged commits carry a recorded review, and just over half of all patches pass through maintainers at five companies.
Daniel Vetter predicted a maintainer bottleneck around 2025; the collapse he feared did not happen, but the concern is now a mainstream process topic.
Burnout, succession planning, and a growing volume of machine-generated patches are all active concerns for the people who run the project.
Teams that write reviewable patches, help review, and budget for longer upstreaming timelines get their code merged with less friction.

Frequently asked questions

What is the kernel maintainer bottleneck?
It is the gap between how much code is submitted to the Linux kernel and how much a relatively small group of reviewers and maintainers can review and merge. Writing patches scales with the number of contributors; review and acceptance do not.

Is the Linux kernel running out of maintainers?
No. Every release still draws record numbers of contributors, including at least 200 first-time contributors. The narrower concern is that review and merge work is concentrated in a small, stable group, and training new maintainers has not kept pace with the growth in contributions.

How long does it take to get a patch merged upstream?
It varies by subsystem, but a non-trivial driver or board-support series often takes several release cycles rather than one. Well-split, clearly described, and tested patches move faster because they cost reviewers less time.

How can an embedded team reduce the impact of the bottleneck?
Budget lead time for upstreaming, make patches small and easy to review, contribute review effort back to the subsystem, prefer subsystems with active multi-person maintainership, and keep a plan for carrying code out-of-tree if it will not land before you ship.

Yocto vs Buildroot: Choosing a Build System for Your Product

Raghu Bharadwaj — Wed, 05 Aug 2026 11:29:48 +0000

For most products you plan to ship, maintain, and update across several years and hardware variants, the Yocto Project is the stronger choice, because its layer model, shared-state cache, generated SDK, and on-target package feeds are built for long-lived maintenance. Buildroot is the better choice when the system is small and fixed, the team is small, the schedule is short, and you will replace the whole image to update it. The trade-off is direct: Yocto asks for a steeper learning curve and heavier build infrastructure in exchange for long-term maintainability; Buildroot gives you simplicity and speed now, and asks for more effort as the product grows.

Every embedded Linux product starts with one decision that shapes years of work: which build system produces the root filesystem. In practice the shortlist is two names. Yocto vs Buildroot is the choice almost every team makes, and it is easy to make it for the wrong reason, such as familiarity or a first-week demonstration, rather than the reasons that matter over the product's life.

This article treats it as an architecture decision: when it arises, both options on their merits, a recommendation for the common case, and the conditions under which the other option wins. It matters because the cost of a build system shows up in maintenance, not in the first build.

The context

The decision arises at the start of a new product, or when an existing product outgrows a vendor's reference image and you decide to own the root filesystem yourself. A few forces are in play, and they usually matter more than any single technical feature.

The first is product lifetime and update obligation: a device you will support and security-patch for five to ten years has different needs from a demonstration unit. The second is the number of variants, since one board and one image is a different problem from a family of products that share most of their software. The third is team size and skills, because a build system you cannot staff is a risk regardless of its features. The fourth is your chosen update strategy, since delivering fixes as packages differs from replacing the entire image. Regulation such as the EU Cyber Resilience Act now adds weight to the security and update forces for connected products.

The options: Yocto vs Buildroot

Both tools are mature and actively maintained; neither is a poor choice. They differ in what they optimise for.

Buildroot

Buildroot is built around simplicity. It uses the same Kconfig and make workflow as the kernel itself, so a kernel or embedded engineer can read a configuration and understand the whole system quickly. It integrates more than 2000 packages and produces a root filesystem image and a toolchain from a single configuration. The current stable series is 2026.05.1 (released 15 July 2026), and the long-term support series is 2025.02.x, maintained until March 2028. Since the 2025.02 release, Buildroot's LTS branches are supported for three years.

A defining design choice is that Buildroot does not include a runtime package manager and does not generate binary packages by default. The project documents this deliberately: safely removing a package and its reverse dependencies at runtime is hard, so Buildroot treats the image as a whole, and a configuration change that affects dependencies is handled by a clean rebuild rather than an in-place update.

Strengths: a short path from zero to a working image; a configuration that is easy to read and reason about; fast to learn; small, predictable output. Limits: no on-target package management; configuration changes often mean full rebuilds; large multi-variant product lines and complex, layered customisation are harder to express than in Yocto.

The Yocto Project

The Yocto Project is built around flexibility and long-term maintenance. It uses the BitBake build engine with metadata organised as layers (the reference distribution is Poky, on top of OpenEmbedded). Software is described by recipes, board and vendor support arrives as layers you add rather than patches you carry, and a shared-state cache (sstate) makes later builds incremental. Yocto can generate a cross-development SDK for application teams and can build on-target package feeds in rpm, deb, or ipk format, which makes package-based field updates possible. The current LTS release is Wrynose (6.0), supported until April 2030 (BitBake 2.18); the previous LTS, Scarthgap (5.0), is supported until April 2028. Yocto designates an LTS every two years, each supported for four.

Strengths: the layer model maps directly onto vendor BSPs and product variants; sstate keeps incremental builds practical on large projects; SDK generation and package feeds support long-lived maintenance and field updates; the ecosystem and commercial support are large. Limits: a steeper learning curve; more moving parts to debug; a larger disk, memory, and build-time footprint, especially for the first build and CI.

The difference shows in the first build; the commands below are illustrative, not a tutorial.

# Buildroot: one configuration, one image
raghu@techveda.org:~$ make qemu_arm_vexpress_defconfig
raghu@techveda.org:~$ make menuconfig
raghu@techveda.org:~$ make

# Yocto: environment, layers, then a BitBake target
raghu@techveda.org:~$ source oe-init-build-env
raghu@techveda.org:~$ bitbake core-image-minimal

The decision

For the common product case, choose the Yocto Project. If you will ship in volume, support for years, patch for security, and produce several hardware or feature variants, the layer model and package feeds pay back the steeper learning curve. Vendor board support that arrives as a Yocto layer keeps your changes separate from upstream, which is what a long maintenance life needs. The same reasoning runs through our note on architecting mainline-friendly products: keep your customisation isolated from the code you do not own.

Choose Buildroot when the system is simple and the constraints point the other way. It is the right decision for a single fixed-function image with few variants, a small team that needs to move quickly, a tight schedule where Yocto's learning curve is a real risk, and prototypes or internal tools. It is also a sound choice for deeply constrained devices that you will always update by replacing the whole image and never need on-target packages. In those situations Buildroot is not a compromise; it is the more honest fit for the requirement.

The common mistake is choosing for the wrong reason: do not pick Buildroot only because it demonstrates faster, or Yocto only because a large vendor uses it. Match the tool to the product's lifetime, variant count, update strategy, and the team you have.

Consequences

Whatever you choose, you accept long-term consequences. Naming them in advance is the point of a deliberate decision.

Maintenance and updates. With Yocto, vendor and board changes live in layers and security fixes flow through recipe updates and package feeds. With Buildroot, customisation tends to live as local configuration and patches, and updates usually mean building and deploying a new image. Yocto matches a package-based update strategy; Buildroot matches whole-image replacement.

Security backporting. Both projects run LTS branches, so both give you a maintained base. Yocto's layer separation makes it easier to track which vendor component a fix belongs to, which helps when you must answer for a specific CVE on a specific board.

Hiring and onboarding. Buildroot is quicker to learn, so a new engineer becomes productive sooner. Yocto skills are scarcer and take longer to build, but they are more valued in the market and let a team take on larger products.

Build infrastructure. Yocto needs more disk, memory, and continuous-integration time, and the first build is long; sstate reduces the cost of later builds but not the initial investment. Buildroot's footprint is smaller and more predictable.

Migration. Reversing the Yocto vs Buildroot choice later is expensive, because recipes and Kconfig customisations do not translate directly. That is the strongest argument for choosing carefully now rather than switching under pressure. Boot time and image size, it is worth noting, are mostly a function of what you put in the image, not which tool assembled it.

Key takeaways

Yocto vs Buildroot is an architecture decision about the product's whole life, not a first-week convenience.
Default to the Yocto Project for shippable products with multi-year support, multiple variants, vendor BSPs, or package-based updates.
Choose Buildroot for simple, fixed-function systems, small teams, tight schedules, prototypes, and whole-image update strategies.
Buildroot by design has no runtime package manager; Yocto can build package feeds and an SDK.
Migrating between the two later is costly, so decide on the merits now.

Frequently asked questions

Is Yocto always better than Buildroot?
No. Yocto is the stronger default for long-lived products with multiple variants and package-based updates, but Buildroot is the better fit for simple, fixed-function systems, small teams, tight schedules, and whole-image update strategies.

Can Buildroot update a device with packages in the field?
Not by default. Buildroot does not include a runtime package manager and does not generate binary packages, so the standard update path is to build and deploy a new root filesystem image. If you need on-target package updates, Yocto's rpm, deb, or ipk feeds support that model.

Which current releases should I target?
On Buildroot, the long-term support series 2025.02.x is maintained until March 2028, with 2026.05.1 as the current stable series. On Yocto, the current LTS is Wrynose (6.0), supported until April 2030, and Scarthgap (5.0) is supported until April 2028.

Does the build system decide boot time and image size?
Mostly no. Boot time and image size are determined by what you include in the image and how you configure it. Both Buildroot and Yocto can produce a small, fast-booting system.

Speculative Decoding: Faster On-Device LLMs

Raghu Bharadwaj — Wed, 05 Aug 2026 02:00:45 +0000

Speculative decoding makes a large language model generate text faster on a constrained device without changing what it produces. A small draft model guesses several tokens ahead, and the large model checks all of those guesses in a single pass. Because single-stream decoding is limited by memory bandwidth rather than by arithmetic, checking many tokens in one memory read is close to free, which is why the technique helps most on memory-bound edge hardware.

Running a language model on a device is usually slower than people expect, and not because the processor is short of arithmetic power. On a phone or single-board computer, generating text one token at a time is limited by how fast the model weights move from memory into the compute units. Speculative decoding is a way to raise that ceiling: it produces the same output as ordinary decoding but pays the memory cost less often.

Why on-device decoding is slow

An autoregressive model generates one token per forward pass. At batch size one, producing that single token means reading every weight in the model from memory, multiplying it by a small activation vector, and discarding it. The arithmetic per weight is tiny, so the processor spends most of its time waiting for data. In roofline terms, single-token decoding is memory-bandwidth bound, not compute bound.

The numbers make this concrete. A Jetson Orin Nano 8GB reaches about 68 GB/s over its LPDDR5 bus. A 7-billion-parameter model stored at 4 bits per weight is roughly 4 GB. Reading 4 GB at 68 GB/s takes about 59 milliseconds, so the memory system alone caps decoding at under 20 tokens per second before a single useful multiply is counted. Faster arithmetic units do not move that ceiling, because the ceiling is set by data movement. This is the constraint the technique is designed to attack.

The key idea behind speculative decoding

Much of the text a large model produces is easy to predict. Function words, common phrasing, and the obvious continuation of a sentence do not need the full model to be guessed correctly, so a much smaller model can propose them quickly. The large model is then used only to verify the proposals, not to generate each token from scratch.

Verification is the important part. Speculative decoding is exact, not approximate. The large model runs once over the whole batch of proposals, and a carefully chosen acceptance rule keeps only the prefix that is consistent with what the large model itself would have sampled. The rule is built so that the final output has the same probability distribution as ordinary decoding from the large model. The text you get is statistically identical to running the large model alone; only the speed changes.

How it works, step by step

One decoding step looks like this:

The small draft model generates a short run of candidate tokens, for example eight. This is cheap because the model is small.
The large target model runs a single forward pass over all eight positions at once, so its weights are read from memory only once for the whole group.
The candidates are compared against what the target model would accept, and tokens are kept up to the first point where draft and target disagree.
At the disagreement point the target model's own prediction is used, so at least one correct token is always produced per step, and the process repeats.

If the draft is good, several tokens are accepted per step, and the device produces multiple tokens for the price of one target-model memory read. If it is often wrong, fewer are accepted and the benefit shrinks toward ordinary decoding.

Why the accepted tokens are almost free

The saving comes back to the memory-bandwidth wall. Reading the target model's weights is the expensive step, and that read happens once per forward pass regardless of how many token positions the pass covers. Processing eight positions instead of one turns a set of tiny vector multiplies into slightly larger matrix multiplies, and on memory-bound hardware there is spare arithmetic capacity to absorb that extra work. So the additional positions cost almost nothing beyond the memory read you were already paying for. Every proposed token that survives verification is one you did not fetch the weights again to produce.

What the numbers look like

The original demonstration reported a 2x to 3x wall-clock speed-up on a large encoder-decoder model, with output identical to the standard implementation. The exact figure on your device depends above all on the acceptance rate: the fraction of drafted tokens the target model keeps. A high acceptance rate on predictable text yields larger speed-ups; a low rate on hard, high-entropy text yields little.

Production inference engines now implement this directly. In llama.cpp, you enable it by passing a small draft model alongside the target model:

raghu@techveda.org:~$ ./llama-cli -m qwen2.5-7b-q4.gguf -md qwen2.5-0.5b-q4.gguf -p "Explain what a spinlock is"

Here -m is the target and -md (also --model-draft) is the draft model. The number of tokens proposed per step is tunable with a separate draft-length flag. No retraining or architecture change is required; the draft only needs to share the target's vocabulary so their tokens line up.

Where it breaks down

It is not a universal win. Several conditions can cancel the benefit:

Low acceptance rate. If the draft model rarely agrees with the target, the extra draft and verification work can make generation slower than plain decoding.
Memory pressure. The draft model must stay resident in DRAM alongside the target model and its key-value cache, which may be exactly the footprint a constrained board cannot spare.
Shared-memory contention. On a unified-memory SoC the draft and target share the same processors and LPDDR bus, so a draft that is too large eats into the bandwidth you were trying to save.
Compute-bound regimes. If you are already serving a large batch, or running a small model on a fast accelerator, the target pass is compute bound rather than memory bound, and the "free" verification is no longer free.
Vocabulary mismatch. The draft and target must use compatible tokenizers, limiting which model pairs you can combine.

What this means for embedded and kernel engineers

The lesson generalizes beyond language models. When you profile an on-device inference workload and find the accelerator idle while memory traffic is saturated, the bottleneck is data movement, and the fix is to move less data or to reuse each transfer for more work. This method reuses a single weight read to settle several tokens; the same reasoning guides weight quantization, operator fusion, and how you place tensors across a shared memory hierarchy. On single-batch edge inference, the common case, memory bandwidth is usually the number to design around. These foundations — where the bytes move, and how the memory subsystem and scheduler behave under that load — are comprehensively covered in our training programs.

Key takeaways

Single-stream, on-device decoding is limited by memory bandwidth, not by arithmetic throughput.
Speculative decoding uses a small draft model to propose tokens and the large model to verify them in one pass, producing identical output to ordinary decoding.
Accepted tokens are nearly free because the target model's weights are read from memory once per verification pass.
Reported speed-ups are around 2x to 3x, but the real figure depends on the draft model's acceptance rate.
The technique can lose its advantage under memory pressure, low acceptance, shared-bus contention, or compute-bound serving.

Frequently asked questions

Does speculative decoding change the model's output?
No. The technique is designed to be lossless. The target model verifies every proposed token with an acceptance rule that preserves its original output distribution, so the generated text is statistically identical to standard decoding from the target model.

Why does speculative decoding help most at batch size one?
Single-stream decoding is memory-bandwidth bound: the processor reads the whole weight set to produce one token and spends most of its time waiting on memory. Verifying several proposed tokens in one pass reuses that single memory read, so the extra work is nearly free. At large batch sizes the target pass is already compute bound and the gain shrinks.

What does the draft model cost on a memory-constrained device?
The draft model must also stay resident in DRAM along with the target model and the key-value cache, and on a shared-memory SoC it competes for the same bandwidth. If it is too large, or its predictions are rejected often, it can erase the benefit.

How do I try speculative decoding without changing my model?
Tools such as llama.cpp support it directly. You pass a small draft model with the -md or --model-draft option alongside the target model. No retraining or architecture change is needed, and the draft only needs a compatible vocabulary.

DEV Community: Raghu Bharadwaj

Kernel & Embedded News: Linux 7.2 Final RC; FFmpeg 9.0 Released

In this edition

Linux 7.2 closes with a large final RC, and Torvalds calls it the new normal

What it means for teams planning on 7.2

FFmpeg 9.0 is released: an ABI break and new defaults that affect devices

What to check before you upgrade

A lockdown gap: legacy PCI sysfs files stayed writable since 2019

How to check your device

Qualcomm's QCE crypto driver faces removal for being slower than the CPU

What to do on Qualcomm-based products

ESP32-S31 enters mass production: one chip for Wi-Fi 6, Thread and Bluetooth

Where it fits in a Linux design

Radxa's rCore SoMs bring Snapdragon and Dragonwing to the edge

How to evaluate them

References

Capturing a Kernel Crash Dump with kdump on ARM64

What you need

What kdump does when the kernel panics

Kernel configuration for a kernel crash dump

Reserving memory with crashkernel=

Loading the capture kernel with kexec -p

Triggering a panic and saving the dump

Reading the kernel crash dump

Where kdump fails on embedded boards

Building kdump into a Yocto image

Key takeaways

Frequently asked questions

Further reading

SoC Kernel Support: How to Tell If Yours Has Already Lapsed

The three clocks behind SoC kernel support

Clock one: the upstream end-of-life date

Clock two: is the vendor branch still moving

Clock three: the silicon longevity commitment

What seventeen SoCs actually look like

What a stale branch does not prove

Why these dates matter

What to do before your next SoC selection

Where to find the current numbers

Key takeaways

Frequently asked questions

Further reading

What fork() Actually Costs, and Why the Process-Builder RFC Does Not Fix It Yet

What happened

What we checked, and what it changes

What posix_spawn() already avoids

What this costs on an embedded target

The seccomp question nobody has answered

What the process-builder API does not do yet

What we could not verify

What to do today

What to watch upstream

Frequently asked questions

References

Linux Kernel CVEs: What to Patch by Device (2 – 8 Aug 2026)

The one action: update to the fixed version for your branch

Mobile and automotive

Embedded and IoT

Cloud and datacenter

Medical devices

How to check which Linux kernel CVEs apply to you

Key takeaways

Frequently asked questions

Further reading

Choosing an A/B Update Layout for Your Product

The context

The A/B update layouts

Symmetric root-fs slots

Asymmetric slots

Split (multiple) slots

Symmetric A/B plus a rescue slot

The decision: matching the layout to the product

Consequences

Key takeaways

Frequently asked questions

Further reading

Embedded Linux Boot Time: The Phase You Are Not Measuring

Why boot time work usually goes wrong

The four phases, and the handoffs between them

Phase 1: ROM and SPL