DEV Community: Raghu Bharadwaj

Kernel & Embedded News: Linux 7.3-rc1 Out; NVMe 2.4 Goes Post-Quantum

Raghu Bharadwaj — Tue, 01 Sep 2026 04:50:25 +0000

Linux 7.3-rc1 shipped on 30 August, closing the second-busiest merge window in kernel history at 15,267 commits and removing two ancient filesystems and a set of unmaintained drivers. This edition covers what the close of the window settles: per-device UBI wear-leveling thresholds for NAND devices, sched_ext sub-scheduler support reaching feature complete, Qualcomm's plan to make its Modular acquisition an open, vendor-neutral AI software layer, and the NVMe 2.4 specification's post-quantum and power-control additions.

The common thread this week is consolidation. The kernel's 7.3 picture is now fixed and can be planned against, and two industry moves — one in AI software, one in storage specifications — aim to reduce how much of a product's software stack has to be rebuilt when the hardware underneath it changes.

In this edition

Linux 7.3-rc1 closes the second-busiest merge window ever — 15,267 commits, a near-41-million-line tree, and a wave of removals now locked in for an October release. — plan your test window
UBI gains per-device wear-leveling thresholds — the threshold stops being one compile-time constant for every NAND device in the system. — evaluate for mixed-flash designs
sched_ext sub-scheduler support is feature complete — a root BPF scheduler can now hand a cgroup subtree to a nested scheduler with revocable CPU grants. — long-term watch
Qualcomm bets on an open AI software stack — Modular's Mojo and MAX aim to make accelerator choice a hardware decision rather than a software rewrite. — planning signal
NVMe 2.4 brings post-quantum security and power controls — the specification set published in August reshapes what storage procurement will ask for. — roadmap input

Linux 7.3-rc1 closes the second-busiest merge window ever

Linus Torvalds released 7.3-rc1 on 30 August and closed the merge window. The numbers are unusual: 15,267 commits were pulled in two weeks, a total only the 6.7 cycle has exceeded, and the tree now stands at 40.98 million lines. About a third of the rc1 patch is one item — AMD's DCN6 display register headers and the code for its next graphics generation — which alone pushed the AMD graphics driver directory to 6.52 million lines, roughly 16 percent of the entire kernel.

We covered the first week of this window as it happened: the 32-bit Arm platform deprecations, and the crypto and AF_ALG changes in the 25 August edition. The second week settled the removals. The EFS filesystem, the read-only driver for SGI's pre-XFS IRIX format, is gone after more than twenty years without a maintainer. The freevxfs driver for Veritas VxFS is gone too; Christian Brauner's removal note records one known user in fifteen years. The SGI XP and GRU drivers, the IBM RSA management driver, the IPWireless PCMCIA driver and the old Moxa serial drivers were also dropped. Moxa PCIe multi-port serial boards, widely used in industrial systems, are not orphaned by this: they get a replacement in the new 8250_mxpcie driver.

The removals have a counterpart on the additions side. The window that deleted EFS and freevxfs — about 3,700 lines between them — added FailFS, a 931-line pseudo-filesystem from Christian Brauner in which every operation fails with EOPNOTSUPP. Paired with a new fchroot() system call and an FD_FAILFS_ROOT sentinel, it lets a process shed its filesystem state completely: with its root in FailFS, every path lookup must be anchored at an explicit file descriptor, and absolute paths, absolute symlinks and AT_FDCWD-relative lookups simply fail. For anyone building sandboxed services, that is a cleaner primitive than the chroot-to-empty-directory conventions it replaces, and unprivileged processes can use it under no-new-privileges.

Torvalds attributed part of the volume to AI coding agents, which are both finding bugs and generating patches at a rate maintainers have not seen before; the networking maintainers used their pull request to say plainly that the volume of machine-generated reports and patches has overwhelmed their review capacity. This is now a structural property of kernel development rather than a one-cycle anomaly, and it showed up in this window both as extra fixes and as extra deletions — several of the removed drivers went because they had become permanent targets for automated bug reports nobody could act on.

On the filesystem side, David Sterba's Btrfs pull brought measured wins. Direct I/O moved onto the IOmap bounce buffer, lifting it from roughly half of theoretical throughput to around 95 percent; replacing an XArray with a local LRU list for tracking inhibited extent buffers yielded around 3x in that path; removing an unnecessary one-jiffy delay in the non-SSD mode with multiple logging tasks raised throughput around 5x on a sample workload; and full fsync of files with many extents but no holes gained a similar factor by skipping hole detection. The old free-space cache v1 code is also now disabled by default, completing a transition that began years ago. A six-week stabilisation series follows, with the release expected in the second half of October.

What it means for engineering teams

Treat rc1 as the feature freeze it is. If your product tracks mainline or you rebase a vendor kernel forward once a year, the 7.3 contents are now fixed enough to plan against: check whether any of the removed drivers or filesystems appear in your defconfig, and schedule a boot test of an rc on your reference hardware before October rather than after. Removals of this kind never announce themselves twice — the next signal you get is a build failure.

A concrete check takes minutes. Grep your defconfigs and any fragment files for the removed symbols — EFS_FS, VXFS_FS, the SGI XP and GRU options, IBMASM — and for out-of-tree modules that touch the old Moxa serial drivers, which now map to 8250_mxpcie. Teams running Btrfs on build servers should also plan a before-and-after benchmark of their own workload during the rc series; the direct I/O and fsync changes are the kind that show up in CI wall-clock times.

UBI gains per-device wear-leveling thresholds

UBI keeps an erase counter for every physical eraseblock it manages. When the difference between the highest and lowest counter on a device exceeds a threshold, the wear-leveling worker starts moving data from little-erased blocks to heavily-erased ones, evening out wear across the flash. Until this cycle that threshold was a single compile-time constant, CONFIG_MTD_UBI_WL_THRESHOLD, default 4096, applied to every UBI device in the system. Merged for 7.3 is work by Ran Hongyun of Huawei that makes the threshold configurable per device at attach time, through a new wl_threshold option on the ubi.mtd module parameter, with a value of zero falling back to the compile-time constant.

The kernel's own configuration help has documented the problem with a single value for years: the 4096 default is sized for SLC NAND and NOR with eraseblock endurance of 100,000 cycles or more, while MLC parts with endurance under 10,000 cycles want something like 128 or 256. A system could always pick one value or the other — by recompiling — but never both at once.

Our UBI on raw NAND piece two days before this landed described wear-leveling as one of the two jobs UBI exists to do. This change addresses the case the single constant handled badly: a system carrying two NAND parts with different erase endurance — say an SLC boot flash beside a cheaper MLC data flash. One threshold tuned for the SLC part lets the MLC part accumulate a wear spread that is a large fraction of its whole life before leveling begins; one tuned for the MLC part makes the SLC device churn data it did not need to move, adding erase cycles and I/O load for nothing.

There is a second, quieter benefit for product families. Because the threshold was a Kconfig constant, board variants with different flash parts previously needed either separate kernel builds or a compromise value; a boot-time per-device option lets one kernel image serve every variant, with the difference expressed in the bootloader arguments alongside the rest of the board configuration.

How to evaluate it for your device

This matters at design time, not as a retrofit. If your board carries a single NAND part, the compile-time default remains fine and there is nothing to do. For mixed-flash designs, work out the endurance ratings of each part from the datasheets and set the boot-time threshold per device accordingly once you are on a 7.3-based kernel — the endurance figures in the flash datasheet, not habit, should set the number. The option arrives with 7.3, so for products on 6.12 or 6.18 LTS this is a note for the next platform revision rather than something to backport. One caution: the exact argument syntax should be read from the merged kernel documentation when you adopt it, since attach-time options have accumulated positional fields over the years and getting the field order wrong attaches the device with defaults silently.

sched_ext sub-scheduler support is feature complete

The sched_ext framework lets a BPF program replace the kernel's scheduling policy. Over the last three cycles it has been growing a hierarchy: 7.1 made the dispatch path hierarchical, 7.2 added the supporting infrastructure, and the enqueue path was still pending when we published our hands-on guide to writing a minimal scheduler last month. The 7.3 pull from maintainer Tejun Heo completes that path. Sub-scheduler support is now feature complete: a root BPF scheduler can hand a cgroup subtree to a nested sub-scheduler along with revocable CPU grants, and the sub-scheduler owns all scheduling decisions for its tasks on those CPUs.

The word "completes" is doing precise work here. The 7.1 and 7.2 cycles could delegate only dispatching — a sub-scheduler could choose which queued task ran next, but the parent still controlled how tasks entered the queues, when one task could preempt another, and CPU frequency selection. With 7.3, parent schedulers grant and revoke all of those per-CPU capabilities, and the grants are enforced on every path a scheduler can reach a CPU through, not just the polite ones. The in-tree scx_qmap example now demonstrates the full hierarchy, which gives anyone building on this a working reference.

The robustness work is as interesting as the delegation. A task whose scheduler has no access to the CPUs it needs used to starve until the watchdog ejected the entire scheduler; the kernel now runs such tasks directly on a small bandwidth budget, converting a scheduler-killing failure into bounded degradation. Tasks migrating across a sub-scheduler boundary are now re-homed to the new owner, closing both a wrong-scheduler-scheduling bug and a use-after-free. The abort path became NMI-safe, arena objects now cross the kernel/BPF boundary as typed pointers instead of manually translated untyped ones, and BPF-writable arena memory is validated before the kernel uses it — with explicit synchronisation rules for task slice and vtime writes — closing corruption paths open to buggy or hostile schedulers.

One further detail matters for LTS users: the pull also carries core scheduling fixes that missed 7.2, including one where interleaved core-wide task selections could corrupt each other's state and hang the machine, and those fixes are marked for stable. They will reach the LTS branches through the normal stable process even if you never run a BPF scheduler.

What it means for scheduler work

Hierarchical scheduling is the piece that makes sched_ext credible for mixed systems — an infotainment stack and a telemetry stack on one SoC, each under a scheduler tuned for it, without either being able to damage the other. The failure-containment work matters as much as the delegation: a sub-scheduler bug now degrades its own subtree instead of taking down scheduling machine-wide, which is the property a safety argument or a multi-tenant design actually needs. If you experimented earlier and stepped back, 7.3 is the version where the sub-scheduler API stops moving.

Prerequisites are unchanged: the kernel needs CONFIG_SCHED_CLASS_EXT and a BPF stack with CONFIG_DEBUG_INFO_BTF, the second of which is the one most often missing from custom embedded kernels. Confirm the framework on your kernel before planning anything:

root@rock-5b:~# cat /sys/kernel/sched_ext/state
enabled
root@rock-5b:~# cat /sys/kernel/sched_ext/root/ops
simple

Qualcomm bets on an open AI software stack

Qualcomm completed its acquisition of Modular, the AI software company co-founded by compiler engineer Chris Lattner, in July. Details of the plan emerged last week, and they are worth the attention of anyone who has done an accelerator bring-up. Modular's stack — the Mojo language, now at 1.0 with its compiler and toolchain under the Apache 2.0 licence, and the MAX inference framework — aims to let the same model code run across GPUs, TPUs and custom accelerators from different vendors. On Nvidia hardware it replaces the CUDA math libraries and kernels wholesale, keeping only a small device-management layer.

Qualcomm's own framing, from Rashid Attar, its head of data center engineering, is that a challenger vendor has two problems to solve in sequence: it must deliver differentiated hardware, and then it must make that hardware usable without the customer assigning engineers to port existing workloads — because, in his account, customers say they will assign exactly none. Modular is the attempt to remove the second problem. The reason it is hard is that the incumbent's advantage is no longer just CUDA: industry analysts point out that the software moat now extends across interconnects, data-processing units, libraries and integrated rack-scale systems, so a CUDA translation layer on its own no longer changes the decision.

The central claim is about bring-up cost. In Modular's own account of porting to AMD's MI355X accelerator, two engineers brought the stack up in fourteen days, with 99.9 percent of the code architecture-agnostic; the hardware-specific remainder concerned new BF16 conversion instructions, larger tensor-core tiles and increased shared memory. The result, Modular says, outperformed AMD's optimised vLLM fork by up to 2.2 times on the workloads tested. Those are the vendor's own figures and should be read as such. But the structural argument stands independently: if the portable layer is thick and the per-silicon layer thin, new silicon competes on merit rather than on the size of its software ecosystem — and the engineering effort concentrates on the small part of the stack that actually changed.

The obvious tension is that a neutrality layer is now owned by one of the vendors it arbitrates between. The mechanisms offered are concrete but partial: Mojo's compiler and toolchain sit under Apache 2.0, which Lattner describes as an irrevocable grant that cannot be withdrawn later; Attar describes organisational firewalls under which Modular engineers may see confidential data about competitors' hardware that Qualcomm's own accelerator teams will not; and an industry alliance giving other hardware companies a governance role is planned for later this year. Lattner's own summary of the neutrality question was that there is no perfect answer to it. The honest reading is that neutrality clearly serves Qualcomm while it is the challenger; the test comes if it stops being one.

How to read it as a planning signal

For teams selecting edge or datacenter AI hardware, the question to track is whether the per-silicon bring-up layer really stays thin for parts beyond the large vendors. If it does, smaller NPU and accelerator vendors — which cannot afford to build a CUDA-class software ecosystem of their own — become viable choices, and hardware selection shifts toward performance per watt, availability and price. There is also a caution in the other direction: teams for whom every microsecond matters will keep optimising against one architecture directly, and an abstraction layer that hides hardware complexity can hide hardware advantages too. Watch the alliance's membership and licence terms when it launches; that list will say more than the benchmarks do.

NVMe 2.4 brings post-quantum security and power controls

NVM Express published the NVMe 2.4 specification set on 4 August — earlier than this edition's window, but a development we had not covered and one with a long tail. The release also restructured the specification family itself, splitting it so that individual capabilities can be revised without republishing the whole base document; the stated aim is faster, simpler development of the specifications, which in practice means features will now arrive more continuously rather than in large multi-year revisions.

The 2.4 feature set groups into three themes. Security: post-quantum cryptographic algorithms in the host-to-SSD data path, TLS 1.3 transport security for NVMe over fabrics, and Key-Per-I/O encryption, which allows a different key per individual I/O command rather than per namespace — the granularity multi-tenant systems have been asking for. Manageability: configuration recovery, and a new virtualization architecture that supports live migration of VMs using locally attached SSDs, removing one of the last reasons cloud operators avoided direct-attached NVMe for migratable workloads. Power and sustainability: controller-based performance controls and voltage monitoring, giving the host standardised control over a drive's draw. Updates to Zoned Namespaces, Key-Value, rotational media and endurance-group management are also included.

Two of these matter directly to embedded and edge builders. The power and performance control features are relevant wherever storage sits inside a fixed thermal budget, which describes most fanless industrial and edge-AI boxes — today that constraint is handled with vendor-specific tooling or by over-provisioning the enclosure, and a standard host-side interface changes both. And the post-quantum additions will flow into procurement: regulated buyers are already writing quantum-resistant requirements into tenders, and drives claiming 2.4 compliance will become the easy way to satisfy them.

What it means for device builders

Nothing ships tomorrow; specification revisions reach real drives over a couple of product generations. The near-term action is procurement language: when specifying storage for a product with a seven-to-ten-year life, ask vendors for their NVMe 2.4 roadmap, and treat post-quantum support in the data path as a feature that will be demanded mid-life even if no one is asking today. Remember also that specification features are individually optional — "NVMe 2.4 compliant" on a datasheet does not by itself mean the drive implements Key-Per-I/O or the power controls, so procurement questions should name the specific features, not the revision number. For Linux support of the new features, watch the NVMe driver changes in the next two or three kernel cycles.

References

— Raghu Bharadwaj

If you work on the Linux kernel or embedded Linux and want structured, instructor-led depth on these subsystems, see TECH VEDA's training programs.

Open-Source Device CVEs: What to Patch by Vertical (August 2026)

Raghu Bharadwaj — Tue, 01 Sep 2026 04:48:00 +0000

August 2026 brought 11 open-source device CVEs and advisories worth acting on across three packages: U-Boot, systemd, and BlueZ. Only five of the 11 carry a CVE number, which is the main thing to take from this month. None is in the CISA Known Exploited Vulnerabilities catalog. Proof-of-concept code is published for two, and we found no evidence of exploitation in the wild. Most have a fix upstream, but several BlueZ fixes exist only as commits and two have no fix at all, so the action is to take your distribution's patched package or cherry-pick, then rebuild your image.

This is the device stack beyond the kernel. It is the bootloader, the C library, the init system, the TLS and networking code, the media pipeline, and the container runtime that sit inside a shipped image. Which of these open-source device CVEs matter to you depends on what is actually in your image and your software bill of materials (SBOM).

The EU Cyber Resilience Act creates software-inventory and vulnerability-handling duties that make SBOM-style tracking operationally necessary, so the SBOM is the list you check this report against. This month the pressure was on the Bluetooth stack. BlueZ published nine advisories in August, more than the rest of our watchlist combined, and most of them carry no CVE number.

A note on coverage. This edition swept the bootloader, C library, init and Bluetooth parts of the stack in full. We did not complete a sweep of the TLS and crypto libraries, the remaining connectivity packages, the remote-access tools, the parsing libraries, the media pipeline, or the container runtime this month. Those are not reported as clear; they are unexamined, and we would rather say so than imply a clean result we did not verify.

We report a package CVE only when it meets one of three tests: a CVSS base score of 7.0 or higher, or a place in the CISA KEV catalog, or clear reachability in a normal device build. Severity numbers come from the source named for each item, because upstream projects, NVD enrichment, and downstream scorers such as the CISA ADP do not always score the same issue the same way.

Kernel CVEs are not here; they are covered by our weekly kernel advisory.

Most of this month's device stack work has no CVE number

The main finding this month is not a single vulnerability. It is that a CVE feed would have shown you almost none of this. BlueZ published nine security advisories in August 2026. Seven of them have no CVE identifier at all. The systemd project fixed a root-level udev issue in the same month, also with no CVE. If your process is to scan an SBOM for CVE identifiers and patch what comes back, that process found two BlueZ issues in August. The project published nine.

The systemd case shows the shape of the problem clearly. The scsi_id helper did not sanitise device metadata read from the kernel, so the ID_WWN, ID_WWN_VENDOR_EXTENSION, and ID_WWN_WITH_EXTENSION values could carry an embedded newline. That allowed an attacker to inject a SYSTEMD_WANTS= line and start a unit of their choosing as root. Plugging in a malicious SCSI or iSCSI device is enough to trigger it, because udev runs at boot as root and parses whatever the device reports.

It was fixed in systemd 261, 260.2, 259.6, and 258.8, and published as advisory GHSA-m8q3-73v4-wvg7. No CVE was assigned, so no CVE-based scan will ever raise it.

This matters for the Cyber Resilience Act specifically. The duty is to know which known vulnerabilities affect your product. A known vulnerability with a published upstream advisory and a published fix is a known vulnerability whether or not anyone minted an identifier for it.

Tracking the advisory feeds of the packages in your SBOM, alongside the CVE feed, is the practical way to meet that. This month is a good argument for doing so: on the CVE feed alone you would have seen roughly a fifth of what BlueZ actually told its users.

Open-source device CVEs to update this month

Package	CVE(s) / advisory	Bug type	Fixed version	Verticals
`U-Boot`	`CVE-2025-70290`, `CVE-2025-70293`	Integer overflows in ZFS and ext4 filesystem parsing at boot. See the note on scoring below	`2026.04`. Recommended baseline `2026.07`	Embedded/IoT, Mobile/Automotive, Medical
`systemd` (`systemd-oomd`)	`CVE-2026-15059`	Local denial of service. A world-writable varlink socket lets any local user make the privileged daemon kill a chosen process (5.5, systemd CNA)	`261`, `260.3`, `259.7`, `258.9`	Cloud/Datacenter, Embedded/IoT, Medical
`systemd` (`udev`)	No CVE. `GHSA-m8q3-73v4-wvg7`	Unit activation as root. Unsanitised SCSI metadata allows `SYSTEMD_WANTS=` injection. Needs physical port access (6.4, systemd CNA)	`261`, `260.2`, `259.6`, `258.8`	Mobile/Automotive, Embedded/IoT, Medical
`BlueZ` (radio range)	`CVE-2026-80186`, `CVE-2026-75032`, and `GHSA-9683-2chf-hfw9` (no CVE)	Stack overflow in `name2utf8` via a crafted discovery packet, needing no pairing (7.2). Out-of-bounds read in AVRCP parsing, needing a paired device (6.3). Stack overflow in the LE Audio BASE parser, pre-authentication and over the air (low)	No release. Commit `381b5d0d`; commits `bd898962` and `58088149`. The LE Audio issue is unfixed	Mobile/Automotive, Embedded/IoT, Medical
`BlueZ` (local, via D-Bus)	No CVEs. `GHSA-4p57-mrcv-r2jc`, `GHSA-75v6-6q44-57hc`, `GHSA-rfcf-9fw3-m9qp`, `GHSA-hhgc-hfgf-8m4x`	Two `sdp-xml` issues via `RegisterProfile`: quadratic parsing denial of service and a heap leak. Two advertisement monitor heap overflows. All need a local caller on the system bus	No release. Commits `308e3536`, `ae6c543e`, `14347c86`	Mobile/Automotive, Embedded/IoT, Medical

A note on the U-Boot scoring. CVE-2025-70290 carries a 9.8, but that score comes only from the CISA ADP auto-scorer, which treated it as network-reachable. There is no NVD primary analysis and no score from the project. The bug parses a filesystem image from local media, so the realistic reading is code execution before the operating system starts, triggered by attacker-supplied storage, rather than anything remote. CVE-2025-70293 has no score from any scorer and is listed here on reachability alone.

Two further BlueZ advisories are outside the table. GHSA-7c3p-mgcr-hf6w, an out-of-bounds access in amp_dump_chanlist, is the one August issue with a real tagged fix: it affects 5.79 and earlier and is fixed in 5.80, so most current images already have it. GHSA-7wjj-8mrm-jhw4, a heap buffer overflow in the BASS broadcast source builder, affects 5.76 and earlier and has no fix, but devices on a recent BlueZ are already past the affected range.

Packages checked this month with nothing that met the inclusion tests: Trusted Firmware-A, GRUB2, musl, and D-Bus. The glibc project published two issues in August, both scored below 7.0 and neither reachable in a normal device build, so they are not listed.

Watch item. BlueZ CVE-2026-80185 is a type confusion in sdp-xml.c, reached through RegisterProfile with crafted nested ServiceRecord data. A local caller who can reach org.bluez.ProfileManager1.RegisterProfile on the system bus can crash bluetoothd with it. The advisory records no released fix and gives the fixed version as unknown, so there is nothing to update to yet. Until there is, you can restrict untrusted local access to org.bluez on the system bus.

One caution on that advisory, GHSA-7mmr-gwqx-vc34. Its title says the issue "leads to arbitrary code execution as root", but its own body says the opposite: "the available evidence establishes local denial of service only", and it is scored 5.5 with no confidentiality or integrity impact. The advisory also states it was generated using AI and asks readers to review it before use. The body and the score are the parts to trust here. If you triage from advisory titles, this one would have sent you after the wrong thing.

Order the work by reachability rather than by score, because the scores this month are unreliable in both directions. Take the two BlueZ issues that need no pairing first: CVE-2026-80186 at 7.2 with a published proof of concept, and the LE Audio BASE parser overflow, which its advisory rates low but describes as pre-authentication and over the air, and which has no fix.

Then the systemd udev issue, if your device has an accessible physical port. Then the U-Boot filesystem issues, if your boot flow reads media a user can replace. The local BlueZ and systemd-oomd issues come last on a single-user embedded device, and considerably higher on a shared or multi-tenant host.

Mobile and automotive

On automotive ARM SoCs that actually ship U-Boot, it runs first in the boot chain. Many platforms instead use a vendor fork, a TF-A chain, or a proprietary first stage, so this applies only where U-Boot is in the image.

The August issues are in filesystem parsing rather than the network path. CVE-2025-70290 is an integer overflow in ZFS metadata parsing and CVE-2025-70293 is an integer overflow in the ext4 block group descriptor table. Both are reached when the bootloader reads a filesystem image that an attacker controls, which in practice means removable media or a service partition.

Before treating either as urgent, confirm that the affected filesystem support is enabled in your U-Boot configuration and that the boot flow actually reads content an attacker could influence. The fixes first landed upstream in 2026.04. We recommend 2026.07 or later as the practical baseline, because it clears these filesystem issues and the July network-path issues together.

Bluetooth is the other exposed surface in a vehicle, because the head unit pairs with phones the manufacturer does not control. Three of the August BlueZ issues are reachable from a device within radio range, and they differ in what the attacker must first achieve.

CVE-2026-80186 is a stack buffer overflow in name2utf8 triggered by a crafted Extended Inquiry Response packet during discovery, so it needs no pairing at all. The LE Audio BASE parser overflow in GHSA-9683-2chf-hfw9 is described by its advisory as pre-authentication and over the air, so it also needs no pairing, and it has no fix. CVE-2026-75032, the AVRCP out-of-bounds read, does need a paired device. Radio range is a real constraint, but a car in a car park is within range of anyone standing near it.

There is no BlueZ release to move to. Every August fix exists as a commit rather than a tagged version, and two of the issues have no fix at all. The practical options are your distribution's patched package, which for Debian covers only the AVRCP issue in 5.87-2, or cherry-picking the commits listed in the table. Note also that one of the advertisement monitor overflows affects 5.87, the newest release, so moving to the latest tag does not by itself clear the month.

The infotainment media pipeline is normally the largest surface in a vehicle, because FFmpeg, GStreamer and a browser engine all decode untrusted input. We did not sweep those packages this month, so treat their absence here as unexamined rather than clear.

Embedded and IoT

On small devices the bootloader and the network client are the parts that touch untrusted input. Update U-Boot to 2026.07 if the device mounts a filesystem the user can replace, for example an SD card or a USB stick used for field updates. Minimal images that use BusyBox init or a cut-down udev replacement are not affected by the systemd issues at all.

If your image does run a full systemd, take that update too. The udev scsi_id issue is most relevant on this class of device, because they often expose a physical USB port and run udev as root at boot.

If your device speaks Bluetooth, the BlueZ items above apply here as much as they do in a vehicle, and small devices are often the ones running an older BlueZ that no one has moved in years. Check which version is actually in your image. The rest of the small-device set, including BusyBox, Dropbear, the embedded TLS libraries and the compression and parsing libraries, was not swept this month.

Cloud and datacenter

The systemd-oomd issue is the one that matters most on multi-tenant hosts. Its varlink socket is world-writable at mode 0666, so any local unprivileged account can ask the privileged daemon to kill a process it chooses, and a working proof of concept is published. On a shared host that is a denial-of-service primitive against other tenants' workloads. Update systemd to 261, 260.3, 259.7, or 258.9.

If you cannot take the update immediately, a drop-in setting SocketMode=0600 on the socket unit closes it, and systemd-oomd is not enabled on many minimal server images in the first place.

The four local BlueZ issues also belong here rather than in the field, because they need a caller on the system bus. On a host where untrusted workloads share a machine, narrow the D-Bus policy for org.bluez if the service is present at all; on most servers it should not be. The container runtime is usually the highest-severity item in this section, and it was not swept this month, so check containerd and runc separately.

Medical devices

Connected medical devices draw their packages from the other three stacks, so the risk is whichever of these is network-reachable or parses untrusted clinical data. Bluetooth is the clearest example this month. Patient monitors, wearables, glucose meters and hospital gateways commonly use BlueZ to pair with sensors and phones, and the radio-reachable issues above need only a device within range. In a ward or a waiting room that is a low bar.

Weigh CVE-2026-80186 first, because a crafted discovery packet reaches it without a completed pairing, and a proof of concept is published. The LE Audio BASE parser issue deserves attention too, because it is also pre-authentication and currently has no fix, which makes it a candidate for compensating controls rather than patching.

The absence of CVE numbers is a particular problem in this sector. Medical device software inventories and regulatory submissions are frequently organised around CVE identifiers. Seven of the nine BlueZ advisories this month have none, so a device file that tracks CVEs alone will not show them, and a supplier questionnaire asking "which CVEs affect this device" will get an answer that is technically accurate and materially incomplete. Ask suppliers about upstream advisories, not only CVEs.

The packages that parse clinical data and medical video, including SQLite, expat, libxml2, FFmpeg and GStreamer, were not swept this month. On a connected medical device those are usually where the untrusted DICOM, HL7 and FHIR content lands.

Of the confirmed items, U-Boot applies to devices that mount a replaceable filesystem during service, and the systemd udev issue applies to any device with an accessible physical port, which includes bedside and cart-mounted equipment. Medical-device updates go through validation and regulatory change control, including US FDA postmarket cybersecurity guidance, EU MDR, and IEC 62304, so the action is to schedule these updates through the device manufacturer's established change-control process rather than patching in place.

How to check if it applies to you

Three questions decide whether any row above is your problem. First, is the package in your image and your SBOM at all? If it is not built into the product, its CVEs are not yours. Second, is the vulnerable feature actually built or enabled? The U-Boot issues need the relevant filesystem support enabled and a boot flow that reads attacker-influenced content; the systemd-oomd issue needs that daemon to be running.

Third, is your installed version below the fixed version? Distributions often backport a fix without matching the upstream version number, so a vendor package that looks older may already carry the patch. Check your distribution's changelog before assuming you are exposed. For most of the BlueZ items this question is harder than usual, because there is no fixed version to compare against, only a commit.

raghu@techveda.org:~$ systemctl --version
systemd 259 (259.4-1)
raghu@techveda.org:~$ systemctl is-active systemd-oomd
active

Here the device reports systemd 259.4 and confirms that systemd-oomd is running. 259.4 is below the fixed 259.7 on that branch, and the affected daemon is active, so the update applies. For the bootloader, read the version from your build manifest or the boot log rather than from the running system. For libraries without a version command, read the version from your SBOM and compare it to the fixed version in the table.

Key takeaways

Most of this month's device stack work carries no CVE number. BlueZ published nine advisories in August and seven have no CVE; the systemd udev fix has none either. A CVE-only process saw a fraction of it.
BlueZ has no release to move to. Every August fix is a commit rather than a tagged version, two issues have no fix at all, and one advertisement monitor overflow affects 5.87, the newest release. Moving to the latest tag does not clear the month.
The two issues that need no pairing are the ones to weigh first: the name2utf8 overflow (CVE-2026-80186, 7.2, proof of concept published) and the LE Audio BASE parser overflow, which is pre-authentication and unfixed.
The U-Boot ZFS score of 9.8 comes from the CISA ADP auto-scorer and treats a local media-parsing bug as network-reachable. Take the update, but do not let that number set your priority order.
Nothing in this roundup is in the CISA KEV catalog. We checked the whole catalogue against the whole watchlist; only OpenSSL and WebKitGTK appear at all, for historic entries from 2014 and 2019.
Advisory titles are not a triage signal. GHSA-7mmr-gwqx-vc34 is titled "arbitrary code execution as root" and its own body concludes local denial of service only.
This edition swept the bootloader, C library, init and Bluetooth stack only. The TLS libraries, remote-access tools, parsing libraries, media pipeline and container runtime were not examined, so treat them as unchecked rather than clear.
Map each CVE to your SBOM, confirm the feature is built and your version is below the fix, then update the package and rebuild the image.

Kernel CVEs Per Release Have Tripled. The Bugs Have Not.

Raghu Bharadwaj — Mon, 31 Aug 2026 13:53:33 +0000

A chart published on 28 August shows kernel CVEs per release climbing from roughly 500 to beyond 1,500 since Linux 7.0. Read carelessly it says the kernel is getting less safe. The stable maintainer confirmed in the same thread that the axis counts CVEs fixed, not introduced, which makes the rise a measure of how much better the tooling has become at finding old bugs. The number that should worry an embedded team is a different one he mentioned in passing: the release that introduced the most CVEs is 5.15, a longterm kernel still shipping in products today.

A kernel CVE chart went around at the end of last week and was widely read as bad news. The shape is genuinely dramatic — a long flat stretch at about 500 CVEs per release, then a step up past a thousand, then past fifteen hundred, with the next release plausibly breaking two thousand. If you ship a Linux product and someone forwarded you that chart, the question you were asked was probably some version of "is the kernel falling apart?" The answer is no, and the reason is in the axis label rather than the curve.

What the chart actually measures

The chart came from Greg Kroah-Hartman, who runs the stable trees and the kernel's CVE assignment team, as a teaser for a talk he is giving at Kernel Recipes in Paris in September. It plots CVEs per release from 6.9 through 7.2. Reading the plot: the 6.9 to 6.19 range sits at roughly 500 per release, every release from 7.0 onward is above one thousand, and 7.2 passes fifteen hundred.

In the replies, someone asked the question that decides what the chart means — is this CVEs fixed, or CVEs found in newly added code? The answer was one word: "Fixed."

That single word inverts the story. A rising count of fixed vulnerabilities in a codebase of roughly 40 million lines is not evidence that the code is getting worse. It is evidence that more of what was already there is being found. A second reply in the same thread made the limitation explicit: the chart shows discovery and fixing, but not how long each flaw sat in the tree being exploitable. Greg Kroah-Hartman also confirmed that longterm kernels are where these fixes actually land, and that the chart does not attempt to show that, because kernel releases form a graph rather than a single line in time.

His own explanation for the jump was dry rather than alarmed — that it looks as though some tools recently got better at finding bugs.

The corroborating numbers

A chart on a slide is one data point, and a kernel CVE story should not rest on one. The independent development statistics for 7.2 show the same inflection from a completely different direction, counting commits rather than CVE identifiers.

The 7.2 release contains 4,830 commits carrying Fixes tags, meaning commits that repair a bug introduced by an earlier commit. Plotted across releases, the count of commits with Fixes tags is roughly level for many releases and then rises sharply at the right-hand edge — the same shape, from a measurement that has nothing to do with CVE assignment. The analysis attributes it to the same cause: large numbers of people pointing language models at the task of finding bugs in existing software.

That analysis also carries a caution worth repeating. There is a crossover point around 5.9 where the kernel appears to start fixing more bugs than it introduces, but that conclusion is not stable, because it takes years for the bugs introduced in any given release to be found. Every year that passes moves the line.

One more figure sets the scale of what is coming. Around 14,000 non-merge changesets were queued in linux-next for 7.3, and more than 1,100 of them carry an Assisted-by tag. That is a lower bound on machine involvement, since it counts only the contributors who declared it.

Why the kernel CVE count is high by policy

There is a second reason the numbers look extreme, and it has nothing to do with tooling. It is written into the kernel's own CVE policy, and most people quoting CVE counts have not read it.

The kernel became its own CVE Numbering Authority so that assignment would be controlled by the people who understand the code. The policy document is blunt about the consequence. Because the kernel sits at a layer where almost any bug might be exploitable, and because exploitability is usually not evident at the time a bug is fixed, the assignment team is deliberately over-cautious and assigns a CVE number to any bugfix it identifies — which, in the document's own words, explains the seemingly large number of CVEs the kernel team issues.

Two further rules in that policy matter for anyone doing kernel CVE triage on a product. No CVE is assigned for an issue found in a kernel version that the stable team is not actively supporting, so a count of assigned CVEs tells you nothing about a kernel that fell off the supported list. And the policy states plainly that applicability is the user's problem, not the assignment team's.

The number that actually matters for embedded products

Buried in a reply, and absent from the coverage, is the figure that should change what an embedded team does this week. Asked for a chart of CVEs introduced per release rather than fixed, Greg Kroah-Hartman declined to publish one before the talk but gave away the result: 5.15 leads as the release that created the most CVEs, and he offered no explanation for why.

Take that seriously for a moment. 5.15 is not a historical curiosity. It is one of the longterm kernels listed as maintained on kernel.org, and it is still the base for a large number of shipping embedded products, because vendor board support packages have long lifetimes and 5.15 was the default choice for several silicon generations.

Two qualifications keep that honest. First, "most CVEs created" is partly a function of how much attention a release has received and how long it has been available to analyse, so it is not a clean quality ranking. Second, those bugs are fixed in the 5.15 stable series as they are found — that is exactly what the stable process does. The conclusion is not that 5.15 is unsafe. It is that a 5.15 product which is not tracking the stable series closely is accumulating known, fixed-upstream defects faster than a product on any other longterm base.

What the published CVE data does and does not tell you

Published kernel CVE counts are a different measurement again, on a different denominator, and mixing them with per-release counts produces nonsense. Counting by publication date from the national vulnerability database rather than by kernel release, 2026 has 5,061 kernel CVEs published so far. The monthly distribution is wildly uneven:

Month, 2026	Kernel CVEs published
January	246
February	220
March	178
April	379
May	1,034
June	514
July	838
August (partial)	1,652

Those swings are mostly publication batching rather than real changes in discovery rate, which is the first reason not to build a metric on this series. Of the 5,061, some 1,033 are still awaiting a severity score, so a fifth of the year's kernel CVEs have no CVSS number at all. And exactly two of them appear in the catalogue of vulnerabilities confirmed as being exploited in the wild.

Hold those two numbers next to each other. Thousands of assigned identifiers, two confirmed exploited. That gap is not a scandal and it is not a reason to ignore CVEs. It is the direct, expected result of a policy that deliberately assigns an identifier to every bugfix that might conceivably have security relevance. A process that treats every kernel CVE as an incident will spend all its time on the wrong ones.

What to do about it on your product

The volume is not going to fall soon, so the practical response is to stop treating the count as the signal. Three things are worth doing this week.

First, find out whether you are even receiving the assignments. They are announced on a dedicated mailing list, and the archive is public, so a team that has been reading vendor advisories has probably been getting a filtered and delayed view:

raghu@techveda.org:~$ uname -r
6.6.155
raghu@techveda.org:~$ curl -s https://lore.kernel.org/linux-cve-announce/ | head

Second, confirm your base kernel is still one the stable team supports, because no kernel CVE will be assigned against a version that is not. A product on an unsupported base does not have fewer vulnerabilities; it has fewer identifiers pointing at them:

root@imx8mp-evk:~# uname -r
5.15.219
root@imx8mp-evk:~# cat /proc/version | cut -c1-60
Linux version 5.15.219 (oe-user@oe-host) (aarch64-poky-l

Check that version against the maintained list on kernel.org rather than against your vendor's release notes. The two disagree more often than teams expect, and the vendor's number is frequently a branch name rather than a supported stable version.

Third, change the question your triage process asks. The official policy already tells you the answer will not come from upstream: applicability is yours to determine, because upstream does not know which subsystems you build. A kernel CVE in a driver you do not compile is not a finding. The tractable version of the work is to derive the set of subsystems your configuration actually enables, and filter against that before anything else:

root@imx8mp-evk:~# zcat /proc/config.gz | grep -c '=y'
1847
root@imx8mp-evk:~# zcat /proc/config.gz | grep -cE '^CONFIG_.*=m'
312

That count is the real scope of your exposure, and it is a small fraction of the tree the CVE numbers are drawn from. The policy also makes the uncomfortable recommendation explicit: it is better to take all released stable changes as a tested whole than to cherry-pick individual fixes, because many problems are only resolved by the sum of several changes, and some relevant fixes never get a CVE at all. A team cherry-picking only CVE-tagged commits is running a strategy the upstream documentation advises against.

What I could not verify

The per-release figures in this article — roughly 500, then above 1,000, then above 1,500 — are readings of a chart. I did not independently recount CVEs per kernel release, and the underlying per-release data has not been published yet; the full set is being held for the talk in September. Treat the shape as well supported, since an independent Fixes-tag analysis produces the same inflection, and treat the individual values as approximate.

I also found no public explanation for why 5.15 leads on CVEs created. The maintainer explicitly said he did not know, and I am not going to supply a theory he declined to offer.

Key takeaways

The chart counts kernel CVEs fixed per release, not introduced. The maintainer confirmed this directly.
An independent count of commits carrying Fixes tags — 4,830 in 7.2 — shows the same sharp rise, so the trend is real even if the exact per-release values are approximate.
The kernel CVE team assigns an identifier to any bugfix that might have security relevance, by policy. High counts are the intended outcome, not a symptom.
5.15 leads as the release that introduced the most CVEs, and it is still a maintained longterm kernel widely used in embedded products.
Of 5,061 kernel CVEs published in 2026 so far, two are confirmed as exploited in the wild and about a fifth have no severity score at all.
No CVE is assigned against an unsupported kernel version, so an old base looks quiet without being safe.
Upstream states that applicability is the user's call and that taking whole stable releases beats cherry-picking individual CVE fixes.

References

I teach Linux kernel, device driver and embedded Linux engineering at TECH VEDA. If your team is working through kernel CVE triage or choosing an LTS base for a product, that is the kind of thing we cover in the Linux kernel and BSP programmes.

UBI on Raw NAND: The Layer That Hides Bad Blocks and Wear

Raghu Bharadwaj — Sun, 30 Aug 2026 08:42:57 +0000

UBI is the volume-management layer between a raw NAND partition and a filesystem such as UBIFS. It hands the layer above it eraseblocks that never go bad and never wear unevenly, and charges for that: one or two NAND pages of every eraseblock for headers, about 2% of the chip reserved for bad blocks, and four further eraseblocks. This article traces those costs, and shows the raw NAND flashing mistake that becomes read errors months after a board ships.

Most engineers meet UBI as three commands in a flashing script somebody else wrote. The script works, the board boots, and nobody looks again until a unit returns from the field with read errors, or the rootfs turns out to be megabytes smaller than the datasheet suggested. Both follow from how UBI behaves on raw NAND.

Why raw NAND cannot be treated as a disk

The kernel's UBIFS documentation states the difference plainly. A block device supports read-a-block and write-a-block; blocks are rewritten in place, do not wear out and do not go bad, because the drive substitutes spares itself. An MTD device supports three operations instead: read at an offset in an eraseblock, write at an offset in an eraseblock, and erase a whole eraseblock. Nothing is rewritten until the eraseblock holding it has been erased, eraseblocks wear out after a bounded number of cycles, and on raw NAND they also go bad.

One mistake follows immediately: putting ext4 on /dev/mtdblock2. It appears to work, because mtdblock emulates a block device by reading an eraseblock, modifying it in memory and writing it back. What it does not emulate is wear levelling or bad-block handling, so every metadata update lands on the same few eraseblocks. The failure is a device that passes every lab test and dies in the field.

UBI removes both problems. The MTD project describes it as a volume management system for raw flash, comparable to LVM except that it maps logical eraseblocks onto physical ones and adds wear levelling and error handling. It is not a filesystem, and explicitly not a Flash Translation Layer.

UBIFS  (or squashfs on ubiblock, read-only)
  |
UBI    volumes: rootfs, data, ...      drivers/mtd/ubi/
  |
MTD    /dev/mtd2, eraseblocks + pages  drivers/mtd/
  |
raw NAND chip

The two layers are configured separately and can disagree. UBI is told the physical eraseblock size and minimum I/O size of the raw NAND; UBIFS is told the logical eraseblock size UBI presents. Get the second wrong and the filesystem will not mount, reporting a size mismatch rather than the mistake behind it.

The two headers, and why your eraseblocks shrank

UBI writes two 64-byte headers at the start of every physical eraseblock (PEB) that is not marked bad. The erase counter header, or EC header, records how many times that eraseblock has been erased. The volume identifier header, or VID header, records which volume it belongs to and which logical eraseblock (LEB) of that volume it holds. Both carry a CRC-32.

Those headers are the whole basis of the design. The EC header lets UBI compare wear across the chip; the VID header means the map need not live anywhere else, because UBI rebuilds it by reading headers. That keeps the on-flash format simple, and it is also why attach time grows with flash size.

The headers occupy space, so a logical eraseblock is smaller than a physical one. The EC header sits at offset 0; the VID header goes at the next minimum I/O unit or sub-page. On raw NAND with sub-page write support, UBI puts both in different sub-pages of the same page, costing one page. Without it the VID header goes in the second page, costing two. On 128 KiB eraseblocks with 2048-byte pages that gives the numbers you will actually see:

Flash	Header cost per PEB	LEB size
NAND, 512-byte sub-pages	2048 bytes (one page)	129024 bytes (126 KiB)
NAND, no sub-page writes	4096 bytes (two pages)	126976 bytes (124 KiB)
NOR, 1-byte minimum I/O	128 bytes	130944 bytes

If the value passed to mkfs.ubifs -e does not match one of these, the image will not mount on the volume it was built for. The gap between 126 KiB and 124 KiB is not a rounding choice; it is whether the controller can write sub-pages. Some controllers read sub-pages but cannot write them, and UBI must then use the two-page layout even though the datasheet advertises sub-pages.

The eraseblocks you never get to use

UBI also withholds whole eraseblocks. Four are fixed: two hold the copies of the volume table, one is reserved for wear levelling, one for the atomic LEB change operation. The rest covers bad blocks, and its size is a Kconfig option. Current mainline drivers/mtd/ubi/Kconfig reads:

config MTD_UBI_BEB_LIMIT
        int "Maximum expected bad eraseblock count per 1024 eraseblocks"
        default 20
        range 0 768

The help text derives 20 from the minimum and maximum valid block counts a NAND datasheet gives for the device's lifetime, which works out at 20 per 1024 for most parts — about 1.9% of the chip. The word chip is doing important work there, and it catches people. The reserve is counted against the whole device, not the partition being attached, because bad blocks are not distributed evenly and one partition could inherit more than its share. So two UBI devices on one chip each reserve the full amount.

Work it through on a 512 MiB SLC part with 128 KiB eraseblocks, 2048-byte pages and 512-byte sub-pages. The chip has 4096 eraseblocks, so the reserve is 20 times 4096 divided by 1024, which is 80 PEBs. Attach one 256 MiB partition of 2048 PEBs and the usable count is 2048 minus 80 minus 4, giving 1964 LEBs. At an LEB size of 129024 bytes that is 253,403,136 bytes — about 241.7 MiB from a nominal 256 MiB, a loss near 14.3 MiB or 5.6%.

Split that chip into two UBI devices and each reserves 80 PEBs, so 160 go to bad-block handling instead of 80: 10 MiB for no benefit. The MTD documentation states the conclusion directly. On one raw NAND chip use one UBI device with several volumes. Volumes are free; devices are not. The same arithmetic prices a controller that cannot write sub-pages, at a further 3.8 MiB here, or roughly 16 MiB per gigabyte.

Wear levelling and scrubbing on raw NAND

Wear levelling works because every PEB carries its erase count in the EC header. UBI rebuilds the erase counter table in RAM at attach time and uses it to decide when data should move. The trigger is a configured threshold:

config MTD_UBI_WL_THRESHOLD
        int "UBI wear-leveling threshold"
        default 4096
        range 2 65536

The parameter is the maximum permitted difference between the highest and lowest erase counter on the device. When the gap is exceeded, UBI moves data out of eraseblocks with low erase counters into ones with high counters. That looks backwards at first. A low erase count means the block holds data that never changes, so moving that cold data into a worn block frees the fresh block for write traffic and parks the worn block under data that will not erase it again.

The default of 4096 suits SLC raw NAND and NOR, with eraseblock lifetimes of 100,000 cycles or more. The help text is explicit that MLC NAND, typically under 10,000 cycles, needs a much smaller threshold, suggesting 128 or 256. If your product ships MLC and nobody revisited this symbol, it is running a policy meant for a different class of flash.

Scrubbing is the other half of the job. Raw NAND accumulates bit flips, which ECC corrects on read. A corrected flip is not an error but it does signal that the eraseblock is degrading, and UBI treats it as a reason to copy that PEB elsewhere and recycle the original. This happens in the background, hidden from the layers above, which is why UBIFS needs no bit-rot handling of its own. That is as far as the documentation goes: I did not read drivers/mtd/ubi/io.c for this article, so I am not quoting internal return codes for the scrub path.

Bad blocks use the same idea with a different trigger. If a write fails, UBI moves the data away and tortures the eraseblock: erase it, confirm it reads back as all 0xFF, then write and verify a series of test patterns. A block that survives returns to service; one that fails, or shows a bit flip during the test, is marked bad. If an erase fails with EIO there is no test — the block is marked bad immediately.

Why attach takes as long as it does

Because the erase counters and the map live only in the headers, UBI must read those headers from every eraseblock to rebuild its tables in RAM. Attaching costs one read per PEB plus a CRC-32 per header, so attach time scales linearly with flash size. The MTD documentation gives measured figures: a 256 MiB OneNAND attaches in under a second, a 1 GiB NAND in about two, and a full scan of a 4 GiB NAND takes several.

Several seconds is a large share of a boot budget, which is why fastmap exists. It stores a checkpoint so attach reads the map instead of rebuilding it, and in mainline it is still experimental and off by default. Its anchor eraseblock must sit within the first 64 PEBs, it keeps a free-PEB pool of 5% of the total, and if the fastmap is corrupt UBI falls back to a full scan — so a failure costs boot time, not data. On a 128 MiB part it solves no problem you have.

The flashing mistake that appears months later

This failure is worth understanding, because it passes every test at the factory. The MTD documentation sets it out step by step. You erase the flash, so every byte is 0xFF. You write a UBI image with a general-purpose tool such as nandwrite. Some eraseblocks in that image end with empty pages, and the tool writes those too, filling them with 0xFF. That makes the controller calculate an ECC code for a page of 0xFF bytes and store it in the out-of-band area — and on many hardware ECC engines that code is not itself all 0xFF. Later, UBIFS uses one of those pages. It writes real data, producing a new ECC code written on top of the old one. Because the page was never erased between the two writes, the out-of-band area now holds a corrupted combination. Nothing fails then. It fails when the page is next read, and the read returns -EBADMSG, which is -74.

The fix is to flash with a tool that understands the format. As the documentation puts it, ubiformat ensures every page is written once and only once after erasure, whereas nandwrite writes some pages twice, once by the flasher and once by UBIFS. It also preserves the existing erase counters, so reflashing a returned device does not discard its wear history.

When the flashing tool cannot be changed — a fixed factory programmer, for instance — the fallback is to set the free-space fixup flag when building the image, with mkfs.ubifs -F. On the first mount the filesystem finds the affected free space and re-erases it before use. It works, and it makes that first boot noticeably slower.

Which image you flash depends on the layer. mkfs.ubifs makes a UBIFS image, which goes into an existing volume with ubiupdatevol. ubinize wraps those into a UBI image, which is what ubiformat writes to the MTD device. Confuse the two and the device will not attach.

Checking raw NAND and UBI on your own board

All of this is observable. On the host, the minimum I/O size and the LEB size are arguments you must get right:

raghu@techveda.org:~$ mkfs.ubifs -q -r rootfs-dir -m 2048 -e 129024 -c 2047 -o ubifs.img
raghu@techveda.org:~$ ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubinize.cfg

The numbers for -m, -p and -s come from the board, so read the geometry rather than trusting a script that hardcodes them:

root@imx6ull-evk:~# mtdinfo /dev/mtd0 -u
mtd0
Name:                           nand
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          4096 (536870912 bytes, 512.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  512 bytes
Bad blocks are allowed:         true

Then attach and read back what UBI made of it. The logical eraseblock size printed here is the value mkfs.ubifs -e needs:

root@imx6ull-evk:~# ubiattach --mtdn 0
root@imx6ull-evk:~# ubinfo -a
ubi0
Volumes count:                           1
Logical eraseblock size:                 129024 bytes, 126.0 KiB
Total amount of logical eraseblocks:     4012
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       6
Count of reserved physical eraseblocks:  80

Read those last two lines together. Six eraseblocks are bad; eighty are reserved. The reservation is not a measurement of damage, it is a budget for damage that has not happened yet, and a fleet whose bad count is climbing toward it is worth investigating.

root@imx6ull-evk:~# zcat /proc/config.gz | grep MTD_UBI
CONFIG_MTD_UBI=y
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
CONFIG_MTD_UBI_BLOCK=y
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set

Three checks are worth running on any product that uses raw NAND. If the flash is MLC and the threshold is still 4096, that is a real problem, not a cosmetic one. If the chip carries more than one UBI device, each pays the full bad-block reserve. And if the factory process uses anything other than ubiformat, change it or build images with mkfs.ubifs -F.

Reading flash geometry off the board and matching the image to it is part of the bring-up work covered in TECH VEDA's Embedded Linux with Yocto training.

Where this does not apply

The MTD project puts the following at the top of its UBI page. UBI works with bare flash. It does not work with managed consumer flash: eMMC, SD, microSD, CompactFlash or USB drives. Those carry a Flash Translation Layer in the controller that already performs wear levelling and bad-block substitution and presents a block device. Stacking UBI on one would be two remapping layers competing for the same job. On managed flash, use a block filesystem and choose it on its own merits — a decision covered in choosing a root filesystem format for embedded Linux.

In the other direction, CONFIG_MTD_UBI_BLOCK exposes a read-only block device over a UBI volume — how a squashfs rootfs runs on raw NAND with UBI still handling bit flips and bad blocks underneath.

Key takeaways

UBI stores the logical-to-physical map, and a per-eraseblock erase count, in two 64-byte headers in every good eraseblock.
Those headers cost one page per eraseblock with sub-page writes and two without: a 126 KiB or 124 KiB LEB on a 128 KiB part.
UBI reserves 20 eraseblocks per 1024 for bad blocks, counted over the whole chip, so use one UBI device with several volumes.
CONFIG_MTD_UBI_WL_THRESHOLD defaults to 4096, which suits SLC raw NAND and NOR. MLC needs a far lower value and nobody adjusts it for you.
Flash UBI images with ubiformat. A general-purpose flasher writes empty pages twice and produces ECC errors long after the device ships.
None of this applies to eMMC or SD, which have their own translation layer.

Frequently asked questions

Why is my UBI volume smaller than the MTD partition?
Three costs are subtracted. Each good eraseblock loses one or two NAND pages to the UBI headers, four eraseblocks are reserved for the volume table, wear levelling and the atomic LEB change operation, and 20 eraseblocks per 1024 of the whole chip are held back for bad blocks. On a 256 MiB partition of a 512 MiB chip that is roughly 14 MiB.

Can I use UBI or UBIFS on eMMC or an SD card?
No. UBI works with raw flash only. Managed flash such as eMMC, SD and USB drives contains its own Flash Translation Layer that already handles wear levelling and bad blocks, and does not expose the eraseblock geometry UBI needs.

Why must I use ubiformat instead of nandwrite?
A general-purpose flasher writes the empty NAND pages at the end of eraseblocks, causing an ECC code to be stored for a page of 0xFF bytes. When the filesystem later writes real data there, a second ECC code is written over the first, and the page returns -EBADMSG when read. Using ubiformat ensures every page is written once after erasure, and it preserves the erase counters.

Linux Kernel CVEs: What to Patch by Device (23 – 29 Aug 2026)

Raghu Bharadwaj — Sat, 29 Aug 2026 05:54:24 +0000

The Linux kernel project published 227 Linux kernel CVEs in the week of 23 to 29 August 2026. That is about five times a normal week, because a full stable cycle landed on 28 August and the records for everything in it were published together. Most are already fixed in current upstream stable releases, but four of the records below name no fix for branches they mark as affected, so the branch update alone does not clear those four. None of the records notes known exploitation, and no public exploit code was found for any of them at the time of writing. The minimum safe upstream release per branch is 5.10.268, 5.15.219, 6.1.186, 6.6.155, 6.12.107, 6.18.48, 7.1.12, 7.2.2, or mainline 7.2.

Last week this series covered six CVEs. This week it is 227. The jump is not a sudden collapse in kernel quality. The CVE IDs show what happened: 21 sit in the CVE-2026-747xx block and 206 in the CVE-2026-805xx to 807xx block, and on 28 August the kernel project cut a new point release on every active branch at once.

Where this advisory quotes a CVSS score, that score is the kernel CNA's own secondary metric; NVD has not finished analysing this batch, so other databases may publish different figures later. Prioritise by reachability and deployment context rather than by CVSS alone.

The one action: update to the fixed version for your branch

raghu@techveda.org:~$ uname -r
6.6.152-lts

Stable branch	Update to at least	Notes
5.10 LTS	5.10.268	`CVE-2026-80590` sets this point release. Four records below name no 5.10 fix.
5.15 LTS	5.15.219	`CVE-2026-80590` sets this. Same four gaps as 5.10.
6.1 LTS	6.1.186	`CVE-2026-80590` sets this. Two of the four gaps apply.
6.6 LTS	6.6.155	`CVE-2026-80590` sets this. One gap: `CVE-2026-74752`.
6.12 LTS	6.12.107	Same as 6.6.
6.18 LTS	6.18.48	Same as 6.6.
7.1 stable	7.1.12	Clears every record discussed below.
7.2 stable	7.2.2	New row this week. Mainline 7.2 shipped 16 August and now has a stable branch.
mainline	7.2	Released 16 August 2026.

All eight stable targets are the newest release of their branch and all eight were cut on 28 August, so on an upstream kernel the check is simple: if you are not on the current release for your branch, you are behind. One record, CVE-2026-80590 (stale GSO state on IPv4 fragments), sets the target on all eight branches by itself and names no mainline fix.

Four records the branch update does not clear

This is the part to read before you close the ticket.

CVE	Introduced	Fixed in	Affected, NO fix named
`CVE-2026-74752` (SCTP, 9.8)	2.6.24	7.1.10, mainline 7.2	5.10, 5.15, 6.1, 6.6, 6.12, 6.18
`CVE-2026-80551` (vfio-ccw, 9.3)	5.3	6.6.154 and later	5.10, 5.15, 6.1
`CVE-2026-74743` (macvlan, 9.8)	2.6.23	6.1.184 and later	5.10, 5.15
`CVE-2026-80635` (wcn36xx)	4.7	6.1.178 and later	5.10, 5.15

Last week's CVE-2026-74582 is also still unfixed on 5.10, 5.15 and 6.1. If you ship an older LTS branch you now have five open items to track separately from the branch bump.

Last week's targets were 5.10.265, 5.15.216, 6.1.183, 6.6.152, 6.12.104, 6.18.45, 7.1.9 and mainline 7.2. Every branch moved up exactly three point releases on 28 August.

On a vendor or BSP kernel the version string will not compare cleanly with the table. A kernel reporting 5.10.110-rk3588 will not become 5.10.268 by any upstream update you apply yourself. Ask your vendor to confirm, and treat a verified backport as equivalent. Until an update is possible, reduce exposure by disabling unused features, restricting the relevant privileges, and documenting residual risk through your product security process.

Mobile and automotive

CVE-2026-80707: CAN J1939 receive buffer was not zeroed

J1939 is the transport protocol on heavy vehicles and agricultural equipment. A new receive session's buffer was allocated but not zeroed, so any part the peer never wrote kept residual kernel memory. The fix zeroes it in j1939_session_fresh_new(). The record states the residual-data problem and the fix; a readback path is our inference, not the record's claim. Affects 4.19+. Gate CONFIG_CAN_J1939. Fixed 5.10.265 → mainline 7.2. Record

CVE-2026-74737: TI AM65 Ethernet used hardware-reserved bits as part of a port index

Only the lower 8 bits of the 16-bit Source Tag carry the MAC port ID; the upper 8 are hardware-reserved and hold an arbitrary value, and the driver used the whole field as an index. The record reports sporadic kernel crashes. The fix masks the tag to its lower 8 bits.

Read the scoring carefully: the record carries CVSS 9.8 with a network attack vector, but its own description attributes the invalid index to hardware-reserved bits rather than attacker-chosen packet content — a reliability defect with a high score rather than a demonstrated remote attack. Gate CONFIG_TI_K3_AM65_CPSW_NUSS. Fixed on all seven stable branches that name a fix, from 5.10.266, plus mainline 7.2. Record

CVE-2026-80722: mac80211 partially validated an S1G TWT setup frame

ieee80211_process_rx_twt_action() only partially validates a received S1G TWT setup frame, so an individual agreement can reach ieee80211_s1g_rx_twt_setup() with twt->length too short for the full params struct. The driver callback and tracepoint then read the complete block. Broadcast agreements are unaffected. Note the S1G qualifier — this is the sub-1GHz path. The affected file has no config option of its own, so it is present wherever mac80211 is built. Affects 5.15+. Gate CONFIG_MAC80211. Fixed 5.15.216 → mainline 7.2. Record

CVE-2026-80721: Bluetooth ISO could leave a dangling connection pointer

Clearing iso_conn::hcon depended on iso_conn_del() happening to drop the last reference. The fix clears conn->hcon explicitly. Isochronous channels carry LE Audio. One of four Bluetooth records this week, with CVE-2026-80692, CVE-2026-80683 and CVE-2026-80666 — all connection-object lifetime issues. Affects 6.0+. Gate CONFIG_BT plus CONFIG_BT_LE. Fixed 6.18.44, 7.1.8, mainline 7.2. Record

Embedded and IoT

CVE-2026-74746: use-after-free in the netfilter flow table

GC could free a flow entry while flow_offload_add() was still inserting it. KASAN reports slab-use-after-free reads and writes. The fix publishes the GC-visible tuple last. Highest priority for anyone running a Linux gateway or router — the trigger is ordinary forwarded traffic once flow offload is enabled. CVSS 9.8, affects 5.13+. Gate CONFIG_NF_FLOW_TABLE. Fixed on all seven stable branches that name a fix, from 5.10.266, plus mainline 7.2. Record

CVE-2026-80603: off-by-one read in the netfilter IRC helper corrupts DCC expectations

parse_dcc() read one byte past the region filled by skb_header_pointer(). The record is explicit that this is not currently an OOB read — the buffer is allocated one byte over — so the real impact is a stale byte consumed by simple_strtoul, producing a wrong DCC IP or port in the conntrack expectation. A firewall induced to open the wrong data connection, not memory corruption. Easy to remove rather than patch. Affects 2.6.12+. Gate CONFIG_NF_CONNTRACK_IRC. Fixed 5.10.261 → mainline 7.2. Record

CVE-2026-80635: out-of-bounds read from a short firmware response in wcn36xx

A shorter-than-expected trigger block-ack response from the Wi-Fi firmware was not rejected, so the parser reads a 22-byte struct past the end. The input is the firmware response, not network traffic. Affects 4.7+. No fix named for 5.10 or 5.15. Gate CONFIG_WCN36XX. Fixed 6.1.178 → mainline 7.2. Record

CVE-2026-80723: out-of-bounds write when the boot device tree declares too many reserved regions

On boot, fdt_scan_reserved_mem() saves each dynamically-placed reserved-memory subnode into a fixed-size local array without bounding the count. This is embedded-specific in a way most kernel CVEs are not: the count comes from whatever FDT the platform hands the kernel at boot, so if your boot chain composes the tree from a base plus overlays, or a carrier board contributes nodes, it is not something you control at build time. Overlays applied later at runtime do not go through this path. Affects 3.15+. Gate CONFIG_OF_RESERVED_MEM. Fixed 6.12.103 → mainline 7.2. Record

Cloud and datacenter

CVE-2026-80551: s390 vfio-ccw double-fetch of the first IDAW

The host reads the first indirect address word to size a buffer, then reads it again while filling it, with nothing guaranteeing the value is the same. A guest that changes it in between makes the host size one buffer and fill another. CVSS 9.3, guest-to-host.

One of nine vfio-ccw records (CVE-2026-80547…80555); all nine carry a CVSS vector with scope marked as changed. CVE-2026-80554 (unbounded recursion processing channel programs) is the other CRITICAL at 9.3. Treat the nine as one unit. Affects 5.3+. No fix named for 5.10, 5.15 or 6.1. Gate CONFIG_VFIO_CCW. Fixed 6.6.154 → mainline 7.2. Record

CVE-2026-80590: stale GSO state left on IPv4 fragments before reassembly

Reassembled packets carried segmentation offload metadata that no longer described them. No CVSS score and no dramatic description, but it is the record that matters most to the table above: fixed on every stable branch, no mainline entry, and its fixed versions are exactly this week's eight targets. Fixed in 5.10.268, 5.15.219, 6.1.186, 6.6.155, 6.12.107, 6.18.48, 7.1.12 and 7.2.2. Record

CVE-2026-74743: macvlan did not inherit headroom from the lower device

The transmit path can underflow reserved skb headroom, which KASAN reports as a slab use-after-free. The precondition decides whether you are affected at all: the record names macsec, IPsec, WireGuard, tunnels, and veth with rx headroom as lower devices requiring extra space. A macvlan over a plain physical NIC is not in scope. CVE-2026-74744 is the identical fix in ipvlan at the same 9.8. Affects 2.6.23+. No fix named for 5.10 or 5.15. Gate CONFIG_MACVLAN. Fixed 6.1.184 → mainline 7.2. Record

CVE-2026-74752: SCTP restored AUTH fields straight from peer-controlled cookie bytes

When cookie authentication is disabled, a COOKIE_ECHO restores fixed-size AUTH fields directly from peer-controlled bytes with no validation. A forged RANDOM length gives OOB reads; a forged HMAC identifier caused a 32-byte write past a zero-length AUTH chunk — which the record describes as a primitive for a local privilege escalation chain, despite the CVSS 9.8 / PR:N score. Affects 2.6.24+. This is the widest gap this week: no fix named for 5.10, 5.15, 6.1, 6.6, 6.12 or 6.18. Gate CONFIG_IP_SCTP. Fixed in 7.1.10 and mainline 7.2 only. Record

CVE-2026-80586: MPTCP processed a second suboption against leftover state

A peer can send a malformed DSS with a wrong size followed by another DSS, or MPC plus data, in the same packet. The malformed suboption is ignored but leaves fields written, so the second is processed against leftover state — inconsistency or reads of uninitialised data. One of four MPTCP records, with CVE-2026-80587 (9.8), CVE-2026-80585 (9.4) and CVE-2026-80588. Affects 5.11+. Gate CONFIG_MPTCP. Fixed 5.10.266 → mainline 7.2. Record

CVE-2026-80557: out-of-bounds read decoding a Ceph watcher list

The record gives an unusually explicit attacker model: a malicious or compromised OSD in a multi-tenant Ceph deployment can trigger this against any kernel client that calls CEPH_OSD_OP_LIST_WATCHERS, with no privileges beyond OSD session establishment. Two more libceph records are the same class — CVE-2026-80558 and CVE-2026-80561, all three at 9.8 — plus CVE-2026-80528 in the Ceph fs client, which needs another mounted ext4 filesystem in play. Affects 3.10+. Gate CONFIG_CEPH_LIB. Fixed 5.10.267 → mainline 7.2. Record

CVE-2026-80713: io_uring dropped per-task restrictions on exec

A task can install io_uring restrictions that apply to every ring it creates. The exec cancellation path called __io_uring_free(), which freed the per-task restriction along with the task context, so any ring created after the exec came up unrestricted. Not memory corruption — a confinement bug: a limit that was applied is silently no longer applied. Affects 5.10+. Gate CONFIG_IO_URING. Fixed 7.1.8, mainline 7.2. Record

Medical devices

Wireless. CVE-2026-80721 and the three other Bluetooth records (80692, 80683, 80666) matter for wearables, patient monitors and hearing devices, LE Audio specifically. CVE-2026-80722 matters on hospital wireless networks, but only where the sub-1GHz S1G path is in use.
Network protocols. DICOM/HL7/FHIR-over-IP devices should review CVE-2026-74752 and CVE-2026-80717, and note the first is unfixed on every LTS branch below 7.1. Devices on TI AM65/Sitara parts should review CVE-2026-74737.
Removable media. A large crafted-image filesystem group: CVE-2026-80672, CVE-2026-80673, CVE-2026-80674 (NTFS), CVE-2026-80598 (NTFS3), CVE-2026-80631 (btrfs), CVE-2026-80656 (HFS+), CVE-2026-80591 (f2fs). Relevant to any device that auto-mounts a USB stick for patient data.

Patching a medical device is not the same as patching a server. The update has to pass validation and regulatory revalidation under US FDA postmarket cybersecurity guidance, the EU Medical Device Regulation (MDR) and IEC 62304. Plan the stable kernel update through the manufacturer's change-control process.

How to check which Linux kernel CVEs apply to you

Version. uname -r against the table. Then read the four-gap list, because on 5.10, 5.15 and 6.1 the branch target is not the whole answer.

Configuration.

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E '^(# )?CONFIG_(MAC80211|BT|NF_FLOW_TABLE|IP_SCTP|MACVLAN|IO_URING)[ =]'
CONFIG_MAC80211=m
CONFIG_BT=m
CONFIG_NF_FLOW_TABLE=m
CONFIG_IP_SCTP=m
CONFIG_MACVLAN=m
CONFIG_IO_URING=y

The pattern is anchored on purpose — a plain grep CONFIG_BT also matches CONFIG_BTRFS_FS. Other symbols worth checking: CONFIG_CAN_J1939, CONFIG_TI_K3_AM65_CPSW_NUSS, CONFIG_BT_LE, CONFIG_NF_CONNTRACK_IRC, CONFIG_WCN36XX, CONFIG_OF_RESERVED_MEM, CONFIG_VFIO_CCW, CONFIG_IPVLAN, CONFIG_MPTCP, CONFIG_CEPH_LIB. Read =m as present-and-loadable, not absent.

Reachability. The records describe code paths, not attacker models, so this ordering is our reading and not a statement from the kernel CVE team. Untrusted guests: the vfio-ccw group. Gateway or router: CVE-2026-74746. Storage client: the libceph group. Radio or bus: CVE-2026-80722, the Bluetooth group, CVE-2026-80707. Then remote protocol records, then local confinement (CVE-2026-80713), then configuration- and image-gated. Cutting across all of it: if you run 5.10, 5.15 or 6.1, the four unfixed records outrank everything, because no update clears them.

Key takeaways

Update to the target release for your branch: 5.10.268, 5.15.219, 6.1.186, 6.6.155, 6.12.107, 6.18.48, 7.1.12, 7.2.2, or mainline 7.2 — all the newest release of their branch, all cut on 28 August.
The branch update does not clear everything. CVE-2026-74752, CVE-2026-80551, CVE-2026-74743 and CVE-2026-80635 each mark a branch affected and name no fix. On 5.10/5.15 all four apply; on 6.1 two; on 6.6/6.12/6.18 one.
Last week's CVE-2026-74582 is also still unfixed on 5.10, 5.15 and 6.1 — five open items on older LTS.
227 is a release cadence, not a collapse in kernel quality.
There is a new 7.2 stable branch; 7.2.2 is the newest stable kernel.
None of the records notes known exploitation, and no public exploit code was found at the time of writing. That is a point-in-time check, not a guarantee.

Inherited a Vendor BSP? Measure These Five Things First

Raghu Bharadwaj — Fri, 28 Aug 2026 04:57:45 +0000

When you inherit a vendor BSP, the useful first move is not to read the code but to measure the distance between what the vendor shipped and what mainline contains, using tools the kernel already gives you. Five measurements — running configuration, mainline delta, externally built modules, the device tree the board actually booted, and the Yocto layer map — take about two days and produce an Inherited BSP Baseline Report, the handover document nobody left you.

At some point in an embedded Linux career, somebody hands you a board and a git repository and leaves the company. The repository builds. The board boots. You now own an inherited vendor BSP you did not write, and within a week somebody will ask how long it would take to move it to a newer kernel.

You cannot answer that yet, but producing a number you can defend is a mechanical exercise: five measurements, run in order, before any other work on the platform. One boundary up front: this maps the Linux side of the platform. U-Boot, TF-A and binary firmware carry their own deltas and are not covered here.

What an unmeasured vendor BSP costs

Here is what happens when nobody runs those five measurements. An engineer takes over an inherited vendor BSP on a 6.6 kernel, reads release notes promising "board support and stability fixes", glances at the driver directories, and estimates six weeks to reach a newer longterm kernel. The estimate goes on a roadmap. Eight months later the work is not finished — not through incompetence, but because the tree held around four hundred non-merge commits, a few dozen touching core memory management rather than board files, and eleven modules built from a source tree nobody could locate. None of that was in the release notes. All of it was discoverable in two days.

The damage is worse than one bad estimate, because the gap repeats. Each vendor release rebases and the delta is never measured, so the debt stays invisible until somebody tries to pay it. Then the SoC vendor ends support, and the number nobody measured becomes the number the schedule depends on.

The five measurements:

Measurement	Question it answers	Risk it exposes
1. Running kernel config	What is the board actually running?	Runtime config has drifted from the repository defconfig
2. Mainline delta	How far is this tree from upstream?	Core-subsystem changes hidden behind a patch count
3. External modules	What code lives outside the tree?	Unknown ownership and maintenance liability
4. Booted device tree	What hardware description did Linux receive?	Bootloader fixups and overlays absent from the source
5. Yocto layer map	What metadata really built the image?	Undocumented overrides and unpinned revisions

1. Recover the running kernel configuration

Be precise about what you are collecting: not the defconfig in the repository, but the configuration of the kernel binary on the board. In an inherited vendor BSP those disagree more often than they agree, because fragments, recipe appends and local builds all mutate the config on its way to the image.

The kernel can carry its own configuration. CONFIG_IKCONFIG embeds the .config into the image; CONFIG_IKCONFIG_PROC exposes it at /proc/config.gz. The distinction catches people: the first can be a module, the second cannot. Seeing CONFIG_IKCONFIG=m does not mean /proc/config.gz will be there.

root@imx8mp-lpddr4-evk:~# zcat /proc/config.gz > /tmp/running.config
root@imx8mp-lpddr4-evk:~# wc -l /tmp/running.config
6314 /tmp/running.config

If it is missing, the configuration may still be embedded. Point the kernel's own extraction script at the binary the board boots:

raghu@techveda.org:~$ scripts/extract-ikconfig /path/to/vmlinux > running.config

Reduce it before comparing. A raw .config holds thousands of symbols nobody chose — they were implied by others — so comparing two raw configs produces a diff nobody reads. make savedefconfig writes a minimised defconfig stripped of everything implied elsewhere, leaving the decisions a human actually made.

raghu@techveda.org:~$ cp running.config .config
raghu@techveda.org:~$ make savedefconfig
raghu@techveda.org:~$ scripts/diffconfig arch/arm64/configs/vendor_defconfig defconfig

The output is a short list rather than a wall of text:

-DEBUG_FS y
 PREEMPT n -> y
+VENDOR_THERMAL_HACK y
+MODULE_SIG_FORCE y

Four lines of real information. Somebody turned off debugfs, switched the preemption model, and added a symbol mainline does not have. Treat that last line as a strong investigation signal rather than proof: it may be a vendor Kconfig addition, a stale fragment no longer consumed by the active build path, or an option renamed upstream. Trace it through Kconfig and the Yocto configuration path before calling it live out-of-tree functionality.

Commit the minimised defconfig into your own repository with today's date in the filename. It is the first written record of what the board runs, and every future argument about "did we change that?" is settled by diffing against it.

2. Measure the distance from mainline

Now size the delta. Add a mainline remote and count commits added on top of the vendor's base:

raghu@techveda.org:~$ git remote add mainline https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
raghu@techveda.org:~$ git fetch mainline --tags
raghu@techveda.org:~$ git log --oneline --no-merges $(git merge-base v6.6 vendor/release)..vendor/release | wc -l

That count is the headline number and on its own close to meaningless. What decides the difficulty of a kernel bump is not how many patches the vendor wrote, but which directories they wrote them in. Ask for that directly:

raghu@techveda.org:~$ git log --format= --name-only $(git merge-base v6.6 vendor/release)..vendor/release | sort | uniq -c | sort -rn | head -20

Read the result as a shape, not a score. Weight sitting in arch/arm64/boot/dts/, drivers/soc/ and a few drivers means an ordinary vendor BSP: large but predictable, because device tree files usually rebase cleanly. Commits in mm/, kernel/sched/ or the DMA core change the character of the job, because those subsystems move internally between releases. As a heuristic rather than a rule, a large device tree delta is often more predictable to rebase than a much smaller delta in core code. Location and dependency depth matter more than commit count.

One limit. This assumes the vendor tree shares history with mainline. Some vendors ship a squashed import with no common ancestor, and git merge-base fails. Then compare trees rather than history — git diff --stat v6.6 vendor/release against the closest tag. Treat that candidate tag as a hypothesis, not a fact: validate it against version metadata, a representative subsystem diff and the vendor's own release context before any estimate rests on it.

3. Inventory the externally built modules

Commits are the visible part of an inherited vendor BSP. The expensive part is usually code that was never in a tree at all, and the kernel gives one signal for it: loading an externally built module sets a taint flag. Bit 12, shown as O, has raw value 4096. Read it for exactly what it is — at least one externally built module has been loaded. It does not enumerate them, say where the code came from, or assign ownership.

root@imx8mp-lpddr4-evk:~# cat /proc/sys/kernel/tainted
4096

To find them, check each loaded module for the intree marker kbuild stamps onto modules built inside the tree. In-tree modules report intree: Y; externally built ones do not carry the field at all.

root@imx8mp-lpddr4-evk:~# for m in $(lsmod | tail -n +2 | cut -d' ' -f1); do echo "$m $(modinfo -F intree $m)"; done

A module without a Y is externally built relative to the running kernel tree. That is all the flag establishes. It does not follow that the vendor wrote it — it may belong to your own product team, a supplier, or an integration nobody documented. Record five things for each: source repository, build path, version, licence and maintainer. However ownership falls, no kernel upgrade carries these modules for you, and because they are invisible in the git history you just counted, they are the part of the estimate engineers consistently forget.

If you cannot say where a module's source lives and who last touched it, you have found the largest risk in the platform — in your first week.

4. Read the device tree the board actually booted

The device tree source in the repository is what the build produced, not necessarily what the kernel saw. The bootloader modifies the blob in transit, commonly inserting memory node sizes, MAC addresses and overlays. Read the real one back out of the running kernel:

root@imx8mp-lpddr4-evk:~# dtc -I dtb -O dts -o /tmp/booted.dts /sys/firmware/fdt

Not every platform exposes /sys/firmware/fdt. Where it is absent, inspect the live tree under /sys/firmware/devicetree/base or /proc/device-tree, and compare semantics rather than expecting decompiled output to match source formatting.

Diff the result against the source. The differences are the bootloader's fixups, and knowing them saves you from a class of confusing bugs where the property you see in the repository is not the property the driver read.

Then ask which devices are still waiting. Deferred probe retries a device whose dependency was not ready at that point in boot, and the driver core exposes the queue:

root@imx8mp-lpddr4-evk:~# cat /sys/kernel/debug/devices_deferred
30a20000.i2c:pmic@25
32c00000.bus:mipi_dsi@32e60000

Each line names a device whose probe was deferred because a dependency — commonly a regulator or clock provider — was not ready when the driver core tried. Some bind later once the supplier appears; others never do. Read the list as dependency chains to investigate, then confirm final binding state separately. A non-empty list is normal, and often the first clear statement of which hardware is described but not working.

5. Map the Yocto layers

If the vendor BSP ships as Yocto metadata, three commands give you the build-side map:

raghu@techveda.org:~$ bitbake-layers show-layers
raghu@techveda.org:~$ bitbake-layers show-recipes
raghu@techveda.org:~$ bitbake-layers show-appends

show-layers lists layers with priorities, which decides who wins when two provide the same recipe. show-appends repays the effort fastest: it lists every .bbappend and the recipe it modifies, answering "who edited this, and where does that edit live". An append in a layer nobody remembers adding is the most common source of behaviour that appears to come from nowhere.

Then pin the kernel. bitbake -e virtual/kernel prints the recipe's expanded variables; you want SRC_URI, SRCREV and KERNEL_DEVICETREE. Those name the tree, commit and device tree files the image is really built from, frequently not what the documentation claims.

What this method does not tell you

A map that overstates its coverage is worse than no map.

Count is not difficulty. The directory breakdown gives the shape; only reading a sample tells you whether patches are mechanical or deep. Read twenty at random before committing to a number.
The config comparison assumes one kernel version. Across versions, renamed symbols produce noise — use make listnewconfig against the newer tree instead.
Deferred probe output needs debugfs, which production vendor configurations often disable.
These commands are verified against documentation, not against every vendor tree. Vendors modify kbuild and strip instrumentation. Where a step fails, the failure itself tells you how far this tree has drifted.

The deliverable: an Inherited BSP Baseline Report

Run the five measurements in order, then write the result up as one document. Name it, because a named deliverable survives handovers in a way scattered notes do not: the Inherited BSP Baseline Report. One page, not a wiki tree. It opens with the board's identity, carries the five measurements in the order you ran them, and closes with what you could not determine:

Board and SoC identity, with bootloader, firmware and kernel versions.
The minimised runtime config snapshot, committed with today's date.
The mainline base hypothesis and the delta by directory, with one sentence on the shape.
External module inventory: repository, build path, version, licence and maintainer for each.
Booted FDT versus source DTS differences, and the deferred-device list with final binding state.
The Yocto layer, append and kernel recipe map, with SRC_URI, SRCREV and KERNEL_DEVICETREE.
Risks, unknowns and next actions — including what you could not determine.

That last line matters most. A baseline that admits what it could not establish is trustworthy; one that quietly omits the gaps will be used to justify an estimate it cannot support.

Do this before you are asked for an estimate, not after. An engineer who answers "how far are we from mainline" with a measured number in week two is treated differently for the rest of the project.

Key takeaways

The first job on an inherited vendor BSP is measurement, not reading code.
Compare minimised defconfigs; patch location beats patch count.
Externally built modules are invisible in git history and no upgrade carries them.
The booted device tree and the Yocto layer map both differ from what the repository shows.
Record what you could not establish. A baseline hiding its gaps will be misused.

Frequently asked questions

Why compare minimised defconfigs instead of the raw .config files?
A raw .config contains thousands of symbols that were implied by other symbols rather than chosen by anyone. make savedefconfig strips those, so the resulting comparison shows only the decisions a person actually made.

What if git merge-base fails on the vendor tree?
That usually means a squashed import with no shared history. Compare trees directly with git diff --stat against the closest matching mainline tag, treating that tag as a hypothesis to validate rather than a fact.

How do I tell which loaded modules are out-of-tree?
Modules built inside the kernel tree carry an intree marker that modinfo -F intree reports as Y. Modules built outside the tree do not carry the field, and loading one sets kernel taint bit 12, which appears as raw value 4096 in /proc/sys/kernel/tainted.

The State of Automotive Grade Linux in 2026: Which Release to Build

Raghu Bharadwaj — Thu, 27 Aug 2026 07:02:15 +0000

The Automotive Grade Linux distribution is not one release on one baseline. As of 26 August 2026 its release-notes page documents three listed release series on three different Yocto Project releases: Ultimate Unagi on Scarthgap 5.0.x, Vibrant Vimba on Wrynose 6.0.x, and the old-stable Quirky Quillback on Kirkstone 4.0.x, which the Yocto Project now marks end of life. Which one you fetch depends on how long your programme has to be supported, and the answer is in the release notes rather than in any announcement.

If you have been asked to stand up an Automotive Grade Linux build this quarter, the first decision is also the one most likely to be wrong: which branch to fetch. The project publishes announcements, a product page and a release-notes wiki, and in August 2026 the three do not agree. An engineer who follows the press release lands on one baseline; an engineer who reads the release-notes table lands on a different one. The gap matters, because the two baselines have Yocto support windows that end two years apart, and a vehicle programme outlives both.

This is a practical problem rather than a criticism; large open source projects carry several series at once on purpose. What follows is what the artefacts say, why the project is structured this way, and what has moved since May.

What the Automotive Grade Linux release notes actually list

The release-notes wiki page at wiki.automotivelinux.org/agl-distro/release-notes records its own last modification as 24 August 2026. Read on 26 August 2026, it lists three named series.

Series	Yocto baseline	Upstream Yocto support	Fetch model	Main implication
Vibrant Vimba	Wrynose 6.0.x	To April 2030	`master` plus milestone manifest	Longest upstream horizon, but maturity and board coverage must be checked separately
Ultimate Unagi	Scarthgap 5.0.x	To April 2028	`unagi` branch plus manifest	Better fit where a cut branch and current board support matter
Quirky Quillback	Kirkstone 4.0.x	End of life	`quillback` branch	Legacy reproducibility only, unless the team owns the backports

Vibrant Vimba appears directly under the heading "Latest Stable Release". Its first listed feature is "Update to Yocto Project 'wrynose 6.0.x'". Its two documented builds are Milestone 1 v21.91.0 and Milestone 2 v21.92.0, and the Jira filter linked from both is labelled "Vibrant Vimba 22.0.0".

Ultimate Unagi is the 21.0.x series. Its feature list opens with "Update to Yocto Project 'scarthgap' 5.0.x" and goes on to name Qt 6.8, an upgrade to VSS 6.0, meta-flutter 3.38.3, the addition of meta-ros, KVM demo updates, and the line "Added sparrow-hawk board". Four point releases are documented. The notable-changes block for v21.0.2 reads "Yocto Project 'scarthgap' 5.0.18"; for v21.0.3 it reads "Yocto Project 'scarthgap' 5.0.19", plus layer and BSP updates and support for the Sifive hifive-premier-p550 board.

Quirky Quillback is the old-stable series, numbered 17.x, and its build-artifacts note reads "Uses Yocto 4.0.35 (YP kirkstone)".

The fetch commands are the clearest signal, because they encode the project's own view of each series. These run on your build machine.

raghu@techveda.org:~$ repo init -b unagi -m unagi_21.0.3.xml -u https://gerrit.automotivelinux.org/gerrit/AGL/AGL-repo
raghu@techveda.org:~$ repo sync

Unagi has a release branch of its own. So does Quillback, with -b quillback. Vibrant Vimba does not.

raghu@techveda.org:~$ repo init -b master -m vimba_21.92.0.xml -u https://gerrit.automotivelinux.org/gerrit/AGL/AGL-repo
raghu@techveda.org:~$ repo sync

The series the page calls "Latest Stable Release" is fetched from master with a milestone manifest, not from a cut release branch. That is a meaningful difference in what you are pinning. The branch it sits on is still the one under active development, whereas Unagi gives you both a manifest and a branch that is not moving underneath you.

Branch, manifest, and what actually counts as a baseline

Branch — a moving line of development.

Manifest — a pinned set of repository revisions, which is what makes a build reproducible.

Production baseline — a manifest, plus validation evidence, plus a maintenance plan naming who fixes what and for how long.

Why the project pins a Yocto LTS at all

Every series above is named after a Yocto Project release, and that is not decoration. A distribution built with OpenEmbedded inherits its recipe set, its toolchain, and its CVE-patching cadence from the Yocto release it is based on. When that release goes end of life, the recipes stop receiving fixes, and the burden of backporting moves to whoever is shipping the product.

The Yocto Project releases table, read on 26 August 2026, gives the windows. Scarthgap 5.0 was released in April 2024 and is long-term supported until April 2028. Wrynose 6.0 was released in April 2026 and is supported until April 2030. Kirkstone 4.0 came out in May 2022, reached 4.0.33 in February 2026, and is now marked end of life. The non-LTS releases show why the distinction exists: Walnascar 5.2 has been end of life since November 2025 and Whinlatter 5.3 is end of life too, both within roughly a year of release. Blacksail 6.1, due October 2026, is scheduled for six months of support.

One qualification matters before those dates are used for planning. They are upstream Yocto Project baseline support horizons and nothing more. They say nothing about how long AGL will maintain a given series, how long a silicon vendor will support its BSP layers, or what a vehicle programme will require. Each is a separate commitment, owned by a different party, and has to be confirmed with that party.

Set those windows against a vehicle programme. A platform entering production in 2028 and staying in service for a decade cannot be anchored to a baseline whose upstream fixes stop in April 2028. It cannot really be anchored to one that stops in April 2030 either, but the difference is two more years of somebody else doing the security backports before that work lands on your team. The same arithmetic governs kernel selection, and it is set out in more detail in Kernel LTS Support: Planning Embedded Products Around It.

The presence of a Kirkstone-based old-stable series in Automotive Grade Linux is not an oversight. Programmes that validated on Quillback years ago are still in the field, and the project keeps the series listed so those teams can reproduce a build. That does not make Kirkstone supported again. A team fetching -b quillback today takes on the backporting itself.

What changed between May and August 2026

Three months is enough to move the Automotive Grade Linux picture, and it has.

On 14 May 2026, at the AGL All Member Meeting in Tokyo, the Linux Foundation announced the initial availability of the AGL SoDeV reference platform, delivered through the Unified Code Base release codenamed Ultimate Unagi. The announcement described SoDeV as combining the Unified Code Base with Linux containers, VirtIO, the Xen hypervisor and the Zephyr RTOS, running on Renesas Sparrow Hawk reference boards, virtual machines and cloud environments. It stated the Yocto baseline as "Scarthgap LTS release 5.0.16", said Ultimate Unagi would be supported for two years, and committed to updates "approximately three weeks after each Scarthgap release by the Yocto Project". SoDeV had been announced in December 2025, so May is when developers could first fetch it.

By 24 August 2026, the date the release-notes page records for its last edit, two things had moved. Ultimate Unagi had reached v21.0.3 on Scarthgap 5.0.19, three point releases beyond the 5.0.16 named in May. And Vibrant Vimba, on Wrynose 6.0.x, had taken the "Latest Stable Release" heading.

The first is the three-week commitment being kept, visible in the artefacts rather than asserted. One caveat on the detail. The AGL notes for 21.0.3 cite Scarthgap 5.0.19, while the Yocto Project's own releases wiki, read the same day, lists 5.0.18 from June 2026 as the current Scarthgap version. The two sources were inconsistent when checked. The AGL entry records the baseline that AGL artefact was built against, which is not the same thing as a statement of upstream Scarthgap status, so confirm the current point release with the Yocto Project directly before resting maintenance assumptions on either number.

The second is a baseline change inside a single quarter. A team that started an Automotive Grade Linux evaluation from the May announcement is on Scarthgap; the page now points newcomers at Wrynose. Neither is wrong, but an evaluation framed as "we are building on what AGL ships" has already gone out of date once.

The product page adds a third answer. The Unified Code Base page on automotivelinux.org, read on 26 August 2026, still states "The latest release, UCB 16.0 (Prickly Pike)" and still describes the Unified Code Base as an in-vehicle infotainment distribution. The release-notes page's own introduction offers a fourth, saying "The latest stable release is Terrific Trout", directly above the heading naming Vibrant Vimba. Prose lags tables, and tables lag manifests.

Where SoDeV appears, and where it does not

This needs care, because the Automotive Grade Linux announcement and the release notes describe the same release in different vocabularies.

The string "SoDeV" does not appear anywhere on the AGL release-notes wiki page as read on 26 August 2026. Neither does the word "kernel", for any series. I checked the release-notes page, the Unified Code Base product page and the 14 May announcement, and found no published kernel version for any Unagi or Vimba point release. This post therefore does not state one, and you should not infer one from the Yocto baseline.

What the release notes do show is consistent with the announcement without confirming its component list. The Unagi layer list includes meta-virtualization, and the feature list includes "KVM demo updates" and the addition of the sparrow-hawk board. Those are the hooks a virtualised, multi-guest platform needs, but they are not proof that Xen and Zephyr are in the default image, and I did not verify that either way. If hypervisor configuration matters to your architecture, fetch the manifest and read the layer revisions.

One more detail is directly useful. Vibrant Vimba lists, verbatim, the known issue "R-Car Gen3 is not yet available (eta from upstream ~Oct) - use sparrow-hawk meantime". The newer baseline has lost a board that older builds had, with recovery expected around October. That is the kind of gap that does not appear in an announcement and does appear in a release note.

Choosing a branch

Reduce the Automotive Grade Linux branch decision to three questions.

How long does the support window need to run? If the programme extends past April 2028, Scarthgap's window ends inside it and Ultimate Unagi inherits that. Wrynose runs to April 2030. That is the strongest argument for Vibrant Vimba.

Do you need a cut release branch? Unagi has one. Vimba is fetched from master with a milestone manifest, and its milestone numbering is 21.9x rather than 22.0 even though the linked issue filter is labelled 22.0.0. Whether it is production-ready despite the "Latest Stable Release" heading is a question to put to the project directly rather than infer, and worth asking before committing a programme.

Which boards do you need today? The Unagi 21.0.3 artefacts table lists a Renesas H3 development board, QEMU on x86_64 and an Intel Up² board. Unagi added sparrow-hawk and, at 21.0.3, the Sifive hifive-premier-p550. Vimba currently lacks R-Car Gen3. If your bring-up target is one of those, board availability decides the branch regardless of support window, at least until October.

Underneath this sits an argument the project itself has been making. In a talk at the May All Member Meeting, published on the AGL blog on 23 June 2026, Hisao Munakata of Renesas argued that the software-defined vehicle transition repeats the Linux kernel's own 2.4 to 2.6 integration crisis, and that what resolved it was not any single technology but time-based release engineering: a fixed merge window, a validation-only period, and the discipline to defer whatever does not fit. He put the kernel's release-interval standard deviation across the v6 and v7 series at 2.83 days, and observed that automotive has no equivalent cadence. The three-week-after-Scarthgap commitment, and the point releases that followed it, are that discipline in practice. It is also why the release notes are the better source: a project on a cadence records where it is in its tables, not in its announcements.

A branch-selection checklist

Steps one and two decide the branch; the rest decide whether you can defend the choice.

Define the programme start date and the maintenance horizon it has to cover.
Confirm the boards you need and whether their BSP layers are present in the series you are considering.
Pin the exact manifest and archive the layer revisions it resolves to.
Confirm the current upstream Yocto support status with the Yocto Project, not a downstream page.
Read the release notes and issue tracker for the target branch, including known issues.
Verify virtualisation, image composition and board support from the actual layers, not announcement text.
Decide, and write down, who owns the backports once upstream support ends.

Key takeaways

Automotive Grade Linux lists three release series on three Yocto baselines: Vibrant Vimba on Wrynose 6.0.x, Ultimate Unagi 21.0.x on Scarthgap 5.0.x, and old-stable Quirky Quillback 17.x on Kirkstone 4.0.x.
Kirkstone is end of life; Scarthgap is supported to April 2028; Wrynose to April 2030. These are upstream Yocto horizons only, not AGL, BSP or programme commitments, but the support window should still drive the choice.
Unagi and Quillback have cut release branches. Vibrant Vimba is fetched from master with a milestone manifest.
Unagi moved from Scarthgap 5.0.16 in May to 5.0.19 by late August, which is the project's stated three-week cadence visible in the artefacts.
The announcement, the product page and the release-notes page each name a different "latest" release. Use the manifest as the reproducibility anchor, then validate maintenance status, BSP coverage and programme readiness separately.
Vibrant Vimba currently lists R-Car Gen3 as unavailable, with an upstream estimate of around October.

Frequently asked questions

Which Automotive Grade Linux release has the longest support window?
Vibrant Vimba, because it is based on Yocto Project Wrynose 6.0, which is supported until April 2030. Ultimate Unagi is based on Scarthgap 5.0, supported until April 2028.

Is Quirky Quillback still safe to build on?
It is still documented and still fetchable, but its Yocto baseline, Kirkstone 4.0, is marked end of life by the Yocto Project. A team building on it takes on the backporting of upstream recipe fixes itself.

What kernel version does Ultimate Unagi use?
The AGL release-notes page does not state one. The word "kernel" does not appear on that page for any series, and no kernel version was found on the Unified Code Base product page or in the May 2026 announcement either.

How do I fetch a specific Automotive Grade Linux release?
With repo init against the AGL-repo Gerrit URL, using the branch and manifest named in the release notes for that point release, then repo sync. Unagi uses -b unagi, Quillback uses -b quillback, and Vibrant Vimba uses -b master with a milestone manifest.

Is the AGL SoDeV platform the same thing as Ultimate Unagi?
The May 2026 announcement said the initial SoDeV reference platform was delivered through the Ultimate Unagi release. The name SoDeV does not appear on the release-notes page itself, so the mapping between the two is documented in the announcement rather than in the release notes.

Why the Inference Process Keeps Growing After It Starts

Raghu Bharadwaj — Wed, 26 Aug 2026 05:17:29 +0000

Written for the person who owns the DRAM budget, the memory cgroup and the OOM postmortem on an embedded board — not the person tuning the model.

To the kernel, an edge AI model is not layers. It is two mappings with very different properties: the weights, usually mapped file-backed and therefore reclaimable, and the tensor arena, a single anonymous mapping holding every intermediate tensor, which on a board without swap cannot be reclaimed at all. The arena is roughly four to ten and a half times smaller than the sum of the tensor sizes, it becomes resident only as its pages are written, and no allocation strategy can take it below a size fixed by one layer of the graph.

Somebody hands you a model file and asks whether it will run on the board. You are the one who has to answer, because you own the DRAM budget, the memory cgroup, and the postmortem when the OOM killer takes the inference process at three in the morning. The weights are the easy half: that number is on disk and you can read it. The hard half is the tensor arena, the block the runtime carves up for intermediate results, and nobody on the model side can tell you how large it will be. This article is about sizing that tensor arena with kernel tools rather than guesses.

What the kernel sees when a model loads

Attach to the process after it has loaded a model and look at its mappings. You will find two objects that matter — the weights and the tensor arena — and the difference between them decides how the system behaves under pressure.

The first is the model file itself. Where the platform supports it, the runtime maps the file rather than reading it into a buffer. In LiteRT this is MMAPAllocation, selected by FlatBufferModel::BuildFromFile after an IsSupported() check, with FileCopyAllocation as the fallback on platforms where mapping is not available. When mapping is used, the weights are file-backed pages. They appear in /proc/<pid>/maps with the model's pathname, they are clean, and the kernel can drop them under memory pressure and fault them back in from storage later.

The second is the tensor arena. It is one large anonymous allocation with no backing file, and it normally exceeds glibc's mmap threshold, so it is served by mmap as its own mapping rather than carved out of the heap. It appears in the maps with no pathname. Two caveats before you rely on that: glibc's threshold starts at 128 KiB but is raised dynamically, up to 32 MiB on 64-bit, once a large block has been freed, and a musl or uClibc-ng rootfs behaves differently again. Check it on your own board rather than assuming.

That difference is the whole reason a kernel engineer should care about this. Under memory pressure the kernel can reclaim clean file-backed pages for free. It cannot do that with the arena's anonymous pages: once written, they have nowhere to go but swap. Most embedded products ship without swap at all. Where zram is configured there is a swap device, so the page frame really is freed and a compressed copy is kept in the zram pool — but activation tensors are dense numeric data and compress far worse than ordinary heap, and every refault costs decompression on the inference path. So of the two halves of your model's memory, the weights are the half the kernel can drop for free, and the tensor arena is the half that stays where it is until the process exits. When you size DRAM, the arena is the number with no give in it.

The weights half is its own problem, governed by the numeric format you ship and by how the memory system moves those bytes, covered separately in 4-Bit Weight Quantization: Why the Memory System Decides. The rest of this article is about the half that cannot be reclaimed.

The number the model team gives you is wrong

Ask for the memory requirement and you will usually get the sum of the tensor sizes, because that is easy to compute. For MobileNet v2 at 32-bit float the intermediate portion of that sum is about 26 MB, roughly twice the size of the model file. Measure the process on the board and you will see nothing like 26 MB of anonymous memory. You will see something near 6 MB. Here is the calculation you were handed, in full, which runs on the host rather than the board because it needs the full framework:

import numpy as np
import tensorflow as tf

interp = tf.lite.Interpreter(model_path="model.tflite")
interp.allocate_tensors()

total = 0
for t in interp.get_tensor_details():
    shape = t["shape"]
    if shape.size == 0:
        continue
    total += int(np.prod(shape)) * np.dtype(t["dtype"]).itemsize

print("sum of all tensor bytes:", total)

Note what that loop actually sums: every tensor the interpreter reports, weights and constants included, for the primary subgraph only. It is the shape of the estimate you will be handed, not a measurement of the arena. The intermediate share of it is what the naive column of the table further down measures.

The gap is the allocation plan, and because it depends on the shape of the graph rather than on the weight count, it produces a specific failure. A team sizes DRAM from one model they measured, then ships a second model with similar accuracy and a similar file size, and the second one needs considerably more anonymous memory because its graph has a different shape. Nothing in the model file warns you, and the arena grows quietly across a project: every skip connection added to improve a metric raises the peak by an amount nobody computed.

What a tensor arena actually is

A network is a directed acyclic graph: nodes are operators, edges are tensors holding intermediate results. The runtime allocates memory for those edges before inference rather than during it, for reasons a kernel engineer will recognise immediately. Calling the allocator inside the inference loop adds time that is neither small nor predictable, and an allocation that can happen mid-inference can also fail mid-inference. So the runtime asks a different question up front: not how much memory all these tensors need, but what is the smallest single block in which all of them can live, given that they do not all need to be alive at once. That block is the tensor arena, and computing it is a scheduling problem rather than an allocation problem.

The answer can be far smaller than the sum because execution is sequential. At any moment one operator is running and only its inputs and outputs must be valid; everything produced earlier and already consumed is dead memory that can be handed to a later tensor. If the graph were a plain chain, two buffers sized for the widest tensor would do. Real graphs are not chains, and residual connections that keep a tensor alive across many operators are what make the problem hard.

The record that makes reuse possible

Every runtime that does this well builds the same small record per tensor: its size, the index of the first operator that touches it and the index of the last, where the indices come from the topological sort that is also the execution order. In LiteRT you can read it directly in tensorflow/lite/simple_memory_arena.h:

struct ArenaAllocWithUsageInterval {
  size_t offset;
  size_t size;
  int32_t tensor;
  int32_t first_node;
  int32_t last_node;
};

That is the idea in five fields. first_node and last_node define a usage interval, and two tensors whose intervals overlap can never share bytes. The planner assigns every tensor an offset into one buffer such that no two overlapping tensors overlap in memory, and makes the buffer as small as it can. The same class tracks a high_water_mark_ and ships a debug dump behind a weak symbol, so a build that links lite:simple_memory_arena_debug_dump can print arena utilisation and live tensors per operator.

One layer decides your minimum arena size

Three derived quantities explain everything downstream. The operator profile of an operator is the set of tensors alive while it runs. The operator breadth is the sum of their sizes. And for a contiguous tensor arena, the theoretical minimum arena size is the maximum operator breadth across the whole graph. It is a lower bound, not a promise: the published measurements reach it on most networks and miss it on some.

If the arena does not fit your budget, one specific layer is responsible. Not the model in general, not the weight count. Every tensor in an operator's profile must be resident while that operator runs, so no allocation strategy can produce an arena smaller than the widest operator. That changes what you say in the review meeting: you can name the layer and hand it back, and the request stops being "please make the model smaller" and becomes "this layer requires N megabytes minimum, can it be restructured".

The planning itself is cheap. These are approximation algorithms for an NP-complete problem, they run in milliseconds, and a runtime can afford to compute several plans at startup and keep the smallest. Be careful where you go looking for them, though. The MemoryStrategy enum — NAIVE, EQUALITY, GREEDY_IN_ORDER, GREEDY_BY_BREADTH, GREEDY_BY_SIZE, GREEDY_BEST and MINCOSTFLOW — belongs to the GPU delegate's planner. The CPU arena you are measuring does not consult it. SimpleMemoryArena::Allocate implements its own search for the smallest gap between already-placed allocations, ordered by ArenaPlanner, so that pair of files is what to read if you want to know what produced your arena.

Where the naive number ranks two models backwards

MobileNet v2 and DeepLab v3 differ by a factor of 1.85 in naive intermediate memory. After packing they need almost the same arena, 5.742 MB against 4.653 MB. Size a board from the naive numbers and you would provision DeepLab for more than ten times the memory it needs, and you would have the relative cost of the two models backwards. The full set is below.

Network	Naive (MB)	Packed arena (MB)	Ratio
DeepLab v3	48.642	4.653	10.5x
Inception v3	54.010	7.914	6.8x
BlazeFace	2.698	0.492	5.5x
MobileNet v2	26.313	5.742	4.6x
PoseNet	28.556	6.271	4.6x
MobileNet v1	19.248	4.594	4.2x

Intermediate tensor memory only, 32-bit float, packed greedy by size. Published measurements; see Further reading. At int8 the absolute figures fall by roughly four times, but the ratios are what the sizing argument rests on and those are a property of the graph.

The ratio column is why a naive estimate is not a conservative estimate. It is wrong by a different amount for every graph, so it does not preserve the ordering between models, and it cannot be scaled into a tensor arena figure by applying a fixed factor.

Greedy by size reaches the theoretical minimum on five of these six networks; on DeepLab v3 a strip-packing method beats it by 7.2 per cent, which is the argument for computing two plans and keeping the smaller. Tighter packing should also mean a smaller working set and better cache behaviour. The source claims up to 10 per cent better inference speed from that effect but publishes no measurement for it, so treat it as something to test on your board rather than a number to budget against.

Why the process keeps growing after it starts

Here is where a straightforward measurement misleads you, and the cause is demand paging. The tensor arena is an anonymous mapping, so its pages are not resident until they are written. The runtime reserves the address range up front, but the kernel allocates physical pages only on the first write, one minor fault at a time. A read of untouched anonymous memory maps the shared zero page and costs nothing.

The consequence is that the process footprint climbs across the first several inferences and settles only once every offset in the tensor arena has been written at least once. A model whose plan places a large tensor in a branch that only executes for certain inputs may not touch those pages for a long time. So a single-shot measurement understates the requirement, sometimes badly, and the number you budget against must be the high-water mark rather than the current value.

Two files give you that. VmHWM in the process status file under /proc is the peak resident set size the process has ever reached, and writing 5 to clear_refs in the same directory resets it, which is how you scope a peak to one run. The smaps_rollup file gives a single summed entry across all mappings, including Rss, Anonymous, AnonHugePages and the rollup-only fields Pss_Anon, Pss_File and Pss_Shmem. That anonymous-versus-file split is exactly the reclaimable-versus-not split from earlier, measured rather than assumed, and it is the single most informative thing you can read about an inference process. Pss_Shmem is worth watching too: dma-buf and most accelerator allocations land there rather than in Pss_Anon. Reading the file walks every page table in the process, so it perturbs what it measures.

Sizing the cgroup instead of guessing

Once you know what the tensor arena costs in practice, put a boundary around it rather than hoping. In cgroup v2, memory.max is the hard limit: reach it and fail to reclaim, and the OOM killer is invoked inside the cgroup. memory.high is the throttle: exceed it and the processes are throttled and put under heavy reclaim pressure, and the documentation is explicit that going over it never invokes the OOM killer.

For an inference workload on an embedded board that distinction needs care rather than a default, and the reason is specific. On a swapless board the kernel does not scan the anonymous lists at all: reclaim is steered onto the file lists instead. So exceeding memory.high does not shrink the arena by a single byte. What it does is evict the other half of your model — the mapped weights — and force them back in from storage, while the task is separately throttled by a penalty sleep. You pay in refault latency and scheduling delay and get nothing back.

The practical arrangement follows from that. Use memory.max as the real boundary. Put memory.high a little below it as a monitoring tripwire rather than as a control, so the counter moves before anything is killed. And set memory.min at roughly the mapped weight size, so the reclaim that does happen cannot evict the model out from under the process.

For measurement, memory.peak records the maximum usage for the cgroup and its descendants since the cgroup was created or since the most recent reset. The reset semantics have a trap in them: writing any non-empty string resets the value only for subsequent reads through that same file descriptor. A shell sequence that writes with tee and then reads with cat opens a second descriptor and quietly reports the never-reset watermark instead. Either hold one descriptor open across the run, or use a fresh cgroup per run and skip the reset. And memory.events carries the counters that tell you afterwards what happened, including high, max, oom and oom_kill; memory.events.local is the non-hierarchical version, which is the one you want for a single-cgroup harness. A rising high count with no oom_kill is a system quietly paying for reclaim, which is the state most teams never notice.

That state has its own instrument, and it is the one to reach for. Pressure stall information reports how much work was lost to waiting on memory: system-wide in /proc/pressure/memory, and per cgroup in memory.pressure. Both carry two lines. The some line is the share of time in which at least one task was stalled on memory. The full line is the share in which every non-idle task was stalled at once, which the kernel documentation describes as thrashing, and which on an inference board means the CPU is not doing your work at all. Each line gives ten, sixty and three hundred second averages as percentages, plus a cumulative total in microseconds that catches short spikes the averages flatten out.

root@rock-5b:~# cat /sys/fs/cgroup/infer/memory.pressure
root@rock-5b:~# cat /proc/pressure/memory

For a latency-sensitive product the number to alarm on is full avg10 for the inference cgroup. Anything sustained there means the board is spending real time refaulting rather than inferring, and given what the previous paragraphs established, what it is refaulting is your model weights rather than the tensor arena. The system-wide file also accepts a trigger, so a supervisor can wait on a threshold rather than sample: writing some 150000 1000000 to it asks to be woken when partial memory stall passes 150 ms within any one-second window, with the file descriptor polled for POLLPRI. Accepted windows run from 500 ms to 10 s. Pressure stall information has to be built into the kernel, so confirm the files exist on your board before designing a supervisor around them.

Why the arena cannot be handed to an accelerator

One more property matters if the model runs on anything other than the CPU. The tensor arena is contiguous in virtual address space, not in physical memory. Its pages are ordinary anonymous pages scattered across whatever the page allocator had, and nothing about a tensor arena guarantees the physical adjacency a device may need.

That is fine for a CPU, and fine for a device behind an IOMMU or one that can scatter-gather from a descriptor list. It is not fine for a master that cannot do either, which needs physically contiguous memory from CMA or a DMA heap. Even an accelerator that could technically reach the arena usually wants its own buffers anyway, for cache-coherency and vendor layout reasons. This is why a hardware delegate allocates separately rather than being pointed at the arena, and why the arena figure and the process footprint diverge once you enable an NPU or GPU path. The arena tells you what the CPU-side plan costs. It does not tell you what the accelerator's buffers cost, and those come from a different allocator with tighter constraints. I have not verified how any specific vendor delegate accounts for its own memory, so measure that path separately rather than reasoning from the arena number.

Where this stops working

The tensor arena scheme rests on one assumption: that every intermediate tensor's size is known before the first inference. The clearest break is dynamic shapes. A graph where a tensor's extent depends on the input cannot be planned in one pass; the workaround is to plan repeatedly, allocating what is known, executing until the first dynamic size resolves and planning again. That works, but it puts allocation back inside the inference path along with the latency variance that pre-allocation existed to remove. If you are consulted on architecture for a latency-sensitive product, this is a real reason to argue for static shapes.

Worse, the arena does not shrink again. ResizableAlignedBuffer::Resize skips reallocation when the new size is smaller, so once a dynamic-shape run has grown the arena it stays grown for the life of the process. One unusually large input sets your footprint for the next three months of uptime.

Two further limits. The execution order is taken as given, so the planner optimises for a fixed topological sort and does not reorder operators to reduce the peak — offline tools exist that do reorder, which is worth knowing before you accept the number as final. And none of this applies to training, where activations must stay alive for the backward pass, so any on-device fine-tuning you are asked to support has a memory profile unrelated to the inference numbers you measured.

Measuring a tensor arena on your own board

The board below runs a Yocto image built with MACHINE = "rock-5b". Put the benchmark tool in the image rather than copying a binary onto a running board, so that what you measure is the rootfs you ship. The community meta-tensorflow-lite layer carries a recipe for it:

raghu@techveda.org:~$ bitbake-layers add-layer ../meta-tensorflow-lite
raghu@techveda.org:~$ echo 'IMAGE_INSTALL:append = " tensorflow-lite-benchmark"' >> conf/local.conf
raghu@techveda.org:~$ bitbake core-image-base

Everything from here runs on the board. Start on the kernel side, because those numbers are the ones you will defend in a design review. The process must still be alive when you read /proc, so run it long and in the background rather than sequentially:

root@rock-5b:~# cd /usr/share/tensorflow/lite/tools/benchmark
root@rock-5b:~# ./benchmark_model --graph=model.tflite --num_runs=2000 & BM=$!
root@rock-5b:~# sleep 5
root@rock-5b:~# grep VmHWM /proc/$BM/status
root@rock-5b:~# grep -E "^(Rss|Pss_Anon|Pss_File|Pss_Shmem|Anonymous|AnonHugePages|Swap):" /proc/$BM/smaps_rollup

Confirm the arena really is one anonymous mapping by listing the mappings with no pathname, largest first. Selecting on field count rather than on the absence of a slash is what keeps the heap, the stack and the vDSO out of the list, since those carry a bracketed name in the sixth field:

root@rock-5b:~# awk 'NF==5 {split($1,a,"-"); print ((strtonum("0x" a[2]) - strtonum("0x" a[1]))/1048576) " MB " $1}' /proc/$BM/maps | sort -rn | head

Watch the first-write behaviour directly. The minor fault count bounds the arena from above rather than isolating it, because it also counts the binary, the libraries and any delegate buffers. Disable the warm-up run so only one inference is measured, and diff a dry run against a real one if you want the arena's own share:

root@rock-5b:~# perf stat -e minor-faults,major-faults ./benchmark_model --graph=model.tflite --num_runs=1 --warmup_runs=0
root@rock-5b:~# perf stat -e minor-faults,major-faults ./benchmark_model --graph=model.tflite --dry_run=true

Bound it with a cgroup. The memory controller has to be enabled in the parent before the interface files exist in the child, and the workload has to actually be moved into the cgroup, or you will read zeroes from a well-formed but empty directory:

root@rock-5b:~# echo +memory > /sys/fs/cgroup/cgroup.subtree_control
root@rock-5b:~# mkdir -p /sys/fs/cgroup/infer
root@rock-5b:~# echo 64M > /sys/fs/cgroup/infer/memory.max
root@rock-5b:~# sh -c 'echo $$ > /sys/fs/cgroup/infer/cgroup.procs; exec ./benchmark_model --graph=model.tflite --num_runs=2000'
root@rock-5b:~# cat /sys/fs/cgroup/infer/memory.peak /sys/fs/cgroup/infer/memory.events.local

A fresh cgroup per run is what makes that final read trustworthy. Do not reach for the reset unless you are driving it from a program that can hold the descriptor open, for the reason given earlier.

Then go to the runtime for the plan itself. This flag prints the interpreter's internal state before the first inference, including the allocated size of each tensor, which is the plan rather than a guess about it. The second samples the tool's own footprint at a fixed interval, 50 ms by default; the sampling perturbs latency, so do not read latency and peak memory from the same run:

root@rock-5b:~# ./benchmark_model --graph=model.tflite --print_preinvoke_state=true --num_runs=1
root@rock-5b:~# ./benchmark_model --graph=model.tflite --report_peak_memory_footprint=true --memory_footprint_check_interval_ms=20

If the model has dynamic-shaped tensors, use the post-invoke state rather than the pre-invoke state, because the sizes are not known until the graph has run. Two flags then change the trade-off, but they are not independent: enabling the second also enables the first, so measure baseline, then the first alone, then both:

root@rock-5b:~# ./benchmark_model --graph=model.tflite --print_postinvoke_state=true
root@rock-5b:~# ./benchmark_model --graph=model.tflite --release_dynamic_tensors=true
root@rock-5b:~# ./benchmark_model --graph=model.tflite --optimize_memory_for_large_tensors=1

Finally, back on the host, read the planner your build actually uses rather than trusting an article about it. For a CPU inference process that means the arena and its planner, not the GPU delegate's strategy enum:

raghu@techveda.org:~$ grep -nE "smallest gap|best_offset" tensorflow/lite/simple_memory_arena.cc
raghu@techveda.org:~$ grep -rn "MemoryStrategy" tensorflow/lite/simple_memory_arena.cc

The second command returns nothing, and that is the point. The CPU arena never consults MemoryStrategy.

Key takeaways

Budget DRAM off the tensor arena figure, not the weights figure. The weights are the half the kernel can drop for free.
A naive estimate is not a conservative one. It is wrong by a different factor for every graph, so it does not even preserve the ordering between two models.
Name the layer instead of asking for a smaller model. The maximum operator breadth is the bound no allocation strategy can beat.
Measure VmHWM over a long run, never a single inference, and reset it with clear_refs to scope a peak to one run.
Set memory.max as the boundary, memory.high below it only as a tripwire, and memory.min at the weight size so reclaim cannot evict the model.
Alarm on full avg10 in the cgroup's memory.pressure. It is the one number that tells you the board is refaulting rather than inferring.
Measure the accelerator path separately. The arena is virtually contiguous only, and delegate buffers come from a different allocator.
Dynamic shapes, training and delegate buffers all fall outside the plan, and an arena grown by one large input never shrinks back.

Frequently asked questions

Why does my inference process keep growing for the first few seconds?

Because the tensor arena is an anonymous mapping whose pages become resident only when first written. The address range is reserved up front, but physical pages arrive one minor fault at a time, so the resident set climbs until every offset in the arena has been touched.

Which part of an edge model can the kernel reclaim under memory pressure?

Only the weights, and only when the runtime maps the model file rather than copying it. Those pages are clean and file-backed, so the kernel can drop them and read them back later. The arena is anonymous, so without a swap device it cannot be reclaimed at all.

Which layer decides how big my tensor arena has to be?

The operator with the largest breadth, meaning the largest total size of all tensors that must be resident while it runs. That is a lower bound no allocation strategy can beat, though a given strategy may not reach it exactly.

Should I use memory.high or memory.max for an inference cgroup?

Use memory.max as the real boundary and memory.high below it only as a tripwire. On a swapless board the kernel steers reclaim entirely onto the file lists, so exceeding memory.high evicts the mapped weights and throttles the task without shrinking the arena at all.

How do I tell whether the board is losing time to memory pressure?

Read memory.pressure in the inference cgroup, or /proc/pressure/memory system-wide. The full line is the share of time in which every non-idle task was stalled on memory, so a sustained full avg10 means the board is refaulting rather than inferring.

Can an NPU or GPU use the tensor arena directly?

Not in general. The arena is contiguous in virtual address space but its physical pages are scattered, so a DMA master without an IOMMU cannot use it. Delegates allocate their own buffers through a different path, which is why the arena figure and the process footprint diverge once acceleration is enabled.

systemd vs BusyBox init: Which Init System Fits Your Device?

Raghu Bharadwaj — Wed, 26 Aug 2026 05:13:43 +0000

For most embedded products with a fixed set of services and no user-installable software, BusyBox init is the right default: small, no dependency chain, and its behaviour fits in one readable file. Choose systemd when the device genuinely needs supervised, interdependent services — readiness ordering, watchdog-backed liveness detection, resource limits — and accept in return a much larger image, a kernel floor of 5.10, a dependency set you maintain for the life of the product, and (if you build with Buildroot) a glibc toolchain for the whole system.

The choice of systemd vs BusyBox init is one of the first structural decisions in an embedded Linux product, and one of the hardest to reverse. PID 1 defines how services declare dependencies, how failures are detected, and which libraries stay in the image forever. Many teams inherit the choice from the vendor BSP without recording why, then work around it for two years. The problem is the unrecorded decision, not the inherited default.

When this decision arises

The decision arises once you move past a single application binary. A device that boots, runs one process and reboots on failure needs nothing beyond a supervising parent. The forces appear when the product gains a second and third long-running service with an ordering relationship, and when support asks why a unit stopped responding without rebooting. Four forces decide it: the image budget from the bill of materials, doubled in an A/B layout; the service topology, meaning whether any service can fail without exiting; the platform floor, meaning the C library and kernel version you are committed to; and the maintenance horizon, meaning how long you must ship security updates.

One clarification first, because it causes real design errors. Init and device management are two decisions, not one. Yocto's INIT_MANAGER values bundle them, but the device manager has its own variable, VIRTUAL-RUNTIME_dev_manager, with documented values udev, busybox-mdev and systemd. Buildroot keeps them in separate menus, and its /dev management choice is hidden only when BR2_INIT_SYSTEMD is selected. Either build system will run eudev under BusyBox init, so a USB modem or a camera that needs real udev rules does not oblige you to take systemd.

# Yocto: distro config or local.conf
INIT_MANAGER = "sysvinit"       # Poky default, SysVinit + udev
INIT_MANAGER = "mdev-busybox"   # BusyBox init + BusyBox mdev
INIT_MANAGER = "systemd"        # systemd + udev

Buildroot's System configuration → Init system menu is longer than most write-ups suggest: four general-purpose options — BR2_INIT_BUSYBOX (the default), BR2_INIT_SYSV, BR2_INIT_OPENRC and BR2_INIT_SYSTEMD — plus four special-purpose ones that are mostly container reapers.

The options: systemd vs BusyBox init

BusyBox init

BusyBox init lives inside the BusyBox multi-call binary you almost certainly already ship. It reads /etc/inittab at startup; entries take the form <id>:<runlevels>:<action>:<process>, with the action one of sysinit, wait, once, respawn, askfirst, restart, ctrlaltdel or shutdown. The runlevel field is unused. Buildroot's default inittab mounts a few filesystems, runs /etc/init.d/rcS and starts a getty.

Myth: "respawn gives me service supervision."

An entry marked respawn is restarted when it exits, and that is the whole of the guarantee. There is no back-off and no start limit, so a service that crashes at startup restarts about once a second, paced only by the one-second sleep in init's main loop, until someone power-cycles the board. BusyBox's own documentation in init/init.c states: "Unlike sysvinit, BusyBox init does not stop processes from respawning out of control." A process that is alive but no longer working never exits, so it is never restarted at all.

BusyBox also ships pieces people assume it lacks. The watchdog applet takes -T N (reboot after N seconds if not reset, default 60), -t N (reset every N seconds) and -F (foreground), so you can run it from inittab as a respawn entry. The syslogd applet accepts -C[size_kb] to log to a shared-memory circular buffer that logread reads — on a flash-based device, often exactly the logging policy you want.

In its favour: negligible size on top of BusyBox; no new libraries; a boot sequence one engineer can read in full before changing it; no C library or kernel constraint beyond BusyBox's own; and failure behaviour that is exactly what you wrote.

Against it: ordering is whatever order you wrote the script in, so a race between two services can remain undetected until a unit fails at a customer site; a hung service stays hung; there is no restart back-off; and supervision beyond respawn is yours to write, test and document.

systemd

systemd replaces init with a service manager. Services are declarative unit files rather than shell scripts, they start in parallel, and the manager tracks them using control groups. Three mechanisms carry most of the value on a device, and each of the three is commonly misread.

Myth: "systemd makes service ordering reliable."

Ordering is not readiness. After= orders unit start-up; it does not wait for the dependency to be usable. Type=notify does: the service sends READY=1 through sd_notify, and systemd "will proceed with starting follow-up units after this notification message has been sent." That is what removes sleep loops from your boot path, and it is also the real migration cost, because it means changing the application. Teams that convert rcS into units without adopting Type=notify keep their race and conclude systemd did not help.

Myth: "systemd automatically detects hung services."

Liveness detection is two independent settings, both off by default. A unit's WatchdogSec= makes the service ping PID 1: it calls sd_notify with WATCHDOG=1, and if the gap between two calls exceeds the configured time, systemd marks the service failed and terminates it with SIGABRT (or WatchdogSignal=), restarting it under a suitable Restart= value. It "defaults to 0, which disables this feature." Separately, RuntimeWatchdogSec= in systemd-system.conf makes PID 1 ping the hardware watchdog — WatchdogDevice=, default /dev/watchdog0 — so the board resets if PID 1 itself stops. It also defaults to 0. Escalating a hung service into a board reset therefore needs three things: the unit setting, the manager setting, and an application that actually calls sd_notify.

The start limit will stop restarting your service. DefaultStartLimitIntervalSec= is 10 seconds and DefaultStartLimitBurst= is 5, so the sixth start inside that window is refused. Units "which are configured for Restart=, and which reach the start limit are not attempted to be restarted anymore."

Myth: "Restart=always means the service always comes back."

The consequence is invisible on a desktop, where an administrator sees the failed state and runs systemctl reset-failed. On a headless device in the field nobody sees it, and the product stops working while PID 1 continues to report a normal system state. StartLimitAction= exists for this and defaults to none; on a product, setting it to reboot and raising RestartSec= above its 100 ms default is usually correct.

[Unit]
Description=Sensor gateway
After=sensor-bus.service

[Service]
Type=notify              # follow-up units wait for READY=1
NotifyAccess=main
WatchdogSec=30s          # also needs RuntimeWatchdogSec= in system.conf
Restart=on-failure
RestartSec=5s
StartLimitBurst=5
StartLimitAction=reboot  # do not give up silently on a headless device

In its favour: readiness ordering rather than implied ordering; detection of a service that has stopped responding, in addition to one that has exited; per-service resource limits and sandboxing, auditable with systemd-analyze security (it reports an exposure level from 0.0 to 10.0, where a high value means little sandboxing); integrated logging; boot analysis through systemd-analyze critical-chain; and one operational interface new engineers already know.

Against it: size and dependencies. The Yocto documentation states that using systemd increases the core-image-minimal image size by 160% for qemux86-64 on Mickledore (4.2), compared to SysVinit. Note what that compares: systemd against SysVinit, not against BusyBox init. A BusyBox-init image is smaller than a SysVinit one, so treat 160% as a lower bound on the gap this decision creates.

Myth: "systemd is too big for embedded."

That lower bound is a percentage of a reference image with almost nothing in it, which is why it travels badly. On a product carrying a graphics stack or a Python runtime, the same absolute delta is a small fraction of the total. The size objection is decisive on a 64 MB flash part and close to irrelevant on a 4 GB one, which is why the number has to come from your own image rather than from a published comparison.

The dependency set is the more durable cost. Upstream requires glibc 2.34 or newer (musl 1.2.6 only when built with -Dlibc=musl), sets a minimum kernel baseline of 5.10 and recommends 5.14, below which it sets an old-kernel taint flag. Buildroot's package/systemd/Config.in is stricter: among other gates it depends on BR2_TOOLCHAIN_USES_GLIBC, an MMU, kernel headers of at least 5.4 and GCC 8, and selects merged /usr, D-Bus (or dbus-broker), libcap, util-linux, kmod and timezone data.

The options this decision does not take

A decision record is judged partly on the alternatives it rejects. SysVinit sits between the two: shell-script services, runlevels, and udev rather than mdev. It suits an inherited body of /etc/init.d/ scripts and is a poor choice starting fresh, costing more than BusyBox init without the supervision that justifies systemd. OpenRC is the genuine middle option; Buildroot ships it as BR2_INIT_OPENRC, depending only on an MMU, dynamic libraries and a non-uClibc toolchain, so it is available on musl where systemd is not. s6, runit and finit offer stronger supervision than respawn on similar terms. All are rejected here for one reason: they add a framework your team must learn and integrate, without the ecosystem, vendor BSP alignment or hiring pool that make systemd worth its cost. If your team already runs one, that reasoning does not apply.

A trimmed systemd deserves naming too, because a reviewer will ask what a minimal build costs. Read the option list for your own release rather than a figure from a blog post: in Yocto, bitbake -e systemd and look at PACKAGECONFIG; in Buildroot, the BR2_PACKAGE_SYSTEMD_* sub-options. Then apply the sentence that matters — trimming moves the size number, not the kernel floor, the C library constraint, or the count of upstream projects you track.

Measure four things before you decide

An argument about systemd vs BusyBox init that contains no numbers from your own product will not survive a design review. Four measurements take an afternoon.

Size delta on your image, not a reference image. In Buildroot, run make clean all then make graph-size with each init selected; it writes output/graphs/graph-size.pdf, package-size-stats.csv and file-size-stats.csv, and size-stats-compare diffs two of those CSVs. In Yocto, set INHERIT += "buildhistory" and BUILDHISTORY_COMMIT = "1", build with each INIT_MANAGER value, then read installed-package-sizes.txt or run buildhistory-diff. Report it twice: absolute bytes on the image you ship, and a percentage of one A/B slot. On a 400 MB image with a graphics stack, a few megabytes is not an argument.

Resident memory, usually the binding constraint below 256 MB. Flash is what teams budget; RAM is what runs out. Under systemd, PID 1, the journal daemon, the D-Bus daemon and udevd are four resident processes BusyBox init does not have. Sum Pss from /proc/<pid>/smaps_rollup, or use systemd-cgtop -m.

Your real service topology. Count long-running processes that must survive a crash; under five, BusyBox init is almost certainly right. For each, write down what must be ready before it starts and count the pairs. Count how many can hang without exiting — anything with a blocking I/O loop or a third-party SDK. Count how many you did not write. Then count the sleep calls and lock files in your current rcS: three or more means you have already built a service manager, badly.

Boot time, before assuming either option helps. Under systemd, systemd-analyze time splits firmware, loader, kernel and userspace, and critical-chain shows the path that gates boot. Under either init, initcall_debug printk.time=1 covers the kernel side, and instrumenting rcS covers user space.

raghu@techveda.org:~$ systemd-analyze critical-chain
raghu@techveda.org:~$ read -r up _ < /proc/uptime; echo "rcS-done $up" > /dev/kmsg

The only number a product review should accept is a GPIO your application toggles, measured from power-on.

The decision

In systemd vs BusyBox init, the default for a fixed-function device — a sensor gateway, a controller, a camera node, a kiosk — with a known set of services is BusyBox init. It matches the product's complexity and leaves the C library and kernel version open. This is also the Buildroot developers' recommendation: BusyBox init is sufficient for most embedded systems.

Choose systemd when at least one of these is true, and be honest about whether it is:

The platform is already systemd-based and has sufficient flash. Diverging from the vendor BSP is not a one-time task; it re-opens at every BSP bump, and it should be budgeted as recurring engineering. Most silicon-vendor reference distributions ship systemd, so verify what yours does before assuming a free choice.
Services depend on each other in ways a linear script cannot express, or you already order them with sleep loops and lock files.
A service can hang without exiting and the product must notice. WatchdogSec= plus RuntimeWatchdogSec= answers this; respawn does not.
You need per-service resource limits or sandboxing, so one service cannot starve another. On a device hosting a customer application this is the strongest argument in systemd's favour.
The device runs software your team does not control: customer applications, containers, third-party agents.

Two conditions rule out systemd regardless of everything above. If your toolchain is musl or uClibc-ng and you build with Buildroot, systemd is unavailable without changing the C library for the whole product, and OpenRC is the middle option there. That is Buildroot's packaging constraint rather than an upstream one, since upstream supports musl when systemd is built with -Dlibc=musl. If your kernel is older than 5.10, upstream systemd does not support it at all, which is a reason to update the kernel before adopting systemd.

Consequences

Boot time does not improve automatically. Parallel startup helps a given service reach ready state sooner, but the Yocto documentation is direct about the whole-system effect: systemd also provides more services by default, therefore increasing the total system boot time.

Myth: "systemd makes the device boot faster."

The honest argument for systemd on boot time is that it provides the measurement tools, not that the device boots sooner. Total boot time is set by what you start and in what order, which is a property of your image rather than of PID 1. Our series on boot time phases and measuring the kernel and user space covers the method under either init.

Where the logs go is decided once, for the life of the product. systemd's integrated logging is a real benefit, and on raw NAND or eMMC a persistent journal is also a wear mechanism. Set Storage=volatile and the journal lives in /run/log/journal, touching no flash. The documented default has changed across releases — older systemd documents auto, current systemd documents persistent — so run systemd-analyze cat-config systemd/journald.conf on your target rather than trusting one figure. If the journal stays in RAM, bound it: RuntimeMaxUse= defaults to 10% of the filesystem it sits on, capped at 4G, which on a tmpfs means RAM. Under BusyBox init you reach the same place by default, with syslogd -C and logread.

The dependency set becomes your security surface. D-Bus, udev, util-linux, kmod and libcap are five more upstream projects to track for the supported life of the device. Produce the number rather than asserting it: build with each init selected and diff the manifests. make legal-info in Buildroot writes output/legal-info/manifest.csv; a Yocto build with CVE checking and an SBOM class gives the equivalent. That difference is the maintenance cost, as a component count you can put in the record.

Layout constraints propagate, including to a read-only root. systemd requires a build prefix of /usr and does not support split-usr systems; a separate /usr must be mounted from the initrd. On a read-only root — standard on A/B devices, and enabled in Yocto with IMAGE_FEATURES += "read-only-rootfs" — the handling of /var becomes an explicit choice, which Buildroot exposes as BR2_INIT_SYSTEMD_VAR_FACTORY (default), _OVERLAYFS and _NONE. This qualifies one argument for BusyBox init: on a read-only root you cannot edit rcS on the target, so the benefit is a boot sequence short enough to reason about, not one editable in the field. On a PREEMPT_RT product, upstream additionally recommends CONFIG_RT_GROUP_SCHED=n under systemd, because real-time group scheduling requires an explicit budget per unit.

Choosing BusyBox init means owning the supervision logic. The unthrottled crash loop described earlier is yours to contain. Set panic= on the kernel command line so a dead PID 1 still resets the board, feed the hardware watchdog from a small dedicated process rather than the application, and write the supervision design down as a document rather than leaving it in rcS. The staffing risk is not that BusyBox init is hard to learn; it is that only its author knows what it does.

When this recommendation is wrong

The recommendation fails when "fixed-function" is doing more work than it can bear. An instrument cluster or a medical monitor is fixed-function in the product sense and still runs a dozen interdependent services with real failure modes. The useful test is not what the product does, it is the topology count from the measurement section. It also fails for a team whose background is server or desktop Linux: the argument that BusyBox init is simpler assumes an engineer who reads shell more fluently than unit files, and that assumption is worth checking rather than presuming.

Key takeaways

In systemd vs BusyBox init, default to BusyBox init for fixed-function devices, as Buildroot's developers recommend — and record why, so the next team does not repeat the argument.
Deviate when the platform is already systemd-based, when a service can hang without exiting, or when the device runs software you do not control.
Neither option supervises by default. BusyBox init does not throttle respawns; systemd's WatchdogSec=, RuntimeWatchdogSec= and StartLimitAction= are all off or inert until you set them.
systemd sets a platform floor: kernel 5.10 minimum, 5.14 recommended, merged /usr, glibc 2.34 — and in Buildroot, a glibc toolchain for the whole product.
Bring four numbers to the review: size delta on your own image, resident memory, service topology count, and boot time measured to application readiness.

Frequently asked questions

Can I use systemd with a musl toolchain?
Upstream systemd lists musl 1.2.6 or newer as a requirement when built with -Dlibc=musl, so upstream support exists. Buildroot's systemd package still declares depends on BR2_TOOLCHAIN_USES_GLIBC, so choosing systemd in Buildroot means choosing a glibc toolchain for the whole product. On musl in Buildroot, OpenRC is the dependency-ordered alternative.

What is the minimum kernel version for systemd?
The upstream systemd README sets 5.10 as the minimum baseline and states that kernels below it are not supported at all. Version 5.14 is the recommended baseline; below it, systemd sets an old-kernel taint flag and upstream support is limited.

Does BusyBox init detect a service that has stopped responding?
No. An inittab entry with the respawn action restarts a process when it exits, but a process still running and no longer doing useful work will not be restarted, and BusyBox's own documentation states that it does not stop processes from respawning out of control. Detecting a hung service needs a watchdog protocol you implement yourself, or systemd's WatchdogSec=.

Does the licence difference affect the decision?
Not materially. systemd is LGPL-2.1-or-later with exceptions noted in its tree, and BusyBox is distributed under version 2 of the GPL only. Both run as separate processes from your application, so the compliance obligation is broadly the same. The one case worth noting is linking libsystemd into a proprietary daemon to obtain sd_notify; the readiness protocol is a documented datagram to $NOTIFY_SOCKET with reference implementations published upstream, so you can implement it without linking anything.

Linux 7.3: AF_ALG Restricted by Default, YOLOX on Etnaviv

Raghu Bharadwaj — Tue, 25 Aug 2026 05:01:05 +0000

Linux 7.3's merge window has produced the clearest security change of the cycle: the AF_ALG user-space crypto interface is now restricted by default, controlled by an allowlist, while new AES library APIs mark out the replacement path inside the kernel. Intel has proposed DRM Fabric, a vendor-neutral topology layer for accelerator interconnects, days after AMD posted its UALink enablement series. The open-source Etnaviv stack now runs YOLOX object detection on NXP i.MX 8M Plus class NPUs, RISC-V gains initial support for eleven ISA extensions in mainline, and a seven-branch stable/LTS release wave plus Buildroot updates make this a heavy patch-intake week.

The theme this week is boundaries moving. The kernel is narrowing what user space may touch (AF_ALG), widening what mainline abstracts (accelerator fabrics, NPU inference), and firming up what platforms must guarantee (RISC-V profile extensions). Each of those shifts lands on embedded teams as either a migration task or a planning signal.

In this edition

AF_ALG is restricted by default in Linux 7.3. A new sysctl limits the user-space crypto interface to an allowlist, and new AES library APIs define the in-kernel replacement path. — audit before you rebase
Intel proposes DRM Fabric. A vendor-neutral topology layer for scale-up accelerator interconnects, posted days after AMD's UALink enablement series. — long-term watch
YOLOX runs on the open Etnaviv NPU stack. New operations bring a stronger object-detection model to Vivante NPUs in the i.MX 8M Plus and Amlogic A311D. — evaluate now
RISC-V lands eleven ISA extensions in 7.3. Indirect CSR access, counter delegation, and QoS tagging move the platform toward server-class expectations. — planning signal
Seven stable/LTS kernels and a Buildroot release wave shipped in one day. 7.1.10 down to 5.10.266, plus Buildroot security point releases. — patch now

Linux 7.3 puts AF_ALG behind an allowlist

The crypto pull for Linux 7.3, sent by maintainer Herbert Xu and now merged, contains the change this audience should read first: a new af_alg_restrict sysctl that defaults to 1. With the default in force, AF_ALG — the socket interface that lets user-space programs drive the kernel's internal crypto algorithms — only accepts the algorithms on a curated allowlist. Eric Biggers authored the series, which also admits the additional ciphers cryptsetup needs and makes cbc(paes) available to privileged callers only.

This is the follow-through on a deprecation that has been building for several cycles. Linux 7.2 formally deprecated AF_ALG after zero-copy support and hardware offload had already been dropped; the maintainers have described the interface as a large attack surface with little remaining justification. 7.3 converts that position from an advisory into a default.

If your product uses AF_ALG, a kernel rebase to 7.3 can break it silently: the interface now defaults to allowlist-only operation. The failure mode is not a build error but a runtime one — socket or algorithm setup fails on a device that worked on 7.2.

The same pull also prunes embedded crypto drivers. The crypto_rng interfaces of the Allwinner sun8i-ce/sun8i-ss and NXP CAAM PRNGs were removed, the Qualcomm RNG moved fully under drivers/char/hw_random, and the Qualcomm qce crypto engine was marked BROKEN even as it gained runtime PM and interconnect-scaling work. Products on i.MX, Allwinner, or Qualcomm silicon that read random numbers or run crypto through these paths should re-check their configs on 7.3.

The constructive half of the story landed in a separate pull from Eric Biggers: library APIs for the AES modes the kernel actually uses — ECB, CBC, CBC-CTS, CTR, XCTR, XTS, GCM and CCM — with full documentation, wired into the traditional crypto API so the self-tests cover them. The old AES-GCM library is gone. Proof-of-concept conversions across kernel subsystems removed roughly 1,900 lines, and the stated plan is to migrate architecture-optimized AES code into the library in later cycles. In-kernel users get a simpler, faster path; user-space users are being pointed back at user-space crypto libraries.

How to check your device

Search your user space for AF_ALG use before you plan a 7.3 rebase, and inspect the new sysctl once you are on a 7.3 kernel:

raghu@techveda.org:~$ grep -rl "AF_ALG" rootfs/usr/ 2>/dev/null
raghu@techveda.org:~$ sysctl -a 2>/dev/null | grep af_alg

Common indirect users include libkcapi and OpenSSL engines configured for kernel crypto. If you find a dependency, migrate it to a user-space implementation, or plan to manage the allowlist deliberately and treat that as product configuration, not a default you inherit. Kernel-module authors using skcipher or AEAD for AES should track the new library APIs — that is where the maintained, optimized path is heading.

Intel proposes DRM Fabric for accelerator interconnects

Intel engineer Konstantin Sinyuk posted a twelve-patch RFC introducing "drm/fabric": vendor-neutral, protocol-agnostic topology infrastructure for scale-up accelerator interconnects. The model is a strict hierarchy — a fabric groups the endpoints of one interconnect instance, an endpoint is one accelerator attachment holding physical ports, a port reports lane capability and operational state, and a peer names the directly adjacent accelerator or switch port. The core records direct adjacency only; route computation, switch forwarding, and the data path stay with vendor drivers and fabric controllers. The uAPI is Generic Netlink rather than sysfs, with query and provisioning operations, a synthetic provider for testing, and KUnit plus netlink selftests in the series.

The timing matters. Days earlier, AMD's Alex Deucher posted a 95-patch series of about 7,000 lines enabling UALink — the open standard for linking up to 1,024 accelerators in one domain — in the AMDGPU driver, including remote memory access without copies and a dedicated address space with remote TLB shootdowns. Two vendors converging on mainline infrastructure for accelerator fabrics in the same fortnight means the interconnect layer is becoming ordinary kernel plumbing rather than a per-vendor SDK concern.

For embedded engineers this is a direction signal rather than a task. The pattern — netlink-based topology objects with vendor providers underneath — is the same shape as devlink in networking, and it is the shape multi-accelerator edge boxes will inherit when today's datacenter interconnects scale down.

What it means for platform teams

Nothing here is merged; an RFC can change shape completely. Track the discussion if you build systems with more than one accelerator, because the uAPI that emerges will be the stable interface your orchestration and monitoring tools consume. Teams evaluating UALink-class hardware should note that topology discovery, error surfacing, and provisioning are being standardized now — the review threads are where objections still count.

YOLOX object detection on the open Etnaviv NPU stack

Tomeu Vizoso, working in partnership with Ideas On Board, has brought YOLOX object detection up on the Etnaviv driver stack for Vivante VIP NPUs — the neural accelerators in the NXP i.MX 8M Plus and the Amlogic A311D. YOLOX, an Apache-2.0 model from Megvii, is a meaningful step up in capability from SSDLite MobileDet, the model the stack supported until now.

The interesting part is what had to be built. FullyConnected now runs directly on the NN convolution cores. Reshape, Split, and Concatenate are handled purely as metadata changes, costing no hardware cycles. ReLU is fused into the output stage, Absolute and Logistic run as lookup-table operations on the tensor-processing cores, Subtract is lowered to a convolution, and Transpose is either fused or executed as a TP operation. Feature maps in signed 8-bit integers were added as well, which for some models increases accuracy at identical computational cost.

Each of those is a general capability, not a YOLOX special case — the operation coverage of the open stack just widened for every model that needs those primitives. Open NPU stacks are moving from proof-of-concept classification demos toward the detection workloads products actually ship.

How to evaluate it

If you ship i.MX 8M Plus or A311D hardware, put the open stack on your benchmark board alongside the vendor SDK and measure your own model, not the demo. Check three things: whether your model's operations are now covered, what accuracy you get from signed INT8 feature maps, and what the end-to-end latency looks like against the blob-based path. A fully mainline inference stack removes an entire vendor-SDK dependency from your BSP lifecycle — that is worth a day of benchmarking to validate.

RISC-V in 7.3: eleven extensions and server-class plumbing

The RISC-V pull for 7.3, sent by maintainer Paul Walmsley and merged, adds initial definitions and discovery for eleven ISA extensions: Smcsrind and Sscsrind (indirect CSR access), Smcntrpmf (privilege-mode filtering for the cycle and instret counters), Ssccfg and Smcdeleg (counter delegation to supervisor mode), Zicclsm, Ziccamoa, Ziccif, Ziccrse and Za64rs (cache, atomicity, and reservation-set guarantees from the profile specifications), and Ssqosid — quality-of-service resource tagging, which also gains CPU context-switch support via the new srmcfg CSR handling.

Around the extensions, the pull carries platform work with a clear direction: restart and power-off through UEFI runtime services when available, ACPI table upgrade support for debugging (matching ARM64 and x86), a smaller CFI shadow-stack allocation (2GB down to 512MB of address space), TLB-flush improvements using ASIDs and the Svinval extension, and build-time extraction of vDSO offsets to cut boot overhead. XIP kernel leftovers were removed.

Read together, this is the plumbing a platform needs before serious server and industrial deployment: delegated performance counters, QoS partitioning, firmware-managed power control, and profile-level guarantees that software can rely on across vendors. The extension list looks obscure item by item; collectively it is RISC-V being fitted for the same expectations Arm servers met a decade ago.

What it means for SoC selection

When evaluating RISC-V silicon for products with multi-year lifetimes, ask vendors which of these extensions their cores implement — counter delegation and Ssqosid in particular separate application-class designs from microcontroller-class ones. For BSP teams, the profile guarantees (Zicclsm, Ziccamoa, Ziccif, Ziccrse, Za64rs) are exposed through the kernel's hwprobe interface, so user space can detect them without vendor-specific code. Devices shipping mixed workloads should watch Ssqosid: cache and bandwidth partitioning is how you keep an inference thread from starving your control loop.

A seven-branch stable wave and a Buildroot release day

On 23 August the stable series updated across every maintained branch in one wave: 7.1.10 on stable, and 6.18.46, 6.12.105, 6.6.153, 6.1.184, 5.15.217 and 5.10.266 on longterm. The same day, Buildroot shipped its 2026.08-rc2 release candidate together with two security bugfix releases, 2026.05.2 and 2025.02.17, covering both maintained stable branches; 2026.08-rc1 had arrived on 18 August. On the Yocto side, the 6.0.1 (Wrynose) and 5.0.17 (Scarthgap) point releases carry matching userspace fixes, including BusyBox CVE patches.

None of these releases is significant on its own, which is exactly why the wave gets missed. A coordinated multi-branch wave usually means fixes the stable maintainers wanted everywhere at once, and a build-system security release the same day means the userspace side of your image has pending patches too. The intake work is the story.

What it means for BSP teams

Diff the changelog of the branch you ship — networking, filesystem, and driver fixes in these waves regularly resolve field issues that were never reported upstream by name. Teams on 5.10 and 5.15 should confirm the projected end-of-life dates for those branches on kernel.org and have a migration plan that does not assume indefinite updates. Buildroot users on 2026.05 or 2025.02 should take the security point releases now, and start validating 2026.08-rc2 in CI so the stable release requires no extra work when it arrives. Treat the wave as one intake task: pin the new versions, rebuild, run your regression suite once.

References

— Raghu Bharadwaj

Want to go deeper than the news? TECH VEDA runs hands-on Linux kernel and embedded Linux training — see techveda.live.

Linux genpd Power Domains: Why a Rail Won’t Power Off?

Raghu Bharadwaj — Tue, 25 Aug 2026 04:59:00 +0000

A genpd power domain removes power only when every device attached to it is runtime-suspended, every subdomain is already off, and the governor agrees. When a rail stays on, the cause is one of the conditions genpd_power_off() checks before it calls the provider's ->power_off() callback. Those conditions are a device that never runtime-suspended, a subdomain still counted as on, an always-on flag, a system suspend in progress, a resume-latency constraint the governor will not accept, or the boot-time stay_on hold introduced in Linux 6.17. Start at /sys/kernel/debug/pm_genpd/pm_genpd_summary, which shows three of those conditions directly and narrows down the rest.

On most embedded SoCs, power is not gated per device. It is gated per block: one switch feeds the GPU and its MMU, another feeds the display controller together with the DSI host and the PHY. The Linux kernel models these blocks as genpd power domains. This article explains, from the code, how genpd power domains decide that a block may lose power, and how to find out which condition is holding your rail on.

It sits one level below an earlier post on the per-device layer, Runtime PM: Why Your Device Never Suspends. Runtime PM tracks whether one device is idle; genpd decides whether the shared rail feeding several devices may be switched off. They fail for different reasons.

Layer	What it decides
Device driver	Calls `pm_runtime_get()` and `pm_runtime_put()`.
Runtime PM (one device)	Is this one device idle? The answer appears as the runtime status column in `pm_genpd_summary`.
genpd (one domain, several devices)	May the shared rail be switched off? This is `genpd_power_off()`, and it is what the rest of this article is about.
Provider driver	The `->power_off()` callback drives the hardware sequence and waits for the acknowledgement.
SoC and PMIC	The rail actually drops. Nothing above this row can confirm that it did.

Where the genpd code lives

The data structures are in include/linux/pm_domain.h. The core logic, the governors and the debugfs interface are in drivers/pmdomain/ — the core in drivers/pmdomain/core.c, the in-tree governors in drivers/pmdomain/governor.c. The core moved there in Linux 6.8; on 6.7 and earlier it is at drivers/base/power/domain.c. Everything quoted below is from Linux 6.17, and one mechanism described later is new in that release. The framework needs CONFIG_PM_GENERIC_DOMAINS, which platform code selects rather than you enabling it in menuconfig; CONFIG_PM_GENERIC_DOMAINS_OF and CONFIG_PM_GENERIC_DOMAINS_SLEEP then default to y when OF and PM_SLEEP are set. The debugfs interface is guarded by CONFIG_DEBUG_FS.

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'PM_GENERIC_DOMAINS|CONFIG_DEBUG_FS='
CONFIG_PM_GENERIC_DOMAINS=y
CONFIG_PM_GENERIC_DOMAINS_SLEEP=y
CONFIG_PM_GENERIC_DOMAINS_OF=y
CONFIG_DEBUG_FS=y

Inside struct generic_pm_domain

A provider driver allocates a struct generic_pm_domain, fills in the two hardware callbacks, and registers it. These are the fields that decide power-off:

/* include/linux/pm_domain.h - fields discussed here, comments condensed */
struct generic_pm_domain {
        struct list_head        parent_links;    /* links where this domain is the parent */
        struct list_head        child_links;     /* links where this domain is the child */
        struct list_head        dev_list;        /* devices attached to this domain */
        struct dev_power_governor *gov;
        const char              *name;
        atomic_t                sd_count;        /* subdomains currently powered on */
        enum gpd_status         status;          /* GENPD_STATE_ON or GENPD_STATE_OFF */
        unsigned int            device_count;
        unsigned int            suspended_count; /* system suspend device counter */
        unsigned int            prepared_count;  /* suspend counter of prepared devices */
        bool                    stay_on;         /* stay powered-on during boot */
        int (*power_off)(struct generic_pm_domain *domain);
        int (*power_on)(struct generic_pm_domain *domain);
        unsigned int            flags;
        struct genpd_power_state *states;
        unsigned int            state_count;
        unsigned int            state_idx;       /* state the domain enters when off */
};

Registration is pm_genpd_init(genpd, gov, is_off), where the third argument states whether the hardware is already off at registration. Hierarchy is declared with pm_genpd_add_subdomain(parent, child). For Device Tree, the provider then calls of_genpd_add_provider_simple() for a single domain or of_genpd_add_provider_onecell() for several, and consumer nodes reference it with power-domains = <&pd 0> against the provider's #power-domain-cells. Attachment happens automatically during probe, but only for a device with exactly one power-domains entry: genpd_dev_pm_attach() counts the phandles and returns without attaching when the count is not one. A device listing two domains is silently left unmanaged unless its driver calls dev_pm_domain_attach_by_name(), which keys on power-domain-names, or devm_pm_domain_attach_list(). Nothing reports this, so it is worth checking early on a new board. Idle states come from a domain-idle-states property in the provider node; without it the domain gets one default state and can only ever report off-0.

Reading pm_genpd_summary

The genpd core creates a debugfs directory named pm_genpd holding a global file called pm_genpd_summary. Read it first on any board where power is higher than expected.

raghu@techveda.org:~$ sudo mount -t debugfs none /sys/kernel/debug
raghu@techveda.org:~$ sudo cat /sys/kernel/debug/pm_genpd/pm_genpd_summary

The two header lines and the separator are printed verbatim by summary_show(). An illustrative result on a board with a display domain and a GPU domain looks like this:

domain                          status          children        performance
    /device                         runtime status                  managed by
------------------------------------------------------------------------------
pd_display                      on                              0
                                                pd_dsi
    fd4a0000.display                active                      0           SW
pd_dsi                          on                              0
    fd4b0000.dsi                    active                      0           SW
pd_gpu                          off-0                           0
    fde60000.gpu                    suspended                   0           SW

The layout is not obvious on a first reading, because subdomains appear on their own indented line rather than beside the domain that owns them. The same output as a tree:

Node	State	Effect on power-off
`pd_display` (domain)	on	Held on by both rows below
`fd4a0000.display` (device)	active	Not runtime-suspended
`pd_dsi` (subdomain)	on	Counted in `sd_count`, checked first
`fd4b0000.dsi` (device)	active	Holds `pd_dsi` on
`pd_gpu` (domain)	off-0	Powered off, in idle state 0
`fde60000.gpu` (device)	suspended	Nothing blocking

The status column shows on, or off-N where N is state_idx, the index of the idle state the domain entered — a domain with several idle states will show off-0, off-1 and so on. Child domain names print on a continuation line indented by 48 characters, so they land under the children heading rather than on the domain's own row. Device rows are indented four spaces and carry the device's Runtime PM status, then its performance state, then HW or SW for hardware- or software-controlled mode.

Read the Runtime PM status closely. active, suspended, suspending and resuming are the normal values, but unsupported means disable_depth is non-zero: the driver never called pm_runtime_enable(), or disabled it again. Such a device can never satisfy pm_runtime_suspended(), so it blocks its domain permanently. Rule that out first.

Why genpd power domains refuse to power off

The decision for all genpd power domains is made in one function, genpd_power_off(). It returns early, without touching the hardware, under any of the following conditions. The checks fall into four stages, and the domain stays on the moment any one of them fails:

Stage	What is tested
1. Domain flags (no lists walked yet)	Already off · `prepared_count > 0` · `GENPD_FLAG_ALWAYS_ON` · `GENPD_FLAG_RPM_ALWAYS_ON` · `stay_on` · `sd_count > 0`
2. Subdomains (walks `parent_links`)	Every child must already be in its deepest state, that is `state_idx == state_count - 1`
3. Devices (walks `dev_list`)	Any device with `rpm_always_on` stops it outright; otherwise counts devices that are not runtime-suspended
4. Governor	`power_down_ok` must agree, then `sd_count` is tested once more
Transition	`GENPD_NOTIFY_PRE_OFF` notifiers, then the provider's `->power_off()`
On success	Sets `GENPD_STATE_OFF`, increments `usage`, then tries each parent domain
On failure	Increments `rejected` for that state and leaves the domain on

The individual conditions, in the order the code tests them:

The domain is already off (status is not on).
prepared_count > 0, meaning a system suspend is in progress for devices in this domain.
The domain has GENPD_FLAG_ALWAYS_ON, or has GENPD_FLAG_RPM_ALWAYS_ON (on except during system suspend).
stay_on is still set — the boot-time hold described in the next section.
sd_count > 0: at least one subdomain is powered on. This is tested again after the governor call, in case a subdomain began powering on in the meantime.
Any child domain is not in its deepest state, that is state_idx is below state_count - 1.
Any attached device has rpm_always_on set, which a driver requests with dev_pm_genpd_rpm_always_on().
More than one attached device is not runtime-suspended. The core counts devices where pm_runtime_suspended() is false, and also counts an IRQ-safe device attached to a domain that is not IRQ-safe, that is, one without GENPD_FLAG_IRQ_SAFE. Exactly one such device is tolerated only when the call came from the domain's own runtime-suspend path, because that device's status has not been updated yet.
The governor refuses. The common choice is simple_qos_governor, and the QoS in the name is the point: its default_power_down_ok does not predict how long the domain will be idle. It adds the state's power_off_latency_ns and power_on_latency_ns, then refuses if any attached device's effective PM QoS resume-latency constraint, or any subdomain's max_off_time_ns, is less than or equal to that total. It refuses because a consumer could not tolerate the wake-up delay, not because the idle window looked short. The governor that does predict idle duration is pm_domain_cpu_gov, for CPU domains under CONFIG_CPU_IDLE. A provider passing NULL to pm_genpd_init() has no governor, skips this check, and always uses state index 0.

If all of that passes, the core runs the GENPD_NOTIFY_PRE_OFF notifier chain and then calls the provider's ->power_off(). If either step fails, genpd increments states[state_idx].rejected and leaves the domain on. Note also that a provider with no ->power_off() callback at all is treated as a success: the domain is marked off in software with no hardware action. On success the core sets GENPD_STATE_OFF, increments the state's usage counter, then walks up through child_links and attempts to power off each parent domain in turn.

One point is frequently misunderstood: the comparison of suspended_count against device_count is not part of this path. That comparison lives in genpd_sync_power_off(), which runs only in the noirq and syscore stages of system suspend. Runtime power-off counts devices by walking dev_list.

The boot-time hold: stay_on and sync_state

Of all the conditions that keep genpd power domains powered, this is the one most likely to be unfamiliar — and the one to check your kernel version against first. The stay_on field, GENPD_FLAG_NO_STAY_ON and of_genpd_sync_state() are new in Linux 6.17. On 6.16 and earlier this mechanism does not exist and grepping for it will find nothing. Among the current long-term branches that means only 6.18.y carries it; 6.12.y, 6.6.y, 6.1.y, 5.15.y and 5.10.y do not, so most products shipping on a long-term kernel today will not see this behaviour at all.

pm_genpd_init() always calls genpd_set_stay_on(). Built with CONFIG_PM_GENERIC_DOMAINS_OF, it sets stay_on = true for any domain already powered on, unless the provider passed GENPD_FLAG_NO_STAY_ON; built without it, a second definition hard-codes the field to false:

#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
static void genpd_set_stay_on(struct generic_pm_domain *genpd, bool is_off)
{
        genpd->stay_on = !genpd_is_no_stay_on(genpd) && !is_off;
}
#endif

While stay_on is true, genpd_power_off() returns immediately. The purpose is to protect devices that firmware left running — a display that is already showing a splash screen, for example — from losing power before their driver has probed. The hold is released by the driver-model sync_state() callback, once every consumer of the provider has probed. The core clears the flag and immediately retries the power-off:

void of_genpd_sync_state(struct device_node *np)
{
        struct generic_pm_domain *genpd;

        if (!np)
                return;

        mutex_lock(&gpd_list_lock);
        list_for_each_entry(genpd, &gpd_list, gpd_list_node) {
                if (genpd->provider == of_fwnode_handle(np)) {
                        genpd_lock(genpd);
                        genpd->stay_on = false;
                        genpd_power_off(genpd, false, 0);
                        genpd_unlock(genpd);
                }
        }
        mutex_unlock(&gpd_list_lock);
}

Lifetime of the hold, on Linux 6.17 and newer:

Point in boot	What happens
Boot	Firmware left the domain powered. The provider registers it, `pm_genpd_init()` calls `genpd_set_stay_on()`, and `stay_on` becomes true.
Drivers probing	`genpd_power_off()` returns immediately every time. The rail is protected while consumers are still coming up.
Every consumer probes	The driver model calls `sync_state()`. `of_genpd_sync_state()` clears `stay_on` and retries the power-off straight away.
One never probes	`sync_state()` never runs, `stay_on` stays true, and the domain remains powered for the whole boot. A missing driver or a permanently deferred probe is enough.

The practical consequence: if one consumer of a genpd provider never probes — a missing driver, a failed regulator, a permanently deferred probe — sync_state() never fires, stay_on is never cleared, and the domain stays powered for the life of the boot. The default policy is the strict one — fw_devlink.sync_state=strict waits indefinitely for consumers to probe successfully — so fw_devlink.sync_state=timeout on the kernel command line is a quick way to test whether this is what you are looking at.

The symptom is a domain that reads on in pm_genpd_summary with every listed device showing suspended. Do not stop there: an always-on flag, a per-device rpm_always_on, a subdomain not in its deepest state, and a governor refusal all produce exactly the same output. The summary cannot separate them, which is why the next section and the checklist above matter.

The per-domain debugfs files

Alongside the global summary, genpd creates one directory per domain containing current_state, sub_domains, idle_states, active_time, total_idle_time and devices. A perf_state file appears only for providers that implement set_performance_state.

raghu@techveda.org:~$ ls /sys/kernel/debug/pm_genpd/pd_gpu/
active_time  current_state  devices  idle_states  sub_domains  total_idle_time
raghu@techveda.org:~$ sudo cat /sys/kernel/debug/pm_genpd/pd_gpu/idle_states
State          Time Spent(ms) Usage      Rejected   Above      Below
S0             412            37         2          0          0

Usage and Rejected are the useful pair. A rising Rejected count means the core decided to power off but the transition failed — usually the provider's ->power_off() callback returning an error, though a GENPD_NOTIFY_PRE_OFF notifier veto increments the same counter. Either way the next place to look is that callback. A Usage count of zero with Rejected also at zero means the domain was never asked to power off at all, which points back to the list above.

Above and Below record how often the chosen idle state was too deep or too shallow for the residency actually achieved, but they are only maintained by pm_domain_cpu_gov, which is the sole caller that sets reflect_residency. On a domain running simple_qos_governor or no governor they stay at zero, as above, so do not read anything into that. Unlike the summary, devices and sub_domains print one bare name per line with no header.

A practical order of investigation

Use this order when genpd power domains stay on and the cause is not yet known.

Read pm_genpd_summary and find the domain that is on when you expect it off.
Any device row that is not suspended is a Runtime PM problem in that driver, not a genpd problem. A row reading unsupported means runtime PM was never enabled for that device.
If every device is suspended, check the continuation line and confirm each subdomain is also off.
If devices and subdomains are all idle, the cause is one of the conditions the summary cannot show. Check them directly:

raghu@techveda.org:~$ cat /sys/kernel/debug/devices_deferred
raghu@techveda.org:~$ grep -rn 'GENPD_FLAG_ALWAYS_ON\|GENPD_FLAG_RPM_ALWAYS_ON' drivers/pmdomain/
raghu@techveda.org:~$ cat /sys/devices/platform/fde60000.gpu/power/pm_qos_resume_latency_us
raghu@techveda.org:~$ cat /sys/kernel/debug/regulator/regulator_summary

The first shows consumers still waiting to probe, which on Linux 6.17 is what keeps stay_on set. The second finds an always-on flag in the provider driver. The third shows a resume-latency constraint tight enough to make default_power_down_ok refuse; this attribute exists only when the driver called dev_pm_qos_expose_latency_limit(), so its absence does not rule the cause out. The fourth is the check most often skipped: off-0 means only that the core believes the domain is off. Confirm the rail actually dropped, through the regulator summary or a meter on the sense point, before concluding the software is correct.

If the domain does power off but not as often as expected, read idle_states. A rising Rejected count means the core did decide to power off and the transition failed, so the problem is in the provider's callback rather than in the checklist.

Working through power domains on real silicon is a large part of driver work, and it is covered in our Linux device drivers training.

Key takeaways

Runtime PM decides whether a device is idle; genpd power domains decide whether a shared rail may be switched off. Both must agree.
genpd_power_off() tests a fixed set of conditions in a fixed order. Match the symptom to one of them instead of guessing.
off-N in pm_genpd_summary is not an error — N is the index of the idle state the domain entered.
A domain that is on while all its devices are suspended has several possible causes that the summary cannot tell apart: an always-on flag, a per-device rpm_always_on, a governor refusal, or, on Linux 6.17 and newer, the stay_on hold.
simple_qos_governor refuses on PM QoS resume-latency constraints, not on a predicted idle window. That changes what you go looking for.
The Rejected counter in idle_states separates a core-level refusal from a transition that was attempted and failed.
genpd reporting off-0 means the core believes the domain is off. Confirm the rail separately.
The core moved to drivers/pmdomain/core.c in Linux 6.8; before that it is drivers/base/power/domain.c.

Frequently asked questions

What does "off-0" mean in pm_genpd_summary?
It means the domain is powered off and entered idle state index 0. The number after the hyphen is state_idx, so a domain with several idle states can show off-0, off-1 and so on. Only a powered-on domain prints a plain on.

Every device in my domain is suspended but the domain still reads "on". Why?
pm_genpd_summary cannot distinguish the remaining causes, so check them in turn: GENPD_FLAG_ALWAYS_ON or GENPD_FLAG_RPM_ALWAYS_ON in the provider driver, a device that called dev_pm_genpd_rpm_always_on(), a subdomain not in its deepest state, and a governor refusing on a resume-latency constraint. On Linux 6.17 and newer there is one more: the boot-time stay_on hold, which is released only when the provider's sync_state() callback runs after every consumer has probed.

What is the difference between GENPD_FLAG_ALWAYS_ON and GENPD_FLAG_RPM_ALWAYS_ON?
GENPD_FLAG_ALWAYS_ON keeps the domain powered at all times. GENPD_FLAG_RPM_ALWAYS_ON keeps it powered during normal operation but allows a power-off during system suspend, which runs through genpd_sync_power_off() instead. Both make genpd_power_off() return before it reaches the device and subdomain checks.

What is the difference between the Rejected and Usage counters in idle_states?
Usage counts the times the domain successfully entered that idle state. Rejected counts the times the core decided to power off but the transition failed, usually because the provider's ->power_off() callback returned an error, though a GENPD_NOTIFY_PRE_OFF notifier veto increments the same counter. A rising Rejected count points at the provider driver and its hardware sequence.

Kernel Cleanup Helpers: What Your LTS Actually Has

Raghu Bharadwaj — Mon, 24 Aug 2026 14:38:01 +0000

The kernel cleanup helpers — __free(), guard() and scoped_guard() — let the compiler release a resource when a variable leaves scope, replacing the goto ladder in a driver's error path. The core arrived in mainline 6.5 and is now in every supported longterm series, down to 5.10. Everything built on top of it arrived later and reached the stable series at point releases rather than at .0, so your major version number does not tell you what you have. On an old vendor tree the practical answer is guard() plus a locally-defined free helper.

Most embedded teams ship a longterm kernel and assume anything added to mainline in the last three years is out of reach until the next BSP jump. For the kernel cleanup helpers that assumption is wrong, but the correction is not simply "you can use them". The core macros are everywhere. The pieces you actually reach for — a free helper for the specific thing you allocated, a guard for a lock that can fail — landed at scattered point releases across the stable series, and a couple of the oldest trees still use a compiler standard that fights the coding pattern these helpers require. This is what that looks like on real trees.

The problem kernel cleanup helpers remove

The kernel documentation calls the "goto error" pattern notorious for introducing subtle resource leaks, and tedious and error prone to extend once a function already carries several unwind conditions. Concretely:

static int mydev_count_channels(struct mydev *priv)
{
        struct device_node *np;
        int ret, count;

        np = of_get_child_by_name(priv->dev->of_node, "channels");
        if (!np)
                return -ENODEV;

        mutex_lock(&priv->lock);

        ret = mydev_sync(priv);
        if (ret)
                goto out_unlock;

        count = of_get_child_count(np);
        if (!count) {
                ret = -EINVAL;
                goto out_unlock;
        }

        ret = count;

out_unlock:
        mutex_unlock(&priv->lock);
        of_node_put(np);
        return ret;
}

The release sequence sits at the bottom, separated from the acquisitions at the top. Adding a third resource means adding a label, placing a release call correctly within the existing sequence, and re-checking every existing goto. The usual way it breaks is smaller than that: someone adds an early-return check near the top, does not notice a resource is already held above it, and returns directly. The compiler accepts it and the refcount leaks quietly.

What the compiler does instead

The kernel cleanup helpers move that bookkeeping to the compiler, using the cleanup variable attribute that GCC and Clang both support. You attach a function to a variable; the compiler calls it when the variable leaves scope, on every exit path, including paths added later by someone who has not read the rest of the function.

DEFINE_FREE() creates the wrapper — a name, a type, and an expression using _T for the variable:

DEFINE_FREE(kfree, void *, if (_T) kfree(_T))

The test looks redundant, since kfree(NULL) is safe. It is there for the code generator. On an ownership-transfer path the pointer is handed to the caller and set to NULL first, so the compiler sees this:

tmp = p; p = NULL; if (p) kfree(p); return tmp;

Value propagation and dead-code elimination reduce that to return p; — the cleanup call disappears. Without the test the compiler cannot make the deduction and the call survives. That elimination applies to the success path specifically; on a real error path the release happens, as it must.

Three helpers exist for that transfer, because ownership is the one thing scope-based cleanup cannot infer: no_free_ptr(p) nulls the variable and returns the value (with must-check semantics); return_ptr(p) is the same as a return statement; and retain_and_null_ptr(p) covers the case where a function you called has already taken ownership.

A conversion that builds on old trees

Here is the same function converted. Note the helper definition — this is the part that matters for portability, and the reason is in the next section:

/* Define once, under a name no kernel header uses. */
DEFINE_FREE(mydev_of_node, struct device_node *, if (_T) of_node_put(_T))

static int mydev_count_channels(struct mydev *priv)
{
        struct device_node *np __free(mydev_of_node) =
                of_get_child_by_name(priv->dev->of_node, "channels");
        if (!np)
                return -ENODEV;

        guard(mutex)(&priv->lock);

        int ret = mydev_sync(priv);
        if (ret)
                return ret;

        int count = of_get_child_count(np);
        if (!count)
                return -EINVAL;

        return count;
}

The labels are gone, every error path is a plain return, and the release is stated where the resource is acquired. The early-return mistake stops being a mistake: a return added by someone unfamiliar with the function still drops the node reference and releases the lock.

The cleanup order is also correct. np is declared before the guard, so the compiler releases in reverse — mutex first, then the node reference — matching the original's tail exactly. That is safe here because dropping a node reference does not need the lock. When it does, the ordering becomes a trap, covered at the end.

guard() and scoped_guard() are not the same thing

guard(name)(args) holds the lock to the end of the enclosing scope. Written inside an if block, it releases at the end of that block, not at the end of the function — what you want if you were thinking about the block, a surprise if you were thinking about the function. The kernel documentation spells this out with an example and the explicit note that the lock is held for the remainder of the if block. scoped_guard(name, args) { … } binds the lock to the braces that follow it instead. Use guard() when the lock covers the rest of the function, scoped_guard() when the critical section is narrower.

Which kernels have which core helpers

Feature	Mainline	In the longterm series
`__free`, `guard`, `scoped_guard`	6.5	All — 5.10.y, 5.15.y, 6.1.y, 6.6.y, 6.12.y, 6.18.y
Conditional guards	6.8	6.6.y from 6.6.70; 6.12.y; 6.18.y. Absent from 5.10.y, 5.15.y, 6.1.y
`retain_and_null_ptr`	6.16	6.12.y in a late point release; 6.18.y
`ACQUIRE`, `ACQUIRE_ERR`	6.17	6.18.y only

The mainline column is from the release tags. The longterm column is a snapshot taken while writing, and stable branches keep moving — which is exactly why the conditional-guard row has to name a point release rather than a series. The 5.10.y, 5.15.y and 6.1.y headers are byte-identical to each other and to the one that shipped in 6.5: those three got the core and nothing since.

The subsystem helpers move separately

The table above covers include/linux/cleanup.h. It says nothing about whether the helper you actually need exists, and those live in the subsystem headers, on their own schedules.

The node-reference helper is the case in point, because it is the one an of_node conversion reaches for. DEFINE_FREE(device_node, …) arrived in mainline 6.9 — four releases after the core. It reached 6.6.y at 6.6.49 and 5.15.y only in the 5.15.21x range. It is not in 5.10.y at all. A vendor tree pinned anywhere below those points has the core macros and no node helper, so __free(device_node) does not compile even though guard() does.

The same unevenness shows up elsewhere. On 6.6.70 the mutex header offers conditional variants for trylock and interruptible acquires, but not the killable one; a file using scoped_guard(mutex_kill, …) fails there despite the conditional-guard check passing. On a 6.6.30 tree the slab header defines exactly one free helper, DEFINE_FREE(kfree, void *, if (_T) kfree(_T)) — no kvfree, no kfree_sensitive, and written in the plain-NULL form rather than the error-pointer-tolerant one used in current mainline.

Why the version number is not the answer

Conditional guards are in 6.6.y — from 6.6.70, not 6.6.0. A BSP pinned at 6.6.30 does not have them, and does not have the node helper either. Vendors pin at whatever point release they forked from, stable series pick features up continuously, and the gap between those two facts is where the time goes. There is no version arithmetic that resolves it. You have to look at the headers.

How to check your tree

raghu@techveda.org:~$ grep -oE 'define (DEFINE_GUARD_COND|scoped_cond_guard|ACQUIRE_ERR)' "$KDIR/include/linux/cleanup.h"
define DEFINE_GUARD_COND
define scoped_cond_guard

Two lines means conditional guards but no ACQUIRE family — a 6.12.y, or a 6.6.y at 6.6.70 or later. No output, with the header present, means the core only: 6.1.y or older, or a 6.6.x below 6.6.70. Current mainline prints five lines, because DEFINE_GUARD_COND is defined three times there under an argument-count dispatch.

For the subsystem helper, grep the header you actually need — and match both spellings, because lock guards are frequently declared through the DEFINE_LOCK_GUARD generators rather than DEFINE_GUARD directly:

raghu@techveda.org:~$ grep -oE 'DEFINE_(FREE|GUARD|LOCK_GUARD_[01])' "$KDIR/include/linux/of.h" "$KDIR/include/linux/mutex.h"

Searching only for DEFINE_GUARD returns nothing for mutex.h on current mainline, which would wrongly suggest guard(mutex) is unavailable.

Writing one file that builds on three kernels

This is the case the helpers make awkward, and it splits in two.

Macro features can be tested directly, because that is what they are:

#ifdef scoped_cond_guard
        scoped_cond_guard(mutex_try, return -EBUSY, &priv->lock) {
                /* ... */
        }
#else
        if (!mutex_trylock(&priv->lock))
                return -EBUSY;
        /* ... hand-written unwind ... */
        mutex_unlock(&priv->lock);
#endif

Free helpers cannot. __free(device_node) expands to __cleanup(__free_device_node), and __free_device_node is a static function generated by the macro, not a macro itself — so there is nothing for #ifdef to test. Defining your own DEFINE_FREE(device_node, …) to fill the gap collides on trees that already have it.

The way out is the one used in the conversion above: define your own helper under a name no kernel header will ever use, and use it unconditionally on every tree. It costs one line and removes the version question entirely.

DEFINE_FREE(mydev_of_node, struct device_node *, if (_T) of_node_put(_T))

The old-toolchain problem on 5.10 and 5.15

One more obstacle applies to the two oldest longterm series, and it is the first thing that will stop a conversion there. Both 5.10.y and 5.15.y still build with -std=gnu89 and -Wdeclaration-after-statement. Scope-based cleanup requires declaring a variable at the point of acquisition, part-way through a function — which is exactly the pattern that warning exists to reject. Every converted function produces warnings on those trees, and a BSP built with -Werror will fail outright.

From 6.1.y onward the kernel builds -std=gnu11 and the warning is gone, so the problem is confined to the two oldest series. If those are in your matrix, the honest position is that the helpers are available but the coding style they require is not, unless you can change the build flags.

Which branch to actually convert

Put those constraints together and the advice narrows usefully.

Convert the branch you intend to upstream. Mainline driver code uses these helpers as a matter of course now, and a submission that hand-rolls unwind paths where a guard would do will attract review comments. That branch has every helper, the newest toolchain, and a reviewer population that expects the style.

Leave the product branches alone unless you have a specific reason. Rewriting every error path in a driver that takes vendor patches on one branch and upstream fixes on another turns clean cherry-picks into three-way merges for the next two years, and on 5.15 you are fighting the compiler standard as well. The leak risk being avoided is real but small; the merge cost is certain.

Where you do convert on an older tree, guard() is the safe half. It is present everywhere, needs no subsystem support, and replaces the most error-prone part of the unwind — the lock. Pair it with a locally-defined free helper and the file stays portable.

The ordering trap

Cleanup functions run in reverse order of definition — last defined, first released. That is usually right, since it mirrors acquisition order. It goes wrong when a function holds a lock and also owns an allocation whose release needs that lock held: if the allocation is declared before the lock is taken, it is released after the lock drops, outside the protected region. The kernel documentation walks through exactly this bug and annotates the offending call as happening with no lock held.

The fix is a habit. Define and initialise each resource in one statement, in the order the resources are acquired, rather than grouping declarations at the top of the function. The __free(...) = NULL-at-the-top pattern is what creates the interdependency, and the documentation recommends against it for that reason. The related rule is not to mix goto and cleanup helpers in one function — a goto can jump between scopes in ways the cleanup attribute does not account for, so convert every resource in a routine or leave the routine alone.

Key takeaways

The core kernel cleanup helpers are in every supported longterm series, down to 5.10. guard() in particular is safe to adopt anywhere.
Everything else — conditional guards, and the per-subsystem free helpers — arrived at scattered point releases. A major version number does not tell you what you have.
__free(device_node) is unavailable on most pinned vendor trees. Define your own helper under a private name and the problem disappears.
#ifdef works for macro features such as scoped_cond_guard, but not for __free helpers, which expand to a generated static function.
5.10.y and 5.15.y build -std=gnu89 with -Wdeclaration-after-statement, which fights the declare-at-acquisition pattern the helpers require.
Convert the branch you intend to upstream; leave product branches alone unless there is a reason beyond tidiness.
Cleanup runs in reverse order of definition, so declare each resource in acquisition order, and never mix goto with cleanup helpers in one function.

References

I teach Linux kernel, device driver and embedded Linux engineering at TECH VEDA. If you work across several vendor kernels and want to go deeper on driver internals, that is what the Linux Device Drivers programme covers.