DEV Community: CodeWithRomi

160 NVMe Drives in a 4U Box: Inside Supermicro's ASG-4116S-NU160R

CodeWithRomi — Sat, 24 Oct 2026 14:54:49 +0000

At FMS 2026 (the Future of Memory and Storage conference, formerly Flash Memory Summit), Supermicro pulled back the curtain on the ASG-4116S-NU160R, a 4U server crammed with 160 NVMe SSD bays. That number is not a typo. Getting that many drives into a standard 4U chassis is an engineering challenge worth understanding, whether you are designing storage infrastructure or just keeping an eye on where the industry is heading.

💡 Why This Matters

Most high-density NVMe servers top out somewhere in the 24 to 48 drive range for a 2U or 4U footprint. Supermicro pushing to 160 bays in 4U changes the math on petabyte-scale flash deployments. Fewer servers mean fewer network ports, fewer power feeds, and less management overhead. For hyperscalers and large storage clusters, that consolidation is a direct cost reduction.

It also shows that density does not require a new drive form factor. Rather than moving to E3.S or other EDSFF drives, Supermicro sticks with 2.5-inch U.2 SSDs and borrows the layout of a top-loading SAS storage server, using PCIe switches where that design would use SAS expanders. The ASG-4116S-NU160R is a preview of what dense all-flash arrays can look like at the server level rather than the appliance level.

🏗️ The Architecture Behind 160 Bays

Fitting 160 NVMe drives into 4U requires a few design choices happening in parallel. Supermicro confirmed at FMS 2026 that the drives are 2.5-inch U.2 NVMe SSDs, not a smaller EDSFF form factor. It reaches 160 bays in 4U with a two-level slide-out tray design: the top tray carries the hot-swap SSDs, and the bottom tray holds the CPU, memory, and networking.

The connectivity side is the harder problem. NVMe drives connect over PCIe, and PCIe lanes are a finite resource. Running all 160 drives at four lanes each would take 640 PCIe lanes, far more than a single CPU provides. Supermicro solves this with PCIe switches in the center of the chassis, fanning out from the host CPU to the drive bays the way SAS expanders do in a top-loading SAS server. Supermicro lists the 160 U.2 bays as PCIe 4.0 x1, so each drive gets a single Gen4 lane rather than the usual x4. The switch chips have not been named, so check the official product page and the ServeTheHome coverage at the source link below for updates.

Power and thermals are the other constraint. 160 NVMe drives all active under load generate substantial heat. Supermicro's spec sheet lists hot-swappable rear fans and redundant (1+1) 2600W Titanium-level power supplies. ServeTheHome estimates roughly 130 to 137W per petabyte of storage at the outlet when the system is filled with 122TB-class drives.

📐 Form Factor and Drive Standards

The small form factor NVMe ecosystem is worth knowing if you are evaluating platforms like this:

E3.S: An EDSFF standard designed specifically for data center density. About 76mm by 112.75mm (7.5mm thick for 1T drives), with a native PCIe connector, NVMe, and hot-swap support. It uses a different connector and shape, so it does not fit 2.5-inch U.2 bays.
E1.S: Even smaller than E3.S, used in ultra-dense JBOF designs. Lower capacity per drive but maximum bays per rack unit.
M.2 (22110): Sometimes used in density builds but limited in hot-swap support and thermal headroom at high drive counts.
U.2 / U.3: The workhorse for enterprise NVMe. The ASG-4116S-NU160R uses 2.5-inch U.2 drives. Supermicro fits 160 of them in 4U with a two-level slide-out tray design rather than moving to a smaller form factor.

Supermicro's product page confirms the design: 160 top-loading hot-swap 2.5-inch NVMe bays running at PCIe 4.0, four front hot-swap E1.S PCIe 5.0 x4 bays, a single AMD EPYC 9005 series processor, up to 6TB of DDR5 across 12 DIMM slots, and three PCIe 5.0 x16 slots for networking or DPUs. See the official ASG-4116S-NU160R product page and the ServeTheHome coverage for the full datasheet.

🖧 Where This Fits in a Storage Stack

A 160-bay NVMe server is not a general-purpose compute node. It is purpose-built for specific workloads:

Distributed storage clusters running Ceph, MinIO, or similar software-defined storage layers
AI training dataset storage where random read throughput matters more than latency
Database tiering where the entire working set needs to live on flash
Video ingest and media asset storage at scale
NVMe-oF (NVMe over Fabrics) targets serving compute nodes over Ethernet or InfiniBand

For homelab use, this is firmly in the "interesting to know about" category rather than the "go buy one" category. But the architectural patterns it uses, PCIe switching in place of SAS expanders and slide-out service trays, show up at smaller scales too.

🔧 What to Watch For in the Spec Sheet

Supermicro's product page is already live, and these are the numbers worth paying attention to:

Total PCIe bandwidth: Supermicro lists the U.2 bays as PCIe 4.0 x1, and PCIe switches share the CPU's lanes across all 160 drives, so per-drive throughput is well below a typical x4 U.2 link. That trade-off favors capacity over peak per-drive speed.
Drive form factor: confirmed as 2.5-inch U.2, with four E1.S bays on the front panel.
Host interface: Onboard networking is just two 10GbE RJ45 ports (Broadcom BCM57416), so plan on adding NICs or DPUs in the three PCIe 5.0 x16 slots to move data at a speed that matches 160 NVMe drives.
Supported CPUs: a single AMD EPYC 9005 series processor (up to 360W TDP) drives the platform. The E1.S bays and expansion slots are PCIe 5.0, while the 160 U.2 bays run at PCIe 4.0 behind PCIe switches. Lane count from the CPU constrains everything downstream.
Redundancy: Drives and rear fans are hot-swap, and the two 2600W Titanium power supplies are 1+1 redundant. Whether the drive backplane and PCIe switch layer have any redundancy is worth asking Supermicro directly.

🛡️ Security Considerations at Scale

When you consolidate 160 NVMe drives into one chassis, the blast radius of a single point of failure goes up. A few things to think through before deploying dense storage like this:

Enable self-encrypting drive (SED) support if the drives offer it, so physical drive removal does not equal data exposure.
Plan your erasure coding or replication factor in software-defined storage to tolerate the loss of an entire chassis, not just a single drive.
Secure BMC/IPMI access on the management interface. A chassis this dense is a high-value target for ransomware via out-of-band management.
Keep firmware update cadence tight. NVMe drive firmware vulnerabilities are rare but real, and patching 160 drives manually is painful without automation.

✅ Wrap Up

The Supermicro ASG-4116S-NU160R is a signal, not just a product. It shows that all-flash density at the server level is catching up to what purpose-built appliances have offered, and that PCIe switching plus a top-loading 2.5-inch U.2 layout is the architecture getting it there. Whether you are planning a petabyte-scale Ceph cluster or just tracking where storage hardware is heading, this is worth bookmarking.

For confirmed specs and drive form factor, go straight to the Supermicro product page and the full ServeTheHome coverage at servethehome.com.

Related from the lab

Originally published at codewithromi.com.

Pi 5 RAM Swaps Have Been Firmware-Locked Since 2024. Here's What That Actually Means

CodeWithRomi — Sat, 24 Oct 2026 14:54:35 +0000

Raspberry Pi enthusiasts have been quietly upgrading the RAM on Pi 5 boards for a while now. The process involved swapping out the LPDDR4X memory package, and for the technically adventurous, it worked. It stopped working two years ago. The restriction is not new: Pi 5 bootloader EEPROM releases dated 2024-09-23 and later refuse to bring up memory that does not match what the board was provisioned with at the factory, and a board with a swapped chip fails SDRAM initialisation and does not boot. What is new is that the behaviour has finally been documented in public. One naming note: this is Raspberry Pi Ltd, the trading company, not the Raspberry Pi Foundation.

Jeff Geerling wrote it up on 21 September 2026, which is why it is being discussed now. His own summary of the timing: "The firmware update that disabled RAM swaps was actually from late 2024, two years ago." Let's break down what the firmware checks, what the changelog does and does not say about it, and what your options actually look like.

💡 Why This Matters

Raspberry Pi lists the Pi 5 in 1 GB, 2 GB, 4 GB, 8 GB and 16 GB variants, all LPDDR4X-4267. For users who wanted more RAM than their board shipped with, unofficial chip swaps were an appealing path. It kept older boards in service and gave power users more headroom for workloads like running local AI models, containers, or memory-intensive self-hosted apps. It was also, according to Raspberry Pi, being abused at the other end of the market: the company's position is that resellers were buying cheap low-RAM boards, soldering larger chips of unknown provenance onto them, and selling them as factory 8 GB units.

Firmware-level lockdowns change that calculus entirely. When the bootloader refuses to initialise memory that does not match the board's factory provisioning, the board does not boot at all. The reported symptom is the green ACT LED flashing eight times in a repeating pattern, which is the SDRAM failure code. This is not a soft restriction you can work around with a config file tweak.

🔧 What the Firmware Actually Does

Modern SBCs rely on their firmware to train and initialize RAM at boot. The firmware contains timing parameters, voltage settings, and compatibility tables specific to the memory chips the manufacturer has validated. When Raspberry Pi updates that firmware to check against an approved list of memory configurations, any chip not on that list can fail initialization entirely or be flagged and rejected.

The practical result is that a board that previously booted fine after a RAM swap stops booting. The restriction lives below the OS, so there is no Linux-side fix. One important caveat on sourcing: the rpi-eeprom changelog entry for 2024-09-23 says only "SDRAM performance tuning", "Allow BANKLOW to be configured by SDRAM_BANKLOW parameter" and "Manufacturing test updates". It never mentions a memory check, so the changelog will not tell you this happened. The clearest primary evidence is rpi-eeprom issue 761, filed 15 October 2025 by a user whose Pi 5, sold as 8 GB, failed to boot on every EEPROM release from 2024-09-23 onward and booted fine on 2024-09-10. Raspberry Pi engineers asked for vcgencmd otp_dump, concluded from the OTP provisioning that the board had been modified after manufacture, and said such a board "won't cope with any changes in firmware / otp provisioning and is stuck on a pre-2712D0 firmware release with old/test SDRAM code." A Raspberry Pi engineer has also stated the policy directly on the company forums, in this thread.

🤔 Why Would Raspberry Pi Do This?

There are a few plausible reasons, and they are not mutually exclusive.

Stability and support: Unofficial RAM swaps can cause hard-to-diagnose instability. Restricting unsupported configurations reduces support burden.
Supply chain and binning: Memory chips are binned and validated for specific performance envelopes. A chip that works in one context may not meet the timing tolerances Raspberry Pi has tested against.
Revenue protection: Selling higher-RAM SKUs is a meaningful part of the business model. Unofficial upgrades cut into that, particularly now that the Pi 5 is widely available.
Per-part memory tuning: The rpi-eeprom changelog is full of entries adding timings for specific memory configurations, including "Improved SDRAM refresh timings for Pi5 16GB" and a Broadcom SDRAM firmware bump "to support additional memory variants." The bootloader has to know which part is actually fitted. That also explains the part people find hardest to accept: a same-capacity chip pulled from another Pi 5 can still be rejected, because the check is not only about size. This has nothing to do with RP1, the Pi 5's I/O controller, which does not sit on the memory interface at all.

None of these reasons are inherently sinister, but the combined effect is a more closed platform than many in the community expected from Raspberry Pi.

🛡️ What This Means for Your Homelab

If you are running a Pi 5 with an unofficial RAM upgrade and it is working today, the immediate risk is a firmware update breaking your setup. Here is how to think about it.

If your swapped board works today, you are already running old firmware. The only bootloader releases that tolerate a swapped chip are 2024-09-10 and earlier, so a board still running a RAM mod is two years behind on bootloader fixes. There is no release channel that keeps you both current and modded. The real choice is old firmware or a factory memory configuration, and it is worth knowing which one you have picked.

To check your current EEPROM version and what update is staged:

sudo rpi-eeprom-update

The release channel is not set with rpi-eeprom-config. That command edits the bootloader's own configuration, things like BOOT_ORDER. The channel is the FIRMWARE_RELEASE_STATUS variable in /etc/default/rpi-eeprom-update:

sudo nano /etc/default/rpi-eeprom-update

In the current rpi-eeprom package that variable defaults to default, and latest is the other channel; the older critical, stable and beta names are historical and no longer what the tooling uses. The same file also holds RPI_EEPROM_ALLOW_DOWNGRADE and STRICT_MIN_VER_CHECK, which matter if you ever need to go backwards. Checked against the rpi-eeprom repository on 29 September 2026.

Going backwards is getting harder. The 2026-09-23 bootloader release added a manufacture-version check: "If the MFG version of the image to update is less than MFG version in OTP then recovery.bin and self update will both fail to write the eeprom update", with bootloader_allow_mfg_downgrade=1 in config.txt as the documented bypass. On newer boards whose minimum version is burned into OTP, rolling all the way back to a 2024 bootloader is no longer a simple reflash. That change landed six days before this post went up, so treat it as current rather than settled.

Consider your upgrade path instead. If you need more RAM for your workload, the official 16 GB Pi 5 is a real option. It costs more than a DIY swap, but it comes with full firmware support and no risk of a future update refusing to initialise your memory.

🧰 Alternatives If You Need More Headroom

If the Pi 5 RAM ceiling is genuinely limiting your homelab, here are paths that do not depend on unofficial hardware modifications.

Raspberry Pi 5 16 GB: The official high-RAM option. Full firmware support, no risk from future updates.
Swap space on fast storage: Not a substitute for real RAM, but an NVMe-backed swap partition on a Pi 5 with an M.2 HAT can extend effective memory for workloads that tolerate it. Add a swap file on your NVMe mount point rather than the SD card.
Move the workload: For memory-intensive tasks like running local LLMs, a small x86 mini PC or a used workstation server often makes more sense than pushing a Pi to its limits.
Orange Pi, Rock Pi, or other SBCs: The broader ARM SBC ecosystem has boards with more RAM and less restrictive firmware policies. They come with their own trade-offs around community support and software compatibility.

🔍 The Bigger Picture

This situation highlights a tension that has been building in the SBC space. As these boards become more capable and more commercially successful, the platforms inevitably become more controlled. The original Pi was almost impossibly open at the hardware level. The Pi 5 is a far more sophisticated piece of hardware, and sophistication tends to come with tighter firmware control.

That does not make the Pi 5 a bad board. It is still one of the most capable and well-supported SBCs you can buy. But it does mean that the "buy a cheap board and hack it into something bigger" model has real limits now. Plan your Pi 5 deployments around the official RAM tiers, and treat any unofficial hardware modification as something that may stop working after a firmware update.

✅ Wrap Up

Raspberry Pi's firmware-level RAM lockdown on the Pi 5 closed the door on a useful, if unofficial, upgrade path, and it did so in September 2024. If you have a working unofficial RAM swap today, you are on a two-year-old bootloader and any update will end that. If you are planning a new Pi 5 build that needs more than 8 GB, buy the 16 GB model and skip the uncertainty entirely.

Jeff Geerling's 21 September 2026 post is the place to start on the community side. For the primary material, rpi-eeprom issue 761 is where Raspberry Pi engineers describe the check themselves, and the Pi 5 bootloader release notes are worth watching for future changes, with the caveat that this particular behaviour never appeared in them.

Related from the lab

Originally published at codewithromi.com.

How Signal Made the Phone Number Optional, and What the Zero-Knowledge Proof Actually Protects

CodeWithRomi — Sat, 24 Oct 2026 14:54:15 +0000

Signal has long had a privacy contradiction at its core: the most secure mainstream messenger still required a phone number to sign up. That number became a de facto identity anchor, linkable to you even if you never shared it with contacts. Signal has now added an optional way around that, and zero-knowledge proofs (ZKPs) are part of it, though not in the way most coverage suggests. Phone-numberless registration landed in the Signal Android 8.28.0 beta on 16 September 2026. It is optional, Android only, and it costs a one-time in-app purchase: 3 USD in the United States, with per-country differences, checked 29 September 2026. The zero-knowledge proof is not what replaces the phone number. It is what stops Signal from linking that purchase to the account it creates. This post breaks down what Signal actually built, and what self-hosters can and cannot take from it.

💡 Why This Matters

Phone numbers are lousy identifiers for a private messaging app. They are tied to real-world identity through carriers, they can be ported or spoofed, and they hand a centralized third party leverage over who can register. Signal has been aware of this tension for years, and their community forums have tracked the demand for phone-number-free registration since at least 2018.

What shipped is narrower than proving eligibility with a zero-knowledge proof. A numberless account is identified by an Account ID and an Account Key, which Signal Android lead Greyson Parrelli describes as "a username and password." Nothing is proved about the person at all. The anti-spam control is the fee, and the zero-knowledge proof exists so that the payment cannot be tied back to the account. That is still a real change to the trust model, but it is worth being precise about which part the cryptography is doing.

🔍 What Zero-Knowledge Proofs Actually Do Here

A zero-knowledge proof lets one party (the prover) convince another party (the verifier) that a statement is true, without revealing anything beyond the truth of that statement itself. In Signal's registration flow, the proof is not about you. It is about the payment. Signal's own description, from Parrelli's 16 September 2026 post on the Signal Community forum: "The payment uses the same zero-knowledge proofs our donation system uses so that there is no link between your payment and your account."

The machinery being reused is the anonymous credential code behind Signal's donation badges, which lives in the zkgroup and zkcredential crates inside libsignal. Note what is not happening: nothing verifies a phone number, and nothing verifies a username either. Registration simply mints an Account ID and Account Key. The credential work is confined to the purchase, so that redeeming it proves a payment was made without identifying which payment.

Concretely: you never give Signal a phone number in this flow, because no number is involved at any point. One caveat that the cryptography does not address. The purchase currently has to go through a Play Store in-app payment, so Google still sees a Signal purchase on your account even though Signal cannot link it to the Signal account. Signal says more payment methods are planned, and that a device without Play Services cannot buy an account today.

🏗️ The Architecture in Plain Terms

Here is the rough flow, based on how ZKP-based credential systems like this typically work:

Credential issuance: A trusted issuer issues a blinded credential to the user. The issuer signs it without seeing the full content. Signal is not federated, so in Signal's case the issuer is Signal.
Proof generation: The client uses the credential to generate a zero-knowledge proof that says "I hold a valid credential issued by this authority" without revealing the credential itself.
Server-side verification: Signal's registration endpoint verifies the proof cryptographically. If it checks out, registration proceeds. No phone number is logged.
One-time use: The proof is typically bound to the session or includes a nullifier so the same credential cannot register multiple accounts.

One-time redemption matters for abuse prevention in any scheme like this. Read the four bullets above as a description of the general pattern rather than of Signal's registration endpoint, because in Signal's case the abuse control is the price, not the cryptography. Parrelli's stated reason for charging anything at all: "Spammers, unfortunately. If it was free, spammers would acquire massive numbers of these accounts and ruin our network." Signal has also said it may raise the price if numberless accounts turn into a spam vector.

📋 Status as of 29 September 2026

What is actually shipping, from Signal's own release notes and forum posts rather than press coverage:

Platform: Android only, and still in beta. It arrived in the 8.28.0 beta on 16 September 2026, and the 8.29 beta notes on 23 September 2026 say Signal is "keeping phoneless registration in beta for another week to get more testing." Signal iOS release notes through 8.30.0 do not mention it, and Signal Desktop has no standalone registration at all.
Cost: a one-time in-app purchase, 3 USD in the United States, with prices differing by country. Play Store in-app payment is the only method at the moment, so a device without Play Services cannot buy an account.
Recovery: none. The Account ID and Account Key are the only credentials, and Signal states plainly that there is no other recovery mechanism if you lose them. These accounts have no PIN, because the Account Key replaces it.
Reachability: a username is optional, and without one the account is, in Signal's words, "completely unreachable" except in conversations you start yourself.
Existing accounts: you cannot remove the phone number from an account you already have. Signal says that may become possible later.
Two-factor: TOTP now, with passkeys and hardware keys promised. It is not yet available on phone number accounts.

🧰 What Self-Hosters Can Learn From This Pattern

You do not need to be Signal to apply this pattern. The ZKP credential model is useful any time you want to gate access to a service without storing identifying data about your users. Here are a few real scenarios where this architecture applies:

Invite-only self-hosted apps: Issue blinded invite tokens. Users redeem them with a ZKP so you know the token is valid but cannot trace which token a specific account used.
Age or membership verification: Prove "user is over 18" or "user is a paid member" without storing birthdate or subscription ID server-side.
Rate limiting without tracking: Issue anonymous rate-limit tokens. Each token proves the holder is allowed N requests without tying requests back to an IP or user ID.

Libraries worth looking at if you want to experiment with this pattern include zk-kit (JavaScript/TypeScript), bellman and arkworks (Rust), and snarkjs for browser-compatible proof generation. For the credential-issuance side, Signal's own zkgroup is open source and worth reading even if you do not use it directly. Check each project's documentation for current API stability before building on them.

🛡️ Security Considerations

ZKPs are not magic. There are real failure modes to understand before you rely on this pattern.

Trusted setup: Some ZKP systems (like Groth16) require a trusted setup ceremony. If that ceremony is compromised, proofs can be forged. Systems using transparent setups (STARKs, or PLONK with a universal SRS) reduce this risk.
Credential theft: If an attacker steals a user's credential before it is blinded, the ZKP does not help. The issuance step still needs to be secured.
Implementation bugs: A ZKP circuit that does not correctly encode the constraint you think it does will verify invalid proofs. Formal audits of ZKP circuits are expensive and specialized. Do not roll your own for production without one.
Metadata leakage: Even with a valid ZKP, timing, IP addresses, and request patterns can still de-anonymize users. Layer your ZKP registration behind Tor or a VPN if anonymity is the goal.

🔧 Practical Next Steps If You Want to Explore This

If you want to get hands-on with ZKP-based authentication, here is a reasonable starting path:

Step 1: Read Signal's anonymous credential code. The standalone signalapp/zkgroup repository is archived and GitHub now points it at signalapp/libsignal, so start there. The rust/zkgroup and rust/zkcredential crates are where credential issuance and verification live. Checked 29 September 2026.

Step 2: Run through the snarkjs tutorial to build a trivial ZKP circuit locally. It uses Circom for the circuit definition and outputs a Solidity or JavaScript verifier. The two tools do not install the same way:

# snarkjs does ship on npm
npm install -g snarkjs

# circom does not. The "circom" package on npm is circom 1,
# which upstream marks deprecated. circom 2 is a Rust
# compiler you build from source:
#   https://github.com/iden3/circom

Step 3: Design your constraint. Write out in plain English what your proof needs to show, for example "the prover knows a value x such that hash(x) equals a public commitment." Translating that into a Circom circuit is the hard part, and the circom documentation is the right reference. Use whichever documentation link the iden3/circom README currently gives, because docs.circom.io failed to resolve from two separate clients when this was checked on 29 September 2026.

Step 4: Wire up your verifier to your registration or access endpoint. The verifier function takes a proof and public inputs and returns true or false. That boolean is your gate:

// Pseudocode for a Node.js endpoint
const { groth16 } = require("snarkjs");

app.post("/register", async (req, res) => {
  const { proof, publicSignals } = req.body;
  const vKey = require("./verification_key.json");
  const valid = await groth16.verify(vKey, publicSignals, proof);
  if (!valid) return res.status(403).json({ error: "Invalid proof" });
  // Proceed with account creation
});

Step 5: Add nullifier tracking. Store the nullifier from each proof in a database table. On each registration attempt, check that the nullifier has not been used before. This prevents credential reuse without revealing who the credential belongs to:

-- PostgreSQL example
CREATE TABLE used_nullifiers (
  nullifier TEXT PRIMARY KEY,
  used_at   TIMESTAMPTZ DEFAULT now()
);

✅ Wrap Up

Signal has not dropped the phone number requirement. It has added an optional, paid, Android-beta alternative to it, and free phone number registration remains the default path. What the feature does demonstrate is narrower and still useful: an anonymous credential can sit between a payment and an account so that the two cannot be correlated, even by the service that issued both. That is the principle worth borrowing, wherever you currently store an identifier only to prove that someone paid or was invited.

The tooling is approachable enough that a small homelab project or self-hosted app can implement a basic ZKP gate without a cryptography PhD. Start with snarkjs, read the zkgroup source for inspiration, and design your constraints carefully. The hardest part is not the math, it is being precise about exactly what you need to prove.

Related from the lab

Originally published at codewithromi.com.

Building a Linux GPU Driver for the M4 Mac mini in One Month, Mostly With Coding Agents

CodeWithRomi — Sat, 24 Oct 2026 14:52:53 +0000

Apple Silicon is powerful hardware, but Linux support on it has always been a community-driven uphill battle. In September 2026 two developers, Cody Ho and Niklas Sheth, published a working, OpenGL ES 3.0 conformant Linux GPU driver for the M4 Mac mini and the MacBook Neo, built in about a month. Two facts frame how to read it, and both are missing from most coverage. First, the bulk of the code was written by LLM coding agents: Ho used Codex for the kernel and firmware reverse engineering and Claude for the user-space work, and he describes the result as "likely the first ever fully LLM-written GPU driver." Second, it ships in Gravity Linux, a Fedora remix that was forked from Asahi Linux over precisely that question. Asahi's contribution policy forbids generative AI; Gravity's permits it under disclosure and provenance rules. Neither of those facts settles whether the work is good, but you cannot read the project accurately without them.

💡 Why This Matters

Most people running Linux on Apple Silicon hardware are doing it through Asahi Linux, which has made incredible progress on M1 and M2 chips. The M4 generation is newer territory. Getting GPU acceleration working on Linux means the difference between a usable desktop and a machine that renders everything in software, burning CPU cycles on tasks the GPU should be handling.

Beyond the Apple Silicon story, this project is a case study in something narrower: what coding agents do when pointed at a hard reverse engineering problem that already has an exhaustive test suite attached. The authors credit that suite directly, writing that "the hardest and most sensitive part of working with LLMs, giving them good tests to ground them, is already done for us" by the Khronos conformance tests.

🏗️ The Core Strategy: Clean-Room RE, Driven by Agents

It is tempting to assume the M4 driver is the Asahi driver patched for a newer chip. It is not. The write-up is titled "Building a GPU Driver From Scratch in One Month", and the team reports reverse engineering the AGX firmware ABI and the M4 user space themselves from live hardware traces, without looking at Apple binaries. They did reuse Asahi's kernel UAPI unchanged, and they credit the M1 and M2 driver for having already settled many of the design questions, but the code is new, including a custom shader compiler and command stream builder. They also report that the A18 Pro firmware ABI has roughly 1.5 times as many structs and twice as many pointers as the M1 and M2 one, so it was not a matter of adjusting register offsets.

The other half of the method is the part the usual framing leaves out. The reverse engineering and most of the implementation ran as long, largely unattended agent loops. On the kernel side, Ho describes the agent capturing full GPU memory state through a hypervisor he had written for an earlier project, replaying that state after a reboot, and then progressively reconstructing the firmware structures in code until nothing was being replayed any more. On the user-space side, Sheth's agent iterated Mesa against the Khronos conformance suite until the OpenGL ES 3.0 tests passed.

What that cost in human judgement is documented too, and is more useful than the headline. Ho reports that one compute capture came back at 336 MB and was impossible to replay, that the agent spent over a week on it without success, and that the fix was a human decision about method: boot into single-user mode so no render work runs, trigger a minimal Metal compute program, and capture that instead. He also reports a day lost because the agent chose the hardest subtask first. Neither developer claims the agents worked without supervision.

🧰 The Toolchain

Reverse engineering a GPU driver without documentation means you need good instrumentation. The general toolchain for this kind of work includes:

A purpose-built hypervisor: Ho had already written a hypervisor to run and observe macOS on the hardware. That is what produced the firmware ABI traces, and the agents drove it directly, capturing full address spaces and diffing them against known-good samples when their own code misbehaved
Self-written Metal programs, not Apple's tooling: the reference workloads were small Metal shaders the team wrote themselves, compiled and run so they could observe what changed in memory. They report not looking at Apple binaries, treating required Apple blobs as opaque objects, and having a third party document those blobs so the implementation could be written clean-room
The Khronos conformance test suite: an existing, exhaustive corpus the driver has to pass. This is what the authors credit for keeping agent-written code grounded
Mesa and DRM: The userspace side of the driver lives in Mesa; the kernel side lives in the Linux DRM subsystem. Both have to agree on the ABI for rendering to work end to end

All of the reverse engineering experiments have been published so that the provenance of the work can be checked independently, which is the point of a clean-room process rather than an extra. On upstreaming, the authors say the Mesa side uses the unmodified Asahi UAPI so there are no policy obstacles there, but that it still needs more testing and human review, and that they expect their code "will be held to a higher standard than human-written code." The kernel driver is further off, because they intend to wait for the M1 and M2 driver to go upstream first.

🔧 How GPU Drivers Actually Fit Together on Linux

If you have not dug into this before, here is the rough stack. The kernel side handles memory management, command submission, power management, and scheduling. The userspace side (Mesa for open-source drivers) handles shader compilation, draw call translation, and OpenGL or Vulkan API implementation. They communicate through ioctls defined by the DRM subsystem.

For Apple Silicon, there is an additional layer. The GPU firmware runs on a separate processor inside the chip and has its own command protocol. The kernel driver does not program the GPU registers directly the way a traditional PC GPU driver might. Instead it constructs command buffers and submits them to the firmware, which then drives the hardware. Reverse engineering that firmware protocol is where most of the hard work lives.

A simplified view of the stack looks like this:

Application (game, desktop compositor, etc.)
    |
Mesa (OpenGL / Vulkan userspace driver)
    |
DRM ioctl interface (kernel boundary)
    |
Kernel DRM driver (memory, scheduling, command submission)
    |
GPU firmware (Apple proprietary, runs on embedded processor)
    |
M4 GPU hardware

Every layer has to be correct for rendering to work. A bug at any level shows up as a crash, a black screen, or corrupted output.

🛡️ Security and Stability Considerations

Running out-of-tree kernel modules on production hardware carries real risk. A buggy GPU driver can cause kernel panics, memory corruption, or silent data errors. If you are experimenting with early driver work on Apple Silicon Linux, keep these things in mind:

Keep your GPU driver experiments on a dedicated machine or a VM, not on anything you rely on day to day
Use kernel lockdown and secure boot awareness appropriate to your threat model. Early driver code often requires disabling module signing, which opens up other attack surfaces
Snapshot your disk before major driver updates. Tools like timeshift or a simple dd image of your root partition will save you a painful recovery
Watch kernel logs closely with dmesg -w when testing new driver builds so you catch soft errors before they become hard crashes

🚀 What a One-Month Timeline Actually Looks Like

"Functional" here means more than a usable desktop. The claimed result is full OpenGL ES 3.0 conformance, with the failing tests being optional extensions, plus OpenGL 3.3 in the shipped Gravity Linux alpha, WebGL working in Chrome and Firefox with compositing, and Minecraft at 200fps. Vulkan 1.4, OpenGL 4.6, OpenGL ES 3.2, OpenCL, Direct3D 12 via Proton and ray tracing are all listed as remaining work rather than as done.

What made a month possible is a specific combination, and the authors are explicit about each part: a hypervisor that already existed for capture, a conformance suite that already existed to grade against, the M1 and M2 driver as a map of which design decisions tend to be correct, and coding agents doing the repetitive reverse engineering and implementation. It was not uniformly fast. One compute capture consumed more than a week and was abandoned, and the productive approach was only found afterwards.

For scale, the M1 and M2 Asahi GPU driver took years and many contributors, and the Gravity authors credit that work in plain terms. The two efforts are now separate projects: Asahi Linux continues on its own terms and its own policies, and Gravity Linux is the Fedora remix where this M4 driver ships.

🔬 Applying This to Your Homelab

Even if you are not writing kernel drivers, there are practical takeaways here for anyone running ARM hardware in a homelab.

First, if you are running Linux in a VM or container on an M-series Mac, check whether your virtualization stack exposes GPU acceleration to guests. Tools like QEMU with VirtIO GPU, or commercial options, vary significantly in how much of the host GPU they surface. Knowing the driver stack helps you understand what you are actually getting.

Second, the habit of iterating inside a snapshotted VM is worth adopting generally, even though it is not what this project did. Its iteration ran against real hardware, observed through a hypervisor written for the purpose. Still, if you are developing anything that touches kernel APIs or low-level hardware, do your iteration inside a VM with snapshots. The time you spend setting that up is paid back the first time a bad config would have bricked a bare-metal node.

Third, be clear about which project covers which hardware, because the two are easy to conflate. Asahi Linux is the long-running effort and is where M1 and M2 support lives. The M4 driver described here is Gravity Linux's, and its first release is an early alpha for the M4 Mac mini published on 18 September 2026, which the authors label "FOR DEVELOPERS" and say is "not yet ready to be used as a daily driver." Suspend, display output over the USB-C ports and Thunderbolt or USB4 do not work in it, HDMI does, and the authors warn there may be no upgrade path from the alpha to the beta. Neither project's Apple Silicon GPU driver is upstream in the kernel yet. Check both projects' blogs and the upstream kernel lists before committing hardware to a production workload.

✅ Wrap Up

A conformant Linux GPU driver for the M4 Mac mini in about a month is a notable result, and the method behind it is specific: clean-room reverse engineering from hypervisor traces, an existing conformance suite as the grader, and coding agents writing most of the code. The code is not ready for general use, it is not upstream, and it ships for now in Gravity Linux rather than Asahi Linux, for policy reasons rather than technical ones. If you want the detail first hand, the write-up is at codyho.dev and the driver repositories are linked from it. The two projects' positions on generative AI in contributions are set out in full at Gravity's LLM Use and Clean-Room Policy and Asahi's Generative AI Policy, and both are short enough to read before forming a view.

Related from the lab

Originally published at codewithromi.com.

An AI Coding Agent Uploaded Whole Git Histories: How to Lock Down Your Homelab

CodeWithRomi — Mon, 12 Oct 2026 13:42:31 +0000

On 18 September 2026 a developer called ferstar published a teardown of ZCode, the desktop coding app from Z.ai (Zhipu AI) built on the GLM models, and the Hacker News thread that followed put it in front of everyone. While clearing disk space, ferstar noticed ~/.zcode holding more than 700 MB, unpacked the app, and found that a logged in client was packaging workspace snapshots and uploading them to Alibaba Cloud object storage. The snapshots were not just the source tree: in one 313 MB example covering 42,411 files, 86.6 percent of the payload was the .git directory, LFS objects and reflogs included. Each snapshot was encrypted with a key wrapped to a server held certificate, so the user could not decrypt their own upload. AI coding tools that run locally or integrate into your editor have deep filesystem access by design, and that access is increasingly being used in ways that are not obvious from the UI. If you run a homelab and use any AI agent that touches your code, this is your wake-up call.

The resolution first, because it changes how you should read the rest. Z.ai responded the same day, initially attributing the uploads to a wiki generation feature. It deleted the Alibaba Cloud bucket on 20 September and on 21 September published ZCode's source under Apache 2.0 at github.com/zai-org/ZCode, citing third party audits. ferstar's follow-up review of the published code found the upload endpoint, the object storage path and the key delivery code all absent from version 3.14.0 onward, with indexing running on locally bundled tools. So the specific behavior is gone. What nobody can audit is what happened to whatever was uploaded before 18 September, which is exactly why the cleanup below is still worth doing. Now: why homelabbers are a soft target, and what to do about it.

💡 Why This Matters for Homelabbers

Your homelab repos are not just toy projects. They often contain .env files, hardcoded API keys, internal IP ranges, VPN configs, Ansible inventories, and Docker Compose files with credentials baked in. Git history makes this worse because a secret you deleted three commits ago is still fully recoverable from the log.

AI coding agents are designed to read your project context, and that context is your entire repo tree. When an agent "understands your codebase," it is reading files and history. The question is whether it keeps that data local or phones home with it.

🧰 Prerequisites

A Linux or macOS machine running AI coding tools (ZCode, Cursor, the GitHub Copilot coding agent, Cline, Continue, etc.). If you see GitHub Copilot Workspace referenced in older guides, that preview was sunset on 30 May 2025 and its ideas moved into the Copilot coding agent.
Basic comfort with the terminal and Git
Optional: a network monitoring tool like mitmproxy, Wireshark, or a Pi-hole with logging enabled

🔍 Step 1: Audit What Your AI Tools Are Actually Sending

Before you lock anything down, you need to know what is leaving your machine. The bluntest tool here is watching outbound connections while your AI agent is active.

On Linux, use ss or nethogs to watch live traffic per process:

# Watch outbound connections from a specific process name
ss -tp | grep zcode

# Or monitor per-process bandwidth in real time
sudo nethogs

If you want to inspect the actual payload, route the tool's traffic through mitmproxy. Install it, set your HTTP_PROXY and HTTPS_PROXY environment variables to point at it, and trust its certificate. Then launch the AI agent from the same shell session and watch what it posts. Check the official docs for your specific tool to understand whether it respects proxy settings.

export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
# Then launch your agent in the same terminal

🔧 Step 2: Scrub Secrets from Git History Before They Can Be Harvested

If you have ever committed a secret to a repo that an AI agent can read, treat that secret as compromised. Rotate it first, then clean the history. The two most practical tools for this are git filter-repo and BFG Repo Cleaner. Check each project's documentation for the exact flags, since the syntax differs and getting it wrong can corrupt history.

A basic git filter-repo invocation to strip a file from all history looks like this:

# Install git-filter-repo first (see https://github.com/newren/git-filter-repo)
git filter-repo --path secrets.env --invert-paths

After rewriting history, force-push to all remotes and immediately rotate any exposed credentials. Do not skip the rotation step. Cleaning history only helps for future reads.

🛡️ Step 3: Use a .gitignore (and a Global One)

The fastest mitigation is making sure sensitive files never enter Git in the first place. Add a thorough .gitignore to every project and also configure a global gitignore that covers your whole machine.

# Set a global gitignore
git config --global core.excludesFile ~/.gitignore_global

# ~/.gitignore_global: common patterns to exclude
.env
.env.*
*.pem
*.key
id_rsa
id_ed25519
secrets/
vault_pass.txt
ansible/group_vars/all/vault.yml

This does not protect history that already exists, but it prevents new secrets from being committed accidentally, which is the most common source of leaks.

🛡️ Step 4: Sandbox AI Agents with Network Policies

If you want a harder boundary, run AI coding tools inside a sandboxed environment that limits outbound network access. There are a few practical ways to do this in a homelab.

Docker with network restrictions: Run the agent in a container with --network none or a custom bridge that only allows connections to your LAN and a specific AI API endpoint.
Firewall rules per user: On Linux, you can use iptables or nftables to restrict outbound traffic from a specific Unix user that you create just for running the agent.
A dedicated VLAN: If you have managed switches and a router that supports VLANs (like pfSense or OPNsense), put your dev machine on a VLAN where you can log and filter all outbound connections at the firewall level.

The VLAN approach is the most powerful because it works regardless of what the agent does inside your OS. You see everything at the network edge.

🛡️ Step 5: Prefer Local or Self-Hosted Models Where Possible

The cleanest solution to data exfiltration is keeping data on your own hardware. Tools like Ollama let you run capable open-source models locally and expose them on a local API endpoint. Many AI coding frontends (Continue, for example) can be pointed at a local Ollama instance instead of a cloud API.

# Example: point Continue or a compatible tool at a local Ollama endpoint
# In the tool's config, set the API base URL to:
http://localhost:11434

You give up some model capability compared to frontier cloud models, but for sensitive internal codebases the trade-off is often worth it. Check the documentation for whichever coding agent you use to confirm it supports custom API base URLs and what model format it expects.

🔧 Troubleshooting

mitmproxy shows no traffic from the agent: Some tools use certificate pinning or ignore system proxy settings. Try monitoring at the network level with Wireshark on your loopback or default interface instead.
git filter-repo says "not a git repository": Make sure you are running the command from inside the repo root, and that you have the tool installed correctly. It is not bundled with Git by default.
Ollama model responses are slow: This is normal on CPU-only hardware. Quantized models (Q4 or Q5) give you a better speed and quality balance on machines without a dedicated GPU. Check the Ollama model library for available quantization levels.

✅ Wrap Up

The ZCode incident is a reminder that AI coding agents are powerful precisely because they have broad access to your project context, and that access can be abused, intentionally or accidentally. ZCode itself was patched and opened up within days, so treat this as a warning about the category rather than about one product, and note what it took to find out: a paying user had to measure a directory, unpack an Electron bundle and watch the traffic. Your homelab repos deserve the same scrutiny you would give a production environment.

Start by auditing what your current tools are sending, scrub any secrets from Git history, and put network-level controls in place so you can see and limit outbound connections. If you are dealing with genuinely sensitive internal code, a local model via Ollama is worth the setup time. These are not exotic measures. They are just good hygiene for an era where your code editor is also a networked agent.

Related from the lab

Originally published at codewithromi.com.

Self-Host BookOrbit on Your NAS: Ebooks, Audiobooks and Comics in One Library

CodeWithRomi — Mon, 12 Oct 2026 13:42:12 +0000

I had ebooks on a laptop, audiobooks in an app that wanted a subscription, and comics in a folder I had stopped opening. BookOrbit collapses all three into one self-hosted library with a browser reader, an OPDS catalogue, and two way progress sync with Kobo and KOReader. It is AGPL-3.0 and Docker only.

This is the setup I actually ran, on a TerraMaster NAS rather than a VM, plus the one permissions mistake that will have you convinced your paths are wrong when they are fine.

Why the NAS and not a VM

My first plan put it on a Linux VM and mounted the library over CIFS. Moving it to the NAS deleted three problems at once.

No network mount at all. The books already live on the NAS disk. The VM plan needed a CIFS mount, stored credentials, and systemd hardening so that a NAS hiccup could not hang the VM at boot. A container on the box that owns the files needs none of that.
Smaller blast radius. BookOrbit falling over does not touch anything else I run.
Space. Terabytes free on the NAS against tens of gigabytes on the VM.

One honest caveat. If your NAS also holds backups, you have just added a workload to the thing that protects everything else. BookOrbit itself is light, under two gigabytes with Postgres, but a runaway library can fill a volume. Keep an eye on free space.

What you need first

Docker on the NAS, which on TerraMaster TOS means installing Docker Engine first and then Docker Manager on top of it. Synology and QNAP have equivalents. Then check over SSH:

docker --version
docker compose version

If docker compose is missing from the CLI, you can still deploy: Docker Manager has a Projects feature that takes a docker-compose.yml and an .env directly.

Create a shared folder for the library, give your user read and write, and make the layout:

mkdir -p /Volume1/Books/{Ebooks,Audiobooks,Comics,_dock}

The setup

Two services, the app and Postgres. The full docker-compose.yml, an .env.example and a setup script are in the download at the end of this post, so I will not paste all of it here. The script creates the directories, generates the four secrets, detects your real UID and GID, and writes the .env with mode 600.

# dry run first, it prints what it would do and changes nothing
BOOKS=/Volume1/Books STACK=/Volume1/docker/bookorbit ./setup.sh
BOOKS=/Volume1/Books STACK=/Volume1/docker/bookorbit ./setup.sh --apply

cd /Volume1/docker/bookorbit
docker compose config --quiet && echo "compose valid"
docker compose up -d
docker compose ps

You want the app running and the database healthy. Then browse to the app, paste the SETUP_BOOTSTRAP_TOKEN the script printed, and create the administrator account. That token is single use and stops working afterwards.

Two details in that compose file worth knowing rather than copying blindly. The app container ships read_only: true with cap_drop: ALL, because upstream has already hardened it, so leave that alone. And the Postgres volume must be on local disk. Books can live on a network share. The database cannot, because Postgres on a network filesystem risks corruption.

The trap: a scan that succeeds and finds nothing

This is the single most common failure with this app and it does not look like a permissions problem. The scan runs, reports success, and your library is empty. You check the paths. The paths are correct. You check them again.

The cause is PUID and PGID not matching whoever owns the library folder. On a NAS the admin user is frequently not uid 1000, which is what every tutorial assumes.

id                      # your real uid and gid
ls -ln /Volume1/Books   # who actually owns the library

Put the real numbers in the .env and bring the stack back up. That is why the setup script detects them rather than letting you guess.

A related one: when you create a library in the UI, the folder you enter is the path as the container sees it. That is /books/Audiobooks, not the host path. The host path appears exactly once, in BOOKS_HOST_PATH.

Naming your library so it scans correctly

The scanner infers author, series and title from folder names. One folder per book, always:

Author - Series NN - Title

Bram Stoker - Dracula                                   <- standalone, no series part
Rebecca Yarros - Empyrean 01 - Fourth Wing
Rebecca Yarros - Empyrean 3.5 - Threshing Day           <- novellas keep the decimal
Sarah J. Maas - ACOTAR 01 - A Court of Thorns and Roses

Four rules, each of which I learned by having it wrong:

Never leave a loose file directly in the media folder. In Folder as Book mode it has no folder to belong to. A single .m4b still gets its own folder.
Never nest a series folder. Crescent City/1 - House of Earth and Blood makes the scanner read the parent as one book containing every volume. Flatten it and put the series in the folder name.
Keep the separator exactly space, hyphen, space. Author- Title and Author-Title parse differently, and you will end up with all three variants by accident.
Zero-pad series numbers. 01 not 1, or book 10 sorts before book 2.

Drop edition noise like (m4b) or Unabridged. It is not part of the title and it confuses the metadata match.

Folder as Book versus File as Book

Make one library per media type, because the allowed formats and metadata priority differ. The setting that matters most is the scan mode.

Folder as Book keeps everything in a book's folder together: alternate formats, covers, sidecar .opf metadata, and every audio track of one audiobook. It understands disc folders like CD 1 and Disc 2. Use this for audiobooks and for any ebook you keep in more than one format.
File as Book means one file equals one title. Only correct for a flat pile of single-file EPUBs.

If an audiobook appears as thirty separate titles, the library is in File as Book. That is the entire diagnosis.

Reading and listening on a phone

There is an official iOS app, BookOrbit: The Official App, free with in-app purchases. It is offline first, streams or downloads audiobooks with chapter navigation, does text to speech, and has Apple Watch support for offline listening. It needs a BookOrbit server to point at, so it is a client and not a standalone reader.

On Android there is no official app yet. The web PWA plays audiobooks but without proper OS integration, so lockscreen controls and background playback are the weak points. Worth knowing before you promise someone in your house that this will work on their phone.

For e-readers, BookOrbit exposes an OPDS catalogue that KOReader, Thorium and Moon+ Reader can subscribe to. OPDS is ebook oriented, so it is not much use for audiobooks.

Away from home, the honest answer is not to expose the NAS. Reach it over a mesh VPN such as Tailscale and browse to its address on the tailnet, for example http://100.101.102.103:3000.

What I left out on purpose

kokoro text to speech. Upstream ships it behind a tts profile. It is CPU heavy and a NAS processor is usually modest, so it is omitted. Add it later with docker compose --profile tts up -d and watch your load average.
Reverse proxy and HTTPS. Unnecessary when it is reachable on the LAN and over a VPN.
OIDC and SSO. Supported, but overkill for a handful of users.

The backup conversation nobody has

If your hypervisor runs nightly backups, those back up guests. They do not back up the NAS. Your books and the BookOrbit database live on the NAS, so unless the NAS has its own backup job, nothing is protecting either of them. Decide that deliberately rather than discovering it.

And dump the database before every update, because BookOrbit runs schema migrations on start, which means rolling the image back is not a rollback once a migration has run:

docker exec bookorbit-db pg_dump -U bookorbit bookorbit > bookorbit-db-$(date +%F).sql
docker compose pull
docker compose up -d

When something is wrong

Symptom	Cause
Scan completes, finds no books	`PUID` and `PGID` do not match the owner of the library folder
An audiobook shows as many separate titles	Library is in File as Book mode, switch to Folder as Book
One book contains an entire series	A nested series folder, flatten it
`docker: command not found`	Engine not installed, or the SSH session needs a re-login after installing
Port already in use	Something else has 3000, check with `ss -lnt
Postgres will not start, or is corrupt	Its data directory is on a network share. It must be local disk
Containers vanish after a NAS firmware update	The update restarted Docker. {% raw %}`restart: unless-stopped` should bring them back

The files below are the compose file, the .env template, the setup script and the naming rules as a separate reference. Every address in them is an example, nothing points at a real host, and there are no secrets to strip because the script generates them on your machine.

Download the files

Both are free, no sign up:

bookorbit-nas-bundle.zip contains the docker-compose.yml, an .env.example with every variable documented, the setup.sh that generates the secrets and detects your UID and GID, plus a README of the four things that will bite you.
bookorbit-naming-rules.md is the folder convention on its own, because that is the part you will want open while you reorganise a library.

Related from the lab

Originally published at codewithromi.com.

Patch These Four Linux Kernel Local Root Vulns Now: DirtyAH6, PPPoEject, TUNderflow, and DiagSpill

CodeWithRomi — Wed, 07 Oct 2026 23:00:49 +0000

Four Linux kernel vulnerabilities went public on 18 September 2026, and they are the kind you should not sit on. DirtyAH6 (CVE-2026-80844), PPPoEject (CVE-2026-68121), TUNderflow (CVE-2026-81000) and DiagSpill (CVE-2026-74469) are all local privilege escalation bugs, meaning an attacker or malicious process with a foothold on your machine can escalate to root. Working proof of concept exploits for all four are public. If you are running Proxmox, a Raspberry Pi, or any self-hosted Linux node, this is a patching weekend.

Security researcher Asim Manizada reported the four bugs in mid July 2026 and published a technical write-up plus working exploits on 18 September 2026, after a coordinated hold so distributions could ship fixes first. The oss-security post and the accompanying write-up carry the per-CVE detail. Upstream, the earliest stable releases that contain all four fixes are 5.10.270, 5.15.221, 6.1.188, 6.6.157, 6.12.109, 6.18.50 and 7.2.4. Note that the 7.0 series reached end of life on 27 June 2026, so a 7.0 kernel only carries these fixes if your vendor backported them.

💡 Why This Matters for Homelabbers

Local root exploits are often dismissed as "low risk" because they require existing access. That framing misses the threat model for homelabs and self-hosted infrastructure. If any service you run gets compromised, a web app, an LXC container with a shared kernel, or even a rogue package, a local root bug turns that into full node ownership.

Proxmox LXC containers share the host kernel. A container breakout combined with a local priv-esc is a realistic attack chain. Raspberry Pi nodes running exposed services are in the same boat. Patch the kernel, then verify.

🧰 Prerequisites

SSH or console access to each node
sudo or root privileges
A maintenance window, since patching requires a reboot
Proxmox nodes: access to the Proxmox web UI or CLI for VM/container shutdown sequencing

🔧 Step-by-Step: Patching Your Nodes

Step 1: Check your current kernel version on every node so you know what you are starting from.

uname -r

Note this output. You will compare it after the update to confirm the new kernel is running.

Step 2: Update your package index and apply all available security updates. On Debian and Ubuntu based systems, including Proxmox and Raspberry Pi OS:

sudo apt update && sudo apt full-upgrade -y

On Proxmox specifically, you also want to pull from the Proxmox repositories. If you are on the no-subscription repo, this is already covered by the command above. Run pveversion -v afterward to confirm the Proxmox packages are current too.

Step 3: If you are running Raspberry Pi OS, check the kernel package by its current name. On Bullseye and earlier the kernel shipped in the raspberrypi-kernel package, but Raspberry Pi OS moved to Debian style linux-image-rpi-* packages with Bookworm, so on a current install you want linux-image-rpi-v8 or linux-image-rpi-2712 depending on the board. The apt full-upgrade command above will still catch it, but confirm with:

# Bookworm and newer
apt-cache policy linux-image-rpi-v8

# Bullseye and older
apt-cache policy raspberrypi-kernel

The "Installed" and "Candidate" versions should match after your upgrade. If they do not, run the upgrade again.

Step 4: Reboot to load the new kernel. This is not optional. The running kernel in memory is still the old one until you reboot.

sudo reboot

For Proxmox hosts, shut down or migrate your VMs and containers before rebooting the host. Use the Proxmox web UI or:

# List running VMs
qm list

# Shutdown a VM by ID
qm shutdown [vmid]

# List running containers
pct list

# Shutdown a container by ID
pct shutdown [ctid]

Step 5: After the reboot, confirm the new kernel is loaded:

uname -r

The version string should be newer than what you recorded in Step 1. Cross-reference it against your distro's security advisory to confirm the patched version is in place. For Debian, check security-tracker.debian.org. For Ubuntu, check ubuntu.com/security/notices. For Raspberry Pi OS, check the Raspberry Pi blog and release notes.

🛡️ Patch Status, Checked 29 September 2026

This is the part of the post that goes stale fastest, so here is a dated snapshot. Everything below was checked against the Debian security tracker, the Ubuntu CVE tracker and the Proxmox kernel changelog on 29 September 2026. Re-check before you decide you are covered.

Debian 13 trixie: all four first fixed in linux 6.12.111-1, shipped as DSA-6528-1.
Debian 12 bookworm: DirtyAH6, PPPoEject and DiagSpill first fixed in linux 6.1.187-1 from bookworm-security. TUNderflow is the exception. The Debian tracker still lists bookworm as vulnerable, 6.1.187-1 included, so there is no fix for CVE-2026-81000 on bookworm yet.
Ubuntu 22.04, 24.04 and 26.04: at the time of this check the Ubuntu CVE tracker showed no released kernel for any of the four on the main linux packages. They sit at needed or pending. Watch ubuntu.com/security/notices instead of assuming an upgrade fixed it.
Proxmox VE: the Proxmox kernel changelog names only TUNderflow, first fixed in proxmox-kernel-7.0 version 7.0.14-19, released 18 September 2026. The newest build at the time of checking was 7.0.14-20 from 24 September 2026. The other three CVEs are not named anywhere in that changelog, and the Proxmox 7.0 kernel follows the Ubuntu kernel, which has not released them, so do not assume DirtyAH6, PPPoEject and DiagSpill are closed on a Proxmox host. Run pveversion -v and compare against the changelog.
Raspberry Pi OS: no Raspberry Pi OS kernel build carrying all four fixes could be confirmed at the time of this check. On the 6.12 branch you need 6.12.109 or newer for the full set, so check the installed package version rather than assuming.

🛡️ What Each Vulnerability Affects

Here is what each bug actually is, taken from the disclosure and the researcher's write-up. Three of the four need CAP_NET_ADMIN in a network namespace the attacker controls, which an ordinary local user obtains by creating an unprivileged user namespace first. DiagSpill is the exception and needs no special privileges at all.

DirtyAH6 (CVE-2026-80844): The IPv6 Authentication Header code in the IPsec and XFRM path does not check that a routing header's segments_left field is no larger than the number of addresses actually present, so pointer arithmetic in ipv6_rearrange_rthdr() walks out of bounds and an oversized memmove() corrupts kernel memory. It needs AH6 and XFRM support plus CAP_NET_ADMIN and CAP_NET_RAW in a namespace the attacker controls. The researcher also reports a remote crash, and in his own lab remote root, against a host acting as an IPv6 router that applies AH in transport mode.
PPPoEject (CVE-2026-68121): A use after free in pppoe_sendmsg(). The function holds a pointer into the socket buffer head across a dev_hard_header() callback that can reallocate that head, then writes six bytes through the stale pointer. It needs a lower device whose header callback expands the buffer head, not merely a loadable PPPoE module, which makes it the narrowest of the four in practice.
TUNderflow (CVE-2026-81000): The TUN/TAP driver stores a receive headroom value without bounding it. A device path that propagates oversized headroom (the proof of concept chains a netkit device through Open vSwitch to a raw TUN port) underflows a size calculation and leaves packet data 64 bytes past the end of the allocation. Simply having a TUN or TAP interface is not enough, you need that headroom propagation path. Worth correcting a common assumption here: Docker's default bridge networking uses veth pairs rather than TUN/TAP, and the in kernel WireGuard module is its own device type, so the usual homelab exposure is QEMU tap interfaces and OpenVPN style TUN devices, not containers.
DiagSpill (CVE-2026-74469): An SCTP association can hold up to 65,536 peer transports but the counter is 16 bits wide, so it wraps to zero at the limit. sctp_diag then reserves buffer space based on the wrapped count and copies the full transport list anyway, writing roughly 8 MiB past the end of the response buffer. This one needs no capabilities and no user namespace, only SCTP and sctp_diag support, which is what makes it the priority of the four.

DiagSpill is the one to move fastest on if you are triaging. It is the only one of the four that an ordinary local user can reach without creating a user namespace, which also means the usual advice to restrict unprivileged user namespaces does nothing for it. If you do not need SCTP, keep the module out of the kernel.

🛡️ Hardening Beyond the Patch

Patching is the fix, but a few additional controls reduce your exposure, which matters more than usual here because some distributions have not shipped all four yet. Restricting unprivileged user namespaces closes the ordinary user path to three of the four. It does nothing for DiagSpill.

Restrict unprivileged user namespaces: Three of these four bugs need a user namespace for an ordinary local user to reach them. Ubuntu introduced AppArmor based restrictions in 23.10 and enabled them by default from 24.04, controlled by kernel.apparmor_restrict_unprivileged_userns, so 24.04 and 26.04 are covered out of the box. Debian is not. The Debian and Ubuntu specific kernel.unprivileged_userns_clone knob has been discussed for deprecation, so confirm it exists on your kernel with sysctl -a | grep userns before relying on it; the portable upstream equivalent is sysctl -w user.max_user_namespaces=0. Either way you will break things that legitimately use user namespaces, including Flatpak, rootless containers and browser sandboxes.
Use AppArmor or SELinux profiles: Proxmox ships with AppArmor. Make sure it is enforcing for your LXC containers. Unprivileged containers with AppArmor confinement are significantly harder to break out of.
Limit local access: Do not run untrusted code or expose shell access to untrusted users on these nodes. Local privesc only matters if someone or something local can trigger it.
Unload unused kernel modules: If you are not using PPPoE or SCTP, keep both out. Add blacklist pppoe and blacklist sctp to a file in /etc/modprobe.d/ and run update-initramfs -u. Blacklisting stops the on demand autoload that a socket call would trigger, and for DiagSpill that is the only mitigation short of patching, since restricting user namespaces does not touch it. Verify with lsmod | grep -E 'pppoe|sctp' afterwards.

🔧 Troubleshooting

Kernel version did not change after reboot: Your package manager may have the patched package available but it was not installed. Run apt list --upgradable | grep linux-image to check, then install explicitly with sudo apt install linux-image-[version] matching your architecture.

Proxmox node is stuck on an older kernel: Proxmox sometimes pins the kernel for stability. Check /etc/default/grub and the GRUB menu to ensure the new kernel is selected as default. Run update-grub after confirming the package is installed.

Raspberry Pi will not boot after upgrade: This is rare but can happen if the bootloader and kernel fall out of sync. Boot from a backup SD card or use rpi-update with caution, and check the Raspberry Pi OS forums for your specific board and OS version before using it.

✅ Wrap Up

Four local root bugs in one disclosure batch is a bad day for Linux security. Updating and rebooting is the right move, but as of 29 September 2026 it is not the whole answer: Debian trixie has all four, Debian bookworm is still missing the TUNderflow fix, and Ubuntu has not released kernels for any of them. Prioritize DiagSpill, since it is the one an unprivileged local user can reach without a user namespace, and check the tracker for your distribution instead of assuming a fresh upgrade covered you.

After you patch, take five minutes to check your other nodes. It is easy to forget a Raspberry Pi tucked behind the TV or a secondary Proxmox node you set up months ago. Cross-reference the patched kernel version against your distro's security tracker to make sure you are actually covered, not just up to date on packages.

Stay subscribed to oss-sec if you want these disclosures in your inbox before they make the rounds on aggregators.

Related from the lab

Originally published at codewithromi.com.

WordPress 7.1.2 Patches a Critical Path Traversal to RCE: Update Now

CodeWithRomi — Wed, 07 Oct 2026 23:00:46 +0000

A serious vulnerability was disclosed in the WordPress core codebase: an unauthenticated path traversal bug that, under the right server conditions, can escalate to remote code execution. That combination, no login required plus potential RCE, puts this near the top of the priority list for any WordPress site owner or host. Here is what the vulnerability is, who is affected, and exactly what to do about it.

💡 Why This Matters

Path traversal vulnerabilities let an attacker reference files outside the intended directory by injecting sequences like ../ into a request. On their own they are dangerous. Chained with a code execution primitive, they become critical.

The "conditional" part of this disclosure means RCE is not guaranteed on every WordPress install. Server configuration, PHP settings, and installed plugins or themes all influence whether the traversal can be weaponised into execution. But "conditional" does not mean "safe to ignore." Attackers will probe for the conditions that make it work, and many shared hosting and default VPS setups are likely to qualify.

The advisory is GHSA-7hp8-65ch-5whp in the WordPress develop repository, and it is CVE-2026-87902, rated Critical at CVSS 4.0 9.2. Every release from 4.7.0 through 7.1.1 is affected. WordPress shipped 7.1.2 on 22 September 2026 along with backports to every branch still eligible for security fixes, down to 4.7.37. Treat this as urgent rather than theoretical: CISA added CVE-2026-87902 to its Known Exploited Vulnerabilities catalog on 25 September 2026, and public scanning tooling plus attempts to write PHP files through pearcmd.php are already being seen in the wild.

🧰 Prerequisites: Who Should Read This

Anyone running a self-hosted WordPress site on 4.7.0 or later, which in practice means anyone running WordPress
Hosting providers and managed WordPress hosts
Homelab builders running WordPress internally (yes, even private sites)
DevOps teams with WordPress in staging or CI pipelines

If WordPress is running somewhere you control, this post is for you.

🔍 How the Vulnerability Works

The flaw is in one specific thing: page template resolution. An unauthenticated request can steer get_page_template() into including a chosen readable local .php file from outside the active theme's directories. That makes it a local file inclusion rather than a generic web root escape, and it is classed as CWE-98, improper control of filename for an include or require statement.

The conditions are narrower and far more specific than 'a misconfigured server', and they are worth checking against your own install rather than guessing. The file inclusion needs the active theme, parent or child, to contain a top level directory whose name begins with page-. That is not exotic: page-templates/ ships in Twenty Twelve and Twenty Fourteen and in popular themes including Neve, Hestia and Sydney. Turning that inclusion into code execution needs two more things: a readable .php file on the server that will act on attacker supplied arguments, in practice pearcmd.php, and register_argc_argv set to On. Official PHP Docker images and default cPanel configurations on PHP before 8.5 both qualify. For the precise code path, refer to the official security advisory.

🛠️ Step-by-Step: Patching Your WordPress Install

The most important action is updating WordPress core. On the current branch that means 7.1.2. If you are held back on an older branch there is a backport for it, 7.0.6, 6.9.9, 6.8.10, 6.7.9 and so on all the way down to 4.7.37, so being unable to move to 7.1 is not a reason to stay unpatched. Here is how to update cleanly.

Step 1: Back up your database and files before touching anything.

# Example using WP-CLI to export the database
wp db export backup-$(date +%F).sql --allow-root

Step 2: Update WordPress core via WP-CLI (recommended for server installs).

wp core update --allow-root
wp core version --allow-root

Step 3: Confirm the installed version matches the patched release listed in the advisory.

wp core version --allow-root

Step 4: If you manage WordPress through a hosting dashboard (cPanel, Plesk, Kinsta, etc.), use the built-in one-click updater or auto-update toggle. Confirm the version number in the WordPress admin under Dashboard, then About.

Step 5: Enable automatic background updates for minor releases if you have not already. Add this to your wp-config.php:

define( 'WP_AUTO_UPDATE_CORE', 'minor' );

This ensures security patches ship automatically without waiting for manual intervention.

🛡️ Hardening: Reduce the Attack Surface Regardless of Version

Patching fixes the specific bug. Hardening reduces the blast radius of the next one.

Block PHP execution in the uploads directory. Worth doing, but be clear that it does not mitigate CVE-2026-87902, which includes a .php file already present on the server rather than one an attacker uploaded. Keep it as general hygiene against the next upload driven bug. For Nginx, add this to your server block:

location ~* /wp-content/uploads/.*\.php$ {
    deny all;
}

For Apache, drop a .htaccess file inside wp-content/uploads/:

<Files "*.php">
    Require all denied
</Files>

Disable file editing in the WordPress admin. Add to wp-config.php:

define( 'DISALLOW_FILE_EDIT', true );

Break the RCE preconditions if you cannot patch within the hour. Set register_argc_argv = Off in your PHP configuration and confirm it with php -i | grep register_argc_argv. Then check whether a reachable pearcmd.php exists at all, with find / -name pearcmd.php 2>/dev/null, and remove it if nothing on the box needs PEAR. Neither of these closes the file inclusion, but together they remove the documented route from inclusion to code execution. Patching is still the fix.

Run WordPress behind a WAF. Cloudflare, Wordfence, or a self-hosted option like ModSecurity can catch traversal patterns at the request layer before they reach PHP. Even a free Cloudflare plan adds meaningful filtering.

Restrict xmlrpc.php and wp-login.php by IP if possible. Unauthenticated attack surface shrinks dramatically when admin endpoints are IP-locked.

# Nginx example: restrict wp-login.php
location = /wp-login.php {
    allow [your-admin-IP];
    deny all;
}

🔧 Troubleshooting Common Update Issues

WP-CLI command not found: Install it by following the official WP-CLI installation guide at wp-cli.org. On most Linux servers it is a single curl download and chmod.

Update fails due to file permissions: WordPress needs write access to its own directory during updates. Check that the web server user (often www-data on Ubuntu/Debian) owns the WordPress files.

chown -R www-data:www-data /var/www/html/

Site breaks after update: Restore from the backup taken in Step 1, then investigate plugin or theme conflicts. Deactivate all plugins and retest. The WordPress support forums and the WP-CLI --debug flag are your friends here.

Managed host shows no update available: Some managed hosts patch at the server level before surfacing a WordPress core update. Contact support to confirm your environment is running the patched version.

✅ Wrap Up

Unauthenticated path traversal leading to conditional RCE is the kind of vulnerability that gets actively exploited fast, especially against WordPress given its enormous install base. The fix is straightforward: update core to the patched release, turn register_argc_argv off if nothing on the server needs it, and put automatic minor updates on so future patches land without delay.

The patched version is 7.1.2, or the backport for whichever branch you are on. The official advisory (GHSA-7hp8-65ch-5whp) has the full backport list and the code path. Patch first, harden second, and since this one is already on CISA's exploited list, check your access logs for requests carrying page- template names or pearcmd while you are in there.

Related from the lab

Originally published at codewithromi.com.

Miri Was Leaking CI Secrets Through GitHub Actions Caches: What to Clean Up

CodeWithRomi — Wed, 07 Oct 2026 23:00:44 +0000

CI/CD pipelines are a common attack surface, and most developers focus on the obvious risks: exposed tokens, world-readable repos, overly permissive workflow permissions. But a subtler class of vulnerability comes from caching, and in September 2026 the Rust project published an advisory that is a textbook case. Miri, Rust's interpreter for detecting undefined behavior, was writing every environment variable it saw into the target/ directory. Cache target/ in GitHub Actions, as most Rust projects do, and any secret handed to that job was sitting in the cache where a later pull request run could restore it and read it.

The good news first, because it changes how you should read the rest: this is fixed. The Rust project published the advisory on 21 September 2026 and the nightly toolchain dated 22 September 2026 stops Miri persisting arbitrary environment variables, keeping only CARGO_* (excluding CARGO_*_TOKEN) and OUT_DIR. No CVE was assigned. What is left for you is cleanup rather than defense: update the toolchain, delete the caches you already have, and rotate anything that was in scope. This post covers the mechanism, that cleanup, and how to stop caching biting you again.

💡 Why This Matters

Miri runs Rust programs in an interpreted environment so it can detect undefined behavior at runtime. To notice when something build relevant changed between runs, cargo miri recorded the environment it was invoked with, and it recorded all of it rather than only the variables it needed. That record lived inside the target/ directory. Note what is not required here: nothing in your code, your test output or your logs had to mention the secret. Passing it to the step was enough for it to land on disk in the directory everyone caches.

That matters because of how GitHub Actions cache scoping works. A workflow run can restore caches created on its own branch and on the default branch, so a pull request run can pull back a cache that a privileged push to main created. The advisory spells out the conditions: you are affected if you run cargo miri in CI, pass secrets as environment variables to that step, cache the target/ directory with something like actions/cache or Swatinem/rust-cache, and allow pull requests to read from that cache.

The broader lesson goes beyond Miri. Any tool that snapshots its invocation environment into a build directory is a candidate for this class of bug, and build directories are exactly what CI caching is pointed at. Rust's toolchain is just where this instance was caught, reported and documented.

🧰 Prerequisites

A GitHub Actions workflow that uses Miri (typically via cargo miri test)
Familiarity with the actions/cache action or the built-in Cargo caching patterns
Repository secrets configured in Settings, such as API keys or tokens passed to your test environment

🔍 How the Leakage Path Works

Here is the setup that creates the risk. A workflow injects a secret as an environment variable, runs Miri, and caches the Cargo target/ directory to speed up future runs. The target/ directory is the part that matters. Caching the registry or the Miri sysroot was never the problem.

# Vulnerable pattern (simplified)
- name: Run Miri
  env:
    MY_SECRET: ${{ secrets.MY_SECRET }}
  run: cargo miri test

- name: Cache build output
  uses: actions/cache@v4
  with:
    path: target
    key: miri-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}

The value of MY_SECRET does not have to appear in any log or test result. cargo miri persisted it into target/ as part of the environment snapshot, and the cache step then packaged target/ up. The next job or pull request run that restores this cache has a file containing your secret, without that secret ever being granted to the run.

No exotic trigger is required. You do not need a panic, a crash, or verbose diagnostics. Passing the secret to the step and caching target/ is the whole bug. The advisory also notes that someone who lifted a secret this way could push further commits to cover their tracks, so a clean-looking workflow log is not evidence you were fine.

🛡️ Step-by-Step: Hardening Your Pipeline

Step 1: Never inject production secrets into Miri test runs. If your tests genuinely need a credential, use a scoped, revocable test credential with no real permissions. Treat anything Miri can see as potentially loggable.

# Prefer a dummy or scoped value for Miri runs
- name: Run Miri
  env:
    MY_SECRET: "dummy-value-for-miri"
  run: cargo miri test

Step 2: Fix your caching scope. The directory that leaked is target/, so either stop caching it on jobs that see secrets, or stop giving those jobs secrets. If you do cache something for Miri, cache the sysroot, and use a distinct cache key so a Miri cache never collides with a standard build cache.

- name: Cache Miri sysroot only
  uses: actions/cache@v4
  with:
    path: ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib
    key: miri-sysroot-${{ runner.os }}-${{ hashFiles('rust-toolchain.toml') }}

Check the exact sysroot path for your toolchain with rustc --print sysroot rather than hardcoding it. The path above is illustrative; confirm it in your environment.

Step 3: Restrict workflow permissions explicitly. Add a top-level permissions block to limit what a workflow can do even if a cache is poisoned.

permissions:
  contents: read

Step 4: Isolate Miri jobs from secret-bearing jobs. Use separate jobs so the job running Miri never sees a real secret. GitHub Actions does not share environment variables across jobs unless you pass them explicitly, so inject secrets at the step level in the job that actually needs them. One correction worth making if you copy patterns off the internet: secrets: inherit is only valid on a job that calls a reusable workflow with uses, not on a job that runs its own steps.

jobs:
  miri:
    runs-on: ubuntu-latest
    # No secrets injected here
    steps:
      - uses: actions/checkout@v4
      - run: cargo miri test

  deploy:
    runs-on: ubuntu-latest
    needs: miri
    env:
      DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
    steps:
      - run: ./deploy.sh

Step 5: Audit your cache keys and paths regularly. Run a search across your workflow files for any actions/cache or cache-dependency-path usage that overlaps with paths your secret-injected steps write to.

grep -r "actions/cache" .github/workflows/

🔧 Troubleshooting

My Miri tests need real environment values to run correctly. Refactor the tests to accept injectable fakes, or use a secrets manager that vends short-lived credentials scoped only to the test run. This is a design problem as much as a security one.

I am not sure what is in my cache. Download a cache artifact locally using the GitHub CLI (gh cache list and gh cache download) and inspect the contents before assuming it is clean. The official GitHub CLI docs cover the exact flags.

We use a third-party caching action, not actions/cache. The same principles apply. Check whether that action stores artifacts in a location your Miri run can write to, and review the action's own permissions model.

✅ Wrap Up

Caching is one of the best levers you have for speeding up CI, but it introduces a persistent artifact that outlives a single workflow run. In this case the artifact was the ordinary Cargo target/ directory, and the tool filling it with secrets was doing so as a convenience feature. Be deliberate about what goes into the cache and which jobs hold real secrets.

Concretely, for this advisory: move to a toolchain from 22 September 2026 or later, delete the existing caches on any repository that ran cargo miri with secrets in scope, and rotate those secrets. Then, as standing practice, keep Miri jobs secret free, lock down workflow permissions, and audit your cache paths as part of any security review. Treat cached artifacts with the same suspicion you would treat a shared filesystem.

The primary source is the Rust project advisory GitHub Actions leaking secrets when Miri output is cached, published 21 September 2026. Read it in full for the exact affected conditions and for any follow-up, and pair it with the GitHub Actions security hardening guide in the GitHub docs.

Related from the lab

Originally published at codewithromi.com.