DEV Community: Osmium

How Tailscale's Route Table Gets Wiped by mwan3 on OpenWrt

Osmium — Sat, 10 Oct 2026 06:43:15 +0000

My trusty Netcore N60 Pro runs on OpenWrt 25.12 with both mwan3 (2.12.0) and Tailscale (v1.98.3) installed. Tailscale would work fine, then randomly stop being reachable. Every time I touched the mwan3 config, tailscale broke immediately after.

Background: routing tables, ip rule, and firewall marks

Before the investigation, it's worth untangling three concepts that are easy to conflate:

Routing table — Linux allows multiple routing tables at once, each with a numeric id. The default main table is 254. Tailscale uses its own table, id 52, to hold peer routes. mwan3 allocates one table per WAN interface.

IP rule — decides which table a packet's lookup goes to. It sits above the routing tables and matches top to bottom by priority, e.g. from all lookup 52 means "send all packets to table 52 for the lookup."

Firewall mark (fwmark) — mwan3's mechanism is to have iptables/nftables mark a packet first (e.g. tag which WAN it came in on), then have ip rule route based on that mark to the matching table. This is why mwan3 depends so heavily on the firewall layer.

The investigation: a few wrong turns first

First guess: mwan3 must be deleting something

mwan3 restart was rebuilding its own policy routing rules and clobbering tailscale's rule pointing at table 52. Testing didn't confirm it:

root@N60Pro:~# ip rule | grep 52
1370: from all lookup 52
root@N60Pro:~# ip route show table 52
root@N60Pro:~#

The rule was still there, but table 52 itself was empty. This narrowed things down: the problem wasn't at the ip rule layer, it was the table itself getting wiped.

Second guess: This nft seems to be doing weird things

mwan3 restart was throwing a screen full of errors:

iptables v1.8.10 (nf_tables): table `mangle' is incompatible, use 'nft' tool.
Error: cmp sreg undef
Error: meta sreg is not an immediate

OpenWrt has used nftables/firewall4 since 22.03, but the official mwan3 package is still iptables-based and relies on the iptables-nft compatibility shim, which frequently fails to translate rules. The suspicion was that this broken compat layer was causing mwan3 to malfunction outright, which led to considering mwan3-nft (a community nftables port) or even downgrading OpenWrt entirely.

But doing a nft list ruleset showed mwan3's own chains and packet counters actively incrementing (packets 3956602 bytes 456220071, etc.), proving mwan3's core policy routing was actually working fine, those errors were log noise, unrelated to table 52 being wiped. So that leads to one precise question: who is clearing table 52, and why?

Finding the code: mwan3's flush loop

With a little help from Claude, grepping mwan3's init script for anything that flushes routing tables:

grep -rn "ip route flush\|route flush" /lib/mwan3/ /etc/init.d/mwan3

That landed on line 77 of /etc/init.d/mwan3:

for tid in $($IP route list table all | sed -ne 's/.*table \([0-9]\+\).*/\1/p' | sort -u); do
    [ $tid -gt $MWAN3_INTERFACE_MAX ] && continue
    $IP route flush table $tid &> /dev/null
done

So, line by line:

$IP route list table all, lists every routing table currently present on the system, not just mwan3's own.
sed extracts the table X number from each route, deduplicates and sorts, giving a list of every table id currently in use.
For each table id: if it's greater than $MWAN3_INTERFACE_MAX, skip it. Otherwise, flush it.

Plain and simple for mwan3, but bad news for Tailscale. The code assumes "any table id below the threshold must be mine". Tailscale's table 52 just happens to sit below that threshold, so every mwan3 restart sweeps it up along with mwan3's own tables.

Why mmdefault - 3: how the threshold is computed

But what exactly is $MWAN3_INTERFACE_MAX? How it's being used and where to configure it?

Digging into mwan3's common.sh, here's where $MWAN3_INTERFACE_MAX comes from:

config_get MMX_MASK globals mmx_mask '0x3F00'

bitcnt=$(mwan3_count_one_bits MMX_MASK)
mmdefault=$(((1<<bitcnt)-1))
MWAN3_INTERFACE_MAX=$((mmdefault-3))
uci_toggle_state mwan3 globals iface_max "$MWAN3_INTERFACE_MAX"

Breaking it down step by step:

Step 1: bitcnt. mmx_mask is the mask mwan3 uses for its fwmark bits, defaulting to 0x3F00. In binary, 0011 1111 0000 0000, which has 6 set bits. This can be verified directly from the device's own nft list ruleset output, where mwan3's rules consistently show meta mark & 0x00003f00.

Step 2: mmdefault. mmdefault = (1 << bitcnt) - 1 is simply the largest value representable with those 6 bits:

mmdefault = (1 << 6) - 1 = 64 - 1 = 63

Step 3: subtract 3 to get the final threshold. The "-3" in mmdefault - 3 isn't arbitrary: mwan3 reserves the top few values in the mark range for special states (blackhole, unreachable, etc., visible in the ip rule output earlier as entries like 5250: from all fwmark 0x80000/0xff0000 unreachable), which can't be assigned to ordinary WAN interfaces and so get carved out of the total. Final result:

MWAN3_INTERFACE_MAX = 63 - 3 = 60

Found it, this is the core of the problem: tailscale uses table 52, which is less than 60, so it falls squarely inside the range mwan3 manages and gets flushed on every mwan3 restart.

Here's a piquant thing, tailscale using table 52, only because the digits 5 and 2 sit above the letters T and S on a QWERTY keyboard, and there is no environment variable or startup flag to change this table number. Which means this conflict can only be resolved from mwan3's side.

The Solution

Since table 52 can't move, the fix is to push the threshold below 52 instead. Tracing back the formula: the threshold is set by the bit count of mmx_mask, and mmx_mask is an ordinary UCI config option under config globals in /etc/config/mwan3, not hardcoded. Lower the bit count, and the threshold drops with it. The default is 0x3f00, so how about a 0x1f00? 28 < 52. Should get the job done.

One small wrinkle

Changing the mask should solve the problem, but it didn't. table 52 was still being wiped, after some more scratching heads and some more Claude-please-help-me-how-it-happened, common.sh's logic for reading MWAN3_INTERFACE_MAX turned up a caching gotcha:

if [ -e "${MWAN3_STATUS_DIR}/mmx_mask" ]; then
    readfile MMX_MASK "${MWAN3_STATUS_DIR}/mmx_mask"
    MWAN3_INTERFACE_MAX=$(uci_get_state mwan3 globals iface_max)
else
    config_get MMX_MASK globals mmx_mask '0x3F00'
    # recompute bitcnt / mmdefault / MWAN3_INTERFACE_MAX ...
fi

This recompute only happens if the cache file doesn't exist. The cache file lives at:

/var/run/mwan3/mmx_mask

Deleting it forces the else branch to run and recompute:

uci set mwan3.globals.mmx_mask='0x1F00'
uci commit mwan3
rm -f /var/run/mwan3/mmx_mask
/etc/init.d/mwan3 restart

Verification

After the restart, both the cache and the threshold had updated:

root@N60Pro:~# cat /var/run/mwan3/mmx_mask
0x1f00
root@N60Pro:~# cat /var/state/mwan3 | grep iface_max
mwan3.globals.iface_max='28'

Then the full test: restart tailscale so table 52 has content, then restart mwan3 and check whether it survives.

root@N60Pro:~# /etc/init.d/tailscale restart
root@N60Pro:~# sleep 8
root@N60Pro:~# ip route show table 52
100.87.123.123 dev tailscale0
100.100.100.100 dev tailscale0
100.107.123.123 dev tailscale0

root@N60Pro:~# /etc/init.d/mwan3 restart
root@N60Pro:~# sleep 5
root@N60Pro:~# ip route show table 52
100.87.123.123 dev tailscale0
100.100.100.100 dev tailscale0
100.107.123.123 dev tailscale0

Finally it fixed! And turns out no hack required at all.

Sources

tailscale/tailscale — router_linux.go, v1.66.1 — the mwan3-detection / policy base priority 1300 logic, and the tailscaleRouteTable = "52" constant with its "TS on a QWERTY keyboard" comment.
tailscale/tailscale PR #12035 — cmd/tailscaled: add ip-rule-pref-base argument — the 2024-07-31 patch that made the ip rule priority base configurable (not a fix for the table-52 constant itself).
openwrt/packages — net/mwan3/files/lib/mwan3/common.sh — official mwan3, where mmx_mask → bitcnt → mmdefault → MWAN3_INTERFACE_MAX is computed.
openwrt/packages — net/mwan3/files/etc/init.d/mwan3 — official mwan3 init script containing the unguarded ip route flush table $tid loop (line ~77 at the time of writing).
OpenWrt wiki — mwan3 (iptables) — official docs, including the "no longer recommended" notice for the iptables-based package on current releases.
OpenWrt wiki — mwan3 (nftables, unofficial) — docs for the community nftables port used in the fix.
dl12345/mwan3 — source of the nftables port; see /etc/init.d/mwan3 in this repo for the ip-rule deletion fix (dual priority+content gate) and the still-unguarded route-table flush loop discussed above.
dl12345/luci-app-mwan3 — companion LuCI frontend built for the nftables port.
jamesmacwhite's mwan3-nft-update.sh (gist) — the install/update script used to fetch and install dl12345's packages via apk.
forum.openwrt.org — Tailscale exit node not working with mwan3 on OpenWrt 25.12.4 — a separate, independently-reported case of the same mwan3/tailscale conflict family.
Tailscale docs — IP routes installed by Tailscale — official confirmation that tailscale routes live in table 52.

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Osmium — Fri, 01 May 2026 14:26:19 +0000

Why Bother

I have a two-node Proxmox VE cluster, and two-node clusters have an inherent distributed systems problem: if either machine goes down, the surviving node can't tell whether it should keep running or not. Running a quorum usually requires a majority vote, which normally means at least three voters. With only two machines, one vote each, lose one and you're down to one vote. No quorum and your cluster stops.

The solution is a quorum device. Proxmox supports corosync-qnetd, a lightweight daemon from the corosync project that acts as a third-party tiebreaker. A qnetd doesn't participate in storage or compute, it just casts the deciding vote during a split-brain scenario. By bringing the total vote count to three, the cluster maintains a majority requirement and consequently, if any node fails, the surviving node and the router’s vote can collectively sustain quorum, making the cluster keeps running.

The ideal host for something like this is a device in a home network that basically never gets turned off, like a router.

I had a Xiaomi Mini router running OpenWrt, MT7620 chip, 128MB DDR2 memory, 16MB SPI Flash, mipsel_24kc architecture. It runs 24/7, draws a few watts, perfect for the job. The problem: there's no corosync-qnetd package for mipsel in the official OpenWrt feeds. :(

Finding a Starting Point

After some searching, I found someone on GitHub who had compiled corosync-qnetd for aarch64 (ARM 64-bit) OpenWrt, targeting GL.iNet routers and using the old opkg/ipk package format. A good starting point.

I had essentially zero experience with C, Makefiles, or cross-compilation packaging. I knew C programs need a Makefile to build, but that was about it. So my approach was simply throw it at Claude, hit an error, ask Claude about the error, hit the next error, ask about that one like a no-brainer. Six hours of tweaking compile parameters over and over, eventually it compiled successfully (phew). I installed it on the router, and it worked!

The Dependency Investigation

But I needed to understand why it works. Time to do a proper investigation.

I started digging into the Makefile syntax, the build parameters, and the dependency structure. But mastering every parameter would be impossible because many of them are deeply encapsulated, documented in their own specific docs with their own quirks.

My approach is simply upstream-first, and I wanted to use official packages whenever possible to maintain generality. The original project compiled custom versions of NSS, NSPR, libknet, and libqb, plus a homebrewed corosync-nss-tools and that's a lot of custom packages to maintain.

To verify what was actually necessary, I used a trick from Claude during our earlier sessions: running ldd against a binary shows you exactly which shared libraries it actually links to at runtime. So I ran it against the corosync-qnetd binary on the router:

ldd /usr/sbin/corosync-qnetd
  /lib/ld-musl-mipsel-sf.so.1
  libnss3.so => /usr/lib/libnss3.so
  libssl3.so => /usr/lib/libssl3.so
  libnspr4.so => /usr/lib/libnspr4.so
  libgcc_s.so.1 => /lib/libgcc_s.so.1
  libc.so => /lib/ld-musl-mipsel-sf.so.1
  libnssutil3.so => /usr/lib/libnssutil3.so
  libplc4.so => /usr/lib/libplc4.so
  libplds4.so => /usr/lib/libplds4.so

And that's pretty much it, NSS and NSPR. Nothing else.

Which means that of the four dependencies bundled in the original project, two of them, libknet and libqb, were completely unnecessary for qnetd.

libknet handles inter-node network communication for the corosync cluster daemon itself; libqb provides logging and IPC for the corosync main process.

But the qnetd server is an independent program, it doesn't use either of them. Not to mention the author's project README said it was for qnetd, yet it compiled and bundled libraries that only the cluster daemon needs. My best guess is that they looked at corosync's upstream README, which lists all dependencies for the full corosync project (including the qdevice client), and just compiled everything at once.

The Portability Principle

My goal became: compile only corosync-qnetd itself, and let the package manager handle everything else through official repositories.

The final dependency chain is clean:

corosync-qnetd (custom-compiled, the only custom package)
├── libnss    ← official OpenWrt package
├── nspr   ← official OpenWrt package (pulled in by libnss)
├── nss-utils ← official OpenWrt package (includes `certutil`, `pk12util` used for certificate management)
└── openssl-util ← official OpenWrt package (needed by the `certutil` signing script)

The Technical Challenges

With the "what" and "why" covered, here are the significant technical problems I encountered and how they were solved, in roughly the order I hit them.

NSS Cross-Compilation: The Deepest Pit

corosync-qnetd depends on Mozilla NSS (Network Security Services) for TLS. NSS doesn't use autoconf, it has its own build system called coreconf, driven by environment variables to determine the target platform. The key variables are defined in coreconf/arch.mk:

OS_ARCH := $(subst /,_,$(shell uname -s))   # e.g. "Linux"
OS_TEST := $(shell uname -m)      # e.g. "x86_64", "mipsel"

When OS_ARCH equals Linux, the build system loads coreconf/Linux.mk for platform-specific flags. The problem is: in a cross-compilation environment, uname -s and uname -m return the host machine's values, not the target's. If you don't override these variables, NSS would happily detects my x86_64 build server, uses the host's cc, and produces x86 object files. The link stage then explodes because you're trying to link x86 objects against mipsel libraries.

The original aarch64 project had set OS_ARCH=aarch64 with USE_64=1, which was non-standard, the OpenWrt official NSS Makefile uses OS_ARCH=Linux and OS_TEST=$(ARCH) instead. The aarch64 build likely worked despite the non-standard OS_ARCH value due to other variables compensating, but that approach couldn't be carried over to mipsel.

Diagnosing this took a while. The build log showed a wall of link errors that initially looked like missing libraries. The real clue was checking the object files with file command, they were all x86 binaries, not mipsel.

The fix was to follow the same pattern as OpenWrt's official NSS package, explicitly tell NSS everything it needs to know about the target:

OS_ARCH=Linux        # load coreconf/Linux.mk
OS_TARGET=Linux      # what NSS actually checks internally
OS_TEST=mipsel       # equivalent to uname -m on the target
CPU_ARCH=mipsel      # target CPU architecture
# USE_64 not set    # mipsel_24kc is 32-bit

Through all of this I also learned some C compilation basics, every dependency's headers and libraries need to be present in the build environment before anything will compile, and how OpenWrt's feed system works to pull in those dependencies automatically.

NSPR Header File Mess

Corosync depends on nss, and nss depends on NSPR (Netscape Portable Runtime), NSS's sub-makefiles hardcode the search paths for NSPR headers and libraries, paths like dist/<OBJDIR>/include and dist/public/nspr. Setting the NSPR_INCLUDE_DIR environment variable isn't enough because different NSS build modules look for NSPR in different places using different methods.

The fix was to manually stage NSPR's headers and libraries into the directory structure NSS expects before compilation:

mkdir -p $(PKG_BUILD_DIR)/dist/target.OBJ/include
cp -fpRL $(STAGING_DIR)/usr/include/nspr/. \
    $(PKG_BUILD_DIR)/dist/target.OBJ/include/
mkdir -p $(PKG_BUILD_DIR)/dist/public/nspr
cp -fpRL $(STAGING_DIR)/usr/include/nspr/. \
    $(PKG_BUILD_DIR)/dist/public/nspr/
mkdir -p $(PKG_BUILD_DIR)/dist/target.OBJ/lib
for lib in libnspr4.so libplc4.so libplds4.so; do \
    cp -fpL $(STAGING_DIR)/usr/lib/$$lib \
        $(PKG_BUILD_DIR)/dist/target.OBJ/lib/ 2>/dev/null || true; \
done

A host-native version of nsinstall (a build tool NSS needs to run on the host during compilation) also had to be compiled first, otherwise it would try to run a mipsel binary on the host machine.

The ar Parameter

All .c files compiled correctly with the cross-compiler, but linking produced an error:

mipsel-openwrt-linux-musl-ar cr target.OBJ/arena.o target.OBJ/error.o ...
mipsel-openwrt-linux-musl-ar: target.OBJ/arena.o: file format not recognized

ar was treating the first .o file as an existing archive. The cause: NSS's makefile adds cr flags when invoking ar, but I had also set AR="$(TARGET_CROSS)ar cr" in the OpenWrt Makefile. Two crs stacked together, mangling the command-line arguments entirely. The fix was changing AR from $(TARGET_CROSS)ar cr to just $(TARGET_CROSS)ar.

Trimming NSS: Cut to the Bone

A full NSS build is painfully slow and drags in sqlite3 and a bunch of other dependencies. corosync-qnetd only uses a small subset of NSS functionality. Only the six .so files qnetd actually needs were kept: libnss3, libssl3, libsmime3, libnssutil3, libsoftokn3, and libfreebl3. The original project already had the idea here, it used NSS_DISABLE_DBM=1 to eliminate the sqlite3 dependency and NSS_DISABLE_LIBPKIX=1 to skip PKIX certificate path validation. I kept both of these flags. But the package relationship required careful handling, so I created a separate package called nss-qnetd instead of reusing the original's NSS build, for two reasons.

First, naming. The package needed to coexist with the official libnss in OpenWrt's package system without conflicting. nss-qnetd declares PROVIDES:=libnss so OpenWrt's package manager can resolve the libnss3.so dependency at compile time. But at runtime on the router, corosync-qnetd's runtime dependency points to the official libnss from the feeds — compile with the stripped version for headers and libraries, run with the package from upstream.

Second, improved robustness. The original project used patch files to modify NSS source files like nss/cmd/manifest.mn. Patches depend on exact line numbers, when I upgraded to NSS 3.112, the line counts didn't match and the patch failed with malformed patch at line 3, so by replacing all patches with direct shell commands in Build/Compile gives it a more robust way of injecting params.

The custom `corosync-nss-tools`

Then there was corosync-nss-tools, a package the original author created that doesn't exist upstream. It bundled certutil and pk12util (NSS command-line tools for managing certificate databases) together with custom shell wrapper scripts that reimplemented the certificate initialisation logic. But OpenWrt's official feeds already have an nss-utils package that provides certutil and pk12util. And the upstream corosync-qdevice project already ships its own corosync-qnetd-certutil script. There was no need to repackage binaries that are already available or to rewrite scripts that upstream already maintains. Removing corosync-nss-tools entirely and depending on the official nss-utils was the straightforward improvement.

The musl dladdr Trap

This fix was inherited from the original project and is worth documenting. NSS's libsoftokn3.so calls dladdr() at runtime to find its own file path, then uses that path to locate and load libfreebl3.so. This works fine on glibc, but musl's dladdr() implementation behaves differently — in certain cases it can't correctly resolve the path of an already-loaded library, causing softokn to fail to find freebl. Mozilla bug tracker Bug 511312 confirmed this.

The original author's fix was to use patchelf after compilation to add an explicit dependency on libfreebl3.so to libsoftokn3.so:

find $(PKG_BUILD_DIR)/dist -name libsoftokn3.so -exec \
    $(STAGING_DIR_HOST)/bin/patchelf --add-needed libfreebl3.so {} \;

This makes the dynamic linker automatically load freebl when loading softokn, bypassing the dladdr() issue entirely. Notably, the OpenWrt official NSS Makefile uses a related approach — it sets FREEBL_NO_DEPEND=1 to handle the same softokn-freebl loading relationship.

OpenWrt 25.12: Big Changes to Package Management

With cross-compilation sorted, I ran into a series of changes in the latest OpenWrt 25.12.

The biggest change was the package manager switch from opkg to apk (Alpine Package Keeper). Output format changed from .ipk to .apk. Fortunately, the OpenWrt SDK 25.12 natively outputs apk format, I just needed to update all .ipk references to .apk in the build scripts.

But after installing the corosync-qnetd apk, more insidious problems were at runtime. Running the certificate initialization script corosync-qnetd-certutil -i produced a barrage of errors:

stat: applet not found
chown: unrecognized option '--reference'
sha1sum: command not found
ps: unrecognized option: e

OpenWrt's userspace tools are mostly BusyBox, stripped-down versions with limited options. BusyBox's chown doesn't support --reference, the stat applet wasn't compiled into this firmware's BusyBox, ps doesn't support the -e flag. The certutil script was written for a full GNU/Linux environment and fell apart in BusyBox land.

Full GNU versions of these tools were needed. But here was another trap: OpenWrt 25.12 splits coreutils into dozens of individual sub-packages (coreutils-chown, coreutils-stat, coreutils-sha1sum, ...). The main coreutils package is an empty shell — installing it installs nothing. apk info -L coreutils outputs exactly one line: lib/apk/packages/coreutils.list.

The final list of runtime dependencies that needed to be installed individually: bash, coreutils-chown, coreutils-stat, coreutils-sha1sum, procps-ng (for a full ps), openssh-sftp-server (PVE's scp defaults to SFTP mode — without sftp-server on the router, certificate copying fails), and openssl-util (the certutil signing script needs the openssl command-line tool).

All of these were added to the Makefile's DEPENDS, so a apk add corosync-qnetd pulls in everything automatically.

The Final Result

After all the pitfalls were navigated, the end product is one clean OpenWrt package. Installation only takes a single command and all dependencies are pulled automatically. Initialise the certificate database (key generation takes about five to ten minutes on mipsel hardware), start the service, then run pvecm qdevice setup <router IP> on any PVE node. Certificates exchange automatically, and the cluster recognises the Qdevice.

The core changes in the final Makefile: corrected NSS architecture parameters from aarch64 to the proper mipsel 32-bit combination; trimmed NSS to compile only the libraries qnetd needs; added all the runtime dependencies and removed unnecessary libknet and libqb; migrated from ipk to apk.

Porting corosync-qnetd to mipsel means thousands of MIPS-based routers running OpenWrt can now serve as quorum arbitrators for Proxmox clusters.

A full day's work, morning to night. Worth it.

GitHub Releases: https://github.com/osmiumsilver/corosync-qnetd-openwrt-mipsel/releases

How a missing @ in a filename broke my Netlify build

Osmium — Mon, 27 Apr 2026 12:18:09 +0000

It was fine three months ago :(

Three months ago I deployed my Qwik site to Netlify. And after some while, I pushed an new, unrelated content change and the build died without touching a thing on it’s codebase:

Edge Functions bundling
────────────────────────────────────────────────────────────────

Packaging Edge Functions from .netlify/edge-functions directory:
 - entry.netlify-edge

[Error: ENOENT: no such file or directory, stat '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'
}
Node.js v22.22.1

No version bumps in package.json. No config changes. The build itself (client, server) compiled cleanly. If it worked three months ago with the exact same versions, why was it broken now?

I did the usual steps: Cleared cache and ran the build, failed. But build process worked fine on local machine. Upgraded all dependencies with ncu to their latest versions, pushed again, still failed. Until I created a brand new Qwik hello-world project, pushed it to Netlify. Still failed, that’s when I knew there must be something wrong on Netlify’s side.

Searching online for "qwik netlify build error ENOENT file not found" wasn't really helpful, found only one forum post describing the same issue. No replies. Auto-closed after seven days. So I posted on Netlify's answers forum and hoping someone could give a solution. A response did come eventually, but hours later. The site was down and needed to come back up ASAP, so I wasn't going to sit there refreshing a forum tab. I started digging deep.

The First Clue: A Missing @

I noticed the failure was in the Edge Functions bundling step, an additional step after everything else was done. I opened the output directory on my local machine expecting to see the missing file:

.netlify/edge-functions/entry.netlify-edge/
├── @qwik-city-not-found-paths.js  ← Here it is!
├── @qwik-city-plan.js
├── @qwik-city-static-paths.js
├── entry.netlify-edge.js
└── ...

That's when it gets interesting, the file existed. It just had an @ in front of it. Yet the error occurs when attempting to call stat on a file with a filename that is almost identical, except for the missing @?

If the bundler knew this file existed, why was it looking for a file with the wrong name?

Something was stripping the @ somewhere in the process.

Following the Trail

I pulled down the source of @netlify/build from Netlify’s GitHub repo including the package edge-bundler, which responsible for that Edge Functions bundling step. I followed the call chain:

edge_functions/index.ts → bundler.ts → formats/tarball.ts

In tarball.ts, I found the function file list being built:

const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

await tar.create({ cwd: bundleDir.path, file: tarballPath, gzip: true }, files)

the code packs bundled edge functions into a tarball for deployment. It calls tar.create() with a list of filenames. And at that point, the filenames still had their @ prefix intact. The correct name was being passed in. So the issue wasn't in the path calculation. The @ was being stripped inside tar.create() itself.

Reproducing the Bug

To confirm, I put together a minimal reproduction with some help from Claude:

import * as tar from 'tar'
import fs from 'fs'
import os from 'os'
import path from 'path'

const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'))
fs.writeFileSync(path.join(tmpDir, '@test-file.js'), 'export default 1')

// Without fix: ENOENT — @ is stripped
await tar.create(
  { cwd: tmpDir, file: '/tmp/out.tar.gz', gzip: true },
  ['@test-file.js']
)

And i got:

[Error: ENOENT: no such file or directory, stat '.../test-file.js']

The bug was confirmed: node-tar WAS doing something on filenames that result in the removal of the @ symbol.

Root Cause: bsdtar (libarchive)’s `@archive` syntax

After some digging online, bsdtar/libarchive has a feature where entries starting with @ have special meaning: @archive.tar means "open that archive and include its contents". node-tar, as a tar implementation, inherited this behavior. When it sees a filename starting with @ in the files list, it strips the @ symbol and tries to open the remainder as a tar archive to read entries from.

So when node-tar anything starts with a @ symbol:

→ Interpret this as an archive-include directive
→ It tried to open qwik-city-not-found-paths.js as a tar archive
→ Run stat on qwik-city-not-found-paths.js
→ ENOENT

The Fix

In tarball.ts, where the files array is passed to tar.create(), prefix each entry with ./:

// Before
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

// After
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => './' + getUnixPath(p))  // ← this line
  .sort()

The Chain Reaction

After the PR was merged into @netlify/build, the maintainer traced the issue further upstream and submitted a PR to node-tar itself — fixing the error handling so that errors from @-prefixed entries are properly catchable instead of becoming unhandled promise rejections that the caller couldn't catch.

Moreover, somehow it also uncovered a CI coverage gap: the test suite for tarball handling had a describe block that was being silently skipped because of a wrong Deno version number in the test configuration. An entire set of tarball tests had been passing CI without actually running.

One missing @, and suddenly there were three fixes flowing in different directions.

Looking Back

Reading someone else's TypeScript to find out why my site wouldn't deploy is not something I'd done before this week. The typical workaround for this kind of issue you would find online, is to duplicate the files without the @. That works. Your build passes. You move on. But somewhere the same bug is still waiting for the next person. And the next. Until someone reads the function that's eating the @.

There's something I've been thinking about since. The experienced developers who maintain these projects, they do this kind of invisible work all the time, and they do it faster than I ever could. They trace a bug, fix it, and the result is that thousands of people simply never encounter the problem. Nobody knows it was ever there.

I'm not a developer by trade. If these projects had been closed-source, my story would have ended at the forum post with no replies, waiting for someone on the other side to notice. But because the code was right there, I could pull it down and find the answer myself. A workaround solves your problem. A root cause fix solves everyone's.

DEV Community: Osmium

How Tailscale's Route Table Gets Wiped by mwan3 on OpenWrt

Background: routing tables, ip rule, and firewall marks

The investigation: a few wrong turns first

First guess: mwan3 must be deleting something

Second guess: This nft seems to be doing weird things

Finding the code: mwan3's flush loop

Why mmdefault - 3: how the threshold is computed

The Solution

One small wrinkle

Verification

Sources

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Why Bother

Finding a Starting Point

The Dependency Investigation

The Portability Principle

The Technical Challenges

NSS Cross-Compilation: The Deepest Pit

NSPR Header File Mess

The ar Parameter

Trimming NSS: Cut to the Bone

The custom `corosync-nss-tools`

The musl dladdr Trap

OpenWrt 25.12: Big Changes to Package Management

The Final Result

More Links

How a missing @ in a filename broke my Netlify build

It was fine three months ago :(

The First Clue: A Missing @

Following the Trail

Reproducing the Bug

Root Cause: bsdtar (libarchive)’s `@archive` syntax

The Fix

The Chain Reaction

Looking Back

Links

DEV Community: Osmium

How Tailscale's Route Table Gets Wiped by mwan3 on OpenWrt

Background: routing tables, ip rule, and firewall marks

The investigation: a few wrong turns first

First guess: mwan3 must be deleting something

Second guess: This nft seems to be doing weird things

Finding the code: mwan3's flush loop

Why mmdefault - 3: how the threshold is computed

The Solution

One small wrinkle

Verification

Sources

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Why Bother

Finding a Starting Point

The Dependency Investigation

The Portability Principle

The Technical Challenges

NSS Cross-Compilation: The Deepest Pit

NSPR Header File Mess

The ar Parameter

Trimming NSS: Cut to the Bone

The custom corosync-nss-tools

The musl dladdr Trap

OpenWrt 25.12: Big Changes to Package Management

The Final Result

More Links

How a missing @ in a filename broke my Netlify build

It was fine three months ago :(

The First Clue: A Missing @

Following the Trail

Reproducing the Bug

Root Cause: bsdtar (libarchive)’s @archive syntax

The Fix

The Chain Reaction

Looking Back

Links

The custom `corosync-nss-tools`

Root Cause: bsdtar (libarchive)’s `@archive` syntax