DEV Community: Osmium

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Osmium — Fri, 01 May 2026 14:26:19 +0000

Why Bother

I have a two-node Proxmox VE cluster, and two-node clusters have an inherent distributed systems problem: if either machine goes down, the surviving node can't tell whether it should keep running or not. Running a quorum usually requires a majority vote, which normally means at least three voters. With only two machines, one vote each, lose one and you're down to one vote. No quorum and your cluster stops.

The solution is a quorum device. Proxmox supports corosync-qnetd, a lightweight daemon from the corosync project that acts as a third-party tiebreaker. A qnetd doesn't participate in storage or compute, it just casts the deciding vote during a split-brain scenario. By bringing the total vote count to three, the cluster maintains a majority requirement and consequently, if any node fails, the surviving node and the router’s vote can collectively sustain quorum, making the cluster keeps running.

The ideal host for something like this is a device in a home network that basically never gets turned off, like a router.

I had a Xiaomi Mini router running OpenWrt, MT7620 chip, 128MB DDR2 memory, 16MB SPI Flash, mipsel_24kc architecture. It runs 24/7, draws a few watts, perfect for the job. The problem: there's no corosync-qnetd package for mipsel in the official OpenWrt feeds. :(

Finding a Starting Point

After some searching, I found someone on GitHub who had compiled corosync-qnetd for aarch64 (ARM 64-bit) OpenWrt, targeting GL.iNet routers and using the old opkg/ipk package format. A good starting point.

I had essentially zero experience with C, Makefiles, or cross-compilation packaging. I knew C programs need a Makefile to build, but that was about it. So my approach was simply throw it at Claude, hit an error, ask Claude about the error, hit the next error, ask about that one like a no-brainer. Six hours of tweaking compile parameters over and over, eventually it compiled successfully (phew). I installed it on the router, and it worked!

The Dependency Investigation

But I needed to understand why it works. Time to do a proper investigation.

I started digging into the Makefile syntax, the build parameters, and the dependency structure. But mastering every parameter would be impossible because many of them are deeply encapsulated, documented in their own specific docs with their own quirks.

My approach is simply upstream-first, and I wanted to use official packages whenever possible to maintain generality. The original project compiled custom versions of NSS, NSPR, libknet, and libqb, plus a homebrewed corosync-nss-tools and that's a lot of custom packages to maintain.

To verify what was actually necessary, I used a trick from Claude during our earlier sessions: running ldd against a binary shows you exactly which shared libraries it actually links to at runtime. So I ran it against the corosync-qnetd binary on the router:

ldd /usr/sbin/corosync-qnetd
  /lib/ld-musl-mipsel-sf.so.1
  libnss3.so => /usr/lib/libnss3.so
  libssl3.so => /usr/lib/libssl3.so
  libnspr4.so => /usr/lib/libnspr4.so
  libgcc_s.so.1 => /lib/libgcc_s.so.1
  libc.so => /lib/ld-musl-mipsel-sf.so.1
  libnssutil3.so => /usr/lib/libnssutil3.so
  libplc4.so => /usr/lib/libplc4.so
  libplds4.so => /usr/lib/libplds4.so

And that's pretty much it, NSS and NSPR. Nothing else.

Which means that of the four dependencies bundled in the original project, two of them, libknet and libqb, were completely unnecessary for qnetd.

libknet handles inter-node network communication for the corosync cluster daemon itself; libqb provides logging and IPC for the corosync main process.

But the qnetd server is an independent program, it doesn't use either of them. Not to mention the author's project README said it was for qnetd, yet it compiled and bundled libraries that only the cluster daemon needs. My best guess is that they looked at corosync's upstream README, which lists all dependencies for the full corosync project (including the qdevice client), and just compiled everything at once.

The Portability Principle

My goal became: compile only corosync-qnetd itself, and let the package manager handle everything else through official repositories.

The final dependency chain is clean:

corosync-qnetd (custom-compiled, the only custom package)
├── libnss    ← official OpenWrt package
├── nspr   ← official OpenWrt package (pulled in by libnss)
├── nss-utils ← official OpenWrt package (includes `certutil`, `pk12util` used for certificate management)
└── openssl-util ← official OpenWrt package (needed by the `certutil` signing script)

The Technical Challenges

With the "what" and "why" covered, here are the significant technical problems I encountered and how they were solved, in roughly the order I hit them.

NSS Cross-Compilation: The Deepest Pit

corosync-qnetd depends on Mozilla NSS (Network Security Services) for TLS. NSS doesn't use autoconf, it has its own build system called coreconf, driven by environment variables to determine the target platform. The key variables are defined in coreconf/arch.mk:

OS_ARCH := $(subst /,_,$(shell uname -s))   # e.g. "Linux"
OS_TEST := $(shell uname -m)      # e.g. "x86_64", "mipsel"

When OS_ARCH equals Linux, the build system loads coreconf/Linux.mk for platform-specific flags. The problem is: in a cross-compilation environment, uname -s and uname -m return the host machine's values, not the target's. If you don't override these variables, NSS would happily detects my x86_64 build server, uses the host's cc, and produces x86 object files. The link stage then explodes because you're trying to link x86 objects against mipsel libraries.

The original aarch64 project had set OS_ARCH=aarch64 with USE_64=1, which was non-standard, the OpenWrt official NSS Makefile uses OS_ARCH=Linux and OS_TEST=$(ARCH) instead. The aarch64 build likely worked despite the non-standard OS_ARCH value due to other variables compensating, but that approach couldn't be carried over to mipsel.

Diagnosing this took a while. The build log showed a wall of link errors that initially looked like missing libraries. The real clue was checking the object files with file command, they were all x86 binaries, not mipsel.

The fix was to follow the same pattern as OpenWrt's official NSS package, explicitly tell NSS everything it needs to know about the target:

OS_ARCH=Linux        # load coreconf/Linux.mk
OS_TARGET=Linux      # what NSS actually checks internally
OS_TEST=mipsel       # equivalent to uname -m on the target
CPU_ARCH=mipsel      # target CPU architecture
# USE_64 not set    # mipsel_24kc is 32-bit

Through all of this I also learned some C compilation basics, every dependency's headers and libraries need to be present in the build environment before anything will compile, and how OpenWrt's feed system works to pull in those dependencies automatically.

NSPR Header File Mess

Corosync depends on nss, and nss depends on NSPR (Netscape Portable Runtime), NSS's sub-makefiles hardcode the search paths for NSPR headers and libraries, paths like dist/<OBJDIR>/include and dist/public/nspr. Setting the NSPR_INCLUDE_DIR environment variable isn't enough because different NSS build modules look for NSPR in different places using different methods.

The fix was to manually stage NSPR's headers and libraries into the directory structure NSS expects before compilation:

mkdir -p $(PKG_BUILD_DIR)/dist/target.OBJ/include
cp -fpRL $(STAGING_DIR)/usr/include/nspr/. \
    $(PKG_BUILD_DIR)/dist/target.OBJ/include/
mkdir -p $(PKG_BUILD_DIR)/dist/public/nspr
cp -fpRL $(STAGING_DIR)/usr/include/nspr/. \
    $(PKG_BUILD_DIR)/dist/public/nspr/
mkdir -p $(PKG_BUILD_DIR)/dist/target.OBJ/lib
for lib in libnspr4.so libplc4.so libplds4.so; do \
    cp -fpL $(STAGING_DIR)/usr/lib/$$lib \
        $(PKG_BUILD_DIR)/dist/target.OBJ/lib/ 2>/dev/null || true; \
done

A host-native version of nsinstall (a build tool NSS needs to run on the host during compilation) also had to be compiled first, otherwise it would try to run a mipsel binary on the host machine.

The ar Parameter

All .c files compiled correctly with the cross-compiler, but linking produced an error:

mipsel-openwrt-linux-musl-ar cr target.OBJ/arena.o target.OBJ/error.o ...
mipsel-openwrt-linux-musl-ar: target.OBJ/arena.o: file format not recognized

ar was treating the first .o file as an existing archive. The cause: NSS's makefile adds cr flags when invoking ar, but I had also set AR="$(TARGET_CROSS)ar cr" in the OpenWrt Makefile. Two crs stacked together, mangling the command-line arguments entirely. The fix was changing AR from $(TARGET_CROSS)ar cr to just $(TARGET_CROSS)ar.

Trimming NSS: Cut to the Bone

A full NSS build is painfully slow and drags in sqlite3 and a bunch of other dependencies. corosync-qnetd only uses a small subset of NSS functionality. Only the six .so files qnetd actually needs were kept: libnss3, libssl3, libsmime3, libnssutil3, libsoftokn3, and libfreebl3. The original project already had the idea here, it used NSS_DISABLE_DBM=1 to eliminate the sqlite3 dependency and NSS_DISABLE_LIBPKIX=1 to skip PKIX certificate path validation. I kept both of these flags. But the package relationship required careful handling, so I created a separate package called nss-qnetd instead of reusing the original's NSS build, for two reasons.

First, naming. The package needed to coexist with the official libnss in OpenWrt's package system without conflicting. nss-qnetd declares PROVIDES:=libnss so OpenWrt's package manager can resolve the libnss3.so dependency at compile time. But at runtime on the router, corosync-qnetd's runtime dependency points to the official libnss from the feeds — compile with the stripped version for headers and libraries, run with the package from upstream.

Second, improved robustness. The original project used patch files to modify NSS source files like nss/cmd/manifest.mn. Patches depend on exact line numbers, when I upgraded to NSS 3.112, the line counts didn't match and the patch failed with malformed patch at line 3, so by replacing all patches with direct shell commands in Build/Compile gives it a more robust way of injecting params.

The custom `corosync-nss-tools`

Then there was corosync-nss-tools, a package the original author created that doesn't exist upstream. It bundled certutil and pk12util (NSS command-line tools for managing certificate databases) together with custom shell wrapper scripts that reimplemented the certificate initialisation logic. But OpenWrt's official feeds already have an nss-utils package that provides certutil and pk12util. And the upstream corosync-qdevice project already ships its own corosync-qnetd-certutil script. There was no need to repackage binaries that are already available or to rewrite scripts that upstream already maintains. Removing corosync-nss-tools entirely and depending on the official nss-utils was the straightforward improvement.

The musl dladdr Trap

This fix was inherited from the original project and is worth documenting. NSS's libsoftokn3.so calls dladdr() at runtime to find its own file path, then uses that path to locate and load libfreebl3.so. This works fine on glibc, but musl's dladdr() implementation behaves differently — in certain cases it can't correctly resolve the path of an already-loaded library, causing softokn to fail to find freebl. Mozilla bug tracker Bug 511312 confirmed this.

The original author's fix was to use patchelf after compilation to add an explicit dependency on libfreebl3.so to libsoftokn3.so:

find $(PKG_BUILD_DIR)/dist -name libsoftokn3.so -exec \
    $(STAGING_DIR_HOST)/bin/patchelf --add-needed libfreebl3.so {} \;

This makes the dynamic linker automatically load freebl when loading softokn, bypassing the dladdr() issue entirely. Notably, the OpenWrt official NSS Makefile uses a related approach — it sets FREEBL_NO_DEPEND=1 to handle the same softokn-freebl loading relationship.

OpenWrt 25.12: Big Changes to Package Management

With cross-compilation sorted, I ran into a series of changes in the latest OpenWrt 25.12.

The biggest change was the package manager switch from opkg to apk (Alpine Package Keeper). Output format changed from .ipk to .apk. Fortunately, the OpenWrt SDK 25.12 natively outputs apk format, I just needed to update all .ipk references to .apk in the build scripts.

But after installing the corosync-qnetd apk, more insidious problems were at runtime. Running the certificate initialization script corosync-qnetd-certutil -i produced a barrage of errors:

stat: applet not found
chown: unrecognized option '--reference'
sha1sum: command not found
ps: unrecognized option: e

OpenWrt's userspace tools are mostly BusyBox, stripped-down versions with limited options. BusyBox's chown doesn't support --reference, the stat applet wasn't compiled into this firmware's BusyBox, ps doesn't support the -e flag. The certutil script was written for a full GNU/Linux environment and fell apart in BusyBox land.

Full GNU versions of these tools were needed. But here was another trap: OpenWrt 25.12 splits coreutils into dozens of individual sub-packages (coreutils-chown, coreutils-stat, coreutils-sha1sum, ...). The main coreutils package is an empty shell — installing it installs nothing. apk info -L coreutils outputs exactly one line: lib/apk/packages/coreutils.list.

The final list of runtime dependencies that needed to be installed individually: bash, coreutils-chown, coreutils-stat, coreutils-sha1sum, procps-ng (for a full ps), openssh-sftp-server (PVE's scp defaults to SFTP mode — without sftp-server on the router, certificate copying fails), and openssl-util (the certutil signing script needs the openssl command-line tool).

All of these were added to the Makefile's DEPENDS, so a apk add corosync-qnetd pulls in everything automatically.

The Final Result

After all the pitfalls were navigated, the end product is one clean OpenWrt package. Installation only takes a single command and all dependencies are pulled automatically. Initialise the certificate database (key generation takes about five to ten minutes on mipsel hardware), start the service, then run pvecm qdevice setup <router IP> on any PVE node. Certificates exchange automatically, and the cluster recognises the Qdevice.

The core changes in the final Makefile: corrected NSS architecture parameters from aarch64 to the proper mipsel 32-bit combination; trimmed NSS to compile only the libraries qnetd needs; added all the runtime dependencies and removed unnecessary libknet and libqb; migrated from ipk to apk.

Porting corosync-qnetd to mipsel means thousands of MIPS-based routers running OpenWrt can now serve as quorum arbitrators for Proxmox clusters.

A full day's work, morning to night. Worth it.

GitHub Releases: https://github.com/osmiumsilver/corosync-qnetd-openwrt-mipsel/releases

How a missing @ in a filename broke my Netlify build

Osmium — Mon, 27 Apr 2026 12:18:09 +0000

It was fine three months ago :(

Three months ago I deployed my Qwik site to Netlify. And after some while, I pushed an new, unrelated content change and the build died without touching a thing on it’s codebase:

Edge Functions bundling
────────────────────────────────────────────────────────────────

Packaging Edge Functions from .netlify/edge-functions directory:
 - entry.netlify-edge

[Error: ENOENT: no such file or directory, stat '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'
}
Node.js v22.22.1

No version bumps in package.json. No config changes. The build itself (client, server) compiled cleanly. If it worked three months ago with the exact same versions, why was it broken now?

I did the usual steps: Cleared cache and ran the build, failed. But build process worked fine on local machine. Upgraded all dependencies with ncu to their latest versions, pushed again, still failed. Until I created a brand new Qwik hello-world project, pushed it to Netlify. Still failed, that’s when I knew there must be something wrong on Netlify’s side.

Searching online for "qwik netlify build error ENOENT file not found" wasn't really helpful, found only one forum post describing the same issue. No replies. Auto-closed after seven days. So I posted on Netlify's answers forum and hoping someone could give a solution. A response did come eventually, but hours later. The site was down and needed to come back up ASAP, so I wasn't going to sit there refreshing a forum tab. I started digging deep.

The First Clue: A Missing @

I noticed the failure was in the Edge Functions bundling step, an additional step after everything else was done. I opened the output directory on my local machine expecting to see the missing file:

.netlify/edge-functions/entry.netlify-edge/
├── @qwik-city-not-found-paths.js  ← Here it is!
├── @qwik-city-plan.js
├── @qwik-city-static-paths.js
├── entry.netlify-edge.js
└── ...

That's when it gets interesting, the file existed. It just had an @ in front of it. Yet the error occurs when attempting to call stat on a file with a filename that is almost identical, except for the missing @?

If the bundler knew this file existed, why was it looking for a file with the wrong name?

Something was stripping the @ somewhere in the process.

Following the Trail

I pulled down the source of @netlify/build from Netlify’s GitHub repo including the package edge-bundler, which responsible for that Edge Functions bundling step. I followed the call chain:

edge_functions/index.ts → bundler.ts → formats/tarball.ts

In tarball.ts, I found the function file list being built:

const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

await tar.create({ cwd: bundleDir.path, file: tarballPath, gzip: true }, files)

the code packs bundled edge functions into a tarball for deployment. It calls tar.create() with a list of filenames. And at that point, the filenames still had their @ prefix intact. The correct name was being passed in. So the issue wasn't in the path calculation. The @ was being stripped inside tar.create() itself.

Reproducing the Bug

To confirm, I put together a minimal reproduction with some help from Claude:

import * as tar from 'tar'
import fs from 'fs'
import os from 'os'
import path from 'path'

const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'))
fs.writeFileSync(path.join(tmpDir, '@test-file.js'), 'export default 1')

// Without fix: ENOENT — @ is stripped
await tar.create(
  { cwd: tmpDir, file: '/tmp/out.tar.gz', gzip: true },
  ['@test-file.js']
)

And i got:

[Error: ENOENT: no such file or directory, stat '.../test-file.js']

The bug was confirmed: node-tar WAS doing something on filenames that result in the removal of the @ symbol.

Root Cause: bsdtar (libarchive)’s `@archive` syntax

After some digging online, bsdtar/libarchive has a feature where entries starting with @ have special meaning: @archive.tar means "open that archive and include its contents". node-tar, as a tar implementation, inherited this behavior. When it sees a filename starting with @ in the files list, it strips the @ symbol and tries to open the remainder as a tar archive to read entries from.

So when node-tar anything starts with a @ symbol:

→ Interpret this as an archive-include directive
→ It tried to open qwik-city-not-found-paths.js as a tar archive
→ Run stat on qwik-city-not-found-paths.js
→ ENOENT

The Fix

In tarball.ts, where the files array is passed to tar.create(), prefix each entry with ./:

// Before
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

// After
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => './' + getUnixPath(p))  // ← this line
  .sort()

The Chain Reaction

After the PR was merged into @netlify/build, the maintainer traced the issue further upstream and submitted a PR to node-tar itself — fixing the error handling so that errors from @-prefixed entries are properly catchable instead of becoming unhandled promise rejections that the caller couldn't catch.

Moreover, somehow it also uncovered a CI coverage gap: the test suite for tarball handling had a describe block that was being silently skipped because of a wrong Deno version number in the test configuration. An entire set of tarball tests had been passing CI without actually running.

One missing @, and suddenly there were three fixes flowing in different directions.

Looking Back

Reading someone else's TypeScript to find out why my site wouldn't deploy is not something I'd done before this week. The typical workaround for this kind of issue you would find online, is to duplicate the files without the @. That works. Your build passes. You move on. But somewhere the same bug is still waiting for the next person. And the next. Until someone reads the function that's eating the @.

There's something I've been thinking about since. The experienced developers who maintain these projects, they do this kind of invisible work all the time, and they do it faster than I ever could. They trace a bug, fix it, and the result is that thousands of people simply never encounter the problem. Nobody knows it was ever there.

I'm not a developer by trade. If these projects had been closed-source, my story would have ended at the forum post with no replies, waiting for someone on the other side to notice. But because the code was right there, I could pull it down and find the answer myself. A workaround solves your problem. A root cause fix solves everyone's.

DEV Community: Osmium

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Why Bother

Finding a Starting Point

The Dependency Investigation

The Portability Principle

The Technical Challenges

NSS Cross-Compilation: The Deepest Pit

NSPR Header File Mess

The ar Parameter

Trimming NSS: Cut to the Bone

The custom `corosync-nss-tools`

The musl dladdr Trap

OpenWrt 25.12: Big Changes to Package Management

The Final Result

More Links

How a missing @ in a filename broke my Netlify build

It was fine three months ago :(

The First Clue: A Missing @

Following the Trail

Reproducing the Bug

Root Cause: bsdtar (libarchive)’s `@archive` syntax

The Fix

The Chain Reaction

Looking Back

Links

DEV Community: Osmium

Turn your OpenWrt router into a quorum device for Proxmox VE cluster

Why Bother

Finding a Starting Point

The Dependency Investigation

The Portability Principle

The Technical Challenges

NSS Cross-Compilation: The Deepest Pit

NSPR Header File Mess

The ar Parameter

Trimming NSS: Cut to the Bone

The custom corosync-nss-tools

The musl dladdr Trap

OpenWrt 25.12: Big Changes to Package Management

The Final Result

More Links

How a missing @ in a filename broke my Netlify build

It was fine three months ago :(

The First Clue: A Missing @

Following the Trail

Reproducing the Bug

Root Cause: bsdtar (libarchive)’s @archive syntax

The Fix

The Chain Reaction

Looking Back

Links

The custom `corosync-nss-tools`

Root Cause: bsdtar (libarchive)’s `@archive` syntax