DEV Community: Alex Georgiev

Bun 1.4's bun test --parallel cuts a 4-second suite to about 1 second

Alex Georgiev — Tue, 06 Oct 2026 10:00:00 +0000

I had a twenty-file test suite that took just over four seconds to run. I added one flag, --parallel, and it took one second. Then I tried to push it further and found the exact point where the flag stops helping, which turned out to depend on what kind of work the tests were actually doing, not on how many files there were.

Bun 1.4 shipped on 20 August 2026 and brought bun test --parallel, which distributes test files across worker processes instead of running them one after another in a single process. I'm running Bun 1.4.2 on a 4-core Intel Xeon VM (KVM, 4 logical CPUs, no cgroup limit), and I wanted to know what the flag actually buys you, where it tops out, and what the "implies --isolate" line in the docs means in practice.

The headline number

I wrote twenty test files, two tests each, where every test awaits a 100ms setTimeout before asserting — a stand-in for the API calls and debounces that make a lot of real integration tests slow. Three runs each, fastest and slowest within 15ms of each other:

Mode	Wall time
`bun test` (serial)	4.04s
`--parallel=1`	4.06s
`--parallel=2`	2.05s
`--parallel=4` (default, = CPU count)	1.03s
`--parallel=8`	0.63s

With no argument, --parallel defaults to the number of CPU cores, which on this box is 4. That alone took the suite from 4.04s to 1.03s, a 3.9x speedup that lines up almost exactly with four workers splitting twenty files. What surprised me is that --parallel=8 kept improving, down to 0.63s, on a machine with only four logical cores. I'll get to why in a minute, because it's the most important part of this post.

The opposite reading: CPU-bound tests hit a wall at the core count

A test that awaits setTimeout isn't actually using the CPU while it waits. It's just occupying a slot. So I built a second suite, same shape, twenty files and two tests each, but this time each test runs a real computation: a prime sieve up to 20 million, which takes genuine CPU time with nothing to wait on.

Mode	Wall time
`bun test` (serial)	6.1s
`--parallel=1`	6.2s
`--parallel=2`	3.3s
`--parallel=4`	1.8s
`--parallel=6`	1.9s
`--parallel=8`	1.8s

This is the number that argues against the headline. Scaling is close to linear up to four workers, matching the four physical cores, and then it stops. Going to six or eight workers bought nothing, and in a couple of runs was a touch slower than four, presumably from context-switch overhead as processes fight for the same cores. --parallel schedules OS processes; it doesn't make more CPU appear. For a suite that's actually CPU-bound, the number that matters isn't how many files you have, it's nproc.

Put the two tables together and the real rule is simpler than either one alone: --parallel can scale past your core count for tests that spend most of their time waiting, and it cannot for tests that spend most of their time computing. Most real suites are a mix of both, so the honest advice is to benchmark your own suite rather than assume either curve.

What "implies --isolate" actually isolates

The docs say --parallel implies --isolate, and describe it as giving "each file... a fresh global object even when two files land on the same worker." I wanted to know exactly what that resets, because the phrasing leaves open whether it's per-file or something coarser or finer.

I wrote a module with a plain top-level counter and four test files that import it and log what they see:

// shared.ts
let count = 0;
export function bump() { count++; return count; }

Under plain bun test, the whole run shares one process and one module cache, so the counter climbs across files in whatever order they happen to run:

file-4 sees counter = 1
file-1 sees counter = 2
file-2 sees counter = 3
file-3 sees counter = 4

Under --parallel, every file saw 1:

file-1 sees counter = 1   worker = 1
file-2 sees counter = 1   worker = 1
file-3 sees counter = 1   worker = 1
file-4 sees counter = 1   worker = 1

All four ran in the same worker process (same PID), so this isn't about separate OS processes — it's a fresh JS global per file inside one process, exactly as documented. Adding --no-isolate to --parallel brought the leak straight back: 1, 2, 3, 4 again, same as serial.

The part the one-line doc summary doesn't spell out: isolation is per file, not per test. Three tests inside a single file, run under --parallel, saw the counter climb 1, 2, 3 with no reset between them. If your tests rely on module-level setup running fresh for every single test, not just every file, --isolate won't give you that, parallel or not.

Bun also sets BUN_TEST_WORKER_ID and JEST_WORKER_ID to the worker's 1-based index, which I confirmed by printing both in the test above — worth knowing if you're porting a Jest setup that keys a database name or port off that variable.

--bail stops starting files, not running ones

The docs describe it precisely: "the coordinator handles --bail at file granularity: once the failure threshold is reached it starts no new files, but files already running finish." I tested this against twenty files where the first one fails a 300ms-long test.

Without --bail, under --parallel, all twenty files ran regardless of the failure:

19 pass
1 fail
Ran 20 tests across 20 files. [1.53s]

With --bail added:

Bailed out after 1 failure
3 pass
1 fail
Ran 4 tests across 4 files. [320.00ms]

Four files ran, not one. That matches four workers already mid-flight when the failure landed; the other sixteen files never started. This is a real time saving in CI (1.53s down to 0.32s here), but it also means --bail under --parallel won't tell you about any other failures in those sixteen untouched files, including ones with nothing to do with whatever broke the first.

Coverage merges correctly, refusals are clean, one bad file doesn't take down the run

Three more things I checked, more briefly:

Coverage merging. I split three functions of one module across three test files, each covering one function, and compared --coverage output. Serial and --parallel --coverage reported identical numbers — 75% functions, 60% lines, same uncovered line range — so the claimed coverage merge across workers held up exactly.

Invalid input is rejected before anything runs. --parallel=0, --parallel=-1, and --parallel=abc all produced the same clean message and exit code 1, with nothing executed:

error: --parallel expects a positive integer, received "0"

A crash in one file doesn't sink the run. I had one test file throw at module load time, independent of any test inside it. Under --parallel, the other three files in that batch still ran and passed, and the crash was reported as a distinct "error" rather than lumped in with assertion failures:

bun test v1.4.2 (744846f84) 4x PARALLEL

tests-crash/cr2.test.ts:

# Unhandled error between tests
-------------------------------
error: boom at module load time
-------------------------------

 3 pass
 1 fail
 1 error
Ran 4 tests across 4 files. [11.00ms]

That 4x PARALLEL line in the banner is also the easiest way to confirm from a CI log how many workers actually ran, without adding anything to your own test code.

What I got wrong on the way

My first attempt at a "CPU-bound" suite used while (Date.now() - start < 100) {} as a stand-in for real work, expecting it to behave like the prime-sieve suite. It didn't: --parallel=8 kept getting faster on a 4-core box well past where real CPU work should have plateaued, the same shape as the IO-bound suite. I spent a while assuming Bun was doing something clever with scheduling before I worked out the actual cause: a spin loop checking the wall clock only needs to be scheduled once after its own deadline to notice time has passed and exit. It behaves like a sleep, not like computation, because the OS can preempt it for as long as it likes without the loop caring, so long as real time has moved on by the time it next gets to check. That's why I rebuilt the CPU-bound suite around an actual prime sieve with no clock check in it at all, which is the version in the table above.

Run it yourself

This is a trimmed version of what produced the first table. It needs Bun 1.4 or later.

curl -fsSL https://bun.sh/install | bash
mkdir bun-parallel-demo && cd bun-parallel-demo

for i in $(seq -w 1 20); do
cat > "t$i.test.ts" <<EOF
import { test, expect } from "bun:test";
test("io-$i", async () => {
  await new Promise((r) => setTimeout(r, 100));
  expect(1 + 1).toBe(2);
});
EOF
done

bun test                 # serial baseline
bun test --parallel      # defaults to your CPU count
bun test --parallel=8    # push past it, since these tests just wait

Swap the body of the generated test for a real computation (a sieve, a sort, anything with no timer in it) to see the plateau instead of continued scaling, and compare against nproc on your own machine.

What to do with this

Before turning --parallel on for an existing suite, work out which kind of suite you actually have. If most of your tests wait on timers, sockets, or a database round trip, raising --parallel past your core count is free speed and worth trying immediately. If your tests are doing real computation in-process, there is no benefit past nproc, and you should set --parallel to that number rather than guessing higher. Either way, check what your tests actually share at module scope before you lean on --isolate for correctness: it resets state between files, not between tests in the same file, and that gap is exactly where flaky suites tend to live.

nginx 1.29.6 moves cookie-based session affinity out of nginx Plus

Alex Georgiev — Mon, 05 Oct 2026 10:00:00 +0000

For as long as I've configured nginx upstream blocks, cookie-based session affinity has been something you paid for. The free build had ip_hash and the hash directive, both of which pin a client to a backend using properties of the connection rather than anything the application controls. Proper sticky sessions, the kind that follow a browser cookie, lived behind an nginx Plus subscription.

nginx 1.29.6, released in March 2026, moves the sticky directive into the open-source build. I pulled it, broke my usual ip_hash setup on purpose, and measured what changes when session affinity stops depending on where a request comes from and starts depending on a cookie.

The setup

Three backend containers, each a plain Python HTTP server that returns its own container hostname so I could tell which one answered. One nginx container in front, reconfigured between runs: default round robin, sticky cookie, and ip_hash, all pointed at the same three backends over a Docker bridge network. Every backend was reachable and healthy throughout; nothing here depends on failure.

The headline number

One client, 30 requests, three configurations:

upstream config	backend 1	backend 2	backend 3
default (round robin)	10	10	10
`sticky cookie srv_id`	30	0	0

With the cookie set, every one of 30 requests from the same client landed on the backend nginx picked on request one. Without it, the same client's requests rotated evenly across all three. That's the entire pitch of the feature, and it held up exactly as advertised on the first test.

What ip_hash actually does to five different users

The reason I'd been using ip_hash for years is that it needed no application cooperation: no cookie, no header, just the client's address. The problem only shows up when several distinct users share one address, which is normal behind NAT, a corporate proxy, or a mobile carrier. I simulated that by running five separate clients, each with its own cookie jar, all issuing requests from the same source container, so nginx saw one IP address for all of them.

upstream config	client 1	client 2	client 3	client 4	client 5
`ip_hash`	backend 1	backend 1	backend 1	backend 1	backend 1
`sticky cookie`	backend 2	backend 3	backend 1	backend 2	backend 3

Under ip_hash, all five clients, despite being genuinely different sessions, landed on the identical backend, every single request, 30 hits out of 30. Under sticky cookie, the same five clients, from the same shared address, spread themselves across all three backends and each one stayed put once assigned. This is the actual argument for the feature. It isn't really about stickiness at all; ip_hash was already sticky. It's about stickiness that doesn't quietly merge unrelated users the moment they share a network path.

I re-ran this twice more to be sure it wasn't an artefact of request order, and both numbers reproduced exactly: five-for-five collapse under ip_hash, correct separation under sticky cookie.

It combines with least_conn, which surprised me

I assumed sticky would be mutually exclusive with another balancing method, the way ip_hash and least_conn can't be used together (nginx rejects that combination outright). I was wrong. sticky cookie layered on top of least_conn passed nginx -t cleanly and behaved correctly at runtime: new sessions were assigned using least_conn's logic, and once assigned, every client stayed on its backend for the rest of the test, five clients, ten requests each, zero migration. The two aren't the same kind of directive. ip_hash and least_conn are both load-balancing methods, and nginx only allows one. sticky is a binding layer that sits on top of whichever method you choose for new sessions. Worth knowing if, like me, you assumed otherwise from the ip_hash precedent.

What's still gated behind a subscription

The open-sourcing isn't complete. sticky learn, which watches upstream responses for an application-issued session cookie rather than inventing its own, now works in the free build:

$ nginx -t
nginx: configuration file /etc/nginx/nginx.conf test is successful

But its sync parameter, which replicates the learned-session table across a cluster of nginx instances, is still commercial-only, and says so plainly:

$ nginx -t
nginx: [emerg] unknown parameter "sync" in /etc/nginx/nginx.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed

So the boundary is specific: cookie, route, plain learn, and the new drain/route server parameters are free as of 1.29.6. Cross-instance session replication is not. If your actual requirement is a cluster of load balancers that agree on where a session lives, this release doesn't get you there by itself.

And none of it exists before 1.29.6 at all. Running the identical config against 1.29.5:

$ nginx -t
nginx: [emerg] unknown directive "sticky" in /etc/nginx/nginx.conf:7

Not a deprecated-but-working directive, not a warning. The whole block fails to parse.

Draining a backend without losing bound sessions

The drain server parameter is the other half of this release, also previously commercial. The documented behaviour is that a draining server keeps serving clients already bound to it via sticky, while refusing any brand-new session. I tested both halves. Across 40 fresh, cookie-less requests against two separately configured draining backends, zero landed on the drained server. Against a session cookie established before drain was switched on, eight replayed requests landed on the drained server eight times out of eight, whether the binding was a plain cookie hash or an explicit route value. The documentation's claim held exactly.

What it costs on every response

The sticky cookie isn't set once and forgotten. nginx resends Set-Cookie on every single response, not just the one that creates the session. Measuring raw response headers with and without it on an otherwise identical request:

	headers, bytes
plain upstream	156
`sticky cookie`	268

112 extra bytes, on every request, forever, for as long as the session exists. That's nothing on a typical API response, but it's not zero, and nobody mentions it because the feature announcement is naturally about the backend behaviour, not the wire cost of the mechanism that provides it.

What I got wrong on the way

My first attempt at testing drain produced a result that looked like a documentation bug: a session already bound to the draining backend got silently redirected elsewhere, which is the opposite of what nginx's docs promise. I was about to write that up as the most interesting finding in this post.

The actual cause was my test harness, not nginx. I'd established the session against one nginx container and replayed its cookie against a second, separately started container, using Python's http.cookiejar, which enforces domain matching the way a real browser would: a cookie set while talking to host A doesn't get sent when the code then talks to host B, even if both containers run identical config. The cookie I thought I was replaying never left the client. Switching to an explicit Cookie: header, bypassing the jar's domain logic entirely, showed the real behaviour, which matched the documentation.

A second, unrelated harness bug showed up when I tried to reproduce the same test with curl's -b/-c cookie-jar file instead of Python: curl refused to load a saved cookie back for a bare, dotless hostname like nx-sticky, with the error cookie 'srv_id' dropped, domain '[file]' must not set cookies for 'nx-sticky', even though it had just set that exact cookie from a live response moments earlier. That's a curl quirk specific to single-label Docker Compose-style hostnames, not an nginx issue, but it's exactly the kind of thing that will quietly break a local reproduction if you reach for curl's file-based cookie jar rather than an explicit -b "name=value".

Both mistakes map to the same lesson: a tool designed to behave like a cautious real browser will sometimes be more cautious than your test setup expects, and the resulting "failure" is the harness, not the thing you're testing.

Run it yourself

This reproduces the headline comparison. It needs Docker and nothing else.

docker network create stickynet

cat > backend.py <<'PY'
import http.server, socket
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200); self.end_headers()
        self.wfile.write(f"{socket.gethostname()}\n".encode())
    def log_message(self, *a): pass
http.server.ThreadingHTTPServer(('0.0.0.0', 8080), H).serve_forever()
PY

for n in b1 b2 b3; do
  docker run -d --name $n --network stickynet \
    -v "$PWD/backend.py:/backend.py:ro" python:3.12-slim python3 /backend.py
done

cat > sticky.conf <<'EOF'
events {}
http {
    upstream backend {
        server b1:8080;
        server b2:8080;
        server b3:8080;
        sticky cookie srv_id expires=1h path=/;
    }
    server { listen 80; location / { proxy_pass http://backend; } }
}
EOF

docker run -d --name nx-sticky --network stickynet \
  -v "$PWD/sticky.conf:/etc/nginx/nginx.conf:ro" nginx:1.29.8

# one client, repeated requests, explicit cookie header (not a jar file --
# curl's jar file rejects bare Docker hostnames, see below)
docker run --rm --network stickynet curlimages/curl:latest sh -c '
cookie=""
for i in $(seq 1 10); do
  curl -s -D /tmp/h.txt -o /tmp/b.txt -b "$cookie" http://nx-sticky/
  cookie=$(grep -oE "srv_id=[a-f0-9]+" /tmp/h.txt | head -1)
  cat /tmp/b.txt
done'

I ran that exact block before putting it in this post; it produced the same backend hostname ten times in a row.

What I measured here applies directly to any upstream block that uses ip_hash for a reason that was never really about load distribution. The failure mode is specific: a shared VPN exit, a corporate NAT gateway, or a mobile carrier's address pool puts several real sessions behind one IP, and ip_hash cannot tell them apart, because it was never given anything to tell them apart with. The fix is a single directive, sticky cookie srv_id expires=1h;, swapped in where ip_hash; used to sit, and as of 1.29.6 it compiles into the open-source binary with no licence file involved. Whether it belongs in a given config is a question about that config's own traffic, not one this post can answer from three containers on one machine.

Valkey 9.2's forkless BGSAVE cuts my memory spike from 350MB to 10MB

Alex Georgiev — Sun, 04 Oct 2026 10:00:00 +0000

Every BGSAVE I have ever watched on a busy Redis or Valkey instance does the same thing: the process forks, and for a few seconds the host's free memory drops like someone pulled a plug. Valkey 9.2.0-rc1, tagged on Docker Hub on 16 September, adds a way to skip the fork entirely. I wanted to know what that actually buys you, not what the release notes say it buys you, so I ran both paths against the same dataset and measured what happened.

The short version: the memory spike nearly disappeared, and the save got noticeably slower doing it.

The setup

I pulled valkey/valkey:9.2.0-rc1 and ran two containers from the identical image, differing only in one setting. The first used the default: bgsave-default-method fork. The second started with --forkless-infrastructure-enabled yes --bgsave-default-method forkless, which is the only way to turn it on — more on that below.

Into each I loaded 3,000,000 keys at 300 bytes each, a little over 1.1GB of data (used_memory reported 1.04GiB). Then, against each container, I ran eight threads hammering SET on random existing keys as fast as they could go, and from a ninth connection I sent a PING every 2ms and timed the round trip. Two seconds into each 12-second run I fired BGSAVE. I repeated this seven times per mode.

This setup matters for one reason: copy-on-write only costs you anything if pages are actually being written while the fork holds them. A BGSAVE against an idle dataset tells you almost nothing. Mine wasn't idle — the eight writer threads pushed roughly 27,000 operations a second into whichever container was running fork mode.

The memory spike, measured twice

I tracked two numbers: the RDB-reported copy-on-write size (rdb_last_cow_size in INFO persistence), and the container's own cgroup memory usage sampled every 150ms through the save.

	fork (default)	forkless
`rdb_last_cow_size`, mean of 7 runs	354.7MB	0MB (always)
cgroup memory delta during save, mean of 4 runs	354.4MB	9.9MB
save wall time (`rdb_last_bgsave_time_sec`)	3s, every run	5-6s

The two memory measurements agree with each other, which is the point of taking both: the kernel's own copy-on-write accounting and an independent cgroup memory sample converge on the same number through two unrelated instruments. On a 1.1GB working set under sustained writes, forking cost roughly a third of the dataset's size in extra memory, every single time, with a tight range of 319.6MB to 367.5MB across the seven runs. Forkless never went above 10.0MB.

That is the headline, and it held up every time I ran it. But it's not the whole story.

What the smoother memory curve cost

The forkless save took 5 to 6 seconds against fork's steady 3. That's not a rounding difference — it's roughly 70% longer for the same data, every time I measured it, over seven runs each.

It also cost write throughput. Across the 12-second window surrounding each save, the eight writer threads delivered a mean of 27,202 operations a second under fork mode and 23,034 under forkless — about 15% less. The background thread doing the serialising in forkless mode isn't free; it competes with the main thread for the same lock and the same CPU core budget, for longer, and the client-facing throughput shows it.

So the trade is real on both sides: fork spikes memory hard and briefly; forkless keeps memory nearly flat but takes longer and leans on your write throughput the whole time it's doing it. Neither is free. If your host is memory-constrained, forkless is the one you want. If your host has memory to spare and your write path cares more about throughput than about a short memory bump, the fork default is doing less damage than its memory graph suggests.

The pause the release notes mention

The one documented caveat I could find for this feature was that if the main thread writes to a key the background serialising thread hasn't reached yet, that client's request briefly stalls while the key gets moved to the front of the queue. I went looking for that stall in my latency samples.

Across seven forkless runs, the worst single round-trip during the 12-second window ranged from 5.1ms to 14.2ms, with a mean of 7.2ms. Fork's worst case ranged from 11.5ms to 25.7ms, mean 20.1ms — consistently about three times rougher. But forkless wasn't perfectly smooth either: one run spiked to 14.2ms, almost twice its own average worst case, which is consistent with hitting that documented stall on an unlucky key. The feature narrows the tail. It doesn't flatten it.

What it refuses, and what else still forks

forkless-infrastructure-enabled cannot be turned on with CONFIG SET against a running server:

$ valkey-cli config set forkless-infrastructure-enabled yes
(error) ERR CONFIG SET failed (possibly related to argument
'forkless-infrastructure-enabled') - can't set immutable config

It has to go on the command line or in the config file at startup, which means adopting this on an existing fleet needs a restart, not a hot config push. Trying to select the method without that flag fails too, with a message that at least tells you why:

$ valkey-cli config set bgsave-default-method forkless
(error) ERR CONFIG SET failed (possibly related to argument
'bgsave-default-method') - 'forkless' can only be selected when the
server was started with 'forkless-infrastructure-enabled yes'

A second BGSAVE while one is already running is refused outright, in both modes, with ERR Background save already in progress — unsurprising, but worth confirming it isn't silently queued.

The more useful limitation is this: enabling forkless-infrastructure-enabled only changes how RDB snapshots are taken. I turned on appendonly on the forkless-enabled instance, which triggers Valkey's automatic initial AOF rewrite, and checked aof_last_cow_size afterwards. It came back at roughly 10MB, not zero. AOF rewrites still fork, forkless RDB setting or not. If your actual pain is AOF rewrite stalls rather than BGSAVE stalls, this feature does not touch that path at all.

Watching it happen

The INFO fields the release notes promised for tracking progress are genuinely there and genuinely populated mid-save, which I didn't expect to hold up as cleanly as it did:

$ valkey-cli info persistence | grep -E 'save_keys|remaining'
current_save_keys_processed:555092
current_save_keys_total:3000000
forkless_estimated_seconds_remaining:3

Polled a second and a half after triggering a save on the same 3,000,000-key dataset, that's a believable in-flight number with a plausible estimate attached, not a placeholder. That's one documented claim that checked out exactly as described, which is worth saying plainly since the other numbers in this post are all pushing back on some part of the pitch.

What I got wrong on the way

My first version of the latency probe recorded each sample's offset by subtracting time.time() (wall clock) from a time.perf_counter() reading (a separate monotonic clock with an arbitrary zero point). The two aren't comparable, so every timestamp in my first batch of CSVs came out as a nonsensical negative number in the billions. The aggregate percentiles I'd already printed to stdout were unaffected, because those only used the latency values, not the timestamps — but the per-sample files were useless for lining a latency spike up against the exact moment BGSAVE started. I fixed it by recording one wall-clock epoch at the start of the run and adding monotonic deltas to it, then re-ran the affected trials. The lesson, again: check that your instrumentation's own clock is the one you think it is before trusting a timing result, even one that looks plausible.

Run it yourself

This is the full loop for one side of the comparison; swap the container flags to switch modes.

docker run -d --name vk-forkless -p 16379:6379 valkey/valkey:9.2.0-rc1 \
  --forkless-infrastructure-enabled yes --bgsave-default-method forkless --save ""

docker run -d --name vk-fork -p 16380:6379 valkey/valkey:9.2.0-rc1 --save ""

python3 - <<'EOF'
import socket, sys
def load(host, port, n=3_000_000, vsize=300):
    s = socket.create_connection((host, port))
    val = b'x' * vsize
    buf, batch = bytearray(), 2000
    for i in range(n):
        key = f'key:{i}'.encode()
        buf += b'*3\r\n$3\r\nSET\r\n$%d\r\n%s\r\n$%d\r\n%s\r\n' % (len(key), key, len(val), val)
        if (i + 1) % batch == 0:
            s.sendall(buf); buf = bytearray()
            got = 0
            while got < batch:
                got += s.recv(65536).count(b'+OK')
load('127.0.0.1', 16379); load('127.0.0.1', 16380)
EOF

docker exec vk-fork valkey-cli bgsave
docker exec vk-forkless valkey-cli bgsave
docker exec vk-fork valkey-cli info persistence | grep -E 'rdb_last_cow_size|rdb_last_bgsave_time_sec'
docker exec vk-forkless valkey-cli info persistence | grep -E 'rdb_last_cow_size|rdb_last_bgsave_time_sec'

Running just this much against an idle dataset gave me an rdb_last_cow_size of about 12MB, nowhere near the 350MB in the table above. That gap is the whole point: copy-on-write only costs you something when pages are being written while the save runs, so to see the real effect you need the eight-thread write load from the full test, not this stripped-down version.

If you're running Valkey on a host where memory headroom during BGSAVE has actually bitten you — not hypothetically, but a real OOM or a real eviction storm triggered by a save — this is worth testing against your own dataset shape before 9.2 ships as stable. If your problem has instead been a save that takes too long and steals too much write throughput while it runs, measure that before switching the default, because this release doesn't make BGSAVE cheaper. It moves where the cost shows up.

Docker Engine 29.7's overlay networking breaks every Swarm task without IPv6

Alex Georgiev — Sat, 03 Oct 2026 10:00:00 +0000

I asked a Swarm service for twenty replicas on an overlay network and got zero. Not slow, not partially up. Zero, forever, with the exact same command that gives me twenty on the engine version one release behind it.

I'd gone looking at the Docker Engine 29 release notes for something smaller: a line in the 29.8.0 changelog about spreading out the daemon's periodic overlay-network gossip so it doesn't burn CPU in bursts. That seemed like a nice, modest thing to measure. While setting up a test rig for it, before I'd measured anything about gossip at all, every overlay-attached service I created came up with 0 running tasks instead of the number I'd asked for.

The setup

I had Docker Engine 29.6.2 already running as this machine's system install. I downloaded the static builds for 29.7.0, 29.7.2 and 29.8.2 from download.docker.com and ran each as its own dockerd, pointed at its own data directory and its own Unix socket, so I could run two or more versions side by side on one host and compare them directly rather than trusting memory of "what it used to do."

Each one got docker swarm init on a single node, an overlay network, and a service with a handful of replicas running sleep 600, nothing fancier.

On 29.6.2:

$ docker service create --detach --name gossip-svc --replicas 20 --network gossip-net alpine:3.20 sleep 600
$ docker service ls
ID             NAME             MODE         REPLICAS   IMAGE         PORTS
e8v48kmvwvya   gossip-svc-old   replicated   20/20      alpine:3.20

Twenty requested, twenty running. I exec'd into two of the containers and pinged across the overlay network to be sure it wasn't just lying about the count:

$ docker exec 17a672fb04a7 sh -c "ping -c3 -W2 10.0.1.16"
PING 10.0.1.16 (10.0.1.16): 56 data bytes
64 bytes from 10.0.1.16: seq=0 ttl=64 time=1.051 ms
64 bytes from 10.0.1.16: seq=1 ttl=64 time=0.066 ms
64 bytes from 10.0.1.16: seq=2 ttl=64 time=0.075 ms

--- 10.0.1.16 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss

Real traffic, real VXLAN tunnel, both containers reachable. Good baseline.

On 29.8.2, the identical sequence of commands, same host, same image:

$ docker service create --detach --name gossip-svc3 --replicas 5 --network gossip-net3 alpine:3.20 sleep 600
$ docker service ls
ID            NAME          MODE         REPLICAS   IMAGE         PORTS
iaki8w2lsuwr  gossip-svc3   replicated   0/5        alpine:3.20

docker service ps named the reason:

network sandbox join failed: subnet sandbox join failed for "10.0.1.0/24":
overlay: cannot determine address family of transport: the local data-plane
address is not currently known

Every one of the five tasks hit this. Not one came up.

It isn't a fluke or a port clash

My first instinct was that I'd done something wrong with the test harness itself — I'd given the two engines different VXLAN data-path ports to avoid a bind conflict, so I reran 29.8.2 on the default port 4789 with the older engine's overlay network torn down first, to rule that out. Same error, same 0/5.

Then I widened the test backwards through the point releases to find where it actually started, since I only had 29.6.2 and 29.8.2 at first:

Engine	Replicas requested	Replicas running	Result
29.6.2	20	20	overlay traffic confirmed with `ping`
29.7.0	5	0	identical "address family" error
29.7.2	5	0	identical "address family" error
29.8.0–29.8.2	5 and 20	0	identical "address family" error

It's not version-specific to 29.8. It was already broken in 29.7.0, the release immediately after the one that works, and it's still broken in 29.8.2, the newest static build on download.docker.com as I write this. I didn't bisect down to a single commit — I don't have the moby source checked out here — but the official 29.7.0 and 29.8.0 release notes both list several Swarm networking changes in that stretch, including one about published ports on the service mesh sharing infrastructure with locally published ports, which touches exactly the code path that's failing here.

What it isn't

Plain containers on this same 29.8.2 install work fine. A docker run with a published port, no Swarm involved:

$ docker run -d --rm -p 18080:80 --name plainweb nginx:alpine
$ docker exec plainweb wget -qO- http://127.0.0.1:80 | head -3
<!DOCTYPE html>
<html>
<head>

That rules out something broad like the bridge driver or the whole networking stack being down. This is specific to the Swarm overlay driver's VXLAN data plane.

The one thing I could find that correlates

This host has no IPv6 stack at all:

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
cat: /proc/sys/net/ipv6/conf/all/disable_ipv6: No such file or directory
$ cat /proc/net/if_inet6
cat: /proc/net/if_inet6: No such file or directory

Not disabled — absent. No IPv6 sysctls, no IPv6 socket table, nothing. Both 29.6.2 and every later engine log the same ipv6-related read failures at startup (failed to read ipv6 net.ipv6.conf.<bridge>.accept_ra), so the daemon itself already knows this host has none. The difference is what each version does with that fact once a container actually tries to join an overlay sandbox: 29.6.2 carries on and completes the join over IPv4; 29.7.0 onward asks something that comes back unanswered and calls it fatal.

I want to be careful here: I didn't read the overlay driver's source to confirm this is the actual cause rather than a correlated symptom. What I can say is that the error text names exactly this — "cannot determine address family of transport" — on the one property of this host that changed nothing between engine versions and that I can independently confirm is true.

I checked whether Docker's own documentation sets an expectation either way. Its swarm networking page lists the ports that need to be open between hosts and says nothing about IPv6 either requiring it or ruling it out:

Port 2377 TCP for communication with and between manager nodes
Port 7946 TCP/UDP for overlay network node discovery
Port 4789 UDP (configurable) for overlay network traffic

Three protocols, all IPv4-shaped, no mention of IPv6 as a prerequisite anywhere on that page. By that documentation, what I ran should have worked on 29.8.2 exactly as it did on 29.6.2.

The workaround that didn't work

docker network create takes an explicit --ipv6 flag, so I tried forcing it off on the overlay network itself, on the theory that the daemon might be tripping over IPv6 address assignment specifically and that turning it off per-network would route around that:

$ docker network create -d overlay --ipv6=false gossip-net3
$ docker service create --detach --replicas 5 --network gossip-net3 alpine:3.20 sleep 600
$ docker service ls
ID            NAME           MODE         REPLICAS   IMAGE
zwuxi4opkw5  gossip-svc3     replicated   0/5        alpine:3.20

Identical failure, identical error text. Whatever is going wrong, it isn't happening at the per-network IPv6-enable flag, so there's no documented flag I found that gets you out of this on an IPv6-less host.

What it actually costs, which surprised me

I expected a service stuck retrying forever to be quietly expensive — some background loop hammering the scheduler. I sampled dockerd's own CPU time once a second for thirty seconds under two conditions: 29.6.2 running twenty real, working replicas, and 29.8.2 sitting on its broken five-replica service.

Condition	Mean CPU	Max CPU (1s sample)
29.6.2, 20/20 replicas running	0.52%	0.98%
29.8.2, 0/5 replicas, stuck	0.30%	1.97%

The broken service was cheaper on average, not more expensive. Watching docker service ps over time explained why: each task gets retried three times, each attempt a few seconds apart, and then Swarm stops trying and leaves it sitting in Assigned forever. No more log lines, no more CPU, no more anything. It fails hard once and then goes completely quiet. That's worse for anyone watching dashboards rather than logs, because nothing about resource usage tells you it's broken.

How you'd actually notice

One command shows it plainly, which is the only genuinely good news in this post:

$ docker service ls
ID            NAME          MODE         REPLICAS   IMAGE
iaki8w2lsuwr  gossip-svc3   replicated   0/5        alpine:3.20

0/5, sitting there indefinitely, next to a service that's genuinely fine. If your deploy tooling checks docker service ls or the equivalent API field for convergence before calling a rollout successful, you'll catch this immediately. If it only checks that docker service create returned exit code 0 — which it does, every time, failure included — you won't.

A smaller thing I checked and it didn't hold up

While I was in there I also tried to reproduce a specific fix listed in the 29.8.0 notes, about service creation failing when an automatically generated name collides with an existing one. I fired thirty concurrent docker service create calls with no --name at both 29.6.2 and 29.8.2, hoping to force a collision in the random name generator. Zero collisions on either version, in thirty concurrent attempts each. The namespace is evidently large enough that this doesn't show up at a scale I can produce by hand in an hour. I'm noting it rather than dropping it, because a finding of "I couldn't reproduce this" is still useful if you were about to spend time worrying about it.

What I got wrong on the way

My first run of the concurrent service-creation test hung indefinitely. I'd used docker service create without --detach, and with --restart-condition none the task is supposed to exit after sleep 1 rather than stay running — which meant the CLI's default wait for "the service has converged to its desired running count" could never be satisfied, since the desired count for a task designed to exit is never going to match "currently running." It wasn't a Docker bug, it was me asking the CLI to wait for a state I'd specifically engineered never to arrive. Adding --detach fixed it in about ten seconds once I noticed what the command was actually doing.

Run it yourself

This needs root and a Linux host with Docker's static builds reachable from download.docker.com. It stands up two engines side by side on their own sockets so you can compare without touching your real Docker install.

# grab a second engine build to compare against your system one
curl -sA 'Mozilla/5.0' -o docker-29.8.2.tgz \
  https://download.docker.com/linux/static/stable/x86_64/docker-29.8.2.tgz
mkdir new && tar xzf docker-29.8.2.tgz -C new
mkdir -p /var/lib/docker-new /run/docker-new

nohup new/docker/dockerd \
  --data-root /var/lib/docker-new \
  --exec-root /run/docker-new/exec \
  --host unix:///run/docker-new/docker.sock \
  --exec-opt native.cgroupdriver=cgroupfs \
  > new.log 2>&1 &
sleep 5

ADDR=$(hostname -I | awk '{print $1}')
export DOCKER_HOST=unix:///run/docker-new/docker.sock
new/docker/docker swarm init --advertise-addr "$ADDR"
new/docker/docker network create -d overlay testnet
new/docker/docker service create --detach --name testsvc --replicas 5 \
  --network testnet alpine:3.20 sleep 600

sleep 15
new/docker/docker service ls
new/docker/docker service ps testsvc --no-trunc | head -5

# check whether this host even has IPv6 at all
cat /proc/net/if_inet6 2>&1 || echo "no IPv6 stack on this host"

If service ls shows 0/5 and your host has no /proc/net/if_inet6, you've reproduced this. If it shows 5/5, either this has been fixed since 29.8.2 or your host has IPv6, and either way that's worth knowing before you plan around it.

If you run Swarm in production on a host or VM image that has IPv6 turned off — which is a common, deliberate choice on plenty of minimal server images and sandboxed CI runners — test an upgrade past 29.6.x on a throwaway node before you roll it out, and check docker service ls for the replica count rather than trusting a clean exit code from service create or service update.

The BMW manual was off-limits, so I built my friend something better

Alex Georgiev — Fri, 02 Oct 2026 20:07:29 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend repairs his own BMW E90 325i. Not professionally, not as a hobby exactly, just the way someone does when they'd rather understand their car than hand it to someone who won't explain it back.

Watching him work, the actual bottleneck is never the wrench. It's that the information is scattered. A torque spec lives in a forum post from 2013. A part number lives in a different tab. Half the "guides" are a video where someone talks for four minutes before touching the car. He's lying under a car on jack stands with dirty hands trying to pinch-zoom a phone screen.

So I built him BMW Repair Workshop: a fast, mobile-first reference for his exact car. Twenty repair procedures, seven reference guides, every torque value and part number laid out in a table instead of buried in prose. And an Ask box where he can describe a problem in his own words and get an answer built only from those procedures.

The interesting part isn't the site. It's what I wasn't allowed to build.

The constraint that shaped everything

My first plan was obvious: feed the service manual into a RAG pipeline and let an agent answer questions from it.

That plan died about an hour in. The real BMW service documentation (Bentley's manual, BMW's own TIS system) is commercial, copyrighted, and sold for money. It's also mirrored absolutely everywhere, a chapter-split copy is one search away, and there are entire sites dedicated to serving it for free.

Wide availability isn't permission. So I didn't use any of it. Not the PDFs, not the mirrors, not "just extracting the facts" from them, which is the same thing wearing a different hat.

That left me with a harder problem and, it turns out, a better project.

Where the content actually came from:

Procedures and specs written from general public automotive knowledge, with a handful of figures cross-checked against publicly published sources (owner forums, retailer install guides)
Every single entry states its own provenance in a Source field, visible on the page
Every torque figure and part number carries an explicit "verify this against RealOEM or your dealer" caveat, because I am not a mechanic and these came from general knowledge, not from his car's documentation

That last point matters more than it sounds. A repair reference that quietly presents an unverified number as fact is worse than no reference, because someone torques a bolt to it.

Demo

Live: https://georgievalex.github.io/bmw-repair-assistant/

Try the Ask box with something vague, the way you'd actually say it:

"my coolant keeps disappearing but there is no puddle"
"clunk from the front over bumps, what should I check first"
"how do I rebuild the automatic transmission?" ← watch it refuse

Code

https://github.com/GeorgievAlex/bmw-repair-assistant

How I Built It

Open-source AI at the core: Gemma (gemma-4-31B-it), Google's open-weight model, served through DigitalOcean Serverless Inference.

DigitalOcean is doing the actual model serving here. I send an OpenAI-compatible chat completion to inference.do-ai.run/v1/chat/completions, authenticated with a DigitalOcean model access key, and Gemma runs on their infrastructure. No GPU to provision, no model to host, no container to keep warm. The spend ceiling is their prepaid Inference & Agents balance with auto-reload off, which means the absolute worst case for a runaway is a capped loss and the Ask box going quiet, rather than a surprise bill.

That combination, an open-weight model served by a provider I'm not locked into, is most of the "open innovation" argument below in one line.

Architecture, deliberately boring:

browser → GitHub Pages (static site, no backend)
            ↓ POST { question }
          Cloudflare Worker  ← holds the model access key
            ↓
          DigitalOcean Serverless Inference → Gemma

The site is 100% static. Content lives as Markdown in the repo, a dependency-free Node script compiles it into a search index plus one static HTML page per procedure. Search runs client-side. There is no database, no framework, no build toolchain.

The design decision I'm happiest with: there is no RAG pipeline.

I started to build one, then measured the corpus. The entire knowledge base is about 11,000 tokens. That fits in a single prompt with room to spare.

So every question sends all twenty-seven entries to the model. No embeddings, no vector store, no chunking, no retrieval step that can silently fetch the wrong chunk and answer confidently from it. The model sees the complete corpus every time.

This is faster to build, free of an entire category of bug, and more accurate than chunked retrieval at this scale. Costs about $0.002 per question. The reflex to reach for a vector database is strong, and measuring first saved me from it.

Keeping it honest. The system prompt does real work here:

Answer only from the provided entries
Never invent a torque value, part number, or step
Carry each entry's verification caveats through into the answer
Keep safety warnings (spring compressors, hot cooling systems, torquing bushings at ride height) rather than trimming them for brevity
Name the procedure the answer came from

Asked to rebuild an automatic transmission, it says it has no such procedure and points at the nearest relevant entry. That refusal is the single most important behaviour in the whole project.

Why Does Open Innovation Matter?

Four reasons, in increasing order of how much I actually believe them.

It's cheap. $0.002 a question. The whole thing runs on free static hosting with a free worker in front of an open-weight model. A closed frontier model would work too, and cost more for a job that doesn't need it.

Model choice is a swap, not a migration. Gemma is one line of config. If something better ships, or Gemma gets cheaper elsewhere, I change a string. Nothing else in the project knows or cares which model answers.

It's inspectable. When the model does something strange, I can read the entire input that produced it: 27 Markdown files in a public repo and a system prompt in a 160-line worker. No hidden retrieval step deciding what the model sees. For something that tells people how tight to torque a brake caliper, I want to be able to explain any answer it gives.

And the one that actually drove the project: the open approach was the only one I could build honestly.

The closed path here isn't a proprietary model. It's the proprietary manual, and the ecosystem of unauthorised mirrors around it. Those mirrors work right up until they don't. They vanish, they get taken down, they're of unknown provenance, and nothing in them tells you where a number came from.

What I built instead is small, honest about its limits, and entirely inspectable. Every entry says where it came from. Every number says to verify it. The model can only speak from content that's sitting in a public repo with its sources stated.

That's a worse product than BMW's actual service manual. It's a far better one than a pirated copy, because you can see exactly what you're trusting.

What I got wrong on the way

I guessed a model ID. gemma-4 seemed reasonable. The real one is gemma-4-31B-it, and a wrong ID fails as a bare 404 from the inference API, nothing that says "check your config." Twenty minutes gone.

I wrote an XSS bug into the answer renderer. My escaping helper used the textContent → innerHTML trick, which escapes <, >, & but not quote characters, because quotes aren't special in text nodes. I then dropped that "escaped" output straight into an alt="..." attribute. One stray " in a caption, even my own typo, would have broken out of the attribute. Fixed with proper attribute-context escaping.

I told my friend CORS would protect the API budget. It won't. CORS is enforced by browsers, so it stops another website calling the worker from page JavaScript, and does precisely nothing against curl. Rate limiting and a capped prepaid balance are the actual controls.

I shipped reference pages that rendered ## Heading as literal text for a day, because my first reference docs were plain prose and never exercised the heading path.

What I didn't do

No VIN decoding yet, so the site can't narrow specs to an exact build, which matters on a chassis spanning several engine variants. No photos of the actual parts, because the only correctly-licensed one I found was a single public-domain engine shot, and a generic stock photo of someone else's brake caliper on a BMW brake page is the kind of small dishonesty this project is supposed to avoid. No testing against a second chassis, though the data structure already carries chassis and engine code per entry so adding one is content work, not a rewrite.

And the big one: every technical figure on that site needs a mechanic's eye. The provenance lines say so plainly. Which brings me to the only part of this I couldn't do myself.

Handing it over

I sent it to him while he was at the garage, with four questions. The most important was the first: is anything on here actually wrong?

He went through the jobs he's done himself and nothing contradicted what he knows. I want to be precise about what that is and isn't: it's a working mechanic reading it and nothing jumping out, which is genuine signal. It is not a line-by-line audit against documentation. Those caveats on every number stay exactly where they are.

On the AI, he was more measured than I expected, in a way I liked:

"The answers it gives are valid, you still need to check them of course, like diagnosis and etc"

He arrived at the project's own position without being told it. That's about the best outcome available for a tool like this: it was useful, and it didn't make him credulous.

Then he corrected an assumption I'd built the whole input design around. I'd been thinking about voice input, on the theory that dirty hands and phone screens don't mix. He doesn't want it:

"I always prefer text as long answers can be quickly forgotten and I work with gloves so taking them off to ask is not a problem"

Two things I hadn't considered. Gloves come off anyway. And more interesting: a spoken answer evaporates, while text stays on screen while you're under the car with your hands busy. The persistence is the feature. I'd have built the wrong thing.

On the look, which I'd worried was too plain:

"I like the idea it looks simple and old style - no weird images, adds and stuff"

And the one real feature request, which he raised and then talked himself out of:

"Perhaps it misses like video tutorial itself but this is not possible I guess, like to search in youtube if someone performed this repair on my car/engine"

It is possible, just not the way he assumed. I can't host or embed video. But I can hand off a search already narrowed to his exact chassis and engine, so he gets BMW E90 N52B25 brake pads and rotors change rather than generic results for a different car. Every procedure now has a "See it done" link doing that. It shipped before I finished writing this post.

He also asked for exhaust and gearbox procedures. Exhaust is a straightforward driveway job and it's added. The gearbox I partly declined: the card covers fluid and pan service, and then says plainly that internal rebuild is a specialist bench job with no honest driveway version. Writing a rebuild procedure from general knowledge, for a job where a wrong clearance is a destroyed transmission, is exactly the failure mode this whole project was built to avoid. Saying "this isn't something I should write" is part of the same discipline as the model refusing the same question.

The line I didn't expect:

"You're my go to AI guy and also good mechanic yourself so I would love if you can work on this in the future so me, you and other petrol heads can use it"

I'm not a good mechanic. I just wrote down what I could verify and was careful about what I couldn't. But "so me, you and other petrol heads can use it" is a better description of why this is worth continuing than anything in my own notes.

What his feedback actually changed

All of this is live on the site now, shipped between his message and this post going up:

He said	What changed
Wanted to find video of the job on his engine, assumed impossible	Every procedure has a "See it done" link, a search pre-narrowed to `BMW E90 N52B25 <job>`
Asked for an exhaust procedure	Added, full job
Asked to "dissamble the gearbox"	Added fluid and pan service; internal rebuild explicitly declined as a specialist bench job
Prefers text over voice, with reasons	Voice input dropped from the roadmap entirely

That last row is the one I'd have got wrong on my own. The feature I was about to build is the feature he actively didn't want, and I'd never have found that out by thinking harder about it.

He had one more request, which I'm not going to be able to ship:

"it will be good if the site can handle those repairs and changes for myself but AI is not there yet :D"

Correct on both counts. And I think that joke is a decent summary of where this sits. The useful version of AI here isn't the one that does the job. It's the one that tells him the caliper guide bolts are 30 Nm, points at the procedure it got that from, reminds him to check it, and then gets out of the way while he does the work he already knows how to do.

Prize Categories

Best Use of DigitalOcean — DigitalOcean Serverless Inference serves the open-weight model behind the Ask feature, via inference.do-ai.run, with spend capped by the prepaid Inference & Agents balance.

Best Use of Gemma — gemma-4-31B-it is the open-weight model doing the reasoning, constrained to the site's own content and instructed to refuse rather than invent.

Credits

The project has no dependencies, the code is all hand-written, and the content was written for this project rather than taken from anywhere. Two things worth naming anyway:

The GitHub mark in the site navigation is GitHub's logo, used only to link to the repository.
The post and the project were built with AI assistance, declared in the front matter. The specs in the site came from general knowledge, and every one of them says on the page that you should verify it yourself.

Git 2.56's fetch.followRemoteHEAD setting fixes stale origin/HEAD for every remote

Alex Georgiev — Fri, 02 Oct 2026 10:00:00 +0000

I have an origin/HEAD pointing at origin/master in a repository I cloned two years ago, long after the project moved its default branch to main. It has never fixed itself. Every git fetch since then has pulled the right commits and left that one symbolic ref exactly where it was.

Git 2.56, released in the past few weeks, adds a config variable called fetch.followRemoteHEAD that is supposed to address this, by giving you one global switch instead of a separate setting per remote. I built three versions of Git from source — the Ubuntu-packaged 2.43.0, a 2.55.0 I compiled myself, and 2.56.0 — and used them against a handful of local bare repositories to see what the setting actually does, what it doesn't, and what happens when you reach for the wrong value.

Reproducing the stale ref

First I confirmed the bug I already own a copy of. I cloned a bare repo while its HEAD pointed at main, then renamed the remote's default branch to trunk, the way GitHub does when you change the default branch in the settings page:

$ git clone remote2 clone-create -q
$ git -C clone-create symbolic-ref refs/remotes/origin/HEAD
refs/remotes/origin/main
$ git --git-dir=remote2 branch -m main trunk
$ git -C clone-create fetch
From /tmp/gittest/remote2
 * [new branch]      trunk      -> origin/trunk
$ git -C clone-create symbolic-ref refs/remotes/origin/HEAD
refs/remotes/origin/main

The fetch worked. The new branch arrived. origin/HEAD still points at origin/main, a ref to a branch that no longer exists on the remote. That's fetch.followRemoteHEAD's default value, create: it will only ever create the symref if none exists. Once it exists, nothing updates it, forever, on every Git version that has the setting at all.

Setting the per-remote value to warn doesn't fix it either, it just tells you:

hint: Run 'git remote set-head origin trunk' to follow the change, or modify
hint: either of the 'remote.origin.followRemoteHEAD' or 'fetch.followRemoteHEAD'
hint: configuration variables to handle the situation differently.
hint:
hint: Using this specific setting
hint:
hint:     git config set remote.origin.followRemoteHEAD warn-if-not-trunk
hint:
hint: will suppress the warning until the remote changes HEAD to something else.
'HEAD' at 'origin' is 'trunk', but we have 'main' locally.

always, set per-remote, does what you'd expect: the next fetch silently repoints the symref to trunk. All three of these modes (create, warn, always, plus never) existed before 2.56. What's new in 2.56 is that you can set the default for all of them at once, with fetch.followRemoteHEAD, instead of writing remote.<name>.followRemoteHEAD into every remote's config by hand.

Does the global setting actually save you per-remote config

This is the part worth measuring rather than describing. I set fetch.followRemoteHEAD always once, in a config file pointed to by GIT_CONFIG_GLOBAL so I wasn't touching anything outside my test directory, then added three remotes to a fresh repository and fetched all of them:

$ git config --global fetch.followRemoteHEAD always
$ git remote add r1 ../remote1   # HEAD -> trunk
$ git remote add r2 ../remote2   # HEAD -> trunk
$ git remote add r3 ../remote3   # HEAD -> main
$ git fetch --all
...
$ git symbolic-ref refs/remotes/r1/HEAD
refs/remotes/r1/trunk
$ git symbolic-ref refs/remotes/r2/HEAD
refs/remotes/r2/trunk
$ git symbolic-ref refs/remotes/r3/HEAD
refs/remotes/r3/main
$ grep -c followRemoteHEAD .git/config
0

Zero lines of per-remote config across three remotes, all three correctly tracking. But that much is also true of the old default behaviour, create, for a brand new remote with no existing symref — I'll come back to that, because it's where I fooled myself for a while.

The real test is what happens later, when a remote I already have changes its default branch again, with no new config written anywhere:

$ git symbolic-ref refs/remotes/r1/HEAD
refs/remotes/r1/trunk
$ git --git-dir=../remote1 branch -m trunk develop
$ git fetch r1
From ../remote1
 * [new branch]      develop    -> r1/develop
$ git symbolic-ref refs/remotes/r1/HEAD
refs/remotes/r1/develop
$ git config --get-regexp 'remote\.r1\..*'
remote.r1.url ../remote1
remote.r1.fetch +refs/heads/*:refs/remotes/r1/*

That's the actual improvement: one global line, set once, keeps every remote's HEAD current indefinitely, without ever touching that remote's own config block. Before 2.56 the equivalent was one remote.<name>.followRemoteHEAD always line added for every single remote you cared about, by hand, after noticing the problem.

It did nothing on last month's Git

To be sure the global key is genuinely new and not just newly documented, I pointed the same GIT_CONFIG_GLOBAL file at Git 2.55.0, which I'd compiled from the official v2.55.0 tag:

$ git --version
git version 2.55.0
$ git fetch r1          # first fetch, symref doesn't exist yet, 'create' kicks in
 * [new branch]      develop    -> r1/develop
$ git symbolic-ref refs/remotes/r1/HEAD
refs/remotes/r1/develop
$ git --git-dir=../remote1 branch -m develop final
$ git fetch r1          # second fetch, remote renamed again
 * [new branch]      final      -> r1/final
$ git symbolic-ref refs/remotes/r1/HEAD
refs/remotes/r1/develop

Same global config file, same rename, same remote. On 2.55 the second rename leaves HEAD stale at develop, because 2.55 only understands the per-remote key. It doesn't error on the unrecognised global setting, it just never consults it. That first fetch succeeding was the default create behaviour doing its ordinary job on a never-before-seen remote, not the global setting working — which is exactly the mistake I made on my first pass through this test, below.

What it refuses

The per-remote setting accepts one value the global one doesn't: warn-if-not-$branch, which behaves like warn but keeps quiet as long as the remote's HEAD still matches the branch you name. I tried setting that at the global level, since the documentation for remote.<name>.followRemoteHEAD says it accepts "the values supported by fetch.followRemoteHEAD" plus this one, which reads as though it might go either way:

$ git config --global fetch.followRemoteHEAD warn-if-not-main
$ git fetch r3
warning: unrecognized fetch.followRemoteHEAD value 'warn-if-not-main' ignored
From ../remote3
 * [new branch]      newmain    -> r3/newmain
$ echo $?
0

No error, no non-zero exit, just a warning and a silent fall-back to create. If you set this in your global config expecting it to suppress warnings for a specific branch name on every remote, it will quietly not do that, and the only sign is a line easy to miss in fetch output you weren't reading closely.

I also checked precedence between the two settings, since the docs state the per-remote value overrides the global one but I wanted to see it fail in both directions:

$ git config --global fetch.followRemoteHEAD never
$ git --git-dir=../remote2 branch -m trunk r2final
$ git fetch r2                                    # no per-remote override: stays stale
$ git symbolic-ref refs/remotes/r2/HEAD
refs/remotes/r2/trunk
$ git config remote.r2.followRemoteHEAD always
$ git --git-dir=../remote2 branch -m r2final r2final2
$ git fetch r2                                    # per-remote override: updates
$ git symbolic-ref refs/remotes/r2/HEAD
refs/remotes/r2/r2final2

That matched the documentation exactly, both ways.

What it costs, and what it doesn't tell you

I timed five fetches each, against a local filesystem remote, with the setting on create and then on always:

mode	run 1	run 2	run 3	run 4	run 5
create	11ms	11ms	11ms	11ms	11ms
always	11ms	11ms	12ms	11ms	11ms

No measurable difference. That's a local filesystem transport, so it only tells you the extra ref write costs nothing on top of a fetch that's already happening; it says nothing about network latency, which this setup can't produce.

The more interesting cost is that always mode gives you no feedback when it does something. Compare the two fetches I ran against r3 with always configured:

$ git fetch r3
From ../remote3
 * [new branch]      finalbranch -> r3/finalbranch
$ echo $?
0

That output is identical whether or not origin/HEAD just got silently repointed. If you want to know it happened, you have to go and check with git symbolic-ref or git remote show <remote> afterwards; nothing in a normal fetch run says so. For a setting whose entire purpose is to change state you didn't ask this invocation to change, that's worth knowing before you turn it on for every clone a CI job makes.

What I got wrong on the way

My first attempt at testing the global default was to add three brand new remotes, set fetch.followRemoteHEAD always beforehand, fetch all three, and check that all three HEAD refs came out correct with no per-remote config. They did, and for a few minutes I treated that as proof the feature worked.

It wasn't proof of anything. create, the old default, already creates a correct HEAD symref for any remote that doesn't have one yet, with no config at all, on every Git version back to whenever the per-remote setting first existed. Three fresh remotes with no prior HEAD are exactly the case where create and always produce the same result. I only caught this rereading the fetch.followRemoteHEAD documentation a second time, which is what sent me back to re-run the test against a remote that had already been fetched once and then changed its default branch again — the case the old default genuinely can't handle and the new global setting can.

Run it yourself

This needs Git built from source, since Ubuntu's packaged Git (2.43.0 here) predates the per-remote setting entirely. Building takes a few minutes on four cores.

# build 2.56.0 and 2.55.0 for comparison
sudo apt-get install -y build-essential libssl-dev libcurl4-openssl-dev libexpat1-dev gettext zlib1g-dev
git clone --depth 1 --branch v2.56.0 https://github.com/git/git /tmp/git-256
cd /tmp/git-256 && make configure && ./configure --prefix=/opt/git-2.56 && make -j4 && make install

mkdir /tmp/gittest && cd /tmp/gittest
git init --bare -b main remote-a
git clone remote-a seed -q && cd seed && git commit --allow-empty -qm init && git push -q origin main
cd .. && rm -rf seed

GIT=/opt/git-2.56/bin/git
$GIT clone remote-a work -q
git --git-dir=remote-a branch -m main trunk   # simulate a default-branch rename

cd work
$GIT fetch                                     # pulls trunk, origin/HEAD stays on main
$GIT symbolic-ref refs/remotes/origin/HEAD     # -> refs/remotes/origin/main, stale

$GIT config remote.origin.followRemoteHEAD always
git --git-dir=../remote-a branch -m trunk final
$GIT fetch                                     # now it follows
$GIT symbolic-ref refs/remotes/origin/HEAD     # -> refs/remotes/origin/final

If you maintain more than a couple of remotes, or you're responsible for a CI image that clones repositories whose owners might rename master to main out from under you, set fetch.followRemoteHEAD to always once in your global config and stop thinking about it per-remote. Just don't expect warn-if-not-$branch to work there, and don't expect any output when it quietly fixes something for you.

SQLite 3.53's SET NOT NULL replaces a full table rebuild with six writes

Alex Georgiev — Thu, 01 Oct 2026 10:00:00 +0000

For as long as I've used SQLite, adding a NOT NULL constraint to an existing column meant the twelve-step dance: create a new table with the constraint you want, copy every row across, drop the old table, rename the new one, and rebuild whatever indexes you just lost in the process. SQLite 3.53.0, released in April 2026, adds ALTER TABLE ... ALTER COLUMN ... SET NOT NULL and DROP NOT NULL, so I built a few test tables and measured what actually changes.

I ran this against the real 3.53.4 CLI, not the one that ships in Ubuntu's apt repositories, which still tops out at 3.45.1 and doesn't understand the new syntax at all.

The rebuild, timed

I generated a 2 million row table with an index on the column I was about to constrain, dropped the OS page cache before each run (echo 3 > /proc/sys/vm/drop_caches) so I wasn't measuring the filesystem cache instead of SQLite, and timed both approaches three times each.

The old way:

CREATE TABLE t_new(id INTEGER PRIMARY KEY, email TEXT NOT NULL, created_at TEXT);
INSERT INTO t_new SELECT * FROM t;
DROP TABLE t;
ALTER TABLE t_new RENAME TO t;
CREATE INDEX idx_email ON t(email);

The new way:

ALTER TABLE t ALTER COLUMN email SET NOT NULL;

Approach	Run 1	Run 2	Run 3
Rebuild (old way)	1.788s	1.495s	1.594s
`SET NOT NULL` (indexed column)	0.0093s	0.0113s	0.0097s

That's roughly 150 to 180 times faster, and it isn't a rounding trick: the rebuild also has to recreate idx_email from scratch, which the old approach needs you to remember to do by hand. Forget that step and your queries silently fall back to a table scan. SET NOT NULL leaves the index exactly where it was.

Why it's that fast: it barely touches the disk

The speed difference made me suspicious, so I ran both under strace -c counting pwrite64 calls on a fresh copy of the same database.

=== OLD WAY ===
% time     seconds  usecs/call     calls    errors syscall
100.00    1.001682          14     69843           pwrite64

=== NEW WAY ===
% time     seconds  usecs/call     calls    errors syscall
  0.00    0.000000           0         6           pwrite64

Six writes totalling 8,716 bytes, against 69,843 writes for the rebuild. The rebuild touches every row twice — once to write it into the new table, once again when the index is rebuilt. SET NOT NULL only rewrites the schema record in sqlite_master; the row data pages never move.

The catch: it still has to read every row, unless there's an index

Here's the finding that argues against the headline number. SQLite's own release notes say ALTER TABLE execution time for a new NOT NULL constraint "is proportional to the amount of data in the table," because every existing row has to be checked. On my first table that wasn't visible at all — 0.01 seconds for 2 million rows looked too good to be true, so I ran it again on a column with no index.

=== indexed column, 8M rows: read syscalls ===
calls
   13   pread64

=== unindexed column, 8M rows: read syscalls ===
calls
25372   pread64

With an index on the target column, SQLite checks for nulls by walking the index itself — NULLs sort first in SQLite's b-trees, so it only has to look at the left edge, 13 page reads regardless of table size. Without an index, it has to scan every data page: 25,372 reads for an 8 million row table.

On a warm cache this difference nearly disappears in wall-clock terms (both finish in well under a second, because Linux is just serving pages out of RAM). Cold-cache numbers tell a clearer story:

Column has index?	Cold-cache time (8M rows, created_at / email)
Yes	~0.010s
No	~0.11s

Still much faster than a rebuild either way, but the documentation's "proportional to table size" claim only really bites when there's no index backing the column you're constraining.

What it refuses, and how unhelpfully

Feed it a table with an existing null and it refuses, correctly:

$ sqlite3 t.db "ALTER TABLE t ALTER COLUMN b SET NOT NULL;"
Error in 2nd command line argument: constraint failed

Compare that with the error you get for the exact same violation through a normal INSERT:

$ sqlite3 t.db "INSERT INTO t VALUES (1, NULL);"
Error near line 1: NOT NULL constraint failed: t.b

The INSERT error names the table and the column. The ALTER TABLE error says "constraint failed" and nothing else — not which column, not which constraint type, not how many rows fail or which row id. On a real table with several nullable columns you're trying to lock down one at a time, that message won't tell you which one is the problem. You have to go find the offending rows yourself, with something like SELECT rowid FROM t WHERE email IS NULL LIMIT 5.

A working feature nowhere in the docs

While testing CHECK constraints I tried the ANSI-SQL-style syntax out of habit, expecting a parse error, because SQLite's own lang_altertable.html page only documents SET NOT NULL / DROP NOT NULL and says CHECK constraints can only be added via ADD COLUMN.

ALTER TABLE t ADD CONSTRAINT age_positive CHECK (age >= 0);

It worked. It validated existing rows, it rejected a negative insert afterwards with a properly named error (CHECK constraint failed: age_positive), and ALTER TABLE t DROP CONSTRAINT age_positive; cleanly removed it again. I checked the raw HTML of the ALTER TABLE reference page for the literal strings "ADD CONSTRAINT" and "DROP CONSTRAINT" — zero matches, in either direction. This is a fully working, round-trippable feature that I couldn't find documented anywhere on sqlite.org.

Locking: reads pass through, writes don't

I used a FIFO to keep one sqlite3 CLI session open with an uncommitted BEGIN IMMEDIATE write transaction, then tried to run the ALTER from a second connection against the locked database.

--- ALTER with busy_timeout=0, while a write lock is held ---
Error in 2nd command line argument: database is locked
immediate-fail attempt wall=0.004s

--- ALTER with busy_timeout=3000, same lock held for 0.3s then released ---
wait-then-succeed attempt wall=0.982s

With no timeout it fails immediately. With a timeout it queues and runs as soon as the lock is free, same as any other write. Unsurprising, but worth confirming, since it means you do need busy_timeout set on whatever runs this migration, same as any other schema change.

The more interesting case is a concurrent reader rather than a concurrent writer. I started the slow (unindexed, 8M row) SET NOT NULL in the background, let it get 150ms into its scan, then fired a SELECT count(*) FROM t from a second connection with busy_timeout=0, in both the default rollback-journal mode and WAL mode:

DELETE mode: reader wall=0.097s, returned 8000000, ALTER total wall=0.658s
WAL mode:    reader wall=0.111s, returned 8000000, ALTER total wall=0.322s

The reader succeeded immediately in both modes, without waiting for the ALTER to finish. The NOT NULL scan only needs a shared lock for the read phase; it doesn't take an exclusive lock until the very end, for that tiny schema write. If your application is read-heavy, running this migration won't stall your readers even on a large table.

The no-op claim checks out

The docs say calling SET NOT NULL on a column that's already NOT NULL is a no-op. I ran it twice in a row:

ALTER TABLE t ALTER COLUMN email SET NOT NULL;  -- succeeds, adds the constraint
ALTER TABLE t ALTER COLUMN email SET NOT NULL;  -- succeeds again, exit code 0

The schema was identical before and after the second call, and the second call didn't trigger the pread64-heavy validation scan a fresh constraint would. DROP NOT NULL behaved correctly too: after dropping it, an INSERT with a null value in that column succeeded again.

What I got wrong on the way

My first attempt at the concurrency test used a background shell loop that checked for the existence of a flag file before writing anything. The loop and the touch of that flag file were two separate commands, and when I ran them back to back, the loop occasionally started, checked for the file, found it missing, and exited — before the touch had actually run. The result was a test that reported "0 writer attempts logged" and looked, misleadingly, like no concurrent activity had happened at all. I rebuilt it using a named pipe holding one persistent sqlite3 session with an explicit BEGIN IMMEDIATE, which gave me a lock I controlled directly instead of a lock I was hoping a loop would acquire in time.

Run it yourself

Get the real CLI, since distro packages lag badly behind:

curl -sS -L -o sqlite-tools.zip \
  https://www.sqlite.org/2026/sqlite-tools-linux-x64-3530400.zip
unzip -o -q sqlite-tools.zip
./sqlite3 --version   # should print 3.53.4 or later

Build a test table and compare the two approaches:

./sqlite3 bench.db <<'EOF'
CREATE TABLE t(id INTEGER PRIMARY KEY, email TEXT, created_at TEXT);
WITH RECURSIVE seq(x) AS (
  SELECT 1 UNION ALL SELECT x+1 FROM seq WHERE x < 2000000
)
INSERT INTO t(id, email, created_at)
  SELECT x, 'user'||x||'@example.com', datetime('now') FROM seq;
CREATE INDEX idx_email ON t(email);
EOF

cp bench.db newway.db
time ./sqlite3 newway.db "ALTER TABLE t ALTER COLUMN email SET NOT NULL;"

cp bench.db oldway.db
time ./sqlite3 oldway.db "
  CREATE TABLE t_new(id INTEGER PRIMARY KEY, email TEXT NOT NULL, created_at TEXT);
  INSERT INTO t_new SELECT * FROM t;
  DROP TABLE t;
  ALTER TABLE t_new RENAME TO t;
  CREATE INDEX idx_email ON t(email);
"

If you want the write-count comparison, run the same two commands under strace -f -e trace=pwrite64 -c instead of time.

What to do with this

If you're on SQLite 3.53 or later and need to tighten a schema that's already in production, use ALTER TABLE ... ALTER COLUMN ... SET NOT NULL without hesitation — it's faster, it doesn't disturb your indexes, and it won't block concurrent readers. But check your target column for nulls yourself first, with an explicit SELECT ... WHERE col IS NULL, before you run it on anything you can't immediately re-run: the error message won't tell you where the violation is. And if you need a CHECK constraint added or removed on a live table, ADD CONSTRAINT and DROP CONSTRAINT already work, even though the manual doesn't mention them yet — just don't be surprised if that's a documentation gap rather than a permanent feature, and keep an eye on future release notes in case the syntax changes before it's formally written up.

MongoDB Community Edition's new search index stayed pending above 89% disk use

Alex Georgiev — Wed, 30 Sep 2026 10:00:00 +0000

MongoDB Search and Vector Search for Community Edition went GA on 30 June 2026. Self-managed deployments get the same mongot search process that used to be Atlas-only, running alongside mongod instead of bolted onto a separate engine. I followed MongoDB's own Docker installation guide word for word, and the search index never left PENDING.

Setting it up

The docs describe two containers on one network: mongod (MongoDB Community Server 9.0.2) and mongot-community (mongot 1.70.5), joined by a gRPC connection and a SCRAM user with the searchCoordinator role. I ran it exactly as documented: a single-member replica set, a mongod.conf pointing at the search host, a mongot.conf pointing back at mongod, then createSearchIndex on a 50,000-document test collection.

The index came back immediately, as expected:

[ { name: 'default', status: 'PENDING', queryable: false } ]

PENDING is normal for the first few seconds. It was not normal five minutes later, still retrying every 30 seconds with the same message in the mongot log:

"msg":"Transient error while trying to sync ... Retrying in 30000 milliseconds"
"stack_trace":"... PauseInitialSyncException: Initial syncs are paused ..."

Finding the actual cause

mongot exposes Prometheus metrics on port 9946, and one of them settled it:

mongot_system_disk_space_data_path_free_bytes 2.8686217216E10
mongot_system_disk_space_data_path_total_bytes 2.70553174016E11

28.7GB free out of 270.6GB, which is 89.4% used. MongoDB's own self-managed troubleshooting page states the rule directly: "Replication stops when disk usage exceeds roughly 90% and resumes after usage drops below roughly 85%. For a new index or rebuild, expect the definition to be accepted but the build to stay stuck if disk pressure is already above the protective threshold." My host had drifted into that band and the index was never going to move.

Nothing about this showed up on the client side. createSearchIndex returned normally, $listSearchIndexes just said PENDING, and df -h on the same path reported 28% used, not 89%, because df was measuring against a quota-limited allowance while mongot was dividing by the underlying device's full reported capacity. Two tools looking at the same disk gave answers 60 points apart.

I moved mongot's data directory onto tmpfs to get a mount with headroom, and the same index went from PENDING to serving queries in under five seconds:

"msg":"Finished a collection scan phase.","attr":{"numDocumentsIndexed":50000}
"msg":"Completed initial sync. Beginning first commit.","attr":{"duration":"4.850 s", ...}
"msg":"Transitioning from INITIAL_SYNC to STEADY_STATE."

The block was real and the fix was disk headroom, nothing else.

A status field that keeps lying

After I recreated the mongot container, $listSearchIndexes kept reporting default as PENDING even though queries against it worked. The full statusDetail array explained why: it still listed the old, now-dead mongot host as PENDING alongside the new host, which was READY. The aggregated top-level status is the worst of all hosts it has ever seen, including ones that no longer exist. A vector index I created fresh, with no dead host in its history, reported READY cleanly. If you restart mongot in place, don't trust the summary status; read statusDetail per host.

What it refuses, and what it doesn't

Querying an index that exists but isn't ready yet is a hard, specific error:

OperationFailure: cannot query search index ... while in state NOT_STARTED (code 8)

Querying an index name that doesn't exist at all is not an error. It returns zero results silently, with only a WARN "No index in catalog" line in mongot's own log, invisible to the client. A typo in an index name and a genuinely empty result set look identical from the application side.

The speed comparison I expected to favour $search

Once the index was live, I ran 15 repetitions each of five single-word and two-word queries against $search, a classic MongoDB $text index, and a plain case-insensitive regex, all on the same 50,000-document collection.

Method	min	median	p95	max
`$search` (mongot)	6.25ms	9.36ms	15.43ms	56.05ms
classic `$text` index	0.48ms	0.67ms	0.89ms	1.57ms
regex scan	0.48ms	0.79ms	2.86ms	3.95ms

For a plain term lookup, the new search stack was 10 to 14 times slower than the mechanism it's positioned to replace. That's the gRPC round trip to a separate process rather than an in-process index scan, and it's not a reason to avoid mongot — it's a reason not to switch to it for cases the old $text index already handled well.

Where $search earns the extra hop is fuzzy matching. I queried all four methods with backpak, a one-letter typo for backpack:

Method	results for "backpak"
`$search` with `fuzzy: {maxEdits: 1}`	5
`$search`, no fuzzy option	0
classic `$text` index	0
regex	0

Only the fuzzy option found anything. That's the actual capability being sold here, and it worked as documented.

Vector search against doing it by hand

I ran $vectorSearch against a 32-dimension cosine-similarity index over the same 50,000 documents, and compared it to scoring every vector in Python after fetching them once:

Method	min	median	max
`$vectorSearch`	10.38ms	15.52ms	93.93ms
brute-force Python cosine scan	123.23ms	134.89ms	243.87ms

$vectorSearch was roughly 8.7 times faster at the median, and that's generous to the brute-force side: I excluded the time to fetch all 50,000 embeddings over the wire, which a real application doing this in-process would also have to pay unless it already held every vector in memory.

Concurrency and cost

Ten concurrent clients issuing $search queries finished 20 total queries in 98.6ms of wall time, against 180.7ms for one client doing the same 20 sequentially — a real throughput gain. But per-query latency degraded: the median went from 8.41ms to 28.77ms, and the worst case rose from 13.07ms to 83.40ms. mongot is a shared, single-threaded-ish gRPC service from the client's point of view, and concurrency buys you overall throughput at the cost of tail latency.

I also checked docker stats while both containers sat idle, with nothing but two small indexes over 50,000 short documents behind them. mongod held 229.5MiB resident. mongot held 1.033GiB, over four times as much, and it hadn't answered a single query yet. Anyone budgeting a self-managed box for mongod alone needs to add a second, JVM-sized line item for mongot, and that line item doesn't shrink if the collection is small.

MongoDB's launch post says $search runs on "the same query model you already know." I wanted to see if that survived contact with a real pipeline rather than a one-line example, so I chained $match, $project, $sort, {$meta: "searchScore"} and $limit after a $search stage. It worked without any special-casing, filtering on price after the text match and sorting on the filtered set, which is a normal thing to want and not something the docs show directly.

What I got wrong on the way

The sync config in the docs sets readPreference: secondaryPreferred, and when the index sat in PENDING for the first few minutes I decided that was the problem: a single-node replica set with no secondary to prefer. So I built one. I added a second mongod container, ran rs.add(), watched rs.status() report a healthy SECONDARY, and waited for the index to move. It didn't. Another five minutes of the same 30-second retry loop passed before I gave up on that theory and went looking at mongot's metrics instead, where the disk numbers had been sitting the entire time. df -h on the same host never once suggested there was a problem.

Run it yourself

This needs Docker with a working daemon, not just the CLI.

docker network create search-community

# mongod.conf: net.bindIpAll true, replication.replSetName rs0,
# plus the setParameter block pointing at mongot-community:27028
docker run -d --name mongod --network search-community \
  --network-alias mongod.search-community \
  -v "$PWD/mongod.conf:/etc/mongod.conf:ro" \
  -v "$PWD/data/db:/data/db" -p 27017:27017 \
  mongodb/mongodb-community-server:latest --config /etc/mongod.conf

docker exec mongod mongosh --port 27017 --quiet --eval '
  rs.initiate({_id:"rs0", members:[{_id:0, host:"mongod.search-community:27017"}]})'
docker exec mongod mongosh --port 27017 --quiet --eval '
  db.getSiblingDB("admin").createUser({user:"mongotUser", pwd:"testpass123", roles:["searchCoordinator"]})'

# mongot.conf: syncSource pointing at mongod, storage.dataPath /data/mongot
docker run -d --name mongot-community --network search-community \
  --network-alias mongot-community.search-community \
  -v "$PWD/data/mongot:/data/mongot" \
  -v "$PWD/mongot.conf:/mongot-community/config.default.yml:ro" \
  -v "$PWD/passwordFile:/passwordFile:ro" \
  -p 8080:8080 -p 9946:9946 \
  mongodb/mongodb-community-search:latest

curl localhost:8080/health          # expect {"status":"SERVING"}
curl localhost:9946/metrics | grep disk_space_data_path   # check your own headroom first

Check your disk metric before you file a bug: (total - free) / total from mongot_system_disk_space_data_path_total_bytes and _free_bytes, not df -h. If you're above roughly 85%, free space before you touch the index definition.

What to do with this

Before you create a first index on a self-managed deployment, pull mongot_system_disk_space_data_path_free_bytes and do the arithmetic yourself rather than trusting df on the same box; the two disagreed by 60 percentage points on my host, and only one of them is the number mongot actually acts on. Don't rip out a working $text index or a regex query just because $search is newer — on plain term lookups mine was an order of magnitude slower. Reach for it when you need fuzzy matching or relevance scoring you can't already get, and reach for $vectorSearch once scoring vectors in application code starts to hurt, which on my 50,000-row test happened well before the collection felt large. If you restart mongot, open statusDetail before you believe whatever the top-level queryable field tells you.

Docker Compose's daily pull policy cuts registry checks by 80% during up

Alex Georgiev — Tue, 29 Sep 2026 10:00:00 +0000

Five docker compose up cycles against the same project produced five HEAD requests to my registry when the service used pull_policy: always, one when it used pull_policy: daily. That part worked exactly as documented. The surprise came when I ran the same comparison against docker compose pull directly: the registry got hit every single time, regardless of the policy or how long I waited between calls.

Docker Compose's pull_policy field has supported time-based refresh windows — daily, weekly, and every_<duration> — since v2.34.0. The idea is straightforward: instead of choosing between "always re-check the registry" and "never re-check unless the image is missing locally", you tell Compose how stale an image is allowed to get before it bothers asking the registry again. The v5.5.0 changelog, released on 17 August 2026, adds this line: "compose pull now honors pull_policy refresh windows (daily, weekly, every_N)." I wanted to know what that was worth in practice, and whether it actually did what the changelog said.

The Docker Engine install I was working with (29.3.1) shipped the Compose plugin at v5.1.1, already older than the release that claims to fix this. So I downloaded the v5.1.1 and v5.5.1 binaries directly from GitHub and ran both against the same setup: a local registry:2 container holding a single small image, with its access log tailed so I could count exactly when a manifest got checked.

What the window actually saves under `up`

With a fresh project — local image removed with docker rmi -f so there was no trace of it left on the daemon — and pull_policy: daily, I ran up -d then down five times in a row, immediately after each other. Compose only checked the registry on the first cycle. The other four reused the image it already had, without a single request leaving the box:

Policy	Cycles	Manifest checks	Image re-pulled
`always`	5	5	every cycle
`daily`	5	1	first cycle only

That's the headline number: four out of five checks avoided, on a workload that's realistic for anyone who runs docker compose up repeatedly during a working session, or as part of a CI job that spins the same stack up and down. Locally, against a registry on the same machine, the wall-clock difference across five cycles was noise — both runs took about 53 seconds, dominated by container and network setup rather than the registry call. That call is not free everywhere, though. Against Docker Hub, resolving an anonymous pull token and checking one manifest took between 0.40 and 1.01 seconds across three tries from this machine:

trial 1: token+HEAD roundtrip 1.014s
trial 2: token+HEAD roundtrip 0.431s
trial 3: token+HEAD roundtrip 0.404s

That is the cost daily is built to avoid, and on a slow or rate-limited registry it adds up over a day of repeated up calls.

I also checked that the window genuinely expires rather than just remembering "already pulled once, never again". With pull_policy: every_3s, an immediate second up skipped the check as expected, but after sleeping six seconds a third up checked the registry again and re-pulled. The mechanism is time-based, not a permanent cache flag.

Where the changelog claim didn't hold up

The part I couldn't reproduce is the specific one the release notes highlight. I built a project with pull_policy: every_3s and ran docker compose pull — the explicit subcommand, not up — three times: cold, immediately again, and a third time after sleeping five seconds so the window had clearly elapsed. Both v5.1.1 (before the fix) and v5.5.1 (the release that claims to have shipped it) hit the registry all three times:

pull #1 (cold): 1 check
pull #2 (immediate): 1 check
pull #3 (after 5s, window elapsed): 1 check

Same pattern with daily in place of every_3s. docker compose pull --help shows no new flag between the two versions, and docker compose up --help is identical too, so whatever changed in v5.5.0 isn't something a flag exposes. I can't tell from the outside what the fix actually touched, but on the exact scenario the changelog names — plain daily/weekly/every_N values, explicit pull, nothing else in play — I saw no difference in behaviour between the pre-fix and post-fix binary. If you were hoping the fix meant your CI pipeline's docker compose pull step would start skipping unnecessary registry calls, it doesn't, at least not in this scenario.

The tradeoff nobody puts on the label

A refresh window that skips checks also skips seeing real changes. I pushed a different image to the same tag while a daily-policied service was still inside its window, then reset the local tag back to the old image ID to rule out Compose picking up the change through the local Docker image store rather than the registry. Running up again produced zero manifest checks and started the container with the old image, silently:

manifest hits: 0
NAME="Alpine Linux"

That's the documented tradeoff, working as intended rather than as a bug, but it's worth being explicit about: any push to that tag during the window is invisible to up until the window closes. If you deploy hotfixes by re-pushing a mutable tag, pull_policy: daily will hide them from you for up to a day.

What it refuses

The pull_policy field is validated against a fairly narrow pattern. hourly and monthly are both rejected outright:

services.app.pull_policy 'monthly' does not match pattern
'^(always|never|build|if_not_present|missing|refresh|daily|weekly|every_([0-9]+[wdhms])+)+$'

every_3x fails the same way — the suffix after the number has to be w, d, h, m or s, nothing else. Neither rejection surprised me once I'd read the regex. What did surprise me was a sixth word sitting in that same pattern, one the docs page never names: refresh, on its own, with no number attached. I set pull_policy: refresh and ran up twice in a row. It checked the registry both times — same as always, no window at all. My guess is that daily, weekly and every_N all compile down to this same internal policy with a duration attached, and a duration of zero just means always trip it. That's a guess, not something I read in source; what I did confirm is that dropping the duration entirely is accepted by the schema and does something, and none of the three docs pages I checked mention it.

What I got wrong on the way

My first attempt at the staleness test looked like it disproved the whole feature: I pushed a new image under the same tag, ran up again inside the window, and the container came back running the new image with zero manifest checks logged. That looked like daily was somehow psychic. It wasn't. Pushing the new image from the same Docker daemon that was also running the compose project updates that daemon's local tag mapping immediately, with no registry round trip involved — docker tag and docker push together mean the local image store already knows about the new image before Compose ever looks at it. Compose was comparing local state, not talking to the registry at all, and getting the right answer for the wrong reason. Resetting the local tag back to the old image ID before the second up — so the only place the new content existed was the registry — is what actually isolated the registry-check behaviour I was trying to measure.

The same mechanism bit me a second time while writing the reproduction script below. I tagged and pushed the test image, then went straight into the five-cycle loop without removing the local image first. Every single cycle came back with zero registry checks, including the first one, which made it look like daily was broken in the opposite direction. It wasn't that either: docker tag stamps the image's LastTagTime the moment it runs, and Compose's freshness window reads that same timestamp. As far as Compose was concerned, the image had already been "checked" a second earlier, by an operation that never went near the registry. The fix was mechanical — docker rmi -f the tag before the loop, so the only honest timestamp left is the one Compose itself sets on a real pull — but it means the window isn't tracking "the last time Compose checked", it's tracking "the last time anything touched this tag locally", which is worth knowing if you also build or docker tag images under the same name your compose file uses.

Run it yourself

This needs Docker with a running daemon; no cloud account or paid registry involved.

# start a local registry and push a tiny test image
docker run -d -p 5500:5000 --name test-registry registry:2
docker pull alpine:3.20
docker tag alpine:3.20 localhost:5500/pulltest:latest
docker push localhost:5500/pulltest:latest

# tagging and pushing just now already stamped this image's local
# LastTagTime, which is what the refresh window reads -- remove the
# local tag so the first cycle below is a genuinely cold check
docker rmi -f localhost:5500/pulltest:latest

# tail the registry's access log in another terminal
docker logs -f test-registry

# a minimal project using the refresh window
cat > compose.yaml <<'EOF'
services:
  app:
    image: localhost:5500/pulltest:latest
    pull_policy: daily
    command: ["sleep", "3600"]
EOF

# five cycles: only the first should touch the registry
for i in 1 2 3 4 5; do
  docker compose up -d
  docker compose down
done

Switch pull_policy to always and repeat to see the difference in the registry log. Swap daily for every_3s and add a sleep 6 between two up calls to see the window actually expire. And to reproduce the part that didn't work as advertised, replace the for loop with two back-to-back docker compose pull calls and watch the registry log show a check both times regardless.

What to do with this

If you're using pull_policy: daily or weekly anywhere and relying on it to make docker compose pull in a CI step cheaper, check your registry's own access logs during that step. On the evidence here, it isn't skipping anything, on v5.5.1 or on the older v5.1.1 it's supposed to have fixed. The saving is real, but it's currently only real under up. If your workflow calls pull directly, and you want fewer registry round trips, you'd need to drop the explicit pull step and let up do the pulling instead — with the tradeoff that any mutable-tag push you make will stay invisible to your stack until the window you chose runs out.

PostgreSQL 19's WAIT FOR LSN command replaces the read-your-writes polling loop

Alex Georgiev — Mon, 28 Sep 2026 10:00:00 +0000

I had ten clients hammering a PostgreSQL standby with the same query in a loop: has my last write landed yet? Standby CPU sat at a mean of 57% across two runs. I swapped the loop for a single blocking call, same ten clients, same workload, and CPU dropped to 38% while throughput went up.

That blocking call is WAIT FOR LSN, new in PostgreSQL 19, which shipped its fourth beta on 24 September 2026. It gives an application a way to ask the database directly: block this connection until a specific write-ahead-log position has reached a named state on this server, then return. Read-your-writes consistency against an async replica has always been possible by hand, by capturing the WAL position after a write and polling for it on the replica. WAIT FOR turns that into one statement.

I built a primary and a streaming-replication standby with postgres:19beta4 and measured the old way against the new one. The honest version of the result is more interesting than "faster": the raw latency is close to a tie, and the real difference only shows up once you put load on it.

Setting it up

Two Docker containers, same image, one network:

docker network create pgwaitnet
docker run -d --name pg-primary --network pgwaitnet -p 15432:5432 \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_HOST_AUTH_METHOD=trust \
  postgres:19beta4 \
  postgres -c wal_level=replica -c max_wal_senders=10 -c hot_standby=on

I created a replication role and a slot on the primary, opened pg_hba.conf for replication and normal connections from the container network, then took a base backup straight into the standby's data directory with pg_basebackup -R, which writes standby.signal and primary_conninfo for me:

docker exec pg-primary psql -U postgres -c \
  "CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD 'replpass';"
docker exec pg-primary psql -U postgres -c \
  "SELECT pg_create_physical_replication_slot('standby_slot');"
docker exec -u postgres pg-standby bash -c \
  "PGPASSWORD=replpass pg_basebackup -h pg-primary -U replicator \
   -D /var/lib/postgresql/19/docker -Fp -Xs -P -R -S standby_slot"

pg_stat_replication on the primary showed the standby streaming within a couple of seconds, replay lag around 0.1ms on this one machine. Everything below ran against that pair, over TCP from the host, not through docker exec (I mention this because it mattered — see the mistake section further down).

The headline number, and why I don't fully trust it

For each of 20 trials, I inserted a row on the primary, captured the resulting LSN, and timed how long it took the standby to satisfy that LSN, three ways: a polling loop with a 1ms sleep between checks, a busy-polling loop with no sleep at all, and one WAIT FOR LSN ... WITH (MODE 'standby_replay') call.

method	median	mean	max
poll, 1ms sleep	0.49–1.65ms	0.96–1.42ms	1.97–2.02ms
busy-poll, no sleep	0.35–0.41ms	0.39–0.40ms	0.55–1.04ms
`WAIT FOR LSN`	0.28–0.36ms	0.31–0.38ms	0.56–0.77ms

(Ranges are across three repeated runs of 20 trials each for the sleeping poll, two for the others.)

Against the sleeping poll, WAIT FOR looks like a clean 3–5x win. Against the busy-poll, the gap almost disappears — WAIT FOR is still slightly ahead on every run, but by tenths of a millisecond, not multiples. A busy loop that never sleeps will notice a replayed LSN about as fast as a purpose-built wait primitive will, on one connection, on an idle standby. The 3–5x number is really measuring my arbitrary 1ms sleep, not the feature.

Where it actually wins: ten clients, continuously

I ran ten worker threads, each looping insert-on-primary, then wait-on-standby, for five seconds straight, and sampled standby CPU with docker stats during the run. Averaged over two runs:

method	completions/sec	standby CPU (mean)	standby CPU (peak)
busy-poll, 10 workers	3,182	57%	67%
`WAIT FOR LSN`, 10 workers	3,552	38%	52%

More completed work, at roughly two-thirds the mean CPU. This is the number I'd actually plan around: a busy-poll loop is fine for one client checking one write, but it doesn't idle — every waiting connection is a backend spinning a query in a tight cycle, and ten of those add up on the machine serving your replica reads. WAIT FOR parks the backend instead of spinning it. I only got two or three CPU samples per five-second run because spawning docker stats --no-stream has its own overhead, so treat the CPU figures as directional rather than to the percentage point — the completions-per-second numbers, counted directly rather than sampled, I trust more.

What it refuses

I fed it every wrong input I could think of.

Garbage LSN:

ERROR:  invalid input syntax for type pg_lsn: "not-a-lsn"

standby_replay mode run against the primary itself:

ERROR:  recovery is not in progress
HINT:  Waiting for the standby_replay LSN can only be executed during recovery.

primary_flush mode run against the standby:

ERROR:  recovery is in progress
HINT:  Waiting for primary_flush can only be done on a primary server. Use standby_flush mode on a standby server.

A made-up mode:

ERROR:  unrecognized value for WAIT option "mode": "bogus_mode"
LINE 1: WAIT FOR LSN '0/3000000' WITH (MODE 'bogus_mode', TIMEOUT '1...
                                       ^

A negative timeout:

ERROR:  timeout cannot be negative

All four of the role and mode errors carry a HINT that names the fix, which is more than the average Postgres error gives you. primary_flush mode itself works fine when run in the right place — I confirmed it on the primary directly, waiting on a just-issued LSN, and it returned success immediately once the WAL was flushed locally.

NO_THROW turns the timeout into data

By default, a WAIT FOR that times out raises an error:

ERROR:  timed out while waiting for target LSN FFFFFFFF/FFFFFFFF to be replayed;
current standby_replay LSN 0/03F18920

Add NO_THROW and the same situation returns a row instead of raising:

 status
---------
 timeout

A successful wait returns status = success the same way. For application code, that's the difference between wrapping every read-your-writes check in exception handling and just branching on a column, which matters if you're calling this from a connection pool path that runs on every request.

The documented claim I could check

The reference page says a TIMEOUT of zero or an omitted TIMEOUT both mean wait indefinitely, not "don't wait." I set up a standby waiting on an LSN that didn't exist yet, with no TIMEOUT clause, checked that it was still running after two seconds, then generated enough WAL on the primary to pass that LSN. The wait returned success right after, in step with the insert, not before it and not with an error. I repeated the same check with TIMEOUT '0' explicitly and got the same behaviour. Both hold as documented, and neither reads as "zero means immediate timeout," which is the more common convention elsewhere and the one I half-expected to trip over.

Watching it happen

While a session is blocked in WAIT FOR, another connection can see exactly what it's doing:

 pid | state  | wait_event_type |    wait_event    |                  query
-----+--------+------------------+-------------------+-------------------------------------------
 110 | active | IPC              | WaitForWalReplay  | WAIT FOR LSN '0/03F7A450' WITH (MODE...

wait_event_type = IPC, wait_event = WaitForWalReplay, sitting right there in pg_stat_activity. A polling loop shows up in that view only for the instant each individual poll query runs; between polls, from the database's point of view, the connection was never doing anything in particular. WAIT FOR gives you one row that tells you, unambiguously, "this connection is blocked waiting for replication to catch up," which is the kind of thing you want in a slow-query or blocked-session dashboard.

Concurrent waiters don't queue behind each other

I was slightly worried that ten or more processes all waiting on the same not-yet-issued LSN might be serialised — woken up one at a time as some internal list gets walked. I started 15 connections waiting on a single future LSN, inserted enough rows on the primary to satisfy it, and measured how spread out their wake-up times were.

Across two runs, the gap between the first and last waiter to return was under 10ms in both, with a standard deviation of 0.5ms and 3.3ms respectively, against absolute wait times in the low hundreds of milliseconds. They wake together, not in a queue.

What I got wrong on the way

My first version of this post would have led with "WAIT FOR is 3 to 5 times faster than polling," because that's what the first benchmark showed, and it looked like a clean, presentable number. I only added the busy-poll variant afterwards, as a sanity check on whether I was measuring the mechanism or my own choice of sleep interval. It was the latter. If I hadn't gone back and questioned the first result, the whole post would have overstated the case for a feature that, on the evidence I actually have, is worth adopting for a completely different reason than the one I started out expecting to write about.

I also lost the first primary container to my own docker rm -f while trying to add a port mapping I'd forgotten at setup, which meant rebuilding the base backup from scratch. Nothing in that was interesting, but it's why every container in the commands above gets its ports mapped on the very first docker run.

Run it yourself

This is the core of the latency comparison, trimmed to what you need; the full concurrency and CPU-sampling scripts are longer but follow the same shape.

import psycopg2, time

PRIM = dict(host='localhost', port=15432, user='postgres', password='postgres', dbname='postgres')
STBY = dict(host='localhost', port=15433, user='postgres', password='postgres', dbname='postgres')

def conn(d):
    c = psycopg2.connect(**d); c.autocommit = True; return c

pc, sc = conn(PRIM), conn(STBY)
pcur, scur = pc.cursor(), sc.cursor()

pcur.execute("INSERT INTO t(v) VALUES ('x') RETURNING id;")
pcur.execute("SELECT pg_current_wal_lsn();")
lsn = pcur.fetchone()[0]

t0 = time.perf_counter()
scur.execute(
    "WAIT FOR LSN %s WITH (MODE 'standby_replay', TIMEOUT '5s');", (lsn,)
)
print(scur.fetchone(), time.perf_counter() - t0)

You need postgres:19beta4 or later on both ends of a streaming-replication pair, psycopg2-binary on the client, and a table called t that exists on both sides. I ran this on one machine, so treat the millisecond figures as belonging to that machine. The CPU ratio under ten concurrent workers is the number I'd actually carry over to a different box, because it comes from comparing two ways of using the CPU, not from network distance.

Postgres 19 is still in beta, due for a release candidate in early October according to the project's own announcement, so nobody should put this on a production standby yet. What I'd do instead is find the polling loop already sitting in a read-replica code path — most teams doing async reads have one somewhere, even if it's disguised as a retry-with-backoff around a stale read — and check how many connections hit it concurrently at peak. If it's one or two, this change buys almost nothing. Past that, the CPU freed up is CPU your standby was supposed to be spending on the reads it exists to serve, not on asking itself the same question forty times a second.

containerd 2.2's mount manager panics on a one-mount mkfs chain

Alex Georgiev — Sun, 27 Sep 2026 10:00:00 +0000

containerd 2.2 shipped a mount manager: a service that can format a file as ext4 or xfs, attach it as a loopback device, and hand the result to a runtime, all from a single Activate call instead of the usual truncate, mkfs, losetup, mount sequence. I wanted to know whether that call is actually faster than doing it by hand, so I wrote a small Go program against the manager's package directly and ran both paths three times each on the same machine.

The manual sequence came out at 23.5 to 25.9 milliseconds. The mount manager's Activate came out at 29.6 to 47.2 milliseconds. It was not faster. Along the way it also panicked once, leaked a raw BoltDB error once, and left a loop device attached with nothing able to find it again.

What the mount manager actually is

There's no ctr subcommand for any of this. The manager lives at github.com/containerd/containerd/v2/core/mount/manager and is meant to be embedded by a snapshotter or a runtime shim, not driven from a terminal. Its job is to let a mount type be built out of steps: a "transformer" can create a file, format it, and format directories, and a "handler" can attach it as a loopback device, before the result gets handed off as an ordinary system mount.

The container running this test was Docker Engine 29.3.1, whose bundled containerd reports itself as v2.2.2. I confirmed the mount manager package is at that exact version by pinning it in go.mod and building against it, not by trusting the daemon's version string.

A working activation for a 200MiB ext4 image looks like this, once I had the templating right:

mounts := []mount.Mount{
    {
        Type:   "mkfs/loop",
        Source: imgPath,
        Options: []string{
            "X-containerd.mkfs.size=200MiB",
            "X-containerd.mkfs.fs=ext4",
        },
    },
    {
        Type:   "format/ext4",
        Source: "{{ mount 0 }}",
    },
}
info, err := mgr.Activate(ctx, "demo1", mounts)

Activate returns two sets of mounts: info.Active, the ones it handled itself (the loopback attach), and info.System, the ones it expects the caller to mount with the ordinary mount(2) syscall (the ext4 filesystem on that loop device). The manager does not mount your rootfs for you. You still call Mount() on whatever comes back in info.System.

The speed comparison

Both paths format a 200MiB ext4 image, attach it as a loopback device, and mount it. I timed the manager's Activate call and, separately, the same steps run as four shell commands: truncate -s 200M, mkfs.ext4 -q -F, losetup -f --show, mount.

Path	Run 1	Run 2	Run 3
Mount manager `Activate`	29.6ms	39.0ms	47.2ms
Manual (truncate + mkfs.ext4 + losetup + mount)	24.4ms	23.6ms	25.9ms

The manual path was faster on every run. That is not a criticism of the manager's design so much as an observation about what it's for: I checked the source of the mkfs transformer (core/mount/manager/mkfs.go) and it shells out to the real mkfs.ext4 and mkfs.xfs binaries with exec.CommandContext, exactly what the manual path calls directly. There's no custom fast-path formatter underneath. The manager's job is composability across snapshotters, not raw speed, and on this measurement it costs a small constant overhead (BoltDB writes, symlink creation, bookkeeping) rather than removing any.

Cleanup showed the same pattern: Deactivate plus umount took 32 to 45ms; umount plus losetup -d by hand took 11 to 13ms.

Disk usage was identical either way. du --apparent-size on both images read 200M; actual usage was 17M on both, since neither mkfs.ext4 nor truncate writes real data to most of the file.

What I got wrong on the way

My first version of the manual comparison used dd if=/dev/zero of=disk.img bs=1M count=200 to create the image before formatting it, because that's the version of this recipe I'd actually run in the past. It came out at 358ms to 1.6 seconds; against that, the mount manager looked ten to fifty times faster, which would have been the headline of this post.

dd was writing 200MB of real zero bytes to disk before mkfs.ext4 ever ran. truncate -s 200M creates the same size file as a sparse hole in under a millisecond, and mkfs.ext4 doesn't need the data pre-zeroed, it only writes its own metadata. The mount manager's mkfs transformer already does the equivalent of truncate, via os.OpenFile plus f.Truncate(size), then calls the real mkfs.ext4 binary on the result. Once I made the manual comparison do the same thing, the ten-times "win" disappeared and mildly reversed. The lesson: when a new API bundles three steps into one call, benchmark it against the shortest correct version of those three steps, not the version you happen to type from muscle memory.

Under concurrency

Ten goroutines calling Activate in parallel against the same manager instance, each formatting its own 50MiB image, completed in 70.1ms of wall time, with individual calls ranging from 30.5 to 69.8ms. All ten succeeded. If activations were serialized behind a lock, ten of them would have taken close to 300-400ms; they didn't, so the manager's internal locking (RLock during normal activation, held exclusively only during garbage collection) does allow real concurrent formatting.

What it refuses

An unsupported filesystem type is rejected before any file gets created:

unsupported filesystem "btrfs": invalid argument

A missing size option is also rejected cleanly:

mkfs requires mkfs.size option: invalid argument

A path outside the manager's configured root is rejected too, but with a misleading error class:

no root "/tmp/not-the-root/disk.img" configured for mkfs: not implemented

That comes back as errdefs.ErrNotImplemented, the same error category containerd uses for "this operation genuinely doesn't exist here." Code that checks errdefs.IsNotImplemented() to decide whether to fall back to a different mount path would treat "you forgot to allow this directory" the same as "this feature isn't built." I read the source (core/mount/manager/mkfs.go) to confirm this isn't a formatting quirk on my end; the transformer returns exactly that wrapped error whenever the source path doesn't match any configured root.

Activating a second time under the same name, without deactivating the first, doesn't get a clean "already exists" either:

bucket already exists

That's a raw bbolt error surfacing straight from the metadata store, with no containerd-level wrapping. It's accurate, but it tells you about the manager's storage engine rather than about your mistake.

The panic

The documentation's own examples always chain at least two mounts: something that produces a loopback device, and something that mounts a filesystem on it. I tried activating a single mkfs/loop mount on its own, with nothing consuming its output, expecting either a successful format-only activation or a clean validation error.

panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/containerd/containerd/v2/core/mount/manager.(*mountManager).Activate(...)
    .../core/mount/manager/manager.go:421 +0x2030

I read manager.go to find out why. The function tracks firstSystemMount, the index of the first mount it expects the caller to handle. When a mount is only ever a transform target (my single mkfs/loop mount, with no second mount to hand the loop device to), that index gets set to i+1, which in a one-mount list equals len(mounts). A later loop indexes into mountConv[firstSystemMount] to apply any pending format templating, and mountConv was allocated with len(mounts) elements. Index 1 into a slice of length 1 panics.

This matters beyond the crash itself. The loopback device gets attached to the backing file before the panic point, and because Activate never returns successfully, nothing gets written to the manager's BoltDB. There is no record of this activation to recover.

Crash recovery, and where it doesn't reach

The manager persists activation state in BoltDB specifically so a restarted process can find and clean up mounts from before a crash. I tested that separately from the panic: one process called Activate and then exited hard with os.Exit(0), skipping Deactivate entirely, to simulate a daemon that died mid-operation.

$ ./mmdemo -mode=crash-activate
activated in 30.7ms, err=<nil>

$ ./mmdemo -mode=crash-recover
List() after simulated crash: 1 activations, err=<nil>
  recovered activation: crashy active=[{loop ... /tmp/mm-crash/targets/1/1}]
cleanup via Deactivate: <nil>

A second process, pointed at the same database and target directory, listed the orphaned activation and deactivated it cleanly, and the loop device it had been holding was released. That documented claim held up.

But it only works because the first Activate call returned successfully and got committed. Compare that against the one-mount panic above: I left that test running separately, and the loop device it opened is still attached to a deleted backing file with no BoltDB record anywhere pointing at it. losetup -a still shows it. No amount of restarting a mount manager pointed at any database will find it, because it was never written down. The crash-recovery mechanism protects against a daemon dying after an activation completes. It has no way to protect against the daemon crashing during one.

xfs, and a size limit that isn't the manager's

I ran the same mkfs/loop chain with X-containerd.mkfs.fs=xfs at 200MiB and got a full mkfs.xfs usage message back:

mkfs.xfs failed: Filesystem must be larger than 300MB.

That's mkfs.xfs itself refusing, not the manager. At 400MiB, formatting succeeded in 48 to 72ms. Mounting the result failed in this environment specifically:

mount source: ".../targets/1/1", target: ".../rootfs", fstype: xfs, flags: 0, data: "", err: no such device

/proc/filesystems on this container's kernel has no xfs entry and there's no modprobe to load one. That's a property of the sandbox this test ran in, not of containerd, and I'm noting it rather than counting it as a finding against the feature.

Run it yourself

This needs Go 1.24+, root (for loopback devices and mounts), and mkfs.ext4.

mkdir mounttest && cd mounttest
go mod init mounttest
go get github.com/containerd/containerd/v2@v2.2.2

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/containerd/containerd/v2/core/mount"
    "github.com/containerd/containerd/v2/core/mount/manager"
    "github.com/containerd/containerd/v2/pkg/namespaces"
    bolt "go.etcd.io/bbolt"
)

func main() {
    base := "/tmp/mm-demo"
    os.RemoveAll(base)
    os.MkdirAll(base+"/targets", 0755)

    db, _ := bolt.Open(base+"/meta.db", 0644, nil)
    defer db.Close()

    mgr, err := manager.NewManager(db, base+"/targets",
        manager.WithMountHandler("loop", mount.LoopbackHandler()),
        manager.WithAllowedRoot(base),
    )
    if err != nil {
        panic(err)
    }

    ctx := namespaces.WithNamespace(context.Background(), "default")
    mounts := []mount.Mount{
        {Type: "mkfs/loop", Source: base + "/disk.img", Options: []string{
            "X-containerd.mkfs.size=200MiB",
            "X-containerd.mkfs.fs=ext4",
        }},
        {Type: "format/ext4", Source: "{{ mount 0 }}"},
    }

    info, err := mgr.Activate(ctx, "demo1", mounts)
    fmt.Printf("info=%+v err=%v\n", info, err)

    os.MkdirAll(base+"/rootfs", 0755)
    for _, sm := range info.System {
        if err := sm.Mount(base + "/rootfs"); err != nil {
            fmt.Println("mount failed:", err)
        }
    }

    // activating the same name again without deactivating first:
    _, err = mgr.Activate(ctx, "demo1", mounts)
    fmt.Println("second activate, same name:", err)
}

go build -o mmdemo .
sudo ./mmdemo
mount | grep mm-demo
sudo umount /tmp/mm-demo/rootfs

I verified every command in this section on a fresh checkout before publishing.

What to do with this

If you're building on the mount manager today, treat it as a composability primitive, not a performance one; it won't beat a shell one-liner on latency. Never construct a mount list that ends in a transform-only type like mkfs/loop without a mount that actually consumes its output, until this specific crash is fixed upstream. Don't branch on errdefs.IsNotImplemented() from this package without also checking the error text, because it currently covers both "unsupported" and "not configured." And if you're relying on its crash recovery for anything in production, test it against a process that dies mid-Activate, not just one that dies after — those are different guarantees, and only one of them is covered right now.

systemd's mstack tool mounts a single-layer directory writable by default

Alex Georgiev — Sat, 26 Sep 2026 10:00:00 +0000

I built a two-file directory, mounted it with systemd's new mstack tool, wrote one line into it, and watched that line permanently overwrite a file I'd marked read-only. No error, no warning, exit code 0.

mstack is one of the features systemd picked up this year: a way to describe an overlayfs mount — several read-only layers plus one writable layer — as an ordinary directory full of symlinks, instead of a long mount -t overlay -o lowerdir=... command line. It landed in systemd 260 in March and systemd-nspawn grew more support for it in 261, released in June and now the current stable release (261.3 is what ships in today's Arch Linux image). I ran everything below against that build, in Docker, since my sandbox has no systemd running as PID 1.

The idea is genuinely useful: put symlinks named layer@0, layer@1 and so on in a directory suffixed .mstack, add a rw/ subdirectory for the writable top, and systemd-mstack --mount that.mstack /somewhere assembles the whole overlay in one call. systemd-nspawn --mstack=, and a service's RootMStack= directive, can point at the same directory. It's meant to make mount stacks shareable and inspectable rather than baked into a shell script.

Building one that works

$ mkdir -p base app app.mstack
$ echo "base content" > base/f.txt
$ echo "app content" > app/g.txt
$ ln -s ../base app.mstack/layer@0
$ ln -s ../app app.mstack/layer@1
$ mkdir app.mstack/rw
$ systemd-mstack app.mstack
TYPE  NAME    IMAGE     WHAT       WHERE SORT
layer layer@0 directory /work/base /     0
layer layer@1 directory /work/app  /     1
rw    rw      directory /work/app.mstack/rw /     -
$ systemd-mstack --mount app.mstack /mnt/target
$ ls /mnt/target
f.txt  g.txt

That worked first try, on a tmpfs-backed directory. Both layers' files showed up merged, as expected.

The overlay-on-overlay refusal, and it isn't mstack's fault

My first attempt wasn't on tmpfs, it was in a plain Docker container's own filesystem, which uses the overlay2 storage driver by default. There, the exact same app.mstack directory produced this:

$ systemd-mstack --mount app.mstack /mnt/target
'(layerfd)' failed with exit status 1.
Failed to apply .mstack/ directory '/work/app.mstack': Invalid argument

That told me nothing useful, and for a while I assumed mstack itself was broken or that I'd built the directory wrong. It wasn't. I ran the equivalent hand-written command against the same two directories:

$ mount -t overlay overlay -o lowerdir=app:base,upperdir=upper,workdir=workdir merged
mount: /work/merged: fsconfig() failed: overlay: filesystem on upper not supported as upperdir.

Same failure, plainer error. The Linux kernel won't let you stack a new overlayfs mount on top of directories that are themselves already on an overlayfs — which is exactly what a Docker container's root filesystem is under the default overlay2 driver. This isn't new in 261 and it isn't mstack's doing; the manual, decade-old mount command hits the identical wall. mstack just explains it worse: it names an internal function, (layerfd), instead of naming the actual constraint. Anyone testing this feature inside a plain Docker container, which is most people's first instinct, will hit it and get the confusing version.

The workaround is the same for both: put the layers on a filesystem that isn't itself overlayfs. A tmpfs works. So does a bind-mounted host directory:

$ docker run --privileged -v /some/host/dir:/work archlinux bash
# (inside)
$ systemd-mstack --mount app.mstack /mnt/target
$ echo EXIT:$?
EXIT:0

I confirmed this against an ext-family host directory and it mounted cleanly, no tmpfs required.

The version-sort claim held up

The documentation says layer ordering is decided by a version sort, not a plain string sort, specifically so layer@10 doesn't get treated as coming before layer@2. I built layers layer@1 through layer@3 and layer@10 through layer@12 and asked for the JSON view:

$ systemd-mstack vsort.mstack --json=short
[{"name":"layer@1","sort":"1"},{"name":"layer@2","sort":"2"},{"name":"layer@3","sort":"3"},
 {"name":"layer@10","sort":"10"},{"name":"layer@11","sort":"11"},{"name":"layer@12","sort":"12"}]

A plain lexicographic sort of those strings would put layer@10 right after layer@1. It didn't. The claim checks out.

No speed advantage, and no extra disk cost

I timed five mounts each of the same two-layer stack, mstack against the hand-written mount -t overlay command, both on tmpfs:

method	run 1	run 2	run 3	run 4	run 5
`systemd-mstack --mount`	5ms	4ms	4ms	4ms	5ms
`mount -t overlay`	4ms	3ms	3ms	3ms	3ms

Roughly a millisecond slower, consistently, which is the extra work of resolving symlinks and building the option string. At this scale it's noise, not a cost anyone would notice. The value of mstack isn't speed. Anyone expecting a performance win from the new tool won't find one here — it's an ergonomics and inspectability feature, not a faster overlay.

I also put a 100MB file inside one layer and mounted it, to check whether mstack copies layer content anywhere before mounting:

$ dd if=/dev/zero of=biglayer/bigfile bs=1M count=100
$ systemd-mstack --mount biglayer.mstack /mnt/bigmount
$ du -sh /tmp/mstack-temporary-*
40      /tmp/mstack-temporary-focKQP
$ stat -c '%s bytes' /mnt/bigmount/bigfile
104857600 bytes

The temporary staging directory mstack creates is 40 bytes, not 100 megabytes. The file is fully visible through the mount, so nothing was lost, and nothing was duplicated. That's a fair result and worth stating plainly since a "self-describing" abstraction is exactly the kind of thing I'd expect to add a copy step somewhere.

One thing I couldn't explain: systemd-mstack --mount on a genuine two-layer stack shows two identical lowerdir+= entries pointing at the same temporary path in /proc/self/mountinfo, rather than two distinct paths for the two distinct source layers:

overlay /work/app.mstack rw,lowerdir+=/tmp/mstack-temporary-focKQP,lowerdir+=/tmp/mstack-temporary-focKQP,upperdir=/tmp/mstack-temporary-focKQP/data,...

The mount worked and both layers' files were present in the result, so it isn't a functional bug I could demonstrate. I just don't know why the kernel reports it that way, and I'd rather say that than guess.

Where it silently gives you nothing

Two failure modes produced no error text at all, which is worse than the opaque (layerfd) message above.

A .mstack directory with a symlink pointing nowhere:

$ ln -s ../does-not-exist bad.mstack/layer@0
$ systemd-mstack bad.mstack
$ echo EXIT:$?
EXIT:1

That's the whole output. --show gives you an exit code and nothing to act on. Only --mount on the same broken directory names the problem, and only because mounting forces it to actually try to open the target:

$ systemd-mstack --mount bad.mstack /mnt/bad
Failed to apply .mstack/ directory '/work/bad.mstack': No such file or directory

A stray file or directory inside a .mstack/ that doesn't match layer@* or rw does the same thing: both the plain and --json show commands exit 1 with no message. If you're building these directories by a script and something goes wrong upstream — a template that leaves a stray file behind, say — you get a bare failure and have to guess.

The part that would actually catch someone out

This is the finding I'd act on. A .mstack directory with exactly one layer, no rw/ subdirectory, and no --read-only flag does not refuse to mount and does not fall back to a safe, discardable overlay. It bind-mounts the source directory directly, writable:

$ ln -s ../base noRw.mstack/layer@0
$ systemd-mstack --mount noRw.mstack /mnt/norw
$ mount | grep norw
tmpfs on /mnt/norw type tmpfs (rw,relatime)
$ echo "MODIFIED via mount" > /mnt/norw/f.txt
$ echo "written via mount" > /mnt/norw/new-from-mount.txt
$ systemd-mstack --umount /mnt/norw
$ cat base/f.txt
MODIFIED via mount
$ ls base/
f.txt  new-from-mount.txt

base/f.txt on disk now says "MODIFIED via mount", permanently, and the new file is sitting there too, after unmount. The mount type even reports as tmpfs, not overlay, which is the tell: with a single layer and no upper directory, mstack skips building an overlay at all and just bind-mounts the layer as-is. Nothing about the directory name, layer@0, or the general framing of "layers" as something you stack read-only content underneath, warns you that this specific shape writes straight through.

I checked whether the same thing happens with two layers and no rw/, since that felt like the more likely real-world mistake — someone forgetting the writable directory on a proper multi-layer stack:

$ systemd-mstack --mount twolayer.mstack /mnt/two
$ mount | grep /mnt/two
overlay ... (ro,relatime,...)
$ echo "written" > /mnt/two/newfile.txt
bash: /mnt/two/newfile.txt: Read-only file system

With two or more layers, the same omission correctly falls back to a read-only overlay. The unsafe default is specific to the single-layer case, where mstack decides an overlay isn't necessary and hands you the raw directory instead. That's a narrow trigger, but a single read-only "base" layer referenced from several application stacks — the exact use case the tool's docs describe — is precisely the shape where this would fire.

What I got wrong on the way

My first read of the (layerfd) error was that mstack had a bug, or that I'd built my test directory wrong, and I spent time double-checking symlink targets and directory names before it occurred to me to try the plain mount -t overlay command against the same two directories. Once that failed with the same underlying cause and a clearer message, it was obvious the fault was the container's storage driver, not the tool. I should have reached for that control test first rather than last.

Run it yourself

This needs systemd 260 or newer for the systemd-mstack binary. Arch Linux's current image ships 261.3, which is what I used:

docker run --rm --privileged -v "$(pwd)/work:/work" archlinux:latest bash -c '
  cd /work
  mkdir -p base app app.mstack
  echo "base content" > base/f.txt
  echo "app content" > app/g.txt
  ln -s ../base app.mstack/layer@0
  ln -s ../app app.mstack/layer@1
  mkdir app.mstack/rw
  mkdir -p /mnt/target
  systemd-mstack --mount app.mstack /mnt/target
  ls /mnt/target
  systemd-mstack --umount /mnt/target
'

Bind-mounting a real host directory as /work, as above, sidesteps the overlay-on-overlay refusal. Drop the -v and it will fail with the (layerfd) error on most default Docker setups, which is itself worth seeing once.

What to do with this

If you're packaging application images with a single, shared read-only base and building .mstack directories by script, add the rw/ subdirectory even when you intend the mount to be read-only, and pass --read-only explicitly rather than relying on the absence of a writable layer to protect anything. Don't test this feature for the first time inside a stock Docker container without bind-mounting a real directory in — you'll spend time chasing a kernel limitation that has nothing to do with the tool. And if a .mstack directory fails to show or mount with no message at all, check for a broken symlink or a stray file before assuming the tool is the problem; in my testing that combination produced silence rather than a diagnostic every time.

I didn't get as far as testing RootMStack= from an actual service unit or systemd-nspawn --mstack= end to end, because that needs systemd running as PID 1, which my sandbox doesn't have. That's the natural next thing to check.