DEV Community: Alex Georgiev

Python 3.14's concurrent.interpreters module turns a 3.6-second GIL bottleneck into 0.9 seconds

Alex Georgiev — Mon, 26 Oct 2026 11:00:00 +0000

A CPU-bound Python job split across four threads took 3.6 seconds on my machine. The same job split across four of Python 3.14's new subinterpreters took 0.9 seconds, roughly what one thread alone takes to do a quarter of the work. That part matched what I expected from the PEP. What I did not expect was that asking one of those subinterpreters to run two things at once does not slow it down. It refuses outright.

concurrent.interpreters is new in the standard library as of Python 3.14, released on 7 October 2026. Subinterpreters themselves are not new - they have existed in the C API since Python 1.5 and CPython has used them internally for years - but PEP 734 is what makes them usable from ordinary Python code, with no C extension required. Each subinterpreter gets its own Global Interpreter Lock, which is the part worth testing: it is the thing that is supposed to let CPU-bound Python code actually use more than one core without reaching for multiprocessing.

I tested it against a CPU-bound workload (counting primes below 400,000, a plain Python double loop with no C extensions to hide behind), compared against threads, processes, and a single thread, three runs each, inside a python:3.14-slim container with four cores available.

The headline number

approach	time for 4x the work
single thread (1x unit, for reference)	0.77 - 0.80s
4 OS threads, classic GIL	3.51 - 3.70s
4 subinterpreters (new in 3.14)	0.89 - 0.98s
4 processes (`multiprocessing`)	0.95 - 1.30s

Four threads doing four units of CPU-bound work took more than four times as long as one thread doing one unit - 3.6s against roughly 0.8s, which is actually a bit worse than the 3.2s you'd get from perfect serialisation, because the GIL has to keep stopping one thread to let another run. Four subinterpreters doing the same four units took about 0.9s: close enough to the single-unit baseline that the extra three units of work were, for practical purposes, free. That is real parallelism, not a scheduling trick, and it lands in the same place multiprocessing already does.

What it refuses: one thread per interpreter, not shared

The PEP's selling point is that each interpreter has its own GIL. I wanted to see what happens when two threads try to use one interpreter's GIL at the same time, expecting it to behave like classic Python: both threads run, the shared GIL makes them take turns.

That is not what happened. A single Interpreter object only tolerates one caller. A second concurrent call doesn't queue, and it doesn't share time with the first - it is rejected immediately:

Exception in thread Thread-2 (_call):
Traceback (most recent call last):
  File "/usr/local/lib/python3.14/concurrent/interpreters/__init__.py", line 226, in _call
    res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
concurrent.interpreters.InterpreterError: interpreter already running

I confirmed the shape of this with three variants of the same interpreter, run in sequence:

scenario	time	outcome
one call	0.84s	fine
two sequential calls, same interpreter	1.63s	fine, roughly 2x
two concurrent calls, same interpreter	0.84s wall-clock	second call raises `InterpreterError`

interpreters.create() is documented as returning "a new (idle) Python interpreter", and that word is accurate: idle or running, nothing in between. If you want N things running at once, you need N interpreters, one each. A thread pool aimed at a single shared interpreter - the pattern every ThreadPoolExecutor tutorial uses with regular threads - does not translate. You get one interpreter's worth of work and an exception, not graceful serialisation.

The opposite number: interpreters are not obviously cheaper to start than processes

The other assumption I walked in with was that subinterpreters, being lighter than OS processes, would be cheap to spin up. I measured the cost of creating four workers and giving each one some real state (a 200,000-element list), steady state after the first warm-up run:

approach	create + populate, 4 workers
`fork` (Python 3.13's Linux default)	0.037 - 0.038s
`forkserver` (Python 3.14's Linux default)	0.057 - 0.065s
4 subinterpreters	0.076 - 0.078s

Creating four subinterpreters and running a small amount of code in each took about twice as long as a classic fork-based process pool, and somewhat longer than the new forkserver default too. None of this matters once the work inside each worker takes more than a few milliseconds - the 3.6s-versus-0.9s number above swallows an 80-millisecond setup cost without noticing. But it means "subinterpreters are lighter than processes" is not something I could verify at creation time; the saving, where it exists, is in not forking a second copy of the process's address space, not in any faster start.

That forkserver row was an accident. I wrote a timing script that called multiprocessing.Process(...).start() directly from the module's top level, the way plenty of tutorial code still does, and it crashed:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

Checking multiprocessing.get_start_method() explained it: Python 3.14 changed the default start method on Linux from fork to forkserver. Python 3.13 reports fork; 3.14 reports forkserver. forkserver re-imports your module in the child, which is exactly the thing if __name__ == "__main__": exists to guard against, and which fork never required you to bother with. Every piece of multiprocessing code I'd written without that guard over the years happened to work by accident, on Linux, until this release. It's a one-line fix, but it is a real break, and it landed in the same release that gives you an alternative that doesn't have this problem at all.

What it costs: closing an interpreter doesn't give the memory back

I ran four rounds of "create four interpreters, give each a 200,000-element list, close them", watching my own process's RSS the whole time:

round 1: 16920 -> 55648 KB
round 2: 55648 -> 57140 KB
round 3: 57140 -> 59476 KB
round 4: 59476 -> 61096 KB

.close() does finalise the interpreter - a destroyed interpreter can't be called again, and I checked that it raises if you try - but the process's resident memory never drops back to where it started. It flattens out (each later round adds less than the first) rather than growing without bound, which looks like CPython's allocator holding onto freed arenas rather than a genuine leak, but I didn't chase that further. If you're planning to cycle through many short-lived interpreters inside one long-running process, budget for that memory to stay resident, not to be returned to the OS the way a finished process's memory is.

What's actually shareable between interpreters

Subinterpreters don't share objects by default - that is the entire point, it's what lets them skip the GIL - so most interesting values need to cross a boundary. is_shareable() gives a yes/no answer per type:

int              -> True
str              -> True
bytes            -> True
None             -> True
bool             -> True
tuple of ints    -> True
float            -> True
list             -> False
dict             -> False

Passing a threading.Lock as an argument to Interpreter.call() fails immediately, with the repr of the offending object right there in the message:

NotShareableError: (<unlocked _thread.lock object at 0x7f61b8b5add0>,) does not support cross-interpreter data

An ordinary custom class instance fails the same way, with a flatter message:

NotShareableError: object does not support cross-interpreter data

But the cross-interpreter Queue that the same module provides does not enforce this. I put a plain custom-class instance onto a queue and it went through without complaint, and came back out as a distinct, working object with the same attribute values - the queue silently pickles and unpickles it:

q = interpreters.create_queue()
q.put(Thing())          # no error
received = q.get()
# received is not the original object, but received.value is unchanged

So "not shareable" is enforced differently depending on how the object crosses over. A direct argument to call() is checked against the strict shareable-types list and rejected on the spot. The same object dropped into a Queue gets the softer treatment the docs describe for put(): pickled going in, unpickled coming out as a new, unrelated object. If your code passes data between interpreters by argument, moving it to a queue instead buys you a much larger set of types - at the cost of identity and, for anything mutable, of staying in sync with the original.

What I got wrong on the way

My first attempt at the shareability tests put several checks in one script with no if __name__ == "__main__": guard, calling interp.call() with a mix of shareable and non-shareable arguments one after another. Running it printed every result twice, and the second copy sometimes disagreed with the first about which error got raised. My best explanation, which I didn't fully verify, is that call()'s documented pickle fallback needs to resolve names against the target interpreter's own __main__, and since that interpreter had never run my module, resolving anything defined at module level meant running the whole module there first - silently re-executing my test file a second time, inside the subinterpreter, with its own fresh state. Splitting each check into its own small script with a proper __main__ guard made every result reproducible, which is what's reported above. I'd treat any subinterpreter test that imports a lot of top-level code as suspect until it survives being run twice.

Run it yourself

This needs Python 3.14 and four-ish cores to see the gap clearly. I ran it in Docker because that's the fastest way to get an exact version:

docker pull python:3.14-slim

The core benchmark:

# bench.py
import time, threading, sys

def cpu_work(n):
    count = 0
    for i in range(2, n):
        is_prime = True
        for d in range(2, int(i ** 0.5) + 1):
            if i % d == 0:
                is_prime = False
                break
        if is_prime:
            count += 1
    return count

def run_threads(n, workers):
    threads = [threading.Thread(target=cpu_work, args=(n,)) for _ in range(workers)]
    t0 = time.perf_counter()
    for t in threads: t.start()
    for t in threads: t.join()
    print(f"threads={workers}: {time.perf_counter()-t0:.3f}s")

def run_interpreters(n, workers):
    import concurrent.interpreters as interpreters
    interps = [interpreters.create() for _ in range(workers)]
    t0 = time.perf_counter()
    threads = [i.call_in_thread(cpu_work, n) for i in interps]
    for t in threads: t.join()
    print(f"interpreters={workers}: {time.perf_counter()-t0:.3f}s")
    for i in interps: i.close()

if __name__ == "__main__":
    mode, n, workers = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
    {"threads": run_threads, "interpreters": run_interpreters}[mode](n, workers)

docker run --rm -v "$PWD":/w -w /w python:3.14-slim python3 bench.py threads 400000 4
docker run --rm -v "$PWD":/w -w /w python:3.14-slim python3 bench.py interpreters 400000 4

And the refusal, which needs no setup at all:

docker run --rm python:3.14-slim python3 -c "
import concurrent.interpreters as interpreters
i = interpreters.create()
t1 = i.call_in_thread(lambda: __import__('time').sleep(1))
t2 = i.call_in_thread(lambda: __import__('time').sleep(1))
t1.join(); t2.join()
"

What to do with this

If you have a CPU-bound Python workload that multiprocessing already parallelises well, subinterpreters aren't an obvious upgrade yet: on this release they start up slower than a plain fork and about the same as forkserver, and you still need one interpreter per concurrent task, with no pooling against a single interpreter. Where they're worth trying now is the code that currently reaches for a thread, because threading is simpler to wire up than multiprocessing, and then quietly gets nothing from the GIL: swap the Thread for an Interpreter.call_in_thread, and check your data either fits the shareable-types list or is willing to cross as a pickled copy through a Queue. And if you have multiprocessing code anywhere that starts a Process without an if __name__ == "__main__": guard, fix that first - 3.14 will find it on Linux even if nothing else in this post applies to you.

Valkey 9.2's new rdbcompression lz4 setting cuts a 1M-row RDB from 178MB to 35MB

Alex Georgiev — Sun, 25 Oct 2026 11:00:00 +0000

I set rdbcompression lz4 on a Valkey 9.2.0-rc1 primary, watched a replica do a full sync, and measured exactly the same number of bytes going over the wire as with compression off. For a few minutes I believed the feature didn't touch replication at all. It does. I just had the wrong setting.

Valkey 9.2.0-rc1, released as a release candidate on 16 September with GA planned for mid-November, adds rdbcompression lz4: whole-stream LZ4 compression for RDB files. Until now, RDB persistence has compressed each string value separately with LZF, a scheme the project's own issue tracker describes as "quite inefficient" because it can only find repetition inside one value, never across the file. I ran both schemes against three kinds of data and then went looking for where the new setting falls over.

The headline number

I loaded a million keys shaped like a typical user-profile hash, each a small JSON document with a handful of repeated field names and a handful of unique values, and ran SAVE three times under each compression setting.

rdbcompression	file size	SAVE time (best of 3)
no	180.3 MB	0.80s
yes (default, per-string LZF)	177.6 MB	1.10s
lz4	35.4 MB	1.16s

The default LZF setting barely moves the needle: 180.3MB down to 177.6MB, a 1.5% saving, for the CPU cost of running compression at all. lz4 takes the same file down to 35.4MB, an 80.1% cut against the default and 80.4% against no compression whatsoever, for about 6% more CPU time than LZF already costs. I reran it on a second, more repetitive dataset — 200,000 synthetic log lines with a shared template — and the gap was the same shape: 31.47MB uncompressed and under LZF, 5.91MB with lz4, an 81.2% cut.

The reason LZF does almost nothing here is exactly the limitation the project describes: each JSON document is 150-200 bytes, with no exploitable repetition inside itself, and LZF never looks past one value's boundary. Whole-stream LZ4 compresses the serialized file as one byte stream, so the repeated "active":true, "roles":["user","member"] and field names across a million records are exactly the kind of redundancy it picks up and LZF structurally cannot.

Where it doesn't help

I tried the same experiment on 200,000 keys holding 32-byte random hex tokens, the kind of value you'd get from session IDs or API keys. There is no cross-record redundancy to find, and it showed:

rdbcompression	file size	SAVE time (best of 3)
no	15.49 MB	0.186s
yes	15.49 MB	0.211s
lz4	14.76 MB	0.283s

A 4.7% size reduction for roughly 35% more wall-clock time than LZF, and 52% more than no compression at all. On a database that's mostly tokens, counters or anything that already looks random, whole-stream LZ4 is close to pure overhead. The win is entirely a property of the data, not the algorithm being generically better than LZF — it just has a wider window to search in.

What I got wrong on the way

My first attempt at measuring the replication effect gave a flat, suspicious result: identical replica-side byte counts under rdbcompression no, yes and lz4, down to a few kilobytes of jitter. I assumed the flag simply wasn't wired into the sync path yet, since this is a release candidate.

Before concluding that, I checked whether my measurement itself was broken, by re-running a plain SAVE on disk with the same dataset and the same setting. That still showed the expected 80% drop. So the instrumentation was fine; the behavior genuinely differed between a local save and a replication sync. Looking at the primary's log line during the sync was the actual answer: it read Starting BGSAVE for SYNC with target: disk using: normal sync, with no mention of compression at all. Setting rdbcompression lz4 on the primary had done nothing for the replication path.

The fix is a second setting, repl-compression, which controls the replication stream independently of the on-disk RDB format. Once I set repl-compression lz4 on the primary, the exact same sync produced this log line instead:

7:M 25 Oct 2026 05:12:15.412 * Disk-based full sync with compression: lz4
7:M 25 Oct 2026 05:12:15.587 * Replication compression enabled for replica 172.17.0.4:6379 (algo=lz4)

and the replica's actual received bytes dropped from 17,627,802 to 3,435,304 — an 80.5% cut, in line with the file-based number. rdbcompression lz4 and repl-compression lz4 are two separate flags controlling two separate code paths, and the release note text I found for this change talks only about the first one. Setting only rdbcompression lz4 on a primary that replicates will leave every full sync exactly as large as before, silently, with no warning anywhere.

Mixed fleets cost more than one BGSAVE

I wanted to know what happens when an old replica and a compression-aware one both attach around the same time. I started a Valkey 8.1 replica and a 9.2.0-rc1 replica configured with repl-compression lz4 against the same primary within the same second. The legacy replica synced cleanly — all million keys landed, master_link_status:up, no errors — so the fallback to plaintext for a replica that can't speak the new protocol works as I'd want. But the primary's log showed two separate BGSAVE passes, not one shared fork:

7:M * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC
7:M * Diskless rdb transfer, done reading from pipe, 1 replicas still up.
7:M * Streamed RDB transfer with replica 172.17.0.5:6379 succeeded (socket)...
7:M * Starting BGSAVE for SYNC with target: disk using: normal sync
7:M * Disk-based full sync with compression: lz4

A plain diskless, uncompressed save served the 8.1 replica, then a second, disk-based, LZ4-compressed save ran immediately after to serve the 9.2 one. A fleet with a mix of old and new replicas pays for a full extra fork-and-save cycle per sync generation rather than sharing one, which matters on a large dataset where a single BGSAVE already costs real CPU and I/O.

Watching it from outside

I expected INFO persistence or INFO replication to expose which algorithm was actually used, the way rdb_last_bgsave_status or master_link_status do. Neither field exists. I checked both sections after every save and sync in this post and found nothing: no rdb_compression, no repl_compression_algo, nothing under INFO ALL either. The only record of which compression ran is the primary's own log line, and that line only appears for a replica that negotiated compression — a plaintext fallback sync produces a log entry identical to one with no compression configured at all. If you want to confirm in production whether a given replica actually got a compressed sync, you grep the primary's log for algo= at the moment that replica attached. There's no way to ask the running server after the fact.

Run it yourself

This needs Docker and roughly 400MB of free disk for the images and test data.

docker pull valkey/valkey:9.2.0-rc1
docker run -d --name vk -v "$(pwd)/vk-data:/data" valkey/valkey:9.2.0-rc1 --save ""

# load ~1M small JSON-shaped values
python3 - <<'EOF'
import subprocess, random
def feed(cmds):
    subprocess.run(['docker', 'exec', '-i', 'vk', 'valkey-cli', '--pipe'],
                    input=cmds.encode(), capture_output=True)
batch = []
for i in range(1_000_000):
    val = '{"id":%d,"name":"user%d","active":true,"roles":["user","member"]}' % (i, i)
    batch.append(f"SET user:{i} '{val}'")
    if len(batch) >= 200_000:
        feed('\n'.join(batch) + '\n'); batch = []
if batch:
    feed('\n'.join(batch) + '\n')
EOF

for c in no yes lz4; do
  docker exec vk valkey-cli config set rdbcompression "$c" > /dev/null
  docker exec vk rm -f /data/dump.rdb
  time docker exec vk valkey-cli save > /dev/null
  echo "$c: $(docker exec vk stat -c%s /data/dump.rdb) bytes"
done

# replication needs a second setting, not just rdbcompression:
docker exec vk valkey-cli config set repl-compression lz4

If you run this on a dataset you actually care about, check your own ratio before assuming it'll be 80%. The size I got is a property of how repetitive your values are across records, not a fixed number this version of Valkey guarantees.

What to do with this

If you're on Valkey and most of your values are small, structured and repeat a lot of their shape across keys — session hashes, user profiles, event records — rdbcompression lz4 is worth turning on for the RDB file alone; the CPU cost over the existing default is small next to an 80% size cut on disk and in backups. If you also replicate, set repl-compression lz4 on every primary explicitly and don't assume the RDB setting covers it — check the primary's log for algo=lz4 on the next full sync rather than trusting the config to have taken effect. And if your values are already high-entropy, measure your own data before turning anything on, because the win there was close to nothing in my tests.

Caddy 2.11.6 turns a silent config failure on /load into an HTTP 400

Alex Georgiev — Sat, 24 Oct 2026 10:00:00 +0000

I posted a broken load-balancer config to Caddy's admin API and it told me the reload had worked. The server behind it stopped answering requests entirely. The API still said 200.

That was on Caddy 2.11.4. I wanted to know whether the two patch releases that followed it, 2.11.6 (1 October) and 2.11.7 (3 October), actually closed that gap, because their changelogs claim "stricter config validation" and a fix for a Unix socket that used to hang on reload. Both claims turned out to be true, and more precise than I expected on the first one and more dramatic than I expected on the second.

The config that lied about succeeding

Caddy's admin API has a /load endpoint: POST a JSON config, it replaces the running one. I sent this, which gives a weighted_round_robin policy a negative weight:

"load_balancing": {
  "selection_policy": {
    "policy": "weighted_round_robin",
    "weights": [1, -3, 2]
  }
}

On Caddy 2.11.4:

$ curl -s -o /tmp/out.json -w 'HTTP %{http_code}\n' -X POST \
    -H 'Content-Type: application/json' --data @bad_weighted.json \
    http://localhost:2019/load
HTTP 200

Then I curled the server's actual listener on port 8080:

$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://localhost:8080/
curl: (52) Empty reply from server

The admin API said the new config was live. The thing it was supposedly serving had stopped responding. On Caddy 2.11.6 and 2.11.7, the same POST gets this instead:

HTTP 400
{"error":"loading config: loading new config: loading http app module: provision http:
server srv0: ... http.reverse_proxy.selection_policies.weighted_round_robin:
invalid configuration: weighted_round_robin requires at least one upstream with a
positive weight"}

And the server that was already running kept running. I reloaded a known-good config on 2.11.7, confirmed it was serving, sent the broken one, got the 400, and checked again:

Step	2.11.4	2.11.7
POST bad config to `/load`	`200`	`400` with error body
Live traffic afterwards	empty reply, server down	unaffected, still serving old config

If you have anything automated that pushes Caddy config over this endpoint and only checks for a 2xx, on 2.11.4 and earlier that check was lying to you.

Caddy logs every admin API request, which is the other way to catch this if you're watching logs rather than status codes. On 2.11.4, the log line for the bad load reads:

{"logger":"admin.api","msg":"load complete"}

On 2.11.7, the same POST produces:

{"level":"error","logger":"admin.api","msg":"request error","error":"... weighted_round_robin:
invalid configuration: weighted_round_robin requires at least one upstream with a positive
weight","status_code":400}

"Load complete" is exactly what it sounds like: as far as 2.11.4 is concerned, nothing went wrong. A log-based alert watching for admin.api errors would have caught this on the newer version and found nothing at all to catch on the older one, same as checking the status code would.

It's a specific list, not a general re-check

The changelog names several newly-rejected categories, and I tested two more of them directly, because I wanted to know whether "stricter" meant "all of it" or "this list, no more."

A Caddyfile with a duplicate uri inside forward_auth:

forward_auth 127.0.0.1:9099 {
    uri /auth1
    uri /auth2
}

On 2.11.4, caddy adapt accepts this with no warning and silently keeps the second value. On 2.11.7:

$ caddy adapt --config Caddyfile.forwardauth
{"level":"error","msg":"parsing caddyfile tokens for 'forward_auth': cannot re-declare uri: /auth2, at /Caddyfile.forwardauth:4"}

Exit code 1. A map handler with a destination placeholder missing its closing brace behaved the same way: 200 on 2.11.4, and on 2.11.7:

{"error":"... http.handlers.map: destination 0 must be a placeholder and only a
placeholder, but got '{http.vars.unterminated'"}

What I got wrong on the way

I assumed "stricter validation" meant any previously-accepted mistake would now be caught, so I tried two more cases expecting the same result and got surprised twice.

First, duplicate @id values on two routes. I expected this to be a new catch. It wasn't — Caddy 2.11.4 already rejects it:

{"error":"loading config: indexing config: duplicate ID 'r1' found at
/config/apps/http/servers/srv0/routes/0 and /config/apps/http/servers/srv0/routes/1"}

Same error, same HTTP 400, on both versions. Second, a file_limit set to 3.5 where the schema wants an integer. Also already rejected on 2.11.4, by Go's own JSON decoder rather than anything Caddy added:

{"error":"... json: cannot unmarshal number 3.5 into Go struct field
Browse.browse.file_limit of type int"}

So the fix is a named list of specific gaps — duplicate named_routes, duplicate forward_auth subdirectives, bad weighted_round_robin weights, ambiguous map inputs, and a handful of others — not a blanket pass over every handler. Worth knowing before you assume a config mistake you haven't tested is now covered.

I also checked a boundary case on the weight fix, because "stricter" sometimes means "stricter than it should be." A weight of 0, as opposed to negative, is accepted on 2.11.7:

"weights": [1, 0, 2]

returns 200. The rule is "at least one positive weight," not "no zeroes," so taking an upstream fully out of rotation with a zero weight still works.

The Unix socket that took two minutes to let go

This is the one I didn't expect to be able to measure cleanly, and it was the most striking result of the day.

The 2.11.7 changelog says a reload that moves a server's listener off a Unix socket used to leave the old socket file in place and leave clients connecting to it hanging for "about two minutes," fixed by closing it immediately. I set up Caddy 2.11.6 listening on unix//tmp/sock_a, confirmed it answered, then POSTed a new config that moves the same server to unix//tmp/sock_b:

$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' -X POST --data @b2.json \
    http://localhost:2019/load
HTTP 200

Then I polled the old socket path every half second or so. For roughly the first 100 seconds, every attempt timed out rather than failing fast:

curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

Around the 110-second mark it flipped to a connection reset, and by about 120 seconds it was a clean, instant refusal. The socket file itself (/tmp/sock_a) was still sitting on disk the entire time — ls kept finding it right up until the connections started failing fast. That is close enough to the changelog's "about two minutes" that I don't think I got lucky with the timing; it's a real, repeatable window.

On Caddy 2.11.7, running the identical reload:

$ ls /tmp/sock_a
ls: cannot access '/tmp/sock_a': No such file or directory
$ curl -s --unix-socket /tmp/sock_a http://localhost/
curl: (7) Failed to connect to localhost port 80 after 0 ms: Couldn't connect to server

The file was gone and the refusal was instant, checked immediately after the /load call returned and ten more times over the next three seconds. No timeout anywhere in that window.

	2.11.6	2.11.7
Old socket file after reload	still present	removed immediately
Connecting to old socket	hangs ~100-120s, then reset/refused	refused instantly, 0ms

If you reload Caddy in a loop as part of a deploy pipeline and something downstream reconnects on a fixed interval, the old version could leave that something stuck for two minutes against a socket nobody is serving on any more. That is a real production cost, not a cosmetic log message.

Run it yourself

This needs Docker and about ten minutes once the images are pulled.

docker pull caddy:2.11.4 && docker pull caddy:2.11.7

cat > good.json <<'EOF'
{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"routes":[
  {"handle":[{"handler":"static_response","body":"hello-good"}]}
]}}}}}
EOF

# Deliberately invalid: a negative weight
cat > bad_weighted.json <<'EOF'
{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"routes":[{"handle":[{
  "handler":"reverse_proxy",
  "load_balancing":{"selection_policy":{"policy":"weighted_round_robin","weights":[1,-3,2]}},
  "upstreams":[{"dial":"127.0.0.1:9001"},{"dial":"127.0.0.1:9002"},{"dial":"127.0.0.1:9003"}]
}]}]}}}}}
EOF

for v in 2.11.4 2.11.7; do
  docker run -d --name caddy-$v caddy:$v caddy run --resume=false >/dev/null
  sleep 1
  docker cp good.json caddy-$v:/good.json
  docker cp bad_weighted.json caddy-$v:/bad_weighted.json
  echo "== $v =="
  docker exec caddy-$v curl -s -o /dev/null -w 'load good: HTTP %{http_code}\n' \
    -X POST -H 'Content-Type: application/json' --data @/good.json http://localhost:2019/load
  docker exec caddy-$v curl -s -o /dev/null -w 'traffic after good: HTTP %{http_code}\n' \
    http://localhost:8080/
  docker exec caddy-$v curl -s -o /dev/null -w 'load bad: HTTP %{http_code}\n' \
    -X POST -H 'Content-Type: application/json' --data @/bad_weighted.json http://localhost:2019/load
  docker exec caddy-$v curl -s -o /dev/null -w 'traffic after bad: HTTP %{http_code}\n' \
    http://localhost:8080/
  docker rm -f caddy-$v >/dev/null
  echo
done

This is the actual output from running it:

== 2.11.4 ==
load good: HTTP 200
traffic after good: HTTP 200
load bad: HTTP 200
traffic after bad: HTTP 000

== 2.11.7 ==
load good: HTTP 200
traffic after good: HTTP 200
load bad: HTTP 400
traffic after bad: HTTP 200

On 2.11.4, the bad config reports 200 and the server stops answering (000, connection failure). On 2.11.7, the bad config is refused with 400 and the server carries on serving hello-good exactly as before. The /load response alone tells you which version you're running, without needing a second request against the data path at all.

What to do with this

If anything you run pushes configuration to Caddy's admin API — a CI job, a control plane, a sidecar — and it treats a 2xx from /load as "the new config is live and correct," go check which version you're running. Before 2.11.6, that check could pass while the server underneath it was already broken. After 2.11.6, it still can't tell you about every possible mistake, only the specific ones listed in the changelog, so don't drop your own post-reload health check just because the API got stricter.

And if you reload Caddy on a schedule and something else in your infrastructure dials a Unix socket Caddy used to own, check your Caddy version before assuming a two-minute stall is your code's fault.

nginx 1.31.5's control API turns a failed reload into an HTTP 422 instead of a log line

Alex Georgiev — Fri, 23 Oct 2026 10:00:00 +0000

I bound nginx's new control socket to a plain TCP port, sent it a bare curl with no headers and no credentials from outside the container, and it handed back the live PIDs of both worker processes. A second curl, still with nothing attached, swapped the running configuration for a different one.

That's the control API, a feature nginx 1.31.5 added to the open-source build on 2 September 2026. It lets you reload nginx over HTTP instead of a kill -HUP or nginx -s reload, and get a structured answer back instead of a line in the error log. I tested it on nginx 1.31.8, the current mainline, which carries the same feature forward unchanged.

Finding it

You turn it on with a new command-line flag, not a config directive:

nginx -g 'daemon off;' -l unix:/tmp/nginx_control.sock

The official nginx:1.31.8 Docker image already ships the binary with --with-control-api compiled in, so no custom build was needed. Querying it from inside the container found the endpoint tree by walking it one GET at a time:

$ curl -s --unix-socket /tmp/nginx_control.sock http://localhost/
["1"]
$ curl -s --unix-socket /tmp/nginx_control.sock http://localhost/1/
["control","nginx"]
$ curl -s --unix-socket /tmp/nginx_control.sock http://localhost/1/control
["processes","config"]

GET /1/control/config returns the live config file as JSON, name and full content. PATCH /1/control/config with no body re-reads the file from disk and reloads, which is the one I cared about.

The headline: a reload failure you can parse

I broke the config on purpose — a typo, retrun instead of return — and reloaded it both ways.

$ curl -s -i --unix-socket /tmp/nginx_control.sock -X PATCH http://localhost/1/control/config
HTTP/1.1 422 Unprocessable Entity
Content-Type: application/json

{"logs":["2026/10/23 05:11:41 [emerg] 1#1: unknown directive \"retrun\" in /etc/nginx/nginx.conf:6\n"]}

$ nginx -s reload
2026/10/23 05:11:41 [emerg] 131#131: unknown directive "retrun" in /etc/nginx/nginx.conf:6
nginx: [emerg] unknown directive "retrun" in /etc/nginx/nginx.conf:6

Both refuse the bad config and keep the old one running — nginx has always done that part safely, and I confirmed the previous config was still being served after each attempt. The difference is what a calling script gets back. The signal route gives you a process exit code and a line that only exists in the error log, which means a deploy script either parses log text or just trusts the exit code and hopes. The control API gives you a 422 and the exact compiler message as a JSON array, in the response body, with nothing to tail.

The opposite reading: it isn't faster

Before I found the error-reporting difference I assumed the API route would at least be quicker, since it skips sending a signal and waiting for the master to notice it. Five reloads each, same container, alternating between two valid configs so every run was a genuine reload rather than a no-op:

Method	Trial 1	Trial 2	Trial 3	Trial 4	Trial 5
`PATCH /1/control/config`	102ms	91ms	90ms	101ms	102ms
`nginx -s reload`	83ms	78ms	85ms	96ms	100ms

Those numbers include docker exec's own overhead, so treat them as a comparison between the two methods rather than nginx's true reload latency. By that comparison, the signal route was if anything a touch quicker, never slower. The control API's benefit is entirely in what it tells you, not in how fast it gets there.

Under concurrency: in-flight requests didn't care either way

I proxied a request to a backend that sleeps three seconds before answering, started the request, and reloaded nginx one second in — once with the API, once with the signal, three trials each:

Trigger	Trial 1	Trial 2	Trial 3
API reload at 1s	3.005s, HTTP 200	3.004s, HTTP 200	3.004s, HTTP 200
Signal reload at 1s	3.003s, HTTP 200	3.004s, HTTP 200	3.003s, HTTP 200

No dropped connections, no difference from a reload-free baseline of the same request (3.012s). I was suspicious of this result at first, because "no difference" is the answer a broken test gives you by default. So I checked the obvious failure: did the reload actually happen during the request, not before or after it? The worker PIDs confirmed it — the old workers showed "exiting":true in /1/control/processes while the slow request was still in flight, and new PIDs appeared, so the drain was genuinely overlapping the request. nginx's graceful worker shutdown applies the same way regardless of which door you reload through, which is reassuring rather than interesting, but it's the kind of thing worth checking rather than assuming.

What it actually exposes

nginx's own release notes call binding this to a network port "strongly discouraged" and say the API carries no authentication of its own. I wanted the exact shape of that risk, not just the warning.

$ docker run -d -p 19000:9000 nginx:1.31.8 nginx -g 'daemon off;' -l 0.0.0.0:9000
$ curl -s http://127.0.0.1:19000/1/control/processes
[{"name":"worker process","pid":30,"exiting":false},{"name":"worker process","pid":31,"exiting":false}]
$ curl -s -X PATCH http://127.0.0.1:19000/1/control/config
{"logs":[]}

Both requests ran from the host, outside the container, with no Authorization header, no API key, nothing but a bare URL. The second one actually replaced the running configuration — I'd already copied a different config file into the container first, and the content being served changed immediately after. There's no login, no token, no IP allowlist built into the feature itself. Anyone who can reach that port can read live process state and push a complete config swap.

The documented mitigation is to bind the control socket to the filesystem instead and rely on normal file permissions. I checked that it actually works rather than taking it on faith. The default unix socket came up as srw-------, owner and group both root, world nothing:

$ ls -la /tmp/nginx_control.sock
srw------- 1 root root 0 Oct 23 05:11 /tmp/nginx_control.sock
$ docker exec --user nobody mycontainer curl -s --unix-socket /tmp/nginx_control.sock http://localhost/1/control/processes
(exit code 7 — couldn't connect)

As nobody, the connection was refused before it ever reached nginx. So the workaround is real: a unix socket with tight permissions genuinely blocks an unprivileged user on the same host, which is more than the API does for itself over a network interface, where it blocks nothing.

What it costs to leave open

I compared the master process's resident memory with and without the -l flag, same image, same otherwise-empty config:

Configuration	Master process VmRSS
No control socket	9,164 KB
Control socket open	9,468 KB

About 304KB, roughly 3.3%, just for keeping the listener alive with nothing connected to it. Small, and nobody's capacity planning is going to notice it, but it's a real, non-zero cost for a feature you might enable once and forget about.

What I got wrong on the way

My first attempt at testing the reload used a host-side bind mount for the config file, editing it with sed on the host and expecting the running container to see the change immediately, the way bind mounts normally work. It didn't: the container kept serving the old content no matter what I wrote to the host file, and I spent a while assuming the control API itself was silently failing to apply changes. It wasn't. The sandboxed Docker setup I was running in didn't propagate host-side edits to an already-mounted file the way a normal Docker host does. Switching to docker cp to push the new config file directly into the container, then triggering the reload, fixed it immediately and every measurement above uses that approach.

Run it yourself

docker pull nginx:1.31.8
docker run -d --name ngxtest -p 18080:8080 \
  nginx:1.31.8 nginx -g 'daemon off;' -l unix:/tmp/nginx_control.sock

# confirm the socket and walk the endpoint tree
docker exec ngxtest curl -s --unix-socket /tmp/nginx_control.sock http://localhost/1/control

# push a new config and reload through the API
cat > new.conf <<'EOF'
events {}
http {
    server {
        listen 8080;
        location / { return 200 "v2\n"; }
    }
}
EOF
docker cp new.conf ngxtest:/etc/nginx/nginx.conf
docker exec ngxtest curl -s -i --unix-socket /tmp/nginx_control.sock \
  -X PATCH http://localhost/1/control/config

curl -s http://127.0.0.1:18080/   # should print v2

I ran that exact sequence again right before publishing this, on a fresh container, and it produced v2.

What to do with this

If you already run nginx -s reload from a deploy script and have ever had it fail silently in a way you only noticed later, the control API is worth wiring in for the structured error alone — a 422 with the compiler's own message beats grepping an error log you may not even be shipping anywhere. Don't bind it to anything but a unix socket, and don't treat the socket's default permissions as optional — they're the only access control this feature has. If you're auditing a host that already runs nginx 1.31.5 or newer, check now whether anyone turned this on with -l pointed at a network address, because nothing in the feature itself will stop a request from working.

Git 2.56's last-modified command is fast on one directory and slow on scattered files

Alex Georgiev — Thu, 22 Oct 2026 10:00:00 +0000

I ran the same command on the same 40 files twice. The only thing I changed was which 40 files. The first run took 861 milliseconds. The second took 6.4 seconds, and lost to the thing it's supposed to replace.

Git 2.52.0 added git last-modified, a command that walks history once and reports, for every path you give it, the closest commit that touched it. It exists because the obvious alternative — running git log -1 -- <path> once per file — is what every repo browser's file listing has always done, and it's wasteful: each call re-walks history from scratch for one path. git last-modified is meant to do that walk once and answer for all the paths at the same time. It's still marked EXPERIMENTAL in its own manual page, and it picked up real bug fixes as late as 2.54, so I wanted to know what the current stable release, 2.56.0 (released 27 September 2026), actually delivers.

I tested against the git/git repository itself, which is also where the original performance discussion happened. I cloned it with --filter=blob:none (full commit and tree history, no file contents, since last-modified never needs blob bytes) and built Git 2.56.0 from source, because no package repository I could reach ships it yet. 82,542 commits, 4,857 tracked files, on a 4-core VM.

The headline number

The realistic case for this command is a repo browser listing one directory: here are these 40 files, show me who touched each one last. I ran it against git.git's top-level listing, three times each, fastest run reported:

Method	Time (40 files, one directory)
`git last-modified -- <40 files>`	861ms
per-file `git log -1` loop	2,981ms

That's 3.5x faster, and I checked the actual output against git log -1 for five of the files by hand — every commit hash matched. It isn't fast because it's wrong.

The opposite reading: same file count, different shape, different winner

The 40 files above all live in one directory, which means a lot of their history overlaps — the same release-prep commits, the same large refactors, touch several of them together. That overlap is exactly what lets a single history walk answer for all of them at once. So I built a second set of 40 files, picked at random from anywhere in the 4,857 tracked files, with no shared directory and mostly unrelated histories, and ran the identical comparison:

Method	Time (40 files, scattered across the tree)
`git last-modified -- <40 scattered files>`	6,402ms
per-file `git log -1` loop	5,793ms

Same number of files, same repository, same machine. last-modified is now the slower option. I pushed the scattered case further to see whether that gap was a fluke:

Scattered file count	`last-modified`	log loop
40	6.40s	5.79s
50	7.48s	6.59s
100	25.7s	12.2s

Going from 50 to 100 scattered files should roughly double the loop's time, and it does (6.59s to 12.2s). last-modified more than tripled over the same step (7.48s to 25.7s). I killed a 500-file run of the same shape after it had burned over four CPU-minutes without finishing — more evidence this isn't linear once the paths stop sharing history, not a timing fluke. Whatever bookkeeping the command does per tracked path seems to compound badly when the paths don't overlap, and the maintainer's own "about 6x faster" claim, which I could reproduce for a real directory listing, doesn't hold at all for a scattered set of the same size.

The practical rule: this command is for "show me the directory," not "show me these files I picked from across the repo." A GitHub- or Forgejo-style file browser is exactly the first case. A dashboard that pulls together files from unrelated parts of a codebase is the second, and the per-file loop will beat it there.

It also loses outright on a single file

The whole appeal of last-modified is spreading one history walk across many paths. Hand it exactly one path and there's nothing to spread, so the comparison just measures whatever setup the command does before it starts answering at all.

Method	Time (1 file, `COPYING`)
`git last-modified -- COPYING`	684ms (min of 5), up to 1,582ms
`git log -1 -- COPYING`	431ms (min of 5), up to 744ms

Five runs each, and last-modified never once beat git log -1 for this single file, even on its fastest attempt. Its spread was wider too, up to 1.58 seconds on a slow run against git log's worst of 744 milliseconds, which looks like the setup cost itself is a little unpredictable rather than just slower on average. If you're fetching the last-touch commit for one file, plain git log -1 is still the right tool, because there's no second path for the new command to amortize anything against.

A pathspec order bug that isn't in any changelog

The manual says the default (--max-depth=0) "shows all paths matching the pathspec without descending into subtrees." I gave it a directory and one of its own children as two separate pathspecs, in both orders:

$ git last-modified -- contrib
a9bd346d8b6570a950724d4196224a493b038243    contrib

$ git last-modified -- contrib/completion
1c1256682a54d6738880e11ed2edf4b36757ef6c    contrib/completion

$ git last-modified -- contrib contrib/completion
1c1256682a54d6738880e11ed2edf4b36757ef6c    contrib/completion
30cc6c91b618a26a355558d6ee6aadbe7a0d5d85    contrib/buildsystems
b837e5f4bc1577f267edfec5459babafc156d7ed    contrib/contacts
[... 15 more rows, one per immediate child of contrib ...]

$ git last-modified -- contrib/completion contrib
a9bd346d8b6570a950724d4196224a493b038243    contrib

Asked alone, each path gives exactly one row, matching git log -1 for the same path. Asked together with the parent listed first, contrib silently expands into all 18 of its immediate children and its own aggregate row disappears entirely — nothing told me it was recursing, and --max-depth=0 didn't stop it, because I'd already passed that explicitly. Reverse the order, child first, and it collapses correctly back to the single contrib row. Nothing in the 2.52 through 2.56 release notes mentions this, and I couldn't find it described anywhere. For a tool meant to back a file browser, where the pathspec list is built from whatever the user clicked on, this is the kind of bug that would show up as an intermittent, hard-to-reproduce wrong listing, because it depends on argument order.

One bug that is actually fixed

The 2.54 release notes claimed "giving git last-modified a tree (not a commit-ish) died an uncontrolled death, which has been corrected." I checked:

$ git last-modified "$(git rev-parse HEAD^{tree})" -- COPYING
error: revision argument '006b28c5d4599d10965d9e5bbe22450db8c24eca' is a tree, not a commit-ish

A clean, named error, not a crash. That one holds up.

What it costs, and how it behaves under load

Peak memory for both approaches landed close together — about 190MB for last-modified against 50 paths, and roughly the same for the single heaviest git log process in the loop. Both numbers are dominated by loading this repository's commit-graph and pack index, not by anything last-modified itself allocates, so there's no memory story here worth claiming either way.

Running four copies of the 40-file directory listing at once, simulating four people loading the same repo page simultaneously, finished in 1.3 seconds wall-clock on my 4-core box, against 3.64 seconds run one after another. It's a read-only command with nothing to lock, so it parallelises about as well as the core count allows, with no contention I could detect.

What I got wrong on the way

My first build failed outright: make died on http.c unable to find curl/curl.h, because I'd skipped libcurl4-openssl-dev on the assumption that a locally-run, locally-cloned binary wouldn't need network transport code at all. Git's Makefile doesn't separate "can clone over https" from "can do anything," so leaving NO_CURL unset means it always tries to compile the curl-backed transport. Adding NO_CURL=1 NO_EXPAT=1 to the build fixed it in under two minutes, and the resulting binary works fine for everything in this post, since none of it touches a remote.

Run it yourself

This needs Git built from source, since 2.56 isn't in most distributions' repositories yet:

git clone --filter=blob:none https://github.com/git/git.git git-source
cd git-source && git checkout v2.56.0
make -j"$(nproc)" NO_GETTEXT=1 NO_TCLTK=1 NO_PERL=1 NO_CURL=1 NO_EXPAT=1
./git --version   # git version 2.56.0

Then the comparison itself, against any repo with real history (git/git works well because it's large and public):

import subprocess, sys, time

GITBIN, REPO = "./git", "./git-source"

def run(cmd, reps=3):
    times = []
    for _ in range(reps):
        t0 = time.perf_counter()
        subprocess.run(cmd, cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        times.append(time.perf_counter() - t0)
    return times

paths = sys.argv[1:]
print("last-modified:", run([GITBIN, "last-modified", "--"] + paths))
print("log loop:", run(["sh", "-c",
    "for p in " + " ".join(paths) + "; do git log -1 --format=%H -- \"$p\"; done"]))

Run it with a real directory's file list first (git ls-tree HEAD --name-only inside some subdirectory), then with the same number of paths picked from across the tree, and compare. The gap flips.

If you're building a file browser or a dashboard on top of this command, test it against your own directory shapes before trusting the "faster" claim, and don't pass a parent directory alongside one of its own children until the ordering bug above is fixed — list each directory's contents as its own call instead.

Kubernetes 1.35's kubelet refuses to start on a cgroup v1 host by default

Alex Georgiev — Wed, 21 Oct 2026 10:00:00 +0000

I tried to bring up a kind cluster on this machine using the current stable Kubernetes, 1.37.0, and the control plane never came up. kubectl get nodes never returned anything, because the kubelet never got past its own startup checks. systemctl status kubelet inside the node container showed this:

Active: activating (auto-restart) (Result: exit-code) since Wed 2026-10-21 05:20:55 UTC; 959ms ago

Restarting every second, forever. The actual error, from the journal:

E1021 05:20:25.129684 1003 run.go:72] "command failed" err="failed to validate kubelet
configuration, error: kubelet is configured to not run on a host using cgroup v1.
cgroup v1 support is unsupported and will be removed in a future release,
path: &TypeMeta{Kind:,APIVersion:,}"

This machine still runs cgroup v1. Starting with Kubernetes 1.35, that alone is now fatal, by default, with no grace period.

Cgroup v1 is not some exotic setup. It's what you get on older kernels, older base images, and any host nobody has gone back to reconfigure since before cgroup v2 was the default. The failure mode here isn't "feature doesn't work as well as hoped." It's "kubelet will not run at all, and it will not tell you that until you're already mid-upgrade."

Confirming the boundary myself

The behaviour is in a kubelet field called FailCgroupV1, and Kubernetes' own blog, published two weeks before I ran this, says the default flipped to true in 1.35. I didn't want to take that on faith, so I pulled three kindest/node images and watched each one's kubelet directly with docker exec and journalctl, rather than waiting on the whole cluster to come up.

image	kubelet on a cgroup v1 host
`kindest/node:v1.34.0`	starts, logs one warning event, stays running
`kindest/node:v1.35.0`	exits immediately, restarts, exits again
`kindest/node:v1.37.0`	same as 1.35

On 1.34.0 the kubelet logged a single warning event and then just ran:

Message:cgroup v1 support is in maintenance mode, please migrate to cgroup v2
Type:Warning

On 1.35.0 and 1.37.0 it never got that far. Both produced the exact same fatal error above and both crash-looped indefinitely. The boundary is real, and it's exactly where the blog post says it is. I didn't test 1.36.1, the release in between, so I'm only reporting the boundary at 1.34 versus 1.35/1.37, not claiming anything about 1.36 specifically.

What the crash loop actually costs

I measured the 1.37.0 node's restart counter twice, thirty seconds apart, with systemctl show kubelet -p NRestarts:

check	elapsed since kubelet first ran	restarts
first	~60s	58
second	~90s	82

That's 24 restarts in 30 seconds, or a full crash-and-respawn roughly every 1.25 seconds. It doesn't back off and it doesn't stop; I left a 1.35.0 node running longer and it reached 137 restarts with no sign of slowing. The node sat at 9.35% CPU doing nothing useful, and its systemd journal had already grown to 8.0MB purely from one error message looping. A single misconfigured cgroup v1 node, left alone, will burn CPU and fill its own log forever rather than failing once and going quiet.

The documented fix stops the kubelet crash, nothing else

The blog's workaround is a kubelet config line:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false

I applied this to a fresh 1.37.0 node through a kind config patch. It worked, exactly as advertised, at the kubelet level:

$ docker exec c137wa-control-plane systemctl show kubelet -p NRestarts
NRestarts=0

Stable, no crash. But the blog's fix only covers the kubelet. On a real cluster you also run kubeadm init, and kubeadm runs its own, separate preflight check before it ever touches the kubelet config. I ran that phase by hand, outside of kind, to see it fire on its own:

$ kubeadm init phase preflight --config=/kind/kubeadm.conf
[preflight] Some fatal errors occurred:
    [ERROR SystemVerification]: cgroups v1 support is deprecated and will be removed
    in a future release. Please migrate to cgroups v2. To explicitly enable cgroups v1
    support for kubelet v1.35 or newer, you must set the kubelet configuration option
    'FailCgroupV1' to 'false'. You must also explicitly skip this validation. For more
    information, see https://git.k8s.io/enhancements/keps/sig-node/5573-remove-cgroup-v1

Read that last sentence again: kubeadm's own error message says you must also explicitly skip its validation, separately from the kubelet setting. The fix is two flags, not one, and the first one I tried, failCgroupV1: false alone, would have left a plain kubeadm init dead on preflight regardless of what the kubelet config said. Adding the second flag does exactly what the error promises:

$ kubeadm init phase preflight --config=/kind/kubeadm.conf --ignore-preflight-errors=SystemVerification
    [WARNING SystemVerification]: cgroups v1 support is deprecated and will be removed
    in a future release. ...

Same text, downgraded from a fatal error to a warning. That's the actual two-part workaround: failCgroupV1: false in the kubelet config, and --ignore-preflight-errors=SystemVerification on every kubeadm init, join, and upgrade. The blog post I checked this against only documents the first half.

The override has a quiet side effect

With failCgroupV1: false applied and the kubelet running cleanly, I still found this in the same startup log, a few lines after the config dump:

I1021 05:21:07.455963 278 server.go:682] "Warning: MemoryQoS feature only works with
cgroups v2 on Linux, but enabled with cgroups v1"

Memory QoS has been a stable kubelet feature for several releases. Keeping cgroup v1 alive with the override doesn't just let an old node limp along unchanged — it silently drops a scheduling feature that's supposed to be on by default, with nothing in the fatal-error path warning you about it. If you adopt the override as a stopgap because migrating every node to cgroup v2 isn't a today problem, you've also quietly turned off memory QoS on all of them, and the only way you'd notice is reading the full startup log rather than grepping it for "error".

kind hides half of this by default

While building the test above I noticed kind never actually runs the kubeadm preflight phase I quoted. The kubeadm config kind generates for every node includes this:

skipPhases:
- preflight

So if your only exposure to this change is through kind, you will never see the SystemVerification error or its warning at all — kind skips that check unconditionally, on every version, cgroup v1 or not. What you'll see instead is purely the kubelet's own internal validation killing the process a layer further down, which is the crash loop at the top of this post. The two checks exist independently in real Kubernetes, kubeadm's and kubelet's own, and testing through kind only exercises one of them.

What I got wrong on the way

My original plan was to demonstrate this against a fully working cluster: bring up kind, show kubectl get nodes going Ready, then break and fix it. I couldn't get that far. Every node I started, on every Kubernetes version including 1.34, failed to actually run its static pods, with containerd reporting:

OCI runtime create failed: runc create failed: unable to start container process:
can't get final child's PID from pipe: EOF

That's a sandboxing limitation of the nested container environment this ran in, not a Kubernetes bug, and it hit 1.34 exactly as hard as 1.37. Once I confirmed the failure was identical and version-independent, I stopped trying to get a live cluster and tested the kubelet and kubeadm directly with docker exec and systemctl/journalctl instead, which turned out to be the right layer for this question anyway, since the thing I was measuring happens entirely before any pod is ever scheduled.

Run it yourself

This needs docker and the kind and kubectl binaries; no cloud account.

curl -sL -o /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
chmod +x /usr/local/bin/kind

# default behaviour: crash loop on a cgroup v1 host
kind create cluster --name c137 --image kindest/node:v1.37.0 --retain || true
docker exec c137-control-plane systemctl show kubelet -p NRestarts
docker exec c137-control-plane journalctl -u kubelet --no-pager | grep "command failed" | tail -1
kind delete cluster --name c137

# the two-part workaround
cat > /tmp/kind-workaround.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: KubeletConfiguration
    failCgroupV1: false
EOF
kind create cluster --name c137wa --image kindest/node:v1.37.0 \
  --config /tmp/kind-workaround.yaml --retain || true
docker exec c137wa-control-plane systemctl show kubelet -p NRestarts
docker exec c137wa-control-plane journalctl -u kubelet --no-pager | grep -i memoryqos
kind delete cluster --name c137wa

Check grep cgroup /proc/filesystems and stat -fc %T /sys/fs/cgroup first; if that prints cgroup2fs you're already on v2 and none of this applies to you.

What to check

Run stat -fc %T /sys/fs/cgroup on every node you plan to run Kubernetes 1.35 or later on. If it says tmpfs rather than cgroup2fs, you're on cgroup v1, and an in-place upgrade to 1.35+ will not degrade gracefully, it will crash-loop every kubelet on every affected node the moment the new binary starts. That's not a staged rollout risk, it's an outage the instant the new kubelet binary lands, with no warning beforehand unless someone already read the 1.35 changelog.

Migrate to cgroup v2 before the upgrade if you can. If you genuinely can't yet, set both halves of the override, kubelet's failCgroupV1: false and kubeadm's --ignore-preflight-errors=SystemVerification, and then go looking in your kubelet logs for the Memory QoS warning, so you know, deliberately, what you gave up to keep the node alive, rather than finding out later from a scheduling decision that doesn't make sense. Cgroup v1's fallback path is on record as scheduled for removal in 1.38, two releases from now, so this buys you a reprieve measured in months, not a long-term plan. Put the migration on the calendar now, while the override still exists to buy you the time to do it properly.

Docker Engine 29.9.1 fixes a docker save bug that left deleted layers on disk

Alex Georgiev — Tue, 20 Oct 2026 10:00:00 +0000

I run docker save sometimes to move an image between two machines that don't share a registry. I never thought about what happens to the layers afterwards once I docker rmi the image. Docker Engine 29.9.1, released on 14 October, fixes a bug where the answer was: nothing. They stay on disk, forever, and no command can see them.

The fix is one line in the 29.9.1 release notes: "docker save no longer leaves layers on disk after image removal when an image repeats a layer." I pulled the moby pull request that made the fix (#53927) to see what "repeats a layer" meant, then built both the broken and fixed engine to check it myself.

The bug

The root cause is a reference count that never gets released. When docker save writes out an image's config and layers, it takes a reference on each layer so it isn't garbage collected mid-write. The code released that reference in a deferred call, but only after checking whether the layer's diff ID had already been saved once in the same run. If an image lists the same diff ID twice in its RootFS.Layers — two layers with byte-identical content — the second occurrence hit the "already saved" branch and returned before the deferred release ran. One layer walks away with a reference nobody ever drops.

That's a strange precondition — why would an image list the same layer twice? It happens if a Dockerfile copies the same file into the image at two different steps. The content is identical, so the diff tar it produces is identical, so the layer digest is identical. I built one on purpose:

FROM busybox:1.36.1
COPY f /f
COPY f /f

f was a few hundred bytes of text. docker inspect showed exactly what the bug needs:

["sha256:97e4ece8f8a4d01b66df2c55c1f1e8a2860597b54dffbbc0df2f9942062cb34d",
 "sha256:5a0a403383bddeef1a6e1d06106756361edc2cf35a15397ecdad4f2d10cc62d7",
 "sha256:5a0a403383bddeef1a6e1d06106756361edc2cf35a15397ecdad4f2d10cc62d7"]

The second and third layers are the same digest.

Reproducing it

I downloaded the static binaries for Docker Engine 29.8.2 (the version right before the fix) and 29.9.1 from download.docker.com, and ran each with its own --data-root so they couldn't interfere with each other. Both used the classic overlay2 graph driver, which the release notes specifically name as the affected store — I had to pass "features": {"containerd-snapshotter": false} in daemon.json to get that, since newer Docker defaults to the containerd image store.

First, a control run on 29.8.2 with no docker save involved at all:

$ docker rmi repeat-layer-test busybox:1.36.1
Untagged: repeat-layer-test:latest
Deleted: sha256:d4ec96e71ec1...
Deleted: sha256:93763116f792...
Deleted: sha256:7698c5712b92...
Deleted: sha256:35dc035c6e26...
Untagged: busybox:1.36.1
Deleted: sha256:b116e1550744...
Deleted: sha256:97e4ece8f8a4...

/var/lib/docker/image/overlay2/layerdb/sha256 went from 3 entries to 0. Clean removal, as expected.

Then the same build, this time with docker save in between:

$ docker save repeat-layer-test -o /tmp/repeat-layer-test.tar
$ docker rmi repeat-layer-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
3
$ du -sh /var/lib/docker/overlay2
4.5M    /var/lib/docker/overlay2

The tarball from docker save was 4.5MB. After removing both images, /var/lib/docker/overlay2 was still 4.5MB. Nothing was freed. The three layer directories just sat there, owned by nothing.

scenario (29.8.2, overlay2)	layerdb entries after rmi	disk left after rmi
no `docker save`	0	0
`docker save` then `docker rmi`	3	4.5MB

I ran the identical build, save, and remove sequence against 29.9.1 in its own data root:

$ docker save repeat-layer-test -o /tmp/repeat-layer-test-291.tar
$ docker rmi repeat-layer-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
0
$ du -sh /var/lib/docker/overlay2
8.0K    /var/lib/docker/overlay2

Zero entries, 8KB left, which is just empty directory structure. The fix works. The moby PR's own author noted they hadn't rerun the end-to-end reproduction against a patched daemon — as far as I can tell, this is that confirmation.

What it costs, and what doesn't show it

The part that bothered me more than the leak itself was that nothing surfaces it. On the broken 29.8.2 daemon, after the leak:

$ docker system df -v
Images space usage:

REPOSITORY   TAG   IMAGE ID   CREATED   SIZE   SHARED SIZE   UNIQUE SIZE   CONTAINERS

Containers space usage:
...
Build cache usage: 0B

Zero images, zero containers, zero build cache. The 4.5MB sitting in overlay2 doesn't appear in any column. I tried both prune commands docker ships:

$ docker system prune -af
Total reclaimed space: 0B
$ docker builder prune -af
Total:  0B

Neither sees it, because both work from the same image and layer records that no longer reference the orphaned layers. I restarted the daemon to rule out an in-memory cache masking the problem; the 4.5MB and the three directories were still there afterwards. On an affected engine version, there's no documented command that gets the space back. The only fix is the upgrade.

The bug is specific to one storage backend

The release notes say this only affects "the graph-driver image store," which on current Docker means you have to be on an upgraded host still running overlay2 rather than the containerd image store that's been the default for fresh installs since Docker Engine 29.0. I checked this rather than taking it on faith, running the same build-save-remove sequence on 29.8.2 with the containerd image store enabled instead:

$ docker system df -v
REPOSITORY          TAG       IMAGE ID       SIZE      SHARED SIZE   UNIQUE SIZE
repeat-layer-test    latest    1e9550303c9d   6.77MB    6.757MB       9.87kB
busybox              1.36.1    73aaf090f3d8   6.76MB    6.757MB       3.821kB

$ docker rmi repeat-layer-test busybox:1.36.1
$ docker system df -v
(nothing listed)
$ du -sh /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs
56K

56KB of directory overhead, no leak. The claim held up: a fresh Docker install, or any host that's already switched to the containerd store, was never exposed to this regardless of engine version. It's specifically upgraded hosts that stayed on the legacy driver — which, going by Docker's own migration notes about upgraded hosts keeping overlay2 by default, is still a lot of production fleets.

The leak doesn't multiply with more repeats

My next guess was that an image with more repeated layers would leak proportionally more. I built a Dockerfile with the same file copied five times instead of two, saved it, and removed it on the broken engine:

$ docker save repeat5-test -o /tmp/repeat5.tar
$ du -h /tmp/repeat5.tar
4.5M    /tmp/repeat5.tar
$ docker rmi repeat5-test busybox:1.36.1
$ ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l
6
$ du -sh /var/lib/docker/overlay2
4.5M    /var/lib/docker/overlay2

The tarball was the same 4.5MB as the two-copy version, and the disk usage after the leak was the same 4.5MB too. The layerdb entry count went from 3 to 6, because overlay2 tracks a distinct chain ID for each position in the layer stack even when the underlying content is identical, but the actual layer bytes are stored once and shared across those positions. The leak is bounded to the size of the unique data, not multiplied by how many times it's repeated. That's the one number here that argues against the headline: this bug is real and the fix matters, but it isn't the kind of leak that gets worse the more you push it. A one-off accidental duplicate costs you one layer's worth of space, not five.

What I got wrong on the way

My first attempt to run 29.8.2 with the classic graph driver just added --storage-driver overlay2 to the dockerd command line, which failed immediately:

failed to start daemon: configured driver "overlay2" not available: unavailable

I'd assumed the flag alone would turn off the containerd image store. It doesn't — Docker 29 defaults to containerd snapshotters regardless of --storage-driver, and I had to explicitly set "features": {"containerd-snapshotter": false} in a daemon.json passed via --config-file before the classic driver would start at all. Small thing, but it meant my first "reproduction" of the bug was actually running against the containerd store the whole time, and obviously showed no leak.

I also expected three RUN true instructions to produce three identical empty layers and hit the same bug through a more ordinary-looking Dockerfile. They didn't — the legacy builder recognised the empty diffs and didn't create layers for them at all, so docker inspect showed only the base image's single layer. That path doesn't reproduce this bug; you need an actual non-empty duplicate, such as a repeated COPY.

Run it yourself

This needs two Docker Engine builds. Static binaries are at download.docker.com; no package manager install is required.

# get both engine versions
mkdir -p /opt/docker-282 /opt/docker-291
curl -sL -o /tmp/d282.tgz https://download.docker.com/linux/static/stable/x86_64/docker-29.8.2.tgz
tar -xzf /tmp/d282.tgz -C /opt/docker-282 --strip-components=1

# force the classic graph driver (containerd snapshotter is the default otherwise)
echo '{"features": {"containerd-snapshotter": false}}' > /tmp/daemon.json
mkdir -p /data/docker-282
/opt/docker-282/dockerd --data-root /data/docker-282 \
  --host unix:///var/run/docker.sock \
  --config-file /tmp/daemon.json --storage-driver overlay2 &

# build an image with a genuinely repeated layer
mkdir /tmp/ctx && cd /tmp/ctx
echo hello > f
printf 'FROM busybox:1.36.1\nCOPY f /f\nCOPY f /f\n' > Dockerfile
DOCKER_BUILDKIT=0 /opt/docker-282/docker build -t repeat-test .

# save, then remove, then check what's left
/opt/docker-282/docker save repeat-test -o /tmp/repeat.tar
/opt/docker-282/docker rmi repeat-test busybox:1.36.1
ls /data/docker-282/image/overlay2/layerdb/sha256 | wc -l   # 3 on 29.8.2, 0 on 29.9.1
du -sh /data/docker-282/overlay2

Swap /opt/docker-282 for a 29.9.1 build with a separate --data-root to see the difference.

What to do about it

Run docker info | grep "Storage Driver" on whatever host actually runs your docker save step. Mine said overlayfs with driver-type: io.containerd.snapshotter.v1 once I enabled the newer store, and plain overlay2 on the legacy one — that one word is the whole difference between leaking 4.5MB per affected image and leaking nothing.

The hosts most likely to still show overlay2 are the ones nobody has touched the storage config on since before Docker Engine 29.0, because that's what upgrading in place leaves you with. If that's what you find, and a backup or image-migration script on that box calls docker save, the 4.5MB I measured here won't show up in docker system df, won't show up in docker system prune -af, and won't go away on its own. I only got rid of mine by installing 29.9.1. There isn't a second option, and I looked.

Redis 8.4's digest-based SET option cuts a 1MB compare-and-set from 4.3ms to 1.6ms

Alex Georgiev — Mon, 19 Oct 2026 10:00:00 +0000

I keep a cached object in Redis that several workers read and occasionally rewrite. Before writing, each worker has to check that nobody else changed the object since it was last read, or the write clobbers someone else's update. Redis never had one command for that. You used WATCH/MULTI/EXEC, or a Lua script that reads and compares server-side, or you accepted the race and moved on.

Redis 8.4 adds four conditions to SET: IFEQ, IFNE, IFDEQ and IFDNE. The first two compare against a literal value. The last two compare against a digest, a 16-character hex hash the server computes with DIGEST. I pulled redis:8.4 and ran all four against the patterns they replace.

The headline number

The point of a digest is that you don't send the value you're comparing against, you send its hash. For a small string that saves nothing. For a large one it matters.

I set a 1MB string, kept a cached digest of it client-side, and timed three ways of doing a conditional overwrite 50 times each, taking the median:

size	classic GET-then-SET	Lua script CAS	`SET ... IFDEQ`
100 B	0.29 ms	0.17 ms	0.21 ms
10 KB	0.32 ms	0.25 ms	0.17 ms
100 KB	0.55 ms	1.10 ms	0.48 ms
1 MB	4.44 ms	7.37 ms	1.77 ms

Those are medians across three separate runs of 50 trials each at every size, against a redis:8.4 container on localhost. At 1MB, the digest-based set was roughly a third of the classic GET-then-SET and under a quarter of the Lua script, while the data crossing the wire for the comparison itself dropped from 1,000,000 bytes to 16.

The classic approach does a GET (full value back to the client) followed by a conditional SET (full value back to the server). The Lua script sends the full old value as an argument so the server can compare it internally. The digest option sends neither; the client already has the digest from the last time it fetched the object, and this SET just needs to check it still matches.

The number that argues against it

At 100 bytes to 10KB, none of the three were reliably faster than the others. The differences at that end were inside normal jitter between runs — sometimes the digest set won, sometimes the classic one did, by fractions of a millisecond either way. The benefit only shows up once the payload is large enough that transfer time, not command overhead, dominates the round trip.

More surprising: the Lua script was the slowest option at every single size I tested, including 100 bytes. I expected it to win, since it does the comparison server-side in one round trip, the same as IFEQ does natively. It didn't. EVAL carries script-lookup overhead that a plain SET with a condition flag doesn't, and that overhead outweighed the round-trip savings until the value got large, and even then it never caught up to the native digest option.

Separately, DIGEST itself isn't free. Its own cost scales with the value, because it has to read every byte to hash it:

value size	`DIGEST` latency (median)
100 B	0.17 ms
10 KB	0.15 ms
1 MB	1.03 ms
10 MB	6.20 ms

So the win from IFDEQ is specifically a network-bytes win, not a CPU-avoidance win. If you have to call DIGEST fresh every time rather than reusing a digest you already hold from an earlier read, you've moved the cost from the wire to the server's CPU, not eliminated it.

What it refuses

A handful of error cases, with the exact messages:

127.0.0.1:6379> DEL mylist
127.0.0.1:6379> RPUSH mylist a b c
127.0.0.1:6379> SET mylist v IFEQ a
(error) WRONGTYPE Operation against a key holding the wrong kind of value

127.0.0.1:6379> SET anotherkey v NX IFEQ x
(error) ERR syntax error

NX and IFEQ are mutually exclusive, which makes sense once you think about it: NX means "only if the key doesn't exist", IFEQ means "only if it holds a specific value", and a key can't satisfy both at once except trivially.

A more useful refusal: IFEQ and IFDEQ never succeed against a key that doesn't exist, even though you might reasonably expect "the value matches what I think it should be" to include "there is no value yet". I deleted a key, then ran SET nosuch v IFEQ anything against it, and got a plain (nil) — no error, just a silent no-op, same as a normal failed condition. If your code checks IFEQ on a key that might legitimately not exist yet, you need NX as well, or a separate existence check; IFEQ alone won't create it.

And a digest with the wrong shape is a hard error, not a silent failure:

127.0.0.1:6379> SET cs1 c IFDEQ zz
(error) ERR must be exactly 16 hexadecimal characters

Under concurrency

The whole point of these flags is optimistic concurrency control, so I tested it directly. Twenty threads, each doing fifty read-increment-write cycles against the same ~950-byte JSON blob with a counter field inside it. Expected final count: 1,000.

Plain GET then unconditional SET, no retry:

naive GET+SET   trial 1: final=109  lost_updates=891
naive GET+SET   trial 2: final=105  lost_updates=895
naive GET+SET   trial 3: final=105  lost_updates=895

Roughly 89 to 90 percent of increments vanished every single run. That's not a surprising result in itself — it's the textbook lost-update problem — but the size of it, with twenty threads on commodity hardware, was bigger than I expected going in.

Same workload, SET ... IFEQ <old-blob> in a retry loop instead of an unconditional SET:

SET...IFEQ loop trial 1: final=1000  lost_updates=0  total_retries=4892
SET...IFEQ loop trial 2: final=1000  lost_updates=0  total_retries=5111
SET...IFEQ loop trial 3: final=1000  lost_updates=0  total_retries=5111

Zero lost updates every run, which backs up Redis's own description of these flags as enabling "optimistic concurrency control patterns". But correctness wasn't free: getting 1,000 successful writes took on the order of 5,000 extra attempts, roughly five tries per success, because twenty threads hammering the same key collide constantly. IFEQ fixes correctness. It doesn't fix contention, and under heavy contention on one hot key, you pay for that in retries, not in lost data.

What I got wrong on the way

I tried the new option through redis-py's low-level execute_command escape hatch first, the same way I'd call any command the client doesn't have a typed wrapper for yet:

r.execute_command('SET', 'gtest4', 'attempted-new', 'IFEQ', 'wrong-guess', 'GET')
# -> False

I read that False as "the server doesn't return the old value when the condition fails, only when it succeeds", and nearly wrote that up as a limitation. It isn't true. I opened a raw socket and sent the same command by hand:

$ python3 -c "..."
failed-condition with GET, raw reply: b'$8\r\noriginal\r\n'

The server was sending the old value back correctly the whole time. redis-py's execute_command path has a response-type table for known commands, and it doesn't yet know that SET with IFEQ and GET together returns a bulk string on a failed condition, so it falls back to treating the reply as a plain boolean and throws the actual value away. redis-py 8.1.0 does have a proper typed wrapper for this — r.set(key, value, ifeq=..., get=True) — and that one returns the real old value correctly. It's marked experimental and prints a UserWarning on every call, which is a fair trade for a command three months old. The bug is specifically in the generic low-level path, which gives no warning at all and just quietly hands you the wrong answer. If you're reaching for execute_command because your client doesn't have the new kwargs yet, check the actual reply type before you trust it.

Watching it in production

INFO commandstats tracks these like any other command, including failed conditional writes:

$ redis-cli INFO commandstats | grep -E "cmdstat_(set|digest)"
cmdstat_set:calls=3,usec=38,usec_per_call=12.67,rejected_calls=0,failed_calls=1
cmdstat_digest:calls=1,usec=13,usec_per_call=13.00,rejected_calls=0,failed_calls=0

failed_calls on set rises every time an IFEQ/IFNE/IFDEQ/IFDNE condition doesn't match. That's a cheap way to watch contention on a hot key in production without instrumenting your own retry loop: if that counter is climbing fast relative to successful set calls, you have the same problem my twenty threads had.

Run it yourself

docker run -d --name redistest -p 16379:6379 redis:8.4
pip install redis

python3 - <<'EOF'
import redis
r = redis.Redis(host='localhost', port=16379, decode_responses=True)
r.set('k', 'v1')
print(r.set('k', 'v2', ifeq='v1'))      # True, condition matched
print(r.get('k'))                        # v2
digest = r.execute_command('DIGEST', 'k')
print(r.set('k', 'v3', ifdeq=digest))    # True, digest matched
print(r.get('k'))                        # v3
EOF

The bandwidth/latency comparison and the twenty-worker concurrency test are both plain Python scripts against that same container; the core of the concurrency test is this loop, run on twenty threads at once:

while True:
    blob = rr.get(key)
    c = parse_counter(blob)
    ok = rr.execute_command('SET', key, make_blob(c + 1), 'IFEQ', blob)
    if ok:
        break

If you're already doing optimistic locking against Redis with WATCH/MULTI/EXEC or a Lua script, IFEQ replaces the common case in one round trip and one command. If the values you're comparing are large and you already keep a digest of the last version you read, reach for IFDEQ instead of IFEQ and skip sending the payload twice. Check whichever client library you use against its raw reply before trusting the boolean it hands back on a brand-new command — the server was right in my case; the convenience wrapper around it was not.

SQLite 3.54's diskused command needs a build flag sqlite3_analyzer never required

Alex Georgiev — Sun, 18 Oct 2026 10:00:00 +0000

SQLite 3.54.0 shipped on 9 October, and the change list has one line that's easy to skim past: the sqlite3_analyzer utility is now deprecated, replaced by a diskused() SQL function built straight into the command-line shell as a new .diskused dot-command. I wanted to know what that actually buys you, and what it costs, so I built both versions from source and ran them against the same database.

sqlite3_analyzer has been the way to answer "which table is eating my disk space" for as long as I can remember using SQLite seriously. It's a separate program you download or build on top of the library, and it opens your database file cold, walks every b-tree page, and prints a report. .diskused does the identical analysis, but it runs inside the shell you already have open, using a virtual table (dbstat) the engine already exposes internally.

Same report, less than half the time

I built a 20MB test database — four tables, one index, 350,000 rows — and ran both tools against copies of the same file, five times each:

=== .diskused (new, 3.54) ===
0.028s  0.023s  0.022s  0.024s  0.029s

=== sqlite3_analyzer (old, deprecated) ===
0.056s  0.061s  0.058s  0.055s  0.076s

	fastest	mean of 5
`.diskused`	0.022s	0.025s
`sqlite3_analyzer`	0.055s	0.061s

That's roughly 2.4 times faster on the mean, and the gap holds at the fastest runs too. Part of that is architectural rather than algorithmic: sqlite3_analyzer is a 3.3MB standalone binary that has to start its own process and open its own connection to the file from scratch, where .diskused runs a SQL function against a connection the shell already has open.

The two reports agree on every number I checked — page counts per table, bytes of payload, unused bytes per page, b-tree depth — which is exactly what you'd want from a replacement. But .diskused's output is 242 lines and 13KB; sqlite3_analyzer's is 432 lines and 21KB for the same database. The new tool drops some of the older report's redundant subsections (separate "all indexes" and "all tables" rollups that .diskused folds into fewer blocks) without losing any of the figures I actually used.

It fails, and tells you nothing about it

diskused() depends on dbstat, a virtual table that SQLite only exposes when it's compiled with SQLITE_ENABLE_DBSTAT_VTAB. The official binaries on sqlite.org have that flag set, so if you download the prebuilt shell, none of this applies to you. But a lot of people don't use the prebuilt shell — they drop sqlite3.c into their own project and compile it with their own flags, which is the whole point of the amalgamation. I did exactly that, with no extra flags beyond -O2, and ran the new command against my test database:

$ ./sqlite3_nodbstat test.db ".diskused"
$ echo $?
0

No output. No error. Just a clean exit. My first assumption was that I'd broken something in my own minimal build — I went back and double-checked the compile line, assuming I'd missed a header. I hadn't. The command had run, found nothing it could use, and said nothing about it.

Calling the underlying SQL function directly, instead of the dot-command wrapper, finally showed the real cause:

$ ./sqlite3_nodbstat test.db "SELECT diskused('main');"
Error in 2nd command line argument: SQL parse error: no such table: dbstat

dbstat isn't there, because I didn't ask for it at compile time. For comparison, running .diskused against 3.53.2, the version before this command existed, gives you an actual error:

$ ./sqlite3-3.53.2 test.db ".diskused"
Error: unknown command or invalid arguments:  "diskused". Enter ".help" for help

So an unsupported command on an old version tells you plainly that it doesn't exist. A supported command on a new version, missing one dependency, tells you nothing at all and exits zero. If you build your own shell from the amalgamation and rely on .diskused in a script to catch a bloated table before it causes problems, that script will quietly do nothing on a build missing one flag, and report success while doing it.

Checking the 4% claim, and why wall-clock time couldn't

The release notes claim 3.54 is "about 4% faster" than 3.53 and "about 7.5% faster" than 3.51, measured as speedtest1 CPU-cycle counts under valgrind. I tried to check that first with ordinary wall-clock timing: a small C program against each version's sqlite3.c, inserting 300,000 rows, building an index, then running 100,000 indexed point lookups, three runs per version.

3.53.2: total 0.554s, 0.586s, 0.602s
3.54.0: total 0.563s, 0.600s, 0.635s

That's not 4% faster. If anything it points the other way. Before writing that down as a real finding, I distrusted the harness rather than the claim: a 4% difference on a workload with real disk I/O, page cache behaviour, and OS scheduling noise is well within the run-to-run jitter I was already seeing between repeats of the same binary. Wall-clock timing on a live filesystem was never going to resolve a few percent.

So I ran the identical workload under valgrind --tool=callgrind, which counts CPU instructions retired rather than elapsed time — the same kind of measurement the release notes describe:

3.53.2: 2,557,436,360 instructions (confirmed on a second run: 2,557,436,359)
3.54.0: 2,504,239,038 instructions (confirmed on a second run: 2,504,239,042)

That's a reduction of about 2.1%, consistent across repeats. The direction matches the documented claim; the size doesn't. My workload is dominated by writes and an index build, not the query-heavy mix speedtest1 actually runs, and the release notes say as much themselves ("results may vary depending on workload"). I can confirm 3.54 does less CPU work on a real workload. I can't confirm it does 4% less, because that number was never about my workload to begin with.

Under concurrency, nothing changed

I ran .diskused against a database with an open write transaction in another connection, once in WAL mode and once in the default rollback-journal mode, expecting the reader to either block or get SQLITE_BUSY in the second case. Neither happened. .diskused only needs a SHARED lock, and BEGIN IMMEDIATE on the writer only takes a RESERVED lock, which is compatible with any number of concurrent readers under SQLite's normal locking rules. This isn't a new behaviour diskused() introduced; it's SQLite doing what SQLite has always done. Worth checking, nothing to report.

What I got wrong on the way

I spent longer than I'd like to admit convinced that the silent, empty output from my no-dbstat build was a bug in how I'd compiled the shell, not a property of the tool. The fact that it exited 0 was the tell I ignored for too long — a crash doesn't give you a clean exit code. Once I stopped assuming my build was broken and started asking the SQL function directly what it thought was wrong, the actual cause (no such table: dbstat) came back in one line.

Run it yourself

This needs a C compiler and the usual build tools. It downloads and builds two SQLite versions from source, so expect it to take a few minutes.

mkdir sqlite-diskused-test && cd sqlite-diskused-test

curl -sS -o 354.tar.gz https://sqlite.org/2026/sqlite-autoconf-3540000.tar.gz
tar xzf 354.tar.gz && cd sqlite-autoconf-3540000
./configure CFLAGS="-O2 -DSQLITE_ENABLE_DBSTAT_VTAB=1"
make -j4
cd ..

# a build with no extra flags at all, to see the silent failure
cc -O2 -o sqlite3_nodbstat sqlite-autoconf-3540000/shell.c sqlite-autoconf-3540000/sqlite3.c -lpthread -ldl -lm

./sqlite-autoconf-3540000/sqlite3 test.db <<'EOF'
CREATE TABLE t(id INTEGER PRIMARY KEY, a INTEGER, b TEXT);
WITH RECURSIVE seq(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM seq WHERE x<300000)
INSERT INTO t(a,b) SELECT x%997, 'padding' FROM seq;
EOF

./sqlite-autoconf-3540000/sqlite3 test.db ".diskused"      # full report
./sqlite3_nodbstat test.db ".diskused"; echo "exit: $?"    # silent, exit 0
./sqlite3_nodbstat test.db "SELECT diskused('main');"      # the real error

What I'd do with this

If you ship your own SQLite build compiled from the amalgamation, check now whether SQLITE_ENABLE_DBSTAT_VTAB is in your flags, rather than finding out the day a disk-space script reports "nothing to analyse" on a database that's actually full of bloat. If you're still shelling out to a separate sqlite3_analyzer binary in a build pipeline or cron job, .diskused on 3.54 does the same job from inside a shell you already have running, in under half the time, and you can drop the extra binary from whatever you're currently downloading to get it.

Docker Compose 5.6's jobs type only starts through docker compose run

Alex Georgiev — Sat, 17 Oct 2026 10:00:00 +0000

Docker Compose 5.6.0 shipped on 2 October with a new top-level jobs: section, sitting next to services: in the compose file. I wanted to know what actually happens when you declare one, because the release notes say almost nothing beyond "partial support for jobs: only manually-triggered jobs are supported for now."

The first thing that happened was my compose file got rejected.

$ docker compose config
validating /home/user/compose-jobs-test/compose.yaml: jobs.migrate missing property 'triggers'

A jobs: entry is not just a service with a different label. It needs its own triggers: block, and compose will not even parse the file without one. Once I added triggers: {manual: true}, it validated and I could start testing what a job actually does differently from a service.

What a job refuses to do

I wrote a small compose file: a long-running db service, and a migrate job that depends on it, sleeps a second and exits 0.

services:
  db:
    image: alpine:3.20
    command: ["sh", "-c", "echo db-up; sleep 600"]

jobs:
  migrate:
    image: alpine:3.20
    depends_on:
      - db
    triggers:
      manual: true
    command: ["sh", "-c", "echo migrating; sleep 1; echo done; exit 0"]

docker compose up -d started db and said this about the job, then moved on:

time="...Z" level=warning msg="jobs are not started by up; trigger them with `docker compose run`: migrate"

Exit code 0. docker compose ps -a afterwards shows only db. The job is simply absent, not failed, not pending, not listed anywhere up reports on.

create and start are less forgiving. Both refuse outright:

$ docker compose create migrate
job "migrate" can only be triggered with "docker compose run"
$ echo $?
1

Same message, same exit code, for start. So of the five commands that normally bring a service into existence, four of them (up, create, start, and as far as I could find no fifth alternative) will not touch a job at all. Only run works.

The opposite reading: declaring the documented future breaks the present

The release notes are honest that scheduled jobs aren't supported yet, only manual ones. I wanted to see what "not supported yet" meant in practice, so I added a second job with a schedule trigger instead of manual:

jobs:
  nightly:
    image: alpine:3.20
    triggers:
      schedule: ["0 2 * * *"]
    command: ["sh", "-c", "echo nightly-run"]

This is where the asymmetry shows up. A manual-only job that up can't start gets a warning and is skipped; the rest of the stack comes up fine. A job with a schedule trigger makes up refuse to run at all:

$ docker compose up -d
scheduled jobs are not supported in this version: nightly
$ echo $?
1

Not a warning this time. The whole up fails, and nothing starts, not even the unrelated db service sitting in the same file. If you're migrating an existing stack to use jobs and you reach for the field that matches what you actually want (a job that runs on a schedule), you get a worse outcome than if you'd done nothing: the entire compose file stops coming up until you remove that trigger. Manual-only jobs are the only kind that currently coexist peacefully with the rest of your stack.

docker compose run --rm nightly still runs the scheduled job by hand without complaint, which matches what the Compose team's own notes say about manual execution being "always available... unless a job sets triggers.manual: false." I checked that half of the claim directly too: a job declared with triggers: {manual: false} is refused by run as well:

$ docker compose run --rm migrate_locked
job "migrate_locked" is declared with manual: false, it cannot be run manually
$ echo $?
1

Since nothing can trigger it on a schedule yet either, a job with manual: false in this release is not deferred to a scheduler. It's unreachable. There is currently no path in Compose 5.6.0 that starts it.

What a job costs you to run

docker compose run on a job starts its declared dependencies first, same as it does for a one-off command on a service. I measured how much that costs by timing docker compose run --rm migrate three times with db already running, and three times after docker compose down -v so db had to be created and started from scratch each time.

Run	Cold (db stopped first)	Warm (db already up)
1	1.716s	1.512s
2	1.718s	1.467s
3	1.717s	1.481s
mean	1.717s	1.487s

The job's own work (a one-second sleep plus container overhead) is identical in both cases, so the roughly 230ms gap, about 15%, is what it costs to create and start the dependency container before the job can begin. That's a small number on a single Alpine container; it's the number to budget for if your job's real dependency is a database image with its own startup routine, multiplied by however many depends_on entries the job declares.

Exit codes pass through exactly. A job that ends with exit 17 hands that 17 straight back to the shell that ran docker compose run:

$ docker compose run --rm migrate_fail
failing
$ echo $?
17

That matters if you're wiring a job into a CI step and checking $? afterwards, which is presumably the point of a dedicated one-off-task primitive.

Under concurrency

I ran the same job twice at once, both in the background, to see whether two manual triggers would collide:

$ docker compose run --rm migrate & docker compose run --rm migrate & wait
 Container compose-jobs-test-migrate-run-4fdc2a756110 Creating
 Container compose-jobs-test-migrate-run-f3445db266a6 Creating
migrating 1
migrating 1
done 1
done 1

Two distinct container names, both ran to completion, both cleaned up by their own --rm. No lock, no queue, no contention on the shared db dependency either, since it was already running. Jobs don't serialise against each other; if you need that, you still have to build it yourself.

A claim I checked and found true

The one documented behaviour I could directly verify was profile activation: a job's own profiles: entry is meant to be enabled automatically when you run it, without needing --profile on the command line. I gave a job profiles: ["migrations"], ran it with no --profile flag, and it ran:

$ docker compose run --rm migrate
migrated-with-profile
$ echo $?
0

That one held up exactly as documented.

What I got wrong on the way

My first attempt at the schedule trigger was triggers: schedule: "0 2 * * *", a bare string, because that's what cron usually looks like everywhere else. Compose rejected it: jobs.nightly.triggers.schedule must be a array. It wants a list of objects, each with a cron: key — schedule: [{cron: "0 2 * * *"}] once compose canonicalises it. I'd assumed the YAML shorthand I wrote would work because it reads naturally; the schema disagreed, and the error message was specific enough that I didn't need to go digging through compose-go source to fix it.

I also nearly reported "jobs leak containers that down can't clean up" as a jobs-specific finding, since a migrate container left over from a run without --rm survived both docker compose down and docker compose rm -f. Before writing that down as a cost of the new feature, I ran the identical test against an ordinary service behind a profiles: ["tools"] guard, the pattern people have used for one-off tasks in Compose for years. It leaked exactly the same way. Jobs didn't introduce this; they inherited it. It's worth knowing either way, but it would have been the wrong feature to blame.

Run it yourself

This needs Docker Compose 5.6.0 or later (docker compose version to check) and a working Docker daemon.

mkdir jobs-test && cd jobs-test
cat > compose.yaml <<'EOF'
services:
  db:
    image: alpine:3.20
    command: ["sh", "-c", "echo db-up; sleep 600"]

jobs:
  migrate:
    image: alpine:3.20
    depends_on:
      - db
    triggers:
      manual: true
    command: ["sh", "-c", "echo migrating; sleep 1; echo done; exit 0"]
EOF

docker compose up -d          # starts db, warns and skips migrate
docker compose create migrate # refused: "can only be triggered with docker compose run"
docker compose run --rm migrate   # this is the one that actually works
docker compose config --services  # lists db only, migrate is invisible here

Add a second job with triggers: {schedule: [{cron: "0 2 * * *"}]} and run docker compose up -d again to see the whole file refuse to start, rather than just skip the one job.

What I'd do with this

If you're reaching for jobs: today, use triggers: {manual: true} and only that. A schedule trigger doesn't give you a scheduled job yet, it gives you a compose file that up refuses to run at all, and manual: false with no scheduler gives you a job nothing can start. Keep using run --rm rather than bare run, because the orphaned-container problem is the same one you already had with profile-gated one-off services, and jobs don't fix it. And if any script you have loops over docker compose config --services to do per-service work, remember it won't see your jobs at all; you'll need to list them separately until Compose grows a command that does.

PostgreSQL 19 beta's parallel autovacuum launches workers but leaves vacuum time unchanged

Alex Georgiev — Fri, 16 Oct 2026 10:00:00 +0000

Autovacuum in PostgreSQL has never been able to split the work of cleaning up a table's indexes across more than one process. A manual VACUUM (PARALLEL n) could do it since PostgreSQL 13, but the background worker that actually runs on your production database at 3am could not. PostgreSQL 19 beta4 changes that, through a new autovacuum_max_parallel_workers setting and an accompanying scoring system that decides which bloated table gets attention first.

I built a table designed to make that matter, bloated it the same way three times over, and measured what parallel index vacuuming actually bought me. The answer, on this workload, was close to nothing on the clock that counts.

The setup

PostgreSQL 19 beta4, run from the official postgres:19beta4 Docker image. A table with six indexes and 400,000 rows, updated twice so that every indexed column changed on every row, leaving 800,000 dead tuples behind:

CREATE TABLE wide_bloat (
    id bigint PRIMARY KEY,
    c1 text, c2 text, c3 text, c4 int, c5 int, c6 timestamp, c7 uuid, payload text
) WITH (autovacuum_enabled = false);
CREATE INDEX idx_wb_c1 ON wide_bloat(c1);
CREATE INDEX idx_wb_c2 ON wide_bloat(c2);
CREATE INDEX idx_wb_c3 ON wide_bloat(c3);
CREATE INDEX idx_wb_c4 ON wide_bloat(c4);
CREATE INDEX idx_wb_c5 ON wide_bloat(c5);
CREATE INDEX idx_wb_c6 ON wide_bloat(c6);

I disabled autovacuum on the table while loading it, bloated it, then flipped autovacuum_enabled back on and let the real autovacuum worker pick it up, reading the outcome from log_autovacuum_min_duration = 0. I rebuilt the table fresh for every trial so each run started from the same 800,000 dead tuples.

The headline number, and why it undersells the result

With autovacuum_max_parallel_workers = 0 (the default — this feature does nothing unless you turn it on), three runs:

Run	Wall time	Leader CPU
1	1.24s	0.57s
2	1.36s	0.57s
3	1.32s	0.55s

With autovacuum_max_parallel_workers = 4, confirmed in the log by parallel workers: index vacuum: 4 planned, 4 launched in total:

Run	Wall time	Leader CPU
1	1.23s	0.42s
2	1.38s	0.40s
3	1.14s	0.38s

Wall-clock time averaged 1.31s serial versus 1.25s parallel — about 4%, well inside the run-to-run noise I was already seeing. The CPU time billed to the autovacuum worker itself dropped by a real margin, roughly 30%, because the index-scanning work for five of the six indexes got handed to launched workers instead of done by the leader alone.

That split matters. If you're watching autovacuum's CPU footprint on a busy database, parallel workers genuinely reduce what the worker process itself burns. If you're watching how long a table stays bloated before the dead tuples disappear, this table's bottleneck was the heap scan and the WAL it generated (official instrumentation logged 101,000+ WAL records and 70,000+ dirtied buffers on every run, serial or parallel), not the index cleanup phase that got parallelised. Autovacuum doesn't parallelise the heap scan. On a table where index cleanup actually dominates, I'd expect a different result; on this one it didn't.

What it costs

The same log line reports dead-tuple (TID) storage. Serial runs used 1.45MB. Parallel runs with 4 workers used 3.00MB — roughly double, for the same 800,000 dead tuples, because each launched worker gets its own share of that bookkeeping. On an 800,000-row vacuum this is nothing. On a multi-terabyte table with autovacuum_work_mem already stretched thin and several autovacuum workers each spawning their own parallel helpers, that multiplication is the real thing to budget for, not CPU.

The setting that doesn't do what I assumed

PostgreSQL has had max_parallel_maintenance_workers since version 11, controlling how many workers a manual VACUUM (PARALLEL) or CREATE INDEX can use. I assumed it would also cap autovacuum's new parallel workers. It doesn't.

I set max_parallel_maintenance_workers = 0 and autovacuum_max_parallel_workers = 4 with max_parallel_workers = 8, and autovacuum still logged parallel workers: index vacuum: 4 planned, 4 launched in total. The actual gate, per the PostgreSQL 19 documentation for autovacuum_max_parallel_workers, is this sentence: "The actual number of parallel workers is further limited by max_parallel_workers." Setting that one to 0 instead produced 4 planned, 0 launched, exactly as documented. max_parallel_maintenance_workers isn't mentioned in that page at all. If you're budgeting parallel workers across a cluster, the knob to turn is the global pool, not the maintenance-specific one that governs manual commands.

The scoring system, confirmed

Beta4's release notes mention "fixes for the new autovacuum scoring system" without saying what it does day to day. I built six tables with dead-tuple counts of 100, 500, 5,000, 40,000, 150,000 and 500,000, in that creation order, and enabled autovacuum on all six in the same transaction. A single autovacuum worker picked them up in this order, by timestamp in the log:

p_massive  (500,000 dead)  05:28:13.989
p_huge     (150,000 dead)  05:28:14.081
p_big      ( 40,000 dead)  05:28:14.122
p_med      (  5,000 dead)  05:28:14.141
p_small    (    500 dead)  05:28:14.147
p_tiny     (    100 dead)  05:28:14.149

Strictly descending by dead-tuple count, the exact reverse of creation order. The scoring system does prioritise the most urgent table first, at least for this one signal.

Under concurrency, two things went wrong at once

I capped max_parallel_workers at 4 globally, then started a separate session running a deliberately slow parallel query (pg_sleep() inside a WHERE clause, three workers, over a 1.5 million row table) to hold three of those four slots. With that session running, I re-bloated wide_bloat and enabled autovacuum.

Three consecutive autovacuum attempts over about 40 seconds each logged parallel workers: index cleanup: 4 planned, 0 launched — no slots free, correctly degrading to zero rather than failing. But the more useful finding was in the same log lines: tuples: 0 removed, 1200000 remain, 800000 are dead but not yet removable. The long-running session wasn't just holding parallel workers, it was holding a snapshot that kept those 800,000 dead tuples below the removable horizon. Nothing was deleted, parallel or not, for the full 40 seconds that session stayed open.

The moment I killed that session, the very next cycle logged 4 planned, 4 launched and cleared all 800,000 tuples in about a second. The lesson isn't really about parallel autovacuum: a long-running query anywhere on the database stalls vacuum regardless of how many workers you give it, and parallel autovacuum doesn't change that. It just means the workers you configured can also sit idle at the worst possible time, for the same old reason.

Watching it happen

pg_stat_progress_vacuum in PostgreSQL 19 reports live phase and index progress. Polling it every 300ms during a 1.5 million row autovacuum run on a three-index table caught this:

05:30:13.949 | scanning heap     | heap_blks_scanned=137335/142858 | indexes_processed=0/0
05:30:14.370 | vacuuming indexes | heap_blks_scanned=142858/142858 | indexes_processed=2/4
05:30:14.791 | vacuuming heap    | heap_blks_scanned=142858/142858 | indexes_processed=0/0

indexes_total and indexes_processed are enough to tell whether a slow vacuum is stuck scanning the heap or working through indexes, which is the question I actually want answered at 3am.

What I got wrong on the way

My first run of that test looked like a bug. I set max_parallel_maintenance_workers = 0 with ALTER SYSTEM, called pg_reload_conf(), got back applied: t from pg_file_settings — and SHOW max_parallel_maintenance_workers still said 8. I spent ten minutes convinced the setting wasn't taking effect at all, before I remembered why: I'd started that container with -c max_parallel_maintenance_workers=8 on the docker run line. A value set that way is fixed for as long as the server runs. ALTER SYSTEM writes to postgresql.auto.conf happily, the file even says it applied, but nothing reads it back over a command-line override. I had to kill the container and start a fresh one without that flag before the test told me anything real.

There's a second, smaller one worth naming: my first contention test used a global max_parallel_workers of 8 against a competing query holding only 3 workers, leaving 5 free for autovacuum's request of 4 — plenty of room, so nothing contended and I nearly wrote up "no effect" as a finding. Dropping the pool to 4 total is what actually forced the two requests to compete, and that's the version that produced something worth reporting.

Run it yourself

docker run -d --name pgq19 -e POSTGRES_PASSWORD=postgres -p 15432:5432 postgres:19beta4 \
  -c log_autovacuum_min_duration=0 -c autovacuum_naptime=5 \
  -c autovacuum_vacuum_threshold=50 -c autovacuum_vacuum_scale_factor=0.0 \
  -c autovacuum_vacuum_cost_delay=0 -c maintenance_work_mem=256MB \
  -c max_parallel_workers=8 -c autovacuum_worker_slots=16 -c autovacuum_max_workers=3

docker exec pgq19 psql -U postgres -c "
CREATE TABLE wide_bloat (id bigint PRIMARY KEY, c1 text, c2 text, c3 text, c4 int, c5 int, c6 timestamp)
  WITH (autovacuum_enabled = false);
CREATE INDEX idx_wb_c1 ON wide_bloat(c1); CREATE INDEX idx_wb_c2 ON wide_bloat(c2);
CREATE INDEX idx_wb_c3 ON wide_bloat(c3); CREATE INDEX idx_wb_c4 ON wide_bloat(c4);
CREATE INDEX idx_wb_c5 ON wide_bloat(c5); CREATE INDEX idx_wb_c6 ON wide_bloat(c6);
INSERT INTO wide_bloat SELECT g, 'v'||(g%5000), 'c'||(g%300), md5(g::text), g%1000, g%97, now()
  FROM generate_series(1,400000) g;
UPDATE wide_bloat SET c1='v2_'||(id%5000), c2='c2_'||(id%300), c3=md5((id+1)::text);
UPDATE wide_bloat SET c1='v3_'||(id%5000), c2='c3_'||(id%300), c3=md5((id+2)::text);
"

# serial baseline — ALTER SYSTEM must be its own statement, not chained with ;
docker exec pgq19 psql -U postgres -c "ALTER SYSTEM SET autovacuum_max_parallel_workers = 0;"
docker exec pgq19 psql -U postgres -c "SELECT pg_reload_conf();"
docker exec pgq19 psql -U postgres -c "ALTER TABLE wide_bloat SET (autovacuum_enabled = true);"

# then rebuild the table and repeat with:
# ALTER SYSTEM SET autovacuum_max_parallel_workers = 4;
# and read the result from: docker logs pgq19 | grep -A20 'automatic vacuum of table'

What I'd do with this

If your autovacuum workers are CPU-starved on wide tables, turning autovacuum_max_parallel_workers up is a real lever on that specific cost, and it's off by default so nobody gets it by accident on an upgrade. It is not a lever on how long your table stays bloated if the heap scan or the WAL it writes is the actual bottleneck, so measure your own table before assuming it helps — pg_stat_progress_vacuum's phase column will tell you which one you've got. And budget the global max_parallel_workers pool, not max_parallel_maintenance_workers, because that's the setting that actually gates this, and a long transaction left open anywhere will still starve a bloated table whether or not you've given autovacuum workers to spend.

RevCheck, compare the cars you want before you buy your next daily driver

Alex Georgiev — Thu, 15 Oct 2026 12:33:30 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Buying a used car in Bulgaria goes like this. You have twelve tabs open on mobile.bg and cars.bg. Half of them say "нов внос" (fresh import), "реални километри" (real mileage) and "без забележки" (nothing to fix). The same car shows up on both sites. Then you lose a Saturday driving to Plovdiv for a car that turned out to have a different engine, a seat that's been through a dog, and a price nobody else is asking.

So I built RevCheck. You paste two listings, from either site, and about 30 seconds later you get a side-by-side report and a verdict:

Every photo is checked by Gemma 4: color, dents, scratches, mismatched panels, and the equipment you can actually see (M Sport bumpers, panoramic roof, leather, navigation). It also flags extras the seller lists but the photos contradict.
The market position. It finds similar cars for sale on both sites (same model, body type, fuel and engine power, year ±1) and tells you "priced higher than 91% of 44 similar listings". A car listed on both sites only counts once.
A fair price range from TabPFN, trained on those comparables during the request: "€4,718 – €8,665, so this one is €2,276 above the expected price".
The known weak spots of that exact engine. DeepSeek V4 names the likely engine code from year, power and displacement (N47D20C, N52 or N53, and so on) and lists what to check at the viewing: timing chain rattle on a cold start, swirl flaps, the ZF gearbox, the high-pressure fuel pump.
Plain checks with no AI: a missing VIN, unusually low or high km per year, too few photos, a US or Canada import (go and run that VIN).
The verdict: which car to go and see, why, what to check on each one, and what to ask the seller before you get in the car.
English and Bulgarian, with the flags at the top. The report is written in whichever language you pick.

Who would pay for this? Anyone about to spend half a day and a tank of fuel on a viewing. People already pay mechanics and inspectors to look at a car in person before they buy. RevCheck is the step before that: it tells you which of the two is worth the trip. I'd keep a couple of comparisons a day free and charge flippers and small dealers, who compare dozens of cars a week, a small monthly fee.

Demo

Live site: https://georgievalex.github.io/revcheck/

Try it with two BMW 320d estates that are for sale right now:
https://georgievalex.github.io/revcheck/?a=https://www.mobile.bg/obiava-11744111052659454-bmw-320-2-0d&b=https://www.mobile.bg/obiava-11787314145036432

Paste two links, tick the box, click Compare:

The verdict comes back first. Each car card fills in as soon as that car is done:

The market position, the TabPFN fair price range and what the photos show:

And the known weak spots for the engine, with what to check at the viewing:

Code

GeorgievAlex / revcheck

Compare two used-car listings from mobile.bg or cars.bg with open-weight AI (Gemma 4 vision + TabPFN)

RevCheck

The pre-purchase check from the RevSofia family. Paste two used-car listings from mobile.bg or cars.bg and get a side-by-side report and a verdict on which one is the better buy.

Built from scratch on 15–18 October 2026 for the DEV Hacktoberfest Open-Source AI Challenge, Week 2: Side Hustle.

Site: https://georgievalex.github.io/revcheck/
Write-up on DEV: https://dev.to/alexgeorgiev17/revcheck-paste-two-used-car-listings-get-a-second-opinion-before-you-drive-across-town-1e7e

How the listings are read

mobile.bg and cars.bg refuse requests from DigitalOcean (a Cloudflare challenge or a 403). From Cloudflare's own network they answer. So the function reads pages through revcheck-reader (cloudflare/reader/), a small Cloudflare Worker:

Honest about who it is: a RevCheck/0.1 (+github link) User-Agent, plus the CF-Worker header Cloudflare adds to every Worker request. Either site can block it with one rule, and if they do, we stop.
Not a public proxy: it requires a shared secret that only the DigitalOcean function has, and it only fetches a mobile.bg listing…

View on GitHub

How I Built It

Same rule as my last two entries: no framework, no database, no npm dependencies. A static page on GitHub Pages, one DigitalOcean Function and one tiny Cloudflare Worker.

browser ── GitHub Pages (static) + Cloudflare Turnstile
   │  start   → bot check, returns a 3-minute signed ticket for these two links
   │  car ×2  → one request per car, in parallel
   │  verdict → the written comparison
   ▼
DigitalOcean Function  revcheck/compare
   ├─ reads listings + search pages ──► Cloudflare Worker ──► mobile.bg / cars.bg
   ├─ Gemma 4 (vision): every photo, 8 per call      ┐
   ├─ DeepSeek V4 Pro: engine + known weak spots      ├ DigitalOcean Serverless Inference
   ├─ Gemma 4: the verdict                            ┘
   └─ TabPFN 3.5: fair price from the comparables ──► Prior Labs API

Job	Model	Why this one
Look at the photos	`gemma-4-31B-it` (open-weight)	Reads a dashboard, a bumper and a seat in one pass, in Bulgarian or English
Engine and weak spots	`deepseek-v4-pro` (open-weight)	I tried four open models on the same car. Two returned broken JSON, and of the other two only DeepSeek brought up the gearbox
Fair price	TabPFN 3.5	Trains on the ~50 comparables inside the request, no training pipeline of my own
The verdict	`gemma-4-31B-it`	Weighs all of the above and writes like a friend who knows cars

A few things I'm happy with:

The browser data is untrusted. The verdict step gets each car's results back from the page, so every field is rebuilt from a known list with size caps before it reaches the model. Photo URLs have to point at the two sites' image servers, because Gemma downloads them.
The scary stuff is decided in code, not by the model. The odometer flag, the VIN check, km per year and US imports are all rules. The model can only suggest something to check (more on why below).
Every slow step has a deadline. If the known-issues lookup hangs, you get the car without it instead of no car at all.
26 tests, all node:test, including the ugly real-world cases from the next section.

The commit history has one commit per lesson below, if you want to see them land.

What I got wrong on the way

The listing sites block DigitalOcean. My first version fetched listings from the function, and on my laptop it worked perfectly. Once deployed, mobile.bg answered with a Cloudflare "Just a moment..." page and cars.bg with a 403. They block cloud servers, which is fair, they're tired of scrapers. The browser can't read them either, because neither site allows other websites to (no CORS headers). I built a small browser extension, and then threw it away, because nobody is going to install an extension from a site they've never heard of. What worked was a Cloudflare Worker that does nothing but fetch a listing, a search page or a model list. It sends RevCheck/0.1 and a link to the repo as its User-Agent, and Cloudflare adds a header naming the Worker, so either site can see exactly who's asking and block it with one rule. If they do, that's their call and I'll stop. It needs a secret only my function has, and it refuses any other URL, so it isn't an open proxy.

Gemma accused a seller of odometer fraud. On a 2012 Audi Q5 listed at 104,000 km, it read the dashboard photo as 266,777 km and the verdict called it "severe odometer fraud". I zoomed into the photo myself: the dash says 104,677. A blurry digit nearly got a real person called a fraudster on a public page. Now the model only reads the odometer and rates the reading clear or unclear, and the code decides. A flag only appears if the reading is clear and more than 15% off, and even then it says "check the dash in person". The prompts now ban accusations outright.

"Not enough photos" on a listing with 10 photos. DigitalOcean's endpoint takes at most 8 image links per call, so I check photos in parallel batches of 8. The last batch had just 2 interior photos, and the model, seeing only those, wrote "very few photos, interior only". The model doesn't get to judge photo count any more. The code does.

A coupe compared against sedans. One cars.bg seller typed the body type as "Kупе", with a Latin K. My body-type filter didn't recognise it and quietly searched without it. Funny enough, cars.bg's own search form spells it with a Latin K too. Body types are now compared after mapping Latin look-alike letters to Cyrillic, and there's a test with that exact string.

The AI invented a recall. I wanted RevCheck to list manufacturer campaigns, the "VANOS bolts" kind. DeepSeek confidently listed a VANOS bolt campaign for an N52 engine, which as far as I know was for different engines. I'd also gotten something wrong myself while testing: I told it a 2007 272 hp 330i had to be an N53, when the N52 made the same power until the changeover that year. So two changes. Recalls aren't generated at all: the page tells you to have a dealer check open campaigns by VIN, and links to a VIN decoder. And when an engine changed mid-year, the model has to give both and say "likely", never "certain".

Requests died at exactly 40 seconds. With photos, TabPFN and the known-issues lookup, a full comparison took 33–45 seconds, and every so often the page said "couldn't reach the comparer". I deployed a test function that just sleeps: DigitalOcean's web gateway answers 202 Response not yet ready after exactly 40 seconds, whatever the function's own timeout is. So a comparison is now three kinds of short requests (start in ~2 s, each car in ~19 s in parallel, the verdict in ~8 s), tied together by a signed ticket so the bot check still covers all of them. As a bonus, the cars now show up before the verdict.

Why Does Open Innovation Matter?

The right model for each job, behind one endpoint. Gemma looks at photos, DeepSeek knows engines, TabPFN does the numbers. With open-weight models on DigitalOcean's serverless inference, switching the known-issues step from Gemma to DeepSeek after testing was one string. With a single closed API, I'd have been stuck with whatever that vendor's one model knows about BMW timing chains.
It has to be cheap per comparison, or the side hustle doesn't work. One comparison is 6 to 8 model calls. Open models on prepaid inference keep that cheap enough to give a couple away free every day, and the prepaid balance with auto-reload off is a hard cap on the bill.
TabPFN trains on today's market, inside the request. No dataset to collect, no model to retrain. The 50 cars for sale right now are the training set, which is exactly what you want for used-car prices.
Open code is how you earn trust when you read other people's sites. The Worker that fetches listings is about 50 lines in the repo. Anyone, including mobile.bg and cars.bg, can read exactly what it fetches and how often.

Prize Categories

Best Use of DigitalOcean: a DigitalOcean Function runs the whole backend, and DigitalOcean Serverless Inference serves both open-weight models (Gemma 4 and DeepSeek V4 Pro). The 40-second gateway lesson above is a DigitalOcean story too.
Best Use of Gemma: gemma-4-31B-it checks every photo, with damage, equipment and odometer readings, and writes the final verdict in English or Bulgarian.
Best Use of TabPFN: TabPFN 3.5 predicts a fair price range (10th to 90th percentile) for each car, trained on live comparables from two sites, and that prediction feeds the verdict. It's one step inside the larger prediction pipeline, not a standalone demo.

What's next

Exact engine codes and campaigns from the VIN. The BMW decoders I use myself don't allow automated lookups, so for now RevCheck just links to them. OEMNavigations sells an official BMW VIN API, which would give the exact engine, build date and campaigns instead of "likely".
Asking mobile.bg and cars.bg for proper access, now that there's something real to show them.
mobile.de, through its official API, for the "what would this car cost in Germany plus import" question that every Bulgarian buyer asks.

Buying a used car in Bulgaria? Paste your two favourites into RevCheck before you drive anywhere.