DEV Community: The Homelab Postmortem

Podman's Docker-compatible API resets a container's restart policy to 'no' on any update that omits it

The Homelab Postmortem — Thu, 17 Sep 2026 23:10:58 +0000

Update 2026-09-21. Fixed upstream. containers/podman#29795 ("compat: Restart policy no longer resets on compat update") was merged into main on 2026-09-19, one day after this post and the comment below it; the maintainers closed #29790 the same day. The change is the guard described in "The fix" — restartPolicy is now only set when options.RestartPolicy.Name != "" — plus a regression test in test/apiv2/20-containers.at. It is not in a tagged release yet: v6.1.2 and v5.8.7 (both 2026-09-16) predate it, so every packaged Podman, including Debian 13's 5.4.2, still has the bug, and the client-side workaround still applies. The toolkit probe reports per machine rather than per version for exactly this reason.

TL;DR: On Podman 5.4.2 the Docker-compatible POST /containers/{id}/update endpoint always writes a restart policy, whether or not the request contained one. A body of {"Memory":67108864} applies the memory limit and changes unless-stopped to no; so does {"CpuShares":512}; so does {}. The response is 200 {"Warnings":null} every time. The consequence is exactly what the policy name says: a container that had been restarting itself after crashes stopped doing so, and a container with always did not come back after a reboot. Docker 26.1.5 given the same three requests keeps the policy — its handler only touches the policy when the body names one. Podman's own podman update CLI and its native libpod endpoint are fine; the reset lives in the compat handler alone, which passes a pointer to an empty string down to libpod, and libpod accepts the empty string as a valid policy. The reporter's fix — always send RestartPolicy in the body — works and is the only fix until containers/podman#29790 is addressed. The toolkit's check-podman-compat-update-restart.sh probes the box you are on and lists the running containers that will not come back.

The symptom

A fresh Debian 13 container on Proxmox, apt install podman (5.4.2+ds1-2+b2, rootful, cgroup v2, crun), the API socket enabled with systemctl enable --now podman.socket. The four requests from the upstream report, sent with curl to the Docker-compatible path:

## 1 create (RestartPolicy unless-stopped)
201 {"Id":"7e20d62370660c7ed26b6c50e92665cfdf47c1da75eafd36c9ecc44cc2ef8937","Warnings":[]}
## 2 start
204
## 3 inspect
RestartPolicy=unless-stopped Memory=0
## 4 update, memory only:  POST /v1.41/containers/t/update  {"Memory":67108864}
200 {"Warnings":null}
## 5 inspect
RestartPolicy=no Memory=67108864

The memory limit is applied. The restart policy is gone. Nothing in the response says so, and the compat GET /containers/t/json agrees with podman inspect: {"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"Memory":67108864}.

It is not specific to memory. Three more containers, each created with --restart unless-stopped --memory 128m:

## C. compat update {"CpuShares":512}
before  RestartPolicy=unless-stopped Memory=134217728 CpuShares=0
after   RestartPolicy=no             Memory=134217728 CpuShares=512
## E. compat update {}
before  RestartPolicy=unless-stopped Memory=134217728 CpuShares=0
after   RestartPolicy=no             Memory=134217728 CpuShares=0
## D. compat update {"Memory":67108864,"RestartPolicy":{"Name":"unless-stopped"}}
before  RestartPolicy=unless-stopped Memory=134217728
after   RestartPolicy=unless-stopped Memory=67108864

C is the telling one: the memory limit set at creation survives a CpuShares-only update — resources are merged into the existing set, as you would expect — and the restart policy does not. E says it more bluntly: an update with nothing in it resets the policy. D is the workaround from the report, and it holds.

Then whether it matters. Two containers whose command exits on its own after four seconds (sh -c 'sleep 4; exit 1'), both created with unless-stopped; g gets a memory-only compat update, f is left alone:

f RestartPolicy=unless-stopped State=running RestartCount=0
g RestartPolicy=no             State=running RestartCount=0
-- after 8 s
f RestartPolicy=unless-stopped State=running RestartCount=2
g RestartPolicy=no             State=exited  RestartCount=0
-- after 16 s
f RestartPolicy=unless-stopped State=running RestartCount=3
g RestartPolicy=no             State=exited  RestartCount=0

And across a reboot, with podman-restart.service enabled: p created with --restart always, q created the same way and then given a CpuShares-only compat update.

p always
q no
(pct reboot)
up 0 minutes
p Up 20 seconds
q Exited (137) 25 seconds ago

Why this is easy to miss

Nobody types this request by hand. It comes out of a Docker client library — dockerode, docker-py, the Docker SDK for Go — called by something that manages containers: a dashboard that lets you drag a memory slider, an autoscaler, a plugin that keeps a fleet of service containers within limits. Those libraries build the body from the fields you passed. If you passed a memory limit, the body has a memory limit in it and nothing else, which on Docker is the correct and documented way to change one thing. The upstream report came from exactly that shape of software: a container-management plugin that updated limits and found, later, that none of its containers had a restart policy any more.

Nothing on the Podman side flags it. The call returns 200. Warnings is null. The memory change is visible immediately and is what the caller checks. The policy change is visible only if you inspect a field you had no reason to touch, and its effect — the container not coming back — arrives at the next crash or the next reboot, hours or weeks later, with no line in any log connecting it to an API call that happened in between. When the container is found dead, podman inspect shows RestartPolicy.Name: no, which looks like it was always that way.

One more thing that hides the affected containers once you go looking: the stored value is not the string no. It is the empty string, which Podman's define.RestartPolicyNone is. podman inspect renders it as no, but the ps filter does not accept that spelling:

$ podman ps -a --filter restart-policy=no
(nothing)
$ podman ps -a --filter restart-policy=none
t c e g h

What is really going on

pkg/api/handlers/compat/containers.go, UpdateContainer, at v5.4.2 and — for these lines — identical on main today. After decoding the body into a Docker-shaped container.UpdateConfig and merging the resource fields one by one (if options.CPUShares != 0 { … }, and so on for every limit), it gets to the policy:

localPolicy := string(options.RestartPolicy.Name)
restartPolicy := &localPolicy

var restartRetries *uint
if options.RestartPolicy.MaximumRetryCount != 0 {
    localRetries := uint(options.RestartPolicy.MaximumRetryCount)
    restartRetries = &localRetries
}

if err := ctr.Update(resources, restartPolicy, restartRetries, &define.UpdateHealthCheckConfig{}); err != nil {

Every resource field is guarded by "did the caller set it". The policy is not: restartPolicy is a pointer to whatever Name was, and when the body had no RestartPolicy, Name is "". libpod's Update treats a non-nil pointer as an instruction:

if restartPolicy != nil {
    if err := define.ValidateRestartPolicy(*restartPolicy); err != nil {
        return err
    }
    …
    c.config.RestartPolicy = *restartPolicy

and ValidateRestartPolicy("") succeeds, because "" is RestartPolicyNone, one of the five accepted values. So the empty string is stored as the policy, and from then on the container has none.

The CLI does not have this problem because cmd/podman/containers/update.go only fills in the pointer when the flag was given:

if cmd.Flags().Changed("restart") {
    policy, retries, err := util.ParseRestartPolicy(updateOpts.Restart)
    …
    opts.RestartPolicy = &policy

Confirmed on the box: podman update --memory 64m a changed the memory limit and left unless-stopped alone. The native libpod endpoint (/v5.4.2/libpod/containers/{id}/update, which takes an OCI-shaped body — {"memory":{"limit":67108864}}; send it the Docker shape and you get a 500 decode error, not a silent reset) also left the policy alone.

Docker, given the identical compat requests from a second throwaway container running docker.io 26.1.5 from the same Debian release: {}, {"CpuShares":512}, {"Memory":67108864,"MemorySwap":134217728} — all 200, policy unless-stopped after every one. (Docker rejected a bare {"Memory":67108864} with a 409 about the memoryswap limit, which is its own quirk and beside the point; the policy was untouched by that too.) That is the comparison the report drew, and it holds: the same request means "change these limits" to one daemon and "change these limits and delete the restart policy" to the other.

The fix

Upstream: the handler needs the same guard the resource fields have — set restartPolicy only when options.RestartPolicy.Name != "". That is the reporter's suggestion and there is no PR yet; #29790 is where it will happen.

Until then, on the client side, every compat update request must carry the restart policy you want to keep, even when you are not changing it:

{"Memory":67108864,"RestartPolicy":{"Name":"unless-stopped"}}

which means reading it back first (GET /containers/{id}/json, .HostConfig.RestartPolicy) unless your code already knows it. If you control the client and can call Podman natively, podman update and the libpod endpoint do not have the bug.

For containers that have already lost their policy, podman update --restart=always <name> (or unless-stopped) puts it back; the CLI path is safe. Finding them is the harder part, since a container that never had a policy and one that lost it look identical. The toolkit's check-podman-compat-update-restart.sh does two things: lists the running containers whose policy is currently none — the set that will not come back — and probes the Podman you are on, because there is no fixed version to compare against yet:

podman             5.4.2  (rootful)
running, policy=no 1 container(s): q
                   (will not restart after a crash or a reboot; if any were meant to,
                    re-apply with: podman update --restart=always <name>)
api socket         /run/podman/podman.sock
probe              create --restart always -> always; POST /v1.41/containers/{id}/update {} -> HTTP 200 {"Warnings":null}; policy now: no

AFFECTED: a Docker-compat update without RestartPolicy resets the policy to 'no' (containers/podman#29790).

The probe creates a throwaway container with --restart always, sends {} to the compat endpoint, inspects, and removes it. If no API socket is active it starts podman system service on a private socket for thirty seconds and stops it afterwards; nothing is pulled and nothing else on the host is touched. --no-probe does only the listing. The probe was run against 5.4.2 with the socket active, with the socket stopped, and on a machine with no Podman at all (exit 2); it has not been run against a fixed Podman because none exists.

The generalisable habit

"Compatible" is a claim about the request format, not about what the daemon does with the fields you left out. Every Docker-shaped client that talks to Podman is relying on the compat layer to have made the same decision Docker made for each absent field, and this is one field where it made a different one. There is no way to see that from the request or the response; the two daemons return the same 200 {"Warnings":null} and diverge in a field nobody asked about. The same runtime, on the same box, publishes a port with a different firewall outcome from Docker while printing the identical 0.0.0.0:8080->80/tcp — there the difference was in Podman's favour; here it is not. Either way it was found by sending the same input to both and reading everything back, not the field that was changed.

When you test a partial update, inspect the whole object, not the field you changed. The report's reproduction is four commands, and the one that finds the bug is an inspect of a field the update had no business touching. Merging semantics — "fields you omit are left alone" — are the kind of thing every API is assumed to have and few document, and the only way to know is to omit a field that matters and see whether it survives.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

One failed WPA handshake and a headless Raspberry Pi on Trixie stays off WiFi until someone types a command

The Homelab Postmortem — Thu, 17 Sep 2026 16:36:22 +0000

TL;DR: Raspberry Pi OS Trixie ships NetworkManager 1.52.1. On that version a WPA-PSK connection gets exactly one 4-way handshake: if it fails, NM asks for a new key — on a profile that has connected successfully many times, and regardless of connection.auth-retries, which 1.52's own documentation scopes to 802.1X. On a headless Pi there is no secret agent to answer, so the request fails with no-secrets, and NetworkManager then blocks the profile from autoconnect. That block survives restoring the key and reloading the profile: two minutes later the device was still disconnected. Only nmcli connection up (or a reboot) clears it. Upstream extended the retry budget to PSK in NetworkManager 1.58 (746a5902) and the commit message names the real-world trigger: a device "can leave the range of an access point and therefore fail a 4-way handshake". Trixie does not have it (raspberrypi/trixie-feedback#102 is the open backport request). A systemd timer that re-activates the profile when the device is disconnected recovered the same failure in 40 seconds; the toolkit's check-wifi-autoconnect-block.sh reports whether you are exposed and installs it.

The symptom

Raspberry Pi 4B, Raspberry Pi OS Lite Trixie (arm64), network-manager 1.52.1-1+rpt4, one WiFi profile that has been the machine's only network for weeks (the ethernet port is empty). The test: set connection.auth-retries 0, replace the PSK with a wrong one, and reconnect — from a script detached with systemd-run, because the SSH session is on that WiFi and is about to die. NetworkManager's log at debug level:

01:08:48.5  device (wlan0): supplicant interface state: scanning -> associating
01:08:48.8  device (wlan0): supplicant interface state: associating -> 4way_handshake
01:08:52.4  device (wlan0): supplicant interface state: 4way_handshake -> disconnected
01:08:52.4  device (wlan0): Activation: (wifi) disconnected during association, asking for new key
01:08:52.4  device (wlan0): state change: config -> need-auth (reason 'supplicant-disconnect')
01:08:52.4  device (wlan0): no secrets: No agents were available for this request.
01:08:52.4  device (wlan0): state change: need-auth -> failed (reason 'no-secrets')
01:08:52.4  policy: block-autoconnect: connection '<wifi profile>' now blocked from autoconnect due to no secrets
01:08:52.4  device (wlan0): Activation: failed for connection '<wifi profile>'

One handshake. 3.6 seconds. No retry. auth-retries had been set to 0 for the test, which is documented as "try indefinitely" — but read the whole comment in 1.52's libnm-core-impl/nm-setting-connection.c:

The number of retries for the authentication. Zero means to try indefinitely; -1 means
to use a global default. If the global default is not set, the authentication
retries for 3 times before failing the connection.

Currently, this only applies to 802-1x authentication.

For a PSK network on this version there is no retry setting at all, and no retry. The upstream report and the research that led here both framed this as "auth-retries is ignored"; the last line of the comment says it more plainly — it was never meant to apply. That does not change what the machine does, which is the rest of this post.

Then the second half, which is the one that matters on a headless box. The correct key was put back by copying the original /etc/netplan/90-NM-*.yaml over the modified one and running nmcli connection reload, with nothing else touched, and the device was polled every five seconds:

01:15:43  restoring profile files + reload only
          connection.auth-retries: -1        ← original setting is back
01:15:54  t+5s     wlan0:disconnected
01:16:19  t+30s    wlan0:disconnected
01:16:50  t+60s    wlan0:disconnected
01:17:21  t+90s    wlan0:disconnected
01:17:52  t+120s   wlan0:disconnected
01:17:52  explicit `nmcli connection up`
          Connection successfully activated

The profile had the right key for two minutes and NetworkManager did not try it once. nmcli connection up connected on the first attempt.

Why this is easy to miss

nmcli connection modify … connection.auth-retries 0 returns 0 and the value reads back, and the property's one-line description in nmcli's help is "Number of retries for authentication" with no qualifier. Only the full comment in the source, or the 1.58 changelog, says PSK is excluded. Someone who sets it to protect a headless Pi has done nothing, and nothing tells them.

On a desktop, the whole thing is a password prompt: the handshake fails, NM asks the agent, the user sees "enter password for network X", types the same password, and is back online with a shrug. The bug reads as "Ubuntu keeps asking for my WiFi password" — and that is exactly how it appears in the threads that exist about it, none of which mention auth-retries.

On a headless Pi there is no prompt and no shrug. The no-secrets failure blocks the profile, nmcli device status says disconnected, and nothing on the machine will change that. From outside, the Pi has dropped off the network and stays off; the natural diagnosis is power, the SD card, or the radio. A reboot would clear it — the block is not persisted — which confirms the wrong diagnosis. The upstream commit that fixed the retry path describes the trigger as ordinary — a device that "can leave the range of an access point" — which for a Pi in a cupboard means the access point rebooting, changing channel, or a microwave running.

The block is the part that turns a nuisance into an outage, and it is invisible: there is no nmcli field for it. It shows up only as one journal line, block-autoconnect … due to no secrets, and as the absence of any retry afterwards.

What is really going on

Two mechanisms in sequence.

The retry that does not exist. In NetworkManager 1.52, a 4-way handshake failure on a WPA-PSK connection lands in the device's auth-failure handler, which requests new secrets directly. The authentication retry budget — nm_device_auth_retries_has_next(), driven by connection.auth-retries — is consulted on the 802.1X path only, as the documentation says. The upstream change, +78/−12 in src/core/devices/wifi/nm-device-wifi.c, makes a previously-connected PSK profile exhaust that budget before asking, and rewrites the documentation to match ("Connections using a pre-shared key to authenticate will only prompt for a new key during the last authentication attempt"). The reasoning in the commit message:

While NetworkManager tries its best to determine whether a new PSK is needed, it can still run into edge cases. One of these edge cases is that a device can leave the range of an access point and therefore fail a 4-way handshake. Because these cases can't be confidently detected, a device which was previously connected should try to exhaust its authentication retries before requesting new secrets.

That landed in 1.58.0 (July 2026). Trixie's 1.52.1 predates it by three stable releases, and the Pi OS package has not backported it. Before it, "one attempt, then prompt" was the designed behaviour for PSK — designed for a laptop with a person in front of it.

The block that never lifts. When NM asks for secrets and no agent answers, the activation fails with reason no-secrets, and policy marks the connection as blocked from autoconnect for that reason. The intent is sensible — do not retry a connection whose password the user has declined to provide. But on 1.52 the request was never justified, and on a headless machine the "user" is nobody. The block is cleared by an explicit activation of the connection — observed three times here. It was not cleared by rewriting the profile with the correct key and reloading it, which is what the two-minute poll above shows. (A reboot should clear it too, since the block is not persisted; that was not tested, and on a headless box it is not much of a remedy.)

So the chain, on a headless Pi on Trixie, is: any single handshake failure → new-secrets request (no PSK retry exists on 1.52) → no-secrets → autoconnect blocked → offline until a human runs nmcli connection up.

The fix

Upstream's fix is the right one and it is not available on Trixie today; #102 is the request. Until it lands, the thing that clears the block is the thing to automate.

A watchdog timer. Every 30 seconds: if the WiFi device is disconnected and the profile is not active, run nmcli connection up for it. The same failure was induced a third time with this installed, the correct key restored by file copy and reload as before, and no manual command:

01:20:28  wifi-autoconnect-watchdog: wlan0 disconnected and '<wifi profile>' inactive — running nmcli connection up
01:20:34  wifi-autoconnect-watchdog: up failed              ← key still wrong at this point; correct
01:20:29  restoring profile files + reload only
01:21:03  wifi-autoconnect-watchdog: wlan0 disconnected and '<wifi profile>' inactive — running nmcli connection up
01:21:15  wifi-autoconnect-watchdog: recovered
01:21:15  t+40s    wlan0:connected

Forty seconds, against two minutes and counting without it. The first attempt failing is the watchdog doing the right thing — the key really was wrong then — and it costs nothing but a log line.

The toolkit's check-wifi-autoconnect-block.sh does the reporting and the install:

$ ./check-wifi-autoconnect-block.sh
NetworkManager     1.52.1-1+rpt4
WiFi profile       <wifi profile>  (device wlan0)
auth-retries       -1   (802.1X-only before NM 1.58; makes no difference to PSK here)
fix in this NM     no  (< 1.58: one handshake failure -> new-secrets request)
blocked this boot  2 time(s)  (journal: 'blocked from autoconnect due to no secrets')
watchdog timer     inactive

EXPOSED: NM 1.52.1-1+rpt4 asks for a new key after one failed PSK handshake (no retry on this version),
         and with no secret agent the profile is then blocked from autoconnect until
         'nmcli connection up' runs. A headless box stays off WiFi.
         Install the watchdog:  sudo ./check-wifi-autoconnect-block.sh --install-watchdog

--install-watchdog writes a 12-line script to /usr/local/sbin, a oneshot service, and a timer, and enables the timer; --remove-watchdog takes all three away. Report mode is read-only and never touches secrets. blocked this boot counts the journal line, so it also tells you whether this has already happened to the machine you are looking at.

Things that do not fix it, for the record: setting auth-retries to anything (on 1.52 it applies to 802.1X only); nmcli connection reload (tested, two minutes); connection.autoconnect-retries (a different budget for a different failure — not tested here, and not the reason the profile is blocked).

The generalisable habit

A setting that is accepted and stored is not a setting that is consulted — and the qualifier is usually in the last line of the comment. auth-retries has a validator, a documented range, a documented meaning, a code path that reads it, and one sentence at the bottom saying it applies to a different authentication type. The research that surfaced this candidate, the upstream report, and the first draft of this post all missed that sentence. The only way to know a setting does nothing on your box is to make it matter and watch the log. That is a five-minute test on any machine that can afford to lose its network for two of them, and it is the same test that found instance-id being ignored by cloud-init and response_format being ignored by llama-server: change the value to something that must change the outcome, and see whether the outcome changes.

Test the recovery, not just the failure. The retry bug is the upstream report. The block is what a headless machine actually experiences, and it only became visible by restoring the correct key and not running the obvious command — waiting to see whether the system would come back on its own. It would not. A reproduction that ends at "yes, it fails" would have shipped a post about a missing retry; the post that matters is about a Pi that stays offline, and the difference was two minutes of watching a disconnected line not change.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama.cpp reads past your pos array on embedding batches for M-RoPE models, and the logits change with the heap

The Homelab Postmortem — Wed, 16 Sep 2026 03:47:05 +0000

TL;DR: decode the same llama_batch.embd batch twice on an M-RoPE model — Qwen3.5, Qwen2.5-VL, Qwen3-VL, anything whose llama_model_rope_type() is MROPE or IMROPE — and the logits differ, on the CPU, one thread, memory cleared, nothing random anywhere. Token ids through the same context are bitwise identical. The cause is a heap over-read: llama.h says every array in a batch "must have size of n_tokens" and llama_batch_init allocates pos that way, but for an embedding batch on an M-RoPE model llama_batch_allocr::ubatch_add reads pos[j*n_tokens + i] for j in 0..3, on the assumption that embeddings are images with per-section positions. Sections 1–3 are whatever sits after your array. valgrind: Invalid read of size 4 … ubatch_add … 0 bytes after a block of size 20 alloc'd … llama_batch_init. Which batch sizes misbehave, and whether BF16 or F16 weights "matter", is heap layout and nothing else — an extra environment variable changed the answer. Allocate pos with 4 * n_tokens entries and fill every section and every case is deterministic. The upstream report (ggml-org/llama.cpp#28963) has the symptom and localises it to the first attention layer; the read that causes it is below, and it is the twin of the pos = NULL over-read from the day before.

Update, 2026-09-16 (same day): the reporter of #28963 confirmed the diagnosis independently on a different platform, model and toolchain — Windows 11, MSYS2 g++ 15.2, Qwen3.5-2B in f16 — applying only the caller-side fix below (pos sized 4 * n_tokens, every section filled). Two decodes of the same 141-position embedding batch went from argmax agreement 126/141 (cosine 0.85–0.99) to 141/141 (cosine 1.000000); the embedding path against the token path went from 132/141 to 141/141. In their real application — a layer-pipeline relay feeding a truncated model through llama_batch.embd — a 141-step greedy generation had been diverging from the whole-model baseline somewhere between steps 46 and 67, differently on each run; after the fix it matches the baseline exactly, repeatedly. They also hit every one of the false variables described below (weight type, fresh context, graph reuse, an unrelated environment variable) before the cause was known.

The symptom

llama.cpp master 930e2fa (2026-09-16), CPU build, Debian 13 in an LXC on an i5-8400T, n_threads = n_threads_batch = 1, flash attention off, n_ctx = n_batch = n_ubatch = 512. A 90-line C program against libllama: load a model, build one batch of N positions with llama_batch_init, fill pos[i] = i the way the header describes, then six times in a row call llama_memory_clear(mem, true) and llama_decode on that same batch, and compare the full N × n_vocab logits block of each run against run 0. In tokens mode the batch carries token ids; in embd mode it carries seeded pseudo-random floats through llama_batch.embd, as in the report.

Qwen3.5-0.8B, BF16 GGUF from ggml-org:

mode=tokens N=5
run 1..5: bitwise_equal=yes  max_abs_diff=0  cosine=1.00000000  argmax_same=yes

mode=embd N=5
run 1: bitwise_equal=NO  max_abs_diff=2.4   cosine=0.99422627  argmax_same=yes
run 2: bitwise_equal=NO  max_abs_diff=2.05  cosine=0.99525302  argmax_same=NO
run 3: bitwise_equal=NO  max_abs_diff=2.11  cosine=0.99615401  argmax_same=yes
run 4: bitwise_equal=NO  max_abs_diff=1.85  cosine=0.99524820  argmax_same=NO
run 5: bitwise_equal=NO  max_abs_diff=2.06  cosine=0.99490347  argmax_same=yes

The report says argmax stays constant and the practical impact is on reproducibility. Here it does not stay constant: at N=5 the greedy token differs from run 0 in two runs out of five, and at N=8 in every run. At N=8 the shape is also different — runs 1 through 5 are identical to each other and only run 0 stands apart:

mode=embd N=8
run 1..5: bitwise_equal=NO  max_abs_diff=2.47  cosine=0.99254973  argmax_same=NO   (all five identical)

Why this is easy to miss

Nothing in the API says anything. llama_decode returns 0, the logits are finite and plausible, and the top token is right most of the time. Greedy generation through this path can match a token-path baseline for a whole sequence and then diverge on the next run where two candidates were close, which reads as "the model is a bit unstable", not "the library read past my array".

It is invisible to almost everyone. llama-cli, llama-server, chat, completion — all tokenise text and go through llama_batch.token, where positions for text are broadcast across the M-RoPE sections and nothing is over-read. llama_batch.embd is for people who inject vectors: multimodal projectors, hidden-state relays between model instances, embedding-space experiments. The report came from a layer-pipeline prototype. And the multimodal code in the tree that does use this path — mtmd — allocates four positions per token itself, so it never sees it.

Then there is the false trail, which cost most of the afternoon and is worth recording because it looked like a result. The first thing tried was the dense model already on the box, gemma-3-1b Q4_K_M: deterministic at every N. Then Qwen3.5 in BF16: not. Then the same Qwen file converted to F16 and Q8_0 with llama-quantize: deterministic. Then gemma converted to BF16: deterministic. A tidy six-cell grid with exactly one failing cell — hybrid architecture, BF16 weights — and a one-command workaround. Meanwhile, which batch sizes failed moved around when the graph was rebuilt instead of reused, moved again with a fresh context per decode, and N=8 failed in every configuration. That grid was going to be the article.

It was heap layout. Every one of those variables changes what malloc puts next to a 20-byte allocation. The proof that it was layout and not weights came from setting MALLOC_PERTURB_, which fills freed and allocated memory with a byte pattern: the results changed, but so did they with X_PERTURB_=1, a variable glibc has never heard of, because a longer environment shifts the initial heap. A "fix" that works for reasons you cannot state is a layout accident, and the F16 conversion was one.

What is really going on

llama_context_params.cb_eval is a public callback that sees every graph node after it is computed. Hashing every node's output on each run and reporting the first one that differs from run 0, with the hashes of its sources, gives this at N=8:

run 1: 948/1441 nodes differ; FIRST differing node = #0
  #0     node_0        op=GET_ROWS  ne=[1024,8,1,1]     srcs: ALL SAME
  #204   Qcur-3        op=ROPE      ne=[256,8,8,1]
          src0 Qcur_normed-3   (in-place, same buffer)
          src1 leaf_55         DIFFERS   data=…9c860..…9c8e0   (128 bytes)
  #211   Kcur-3        op=ROPE      ne=[256,2,8,1]
          src1 leaf_55         DIFFERS

node_0 is the unselected token-path branch that ggml_build_forward_select leaves in the graph without computing; nothing reads it, and zero nodes list it as a source. The first divergence that matters is Qcur-3, the RoPE on layer 3 — the first full-attention layer in this model, which is exactly where the reporter's per-layer comparison put it. Its output differs because its second input differs: leaf_55, 128 bytes, is int32 × 32 = 8 tokens × 4. That is the M-RoPE position tensor, and it is different on every run with the same batch.

Positions come from the batch through llama-batch.cpp:

for (size_t j = 0; j < (size_t)n_pos_per_embd; ++j) {
    // if we are using M-RoPE
    //     if the current batch is text, we need to broadcast the same position across all RoPE sections
    //     otherwise, the input batch is image embeddings, we copy the positions as-is
    size_t src_off = batch.token ? 0 : j*batch.n_tokens;
    udata->pos[j*n_tokens + i] = batch.pos[src_off + idxs[i]];
}

For a token batch, src_off is 0 and the one position per token is broadcast into all four sections. For an embedding batch, the code assumes "image embeddings" and copies four sections from batch.pos[j*n_tokens + i] — indices up to 4*n_tokens - 1. And the header, include/llama.h:

// The provided arrays (i.e. token, embd, pos, etc.) must have size of n_tokens

with llama_batch_init documented, and implemented, to allocate pos with n_tokens entries. So a caller who does what the header says — the reporter, this probe, anyone relaying hidden states — hands over N positions, and the library reads 4N. valgrind on the unmodified library says it in one line:

Invalid read of size 4
   at llama_batch_allocr::ubatch_add(...)
   by llama_batch_allocr::split_equal(...)
   by llama_memory_hybrid::init_batch(...)
   by llama_context::decode(...)
 Address 0xb4e4244 is 0 bytes after a block of size 20 alloc'd
   at malloc
   by llama_batch_init

Twenty bytes is five positions. The next twelve ints are the heap. What lives there depends on everything that was allocated before and after the batch — the model's tensors, the context's buffers, the environment — which is why the symptom tracked the weight type, the batch size, graph reuse, a fresh context and an unrelated environment variable, and why at N=8 run 0 differed from runs 1–5 (the neighbour was written once, early, and then left alone). It is also why the zero-input control in the report is bitwise stable: garbage positions rotate a zero vector to another zero vector.

Dense models have n_pos_per_embd = 1; the loop runs once and never leaves the array. The over-read needs an M-RoPE model and an embedding batch, and nothing else — not the weight type, not the architecture beyond the RoPE mode, not the batch size.

This is the second over-read in the same fifteen lines in two days. Yesterday's was pos = NULL, where the library auto-generates positions and sized its own buffer to n_tokens before reading 4*n_tokens; that one had a PR, #28910, still open. This one is pos != NULL, where the buffer is yours and the library reads past it. Same loop, same assumption — that an embedding batch on an M-RoPE model carries four positions per token — applied to a buffer the header told you to make one-quarter the size.

The fix

In your code, today. Allocate four positions per token for any embedding batch on an M-RoPE model, and fill every section — for text-like input, the same position in each:

enum llama_rope_type rt = llama_model_rope_type(model);
int n_pos = (rt == LLAMA_ROPE_TYPE_MROPE || rt == LLAMA_ROPE_TYPE_IMROPE) ? 4 : 1;

struct llama_batch batch = llama_batch_init(N, n_embd, 1);
free(batch.pos);
batch.pos = calloc((size_t)n_pos * N, sizeof(llama_pos));
for (int j = 0; j < n_pos; j++)
    for (int i = 0; i < N; i++)
        batch.pos[j * N + i] = i;   /* position of token i */

That is the single-variable test. With it and nothing else changed — same BF16 file, same build, same thread count — every case that failed above is bitwise deterministic across six runs, including N=7–10 with graph reuse disabled and with a fresh context per decode, the configurations nothing else fixed:

### pos allocated 4*N, every section filled
RESULT embd N=3: DETERMINISTIC
RESULT embd N=5: DETERMINISTIC
RESULT embd N=8: DETERMINISTIC
RESULT embd N=10: DETERMINISTIC
### + LLAMA_GRAPH_REUSE_DISABLE=1 / fresh context
RESULT embd N=7..10: DETERMINISTIC (all)

llama_batch_free frees pos with free(), so replacing the pointer with your own calloc is safe.

Upstream. The header and the loop disagree, and one of them has to move. Either the header says that embedding batches on M-RoPE models carry n_pos_per_embd * n_tokens positions and llama_batch_init grows to match (which needs to know the model, and today it does not), or the batch code stops guessing that every embedding is an image and broadcasts when it is handed one position per token — which is what #28910 does for the pos = NULL case. The mtmd code that legitimately passes four sections would need a way to say so. That is a design decision for the maintainers; the comment in the thread has the trace, the valgrind line and the probe.

Before you build on this path. The toolkit's check-embd-determinism.sh compiles the probe against your llama.cpp checkout, runs the token path as a control, sweeps the embd path over a list of batch sizes with pos sized as documented, and for any N that is not deterministic re-runs it with pos sized n_pos_per_embd * N to confirm that the over-read is the cause on your build rather than something new:

control  tokens N=5   deterministic   (rope_type=40 n_pos_per_embd=4)
embd     N=2          deterministic
embd     N=5          NON-DETERMINISTIC   worst cosine 0.99631981   argmax changed in 3 of 3 runs
         N=5   with pos sized n_pos_per_embd*N: deterministic  -> over-read confirmed
embd     N=8          NON-DETERMINISTIC   worst cosine 0.99226277   argmax changed in 3 of 3 runs
         N=8   with pos sized n_pos_per_embd*N: deterministic  -> over-read confirmed

FAIL: the embd path gave different logits for the same input at N = 5 8,
      while the token path did not.
      Cause on this build: llama-batch.cpp reads n_pos_per_embd*N positions from an
      embd batch on an M-RoPE model, past the N that llama.h documents and
      llama_batch_init allocates.

Against gemma-3-1b, or against Qwen3.5 with the fix in place, it prints OK. Note that "N=2 deterministic" above is not safety: the twelve bytes past a two-position array happened to be stable on this heap. The over-read is there at every N.

The generalisable habit

Two, and the second is the one that cost time.

When a header states a size, find the loop that reads the array and check its bound against the same expression — the habit from yesterday, and it would have found this in ten minutes if applied to batch.pos instead of udata->pos. Yesterday's loop and today's are the same loop. The fix for one over-read was a resize of the library's own buffer; the buffer in the other over-read belongs to the caller, so the same loop needed the same look twice.

A workaround you cannot explain is a measurement of your heap, not of the bug. The six-cell weight-type grid was real data: every cell was measured, reproducible, and wrong about what it meant. What gave it away was not a failed reproduction but a successful one from a variable that could not possibly matter — an environment variable with a made-up name flipped the result, which meant the real variable was memory layout, which meant an uninitialised or out-of-bounds read, which meant stop characterising and start tracing. cb_eval found the tensor in one run and valgrind found the line in one more. Neither needed a fork, a hidden-state API or a sanitiser build; both were available the whole afternoon. The order that would have saved three hours is: reproduce, then locate, then characterise — and treat any characterisation that depends on things like weight type or batch size with suspicion until the located cause explains why it should.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Ollama's gemma4 renderer silently drops tool parameters named type or description, and the model invents a value

The Homelab Postmortem — Wed, 16 Sep 2026 02:02:30 +0000

TL;DR: define a tool for a gemma4 model in Ollama with a parameter whose name is type, description, properties, required or nullable, and the model is never shown that parameter's definition. It is still listed under required:[…] in the rendered declaration, so the model knows it must supply something and has no idea what. On gemma4:e2b at temperature 0 the result is an invented value — "type": "urgent" for a parameter whose only legal values are urgent_A7 and routine_B3 — and on the reporter's gemma4:26b it is an omitted argument. HTTP 200, nothing in the log. The cause is a Go port of the reference Jinja template that dropped one argument: the template's format_parameters macro takes filter_keys=false and filters schema keywords in exactly one call site; the Go writeSchemaProperties filters unconditionally and serves all four. Twelve lines restore the argument, and the same request then returns urgent_A7. Until that lands, rename the parameter.

The symptom

Ollama 0.33.3, Debian 13, CPU only, gemma4:e2b. The tool is the one from ollama/ollama#18468: a ticket creator with two required string parameters, one of which has an enum. The user message is Open an urgent ticket titled 'Water leak'. Temperature 0, think: false, two runs each. The only thing that changes between requests is the name of the second parameter.

"parameters": {"type": "object", "required": ["title", "<name>"],
  "properties": {"title":  {"type": "string", "description": "Ticket title"},
                 "<name>": {"type": "string", "description": "Priority code",
                            "enum": ["urgent_A7", "routine_B3"]}}}

kind         run1  {"kind":"urgent_A7","title":"Water leak"}
kind         run2  {"kind":"urgent_A7","title":"Water leak"}
type         run1  {"title":"Water leak","type":"urgent"}
type         run2  {"title":"Water leak","type":"urgent"}
description  run1  {"description":"Urgent water leak.","title":"Water leak"}
description  run2  {"description":"Urgent water leak.","title":"Water leak"}

urgent_A7 is a string that appears nowhere in the prompt except inside the parameter's definition. When the parameter is called kind, the model produces it every time. When the same definition is called type, the model produces urgent — the word from the user's sentence — and when it is called description, a sentence. It is filling a slot it was told exists with whatever the name suggests, because the definition that would have told it otherwise was never in the prompt.

The reporter, on gemma4:26b, saw the argument left out entirely instead. A bigger model declines to guess; a smaller one guesses. Both are downstream of the same missing text.

Why this is easy to miss

Nothing fails. The request is accepted, the response is 200, the tool call is well-formed JSON with the right function name, and one of its two arguments is correct. If your tool handler validates against the schema, you get a validation error on the model's output and start debugging the model: prompt wording, temperature, whether gemma4 is any good at function calling. The reporter's numbers were 1 ticket created out of 14 attempts with 35 schema-validation failures, all on a required parameter called description, and it took renaming the parameter to find out the model had never seen it.

If your handler does not validate, you get "type": "urgent" stored somewhere as a priority code, and you find out later.

The names involved are not exotic. type is the most natural name for a categorical field. description is what you call the free-text field on a ticket, an event, a product. JSON Schema has no rule against property names that happen to be schema keywords — they live in a different namespace, inside properties, and every validator handles them correctly. This renderer does not.

Ollama has no debug setting that prints the rendered prompt, so the declaration the model actually receives is not observable from outside the process. The only ways to see it are to read the renderer or to run it.

What is really going on

Ollama renders gemma4 prompts in Go, in model/renderers/gemma4.go, and checks that renderer against the model's reference Jinja template in model/renderers/testdata/. The template's parameter macro looks like this:

{%- macro format_parameters(properties, required, filter_keys=false) -%}
    {%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
    {%- for key, value in properties | dictsort -%}
        {%- if not filter_keys or key not in standard_keys -%}
            {{ key }}:{

filter_keys defaults to false, and the macro is called four times. Three of them pass nothing — top-level parameters, nested object properties, array items — because in those calls the keys are parameter names. One call passes filter_keys=true: the branch where an object has no properties map and the macro walks the object's own keys, where type and description really are schema keywords and must not be rendered as if they were parameters.

The Go port has the same four call sites and the same skip list. It does not have the argument:

func (r *Gemma4Renderer) writeSchemaProperties(sb *strings.Builder, props map[string]any) {
    …
    for _, name := range keys {
        if isSchemaStandardKey(name) {
            continue
        }

So the one-branch filter became an every-branch filter. A same-package test that calls Render with the ticket tool and prints the declaration shows the consequence directly, on 0.33.3 (gemma4's <|"|> quote token shown as "):

kind        properties:{kind:{description:"Priority code",enum:["urgent_A7","routine_B3"],type:"STRING"},title:{…}}  required:["title","kind"]
type        properties:{title:{…}}                                                                                   required:["title","type"]
description properties:{title:{…}}                                                                                   required:["title","description"]
properties  properties:{title:{…}}                                                                                   required:["title","properties"]
required    properties:{title:{…}}                                                                                   required:["title","required"]
nullable    properties:{title:{…}}                                                                                   required:["title","nullable"]
typo        properties:{title:{…},typo:{description:"Priority code",enum:[…],type:"STRING"}}                          required:["title","typo"]

The required list is written by a different function that does not filter, which is why the name survives there. The model is told "you must supply type" by one line and shown no type by the other. typo is in the table as a control: it is not on the list, so it renders. The problem is the five words, not anything about the schema.

The existing reference tests in the package all pass with this behaviour and all still pass after the fix below, which means none of them ever used a parameter named after a keyword. That is why the port shipped.

Upstream main at a43fad18 (2026-09-15) has the same code.

The fix

Two, in order of when you can have them.

Today: rename the parameter. This is the whole workaround and it is enough — the runs above with kind are the fixed state. If the name is part of an external contract you cannot change, map it in your tool handler: declare kind to the model, translate to type before you call the real function.

The toolkit's check-tool-param-names.sh finds every colliding name in a tool definition file, including nested objects and array items, and prints the JSON path of each:

$ ./check-tool-param-names.sh tools.json
tools.json: 3 colliding parameter name(s)
  create_ticket.parameters.properties.type
  create_ticket.parameters.properties.meta.properties.description
  create_ticket.parameters.properties.meta.properties.tags.items.properties.required

It also has a live mode, --probe http://host:11434 --model gemma4:e2b, which sends the ticket request twice — kind as the control, then type — and reports FAIL when the type call does not carry the enum value. The control is there because a model that cannot make tool calls at all would otherwise look like a renderer bug; that case is reported as UNKNOWN, not as a pass or a fail.

Upstream: put the argument back. The change that mirrors the template is a filterKeys bool on writeSchemaProperties, passed true at the one call site where the object itself is being walked and false at the other three:

-func (r *Gemma4Renderer) writeSchemaProperties(sb *strings.Builder, props map[string]any) {
+func (r *Gemma4Renderer) writeSchemaProperties(sb *strings.Builder, props map[string]any, filterKeys bool) {
    …
-       if isSchemaStandardKey(name) {
+       if filterKeys && isSchemaStandardKey(name) {
            continue
        }

Twelve changed lines in total. With that applied, the same-package test renders all seven names, the package's existing tests still pass, and a Go binary rebuilt from the v0.33.3 tag with the patch — dropped in over the stock one, same runner libraries — answers the same requests differently:

type         run1  {"title":"Water leak","type":"urgent_A7"}
type         run2  {"title":"Water leak","type":"urgent_A7"}
description  run1  {"description":"urgent_A7","title":"Water leak"}
description  run2  {"description":"urgent_A7","title":"Water leak"}

That is a one-variable change producing the enum value where the unpatched binary produced a guess, so the mechanism is not in doubt. The probe script reports OK against the patched server and FAIL against the stock one.

One thing this session did not test: nested object parameters and array items end to end with a model. The code reads the same way for those paths — they call the same function with the same unconditional filter, and the static test confirms the top-level case only — but the E2E runs here were top-level parameters.

The generalisable habit

When a template is ported from one language to another, the arguments with default values are the ones that go missing, because at every call site that uses the default the port looks complete. format_parameters(properties, required, filter_keys=false) has three call sites that never mention filter_keys and one that does. A port that reads the three and generalises gets the filter wrong; a port that reads the one and generalises gets it wrong the other way; only a port that carries the parameter gets all four right. The test suite did not catch it because the test cases were written from the same three call sites.

This is the third post here where a local-model server accepted a request, returned 200, and quietly did something other than what the request said — llama-server's README response_format and Ollama's previous_response_id are the other two. The common thread is that the thing being dropped is a name: a JSON key the server does not read, a request field the struct does not have, a parameter the renderer decides is not a parameter. Names are the cheapest thing to get wrong and the last thing a 200 will tell you about. When a model is "bad at" something that depends on it having seen a specific piece of text, the first question is whether it saw the text — and for a renderer with no debug output, the only way to answer that is to render it yourself.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama.cpp reads past its own pos buffer for M-RoPE embeddings, and nothing reports it

The Homelab Postmortem — Tue, 15 Sep 2026 15:06:03 +0000

TL;DR: include/llama.h says the pos array "must have size of n_tokens", and that if you pass NULL "the token position will be tracked automatically". For a model that uses multiple positions per embedding — M-RoPE, which is Qwen2.5-VL and Qwen2.5-Omni — the automatic path sizes its own vector to n_tokens and the batch splitter then reads 4 × n_tokens from it. On master as of today, AddressSanitizer reports a heap-buffer-overflow, READ of size 4, 0 bytes after 8-byte region, at llama-batch.cpp:787, on a vector allocated at llama-batch.cpp:91. In a normal build nothing fires; the bytes that follow the allocation become positions and the model runs on them. An open pull request fixes the NULL path and I have verified it does; the caller-allocated path is fixed only by changing what the header asks for.

The symptom

This one did not present as a symptom to me. It presented as a code-read on the llama.cpp tracker, #28902, where two people had converged on the same diagnosis from the source and a third, running Qwen2.5-Omni under Metal, had the field data: a 750-token audio prefill that "allocated 3,000 bytes and read 12,000", output that was intermittently incoherent, and a failure rate that moved with what else was running on the machine — the signature of a read past the end of an allocation, where the extra bytes depend on what happens to be next to it.

What the thread did not have was a sanitiser trace. "The ASan build would help," one of them wrote. "A heap-buffer-overflow trace turns two code reads into a reproducible report." So that is what this is.

What is really going on

Two lines from include/llama.h, as shipped:

// The provided arrays (i.e. token, embd, pos, etc.) must have size of n_tokens          (line 248)
// - pos : the positions of the respective token in the sequence
//         (if set to NULL, the token position will be tracked automatically ...)         (line 253)

The automatic path, src/llama-batch.cpp, line 91:

if (!batch.pos) {
    pos.resize(batch.n_tokens);

And the batch splitter, lines 780–787, which for an embeddings batch (batch.token == NULL) reads one position per RoPE section:

for (size_t j = 0; j < (size_t)n_pos_per_embd; ++j) {
    // if we are using M-RoPE
    //     if the current batch is text, we need to broadcast the same position across all RoPE sections
    //     otherwise, the input batch is image embeddings, we copy the positions as-is
    size_t src_off = batch.token ? 0 : j*batch.n_tokens;
    udata->pos[j*n_tokens + i] = batch.pos[src_off + idxs[i]];
}

n_pos_per_embd is 4 for M-RoPE. The loop reads batch.pos[0 … 4·n_tokens − 1]. The vector it reads from has n_tokens entries. Both halves of the header are wrong for this model family: a caller who allocates n_tokens gets overread, and a caller who passes NULL gets overread by the library's own fallback. The comment on the loop describes the intent correctly; the intent was never matched by an allocation.

The trace

The fix PR, #28910, adds a 24-line test to the existing tests/test-batch-alloc.cpp: a two-token embeddings batch with pos = NULL against a mock vocab with n_pos_per_embd = 4. That test does not need a model, a GPU, or a Linux box, so I ran it on a Mac mini — master 4c9233c, Apple clang 21, Metal off, -DLLAMA_SANITIZE_ADDRESS=ON — first with only the test applied and the fix left out:

==8100==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000007c18
READ of size 4 at 0x602000007c18 thread T0
    #0 llama_batch_allocr::ubatch_add(...)      llama-batch.cpp:787
    #1 llama_batch_allocr::split_simple(...)    llama-batch.cpp:507
    #2 test_mrope(...) auto_pos_broadcast_for_embd   test-batch-alloc.cpp:640

0x602000007c18 is located 0 bytes after 8-byte region [0x602000007c10,0x602000007c18)
allocated by thread T0 here:
    ...
    #8 llama_batch_allocr::init(...)             llama-batch.cpp:91

SUMMARY: AddressSanitizer: heap-buffer-overflow llama-batch.cpp:787

Eight bytes is two int32 positions — n_tokens = 2. The first read past the end is pos[2], the start of the second M-RoPE section. That is the diagnosis from the thread, with an address on it.

Then with the PR's src/ and include/ changes applied, same build flags:

  auto_pos_broadcast_for_embd (11 assertion(s))                                 [PASS]
failures   : 0

The fix resizes the fallback vector to n_tokens × n_pos_per_embd and broadcasts each auto-generated position across the sections. Eleven assertions, no sanitiser output, exit 0.

Why this is easy to miss

A READ past the end of a heap allocation does not crash. It reads whatever is there. In a release build the next eight, sixteen, or several thousand bytes of the heap become RoPE positions, the model computes attention against them, and the output is a little wrong, or a lot wrong, or fine, depending on what the allocator happened to place there. That is exactly the field report: intermittent, load-dependent, silent. A bug that presents as "the model is flaky on long audio" is not one anybody goes looking for in the batch allocator.

It also survives the obvious defence. Someone who reads the header and allocates pos to exactly n_tokens has done the documented thing. Someone who reads the header and passes NULL has done the other documented thing. Neither one is a bug in their code, and neither one can be fixed in their code without knowing the layout the library actually wants — four section-major planes of n_tokens each, which the header did not say until this PR.

The fix

Upstream, when #28910 merges: the NULL path is repaired, and llama.h gains a note that embeddings batches on n_pos_per_embd > 1 models must supply n_tokens × n_pos_per_embd positions in consecutive sections. Note what that second half is: a documentation change. llama_batch_init() still allocates pos at n_tokens; the caller-side overread is closed by telling callers to allocate more, not by the library allocating more. The thread calls this an API question for the maintainers, and it is.

Until then, if you feed embeddings to an M-RoPE model through the batch API directly — not through mtmd, which already lays positions out correctly — allocate pos yourself at n_tokens × 4 and fill the four planes, as tools/mtmd/mtmd-helper-common.h does. That is what the reporter of the field data did, and it is what turned four intermittently broken runs into four coherent ones.

If you want to know whether your build has it, the test is the check. Apply the PR's tests/ hunk to a checkout, build test-batch-alloc with -DLLAMA_SANITIZE_ADDRESS=ON, and run it. Under a minute after the build, no model required.

There is no toolkit script for this one. The detection is "build with a sanitiser and run the test", and that is not something a script would improve on.

The generalisable habit

The narrow one: when a header documents a size, find the loop that reads the array and check its bound against the same expression. Here they were n_tokens and n_pos_per_embd × n_tokens, in files a few hundred lines apart, and both were individually reasonable.

The wider one is about what a code-read is worth. Two competent people agreed on this bug from reading the source, and they were right — but agreement is not a reproduction, and the thread knew it. A sanitiser trace took twenty minutes and settled it with an address and a line number. When a finding is "we read the code and it looks like it overreads", the next step is not a third reader. It is the same move as removing a constraint to see whether the bytes change: arrange for the thing to be observed rather than inferred, then observe it. The same fifteen lines had a second over-read waiting: when the caller supplies pos, the library reads four times what the header told the caller to allocate, and it took a heap-layout wild-goose chase and a valgrind line to find it.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Ollama's Responses API accepts previous_response_id, returns 200, and forgets the whole conversation

The Homelab Postmortem — Mon, 14 Sep 2026 02:10:04 +0000

TL;DR: send previous_response_id to Ollama's /v1/responses and it is accepted, the reply is HTTP 200 with "status": "completed" and "error": null, and the model has never seen the previous turn. A request that carries previous_response_id costs the same number of input tokens as a request that carries no history — 41 and 41 on this box — and produces the same kind of answer. Sending the full history instead costs 68 and works. The reason is in the source: the request struct has no previous_response_id field, so the JSON decoder throws it away, and the response struct sets it to nil with the comment // Not supported. The bug report that led here, an empty completion from a hosted model after a tool call, is what a bare function_call_output with no context looks like when the backend has nothing to say about it.

The symptom

Ollama 0.34.0, Debian 13, CPU only, qwen2.5:1.5b. The Responses API, which is the OpenAI-compatible endpoint Codex and similar clients use. Turn 1 plants something the model could not guess:

POST /v1/responses
{"model": "qwen2.5:1.5b",
 "input": "My secret word is PINEAPPLE. Remember it. Reply with just OK."}

→ 200  status: completed   input_tokens: 45   "OK"

Turn 2 continues the conversation the way the API is documented to work, by referencing the previous response:

{"model": "qwen2.5:1.5b",
 "previous_response_id": "resp_578667",
 "input": "What is my secret word? Reply with just the word."}

→ 200  status: completed   error: null   input_tokens: 41   "password"

password. Now two controls. The same question with the full history sent in the request body:

input: [user: "My secret word is PINEAPPLE…", assistant: "OK", user: "What is my secret word?…"]

→ 200   input_tokens: 68   "PINEAPPLE"

And the same question with no history at all, just the bare turn-2 text:

input: "What is my secret word? Reply with just the word."

→ 200   input_tokens: 41   "password"

The previous_response_id request and the no-history request cost the same 41 input tokens and produce the same guess. The model was shown the same thing both times: nothing but the question.

The same holds through /api/codex/v1/responses, the proxy path that ollama launch codex sets up for Codex Desktop. 41 tokens, a guess (Qwen, this time), completed.

Why this is easy to miss

Nothing fails. The response is not an error, not incomplete, not empty. It is a well-formed completed response with a plausible answer in it. A conversation client that sends previous_response_id gets a reply to every turn; the replies are just to a different conversation than the one it thinks it is having — one that starts fresh every time.

With a tool-calling loop the shape is worse and more specific. Turn 1: the model emits a function_call. The client runs the tool and sends back only what the protocol says to send back — previous_response_id plus a function_call_output:

turn 1   "Call the check tool once, then reply exactly DONE."   → function_call    input_tokens 144
turn 2   previous_response_id + function_call_output "test passed"
         → 200  completed  "The test passed successfully! Is there anything
           else you need help with?"                              input_tokens 144
control  full history, no previous_response_id
         → 200  completed  "DONE"                                 input_tokens 179

The turn-2 reply is friendly, on-topic, and wrong: the instruction was to say DONE, and the model never saw it. It saw a tool result arrive from nowhere and did its best. That is the local-model version. The upstream report that started this used a hosted :cloud model through the same Codex proxy and got, for the same bare function_call_output, an empty output_text with input_tokens: 0 — which Codex Desktop then read as a cleanly finished turn. That exact symptom did not reproduce here with a local model, and I did not test a hosted one. What did reproduce is the thing underneath it: the context is gone, and what the backend does with a context-free tool result is up to the backend.

What is really going on

openai/responses.go, Ollama 0.34.0. The request type:

type ResponsesRequest struct {
    Model        string
    Input        json.RawMessage
    Instructions *string
    Tools        []ResponsesTool
    ...
    // no PreviousResponseID field
}

Go's encoding/json ignores keys that have no matching field unless DisallowUnknownFields is set. It is not set. previous_response_id is discarded at decode, before any handler sees it.

The response type, on the other hand, does know the field exists:

PreviousResponseID *string `json:"previous_response_id"`
...
PreviousResponseID: nil, // Not supported

So every response carries "previous_response_id": null. That is the one honest signal in the whole exchange, and it is a field no client reads back — OpenAI's implementation echoes the id you sent, so there is nothing to check. The feature request to implement it, ollama/ollama#15954, has been open since May.

A note on the Codex path, because it is easy to get wrong. /api/codex/v1/responses is not a separate implementation. It is a router: a request whose model appears in ~/.codex/ollama-launch-codex-routing.json is forwarded to Ollama's own /v1/responses; any other model is forwarded to OpenAI. On a fresh install with no routing file the endpoint answers 503 read Codex Ollama model catalog … no such file, which is not the bug, it is just ollama launch codex not having run. Write a one-line catalog by hand and the proxy routes a local model exactly as it would a hosted one — and drops previous_response_id exactly the same way, because the drop happens downstream.

The fix

Nothing in Ollama restores the state, so the client has to stop expecting it to. Send the full conversation on every turn:

{"model": "qwen2.5:1.5b",
 "input": [
   {"role": "user", "content": "…turn 1…"},
   {"role": "assistant", "content": "…turn 1 reply…"},
   {"type": "function_call", "call_id": "call_x", "name": "check", "arguments": "{}"},
   {"type": "function_call_output", "call_id": "call_x", "output": "test passed"}
 ]}

That is the 68-token / 179-token control above, and it works on both paths. Most OpenAI-compatible clients already do this; the ones that do not are the ones built against the Responses API's stateful mode — Codex being the prominent case, which is why the upstream report came from Codex Desktop.

Check before you trust a Responses server with state. It takes four requests:

import json, secrets, urllib.request
URL, MODEL = "http://127.0.0.1:11434/v1/responses", "qwen2.5:1.5b"
def post(b):
    r = urllib.request.Request(URL, data=json.dumps(b).encode(), headers={"content-type":"application/json"})
    return json.load(urllib.request.urlopen(r))
def text(d): return " ".join(c["text"] for o in d["output"] if o["type"]=="message" for c in o["content"])
word = "".join(secrets.choice("BCDFGHJKLMNPQRSTVWXZ") for _ in range(7))
plant, ask = f"My secret word is {word}. Reply with just OK.", "What is my secret word? Reply with just the word."
r1 = post({"model": MODEL, "input": plant, "stream": False})
r2 = post({"model": MODEL, "previous_response_id": r1["id"], "input": ask, "stream": False})
r4 = post({"model": MODEL, "input": ask, "stream": False})
print(word, "| via id:", r2["usage"]["input_tokens"], repr(text(r2)), "| no history:", r4["usage"]["input_tokens"], repr(text(r4)))

LNLPFVF | via id: 41 'Unlimited possibilities' | no history: 41 'Password'

If the id-referencing request costs the same as the bare one and does not know the word, the server is not keeping state, whatever the status field says. The toolkit's check-responses-state.sh runs this with a full-history control as well, so a model too small to remember the word is reported as unknown rather than as a pass or a fail.

The generalisable habit

The narrow one: a status of completed describes the request the server processed, which may not be the request you sent. Here the server processed "what is my secret word?" and completed it perfectly. The previous_response_id was not rejected, not warned about, not incomplete_details-ed. It was not there by the time anything could have complained. The same server does it one layer down, too: its gemma4 renderer drops any tool parameter named type or description from what the model sees while still listing it as required, and the model fills the gap with a guess — same 200, same silence, a name dropped in the prompt instead of in the request.

The wider one is about controls that cost nothing. The token count did all the work in this diagnosis. Two requests that should have carried different amounts of context cost the same, and one that should have carried the same as the first cost more. That number is in every response, it is exact, and it does not depend on interpreting model output. When a stateful API might be stateless, count the tokens before reading the words. It is the same move as holding the prompt fixed and removing the constraint to see whether the bytes change: find the one number that has to move if the feature is real, and watch whether it moves.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

docker logs stops at a NUL byte, exits 0, and --since says nothing happened

The Homelab Postmortem — Sat, 12 Sep 2026 00:46:48 +0000

TL;DR: with the default json-file log driver, a run of NUL bytes anywhere in the log file makes docker logs <container> stop at that point and report success. On Docker 29.8.0 with one of 50 records zeroed, the full read returned 17 lines, exit 0, stderr empty. --tail N works only when its window begins after the corruption. --since <time> reads forward from the start, hits the NUL, and returns zero lines for any timestamp before the end of the file — which after a crash reads as "nothing happened since then." The daemon logs exactly one warning that never reaches the client. The reader code returns EOF on a decode error with a comment saying this is so it can move to the next file; when the corruption is in the last or only file, there is no next file, and the read just ends.

The symptom

Docker Engine 29.8.0, Debian 13, a throwaway LXC container, json-file driver with no options — the default. A busybox container printing numbered lines every 100 ms, stopped after 50:

$ docker logs logtest | wc -l
50
$ docker logs logtest | tail -1
line 49

Then one record in the middle of the log file — number 17 of 51 — overwritten with NUL bytes of the same length, so the file size does not change:

record index 17 of 51: b'{"log":"line 17\n","stream":"stdout","time":"2026-09-12T00:1'
size before: 3889 bytes   size after: 3889 bytes
NUL bytes in file: 77

Now the same read, with stdout, stderr and the exit code captured separately rather than through a pipe:

$ docker logs logtest > out 2> err; echo $?
0
$ wc -l < out; wc -c < err
17
0
$ tail -1 out
line 16

Seventeen lines, exit zero, nothing on stderr. The other thirty-three records are intact on disk, and the client has no idea they exist.

--tail looks like it works, and that is the trap:

docker logs --tail 5    logtest    exit 0   5 lines    line 45 … line 49
docker logs --tail 30   logtest    exit 0   30 lines   line 20 … line 49
docker logs --tail 40   logtest    exit 0   7 lines    line 10 … line 16
docker logs --tail 100  logtest    exit 0   17 lines   line 0  … line 16

A window that starts after record 17 is fine. A window that includes it stops at it. You do not know where the corruption is, so you do not know which of those you are getting.

Why this is easy to misdiagnose

Two things make this worse than a truncated file.

The tail-vs-full disagreement points at the wrong thing. --tail 50 shows fresh lines. The plain read ends in the past. The natural conclusion is that something is wrong with the reader — a client version mismatch, a buffering problem, a terminal issue — because the file is evidently still being written. It is not a reader problem. The file has a hole in it, and only one of the two commands walks through the hole.

--since gives an answer that is wrong in the most plausible direction. This is the one the upstream report did not mention. Take the timestamp of record 20, three records past the corruption, and ask for everything since:

$ docker logs --since 2026-09-12T00:11:23.772318762Z logtest > out; echo $?
0
$ wc -l < out
0

Zero lines, exit zero. There are thirty records after that timestamp on disk. --since still reads forward from the beginning of the file, hits the NUL region at record 17, and stops before it reaches anything new enough to match. The control, with the corrupt record removed and the same timestamp:

$ docker logs --since 2026-09-12T00:11:23.772318762Z logtest | wc -l
30

So the exact command you would run after a power loss — "what was this container doing in the minutes before it died?" — returns the same output as "it was doing nothing." Not an error. Not a truncated list. Empty.

The daemon knows. journalctl -u docker on the host has one line:

level=warning msg="Error decoding log file" error="invalid character '\x00' looking for beginning of value"

Nothing carries that to the client, and a warning-level line in the daemon journal is not where anyone looks when docker logs returns cleanly.

What is really going on

daemon/logger/loggerutils/logfile.go, in the function that tails the log files, as of master on 2026-09-11:

ok := fwd.Do(ctx, watcher, func() (*logger.Message, error) {
    msg, err := dec.Decode()
    if err != nil && !errors.Is(err, io.EOF) {
        // We have an error decoding the stream, but we don't want to error out
        // the whole log reader.
        // ...
        // Instead just log the error here and return an EOF so we can move to
        // the next file.
        log.G(ctx).WithError(err).Warn("Error decoding log file")
        return nil, io.EOF
    }
    return msg, err
})

The comment describes the design: a decode error is converted into EOF so that the reader can give up on this file and continue with the next one in the rotation. That is reasonable when there is a next file. When the corruption is in the last file — or the only file, which is every container that has not rotated — "move to the next file" means "finish." The warning is logged, the EOF is returned, the read completes, and completion is success.

--tail escapes this because it seeks backward from the end of the file and only decodes forward from there. If the seek lands after the NUL region, the decoder never sees it. --since does not seek; it filters, and the filter never gets input past the hole.

Where the NUL bytes come from is not something I reproduced. I wrote them into the file with Python. What I can say is what the upstream tracker says: the issue this came from, moby/moby#53631, got its zeroed region from a Docker Desktop VM stopping uncleanly. The eight-year-old sibling issue about half-written records, moby/moby#29511, has a maintainer comment from 2024 that is the part relevant here:

There are other reports of the exact behavior with the 'local' log driver on rpi's after abrupt power loss … The fact that it is writing null bytes when this occurs … makes it extra suspicious

A filesystem that has extended the file but not yet written the data — ext4 with delayed allocation, power gone before the flush — leaves exactly this: the right length, zeros where the bytes should be. On a Raspberry Pi on a wall adapter with no UPS, that is not an exotic scenario. It is the scenario.

The fix

Upstream has not fixed it; the decode-error path has looked like this for years and #53631 is a day old with no comments. Two things you can do on your own machine.

Repair a stopped container's log by dropping the records that contain NUL bytes. Those records were never recoverable — the bytes are zeros — but everything after them is, and this is what brings it back:

LOG=$(docker inspect --format '{{.LogPath}}' logtest)
cp -p "$LOG" "$LOG.bak-$(date +%Y%m%d-%H%M%S)"
python3 - "$LOG" <<'PY'
import sys
p = sys.argv[1]
lines = open(p, 'rb').read().split(b'\n')
kept = [l for l in lines if b'\0' not in l]
open(p, 'wb').write(b'\n'.join(kept))
print(f"removed {len(lines) - len(kept)}, kept {len(kept)}")
PY

removed 1, kept 50
$ docker logs logtest | wc -l
49

Restoring the corrupt file brings the 17-line read straight back, so the repair and the fault are cleanly attributable to that one record.

Check before you trust a read, especially after a crash. The file is the ground truth and it is readable without going through the daemon at all:

LOG=$(docker inspect --format '{{.LogPath}}' "$C")
echo "NUL bytes: $(tr -cd '\000' < "$LOG" | wc -c)"
echo "last on disk: $(grep -avP '\x00' "$LOG" | tail -1 | python3 -c 'import sys,json; print(json.loads(sys.stdin.read())["log"].rstrip())')"
echo "last from docker: $(docker logs "$C" 2>/dev/null | tail -1)"

On the corrupted container above:

NUL bytes: 77
last on disk: line 49
last from docker: line 16

(grep -P '\x00' rather than a shell $'\0' — a NUL cannot be passed as a shell argument, so the latter silently matches the empty string and excludes everything. That version was wrong the first time I wrote it.)

If the NUL count is nonzero, or the two last lines disagree on a stopped container, the full read is stopping short. The toolkit's check-docker-log-integrity.sh is this comparison across rotated files with the exit code read directly rather than through a pipe, and the repair printed for you.

The generalisable habit

The narrow lesson is that docker logs reports whether the read finished, not whether it read everything, and those diverge exactly when you most need them not to. After a power loss, treat the log file as the record and docker logs as one view of it that may end early.

The wider one is about comments that describe a design and code that runs outside it. The EOF-on-decode-error path is correct for the case its author had in mind — rotated files, skip the bad one, keep going. Nobody wrote a lie. The code simply also runs in the case where there is nothing to keep going to, and in that case the same three lines mean "stop and say nothing." The same shape as a cloud-init module that reports SUCCESS for the run it skipped: the status word is true about the control flow and says nothing about your intent. And it sits next to a journal that is discarded on every reboot as a second way to find, after the crash, that the logs you were counting on are not there — this time with the bytes still on disk.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama-server ignores the response_format its own README shows, and returns 200

The Homelab Postmortem — Fri, 11 Sep 2026 17:16:33 +0000

TL;DR: tools/server/README.md gives {"type": "json_schema", "schema": {...}} as an example of schema-constrained output. On a build from this week the reply to that form is byte-identical to the reply when you send no response_format at all — same SHA-256, same 248 characters of prose. HTTP 200, nothing in the log. The json_schema branch of the request parser reads only response_format.json_schema.schema; a schema sitting directly under response_format is never consulted in that branch. The March 2025 fix that people cite for this fixed the nested form and never touched this one, and the README example was already there when it merged. Two other forms constrain correctly and are the workaround.

The symptom

llama.cpp b10868 (commit 304665fe7, released 2026-09-09), the official Linux x64 binary, on Debian 13, CPU only. Model is gemma-3-1b-it-Q4_K_M, small enough that the whole experiment takes a minute. One prompt, one schema, temperature 0, seed 42, four ways of asking:

prompt   What is 2+2? Explain your reasoning in a few sentences.
schema   {"type":"object","properties":{"answer":{"type":"integer"}},
          "required":["answer"],"additionalProperties":false}

The prompt is chosen to produce prose if nothing stops it. The schema allows exactly one integer field and nothing else. Here is what came back, hashed:

                                                         sha256        result
no response_format                                       5dc606573325  free text
{"type":"json_object","schema":{...}}                    72241d50124b  {"answer": 4}
{"type":"json_schema","schema":{...}}         [README]   5dc606573325  free text
{"type":"json_schema","json_schema":{"schema":{...}}}    72241d50124b  {"answer": 4}

Two hashes for four requests. The README form did not produce worse structured output, or partially constrained output. It produced the control, to the byte:

2 + 2 equals 4. This is a fundamental mathematical concept that relies on
basic addition – combining two individual units to create a combined unit
with a value of four. It's a core principle used in …

Every request returned HTTP 200. The server log for the four of them is four ordinary launch_slot_ / print_timing / release triples with nothing about a schema, a grammar, or an empty anything.

Why this is easy to miss

You would not notice, because models cooperate. Ask a capable model for JSON in the prompt and it usually gives you JSON, so a response_format that does nothing looks like a response_format that works — right up to the request where the model decides to add a sentence of explanation, or a trailing comma, or a field you did not ask for, and something downstream falls over. At that point the natural question is "why is the model misbehaving?" and not "was the constraint ever applied?", because the server has been answering 200 the whole time.

The second false trail is the issue tracker. Search for this and you find #10732 ("server provides structured output for json_object, but not for json_schema", December 2024) and, next to it, #11988, closed as completed in March 2025 with a merged fix, #12168, and a reporter confirming "the issue is not present in the new b4820." That reads like the end of the story. It is not, and the reason is in the diff.

What is really going on

This is the request parser, tools/server/server-common.cpp, as shipped in b10868:

if (response_type == "json_object") {
    json_schema = json_value(response_format, "schema", json::object());
} else if (response_type == "json_schema") {
    auto schema_wrapper = json_value(response_format, "json_schema", json::object());
    json_schema = json_value(schema_wrapper, "schema", json::object());
} else if (!response_type.empty() && response_type != "text") {
    throw std::invalid_argument(...);
}

The json_object branch reads response_format.schema. The json_schema branch reads response_format.json_schema.schema and only that. Send {"type":"json_schema","schema":{...}} and schema_wrapper defaults to {}, json_schema defaults to {}, and generation proceeds with an empty schema — which is to say, no grammar. No branch throws, because the type was recognised.

Now the 2025 fix. This is the whole of what #12168 changed in that function:

-            json json_schema = json_value(response_format, "json_schema", json::object());
-            json_schema = json_value(json_schema, "schema", json::object());
+            auto schema_wrapper = json_value(response_format, "json_schema", json::object());
+            json_schema = json_value(schema_wrapper, "schema", json::object());

A shadowed variable. The inner json_schema was hiding the outer one, so the nested wrapper form silently did nothing. The fix un-shadowed it, the nested form started working, the test it added uses the nested form, and the issue closed. The top-level form was not part of that bug and not part of that fix. It has not regressed; on this code path it has never worked.

And the README already showed it. At the merge commit of #12168, examples/server/README.md line 1076 read:

The response_format parameter supports both plain JSON output (e.g. {"type": "json_object"}) and schema-constrained JSON (e.g. {"type": "json_object", "schema": {...}} or {"type": "json_schema", "schema": {...}})

The same sentence is at tools/server/README.md line 1316 today. Eighteen months of a documented example that the parser does not read, sitting one clause away from a documented example that it does.

There is a pull request that fixes it, #28697, which adds a fallback to response_format.schema when the wrapper is absent. As of this writing it is open with no reviews. I built its branch (48f9bfd) on the same machine and ran the same four requests: the README form now hashes 72241d50124b, identical to the two forms that already worked, and those two are unchanged. The patch does what it says.

The fix

Until that merges, use either of the two forms that do constrain on this build. Both were verified above with the same hash:

{"type": "json_object", "schema": { ... }}

{"type": "json_schema", "json_schema": {"name": "x", "schema": { ... }}}

The second is the OpenAI wrapper shape and is what most client libraries emit, which is probably why this has survived so long — the people who would hit it are the ones who read the README and wrote the request by hand.

Check the server you actually run, not the one in the docs. Four requests, compared by hash, is the whole test, and it takes seconds against any running llama-server:

import json, hashlib, urllib.request
S = {"type":"object","properties":{"answer":{"type":"integer"}},"required":["answer"],"additionalProperties":False}
forms = {
  "none":        None,
  "json_object": {"type":"json_object","schema":S},
  "readme":      {"type":"json_schema","schema":S},
  "openai":      {"type":"json_schema","json_schema":{"name":"x","schema":S}},
}
for name, rf in forms.items():
    body = {"messages":[{"role":"user","content":"What is 2+2? Explain your reasoning in a few sentences."}],
            "temperature":0,"seed":42,"max_tokens":200}
    if rf: body["response_format"] = rf
    req = urllib.request.Request("http://127.0.0.1:8080/v1/chat/completions",
            data=json.dumps(body).encode(), headers={"content-type":"application/json"})
    c = json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]
    print(f"{name:12} {hashlib.sha256(c.encode()).hexdigest()[:12]}  {c[:40]!r}")

Output on b10868, gemma-3-1b:

none         5dc606573325  '2 + 2 equals 4. This is a fundamental ma'
json_object  72241d50124b  '{\n \t \t \t \t \t \t \t \t \t \t"answer": 4\n \t \t \t'
readme       5dc606573325  '2 + 2 equals 4. This is a fundamental ma'
openai       72241d50124b  '{\n \t \t \t \t \t \t \t \t \t \t"answer": 4\n \t \t \t'

If any form with a schema hashes the same as null, that form is not being enforced on your build. The toolkit's check-llama-response-format.sh does exactly this, with the control checked first so a model that happens to answer in JSON unprompted cannot fake a pass.

The generalisable habit

The narrow one: a fix for a sibling form is not a fix for your form. #12168 was real, was merged, and was confirmed by the person who reported it. It also had nothing to do with the request shape the README shows. When a closed issue seems to cover your case, read the diff and find your exact input in it; if it is not there, the issue was about something else that happened to share a title.

The wider one is about what "it worked" is evidence of. A structured response from a model can mean the server constrained the output, or it can mean the model produced it unconstrained because you asked nicely — and from the outside those are the same bytes. The only way to tell them apart is to hold the prompt fixed and remove the constraint: if the output does not change, there was no constraint. That is the same move as running the control before the suspect and the same shape as an adapter that is accepted and applied to zero layers: the system reports that it did what you asked, and the output looks right, and neither of those is the thing you needed to know.

Two days later the same shape showed up one layer over: Ollama's Responses API accepts previous_response_id, returns 200 and "completed", and starts every turn from nothing. There the number that has to move is the input token count, and it does not.

And three days after that, the layer below: llama.cpp's batch API says pass pos as NULL and positions are tracked automatically; for M-RoPE embeddings it reads past its own buffer. No number to watch that time — a sanitiser, and an address.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

cloud-init never reads the instance-id Raspberry Pi OS sets, so your second user-data is skipped

The Homelab Postmortem — Fri, 11 Sep 2026 14:56:30 +0000

TL;DR: write a plain Raspberry Pi OS image to a disk, boot it once, then add a user-data with a users: block and boot again. The user is not created. The file is not the problem: cloud-init parses it, merges it, and writes the merged result to disk with your block in it. It is skipped because users_groups is a once-per-instance module and the first boot already ran it — against the stock user-data, which is 3,277 characters of comments and parses to nothing. The instance never changes, because the stock meta-data writes instance_id and cloud-init reads instance-id. The skipped modules are logged as SUCCESS.

The symptom

A USB SSD, freshly written with Raspberry Pi OS Lite (Trixie, arm64) and booted once to confirm it came up. Then a user-data placed on the boot partition, by hand:

#cloud-config
hostname: mitoyosh-pi4b-ssd
users:
  - name: mito
    groups: users,adm,dialout,audio,netdev,video,plugdev,cdrom,games,input,gpio,spi,i2c,render,sudo
    shell: /bin/bash
    lock_passwd: true
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyMaterialHere
ssh_pwauth: false

Boot. No SSH. Mount the disk read-only from another machine and the reason is plain:

$ grep -E "^mito:" /mnt/ssdroot/etc/passwd
  mito は存在しない
$ ls -l /mnt/ssdroot/home/mito/.ssh/authorized_keys
  ls: cannot access: No such file or directory

The obvious next move is to suspect the file. It survives every check:

file        Unicode text, UTF-8 text      (no BOM)
CR count    0                             (no CRLF)
line 1      #cloud-config$                (cat -A — nothing before it)
yaml        type: dict
            keys: ['hostname', 'packages', 'ssh_pwauth', 'users']

And cloud-init agrees. This is the merged config it wrote to /var/lib/cloud/instances/<id>/cloud-config.txt on that boot:

#cloud-config
# from 1 files
# part-001
hostname: mitoyosh-pi4b-ssd
users:
-   name: mito
    ssh_authorized_keys:
    - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyMaterialHere
    sudo: ALL=(ALL) NOPASSWD:ALL
...

The block is there. It was read, parsed, merged, and written back out. And the user does not exist.

Why this is easy to misdiagnose

Two things point you away from the answer.

The hostname changes. /etc/hostname really does become mitoyosh-pi4b-ssd. update_hostname runs with frequency always, so it applies on every boot regardless of instance. users_groups runs once-per-instance, so it does not. Half your user-data takes effect, which is a strong signal that the file is being read — and it is, just not acted on.

The failure surfaces as a network problem. No SSH looks like no network, and if the machine has no route out, you will find plenty of networking to be wrong. On this disk /etc/netplan/ was empty and NetworkManager.state had WirelessEnabled=false, both true and both irrelevant to the missing user. I spent an hour building a cloud-init explanation for the network symptom before checking whether an ethernet cable was plugged in. It was not.

The log is no help either, because it reports success:

09:32:20  Attempting to load yaml from string of length 773
09:32:20  Merging by applying [('dict', ['replace']), ('list', []), ('str', [])]
09:32:20  config-users_groups already ran (freq=once-per-instance)
09:32:20  finish: init-local/config-users_groups: SUCCESS: config-users_groups previously ran

It loads your YAML. It merges your YAML. Then it declines to act on it and records SUCCESS.

What is really going on

Every once-per-instance module leaves a semaphore. Theirs are all dated from the first boot:

$ ls -l /var/lib/cloud/instances/nocloud/sem/
config_locale          Jun 18 09:27
config_mounts          Jun 18 09:27
config_seed_random     Jun 18 09:27
config_set_hostname    Jun 18 09:27
config_set_passwords   Jun 18 09:27
config_ssh             Jun 18 09:27
config_ssh_import_id   Jun 18 09:27
config_users_groups    Jun 18 09:27
consume_data           Jun 18 09:27

That first boot had the stock user-data, which ships as pure comments:

09:27:16  Attempting to load yaml from string of length 3277
09:27:16  loaded blob returned None, returning default.
09:27:16  WARNING: Failed at merging in cloud config part from part-001: empty cloud config

Nine modules ran against an empty configuration and marked themselves done. The seats were taken before you sat down.

Normally that is fine, because a new instance gets a new id and a fresh set of semaphores. Here the id never changes. This is the stock meta-data, verbatim:

# Specifies the "unique" identifier of the instance. Typically in cloud-init
# this is generated by the owning cloud and is actually unique (to some
# degree). Here our data-source is local, so this is just a fixed string.
# Warning: changing this will cause cloud-init to assume it is running on a
# "new" instance, and to go through first time setup again (the value is
# compared to a cached copy).
instance_id: rpios-image

The comment is correct. The key below it is not. cloud-init reads instance-id, with a hyphen. From cloudinit/sources/DataSourceNoCloud.py:

# line 65 — the default when nothing supplies one
    "instance-id": "nocloud",

# line 284
    iid_key = "instance-id"

So instance_id: rpios-image is never seen, the id becomes the literal string nocloud, and the instance directory is /var/lib/cloud/instances/nocloud on every such disk. Change the value the comment tells you to change and nothing happens, because the line it is attached to was never read in the first place.

The control is the same OS on a card written by Raspberry Pi Imager:

$ cat /boot/firmware/meta-data
instance-id: rpi-imager-1787993916805

Hyphen, and unique. Imager also passes it on the kernel command line, which is a second route to the same field:

$ cat /boot/firmware/cmdline.txt
... rootwait ds=nocloud;i=rpi-imager-1787993916805 cfg80211.ieee80211_regdom=JP

That i= maps to instance-id too — s2l = {"h": "local-hostname", "i": "instance-id", "s": "seedfrom"}, line 360 of the same file. A customised image has two ways to set the id. A plain image write has none.

The fix

Change the key, and give it a value that has not been used before:

# /boot/firmware/meta-data
dsmode: local
instance-id: ssd-boot-test-20260909

That is the whole fix. /var/lib/cloud was deliberately not deleted, so that the result attributes cleanly to the one character:

before                          after
instances/nocloud               instances/nocloud
                                instances/ssd-boot-test-20260909
sem: 9 files                    sem: 18 files
no mito                         mito:x:1000:1000::/home/mito:/bin/bash
no authorized_keys              /home/mito/.ssh/authorized_keys  0600 mito:mito
no boot-finished                boot-finished
                                cloud-init status: done

Nine semaphores became eighteen, the user was created, the key was installed, and cloud-init ran through modules-final to completion. The old instance directory is still sitting next to the new one.

Check before you boot, not after. Both of these are cheap:

# 1. Is the key name one cloud-init will read?
grep -E '^instance[-_]id:' /boot/firmware/meta-data
# instance_id: rpios-image   <- underscore. Never read.

# 2. Has this disk already burned its once-per-instance modules?
#    (mount the rootfs read-only somewhere first)
ls /mnt/root/var/lib/cloud/instances/
# nocloud   <- the generic id, so yes, and a new user-data will not run

If the second one lists anything, a fresh user-data will be read and ignored unless you change the instance id.

The generalisable habit

The narrow lesson is to check the key name. The wider one is about which artifact you are trusting.

This file documents its own mechanism correctly, in a comment written by someone who understood it, and then names the key wrong on the next line. Following the instructions in a file is not the same as the file working. The prose and the code in a config template are maintained by the same hand but validated by different things — the prose by nobody, the key name by whatever reads it, silently, at boot.

That is the same shape as a cloud-init key whose valid placement is exactly opposite between two documented forms, where cloud-init schema calls the file valid either way. Both cases pass every check that looks at the file, because the thing that is wrong is the relationship between the file and the reader, and no validator holds both.

And the operational habit, which cost the most time here: when a tool reports SUCCESS for work it skipped, the word is describing the module's control flow, not your intent. config-users_groups previously ran is a true statement. It is also the only notice you will get that the configuration you just wrote is never going to execute.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

vLLM accepts a LoRA it will never apply, and answers with the base model instead

The Homelab Postmortem — Mon, 07 Sep 2026 07:26:11 +0000

Update (2026-09-13): the fix has landed. vllm-project/vllm#55310 merged on 2026-09-09 and closed the issue, so vLLM built from main after that commit rejects an adapter whose modules fall entirely outside --lora-target-modules instead of accepting it and applying it to nothing. Released versions up to and including 0.28.0 — the one this post reproduced on — still have the behaviour described below. The toolkit's check-lora-target-overlap.sh remains the way to catch it on those.

TL;DR: vllm serve --lora-target-modules qkv_proj plus an adapter that only touches down_proj is a combination vLLM accepts. It loads the adapter, compiles the LoRA kernels, and then wraps zero layers with it — every request comes back as the unmodified base model, byte for byte, with no error, no rejection, and no warning that mentions target_modules. The acceptance path reads supported_lora_modules; the application path reads that and lora_config.target_modules. The second list is the one you set on the command line, and only one of the two checks has heard of it.

The symptom

Same adapter file, same prompt, temperature 0, seed 42. The only thing that changes between these two runs is the deployment's allow-list.

--lora-target-modules down_proj      (overlaps the adapter)

  base : " Paris. It is the largest city in the world by population. It is
          located in the south of France. It is"
  lora : ":\nA. Paris\nB. London\nC. Rome\nD. Moscow\nAnswer:\n\nA\n\nWhich of the"

--lora-target-modules qkv_proj       (disjoint from the adapter)

  base : " Paris. It is the largest city in the world by population. It is
          located in the south of France. It is"
  lora : " Paris. It is the largest city in the world by population. It is
          located in the south of France. It is"

The second pair is byte-identical. The request carried a LoRARequest, it returned 200, and it was served by the base model.

Verified on vLLM 0.28.0, Qwen/Qwen2.5-0.5B-Instruct, an RTX 2070 under WSL2.

It is not that the adapter failed to load

That was the first hypothesis and it is wrong, which matters, because "my adapter path was bad" sends you looking in a place where there is nothing to find. vLLM stands up the entire LoRA machinery for a request it is about to serve without any LoRA in it:

WARNING [jit_monitor.py:141] Triton kernel JIT compilation during inference: _lora_shrink_kernel
WARNING [jit_monitor.py:141] Triton kernel JIT compilation during inference: _lora_expand_kernel

Those two kernels are LoRA. A LoRA layer computes y = Wx + B(Ax): shrink is Ax, projecting down to the adapter's rank, and expand is B(...), projecting back up. Both were compiled, for this request, on the run that returned base-model output. The adapter was read, registered and activated. It was simply attached to nothing.

So the search space is not "why was my adapter rejected". Nothing was rejected.

What's really going on

Two checks, and they do not read the same thing. From the installed 0.28.0, not from the report:

>>> "supported_lora_modules" in inspect.getsource(WorkerLoRAManager._load_adapter)
True
>>> "target_modules" in inspect.getsource(WorkerLoRAManager._load_adapter)
False

The acceptance path validates the checkpoint against supported_lora_modules — the set of module types vLLM can wrap for this architecture at all. qkv_proj, o_proj, gate_up_proj, down_proj for a Qwen2 model. An adapter targeting down_proj passes, because down_proj is a thing vLLM knows how to wrap.

The application path asks a second question:

def _match_target_modules(self, module_name: str) -> bool:
    if not is_supported_lora_module(module_name, self.supported_lora_modules):
        return False
    return is_in_target_modules(
        module_name,
        self.lora_config.target_modules,
        self.packed_modules_mapping,
    )

lora_config.target_modules is --lora-target-modules. It is consulted here and nowhere upstream of here. So the adapter is judged twice, against a permissive list and then a restrictive one, and only the permissive judgement can produce a message. By the time the restrictive one runs, the answer is not "reject this adapter" — it is "wrap this module: no", asked once per module, and every answer is no.

Zero wrapped modules is not an error state anywhere. It is just a loop that did nothing.

Who this reaches, and who it does not

Not everyone, and the shape of who matters, because it explains why a bug this loud in its consequences is this quiet in the wild.

If you never pass --lora-target-modules, you cannot hit this. The allow-list then defaults to everything the architecture supports, so the restrictive list and the permissive list are the same list and the two checks agree. Most single-adapter deployments look like this.

You hit it by deliberately narrowing the deployment — restricting to qkv_proj to bound memory, say, or to keep a serving profile stable across adapters — and then loading an adapter someone else trained, whose modules you did not check against your own restriction. That is a multi-tenant, many-adapters shape. It is also exactly the shape where nobody is reading individual responses closely enough to notice they got the base model.

The fix

Upstream has fixed it: vllm-project/vllm#55310, "Reject adapters with no matching target modules", was open when this was written and merged on 2026-09-09. On any release up to 0.28.0, nothing in vLLM will tell you.

Check the two lists against each other before you deploy. The adapter states its own in adapter_config.json:

python3 -c '
import json,sys
cfg = json.load(open(sys.argv[1] + "/adapter_config.json"))
adapter = set(cfg["target_modules"])
deployed = set(sys.argv[2].split(","))
print("adapter :", sorted(adapter))
print("deployed:", sorted(deployed))
print("overlap :", sorted(adapter & deployed) or "NONE — this adapter will be ignored")
' /path/to/adapter qkv_proj,o_proj

Note that packed modules make the comparison less obvious than it looks: an adapter naming q_proj, k_proj and v_proj overlaps a deployment naming qkv_proj, because vLLM fuses them. A plain string comparison will report a false alarm there.

And confirm it end to end, once, with a prompt you know the answer to. Send the same prompt with and without the LoRARequest at temperature 0:

identical output  ->  the adapter is doing nothing
different output  ->  it is applied

That is the only signal that cannot be faked by a mechanism that reports success. It costs two requests.

If you are building the adapter yourself to test this, know that peft initialises lora_B to zeros, so a freshly created, untrained adapter produces identical output whether or not it is applied. Mine had to be given non-zero lora_B deliberately, or the control would have proved nothing.

The generalisable habit

The narrow lesson is about validation that runs before the decision it is supposed to guard. Accepting an adapter and applying an adapter were separated by a config field that only the second step could see, and only the first step had a voice. Any time a system says yes in one place and acts in another, the interesting question is not "does it validate?" but "does the thing that validates know everything the thing that acts knows?" Here it did not, by one field.

The wider one is about what counts as a control. My first instinct was to load the adapter, see base-model output, and call it reproduced. That would have been worthless: an adapter that does nothing when ignored also does nothing when applied, if it was never trained. The result only means something because the same adapter file changed the output under a different allow-list. A negative result is evidence only after the positive one has been shown on the same setup — otherwise you have measured your own test rig.

It is the same trap as an Ollama tag that pulls and runs and has no working code in
it,
where the control had to run first to catch a broken harness. Both times the failing observation was
available immediately and agreed with the report, and both times it would have been the wrong reason.

The same "accepted, then not applied" shape turned up in llama.cpp a few days later:
llama-server takes the json_schema form its own README documents, returns 200, and
generates as if no response_format was sent.
There the control is even simpler — remove the constraint and see whether the bytes change —
and it is the only way to tell a server that enforced from a model that cooperated.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

An Ollama tag that pulls and runs and has no working code in it. The same model at the same quantisation, elsewhere, is fine.

The Homelab Postmortem — Sun, 06 Sep 2026 23:08:12 +0000

Update (2026-09-09): the reporter has published the full audit as a
preprint — Broken on Arrival: Silently Defective LLM Artifacts in Public Model
Registries and How to Catch Them (Aditi
Patodiya, 2026), with the dataset and tooling at
aditi-p31/quantcheck. It executed 327
quantised code-capable artifacts, 305 of them from the official Ollama library,
and confirmed silently defective artifacts in the official library, including
the batch of four Qwen2.5-Coder-3B conversions that the q3_K_M below belongs
to. This post only ever tested q3_K_M — the three siblings are the
census's result, not mine.

Correction (2026-09-10): the paragraph above first said five defects,
including phi3.5:3.8b-mini-instruct-q2_K. The author has since retracted the
phi3.5 case and the count is four. The retraction came out of the control this
site asked for: an independent conversion of phi3.5 at q2_K with no
importance matrix fails identically (0/15), which is genuine capability
collapse at that quantisation, not a bad file. The original referee had been an
imatrix build, and that is what made phi3.5 look like an outlier. The four
Qwen artifacts came through the same re-run stronger, not weaker — the model
author's own no-imatrix conversions score 14/15 at q2_K and 15/15 at
q3_K_M, where the library artifacts score zero. A revised preprint is going
to arXiv.

Worth carrying away, because it is what caused the error: the replacement
referee turned out to be the same file under a different name. The library's
phi3.5 q2_K blob is byte-identical to uploads in two separate HuggingFace
repositories — verified here without downloading anything, since Ollama's
registry manifest gives the layer digest and HuggingFace returns the file's
SHA-256 in the x-linked-etag header:

ollama library  phi3.5:3.8b-mini-instruct-q2_K   54d47caa8bf3…
QuantFactory/Phi-3.5-mini-instruct.Q2_K.gguf     54d47caa8bf3…
neopolita/phi-3.5-mini-instruct_q2_k.gguf        54d47caa8bf3…
bartowski/Phi-3.5-mini-instruct-Q2_K.gguf        7425cb5fec0d…   (imatrix build)

A different repository is not evidence of a different conversion. Only the
hash settles it, and you can get the hash without transferring the file.

One of its findings is a caveat on my own advice, so it goes at the top rather
than the bottom. Two of the five defects produce output whose surface
statistics sit inside the healthy range. The ten-second check further down
this page — look for a missing def and return — is calibrated against the
artifact I actually had, where the output was a stray . and a stray 00. It
would not catch those two. Reading the output is a filter, not a test; if you
want a test, run what comes back, which is what the toolkit
script does and what the census does.

TL;DR: ollama pull qwen2.5-coder:3b-instruct-q3_K_M succeeds, ollama run streams at normal speed, and nothing anywhere reports a problem. On three trivial HumanEval-style tasks it produced no implementation at all — 0/3. The obvious conclusion is that 3B at q3 is simply too degraded to code. That conclusion is wrong, and one control disproves it: the official Qwen GGUF at the same q3_K_M level, imported with the library model's own template so only the weights differ, scored 3/3 on the same tasks. The failure is in that conversion, not in the quantisation level.

The symptom

Debian 13, Ollama 0.33.3, CPU only. Ask for the simplest function in HumanEval:

Please provide a self-contained Python script that solves the following
problem in a markdown code block:

def sum_to_n(n: int) -> int:
    """ sum_to_n is a function that sums numbers from 1 to n. """

qwen2.5-coder:3b-instruct-q3_K_M answers, deterministically, with 85 characters:

.
00











 to_n: function to_n:

A stray ., a stray 00, two floods of newlines, and to_n: function to_n: —
the identifier from the prompt echoed back. No def. No return. Nothing to run.

It is not a fluke or a sampling artefact. At temperature 0 and seed 42, two
runs produced byte-identical output (sha256 ec98f76dfeeda2a3…).

Everything around it reported success. The pull completed and verified, the
model is 1.6 GB on disk as advertised, load time and token rate look normal, and
the HTTP API returns 200 with a populated response field. The only thing wrong
is the content.

Why "q3 is just too small" is the wrong conclusion

That was the first hypothesis here, and it is the one most people will reach,
because it is usually right. Low-bit quantisation of a small model genuinely
does fall apart — it is a known trade-off, it is discussed everywhere, and a
3B model at 3 bits is exactly where you would expect it.

Two controls, same machine, same prompts, same temperature 0 / seed 42,
generated code extracted and actually executed against assertions rather
than eyeballed:

model                                                    result
qwen2.5-coder:3b-instruct-q3_K_M   (Ollama library)      0/3
qwen2.5-coder:3b-instruct-q4_K_M   (Ollama library)      3/3
Qwen official qwen2.5-coder-3b-instruct-q3_k_m.gguf      3/3

The third row is the one that settles it. Same model, same quantisation
level, a different conversion — and it works. Here is what it returns for the
prompt above:

def sum_to_n(n: int) -> int:
    """ sum_to_n is a function that sums numbers from 1 to n. """
    return n * (n + 1) // 2

So "3B at q3 cannot code" is false on this machine. Something specific to that
artifact is broken.

The control had to be chosen carefully

The obvious control is one of the popular community re-quantisations, and it is
the wrong one. Those repositories ship an .imatrix file — they use
importance-matrix quantisation, which is a different process. If the working
control is an imatrix quant and the broken one is not, then a difference between
them supports two stories at once: "this conversion is broken", and "imatrix
matters enormously at 3B/q3". You cannot tell which you are looking at.

The official Qwen GGUF repository has no .imatrix, which makes it the control
that answers only one question:

curl -s https://huggingface.co/api/models/bartowski/Qwen2.5-Coder-3B-Instruct-GGUF \
  | grep -o 'imatrix'          # present
curl -s https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-3B-Instruct-GGUF \
  | grep -o 'imatrix'          # nothing

One more thing had to be held constant. A raw GGUF imported into Ollama needs a
chat template supplied by hand, and a wrong template produces garbage that
looks exactly like a broken quantisation. So the template was not written from
scratch — it was lifted from the working library model, verbatim:

ollama show --modelfile qwen2.5-coder:3b-instruct-q4_K_M \
  | grep -v '^#' \
  | sed 's|^FROM .*|FROM /path/to/qwen2.5-coder-3b-instruct-q3_k_m.gguf|' \
  > Modelfile
ollama create qwen-official-q3 -f Modelfile

With that, the only difference between the 0/3 model and the 3/3 model is the
weights.

What you can check, and what it costs

The artifact is unchanged since it was first reported. All the relevant tags
still read "1 year ago" on the model's tag page, and the manifest being served
right now hashes to the digest that page shows:

$ curl -s https://registry.ollama.ai/v2/library/qwen2.5-coder/manifests/3b-instruct-q3_K_M \
    | sha256sum | cut -c1-12
65ff2bc170f3

ollama list shows the same 65ff2bc170f3 after pulling, so what is described
here is what you would get today.

Reproduced on Ollama 0.33.3, which is newer than the 0.32.6 in the original
report — upgrading is not the fix.

The fix

There is no flag for this. The artifact is what it is, and the repair belongs
upstream. What you can do on your own machine is stop using that tag:

Move up one level. q4_K_M from the same library works and costs about
300 MB more. If you were on q3 for size reasons, check whether you actually
needed to be.

Or convert from a source you can name. The model author's own GGUF at the
same level works. Import it with the library model's template as above, so you
are not trading one silent failure for another.

And check before you trust it. The whole test is three prompts and running
what comes back:

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "your-tag-here",
  "prompt": "Please provide a self-contained Python script that solves the following problem in a markdown code block:\n\ndef sum_to_n(n: int) -> int:\n    \"\"\" sums numbers from 1 to n \"\"\"\n",
  "stream": false,
  "options": {"temperature": 0, "seed": 42}
}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["response"])'

If what comes back has no def and no return in it, you have this. It takes
about ten seconds.

It is a filter, not a test. It catches the shape of failure I had, where
there was nothing resembling code in the response at all. It does not catch a
defective artifact that emits a plausible-looking function which does not work
— and per the census in the update at the top, two of the five confirmed
defects are exactly that. The only check that separates them is to paste the
returned block into a file and run it against assertions you wrote yourself.

The generalisable habit

The narrow lesson is that a quantised model is a build artifact, not a
property of the model. "Qwen2.5-Coder-3B at q3_K_M" names a recipe; the file you
downloaded is one execution of that recipe by one party, and it can be wrong on
its own without the recipe being wrong. When a model underperforms, the question
"is this quantisation level too low?" and the question "is this file bad?" feel
like the same question and are not. Converting it yourself, or fetching
another party's conversion at the same level, separates them for the price of
one download.

The wider one cost me the most time today, and it is about the order you run
things in.

The first control run reported the working model as 0/3. The harness was
broken: this model emits its language tag on the line after the code fence
rather than on the fence line, so the extractor was handing python to the
interpreter as if it were code. Twenty minutes of a perfectly good model looked
like a second broken one.

I only caught it because the control ran before the suspect. Had I started
with q3_K_M, I would have seen 0/3, matched it against a report that predicted
0/3, and stopped — with a broken measurement and a conclusion that happened to
agree with it. The error would have survived, because the evidence for it looked
exactly like the evidence for the truth.

So: run the case you expect to pass first, and treat its failure as a bug in
your instrument until proven otherwise. A test that has never been seen to
pass has not been shown to work — it has only been shown to produce the answer
you were hoping for. That is the same trap as
a build that confidently reports a commit hash from a repository it has never
heard of:
the thing doing the reporting is not the thing you are trying to measure, and
when they disagree, you will believe the wrong one unless you have arranged in
advance to tell them apart.

The same ordering saved a second finding the same day. vLLM accepts a LoRA
adapter it has already decided not to apply
anywhere
returns base-model output for the ignored adapter — which is also what an
untrained adapter returns when it is applied. Only running the overlapping
allow-list first, and watching the same file change the output, made the
identical result mean anything.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama-cli reports a commit hash that is not in llama.cpp. It belongs to whatever repository you unpacked the source inside.

The Homelab Postmortem — Sat, 05 Sep 2026 03:10:28 +0000

TL;DR: cmake/build-info.cmake runs git rev-parse --short HEAD with WORKING_DIRECTORY set to the source directory and accepts the result if the command succeeded. Git searches upward, so a release tarball unpacked anywhere inside an unrelated git work tree gets that repository's HEAD stamped into the binary, llama-config.cmake, and ggml-config.cmake. Nothing warns: configure exits 0, and the -dirty suffix never fires because the entire source tree is untracked and untracked files are not diffs. -DLLAMA_BUILD_COMMIT= looks like the fix and only half is — ggml/CMakeLists.txt runs its own unguarded probe and overwrites it, and -DGGML_BUILD_COMMIT= is ignored.

The symptom

Built from the b10816 release tarball, on Debian 13, CMake 3.31.6, GCC 14.2.0:

$ ./llama-cli --version
version: 0.4.0-dev (build 1, commit 00621a7)
built with GNU 14.2.0 for Linux x86_64

That is a well-formed seven-character hash in the field where a well-formed
seven-character hash belongs. It is not a llama.cpp commit:

$ gh api repos/ggml-org/llama.cpp/commits/00621a7
No commit found for SHA: 00621a7 (HTTP 422)

00621a7 is the HEAD of a repository I created two directories up, with one
empty commit, whose commit message is "totally unrelated repo, not llama.cpp".
The build found it, believed it, and printed it with no hedging.

The tag actually being built, b10816, is commit 427291b. That value appears
nowhere in the build.

Reproducing it takes three commands and no compiler

The whole thing is decided during cmake configure. You do not have to build
anything to see it:

mkdir -p /tmp/demo/outer && cd /tmp/demo/outer
git init -q . && git commit -q --allow-empty -m "totally unrelated repo"
git rev-parse --short HEAD          # -> 00621a7

curl -sSL https://github.com/ggml-org/llama.cpp/archive/refs/tags/b10816.tar.gz | tar xz
cmake -S llama.cpp-b10816 -B build -DGGML_CUDA=OFF -DLLAMA_CURL=OFF >/dev/null
grep LLAMA_COMMIT build/src/llama-version.h

#define LLAMA_COMMIT  "00621a7"

Configure exits 0. There is no warning in the output.

It is not that build info is unreliable in general

That was worth checking before writing any of this down, because "the commit
field is junk" is a much less useful thing to know than what is actually
happening. Three builds, same tarball where applicable, same machine, same
CMake:

source                                    LLAMA_COMMIT   BUILD_NUMBER
tarball, inside an unrelated git repo     "00621a7"      1
tarball, not inside any git repo          "unknown"      0
real git clone of the b10816 tag          "427291b"      1

The honest case and the correct case both work. Only the middle situation
lies, and it is the only one of the three that lies confidently. With no git
repository anywhere above it, the build says unknown — which is exactly the
right answer and is impossible to misread. With a real clone it says
427291b, which is right. The failure needs a git repository that exists, is
readable, and has nothing to do with the source.

Note the third column, because it removes the tell you would hope for. A
legitimate shallow clone reports BUILD_NUMBER 1 too — git rev-list --count HEAD on a --depth 1 clone is 1. So "the build number looks too small" is not
evidence of anything.

Why nothing catches it

The dirty flag is structurally blind to this case. ggml/CMakeLists.txt
checks whether the tree is modified and appends -dirty if so. That check is
the one thing in the build that could plausibly notice that the source does not
belong to the repository being credited. It cannot:

$ git status --porcelain llama.cpp-b10816
?? llama.cpp-b10816/

$ git -C llama.cpp-b10816 diff-index --quiet HEAD -- . ; echo $?
0

Exit 0 means clean. The source tree shares zero files with the repository
whose commit is about to be stamped on it, and the modification check reports
no modifications — because untracked files are not diffs. A tree with nothing
in common reads as pristine, so the hash goes out unqualified.

The wrong value does not stay in the binary. It is written into the CMake
package files that downstream projects consume after make install:

$ grep BUILD_COMMIT build/llama-config.cmake build/ggml/ggml-config.cmake
build/llama-config.cmake:set(LLAMA_BUILD_COMMIT 00621a7)
build/ggml/ggml-config.cmake:set(GGML_BUILD_COMMIT "00621a7")

Anything that later does find_package(llama) inherits the claim.

What's really going on

cmake/build-info.cmake asks git a question from inside the source directory
and accepts any answer that came back successfully:

execute_process(
    COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
    OUTPUT_VARIABLE HEAD
    OUTPUT_STRIP_TRAILING_WHITESPACE
    RESULT_VARIABLE RES
)
if (RES EQUAL 0)
    set(BUILD_COMMIT ${HEAD})
endif()

The guard is RES EQUAL 0 — did git succeed anywhere — and git's search for a
repository walks up the directory tree until it finds one or hits a mount
point. There is no check that the repository it found contains these files, is
named llama.cpp, or has ever heard of them. Extracting a tarball two levels
below someone else's .git is enough.

This is reported upstream as
ggml-org/llama.cpp#28397,
opened 2026-09-04 with a real-world case: the Arch AUR llama.cpp-cuda package
builds inside the cloned packaging repository, so two published versions
credited the packaging repo's commits. As of this writing the issue is open
with no comments and nothing referencing it.

The fix, and the half of it that looks like the whole

The obvious move is the documented override. CMakeLists.txt guards both
values with if (NOT DEFINED ...), so passing them on the command line wins:

cmake -S llama.cpp-b10816 -B build -DLLAMA_BUILD_COMMIT=427291b -DLLAMA_BUILD_NUMBER=10816

Check it and it worked:

$ grep LLAMA_COMMIT build/src/llama-version.h
#define LLAMA_COMMIT  "427291b"
$ grep BUILD_COMMIT build/llama-config.cmake
set(LLAMA_BUILD_COMMIT 427291b)

Now check the other header.

$ grep GGML_COMMIT build/ggml/src/ggml-version.h
#define GGML_COMMIT  "00621a7"

Still the foreign repository. CMakeLists.txt does pass the corrected value
down as GGML_BUILD_COMMIT immediately before add_subdirectory(ggml) — and
then ggml/CMakeLists.txt runs its own probe, which has no if (NOT DEFINED)
guard at all, and assigns straight over the top of it:

find_program(GIT_EXE NAMES git git.exe NO_CMAKE_FIND_ROOT_PATH)
if(GIT_EXE)
    execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD
        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
        OUTPUT_VARIABLE GGML_BUILD_COMMIT
        OUTPUT_STRIP_TRAILING_WHITESPACE
        ERROR_QUIET
    )

There is no command-line escape from it either. Passing the variable directly
is silently discarded, because the execute_process result is a normal
variable that shadows the cache entry:

$ cmake -S llama.cpp-b10816 -B build -DGGML_BUILD_COMMIT=427291b 2>&1 | grep 'ggml commit'
-- ggml commit:  00621a7

CMake prints the wrong value to the terminal, as a status line, in the middle
of a configure run nobody reads to the end.

What works is denying the upward search. Give the extracted source its own
repository, so git stops there instead of reaching the stranger:

git -C llama.cpp-b10816 init -q
cmake -S llama.cpp-b10816 -B build -DGGML_CUDA=OFF -DLLAMA_CURL=OFF >/dev/null

#define LLAMA_COMMIT  "unknown"
#define GGML_COMMIT   "unknown"

A fresh repository has no HEAD to resolve, so both probes fail and both fall
back to their honest default. unknown is not a cosmetic loss — it is the
correct statement about a tarball, and it is the value you would have got by
building in a directory that was not inside anyone's repository in the first
place. Moving the source tree out from under the foreign .git does the same
thing and is better when you control the layout.

If you package llama.cpp, use both: git init the extracted tree and pass
-DLLAMA_BUILD_COMMIT with the real hash. The first stops ggml lying, the
second gives llama the true value.

To find out whether any of this applies to a build you already shipped, ask
your binary and then ask upstream:

COMMIT=$(./llama-cli --version 2>&1 | sed -n 's/.*commit \([0-9a-f]*\).*/\1/p')
git ls-remote https://github.com/ggml-org/llama.cpp.git | grep -q "^$COMMIT" \
  || echo "reported commit $COMMIT is not in llama.cpp"

The generalisable habit

The narrow lesson is about provenance derived from ambient state. A build that
reports what it is by looking at its surroundings will report whatever the
surroundings say, and directory nesting is not a statement of identity. The
check that was missing is one line — does the repository git found actually
track these files — and it is missing in both probes.

The wider one is the reason this took longer than the reproduction did.
Overriding LLAMA_BUILD_COMMIT produces two pieces of evidence that the fix
worked, in the two places you would naturally look, and a third file that is
still wrong. If the habit is "apply the fix, check the field, move on", this
ends with a build that is half-corrected and believed to be corrected —
which is worse than the original, because now there is a memory of having
handled it. The same shape as a journald drop-in that is correct and still
does nothing until you flush:
the fix being real is not the same as the fix being complete, and only the
second one is worth anything.

And it is the second time in a week that this project's build and CLI have
reported success in a field that was not true —
llama-cli prints a file error and exits 0
is the same failure at runtime. When you report a bug upstream, the first thing
you are asked for is --version output. It is worth knowing that the line can
be confidently, specifically wrong.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →