DEV Community: The Homelab Postmortem

Ollama: a keep_alive on the loaded model cancels the swap another request waits for

The Homelab Postmortem — Thu, 22 Oct 2026 05:49:28 +0000

TL;DR: With OLLAMA_MAX_LOADED_MODELS=1, a request for a model that isn't loaded makes Ollama schedule the loaded model to unload as soon as it's idle, and the request waits. If a request for the loaded model arrives during that wait and carries keep_alive, that value replaces the pending unload: when the loaded model goes idle, Ollama sets a keep-alive timer instead of unloading, and the request for the other model waits for the whole keep_alive. On Ollama 0.40.8 on CPU, the second model answered in 9 seconds when the interfering request had no keep_alive, and didn't answer within 150 seconds when it had "keep_alive": "30m". Nothing errors, and /api/ps shows a healthy loaded model. Sending keep_alive: 0 for the loaded model released the waiting request in 1.5 seconds. Upstream: ollama/ollama#19182, with a fix proposed in #19190.

The symptom

A disposable LXC container, Ollama 0.40.8 from the GitHub release, CPU only, two small models, and the reporter's settings:

OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=4 OLLAMA_KEEP_ALIVE=30m OLLAMA_DEBUG=1

qwen3:0.6b is loaded. The same three requests as in the report:

a long generation on the loaded model (num_predict 400),
three seconds later, a short request for gemma3:270m, which needs the slot,
two seconds later, a one-token request for the loaded model.

Run once with step 3 as a plain request, and once with "keep_alive": "30m" added to it:

=== control: step 3 without keep_alive
  step3 (resident, no keep_alive) done at +5.129s
  step1 (resident long gen) done at +7.646s
  step2 (other model) http=200 at +9.156s
  ps after: [('gemma3:270m', ...)]
=== step 3 with keep_alive 30m
  step3 (resident, "keep_alive":"30m",) done at +5.113s
  step1 (resident long gen) done at +7.610s
  step2 (other model) http=000 at +153.0s
  ps after: [('qwen3:0.6b', ...)]

http=000 is curl giving up at its 150-second limit. The scheduler's debug log for the second run:

sched.go:309 msg="max runners achieved, unloading one to make room"
sched.go:382 msg="resetting model to expire immediately to make room"
sched.go:393 msg="waiting for pending requests to complete and unload to occur"
sched.go:1424 msg="evaluating already loaded"
sched.go:437 msg="runner with non-zero duration has gone idle, adding timer"

The healthy run instead ended with runner with zero duration has gone idle, expiring to unload and unload completed.

Why it's easy to miss

The server is fine from every angle you'd check. /api/ps shows the loaded model, requests for it are answered, and the scheduler messages that show what happened are debug-level (they were visible here with OLLAMA_DEBUG=1). Only requests for other models hang, and they hang without an error: the client just waits, until its own timeout or the keep_alive runs out.

The trigger is ordinary client behaviour. Many clients and UIs send keep_alive on every request; the reporter lost two hours to a client sending "2h" and thirteen minutes to a health probe. One slot is also the normal setting on a machine whose memory holds one model at a time, which is most homelab GPUs.

And it depends on timing: the keep_alive request has to land while the swap is waiting. Under light use it may never happen; under a steady stream of requests for the loaded model it happens again and again, and the other model effectively never gets loaded.

What's really going on

With one slot full, Ollama picks the loaded runner to evict, sets its remaining time to zero so it unloads as soon as its last request finishes, and the waiting request sits until that happens. A request for the same model that arrives in between is served by the runner that's already loaded, and, according to the report's reading of server/sched.go, its keep_alive overwrites the runner's session duration. When the runner then goes idle it's no longer marked to expire, so it gets an ordinary keep-alive timer and stays. The waiting request stays queued for it.

The debug log above shows exactly that sequence: the eviction is set up, the loaded model is evaluated for the new request, and the runner goes idle with a "non-zero duration" and a timer instead of unloading. Without keep_alive on that request, the duration stays at zero and the unload goes through.

The fix

The proposed fix, #19190, stops a request's keep_alive from cancelling a pending eviction. It's open and not in a release, and none of the commit messages between v0.40.8 and v0.40.9-rc1 mention it.

Until then:

Release a stuck swap by unloading the loaded model:

curl http://127.0.0.1:11434/api/generate -d '{"model":"qwen3:0.6b","keep_alive":0}'

Here that let the waiting request finish 1.5 seconds later, with gemma3:270m loaded afterwards.

Avoid the trigger: stop clients from sending keep_alive on every request (set it once on the server with OLLAMA_KEEP_ALIVE instead), or give the server more than one slot if memory allows, so a second model doesn't need an eviction at all (the three-request sequence wasn't rerun with two slots here).

The toolkit's check-ollama-eviction-stall.sh reads the server's OLLAMA_MAX_LOADED_MODELS and loaded models, and if the server logs at debug level, counts evictions that were turned into keep-alive timers. On the test server:

  Ollama 0.40.8 at http://127.0.0.1:11434, OLLAMA_MAX_LOADED_MODELS=1, loaded: qwen3:0.6b
  AT RISK  one model slot: a request for another model can wait for the whole keep_alive of the loaded one
  STALLED  2 unload(s) since -1day were replaced by a keep-alive timer

--release <model> sends the unload. With two slots and no stall it reports "Not at risk here" and exits 0.

The generalisable habit

When a per-request option and a scheduler decision touch the same field, ask which one wins. Here a client's preference ("keep this model around") silently overrode the server's own plan ("unload it now, someone is waiting"), and the request that paid for it was a different client's. Options that look harmless on their own, like keep_alive, can carry the most surprising side effects precisely because nobody thinks of them as instructions to the scheduler.

And when something hangs with no error, measure the alternatives side by side: the same three requests with and without one field made the difference between 9 seconds and never.

Another Ollama setting that doesn't reach the place you'd expect: OLLAMA_GPU_OVERHEAD is logged but never reaches the runner.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Ollama drops tool calls to names not in tools and returns an empty reply

The Homelab Postmortem — Thu, 22 Oct 2026 05:47:32 +0000

TL;DR: When a model served by Ollama calls a function that isn't in the request's tools list, Ollama drops the call. The reply has no content and no tool_calls, but an eval_count above zero, so tokens were generated and then discarded. On Ollama 0.40.8 with qwen3:0.6b, a system prompt saying to call lookup_order, with only get_time in tools, gave 8 of 8 empty replies, each with eval_count 25. Generating the same prompt raw produced <tool_call>{"name": "lookup_order", "arguments": {"order_id": "48213"}}</tool_call>, also 25 tokens. With lookup_order added to tools, 8 of 8 replies were that call. It happens in the generic, template-based tool parser that Ollama uses for models without a dedicated parser, which includes the library qwen3. The client-side fix is to list every tool the model may name. Upstream: ollama/ollama#19187.

The symptom

A disposable LXC container, Ollama 0.40.8 from the GitHub release, CPU, qwen3:0.6b from the Ollama library. Ollama's log shows which path the model takes:

msg="template selection" model=registry.ollama.ai/library/qwen3:0.6b selected=go_template renderer="" parser=""

so tool calls are parsed by the generic parser built from the chat template.

The request, to /api/chat with stream: false and think: false:

system: You are a shop assistant. To answer questions about an order, always call the lookup_order
        tool with the order id. Never answer order questions from memory.
user:   What is the status of order 48213?

Eight runs each (seeds 0 to 7, temperature 0.7), with two tools lists:

qwen3:0.6b tools=[get_time] (lookup_order not listed): EMPTY x8  eval_count of EMPTY: [25, 25, 25, 25, 25, 25]
qwen3:0.6b tools=[get_time, lookup_order]: tool_call lookup_order x8

EMPTY means content was an empty string and there was no tool_calls field. The OpenAI-compatible endpoint did the same: finish_reason stop, empty content, tool_calls null.

To see what the model actually generated, the same conversation went through /api/generate with raw: true and a prompt written out by hand following the model's template, with get_time as the only tool:

raw seed 0: eval_count=25 '<tool_call>\n{"name": "lookup_order", "arguments": {"order_id": "48213"}}\n</tool_call>'
raw seed 1: eval_count=25 '<tool_call>\n{"name": "lookup_order", "arguments": {"order_id": "48213"}}\n</tool_call>'
raw seed 2: eval_count=25 '<tool_call>\n{"name": "lookup_order", "arguments": {"order_id": "48213"}}\n</tool_call>'

A complete, well-formed call, 25 tokens, the same count as the empty replies.

Why it's easy to miss

An empty reply looks like the model's fault. Small models do sometimes return nothing, and a client that gets an empty assistant message usually retries or shows nothing, without logging that tokens were generated. Only eval_count says the model produced something.

Calling a tool that isn't listed sounds like an edge case, but it comes up in ordinary agent setups. Tools that are loaded on demand, tools named in another tool's description or in the system prompt, a tool list trimmed per request to save context, or a model that remembers a tool from earlier in the conversation. The report came from an app where the unlisted tool was offered on demand; 28 of 32 replays of its real requests came back empty.

And there's no signal on the server either: no warning, no done_reason, nothing in the log about a discarded call.

What's really going on

Ollama's generic tool parser (tools/tools.go at v0.40.8) watches the output for the template's tool-call tag, here <tool_call>. Once the tag appears, the parser switches to tool-calling mode and holds everything after it in a buffer. It then looks for one of the names in the request's tools at the start of that buffer (findTool); a name that isn't in the list matches nothing, so no call is produced. For tags other than { or [ the parser never decides it's finished, and at the end of the reply Content() returns an empty string unless the tag is { or [. So the buffered text, which is the entire tool call, is neither returned as a call nor passed through as content.

That matches every observation above: the tag and the call were generated (the raw output), the tokens were counted (eval_count 25), and nothing came out.

The fix

There's no fix upstream yet; the issue had no reply as of 22 October.

List every tool the model may call. With lookup_order listed, every reply here was the call. The reporter found that entries with empty parameter schemas were enough (30 of 32 calls came back); here the listed tool had a full schema. If tools are loaded on demand, include their names up front and load the implementation when the call arrives.

Treat an empty reply with eval_count > 0 as a dropped call, not as "the model said nothing". Log it, and retry with the full tool list.

The toolkit's check-ollama-dropped-tool-calls.sh runs this probe against a model on your server: first with lookup_order listed, to check the model calls tools at all, then without it:

  control, lookup_order listed:     ['call:lookup_order', 'call:lookup_order', 'call:lookup_order', 'call:lookup_order']
  probe, lookup_order not listed:   ['EMPTY(eval_count 25)', 'EMPTY(eval_count 25)', 'EMPTY(eval_count 25)', 'EMPTY(eval_count 25)']

4 of 4 replies came back empty: the model called an unlisted tool and Ollama dropped the call.

It exits 1 there, 0 when the unlisted call comes back as a call or as text, and 2 when the model doesn't call tools at all (gemma3:270m returned HTTP 400 for a request with tools).

The generalisable habit

Check that a parser has a path for input it doesn't recognise. A parser that only knows how to emit the things it can match will, on anything else, either fail loudly or drop it, and dropping is the worse of the two because nothing downstream can tell. The question to ask of any structured-output layer is: what comes out when the model writes something valid that isn't on the list?

And count what was generated, not just what was returned. eval_count against an empty reply is the whole diagnosis here, and it's sitting in every response.

The Responses API in llama-server has a related gap in the other direction, returning a cut-off tool call as completed.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Docker's graphdriver store re-pushes zstd layers labelled gzip

The Homelab Postmortem — Thu, 22 Oct 2026 05:47:31 +0000

TL;DR: When a Docker daemon uses the graphdriver image store (overlay2: every Docker 28, and Docker 29 installs that were upgraded from 28 or have "containerd-snapshotter": false), docker push of an image it pulled with zstd-compressed layers writes a manifest that labels those layers application/vnd.docker.image.rootfs.diff.tar.gzip. The blobs are reused as they are, so they're still zstd. The push succeeds, the registry accepts it, and docker run of the image works, because Docker detects the compression from the data. Tools that trust the label don't: a BuildKit build FROM the relabelled image with compression=zstd,force-compression=true failed with failed to get compression blob "zstd": not found, and without force-compression it succeeded and copied the gzip label into the new image. Reproduced on Docker 29.9.1 and 29.9.2; the containerd image store labels the same layers correctly. Upstream: moby/moby#53999 and moby/buildkit#7308.

The symptom

A disposable Debian 13 VM with Docker CE 29.9.2 from download.docker.com. First the reporter's own reproducer, after reading it: Compose runs registry:3.1.2, two docker:<version>-dind daemons (one with each image store) and an Alpine container with curl, jq and zstd. It pushes a one-layer zstd image to the registry with curl, then has each daemon pull it, retag it and push it, and compares each manifest's media type with the blob's first bytes:

== 1. Push a one-layer zstd image built with curl and the zstd CLI (registry:5000/repro/app:zstd)
  application/vnd.oci.image.layer.v1.tar+zstd        sha256:b520d8eb3b4b…  content: zstd

== 2. Graphdriver store: pull it, then push it under new tags
  registry:5000/repro/app:graphdriver (blob already in the repo):
  application/vnd.docker.image.rootfs.diff.tar.gzip  sha256:b520d8eb3b4b…  content: zstd
  registry:5000/repro/other-graphdriver:latest (blob mounted from repro/app):
  application/vnd.docker.image.rootfs.diff.tar.gzip  sha256:b520d8eb3b4b…  content: zstd

== 3. containerd store: the same, for comparison
  registry:5000/repro/app:containerd (blob already in the repo):
  application/vnd.oci.image.layer.v1.tar+zstd        sha256:b520d8eb3b4b…  content: zstd

Same blob digest everywhere; only the graphdriver pushes call it gzip. It reproduced with docker:29.9.1-dind and with docker:29.9.2-dind.

Then the part the reproducer doesn't show: what happens downstream. On the VM's own daemon, switched to the graphdriver store, with a registry on localhost:5000 and a BuildKit builder (docker-container driver):

1. buildkit pushed base:zstd rc=0  layers: tar+zstd tar+zstd
2. graphdriver dockerd re-pushed base:relabeled  layers: gzip gzip
3. FROM relabeled, compression=zstd,force-compression=true rc=1
   ERROR: failed to build: failed to solve: failed to get compression blob "zstd": not found
4. FROM relabeled, compression=zstd (no force) rc=0  layers: tar+gzip tar+gzip tar+zstd
6. docker run of relabeled image: base

The relabelled layer's first bytes are 28 b5 2f fd, the zstd magic. Step 4 is the quieter failure: the build works and the new image inherits the two wrong labels, so whoever builds on it next gets them too. docker run doesn't care (step 6).

Why it's easy to miss

Nothing in Docker's own workflow ever fails. Push succeeds, pull succeeds, run succeeds, because Docker and containerd look at the blob's magic bytes rather than the label. The wrong label only matters to something that trusts it, and that's usually another tool, later, maybe on someone else's machine. BuildKit's recompression, below, is one; any tool that decides how to decompress a layer from its media type would be another.

The store you're on isn't obvious either. Fresh Docker 29 installs use the containerd image store; according to the report, a Docker 29 that was upgraded from 28 keeps overlay2 ("using prior storage driver: overlay2" at startup), and any daemon with containerd-snapshotter turned off uses it too, as the test daemon here did. docker info shows it as the storage driver: overlay2 is the graphdriver store, overlayfs the containerd one.

And there was one more trap when testing. After BuildKit had built from the relabelled image once, a build FROM the correctly labelled original also failed with the same error, because the builder's cache now held the layer under the gzip label. After docker buildx prune -af, the correctly labelled base built fine and the relabelled one still failed. So a single mislabelled image can make unrelated builds fail on the same builder until its cache is cleared.

What's really going on

When the graphdriver store pushes a layer that the registry already has, or can mount from another repository, it doesn't upload anything, it just writes the manifest entry. According to the report, the media type for both of those paths is hard-coded to the gzip Docker type in daemon/internal/distribution/push_v2.go. For a layer that was originally gzip that's correct; for a zstd layer pulled from a registry it's wrong, and nothing checks.

The containerd image store keeps the layer's original descriptor, media type included, which is why it pushes the same layers correctly.

On the BuildKit side, according to buildkit#7308, force-compression decides whether a layer needs converting from its media type, so a zstd layer labelled gzip is treated as needing conversion to zstd, and the lookup for a zstd version of it fails, which is the failed to get compression blob "zstd": not found above.

The fix

There's no fix in a release; the issue is labelled help wanted.

Push from a containerd-store daemon. Either a fresh Docker 29 install, or set "features": {"containerd-snapshotter": true} in /etc/docker/daemon.json and restart. Images stored under the old store don't appear in the new one, so pull them again after switching. With that store the same push kept tar+zstd.

Or don't re-push pulled zstd images from a graphdriver daemon; rebuild them instead.

Clear the builder cache after hitting the BuildKit error: docker buildx prune -af on the builder involved.

Check what's already in your registry. The toolkit's check-image-layer-compression.sh fetches an image's manifest (each platform of an index, or one with --platform) and the first bytes of every layer, and compares them with the label. Anonymous pulls, read-only:

  ok       localhost:5000/base:zstd e2975860bc76  label zstd  content zstd
  MISLABEL localhost:5000/base:relabeled e2975860bc76  label gzip  content zstd
  MISLABEL localhost:5000/app:noforce e2975860bc76  label gzip  content zstd
  ok       localhost:5000/app:noforce 0cc3546756f5  label zstd  content zstd

It exits 1 when any layer is mislabelled. Against Docker Hub's alpine:3.24 and nginx:alpine it reported every layer ok.

The generalisable habit

A label is a claim, not a fact. When a format records what something is in metadata next to the data, the two can drift apart, and the tools that read only the metadata are the ones that break. If you consume artifacts produced by other people's tooling, check the content when it's cheap to, the way Docker does here with magic bytes.

And test the next step, not just the one that was reported. The reproducer showed a wrong label; the cost of that label only appeared one build later, as an error in a different tool and a cache that went on failing for a correctly labelled image.

Another Docker 29 behaviour that depends on which store the daemon uses: type=image mounts come back empty after a reboot, differently on overlay2 and on the containerd store.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Proxmox disk move with Delete source deletes the source even when the copy is missing

The Homelab Postmortem — Wed, 21 Oct 2026 04:59:16 +0000

Update 2026-10-23. Proxmox has a fix in review. The report is now Bugzilla #8209 (status PATCH AVAILABLE), and Fiona Ebner posted a four-patch series for qemu-server to pve-devel, "fix #8209: clone disk: do not ignore failure to query size of target". As the patch describes it, a failed size query of the target is now logged as a warning and reported to the caller so it can avoid the destructive step, and because the file-based storage plugins return nothing instead of failing, the size is checked explicitly. The series isn't applied or in a package yet, so qemu-server 9.1.15 behaves as described below. Why the reporter's target disappeared on the NFS share is, in the Bugzilla thread's words, still unclear.

TL;DR: In Proxmox VE 9.2, moving a VM disk with "Delete source" (qm disk move ... --delete 1) deletes the original as soon as the copy step returns, without checking that the copy exists. On a test VM, the target file was removed while the copy was running. The move finished with exit 0 and TASK OK. The task log had one line, failed to stat '<target>', followed by removing source volume ... after successful copy, and the source volume was gone. The VM's config pointed at the missing file, still showing size=2G, so nothing in the config looked wrong. With "Delete source" off, the same failure kept the original as unused0. Until this changes: move without "Delete source", check the copy against the original, then remove the original yourself. The toolkit has a script that searches past task logs for this and one that does the checked move. Report: forum.proxmox.com thread 187166, from someone who lost a disk this way moving to an NFS share.

The symptom

The Proxmox host behind this site's lab, pve-manager 9.2.2, qemu-server 9.1.15, libpve-storage-perl 9.1.5. A throwaway directory storage hpmtest and a stopped test VM with a 2 GiB disk on local-lvm, 1 GiB of it filled with random data so the copy takes a few seconds at --bwlimit 51200. A second shell deleted the target file 3 seconds after it appeared, which is a stand-in for whatever made the reporter's target disappear:

# qm disk move 9901 scsi0 hpmtest --format qcow2 --delete 1 --bwlimit 51200
...
transferred 2.0 GiB of 2.0 GiB (100.00%)
failed to stat '/var/lib/hpmtest/images/9901/vm-9901-disk-0.qcow2'
removing source volume 'local-lvm:vm-9901-disk-0' after successful copy
  Logical volume "vm-9901-disk-0" successfully removed.
rc=0

The task log ends TASK OK. Afterwards:

config: scsi0: hpmtest:9901/vm-9901-disk-0.qcow2,size=2G
source LV: 0 volume(s) for vm 9901
target dir: (empty)

The source is deleted, the target doesn't exist, and the config still looks complete. The same run with --delete 0:

failed to stat '/var/lib/hpmtest/images/9901/vm-9901-disk-0.qcow2'
adding source volume 'local-lvm:vm-9901-disk-0' as 'unused0' to config
rc=0
config: scsi0: hpmtest:9901/vm-9901-disk-0.qcow2,size=2G unused0: local-lvm:vm-9901-disk-0

Also TASK OK, also pointing at a missing file, but the original is still there as unused0, and the VM can be pointed back at it.

Why it's easy to miss

The task succeeds. In the GUI it shows OK, the CLI exits 0, and the only sign is one failed to stat line in a log of progress lines, immediately followed by a line that says "after successful copy". Nobody reads a long move log that ended OK.

The config doesn't help either. The disk line keeps its size=2G, because that comes from the old disk's entry; only a missing size would hint at a problem, and there isn't one. The problem only shows when something needs the disk, most likely the next start of the VM (not tried here), which after an offline move of a stopped VM may be days later, when the space on the old storage has been reused.

And the trigger is not something you'd test for. Here the file was removed by hand. In the forum report it was a move to an NFS share on a Synology, where the copy ran for about nine hours and the target was not there at the end. Why it vanished there is a separate question, but anything that can make a target disappear or become unreadable, a network share, a full or remounted filesystem, a cleanup job, turns "Delete source" into data loss.

What's really going on

The installed code on this host, read directly:

In PVE/API2/Qemu.pm, the disk move runs the clone inside an eval, and if that eval didn't die, the delete block runs:

if ($param->{delete}) {
    print("removing source volume '$old_volid' after successful copy\n");
    eval {
        PVE::Storage::deactivate_volumes($storecfg, [$old_volid]);
        PVE::Storage::vdisk_free($storecfg, $old_volid);
    };

"Successful" means only that nothing died. The one place the target is looked at after the copy is in PVE/QemuServer.pm, where the clone asks for its size:

my $size = eval { PVE::Storage::volume_size_info($storecfg, $newvolid, 10) };

For a file-based target, that ends up in the storage plugin's file_size_info, which, when stat fails, prints failed to stat '...' as a warning and returns nothing, rather than dying. The eval would catch a death anyway. So a missing target produces a warning and an undefined size, the clone returns normally, and the delete goes ahead. The size=2G in the config is the old disk's size, kept because the new one came back undefined.

The copy itself (qemu-img convert) didn't fail either: it keeps writing to the file it has open, even after the name has been removed, and finishes normally.

The fix

There's no fix in Proxmox yet; the forum thread had no reply as of 21 October and there's no Bugzilla entry for it.

Don't use "Delete source" for moves you can't watch. Move with it off, so the original stays as unusedN.

Check the copy before removing the original. For a stopped VM, qemu-img compare reads both and checks the guest-visible contents match, across formats:

qemu-img compare -f raw -F qcow2 /dev/pve/vm-101-disk-0 /mnt/pve/nas/images/101/vm-101-disk-0.qcow2

Exit 0 means identical. Then remove the original with qm disk unlink <vmid> --idlist unused0, which deletes the volume. If the target is missing or differs, point the VM back at the original with qm set <vmid> --scsi0 <old volume>.

The toolkit's pve-move-disk-verified.sh does exactly that for a stopped VM: moves without delete, checks the target exists, compares, and removes the source only if they match and you passed --delete-source. With the target removed mid-copy it printed:

TARGET MISSING: hpmtest:9901/vm-9901-disk-0.qcow2 does not exist after the move. The source is kept as unused0 (local-lvm:vm-9901-disk-0).

and exited 1, with the original intact. A normal move compared identical and removed the source.

Check whether it has already happened to you. check-pve-move-delete.sh reads every disk-move task log on the node and lists the ones with failed to stat, as LOST if the source was then removed, and checks every VM disk in the current configs for a volume that doesn't exist:

  LOST  vm 9901  2026-10-21 09:06:26  source local-lvm:vm-9901-disk-0 was deleted; target /var/lib/hpmtest/images/9901/vm-9901-disk-0.qcow2 could not be stat'ed (TASK OK)
  KEPT  vm 9901  2026-10-21 09:07:25  target /var/lib/hpmtest/images/9901/vm-9901-disk-0.qcow2 could not be stat'ed; source kept: as 'unused0'
...
  MISSING  vm 9901  scsi0  hpmtest:9901/vm-9901-disk-0.qcow2

A LOST source can't be recovered from Proxmox; that's what backups are for.

The generalisable habit

Never let a tool delete the original on the strength of its own success message. "After successful copy" is a claim made by the same code that did the copy, and here it meant only that nothing threw an error. Before deleting a source, verify the destination independently: it exists, it has the right size, and its contents match. That's one command, and it's the one step the tool skipped.

More generally, a warning followed by success is a pattern worth searching your logs for. A warning means some check failed and the program decided to carry on; when the next step is destructive, that decision deserves a look.

Another case of a status that said everything was fine while the data was gone: a Docker container that came back running after a reboot with its image mount empty.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama.cpp splits Hindi and Thai differently for Nemotron 3 because its GGUF says pixtral

The Homelab Postmortem — Wed, 21 Oct 2026 04:59:15 +0000

Update 2026-10-23. Fixed in llama.cpp. #30960 ("vocab : use Tekken pre-tokenizer for pixtral rather than the Llama 3 one") was merged on 2026-10-22 and closed #30894; the first release with it is b11813. With the b11829 release and the same unmodified GGUF, which still says pixtral, llama-server with no override tokenized every sample here exactly as Hugging Face does (Hindi 19 tokens, Tamil 15, Thai 15, दी as [44213]), and llama-tokenize agreed. On b11813 or later nothing below is needed; on older builds, and in front ends that bundle an older llama.cpp, the override still is.

Update 2026-10-22. A llama.cpp maintainer suggested fixing this in llama.cpp itself by treating pixtral as Tekken, since the pixtral hash came from a community copy of the Pixtral tokenizer and the Llama 3 rules may be wrong for Pixtral too; the reporter agreed. That would fix GGUFs already published without rewriting them. No pull request yet, so the override below is still needed.

TL;DR: NVIDIA's Nemotron-3-Nano-4B GGUF says tokenizer.ggml.pre = "pixtral". In llama.cpp, pixtral selects the Llama 3 pre-tokenizer, but Nemotron's own tokenizer uses the Tekken split rules. The result: on llama.cpp master (339f324), English, Japanese, Arabic and code tokenized exactly as Hugging Face's tokenizer does, while Hindi took 30 tokens instead of 19, Tamil 20 instead of 15 and Thai 16 instead of 15, and दी became two tokens instead of one. Nothing errors; the model loads and answers. Starting llama-server with --override-kv tokenizer.ggml.pre=str:tekken, or rewriting the file's metadata to tekken, made every sample match Hugging Face. One trap: llama-tokenize, the obvious tool to check this with, accepts --override-kv and ignores it, so the fix looks like it doesn't work. Upstream: ggml-org/llama.cpp#30894, which lists other affected uploads.

The symptom

A disposable LXC container, llama.cpp built from master at 339f324, NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf from nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF (2,837,072,864 bytes, matching the size on Hugging Face), and the same model's tokenizer.json from nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 as the reference. The GGUF's tokenizer metadata:

general.architecture = nemotron_h
tokenizer.ggml.model = gpt2
tokenizer.ggml.pre = pixtral

Token IDs for the same text, without special tokens, Hugging Face tokenizers against llama-server's /tokenize:

  english    HF  19  server  19 ==
  hindi-one  HF   1  server   2 !=  [4894, 2063]
  hindi      HF  19  server  30 !=
  tamil      HF  15  server  20 !=
  thai       HF  15  server  16 !=

hindi-one is दी, which Hugging Face encodes as [44213]. Japanese (13), Arabic (5) and a line of Python (12) matched exactly.

The same server started with --override-kv tokenizer.ggml.pre=str:tekken:

  english    HF  19  server  19 ==
  hindi-one  HF   1  server   1 ==  [44213]
  hindi      HF  19  server  19 ==
  tamil      HF  15  server  15 ==
  thai       HF  15  server  15 ==

A copy of the GGUF with the metadata rewritten to tekken gave the same full match.

Why it's easy to miss

Everything a quick check looks at is fine. The file loads, llama-server starts, chat works, and English, the language most people test with, is token-for-token identical. Code and Japanese were identical too. The difference is confined to scripts where the two split rules disagree, here the Indic scripts and Thai, and the only symptoms are that those prompts use more tokens and the model sees them split in a way it never saw in training. How much that hurts answers wasn't measured here, and it may be small for a short prompt.

It also hides behind something people already expect: Indic scripts take more tokens than English with most tokenizers, so a high count for Hindi doesn't look unusual. The extra from llama.cpp only shows up next to the Hugging Face count.

And the obvious way to test a fix gives the wrong answer. llama-tokenize is llama.cpp's tool for exactly this, and it accepts --override-kv without complaint:

$ llama-tokenize -m NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf -p "दी" --ids --log-disable --no-bos \
    --override-kv tokenizer.ggml.pre=str:tekken
[4894, 2063]

Still two tokens, exit 0. Its source (tools/tokenize/tokenize.cpp) has no override handling at all, while an unknown flag such as --no-such-flag is rejected with an error, so the override is parsed and then dropped. The same flag on llama-server works, as shown above.

What's really going on

When a model is converted, llama.cpp's converter identifies the tokenizer by hashing how it splits a fixed test string, and looks the hash up in a table of known tokenizers to choose the pre name it writes into the GGUF. According to the report, the Nemotron tokenizer's hash is the same as Pixtral-12B's, so the converter writes pixtral. In src/llama-vocab.cpp, pixtral is grouped with llama3 and gets the Llama 3 pre-tokenizer, which is right for Pixtral. Nemotron's tokenizer.json uses the Tekken split pattern, which llama.cpp has under the separate name tekken. The two patterns behave the same on English and code and differ on the scripts above.

The test here can't see the converter's hash, but the results fit that explanation exactly: everything else in the file is unchanged, and changing only the pre name, by override or by rewriting the file, makes every sample match.

The report proposes a load-time fix (treat pixtral as Tekken when the architecture is Nemotron), which would also correct GGUFs already published. As of 21 October the issue has no replies and there is no pull request.

The fix

At load time: add the override to whatever runs the model.

llama-server -m NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf --override-kv tokenizer.ggml.pre=str:tekken

Or rewrite the file once, with the script that ships in llama.cpp's gguf-py:

python gguf-py/gguf/scripts/gguf_new_metadata.py --pre-tokenizer tekken \
    NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf nemotron-tekken.gguf

That writes a new file and leaves the original alone; it took about five seconds for the 2.8 GB file here.

Check with llama-server's /tokenize, not llama-tokenize, for the reason above.

Other front ends that bundle llama.cpp, such as Ollama and LM Studio, have their own ways of passing metadata overrides, if any; those weren't tested here. Only NVIDIA's Nano 4B upload was tested; the report lists other Nemotron 3 and 3.5 GGUFs from NVIDIA, ggml-org, Unsloth, bartowski and LM Studio as carrying pixtral too, and one Unsloth upload as carrying tekken.

The toolkit's check-gguf-pretokenizer.sh reads the GGUF header directly (no model load, 0.05 s for this file) and flags this combination:

  WRONG    m/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf: architecture nemotron_h, pre-tokenizer 'pixtral' (llama.cpp splits it as Llama 3); should be 'tekken'
  ok       m/nemo-tekken.gguf: architecture nemotron_h, pre-tokenizer 'tekken'

With --server <url> --hf-tokenizer tokenizer.json it sends multilingual samples to a running llama-server and compares them with Hugging Face's tokenizer, which catches this kind of mismatch for any model, not just this one.

The generalisable habit

Test a local model in the languages you'll actually use, against the reference tokenizer, not just by chatting with it. A tokenizer mismatch doesn't break anything visibly; it shifts what the model sees. Comparing token IDs with the original tokenizer.json for a few sentences in each language takes a minute and settles it.

And when a check says a fix didn't work, check the checker. Here the tool silently ignored the option being tested. A tool that accepts a flag it doesn't implement will make a correct fix look wrong; trying the same thing a second way, through the server, showed it.

A different metadata field steering a model to the wrong behaviour came up with Ollama: it picks the Gemma 4 prompt template from the model's name.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama-server's Responses API marks truncated replies and tool calls as completed

The Homelab Postmortem — Tue, 20 Oct 2026 22:41:50 +0000

TL;DR: llama-server's OpenAI-compatible /v1/responses endpoint never reports a reply as cut off. On llama.cpp b11753, a request with max_output_tokens: 8 came back with "status": "completed", incomplete_details: null and the text 1 2 3 4, stopped mid-count. A tool call cut off by the limit came back as a function_call item with arguments {"path": "/, which is not valid JSON, and both the item and the response said completed. The streaming version ended with response.completed and never sent function_call_arguments.done. The same requests on /v1/chat/completions said finish_reason: "length". A client that trusts status, which is how the Responses API is meant to be used, will treat half an answer as a whole one, or run a tool with broken arguments. The one signal left is usage.output_tokens equal to the limit you sent. Upstream: ggml-org/llama.cpp#30773.

The symptom

A disposable LXC container, the b11753 release binary, gemma-3-1b-it-Q4_K_M and Qwen3-0.6B-Q8_0, --jinja -c 4096. A request that can't finish within its limit:

POST /v1/responses
{"model": "x", "input": "Count from 1 to 200, separated by spaces.", "max_output_tokens": 8}

{"status": "completed", "incomplete_details": null, "output_tokens": 8, "text": ["1 2 3 4 "]}

The same question with room to finish used 52 tokens and also said completed, so status is the same whether the reply ended or was cut. On the chat endpoint:

POST /v1/chat/completions   ... "max_tokens": 8
finish_reason= length completion_tokens= 8

With streaming, the last events of the 8-token request were:

response.content_part.done, response.output_item.done, response.completed status=completed incomplete_details=None

There is no response.incomplete event.

A tool call is where it matters most. One tool, write_file(path, content), Qwen3-0.6B, temperature: 0, and /no_think in the prompt so the model calls the tool straight away:

responses limit=20:  status=completed usage.output_tokens=20 args='{"path": "/'
chat      limit=20:  finish_reason=length completion_tokens=20 tool_calls=[('write_file', '{"path": "/')]
responses limit=400: status=completed usage.output_tokens=89 args='{"path": "/tmp/poem.txt", "content": "Th...'
chat      limit=400: finish_reason=tool_calls completion_tokens=89

At a 20-token limit the Responses API returned a function_call item marked completed whose arguments end after {"path": "/. At 30 they ended after "content": ". Both endpoints produced the same truncated arguments; only the chat endpoint said so. In the stream there were five response.function_call_arguments.delta events, then response.output_item.done and response.completed. No response.function_call_arguments.done, and that event was also missing from a call that completed normally.

Why it's easy to miss

Short limits are rare in interactive use, so most replies really are complete, and completed is the right answer most of the time. The ones that aren't are the long ones: a big file write, a long code edit, a model that thinks for a while first. Those are exactly the replies where an agent is about to do something with the output.

The model also isn't wrong. The text is what the model generated, and in the tool-call case the arguments are a correct prefix of a correct call. Nothing errors until the client tries to parse the arguments, and a lenient JSON parser or a retry loop can hide even that. When the call does fail, it looks like the model produced bad JSON, which small models are known for.

And the chat endpoint is fine. Anyone who checked finish_reason on /v1/chat/completions has seen length work. The Responses API is newer, it's what Codex CLI and other recent agent tools speak, and the only thing llama-server's README says about how it works is that it "works by converting Responses request into Chat Completions request."

What's really going on

The conversion is only one way. The request is turned into a chat completion and generated normally, and the server knows the reply stopped at the limit, because the chat endpoint reports length from the same generation. When the result is turned back into a Responses object, in tools/server/server-task.cpp, the response status and the status of every output item (message, reasoning, function call) are written as the literal string "completed". At the b11753 tag that is seven {"status", "completed"} entries across to_json_oaicompat_resp() and to_json_oaicompat_resp_stream(), the non-streaming and streaming builders. Neither reads the stop reason, and the string incomplete does not appear anywhere in the file. The Responses specification has status: "incomplete" with incomplete_details.reason: "max_output_tokens" for exactly this case; llama-server never produces it.

The fix

There's no fix upstream yet; the issue has no replies or PR as of 20 October. On the client side:

Treat usage.output_tokens == max_output_tokens as cut off. In every truncated reply here it was exactly the limit (8 of 8, 20 of 20), and in every complete one it was lower (52 of 400, 89 of 400). It's a heuristic: a reply that ends naturally on its last allowed token would look the same, which errs on the safe side.
Validate tool-call arguments before running anything. A function_call whose arguments don't parse as JSON was cut off, whatever its status says.
Use /v1/chat/completions if your client can. finish_reason: "length" is reported correctly there, for text and for tool calls.
Give agent requests a generous limit, so the truncation case is rare in the first place.

The toolkit's check-llama-responses-status.sh sends one 8-token request to /v1/responses and one to /v1/chat/completions, and compares:

  /v1/responses        max_output_tokens=8: status='completed' incomplete_details=None output_tokens=8
  /v1/chat/completions max_tokens=8:        finish_reason='length'

http://127.0.0.1:18090: a reply cut off at max_output_tokens is reported as 'completed'.

It exits 1 there, 0 when a cut-off reply is reported as incomplete, and 2 when it can't tell.

The generalisable habit

A status field is only useful if something can make it say no. When you rely on one, find out whether the implementation has a code path that ever produces a value other than success, and test the failure case on purpose: set a limit too small for the answer and look at what comes back. A field that is hard-coded to completed passes every test that only checks successful replies.

And when an API is implemented by translating to another one, check that the translation carries the failure signals back, not just the content. Here the information existed one layer down, in the chat endpoint's finish_reason, and was dropped on the way out.

This is the second place this month where llama-server's compatibility layers lose something on the way through: its Anthropic endpoint ignores which tool you ask for and stops inside the model's reasoning.

Ollama loses tool calls in a different way: a call to a tool name that isn't in tools is dropped, and the reply comes back empty.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Docker 29.6+: docker exec creates world-writable files in dind and rootless Docker

The Homelab Postmortem — Tue, 20 Oct 2026 22:40:03 +0000

TL;DR: Since Docker 29.6.0, dockerd runs with umask 0000, so that modes you ask for explicitly (such as COPY --chmod=777) are not filtered. A process started with docker exec inherits its umask from the container's shim, which inherits it from containerd. On the standard docker-ce packages containerd is a separate systemd service with umask 0022, and exec'd processes still got 0022. But in docker-in-docker (docker:29.9.1-dind) and in rootless Docker, dockerd starts containerd itself, and there every docker exec ran with umask 0000: touch made -rw-rw-rw- files and mkdir made drwxrwxrwx directories, with no warning. docker run's main process still got 0022. docker:29.5.3-dind was fine; docker:29.6.0-dind was not. Creating the container with --umask 0022 fixes exec too. Upstream: moby/moby#53985, filed from OrbStack on macOS, which has the same structure.

The symptom

A disposable Debian 13 VM with Docker CE 29.9.1 from download.docker.com. The test is the one from the upstream report: start a container, exec umask in it, create a file and a directory.

On the plain install, nothing looks wrong:

dockerd: Umask: 0000
containerd: Umask:  0022
shim (after run): Umask:    0022
exec umask: 0022
  exec created: drwxr-xr-x /tmp/d
  exec created: -rw-r--r-- /tmp/f

dockerd itself is at 0000, but containerd isn't, and the shim and the exec'd shell follow containerd.

Inside docker:29.9.1-dind, which runs its own dockerd and lets it start containerd, the same steps:

dockerd: Umask: 0000
shim: Umask:    0000
inner exec umask: 0000
  inner exec created: drwxrwxrwx /tmp/d
  inner exec created: -rw-rw-rw- /tmp/f
inner run umask: 0022

And with rootless Docker, set up with dockerd-rootless-setuptool.sh install on the same VM:

dockerd: Umask: 0000
containerd: Umask:  0000
exec umask: 0000
  exec created: -rw-rw-rw-
run umask: 0022

The version boundary, in dind:

=== docker:29.5.3-dind
server: 29.5.3  dockerd Umask:  0022
inner exec umask: 0022
=== docker:29.6.0-dind
server: 29.6.0  dockerd Umask:  0000
inner exec umask: 0000

Why it's easy to miss

On the setup most people test with, the docker-ce packages on a Linux host, it doesn't happen. That is probably why the change, merged in June, was only reported in October.

Where it does happen, nothing fails at the time. The files are created, the command exits 0, and everything works, because world-writable is more permissive, not less. The trouble shows up later and somewhere else. In the upstream report, files written by a docker exec hook were 0666, and OpenSSH refused to use them (Bad owner or permissions). Other tools that check permissions refuse such files the same way; SSH private keys are the usual example. And anything written to a volume shared with other containers or users is writable by all of them.

The places it does happen are common in homelabs and CI: Docker-in-Docker CI runners, rootless Docker on a shared machine, Docker Desktop alternatives on a laptop. docker run behaves normally in all of them, so a quick test that starts a container and creates a file looks fine. Only exec is affected, and setup scripts, CI steps and maintenance commands are exactly what people run with exec.

What's really going on

moby#52892, "daemon: Set umask to 0000", merged in June for 29.6.0, clears the daemon's umask so that "explicitly requested file modes are preserved instead of being filtered by the previous 0022 umask". That fixes things like COPY --chmod=777 producing 0755.

A process's umask is inherited by its children. The container's main process doesn't care: runc sets 0022 for it explicitly. An exec'd process gets no such default when the container spec has no umask, so it keeps whatever its parent, the shim, has. The shim is started by containerd and inherits containerd's umask.

So the outcome depends on who started containerd. With the docker-ce packages, containerd is containerd.service, started by systemd with the usual 0022, and dockerd's 0000 never reaches it. In dind and rootless Docker, dockerd starts containerd as its own child, containerd inherits 0000, and so does every shim and every exec. The upstream reporter, on OrbStack, saw dockerd, containerd and every shim at 0000 as well.

The open fix, moby#53986, sets 0022 for exec processes when the container spec has no umask.

The fix

Until that lands, either give the container its own umask or set one in each exec.

Create the container with --umask 0022 (the flag came with moby#53463 in Docker 29.8.0). Exec'd processes then use it too:

inner --umask 0022: exec umask 0022              (dind 29.9.1)
rootless --umask 0022: exec umask 0022           (rootless 29.9.1)

It needs a recreate, and it applies per container.

Or set the umask in the exec'd command:

docker exec <container> sh -c 'umask 022; <command>'

That gave 0022 in rootless Docker and works on any version, but every caller has to remember it.

A plain install of the docker-ce packages doesn't need either. Check rather than assume, since a distribution package or a custom setup may start containerd differently.

The toolkit's check-docker-exec-umask.sh reads the shim's umask for every running container from /proc, notes which containers were created with their own --umask, and with --probe runs umask in a throwaway container with docker exec. In rootless Docker on the VM:

  ok       r2: created with --umask 0022; exec uses it
  AFFECTED r1: shim umask 0000, so docker exec processes create 0666 files and 0777 directories
  probe    docker exec in a new container without --umask: umask 0000

It exits 1 there and inside docker:29.9.1-dind, and 0 on the plain install and on 29.5.3.

The generalisable habit

When a change sets something process-wide, ask what inherits it. A umask, an environment variable, a resource limit or an open file descriptor is passed down to every child, and the change's author tested the processes they had in mind. Here that was the daemon's own file creation; the exec'd process three generations down was not.

And reproduce on the reporter's setup, not just on the usual one. On a standard install this looked like a bug that didn't exist. The process tree is what differs, so the first thing to check was who starts containerd.

Rootless Docker has another quiet difference from rootful on this same release: docker buildx build --resource limits are ignored there.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

COPY --chown with a missing user builds fine and leaves the files owned by root

The Homelab Postmortem — Tue, 20 Oct 2026 22:40:01 +0000

TL;DR: The Dockerfile reference says a COPY --chown name that isn't in the image's /etc/passwd or /etc/group makes the build fail. With Docker 29.9.1 and buildx v0.38.0 (BuildKit v0.34.0 inside), it didn't. COPY --chown=nosuchuser file.txt /a built with exit 0, printed no warning, and /a was owned by 0:0. A numeric ID of 2^31 or more (--chown=2147483648) also became 0:0. A missing group with a numeric user (--chown=1000:nosuchgroup) became 1000:1000. The only cases that failed were COPY --link --chown=nosuchuser, with a clear error, and a FROM scratch stage with no passwd file at all. So a typo, or a user created in a different build stage, quietly gives files to root, and you find out when the non-root process that was supposed to own them gets Permission denied. Upstream: moby/buildkit#7292.

The symptom

A disposable Debian 13 VM, Docker CE 29.9.1 with buildx v0.38.0 from download.docker.com, the default docker builder. One Dockerfile with four --chown forms:

FROM alpine:3.24
COPY --chown=nosuchuser file.txt /a
COPY --chown=2147483648 file.txt /b
COPY --chown=1000:nosuchgroup file.txt /c
COPY --chown=nobody file.txt /d

$ docker buildx build --no-cache --progress=plain --load -t chtest .
...
build rc=0
$ docker run --rm chtest stat -c '%n %u:%g' /a /b /c /d
/a 0:0
/b 0:0
/c 1000:1000
/d 65534:65534

No warning in the build output. /d is the control: nobody exists in alpine, and the file got 65534:65534. Every other line asked for an owner that doesn't exist and got something else without a word. With both names missing (--chown=nosuchuser:nosuchuser), a RUN stat later in the same build printed /a 0:0. With a known user and a missing group (--chown=nobody:nosuchgroup), the result was 65534:65534, the user's own group.

The two cases that did fail:

COPY --link --chown=nosuchuser file.txt /a
ERROR: failed to build: failed to solve: --chown=nosuchuser: user and group names can't be resolved
when used with --link, use numeric uid[:gid] instead

FROM scratch
COPY --chown=nosuchuser file.txt /a
ERROR: failed to build: failed to solve: failed to load cache key: invalid user index -1

The Dockerfile reference, read on 20 October, says that if the passwd and group files "don't contain the specified names, the build fails". Here that held only for a base with no passwd file at all.

Why it's easy to miss

The build succeeds and the image runs. Ownership only matters when a process other than root touches the file, and in a typical image that happens at run time: an app running as USER app can't write to a directory that was supposed to be its own, or a config file meant to be private belongs to root instead. The error points at the running container, not at a COPY line in the Dockerfile.

The ways to get a missing name are ordinary. A typo. A user created in a build stage, and the COPY --chown in the next stage, which starts from a different base without that user. A base image changed from Debian to alpine, where the same user doesn't exist. A distroless base, where few names exist at all.

What's really going on

When BuildKit applies a --chown name, it reads the destination stage's /etc/passwd and looks the name up (readUser in solver/llbsolver/ops/user_linux.go, the same code in BuildKit v0.34.0, which Docker 29.9.1 vendors). The result starts as uid 0, gid 0, and is only filled in if the lookup finds something:

if len(users) > 0 {
    us.UID = users[0].Uid
    us.GID = users[0].Gid
}

There's no else. A name that isn't found leaves 0:0 and returns no error. Groups work the same way, which is why 1000:nosuchgroup kept the gid that the numeric user had set, and nobody:nosuchgroup kept nobody's group.

The numeric case is a parsing limit. In BuildKit's LLB client, parseUID parses a --chown value with strconv.ParseInt(str, 10, 32), a 32-bit signed parse, so 2147483648 and above don't parse as numbers, are treated as names, aren't found, and end up as root.

--link refuses names outright, and its error says why: they "can't be resolved when used with --link". The scratch failure comes from a different place in the build, not from this lookup. A fix for the silent fallback was proposed in buildkit#7295, but it was closed without being merged on 20 October, and no other fix is open.

The fix

Make sure the user exists in the stage that does the COPY. In a multi-stage build, create it again in the final stage, or start the final stage from a base that has it:

FROM alpine:3.24
RUN addgroup -S svc && adduser -S -G svc svc
COPY --chown=svc:svc file.txt /a

Or use numeric IDs below 2^31, which never need a lookup: --chown=1000:1000. /c above shows that a numeric user with a named group can still go wrong, so make both numeric.

Check the result, not the Dockerfile. After a build, docker run --rm <image> stat -c '%n %u:%g' <paths> on the paths you chowned takes a second and catches all of the above.

The toolkit's check-dockerfile-chown.sh reads a Dockerfile, finds every COPY/ADD --chown without --link, and resolves each user and group against the stage's base image (read with docker create and docker cp, nothing is run) plus any users created with useradd/adduser/groupadd/addgroup earlier in the stage or its parents. On a test Dockerfile (excerpt):

  ok       app
  SILENT   nosuchuser: user 'nosuchuser' is not in alpine:3.24's /etc/passwd. The build succeeds and the files get root (0:0).
  SILENT   2147483648: numeric user 2147483648 is 2^31 or more and is treated as a name. The build succeeds and the files get root (0:0).
  SILENT   1000:nosuchgroup: group 'nosuchgroup' is not in alpine:3.24's /etc/group. The build succeeds and the files get a group other than the one named.
  ok       nobody
  ok       nosuchuser: used with --link, which fails the build if a name does not resolve

It exits 1 when any --chown would fall back. Building that Dockerfile confirmed each line: the paths it flagged came out as 0:0, 0:0, 1000:1000 and, for nobody:nosuchgroup, 65534:65534, and the app user created with adduser earlier in the build owned its file as expected.

The generalisable habit

When the documentation promises a failure, test the failure. "The build fails if the user doesn't exist" is the kind of sentence that's easy to believe and never check, because you only discover it's wrong by making the mistake it's supposed to catch. One deliberately wrong --chown would have shown it.

The code pattern is worth recognising too: a value that starts as a safe-looking default (zero) and is only overwritten when a lookup succeeds. Zero is not safe for a uid; it's root. A lookup that can fail should either fail loudly or have a default chosen for that purpose.

Ownership going quietly wrong between build and run came up before with Podman, where podman export of a keep-id container shifted every file owner.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Docker 29.8+: Swarm containers can't reach their own node's ports under ufw

The Homelab Postmortem — Tue, 20 Oct 2026 07:29:24 +0000

Update 2026-10-23. A Docker maintainer opened a fix for the Swarm case: moby#54006, "libnetwork: DNAT gwbridge packets to ingress ports", set for 29.10.0, not yet reviewed or merged. As its description reads, IPv4 packets from a task to an ingress port on its own node would be DNATed on docker_gwbridge again, as before 29.8, so they no longer pass through INPUT, while docker-proxy keeps handling IPv6 clients and the host's loopback addresses. It also says that hairpin connections to ordinary containers' published ports still go through docker-proxy, so the plain bridge-network case in the update below is not changed by it. Until a release has it, the per-port rules below still apply.

Update 2026-10-21. It isn't only Swarm. A Docker maintainer explained in #53962 that since 29.8.0 Swarm ports are published through the same machinery as ordinary bridge-published ports, and that ufw's default policy also blocks a container from reaching a bridge-published port when the client is on the same bridge, an existing bug that Swarm inherited. Tested on a fresh Debian 13 VM with ufw as below and no Swarm: a container on a user-defined network appnet calling <host-ip>:8080, published by another container on appnet, timed out, as did a container on the default bridge calling a port published from the default bridge. A container on a different network, and the host itself, got through. The same per-port rule fixed it, on the bridge the client comes from (ufw allow in on br-<network id> to any port 8080 proto tcp, or on docker0 for the default bridge), and so did "userland-proxy": false. Docker 29.9.2 and 29.7.2 behaved the same; on 28.5.2 a container on a different network was blocked too. check-swarm-ufw-ingress.sh now checks bridge-published ports as well and names the bridge each rule needs.

TL;DR: On Docker 29.9.1 with Swarm and ufw (default deny incoming), a container on an overlay network that connects to an ingress-published port on the same node times out. The host itself, connecting to the same address and port, gets HTTP 200. The kernel log shows ufw dropping the SYN as it arrives from docker_gwbridge. A Swarm and service created on 29.7.2 passed the same test. The upstream report (moby/moby#53962) traces it to 29.8.0, which hands this connection to docker-proxy through the host's INPUT chain instead of DNATing it. The workaround in that thread, ufw allow in on docker_gwbridge, works but opens every host port to every container on the node. A per-port rule works too: ufw allow in on docker_gwbridge to any port 8080 proto tcp.

The symptom

A disposable Debian 13 VM, Docker CE 29.9.1, ufw with nothing but SSH allowed:

$ ufw default deny incoming && ufw allow 22/tcp && ufw enable
$ docker swarm init --advertise-addr 192.168.128.179
$ docker network create -d overlay --attachable appnet
$ docker service create --name web -p 8080:80 --network appnet nginx:alpine

A single node, so the service's task always runs on the node being connected to. From a container on the overlay network, through the published port:

$ docker run --rm --network appnet alpine wget -T 5 -qO- http://192.168.128.179:8080/; echo rc=$?
rc=1                  (nothing back within the 5 s timeout)

The kernel log at the same moment:

[UFW BLOCK] IN=docker_gwbridge OUT= ... SRC=172.18.0.4 DST=192.168.128.179 ... DPT=8080

From the host, same address and port:

$ curl -s -o /dev/null -w '%{http_code}\n' http://192.168.128.179:8080/
200

ss -tlnp showed docker-proxy listening on 0.0.0.0:8080 and [::]:8080. ufw disable made the container's request succeed, and so did "userland-proxy": false in daemon.json with ufw left on.

On 29.7.2 with a Swarm created on that version, the same probe succeeded with ufw on and nothing allowed. No UFW BLOCK line, no docker-proxy on 8080, and a DNAT rule for the port in the nat table's DOCKER-INGRESS chain instead. One detail from getting there: downgrading the packages to 29.7.2 and keeping the Swarm and service created on 29.9.1 still failed the same way. Only after docker swarm leave --force and re-creating them on 29.7.2 did it pass.

Why it's easy to miss

Every check from outside the container passes. The service is up with 1/1 replicas, the port answers from the host and, according to the report, from other nodes.

What fails is one container calling another service by the node's address and published port, which is common in homelab setups: a reverse proxy pointing at host:port, or an app configured with the host's IP because that's what the documentation showed. It shows up as a timeout, not a refusal, so it looks like a routing or overlay MTU problem. Those are what search results are full of.

And the trigger was a minor Docker upgrade on a machine where ufw had been configured long before. Nothing about the firewall changed.

What's really going on

Up to 29.7, a connection to an ingress-published port from a container on the same node was DNATed by Docker's rules in the nat table and never reached the host's INPUT chain. ufw's rules live in INPUT and never saw it.

According to the reporter's analysis in #53962, since 29.8.0 the per-port DNAT rule excludes traffic arriving from docker_gwbridge, which is where a container's connection to the node address comes from. With the userland proxy on (the default), that connection is delivered to the host and accepted by docker-proxy instead. Being addressed to the host, it goes through INPUT, where ufw's default deny drops it. The test here matches every part of that: the drop logged with IN=docker_gwbridge, docker-proxy listening on 29.9.1 and not on 29.7.2, the DNAT rule on 29.7.2, and both ufw disable and userland-proxy: false fixing it. The connection from the host itself never crosses docker_gwbridge, which is why it still works.

moby#53723, merged in September, fixed the same kind of path with the userland proxy disabled. A maintainer asked in the issue whether 29.9.1 still shows the problem. On this single-node test, it does.

The fix

The workaround in the issue is to let everything in from docker_gwbridge:

$ ufw allow in on docker_gwbridge

It works (the probe passed), but docker_gwbridge is where every container on the node reaches the host from, so it exposes every port the host listens on, including SSH and anything bound to 0.0.0.0, to every container on that node.

Allowing only the published ports is enough. With two services, web on 8080 and web2 on 9090:

$ ufw allow in on docker_gwbridge to any port 8080 proto tcp
  probe 8080: rc=0   probe 9090: rc=1
$ ufw allow in on docker_gwbridge to any port 9000:9100 proto tcp
  probe 8080: rc=0   probe 9090: rc=0

Each published port needs its own rule (or a range), and a new service needs a new one. Two alternatives, both observed to work: connect to the service by name on the overlay network (http://web/, which never leaves the overlay), or set "userland-proxy": false, which has wider effects on how Docker publishes ports and is worth reading up on before changing.

The toolkit's check-swarm-ufw-ingress.sh lists the node's ingress-published ports and says which ones have no matching ufw rule on docker_gwbridge. With no rules:

  DROPPED  web  8080/tcp
  DROPPED  web2  9090/tcp

UFW BLOCK lines from docker_gwbridge in the last day: 21
Docker 29.9.1, swarm + ufw: containers on this node cannot reach 2 of 2 ingress-published port(s) on this node.

It exits 1 until every port is covered, and 0 with the per-port rules or the interface-wide one. The ports come from docker service inspect, so it needs to run on a manager.

The generalisable habit

Test from where the client actually is. "The port answers" checked from the host and "the port answers" checked from a container are different claims, and here only the first one was true. When something works from the host and times out from a container on the same machine, look at the firewall's log for the bridge the container comes from before looking at routing.

The other habit is about workarounds. The first one that works is often the widest one, and an interface-wide allow on a bridge is a much bigger change than the bug it works around. Narrow it, and test the narrow version too.

This is the reverse of an older problem on this site: Docker publishing ports straight past ufw, where ufw said a port was closed and Docker made it reachable anyway. Here ufw is consulted where it wasn't before, and the traffic it now drops is Docker's own.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

Rootless Docker builds silently ignore --resource memory limits

The Homelab Postmortem — Tue, 20 Oct 2026 07:28:22 +0000

TL;DR: Per-step resource limits in BuildKit (docker buildx build --resource memory=..., Bake resources) are silently ignored when the builder is rootless, including the builder inside rootless Docker. On rootless Docker 29.9.1 with buildx v0.38.0, a RUN step limited to 256 MB allocated 600 MB and printed "not killed"; the build exited 0 and said nothing about the limit. On the same rootless daemon, docker run --memory 256m with the same allocation was killed (exit 137), so the cgroup controllers were delegated and limits were possible. On rootful Docker the build failed with 137, as intended. The upstream report (moby/buildkit#7287) points at BuildKit's rootless spec conversion, which removes the resource settings. The toolkit's check-rootless-build-limits.sh tells you which kind of builder you have and can measure it.

The symptom

A disposable Debian 13 VM, Docker CE 29.9.1 installed rootless for a normal user with dockerd-rootless-setuptool.sh install. cgroup v2 with the systemd driver, and the user's service cgroup has the controllers delegated:

$ cat /sys/fs/cgroup/user.slice/user-$UID.slice/user@$UID.service/cgroup.subtree_control
cpu memory pids

A Dockerfile whose only step pushes 600 MB through a pipe buffer:

FROM alpine
RUN head -c 600m /dev/zero | tail > /dev/null && echo "not killed"

Built with a 256 MB limit on its steps:

$ docker buildx build --no-cache --progress=plain \
    --resource memory=256m --resource memory-swap=256m .
...
#5 11.79 not killed
...
rootless build rc=0

No warning about the limit anywhere in the output. The same allocation with docker run on the same rootless daemon:

$ docker run --rm --memory 256m --memory-swap 256m alpine sh -c 'head -c 600m /dev/zero | tail > /dev/null'
Killed
rc=137

And the same build on rootful Docker on the same VM:

rootful build rc=1      (the RUN step failed with "exit code: 137")
rootful build without --resource rc=0

Why it's easy to miss

A limit that works shows up as a failure, and a limit that is ignored shows up as nothing. A build that stays within its limit looks the same either way, so the only way to see whether the limit exists is to exceed it on purpose. Most people set a limit to protect a small machine from one runaway step and never see it trigger.

The rootless daemon also behaves correctly in every other place you'd check. docker run --memory is enforced, docker info shows cgroup v2 with the systemd driver, and the limits are delegated. Nothing suggests the build path is different.

What's really going on

Before BuildKit runs a step in rootless mode, it converts the container spec for rootless use. According to the issue, and in BuildKit's v0.34.0 source, that conversion (ToRootless()) sets the spec's Linux Resources to nil and clears its cgroup path, with a comment about cgroup v1, where an unprivileged user can't set limits. On cgroup v2 with delegation an unprivileged user can, which is how rootless docker run --memory works. The build path never gets to use it. The step runs with whatever cgroup the builder has, and the request is dropped before it reaches the runtime, so there is nothing to warn about at that point.

Which code path Docker's embedded builder takes was not traced here. What was measured is the effect: the limit is gone in rootless mode and present in rootful mode, with everything else equal.

The reporter has opened two fixes, buildkit#7288 (warn when limits are dropped) and #7289 (apply them on delegated cgroup v2). A maintainer approved #7288 on 19 October; neither is merged yet.

The fix

There's no setting that makes rootless BuildKit apply them yet. The options:

Build on a rootful builder where the limits matter. On this VM the rootful build was killed at the limit as intended.
Limit the builder as a whole. In rootless Docker the builder runs inside the user's systemd service, so a MemoryMax= on that service (for example systemctl --user edit docker or a limit on user@<uid>.service) caps every build and container together. This was not tested here, and it is a ceiling for everything rather than a limit per step.
Know which one you have. check-rootless-build-limits.sh reports whether the current Docker endpoint is rootless (exit 1 if it is). With --probe it builds a one-step image that allocates 300 MB under a 128 MB limit and reports what happened. On the rootless daemon:

Docker 29.9.1 (rootless): a step limited to 128 MB used 300 MB and the build succeeded.
--resource limits are not applied here.

exit 1. On the rootful daemon: the step was killed at its 128 MB limit (137). Limits apply., exit 0.

The generalisable habit

Test a limit by breaking it. "The flag was accepted" and "the build passed" say nothing about whether a limit exists; only a step that exceeds it and gets stopped does. Keep a tiny over-the-limit probe next to any limit you rely on, and run it whenever the environment changes, here from rootful to rootless.

And when one path enforces a limit, don't assume the neighbouring path does. docker run and docker build on the same daemon took the same 256m and did different things with it.

This is the same shape as a Pi that accepted every Docker memory limit and enforced none. There it was the memory controller that was missing, so docker run lied as well. Here the controller works and one code path throws the request away.

Rootless Docker differs from rootful in one more way since 29.6: docker exec there creates world-writable files, because dockerd's umask reaches the exec'd process through containerd.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama-server can leave a keep-alive connection unread until another closes

The Homelab Postmortem — Tue, 20 Oct 2026 07:27:19 +0000

TL;DR: When clients keep more keep-alive connections open to llama-server than it has base HTTP threads, a newly accepted connection can sit with its request unread until some other connection closes. With --threads-http 4 and 32 keep-alive clients on llama.cpp b11717, 3 of 9 runs on a freshly started server had exactly one such connection: its first response took about 42 seconds while the median was 1.6 to 1.7 seconds, and ss showed its 209 request bytes sitting in the socket's Recv-Q for the whole wait. No error, no log line, and the response was correct when it arrived. With --threads-http 64 it never happened in 8 runs. The cause was in cpp-httplib's thread pool, which llama-server uses for HTTP. Its maintainer fixed it on 18 October, llama.cpp took the fixed library into build b11737 the next day, and b11750 ran the same test 12 times without a single stall. Update to b11737 or later; on an older build, set --threads-http to at least the number of connections your clients keep open. Upstream: ggml-org/llama.cpp#30675 and yhirose/cpp-httplib#2633.

The symptom

A disposable LXC container with 6 cores, the b11717 release binary, gemma-3-1b-it-Q4_K_M:

llama-server -m gemma-3-1b-it-Q4_K_M.gguf --parallel 8 -c 4096 --threads-http 4 --port 18089

The client opens 32 HTTP/1.1 keep-alive connections back to back, then each one sends POST /completion (4 tokens) in a loop for 40 seconds and records how long its first response took. That is roughly what a proxy, a chat UI backend or an agent framework with a connection pool does. A typical run:

clients 32, 40s: requests 584, median first 1.75s, max first 2.5s, slowest later 2.4s, stuck 0

And one of the bad ones:

clients 32, 40s: requests 584, median first 1.70s, max first 42.1s, slowest later 2.5s, stuck 1 #31 42.1s

Connection #31, the last one opened, waited 42.1 seconds for its first answer. Every later request on it was as fast as the rest. Sampling ss once a second during that run showed the same socket the whole time:

05:19:29 209    0          127.0.0.1:18089    127.0.0.1:33046
05:19:30 209    0          127.0.0.1:18089    127.0.0.1:33046
05:19:31 209    0          127.0.0.1:18089    127.0.0.1:33046
...

209 bytes in the Recv-Q is the whole request, delivered by the kernel and never read by the server. That went on for about 40 seconds.

The count over all runs:

Server	`--threads-http`	Runs with a stuck connection
fresh for each run	4	3 of 9 (41.8 s, 42.1 s, 42.1 s)
reused across runs	4	0 of 4
fresh or reused	64	0 of 8

The server log had nothing for any of them.

Why it's easy to miss

It is rare, and it only hits one connection. On average everything is fine: throughput is normal, every request succeeds, and the slow one eventually gets a correct answer. Unless something records per-request latency, the only symptom is an occasional user whose message hangs for most of a minute, and that looks like a slow model or a busy GPU.

It also didn't show in 4 runs on a server that had already handled a run. All three stalls were on a freshly started server, which in a homelab is the moment right after a restart or a model swap, when a proxy reconnects its whole pool at once.

And --threads-http doesn't sound like it matters. Generation is bounded by --parallel slots, so it's natural to assume the HTTP threads only parse requests and a handful is plenty.

What's really going on

cpp-httplib gives each accepted connection to a thread for the life of that connection: a keep-alive connection is one job, not one job per request. The pool starts with a base number of threads and can add dynamic threads when it runs out. llama-server sets the base to --threads-http if given, otherwise max(--parallel + 4, CPUs - 1), which is 12 for --parallel 8 on 6 cores.

The reporter's analysis in cpp-httplib#2633, which the maintainer confirmed, is that the pool only adds a dynamic thread when its idle-thread counter is zero, but that counter still includes threads that have already been woken for an earlier job and haven't picked it up yet. When connections are accepted in a burst, the pool sees "idle threads > 0" for a job that no idle thread will actually take, adds nothing, and the job waits in the queue. The queued connection is then served only when some thread finishes a whole connection: the client closes it, it hits the idle timeout, or it reaches cpp-httplib's limit of 100 requests per connection. In this test each connection made about 18 requests in its 40 seconds, so none reached the limit, and the stuck connection was served at about 42 seconds, right after the other clients finished their 40-second loops and closed their connections. So the 42 seconds is the length of the test, not a property of the bug. With clients that never close, the wait would last until one of them did.

The test here can't see inside the pool, but the observable parts match: the stall only happened when connections outnumbered base threads, it was always the last connection accepted, its request was unread rather than slow to process, and it ended when other connections closed.

The fix

Update llama.cpp to b11737 or later. The library fix went into cpp-httplib on 18 October as e97165b, "Spawn a dynamic thread when queued jobs outnumber idle threads". It changes that one condition from idle_thread_count_ == 0 to jobs_.size() > idle_thread_count_ and adds a regression test, and it shipped in cpp-httplib 0.62.0. llama.cpp updated its vendored copy to 0.62.0 in #30785, merged on 19 October, which is build b11737; b11738 is the first release with binaries.

On the same container, the b11750 release binary, the same model and client, --threads-http 4, a fresh server for every run:

fresh threads-http 4 run 1: clients 32, 40s: requests 576, median first 1.66s, max first 2.4s, slowest later 2.5s, stuck 0
...
fresh threads-http 4 run 12: clients 32, 40s: requests 656, median first 1.50s, max first 2.0s, slowest later 2.2s, stuck 0

0 of 12 runs stuck, the slowest first response 2.4 seconds, and no request left in a Recv-Q in any sample. b11717 had stalled in 3 of 9.

On an older build, start llama-server with enough HTTP threads for every connection your clients keep open:

llama-server ... --threads-http 64

With 64 for 32 connections, no run stalled on b11717 either. Size it from the client side: the connection pool of your proxy, UI or agent framework, plus some headroom for other clients. An idle thread costs little.

The toolkit's check-llama-http-stall.sh reads each running llama-server's build number and command line. A build before b11737 with more established connections than base HTTP threads is flagged:

  TOO MANY llama-server 9677 port 18089 (build 11717): 32 client connection(s), 4 base HTTP thread(s)

and the same load on b11750 passes:

  ok       llama-server 9783 port 18089: build 11750 has the fix; 32 client connection(s), 4 base HTTP thread(s)

With --watch SECONDS it also samples the server's sockets every second and reports any connection whose request stays unread for 5 seconds or more. In the b11717 run that stalled, it printed STALLED 127.0.0.1:18089<-127.0.0.1:54264: request unread for 38 s and exited 1.

The generalisable habit

Look at the maximum, not the average. A median of 1.7 seconds and a maximum of 42 came out of the same run, and only the second one is what a user remembers. Any benchmark of a server that only reports averages or throughput will miss this whole class of problem.

When a request is slow, find out where it was waiting. A non-zero Recv-Q on the server's socket means the request was delivered and nobody read it, which rules out the network and the model in one look and points at the server's own scheduling.

Another llama-server setting that doesn't mean what it seems to: --cache-ram -1 is not "no limit".

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →

llama-server's Anthropic API ignores tool_choice and stops inside reasoning

The Homelab Postmortem — Tue, 20 Oct 2026 07:26:17 +0000

TL;DR: llama-server's Anthropic-compatible /v1/messages endpoint accepts two request controls and doesn't apply them. A forced tool, tool_choice: {"type": "tool", "name": "get_time"}, is treated as "call some tool": with get_weather and get_time available and a weather question, the model called get_weather. tool_choice: {"type": "none"} is not applied at all, and the model called get_weather again. No error, no log line. Separately, stop_sequences are matched against the model's reasoning as well as its answer, so a reasoning model that writes the stop string while thinking ends the request before it answers: only a thinking block comes back, and stop_reason is end_turn while stop_sequence names the match. Both observed on llama.cpp b11717 with Qwen3-0.6B. The stop behaviour has the same shape as a vLLM report filed on 16 October, vllm-project/vllm#62085, where it breaks Claude Code's auto mode. I found no llama.cpp report of either.

The symptom

A disposable LXC container, the b11717 release binary, Qwen3-0.6B-Q8_0.gguf:

llama-server -m Qwen3-0.6B-Q8_0.gguf --jinja -c 4096 --port 18090

tool_choice

Two tools, get_weather(city) and get_time(city), the user message "What is the weather in Vienna?", temperature: 0, every documented tool_choice form:

anthropic tool_choice=null: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')
anthropic tool_choice={"type": "auto"}: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')
anthropic tool_choice={"type": "any"}: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')
anthropic tool_choice={"type": "tool", "name": "get_time"}: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')
anthropic tool_choice={"type": "tool", "name": "get_weather"}: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')
anthropic tool_choice={"type": "none"}: ([('tool_use', 'get_weather', '{"city": "Vienna"}')], 'tool_use')

All six came back with stop_reason: tool_use and the same call. Asking for get_time by name changed nothing, and neither did none. The server log had no line for any of them.

The OpenAI-compatible endpoint on the same server has its own version of this, already reported upstream (llama.cpp#30369): the named-function form {"type": "function", "function": {"name": "get_time"}} produced get_weather, and this time the log said so:

W Wrong type supplied for parameter 'tool_choice', using default value: [json.exception.type_error.302] type must be string, but is object

With tool_choice: "none" on that endpoint, no tool call was parsed, but the model still wrote one, and it arrived as text in content: '<tool_call>\n{"name": "get_weather", "arguments": {...'.

stop_sequences

A question that makes Qwen3 think first, with a stop string that shows up in its reasoning. temperature: 0.6, two seeds, both endpoints:

stop='answer should be' seed 1 OpenAI finish=stop content='' reasoning_len=1639 tail='...7 is the only number that can divide it. Therefore, the '
   Anthropic stop_reason=end_turn stop_sequence='answer should be' blocks=[('thinking', 807, '...7 is indeed a prime number. Therefore, the ')]
stop='prime' seed 1 OpenAI finish=stop content='' reasoning_len=42 tail='Okay, so the question is asking if 7 is a '
   Anthropic stop_reason=end_turn stop_sequence='prime' blocks=[('thinking', 40, 'Okay, so the question is whether 7 is a ')]

Seed 2 gave the same shape. The reasoning stops right before the stop string, the answer is empty (OpenAI) or absent (Anthropic, a single thinking block and no text block), and the request reports a normal finish. Without a stop string the same request answered <block>yes</block> after about 1,660 characters of reasoning.

On the Anthropic endpoint the reply also contradicts itself: stop_sequence says which sequence matched, while stop_reason says end_turn instead of stop_sequence. That happened in the other case too, when the stop string </block> cut the visible answer.

Why it's easy to miss

Every reply is well-formed. A tool call that is the wrong tool is still a valid tool call, and with only one tool defined, or a model that happens to pick the forced tool anyway, the bug is invisible. get_weather was also the right answer to the question here, so the forced-tool test only shows the bug because the forced tool was deliberately the wrong one.

The stop case looks like the model's fault. An empty answer with finish_reason: stop reads as "the model decided it was done", and small reasoning models do sometimes ramble and stop oddly. The only clue is that the reasoning ends mid-sentence exactly where the stop string would have started.

Both matter most for agent tools pointed at a local server. Claude Code can be pointed at any Anthropic-compatible endpoint. According to vllm#62085 its auto mode sends a classifier request with stop_sequences: ["</block>"]. In my runs Qwen3 never wrote </block> inside its reasoning, so I didn't see that exact request fail here. But any stop string a model can write while thinking will do it.

What's really going on

For tool_choice, llama-server converts the Anthropic request to its OpenAI-style internal form (tools/server/server-chat.cpp). Reading that conversion in the source (master and b10703), {"type": "tool", "name": ...} becomes OpenAI "required" and the name is dropped, and {"type": "none"} has no mapping, so the default "auto" stays. "Required" means "call one of the tools", which the model did, and nothing tells it which one. The results above match that: any and the named forms behave identically. The same conversion has been reported dropping other fields too: output_config.effort (llama.cpp#30564), and per its fix PR #30608, thinking: {"type": "disabled"}. Neither was tested here, and #30608 does not touch tool_choice.

For stop strings, the server checks them in process_token against the slot's whole generated text (find_stopping_strings(slot.generated.text, ...) in tools/server/server-context.cpp). The split into reasoning and answer happens later, when the reply is built. So a stop string is found wherever it appears in the generated text, including the reasoning section, and the reply is then built from whatever was generated before it.

Whether the Anthropic API defines stop_sequences to apply to thinking is a fair question I haven't settled. But an empty answer is clearly not what a client asking for a stop sequence wants, and a reply with stop_sequence set should have stop_reason: "stop_sequence".

The fix

There's no server setting for either one. What helps on the client side:

tool_choice: don't rely on forcing a tool through /v1/messages. If a step needs one specific tool, send only that tool in tools for that request, with {"type": "any"}; since any does work as "call some tool", that should force it, though it wasn't tested here. To prevent tool calls, leave tools out of the request instead of sending {"type": "none"}.
stop_sequences with a reasoning model: pick stop strings the model won't write while thinking, such as a closing tag of your own format, or turn reasoning off for requests that depend on a stop string. With llama-server, --reasoning-budget 0 turns it off server-wide. That wasn't tested in this run.
Detect it: treat a reply with no text block (or empty content) plus a set stop_sequence as this failure, not as an empty answer, and retry without the stop string.

The toolkit's check-llama-anthropic-controls.sh sends four small requests to a server and reports each control. Against this server:

  IGNORED  tool_choice {type: tool, name: get_time} -> called ['get_weather']
  IGNORED  tool_choice {type: none} -> called ['get_weather']
  MATCHED IN REASONING  stop_sequences ['prime'] ended the reply inside thinking: blocks ['thinking'], stop_reason 'end_turn', stop_sequence 'prime'
  note     stop_sequence is 'prime' but stop_reason is 'end_turn'

exit 1. With a model that doesn't call tools or doesn't reason (gemma-3-1b here), it says the tests are inconclusive and exits 2 rather than passing.

The generalisable habit

A compatibility layer is a translation, and translations lose things silently. When a server says it speaks another vendor's API, test the parameters you actually depend on, and design the test so the right answer is different from the default: force the tool the model wouldn't pick on its own, and choose a stop string the model will write where you don't want it to stop. A test where the correct behaviour and the broken one give the same output proves nothing.

The same pattern of a control being accepted and then quietly overridden by the model's reasoning came up with Ollama: a thinking model skipped the JSON schema when it answered directly.

The OpenAI-compatible Responses endpoint on the same server has a related gap: a reply cut off by max_output_tokens is reported as completed, tool calls with half their arguments included.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.

See the toolkit →