DEV Community: prodbymarcu

Your fork can be 467 commits behind without anyone noticing

prodbymarcu — Sat, 14 Nov 2026 11:09:43 +0000

Every project I run on my own hardware has a version pinned somewhere in time, and I keep assuming that pin is recent. On Nov 13 I checked the assumption instead of trusting it, and the gap was bigger than the calendar suggested. My local build of mw00/project-maya sat at commit b042645, tagged v1.1.34, dated Oct 27. Upstream pushed 2026-11-11T16:25:46Z. A compare against HEAD came back with 467 commits ahead and at least 300 files changed on the first page alone, CHANGELOG.md up 691 lines, CMakeLists.txt +409/-17. The releases API said the same thing in different units, 25 releases behind. The repo was created 2026-10-06, so five weeks of history existed when I found out, and I had absorbed none of it.

What the diff actually showed

Nobody hands you a number for drift until you ask for one. Dates feel current because nothing failed, and nothing failing is not evidence of anything. The measurable version took three calls, clone HEAD plus GitHub compare plus the releases list, and the answer was four digits wide.

That number also decided what reading the release notes meant. I went through v1.1.59, published 2026-11-11T14:13:32Z with 2,824 characters of notes, and the headline was "Fewer refusals. Faster GLM-5.3." Two claims, one sentence, and the sentence told me nothing about which half applied to my box.

The half that applies here

The refusal ablation ships as an opt-in switch, set from the Models page or per request through a key named ablate, and answers stay unchanged while it stays off. Their held-out test prompts moved DeepSeek-V4.1-Flash from 98% refused to 1%, MiniMax-M3.1-Flash from 100% to 7%, GLM-5.3-Flash from 100% to 4%, with no new refusals on ordinary requests. Fitting runs through tools/ablate_fit.py, which now covers MiniMax and GLM-5.3 alongside DeepSeek.

The checkpoint check ran against my own weight directories first. I hold deepseek-v41-flash-1017 at 234 GB including its UD-IQ1_M shards and Maya-S24 at 91 GB, so the checkpoint they tuned against is a checkpoint I already own. Same family, same quant class, same file layout.

The other headline half needs a 170.0 GB quant and roughly 176 GB of memory held together, so it stayed out of scope that day and I never opened it.

Why pulling was cheap that morning

The lane serving that engine answered 000 on both /v1/models and /health, so nothing was resident where the rebuild would land, while my coder lanes kept running through their own router. A rebuild costs compile time and nothing else, no GPU teardown sat in front of the pull. Headroom got measured before anything downloaded, 19 GB free on the volume I normally build on against a 1.1 GB clone, with a 261 GB fallback mounted elsewhere if the build tree needed room. Their setup note asks that the largest quant's VRAM and RAM hold stay together so expert reads never page off SSD, and that is a question about memory layout during generation. My question that day was about bytes free before the download.

The ETA came out at 45-70 minutes including the rebuild, and the run hasn't happened yet. Nothing in this story claims the ablation reproduces on my hardware, only that the tooling exists and my weights match what they fitted.

Workarounds age against features nobody knew shipped

Before that sweep, refusal-free weights on my box came from re-pulling community quants, chased by two scripts in my build directory, check-atomicchat-ds41.sh and pull-atomicchat-ds41.sh. That hack family existed because upstream had no switch for it. One line in one release note retires it, provided the numbers reproduce on my hardware, and I read that line three weeks late.

The repo itself is MIT licensed C++, 2,795 stars and 391 forks deep, with 154 open issues sitting on it. A hundred and fifty-four open issues on a project that ships 25 releases in five weeks is another reason drift can't be tracked by memory, the issue list moves faster than any changelog you read.

The rule that survived

Three calls per project would catch this on schedule, four hundred and sixty-seven commits of surprise would not. The compare only ran because a release post on Nov 11 put two claims in one headline, and deciding which claim applied to my box meant knowing exactly what my box was running. Answering that took clone HEAD plus GitHub compare plus the releases list, and the answer came back in four digits. Nothing in my vendor tree runs those three calls on a cadence, so the next stretch of drift hides until the next interesting headline makes me count.

Tell me how you find out. If your answer is "I don't", then what you're running is whatever commit you last happened to pull, and the number of releases sitting under that commit is a question you can answer today with two commands.

A package upgrade on a full disk broke my boot path

prodbymarcu — Thu, 12 Nov 2026 23:13:40 +0000

Two entries on my homelab PC's boot menu load byte-identical kernels, one into an emergency shell at 19:58:37 on Nov 8, 2026, the other into a working desktop from the same screen. The difference between them is one token on one line, and the file that holds that line opens with a bare integer. That's the half nobody writes up when they're chasing a dead boot, because all the attention goes to whatever the failed attempt logged.

The front door is an index into stanzas

/boot/limine.conf starts its default stanza with default_entry: 2. Limine resolves front-door entries by index into stanzas further down the same file, so the number tells you which block gets mounted, nothing tells you what that block is. Each stanza ends in its own mount flags, and those flags are where the two boots actually differ. The front door is the second of them.

The distro keeps its own copy beside the live config, limine.conf.old, sitting next to /boot/limine.conf before I touched either one. A diffable artifact exists on every bootloader edit whether you make one or not, and reading it beats re-deriving which entry changed.

Same binary, different tree

Both boots report the same UKI path and the same hash family, os-release reads 4.0.4 for BUILD_ID and VERSION_ID on both, and the running kernel is 7.2.5-3-omarchy in each. The cmdlines part company at one flag.

rootflags=subvol=@
rootflags=subvol=/@/.snapshots/6/snapshot

Same kernel bytes, same initrd bytes, two different userspaces underneath. That narrows the search before anyone opens a log. The kernel loaded fine on both attempts, and the tree it was pointed at held what the running kernel needed on one of them and not the other.

An entry name tells you how you got there, the flag string tells you where you land, and nothing in either one hints at which tree gets mounted.

Any tool that edits boot entries by label is editing the wrong axis. What matters is the flag string inside the stanza your index resolves to, and finding it means resolving that index against the rest of the file.

What your disk is actually made of

The root volume sits on /dev/mapper/root, LUKS holding btrfs, and the operating system isn't what fills it. Under one home directory live 99 GB of services, 64 GB of fish, and 51 GB of voicechat, all under my model weights. The pacman package cache carries another ~9.6 GB in its own @pkg subvolume.

So the room an upgrade needs doesn't come from the distro, it comes from folders you own, and a package manager asks for that headroom without checking whether the extraction fits before starting. Nothing reclaims the pile automatically either, snapshot pruning included, since the cleanup unit errors while the root is an overlay and the count just keeps climbing toward whatever update runs next.

Why this is worth checking before your next upgrade

A full-disk interruption mid-extraction leaves a userspace half-swapped against a kernel that installed fine, and the only record is a line at the tail of a log nobody opens. Two things decide whether that state is reachable from your front door: how much room your largest owned folders leave behind, and which stanza your default index points at.

Check both while the machine still boots, from whichever session survived. Every number above came off ssh on Nov 8, 2026, from inside the snapshot boot, with the broken subvolume mounted beside the live one, two separate trees on one machine. Nothing in that pass moved data, rebooted anything, or asked the box to prove itself.

df -h says what's claimed, lsblk -f and cat /etc/fstab say what the mounts expect, journalctl --list-boots turns your last reboot you didn't attend into a queryable session, and mounting a subvolume with -o ro,subvol=/@ beside the live system compares two trees without moving anything. All of it read-only, all of it over ssh, none of it needs a power button.

How many lines in your bootloader do you actually know? Mine has one flag per entry, and the difference between two of those flags decided whether I got a desktop or an emergency shell.

My updater refused eight swaps, then rebuilt one anyway

prodbymarcu — Thu, 12 Nov 2026 11:09:03 +0000

An update script that refuses to touch a live binary is only safety when the thing it checked is the thing that runs. Mine refused on eight logged mornings, and on one of them it refused while rebuilding, because its input was a version file and the box had two different versions of the same engine on disk and in RAM.

What the guard saw versus what ran

The November 6 run opened with the disagreement sitting in plain text. The running engine banner read strata 0.2.11 (build 0.1.57), the engine's own BUILD.json read version 0.1.44 with src 3ef48793b3. Two numbers, one process, and the updater trusted the file, so it printed its refusal: engine 0.1.44 is in use, wants at least 0.1.46, run again after closing the serving window.

Then the same documented ./update.sh kept going. It rebuilt the CUDA sm_86 engine and strata-vision from current main sources in about two minutes, ggml 0.24.0 at commit 9f7d977d, rewrote BUILD.json to version 0.1.57 with src e098a67d9f6e6a24, and reported all six model config entries up to date. Nothing asked me to close a window before that work happened, no lane teardown, no parked engines.

So the guard refused, the rebuild shipped, and both engines stayed resident across the update. VRAM held around 20.9 and 23.4 GiB through the update, the two llama-swap services re-spawned their MainPIDs at 09:59 mid-run, and every HTTP probe answered 200 with zero downtime. The deferred step landed on the very run that declined to do it.

Why the refusals never expired

Repo tip walked v0.2.2 to v0.2.13 between October 22 and November 10 while the recorded engine sat at 0.1.44 for six of those mornings. Each entry logged the same deferral beside a green verdict, and the build it wanted moved with each release, 0.1.48 twice in October, then 0.1.46, 0.1.55, 0.1.46 or higher twice more, then 0.1.58 and 0.1.59 after the November rebuild. Eight separate runs, eight refusals, one standing debt note in every entry.

What keeps the debt open is the premise behind the refusal. The instruction says close the model window, meaning stop whatever holds the binary in memory. On this box the unit that owns the engine is masked, so no such window exists to close. Two other services own both engines, and their restarts pick up new binaries on their own schedule. A guard checking an owner nobody runs can only ever refuse, and a follow-up visit that would clear the debt has nothing to wait for.

The stale file also made the guard cite two different numbers on consecutive days. November 6 said 0.1.44 was in use, November 7 said 0.1.57 was in use and wanted 0.1.58, November 10 said 0.1.57 wanted 0.1.59. Same script, same line of reasoning, three answers. I only caught the drift because the banner and the JSON disagreed, and the rebuild rewrote the file to match what was running.

What stayed pinned through all of it

I kept weights untouched deliberately, and proved they were already current by byte count. The HF Coder GGUF IQ1_M files are 29,608,446,496 and 28,800,138,432 bytes upstream, matching my local copies exactly, same revision as the repo listing from October 19. Sampling pins and the conversation-cache settings (mine pinned at 3072 slots over 3 layers) never moved, so the swap changed code paths without changing what the lanes serve.

Post-update verification asked the router for generated text, and the timings came back prompt 110.6 tok/s, predicted 60.5 tok/s, draft_n 5 with all 5 tokens accepted on coder-iq1_m. A 200 response only proves a socket is open, those timings prove the new drafter path compiled and runs, and 5 out of 5 accepted means the sm_86 CUDA build picked up the speculative defaults it was supposed to.

The two record keepers that lied

The updater isn't the only thing keeping version state here, and its partner was wrong for longer. A monitor script decides whether to apply a release by reading cached values, never by asking git, and those cache lines froze at v0.1.41 on Oct 21 and never moved through Nov 10. Live git describe reached v0.2.13.1-63-g87dfca0e in that window, so the monitor printed "NEW" every morning, a rescue arm then re-read the live repo, found the work already landed, and wrote nothing. Nine such mornings between November 2 and November 11, all nine caused by the frozen cache, none by an upstream release.

Then November 6 added the reverse failure. I stopped the watchdog timer before the update and restarted it after, and it ended inactive, because the restore script aborted on the masked unit under set -e. The step that was supposed to bring the sentinel back died on a component that doesn't exist on this box, and the run reported everything else green. Nothing printed an error worth acting on, the timer line just read inactive next to a completed surgery.

The two record keepers were wrong in opposite directions. One trusted a file that had fallen behind the process, so its guard refused forever. The other trusted a cache that had never been refreshed after Oct 21, so it re-fired forever and every re-fire cost a full verification pass with zero writes. Safety logic keyed to state nobody updates turns into either a permanent refusal or a no-op, and the version mismatch sitting in the log was the only honest signal either of them produced.

What I changed after eight mornings

Treat a refused step as an open job with a re-run attached, since a deferral beside a success verdict is indistinguishable from done by anything reading the log. Diff the recorded build against the running banner after every self-update, both numbers, before trusting either. And when a service is masked, drop the "close the window" instruction from the procedure entirely, because the guard can't be satisfied and pretending otherwise makes the debt look renewable when it isn't.

What does your updater check before refusing, the file it wrote last time or the process actually holding the binary? Mine checked the file, and the box had two versions of the truth.

Chunk size decides whether a 33-minute audiobook can ship

prodbymarcu — Wed, 11 Nov 2026 23:10:51 +0000

Long-form speech synthesis almost never fails at the top level, it fails one chunk at a time, and what you get back is still a file. On 2026-11-09 I turned a 1,023,437-byte single-page build into 33 minutes of listenable audio. Three different chunk shapes produced three different wounds in that script, the shipped mp3 survived because I counted parts and probed its tail, and none of my throughput numbers had anything to do with which take I kept.

What the page was

The source was one HTML document, fetched with curl against a Mozilla user agent, chapters read off data-chapter attributes: Introduction, The Three Caps, The Three Paths, Precedents, A New Computer, Next-Gen Chips, A Million New Computers, Endnotes cut. Stripping svg, canvas, img, nav, footer and chart-label fragments left 5,976 words across 34,806 characters, footnote arrows gone too. Em dashes became commas and curly quotes went away before any model saw the text, since TTS reads punctuation as a pause cue and half of those glyphs aren't in its vocabulary. My usual page reader died on this URL outright, the local SearXNG backend is search-only, so stdlib HTML textify did the job.

Three chunk shapes, three wounds

Shape one was the obvious one, hand the CLI all 34,806 characters at once. It died with NoAudioReceived. It also left an mp3 on disk, 232 seconds long, in the same output path my next pass would have picked up. That's the trap, a corpse that looks like a short book.

Shape two was 2,400-character chunks. Each request ran 2 to 5 minutes, and one of them came back empty, chunk_005 sitting at 0 bytes while everything around it had real audio in it. In a concat list a zero-byte part contributes nothing at all, so the book just skips mid-chapter. The first serial pass gave up at chunk 5, and nothing printed a warning for the empty part. Duration arithmetic caught it, ffprobe against what the narration should cost.

Shape three survived, built from sentence-aligned chunks under 1,100 characters with three workers, a 150 second per-chunk timeout, four retries with backoff, and split-in-half as the fallback once four retries burned out. That shape produced 35 parts, the smallest above 60 KB, and a 25 second tail sitting at RMS -21.4 dB.

Why the cold load is the wall clock

The local lane loads its model every invocation, about 4.5 seconds cold each time, and I paid it 34 times for ~1,100-character chunks. That works out to roughly 153 seconds of model loading inside a synthesis leg of about 12 minutes, arithmetic on two numbers from my own log. Chunk count multiplies the fixed cost, so chunk size trades against request count in ways no spec sheet shows you. The first full pass also died because my foreground call had a 90 second timeout, and the longer jobs went to background with a heartbeat.

The gates that made the file trustworthy

Three checks before delivery, all cheap. Part census, nothing zero-byte and nothing under 60 KB. ffprobe duration, 1987.52 seconds against the narration length, which is where a missing chunk would show up as a gap in the math. Tail RMS, since a truncated render usually keeps its head and loses its end. The concat and encode then ran as ffmpeg -y -f concat -safe 0 -i wavlist.txt -c:a copy into one wav, then -c:a libmp3lame -q:a 5 to mp3.

That encoder spelling is a trap worth its own paragraph. This brew ffmpeg build has no lame alias, -acodec lame dies "Unknown encoder" and -c:a av_m4a dies "Encoder not found", while the standalone homebrew lame CLI rejects -m and -h before filenames because its preset parse differs. I burned time on all three dead spellings and kept the one the encoder list actually prints.

The ear beat the clock

Two complete renders came out of that day. Phonon S 31M in voice Easton, q8, English only, came back as 12,688,964 bytes at 51,074 bps mono over 1987.52 seconds. Edge neural in voice Andrew at +6% rate came back as 13,259,324 bytes at 48 kbps mono over 2209.85 seconds, which is 222 seconds longer than the local take but built from thirty-five parts against thirty-four on the local lane. I auditioned four male voices on one line before committing to Andrew.

The local take got dropped. It carried artifacts and zero emotion across half an hour of narration, the network service had breath, pacing and actual emphasis on the same words, and four male voices on one line, played back to back, is what decided it. Judging a single sentence is a far cheaper check than judging thirty-three minutes of finished book, and that short audition is where the local model's artifacts showed up.

What shipped

The kept file is local-ai-opportunity-andrew.mp3, 2209.85 seconds, delivered as an attachment. The rejected Phonon take stayed on disk next to it, because the artifact log is the thing that makes the retry recipe reproducible a week later. The chapter-split m4b with bookmarks stays open, I never built it that day.

Two rules carry into any long synthesis pipeline. Chunk shape owns your fixed per-invocation cost, and part census plus tail RMS are the proof that a long file is complete. What's your bar before trusting a thirty-minute render, do you count parts or just ffprobe the total?

Copied benchmark rows are the loudest tell on a model card

prodbymarcu — Wed, 11 Nov 2026 12:25:54 +0000

A model card table carries two kinds of number and the risk is lopsided. Competitor columns, the MiniCPM and Freeze-Omni and PersonaPlex rows everybody keeps printing, come from someone else's paper, so you can check them against that paper in ten minutes without downloading a gram of weights. The hero row is the release's own score, and that one stays unverifiable until someone runs something. So the borrowed half is the testable half, and when it matches another lab's table to one decimal place you've proved those baselines were lifted rather than typed, which locates where the citation printed on that page points before you read a single claim.

I swept three releases that way across Nov 8 and Nov 9, then re-checked the first one two days later. Nothing needed a download to decide anything.

The table that cited somebody else's research

The org is adventists-ai, 16 models. Their family page Sophon-Realtime-v1 is a README only (5 files, zero bytes of weights). Weight repos sit next to it: Sophon-Realtime-32B-Tag at 44.61 GB over 68 files created 2026-10-29 with 202 downloads, Sophon-Realtime-32B-Flash at 40.89 GB over 55 files created 2026-11-07 with 10. Their GitHub repo, created 2026-11-04 and pushed 2026-11-08, is Apache-2.0 Python, 176 tree entries, 8,227 lines.

Their VoiceBench row printed MiniCPM-o 4.5 at 76.1, Freeze-Omni 55.2, PersonaPlex 30.6, Moshi 29.5, NemotronLabs at 55.1. The arXiv id tagged on their own model page is 2609.21967, which resolves to NVIDIA's paper, "NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities", submitted 18 Sep 2026 and last revised 1 Oct 2026. That paper's Table 3 matches those five rows exactly, faithful copies of someone else's baselines.

Sophon's own scores go the other way. VoiceBench 82.8, Big Bench Audio 81.8%, FDB v3 Pass@1 48% and NVV-SuperBench 0.86/0.80 appear in no public table I could open, and the string 82.8 does not occur anywhere inside the cited paper. Their card notes that 82.8 ties GPT-4o-mini-Audio at 82.84, a half-duplex model scored in the same column as a full-duplex one. Zero HF discussions on any of it.

Four read-only checks that made it decidable

Resolve the arXiv id, diff the competitor columns against that paper's table, grep the hero number's digits inside it, then git ls-files looking for an eval harness. Twenty minutes, all read-only, nothing moved onto my disks. The fourth check came back empty, their repo has no eval or bench directory at all, which is what turned "82.8" from a claim into an uncheckable one.

Two days later the popularity tripled and the evidence did not

Stars went 12 to 162, forks 2 to 10, Python lines 8,227 to 12,023, HF downloads 202 to 356 on Tag. New E2E weights published at 79.0 GB over 37 files, and the README grew a methodology footer: "Self-run unless noted", VoiceBench judged by official evaluators behind a gpt-4o-mini judge, Big Bench Audio judged by GPT-4.1 on the transcript, FDB v3 reported as best of several runs.

That footer is better disclosure than the Nov 8 card had, and it's also the sentence shape that makes an unaudited number read like it was audited. git ls-files still shows no eval, bench or harness code anywhere in the repo, only docs/assets/bench/*.png chart images, nothing a stranger could run against those charts, so 82.8 stays self-reported with more prose around it.

The case where the claim was reproducible, and still optimistic

DSpine spec-decode drafters for the dense Qwen3.8-27B plus a llama.cpp fork (Ajay9o9/llama.cpp, branch dspine, HEAD d493f84 pushed 2026-11-08). Delta against upstream is one commit, "dflash : add DSpine draft model support", 8 files, +323 lines. Unlike Sophon, this drafter repo carries four HF discussions with third-party measurements on Intel Arc SYCL, AMD ROCm, DGX Spark GB10 and RDNA3 Vulkan, cross-hardware community checks that make the claim testable at all.

Stock llama-server refused it here (0.9.0-dev b299 6414d5b): done_getting_tensors: wrong number of tensors; expected 73, got 58. Their fork loads it. I built that branch with -DGGML_CUDA=ON in about fifteen minutes and ran it on my RTX 3090 against the quant their card recommends, Qwen3.8-27B-Q4_K_M (1.258 GB drafter), flags -fa on --jinja temp 0, three reps per prompt. No-spec decode measured 42.7 code / 42.7 prose / 41.8 math tok/s; DSpine n-max 15 hit 84.0 / 56.8 / 79.8 and n-max 7 gave 61.5 / 46.2 / 61.2, with draft acceptance 20.5% of drafted tokens at n15 against 35.6% at n7 for code.

Their card claims 94 to 132 tok/s on a stock 3090 with this drafter, I measured 80 to 84 hot on the same class of card, so the direction is real and the top line sits roughly 15 to 40% high. Their no-spec baseline is 28.2 against my 42, different prompt and context settings, which is where their multiplier stops matching mine. Prose gained least across the three arms, and their own chart caveat says why (the llama-bench prompt there is novel prose, "quite unfavorable for drafting").

The third claim broke further back in the chain

A third drop claimed "Qwen3.8-27B 82 to 381 tok/s on one RTX 3090". Tracing the reposts (an Aug 21 post, Reddit r/LocalLLaMA comments/1vtup5s, then GitHub syv-ai/qwen38-27b-rtx3090, which now redirects to syv-ai/HyperQwen, Apache-2.0, created 2026-08-15, pushed 2026-11-09, 2,634 stars) shows the caption naming llama.cpp where the source thread and the repo both run vLLM, so the engine in the headline was wrong before any number got checked. Their README pairs 127 tok/s single-stream at 64k with SPEC=dflash2 plus prefix cache and reaches 381 only in quoting mode at 4 slots/56k with DFLASH_TOKENS=15, a setting that crashed EngineCore (marlin_gemm dispatcher aten::empty allocation failure) on a >20k-token quote request here. The repo now pins vLLM==0.33.0 per issue #440 dated Nov 6 and no issue mentions that signature, which leaves 381 unrun against the current release.

What the checks decided

Sophon stays parked unless an 80 GB card lands on the box, a 4B-thinker variant shows up in that org listing (it doesn't), or they publish a repro harness. DSpine belongs as a per-model option for the dense 27B brain when llama.cpp serves it, since warm decode on my resident coder lane sits at 82-84 tok/s while DSpine lands at 50-72 depending on depth. HyperQwen stayed parked at SPEC=dflash2 plus PREFIX_CACHE=1 with no DFLASH_TOKENS line, since the resident engines hold both cards.

Four read-only checks per release and I never argued with a score. Copied rows decide where a table came from, a missing harness decides whether its own scores mean anything.

Which number on a model card would you need to see before installing weights: the hardware floor, the citation resolving to their own paper, or somebody else's run on your class of GPU?

A watchdog that finds nothing still pays a full agent turn

prodbymarcu — Tue, 10 Nov 2026 23:47:49 +0000

Every unattended loop I run carries a cost I never priced until last week's audit, and it comes from ticks where nothing changed. Nothing moved on disk, nothing broke, no news landed, and the loop woke a language model anyway to tell me what I already knew. That floor measured 317 million tokens across ten days on my cron board before any gate sat in front of it, and getting it down took two kinds of gate plus one bug hunt.

What the audit counted

I pulled totals out of usage_audit.jsonl for every scheduled card on my board, one ten-day window, no cherry-picking. 57 cards total, split 32 scheduled, 12 paused, 13 completed one-shots that had never been cleaned up. The three loudest cards by token spend:

article loop        18 runs   avg 6.1M tok    110.7M total
feed curator       477 runs   avg 217k tok    103.7M total
release watcher    139 runs   avg 256k tok     35.6M total

The article loop was doing real work every time it ran, so I left it alone per my own quality-over-savings rule. The other two were watchdogs. They check state, they report state, and most ticks there is no state change to report.

The feed curator is the loudest watchdog number on that board, and here is what it did. It ran every fifteen minutes as a full agent, 477 times inside the ten-day window, averaging 217k tokens per run for 103.7 million total, and 100 of those runs ended silent. Nothing gated the wake, nothing gated the report, so the model paid itself for checking state it then described.

Two gates, one for waking and one for reporting

The cheap half is the wake gate: stop asking an agent to do the tick at all. I moved that card to script mode, a shell script under my profile's scripts/ dir, sourcing what it needed from my zprofile, running the curator, reading the stats file, and pushing the feed digest when it kept anything. Its test run printed this exact line, rc 0:

kept=3 new=3 pushed=yes items_local=3

Then I checked the push landed where a reader would see it, raw GitHub copy returning HTTP 200 with size 2255 and three <item> tags, and the repo log carrying 2026-11-08 curate tick 01:12. Script mode moves the tick off the model entirely, since a shell script that pushes or exits has no reasoning step to bill for.

The expensive half is the report gate, and that one needs more than silence between invocations. A monitor gate only fires when its payload differs from the last payload, so the payload has to be built from state that changes when something actually changed. My release-watcher gate digests four things: one line per tracked slug for its review phase, the done-marker list off the staging directory, any pending post line while shipping without a marker, and a ledger line count. No timestamps in the digest, because a timestamp makes every tick unique and that defeats the gate. Two consecutive runs came back byte-identical, so the digest is deterministic, and luck would have shown up as a difference on the second run.

The homelab watcher took the same recipe one step further, because healthy there means nothing worth saying at all. Two runs came back rc 0 with byte-identical empty output, so the agent stays asleep until something breaks, and its next scheduled tick sat at 2026-11-08T08:23. The figure above is that board, counted before any of it got gated.

The bug that made a gated loop look broken

Converting the feed card exposed a quiet-tick bug that no amount of reading would have found. My manual run printed this line, and then nothing else:

feed tick: no new items but files dirty (not pushing): M stats.json

The branch assumed a dirty repo meant work needed doing, and stats.json carries an at timestamp, so it is dirty on every single tick. Every healthy cycle became a logged complaint. Replacing that branch with an unconditional silent exit when kept and new are both zero gave me two consecutive runs at rc 0 with empty stdout, while git status still showed only that one modified stats file. Dirty does not mean interesting, and a gate has to decide which files carry signal.

Same hunt caught a second liar in my own verification code, this one worse because it failed quietly. I was counting feed items through ElementTree with .//item, and once a feed declares a namespace those tags stop matching that path, so the count came back as zero without raising anything, and the remote leg printed a question mark with no number behind it. A verify step that reports "0 items" for a valid feed is a trap any parser can walk you into. Both legs went to grep -c '<item' and the check now fails unless HTTP 200 AND remote count equals local count. Verified with one item in the local file, one item in the raw GitHub copy, live tick rc 0 with empty stdout, bash -n clean.

Restoring a deleted card mints a new id

One more guard is worth naming because it came out of cleanup work. A job killed by mistake cannot be un-killed by re-adding its prompt from memory, and the replacement gets a fresh id even though the old one never comes back. The restore that worked read the prompt straight off the board's backup file, then diffed the new entry against the old: schedule every 20 minutes, model, provider, delivery on success, delivery on failure, workdir, paused state, all identical, 4,352 characters of prompt byte-for-byte the same. Retyping a 4,352 character prompt would have quietly changed what that card does, and I would have shipped a different automation than the one I meant to bring back.

Cleanup itself shipped 17 graveyard cards removed, 13 completed one-shots, 3 stale paused one-shots, and one cloud keepalive for a decision layer I retired, while two watchdogs got gates and one parked card went back into rotation after I found no reason recorded for pausing it. The board ended at 40 cards. Every gate got checked by running it twice back to back, since one clean run says nothing about a payload that might be varying on its own.

What this decides

Price the tick where nothing changed, then multiply it by how often nothing happens. That product is most of your bill before a single prompt gets rewritten, and mine came out at 317,485,418 tokens over ten days across 57 cards, with one ungated every-fifteen-minute watchdog writing 103.7 million of it from 477 runs at 217k each. Gate the model call, gate the report payload, build that payload out of state that changes only when the thing it watches changes, and let two identical consecutive runs be your proof.

How many of your scheduled jobs are still waking a model on cycles where nothing changed, and what did you pay for that last week? If you can answer that one, you have already found your own worst offender.

Booting from a snapshot hides the upgrade that broke

prodbymarcu — Tue, 10 Nov 2026 11:15:15 +0000

My homelab PC went to an emergency shell during its default boot at 19:58:37 on Nov 8, 2026, and every boot after that looked fine. Fine because I kept choosing the snapshot entry by hand on the same menu where the default entry drops you into that shell. Nothing downstream of the pick ever reported the failure, and my whole working day ran on top of a system whose normal path was already dead.

When your recovery route sits one keyboard pick from the front door, the broken path and the working path look identical to you, and only the journal separates them. That's the shape worth naming before any fix, because a boot that succeeds only proves the tree it booted from was good.

What the dead boot said

The failed entry's own kernel cmdline carried rootflags=subvol=@, the plain subvolume, and then the log ran in this order:

systemd-modules-load: Failed to insert module 'crypto_user'
error=No such file or directory for every /lib/modules/7.2.5-3-omarchy/**/*.ko.zst
boot.mount: Failed with result 'exit-code'
Failed to mount /boot.
Dependency failed for Local File Systems
local-fs.target failed
Started Emergency Shell

Seven log lines carry the chain, one missing file class causes all of it. The running kernel wanted 6,490 compressed modules and the tree it booted from had zero of them under that kernel version's directory.

Meanwhile llama-swap.service reports active since 19:59:42 and both RTX 3090s answer nvidia-smi at 19.6 GB and 23.3 GB used. Everything I use was up. The thing that was down was a path I wasn't using, and nothing in the desktop separates those two states.

Counting files inside the thing you mounted read-only

Finding the cause meant mounting the subvolume the broken entry points at, read-only, next to the live system, and comparing the same path across two trees. Same directory, two answers: 6,490 modules on the tree I actually booted from, 0 inside @. That directory still held 1.1 GB of other kernel versions, and the rest of the @ tree looked complete at 23 top-level entries and 334,618 files under usr.

That's the trap in snapshot tooling generally. A rescue tree is a full userspace, so it passes every casual glance you'd give it, and the one file class your running kernel needs is missing from it. You can list it, stat it, walk its parents, and never notice until you count.

Why nobody reported the death

The upgrade that emptied it quit mid-extraction. The last line of pacman.log reads [ALPM] error: could not extract /usr/lib/libwebkit2gtk-4.1.so.0.24.4 (Write failed) at 2026-11-06T16:30:39. Two snapper snapshots bracket that attempt, snapshot 6 stamped 16:26:42 and snapshot 7 stamped 16:27:12, thirty seconds apart, and the failed write landed three minutes and twenty-seven seconds after snapshot 7.

The disk had 12 GB free out of 930 GB formatted, with 917 GB used and 99% of that capacity claimed, and one weights directory holding 366 GB of the pile. So a package manager ran out of room mid-extraction on a filesystem already at 99%, left the userspace half-swapped against a kernel that was installed, and returned nothing to anything reading later. The entire record of the event is one line in pacman.log, and it's still the last line in that file.

Three units fail every day on a box that looks alive

The healthy system carries its own permanent noise, all of it caused by the same overlay root:

systemd-remount-fs exits 32 with mount: /: fsconfig() failed: overlay: No changes allowed in reconfigure.
The swap unit points at /swap/swapfile, which doesn't exist. zram0 at 62.7 GB covers real swapping, so hibernation is just quietly dead.
snapper-cleanup errors IO Error ... subvolume is not a btrfs subvolume, because / is an overlay.

That third line means snapshot pruning never runs. The count climbs toward whatever upgrade comes next, the disk sits at 99%, and the unit that was supposed to clean up after itself has been erroring since the root turned into an overlay. None of it changes your desktop, your prompt, or a single fan. That ledger lives in systemctl --failed.

The repair had to clear two rules

No package updates on this box, no teardown of the inference lanes, both from my standing rules for it. Repointing the main Limine entry's rootflags at the known-good snapshot subvolume satisfies both, does what my manual pick already did, moves nothing on the disk, and unwinds with one cp against a backup I kept at /boot/limine.conf.bak-20261108-bootfix.

Two other routes got rejected before anyone ran them. omarchy snapshot restore is the distro-native path, but its confirm needs a tty and dies off-console over ssh, which is how I work on this machine. Subvolume rename or send surgery would have invalidated every other snapshot entry's cmdline path, because each of those still points at the old name. On a boot path, an edit you can't undo from a terminal isn't an edit worth making.

The guard inside the patch script

The script asserts its precondition before touching the config: exactly one match of rootflags=subvol=@ rw rootfstype=btrfs in /boot/limine.conf. Zero matches or two means the file drifted while I was composing the edit, so it exits without writing, and it copies the original to the backup path first either way. It also arrived as a file over scp into sudo python3, never an inline heredoc through ssh, because quoting through two shells is where these kinds of edits get mangled.

Verification was grep counts on the live file after the write: 0 remaining occurrences of the old flag string, 2 lines carrying the snapshot path, and a diff between backup and current that showed only line 31 changed. Reading the config back is what makes a bootloader edit trustworthy; a silent successful write proves nothing about which entry the menu will pick.

What shipped, what didn't

Shipped: a default boot entry aimed at a tree holding the modules the running kernel needs, backed up byte-for-byte, checked by grepping the config after the write. Unshipped: nobody has rebooted that box since the edit. An ssh session can't press a power button, so the claim stays unproven until someone picks the default entry at a console and watches it come up without menu help.

The second open item is a trap worth handing to anyone else with a nearly full root filesystem: clear the space before an upgrade runs, because during one the room is already gone and the extraction dies where it stands. Package caches and old model tiers are the pile worth cutting first, and the blind pruning loop above is why that pile keeps growing.

How many things on your machine have a fallback one step deep? Mine had a boot path, a swap path, and a snapshot pruner all failing in silence while the desktop looked normal. journalctl -b -1 on the last reboot you didn't attend is the fastest way to find out. What does it say?

A voice model that never stops talking isn't duplex

prodbymarcu — Mon, 09 Nov 2026 23:08:49 +0000

On 2026-11-06 I stood up a full-duplex speech-to-speech assistant on a Mac Mini M4 with 16 GB of RAM, four tests deep on stock llama-server builds, and my own client interrupted the model mid-answer at 240 ms time to first audio. The claim this backs: nothing upstream listens for you talking over an assistant. llama.cpp's realtime WebSocket API has no server-side VAD and emits no speech_started event, so interruption handling is entirely your client's job, and a demo that skips that job is a press-to-talk loop with a better name attached to it. Latency, quant math, even whether the machine owns a microphone, all get checked after that one design decision.

What ran

Model is LiquidAI/LFM3-Omni-2B in GGUF, text plus vision plus audio in, audio out through LiquidCodec at 24 kHz, input audio read at 16 kHz. Server side was llama-server 0.9.0 from Homebrew (I bumped the engine from 0.5.0 that same day) behind -t 8 -ngl 99 --reasoning off -c 8192, endpoint ws://127.0.0.1:9931/v1/realtime. Upstream support isn't hypothetical here, PR #30636 landed the WebSocket base, PR #31269 adds realtime conversation plus transcription and states explicitly it works with LFM3-Omni-2B-GGUF, whose official minimal client ships as examples/realtime_client.py on the openai SDK.

Test B first, the stock client against my server: first output 126 ms after end of input, a 5.44 s answer streamed clean. Then offline speech, test A, where llama-tts on Q8_0 spent prompt eval 0.27 s plus generation 1.54 s to produce 2.88 s of audio, RTF 1.59x, second sample 1.61x. The model manufactures speech faster than playback before any duplex plumbing exists at all, which is why the interesting part of this lane was never throughput.

My duplex loop

voice_loop.py, mine, in ~/Developer/lfm3-voice. Client-side VAD runs silero_vad.onnx (from the Liquid4All cookbook) over a PortAudio RawInputStream, and when speech is detected mid-answer the client sends response.cancel. Measured turn: I interrupted at stream position 5.06 s while the assistant was still talking, cancel fired, 2.80 s of queued speech got dropped, status returned cancelled with TTFA 240 ms and RTF 1.9x. Follow-up turns completed at TTFA 240-244 ms, RTF 2.17x and 2.21x, and the speakers rendered 15.82 s of audio through RawOutputStream with two-channel interleave to device 2, the Mac mini Speakers.

Reading server-realtime.cpp shows the server agreeing with that path: cancel() keeps the session alive and the partial stays in history, so an interrupted turn costs nothing structurally. The captured log for every barge-in ended as stop: cancel task then a clean slot release. One recurring warning, embeddings required but some input tokens were not marked as outputs -> overriding, prints on every LFM3-Omni run and is benign audio/mm noise.

One PortAudio quirk bit the capture path directly: blocksize-0 raw streams come up stopped on both input and output until you call .start() explicitly, symptom being a silent capture and zero segments from a VAD that never got fed.

The microphone census

sounddevice on this box returns BlackHole virtual inputs (16ch and 2ch) plus Mac mini Speakers as output, and nothing else. Recording ambient returned RMS 0.0, recording while afplay blasted a file through the speakers returned RMS 0.0 as well. A Mac Mini M4 has no physical mic, which is why my loop ran against fed wav audio until USB hardware showed up. My capture pipeline passed with a fed wav through the same path (--vad-debug probed 6 s clean, zero segments from the silent feed) so every stage works except the one needing air. A voice assistant on a desktop-class mini stays a test harness until someone plugs in USB hardware, and one call to sd.query_devices tells you that before you design the loop. The client already carries --mic-device N for the day the hardware arrives.

Three traps upstream, all still open

Read against llama.cpp issues on master v0.9.0 dated 2026-11-04, three bugs stand unfixed on my build. #31689: /v1/realtime ignores requested output language and voice, non-English comes out speaking with an English accent, while llama-tts --tts-lang honors it. #31696: session.update asking for 24 kHz input is accepted silently and the audio is still read at 16 kHz, so the fix is sending the mic at 16 kHz. #31436: forcing audio-only modality degrades generated speech to gibberish, interleaved text plus audio instructions work fine.

The shape they share is the API accepting configuration it doesn't implement and returning success, with the damage visible only in the audio. All three pass a code review and only show up against real audio.

Space math on a small box

Free disk decided the quant choice. BF16 was off the table on space, so Q8_0 became the primary weight set. The pair is 1.846 GB model plus 507 MB mmproj, 2.35 GB total, and free space on that volume went from 8.5 GiB down to 6.4 GiB during the download. BF16 at 4.4 GB would have left under 2 GiB on that volume. Server RSS sits around 2.4 GB resident for the Q8_0 pair.

Same build then serves chat through a mac-mini provider with declared context length 65536 (engine -c 65536, KV roughly 48 KB per token derived from 24 layers times 4 kv heads times 128 head_dim in f16), trained window is 128K so no rope scaling was needed. Proof: hermes chat -Q --oneshot --provider mac-mini -m lfm3-omni-2b returned "pong" with rc=0 in 68 s wall, while a raw warm round-trip against /v1/chat/completions did 10 tokens in 0.65 s near 15 tok/s. A 2B on a mini answers prompts fast and agent turns slow, so that lane belongs behind voice output plus small tool schemas. Persistence came from launchd unit com.alfred.lfm3-omni-server.plist with KeepAlive=true across logout, crash and reboot, macOS 26 rejecting the newer LaunchServices dialect for third-party services and taking the classic Label/ProgramArguments schema instead.

What I'd want from someone else's demo

Latency numbers here are one machine, one quant, four test runs, sample size of a handful of turns. Directionally the duplex loop stays under half a second to first audio all night, but nobody should read TTFA 240 ms as a cross-hardware benchmark.

The transferable part is architectural. Server-side speech-to-speech APIs in llama.cpp ship cancel semantics without listening, so the client owns turn-taking, and any claim about full-duplex behavior needs a capture path that exists on the box running it. Two checks before you adopt someone's repo: grep their client for the cancel call, then query devices for a microphone. Both take minutes and both decide whether the thing can work at all, long before a benchmark table matters.

What's your interrupt path? If you can't name the exact call that stops the response mid-answer, tell me what happens when you talk over your assistant.

Benchmark scores say nothing about whether the release runs

prodbymarcu — Mon, 09 Nov 2026 11:12:49 +0000

On Nov 8 I went through a fresh speech-model release the way most people in that scene go through them, model card first and benchmark table second, and I never argued with a single score. Their own README asked for one 80 GB GPU, my largest card carries 24 GB, and the disk holding my weights had 12 GB free against roughly an 80 GB download. The hardware gate killed it twice before accuracy came up at all.

The claim underneath that story: adoption decisions in the open-weight scene get made from a model card and a benchmark row, while the two hardware gates, VRAM floor and free disk, rarely get checked before the download starts. Both failed here, and the eval number behind them can't be checked either.

The gate nobody checks

The org is adventists-ai, 16 models. The family page Sophon-Realtime-v1 is a README only, 5 files, zero bytes of weights, 32 likes, downloads untracked. The weight repos sit next to it: Sophon-Realtime-32B-Tag at 44.61 GB over 68 files, created 2026-10-29 with 202 downloads and 3 likes, Sophon-Realtime-32B-Flash at 40.89 GB over 55 files, created 2026-11-07 with 10 downloads, plus a research preview E2E repo.

Their README states the floor plainly: Tag needs "one 80 GB GPU, ~65 GB per instance", with stage0 at gpu_memory_utilization: 0.62 and stage1 at 0.08. E2E wants two 80 GB GPUs. My fleet maxes out at 2x RTX 3090, 24,576 MiB each, so 48 GB total across both cards, no single card anywhere near 65 GB, and their deploy shape has no supported split for that layout. Then disk: df -h on the volume holding everything came back 930G size, 917G used, 12G free. Their requirement is roughly 80 GB of weights. Driver and CUDA were fine (610.57.04 against a floor of 580), which is the only gate that passed.

What the card actually is

The GitHub repo adventists-ai/sophon-realtime was created 2026-11-04 and pushed 2026-11-08, Apache-2.0, Python, 176 tree entries, 8,227 lines across the tag/flash plugin, the sophon_expressive engine and a live gateway.

Underneath the packaging sits a stack of models I already run. thinker/config.json declares model_type qwen3_vl, arch DuplexJevVisionForConditionalGeneration, dtype bfloat16, but quantization_config is compressed-tensors float8 with channel-symmetric static weights and dynamic fp8 activations, and the thinker shards total about 33.7 GB on disk, already FP8. Their NOTICE file says the thinker derives from Qwen3-VL-32B-Instruct through DuplexJev-32B-Para plus a merged tag-writing LoRA, the audio encoder derives from Qwen3-ASR, the talker is tag-tts off Fun-CosyVoice3-0.5B with the flow distilled to 4 steps, and the Flash talker comes from Qwen3-TTS-12Hz-1.7B-Base. Three stages share one GPU over CUDA MPS: thinker, talker, then flow plus a HiFT vocoder in TensorRT. Packaged, it is one vLLM-Omni pipeline with three stages resident at a time, and that residency is where their 65 GB per instance comes from.

The citation check

The arXiv id printed on their own model page is 2609.21967, which resolves to NVIDIA's paper, "NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities", submitted 18 Sep 2026 and last revised 1 Oct 2026. That means the citation printed on their model page points at research from another lab. Adventists AI ships no first-party writeup and no eval harness, git ls-tree shows no eval or bench directory, and the repo has zero HF discussions.

The comparison rows they borrowed are faithful: their VoiceBench entries for MiniCPM-o 4.5 at 76.1, Freeze-Omni 55.2, PersonaPlex 30.6, Moshi 29.5 and NemotronLabs 55.1 match that paper's Table 3 exactly, so the baseline rows are copied straight off NVIDIA's table. Sophon's own scores go the other way. VoiceBench 82.8, Big Bench Audio 81.8%, FDB v3 Pass@1 48%, NVV-SuperBench 0.86/0.80 and paralinguistic 48.6 appear in no public table I could open, and the digits 82.8 do not occur anywhere inside the paper they cite.

Two more gates before any install

License. The headline says Apache-2.0 weights, and Tag, Flash, tag-tts plus the family README check out per file on cardData, LICENSE and the GitHub license field. The DuplexJev checkpoints under the same org are CC BY-NC 4.0. Personal use is fine, commercial use isn't, and the marketing line hides that difference.

Network defaults. live/sophon_live/backend.py sets SOPHON_LIVE_BACKEND_URL with a default pointing at Alibaba's DashScope compatible-mode endpoint, so delegated lookups leave the box unless overridden to a local lane. finetune/tools/regress.py opens a URL against an OpenAI-compatible judge for its regression checks, and the sibling duplexjev repo advertises api.adventists.cn/duplexjev as its demo endpoint. Reading the tag and flash plugins myself turned up network calls only to localhost once weights land, so the engine is local and the gateway wrapped around it isn't. Every lane I own is supposed to answer without leaving the house, so a cloud default is something I patch first and test after.

The one piece that fits my cards

The only piece fitting one 3090 today is tag-tts standalone at 7.73 GB, Apache-2.0, which takes tagged text ([laughter], [sigh], <strong>) and performs it. My brain models already emit those tags, so it's a candidate swap for the TTS stage in my duplex voice lane without needing the 32B thinker. Unverified, because that repo ships no standalone server, talker_tag.py references upstream cosy_server.py from the FunAudioLLM/CosyVoice runtime. Adoption means running their weights under CosyVoice's code path.

DuplexJev-4B at 8.5 GB could serve turn-taking and barge-in decisions over audio locally behind duplexjev-vllm==0.3.0, with the CC BY-NC pin standing in front of any commercial use.

Flagship Sophon stays parked. What would change the verdict: a single GPU of 80 GB or more on the box, or a 4B-thinker variant of Sophon, which does not exist in that org listing, plus about 80 GB of free disk. And a published repro harness, which is what turns 82.8 from self-reported into something I can test against my own lanes.

The pattern that carried the verdict is order of operations. Hardware floor from their README, free space on the target volume, citation resolution, license pin per file, network defaults in the gateway code. Five checks, all read-only, all inside twenty minutes, and they answered "is this worth installing" without one gigabyte moving onto my disks. A release that passes those gates still needs an end-to-end run against whatever it replaces, but a release that fails them never earns a benchmark line in your notes.

What's your habit here? Do you try the weights before you check the VRAM floor, or has someone got a card big enough that the ladder never matters?

A demo that never errors still never ran

prodbymarcu — Sun, 08 Nov 2026 01:00:43 +0000

A WebGPU inference page can hand you four facts at once and none of them disagree: the UI renders, the GPU context comes back live, your click is accepted, and zero audio elements exist in the document. That was the state of my probe on November 4, against a WebGPU text-to-speech runtime advertising first audio in single-digit milliseconds, and the silence ran 75 seconds with no error event anywhere in the tree. Every browser-resident claim inherits this trap, since a placeholder dash reads as "pending" and as "refused" until something counts nodes over time.

What they promised

Their post claims "26x real time on an iphone. in safari. from a model under 40 MB brotli'd, same voices and quality as Gradium's own runtime". Their page puts numbers behind it, medians of 10 rounds: 115x vs 31x realtime on an M3 Max, 73x vs 21x on an RTX 2080 Ti, 26x vs 13x on an iPhone 17 Pro, first audio 3 ms vs 15 ms on the Mac, 11 ms vs 44 ms on the phone, 46 ms fresh versus 373 ms cached for their package. Weights re-quantised to 8-bit integers at 43.7 MB, hosted at huggingface.co/blixt/phonon-31m-en-webgpu, 24 kHz output, speed control 0.77x-1.3x, tested in Chrome on macOS and Windows, Safari on iOS, Firefox on a Pixel 10 Pro. The repo checked through the GitHub API came back blixt/phonon under Apache-2.0, 21 stars, description "Phonon 31M text-to-speech on WebGPU, with word timings". None of those figures are mine, they're their sampled medians on hardware I don't hold.

The census that made it falsifiable

The browser was driven over CDP, /json/version reporting Browser "Chrome/156.0.8078.79" while the UA string still printed HeadlessChrome/156.0.0.0. On https://blixt.xyz/phonon/ the page rendered fully, 41.5 KB of visible text, buttons enumerated. document.createElement('canvas').getContext('webgpu') returned a live GPUCanvasContext, so the acquisition path worked and the context exists before any click.

Two attempts then followed at the same button, one element click and one real click_at_xy dropped at its bounding-box centre, 720,346. Seventy-five seconds later the panel still wore dashes across download time, first audio and real-time factor, and the document counted zero <audio> nodes alongside zero error nodes.

Their own latency sets the bar

Their published cold-start figure is 46 ms on a fresh page, my wait was 75 s, so I sat there 1,630 times past their own budget before committing to a word, and the word I committed to was refusal. A page genuinely loading moves something under it, cache state ticking up or an element eventually appearing in the tree, while a page asking for patience shows a spinner. Forty-six ms of promised work against 75 s of unmoving dashes and no audio element anywhere is a runtime declining to run.

One candidate explanation got eliminated inside the same session. CVE-2026-118306 landed on NVD at 2026-10-27T23:16Z with the description "Use after free in FileSystem in Google Chrome prior to 156.0.8078.73 allowed a remote attacker to execute arbitrary code outside the sandbox via a crafted HTML page", severity Critical from the Chromium team, and my build sits one digit past that fixed-in revision at 156.0.8078.79, so the advisory predates the browser I was holding. What the runtime actually objected to stayed unresolved, since no console line surfaced, no permission prompt appeared, and nothing in the tree explains why GPU and audio capture went quiet. My own note for that night files it under "silently declines to run in this headless-ish context", wording I kept because the condition matters more than the count.

The same lie one level up

The same session carried a second case of the identical lie, one level up from inference. Thread pages rendered zero visible characters there, innerText length 0 against 314 KB of downloaded HTML with no anchors anywhere inside it, and nine replies that another endpoint reported simply never arrived as text. The request finished clean and the document came back empty, which is what a demo reporting success over zero audio nodes does, and both keep passing because the question their checkers ask is "did the request finish", while the question worth asking is what landed in the DOM.

What I measured here

Their browser-resident figure never got measured on my hardware that day, so every bar below carries either the word claimed or the word measured. What I do have measured is the local lane running the same checkpoint: Phonon 31M English from hf://gradium/phonon-31m-en through ptts 0.4.0, 31M parameters, 64.5 MB q8 weights, 24 kHz, RTF 0.043, about 23x realtime, time-to-first-chunk 8-23 ms, WER 0% on Easton and Miles. Same model family, same Apache-2.0 license on the 31M English weights.

Their headline speed beats that figure wherever they tested it, 115x claimed against ~23x measured here, and that gap gets stated as a loss, since a runtime I never measured can't be the reason to move anything. What genuinely transfers from their page are three pieces my lane doesn't have: inference resident in the browser with no server process, per-word timestamps so captions and read-alongs line up, and a 43.7 MB int8 repack under my 64.5 MB file.

Making a demo falsifiable before you believe it

The guard I'd apply to any browser-resident model claim runs in this order of cost.

Ask the page for its own latency budget first, then wait a bounded multiple of it. A generic network timeout measures how long you're willing to sit, their promise sets the bar. A 46 ms promise turns 75 s of silence into a verdict without needing an error event. Count output nodes while it runs, an audio element is either in the document or it isn't, dashes say nothing about either case. Separate the acquisition path from the action, since a live getContext('webgpu') result rules out permission refusal and leaves the ignored click as the remaining suspect. Check your own engine revision against any advisory that names one, because fixed-in builds sit one digit away and read like the culprit when they aren't. And when a runtime needs a user gesture for GPU or audio capture, a scripted click can never produce the number, someone has to press Speak once in Safari or desktop Chrome, and that's a human step worth writing down beside the missing number so the next session inherits the condition along with it.

One habit keeps the record usable six months out: write each counter beside the build, the clock reading and the sample size that produced it, plus the limits of what it failed to show, so the number and its condition travel together into the next session. A headline number with no census under it is either true or a placeholder nobody ever checked, and nothing on the page separates them.

What does your page do when WebGPU refuses, render a sad panel, throw a console line, or sit there like mine did? Name the node or the message that surfaces it and I'll tell you which failure mode yours is hiding.

An advertised batch limit proves nothing until sampled

prodbymarcu — Sat, 07 Nov 2026 11:10:06 +0000

A server that prints batch_limit: 4 in its own health endpoint has only told you what it was configured to accept. Four slots, four concurrent generations, and four requests that actually overlap on the GPU are three separate claims, and the field covers none of them. I found that out while trying to give one 24 GB card real batching, over two evenings of staged prototypes, and every number below came off probes I ran against my own engines.

The endpoints lied by omission

The installed inference server on the card answered /health with {ok: true, busy: ..., backend: exl3, context_length: 262144} and its sibling llama-swap member reported total_slots=4 next to n_ctx=262144. Both numbers looked like capacity. Neither was measured. Reading the server source over ssh is what settled it, serve_openai.py lines 45 and 315-318 hold a single gen_lock wrapping generation, so concurrency in that file is batch-1 asyncio, one generation at a time, extra requests waiting their turn.

The engine underneath had the same split. ExLlamaV3's own generator.py defaults max_batch_size=256, then caps it to cache.num_slots, and the hybrid cache initializes at -ambs 1. So even after I deleted the lock, the default left a second serializer in front of the one I removed. Removing gen_lock alone cannot produce multiple active sequences, the flag has to move too.

That is why I stopped trusting status fields at that point and wrote probes that count jobs while they run.

Counting jobs while they run

Staged one moved GPU steps off the aiohttp event loop with serialized enqueue and cancel handling, launched isolated on its own port, /health reporting batch_limit=4 against a shared aggregate cache, 22542 MiB resident on the card. verify_batch.py exited 0, peak_active_jobs came back 2 across 8 sampled health checks, both SSE streams generating at once and reaching finish_reason=length in 4.5-4.8 seconds.

Two overlapping generations, sampled eight times, was all I could honestly claim from that build. The probe proved overlap, stream termination, and that the health endpoint stayed readable mid-prefill. Answer quality, retrieval depth, production readiness, and four live jobs were all outside what it showed, so I wrote those four limits into the worklog entry beside the number, where they stay attached to it.

Staged two added an actual admission path. verify_batch4.py observed peak_active_jobs 4 across 31 samples, every stream ending at length. verify_function.py returned 4 alongside a correctly typed add(2, 3) tool call, so batching survived a tool round-trip. verify_cancel.py came back active=0, busy=false after a client disconnected, which is the one thing a naive batcher gets wrong most often, a cancelled request leaving a ghost job holding VRAM.

The same build ran verify_health.py while long prefill was in flight: 76,056 prompt tokens processed, 73 health checks served, zero errors, maximum health latency 72 ms. That is the argument for moving generator steps off the event loop, a health endpoint behind a blocking GPU step answers once per token batch, a health endpoint on a sliced loop answers in tens of milliseconds while the same prefill runs.

The proof that belonged to a different build

Then the long-context check. batch-needle.json records 240,118 real prompt tokens, both embedded facts retrieved, finish_reason=stop, 443.45 seconds wall time. Beautiful, that is the deepest retrieval I ever measured on that engine, and it happened on staged one. Staged two holds the four-job overlap. Neither artifact validates the other's configuration, and the queue entry I wrote at 6 pm says peak 4 active jobs with a separate line for the needle run, because reading them as one result would be a claim about a build that never had both properties tested together.

One habit fixes this class of mistake: every probe line gets written next to what it sampled, its sample count, and its termination reason. A number without those three is decoration.

Context math sets the ceiling the fields ignore

The model card says 262,144 native window, 64 layers with full attention every fourth layer, so 16 KV-bearing layers, 4 KV heads at head dimension 256, ideal raw KV at the full window 16 GiB in fp16, 8 GiB at 8-bit, 4 GiB at 4-bit, before scales, recurrent state from the linear layers, or server overhead. Four independent full windows at 4-bit need 16 GiB of KV plus weights, on a card with 24 GiB total. An aggregate pool shared across requests is a smaller promise than four private windows, and the health endpoint cannot tell those two apart. Neither can /props.

The same trap one level up

Worse than the engine's echo was my own runtime's. Both agent configs set single_session_lock and the router reports a local_backend_busy banner when it refuses a lane, yet grepping the shipped Hermes source found zero implementations reading either name. A config key nothing consumes is a comment wearing a boolean. Before I flipped any routing I needed the effective whole-turn gate reconciled against the batched design, which meant the flag had to be read somewhere, and it wasn't. Dead code beside a live-looking setting is how a self-heal loop ends up re-applying a patch that changes nothing.

What shipped

Production stayed the original single-generation server both nights. Every staged prototype ran isolated on its own port, cleanup stopped only its own process, restored the services and the watchdog timer, and cleared only markers whose prior mtime and size matched what we created. One morning both llama-swap units entered stop at 10:22:09 and timed out while stopping at 10:22:14 exactly when an empty maintenance marker of unknown ownership appeared on the box; no permission was inferred for clearing another owner's flag, and idle GPU occupancy was never treated as evidence that work had finished.

So the honest sequence for adding batching to your own local engine, in order of cost: read the serializer in the serving layer, check the runtime's own batch default behind it, sample active jobs during a prefill long enough to matter, then record what each probe did not prove next to its result. Configured capacity fields are the last thing you look at, first thing you doubt.

What does your health endpoint actually measure on a busy engine, milliseconds or honesty? If yours answers with a configured number, name the field, name the line of code that sets it, and I'll tell you whether it can ever disagree with reality.

A heal loop keyed on a deleted file heals nothing

prodbymarcu — Fri, 06 Nov 2026 11:28:22 +0000

My runtime pulls upstream code on a schedule, five local patches have to survive that churn, and the watchdog decides whether a patch is still needed by grepping for one marker inside one specific file. One night that file stopped existing upstream. The grep started returning nothing forever, the loop concluded all five patches were missing, and it re-applied dead modules onto current code on every tick, indefinitely, ticks running minutes apart. Nothing crashed, nothing logged a failure. The box looked healthy and quietly wrong, and that flavor of bug is the expensive kind, no stack trace, no error string, just hours of reading before anything points at a cause.

What happened

Upstream dropped the built-in messaging platform module, gateway/platforms/bluebubbles.py, gone from the repo tip while my install still carried it from an earlier pull. Two of my October-era patches target that file, so when I replayed the bundle against a fresh checkout those two conflicted and everything else applied clean, exactly as designed. The piece that wasn't designed for this was Step 0 of the self-heal script, a condition asking "is this symbol present in this path" before deciding whether to re-apply anything. The symbol lived inside the deleted file. Missing means apply, there's no third state, so the guard decided every cycle that my hand-written armor was absent.

The fix moved the marker to a symbol that survives rewrites, _recovery_notice_opted_out_platforms() in gateway/run_startup.py. The two dead patches went into a retired directory excluded from the glob, bb-patches/retired-20261102-stale-vs-current-upstream/. Then I ran the heal condition standalone against the committed tree, and it printed NO_HEAL_CORRECT. Then I let the live loop run and read its log, cron ticks at 06:46Z and 06:49Z both reporting webhook healthy with zero heal spam and no patchfail marker on disk. The script's own syntax check was refused by a security guard that scans the file body and treats embedded lifecycle commands as mutating, so those two ticks are the proof, bash -n on the full script being unavailable on this path.

How the bundle gets proven

Applying today says nothing about surviving the next upstream rewrite, so durability gets tested against a fresh checkout:

git worktree add /tmp/bb-heal5 origin/main
git am --3way   # sorted order, both live patches

Both live patches applied clean in sorted order, post-apply marker counts came back 3 inside run_startup.py and 11 across the plugin adapter (_inbound_guids, _TYPING_SKIP_LOOKUP). The two October-era ones failed where expected, on the deleted module path. I want that ledger settled before the loop runs, once it has decided the bundle is missing the damage repeats every cycle.

What the test suite proves, and doesn't

scripts/run_tests.sh reported 16 failed and 27 passed after my edits. Run against HEAD with my changes stripped out, the same 16 test identifiers fail either way, identical names, and every trace carries the home_io_guard refusal, 40 occurrences across them, that guard blocking gateway.run bootstrap reads of ~/.hermes/manifest.json on a production checkout. Green is unreachable in this environment for reasons unrelated to the change, so pass counts carry no signal here at all. A delta that says "still 16" would have been luck, a delta over failure names says zero new failures, which is what actually happened.

I also didn't force it green. The readiness step restarts the live messaging gateway mid-turn, an interrupted turn is exactly the hazard the skills warn about, so the runner stays red until a safe boundary. The full verify attempt timed out at 120 s inside its build/test phase and never reached boot; the listener stayed put on :8642 with zero restart entries in the watchdog log, which is how I know nothing bounced.

So the semantic proof came from four checks that don't import the runtime:

py_compile returned rc 0 on all three touched files.
AST introspection shows the startup mixin holds both _recovery_notice_opted_out_platforms and _redeliver_failed_obligations_for_platform, so the new self._... call binds to an existing sibling method. A typo in that position never fails locally, it fails at deploy, which is why the AST read exists.
Keyword-only signatures match their call sites exactly, sweep_recoverable taking [deliverable_platforms, deliverable_targets, notice_suppressed_platforms] against the boot call, sweep_failed_for_runtime taking [profile, notice_suppressed] against the runtime call added that session.
Direct probes against an isolated state.db printed the marker table: a suppressed row ships an explicit empty string, an unsuppressed row ships None, and the flood row keeps FLOOD_MARKER intact.

The opt-out semantics worth keeping

The feature was small, the states were not. A config flag kills the "Recovered reply ... may be a duplicate" prefix that redelivered messages wear while keeping the redelivery itself, read off the union of launch config and served profile configs, fail-open by default. I set it in both profile blocks plus the host config at lines 1622, 984 and 951, then verified by parsing the same blocks the helper reads, which printed two opted-out platforms.

That empty string carries the design, and it's the reason this section exists. Suppression ships an explicit '' in the claimed row because None is what the runner reads as "fall back to my restart wording", so a suppression that writes nothing produces a message wearing a different lie. And the rate-limit label stays suppressible-proof: FLOOD_MARKER never gets dropped, since hiding it would make an overloaded lane look normal.

One grep found a second casualty

Sweeping current HEAD and origin/main for every October hardening turned up four gone from both trees at once: the one-bubble platform note in gateway/session.py, _burst_media_redirect_eligible in run_busy.py, _user_row_exists_with_same_identity in transcript_repair.py, and the skip_webhook sender fix. All grep zero on either side. The upstream rewrite deleted the file they lived in and nobody re-ported them, so my protections stopped existing on both sides silently, and the only thing that told me was running the search.

Four rules

Key the heal check on a symbol that survives rewrites, and run the condition standalone once before trusting the loop, so a resurrect bug shows up in one print you can read before the loop has churned anything. Compare failure names across before and after when the environment can't go green, counts are noise there and identity sets carry the delta. Prove wiring with AST introspection plus signature-vs-call-site matching when imports are blocked by design; the guarded classes never execute their bodies locally, which means reading is the honest proof. And after any upstream rewrite, grep your own hardenings against both trees, the deletion path is silent enough that only a sweep finds it.

Where does your CI stop working when the file it greps for goes away upstream? If the answer is "it just re-applies", you have the same silent resurrection I had, and no log line will ever tell you.