DEV Community: John

Reading a SQLite Database With the macOS CLI Left an Empty WAL File and Tripped My Freshness Alarm

John — Fri, 13 Nov 2026 00:00:05 +0000

Originally published on hexisteme notes.

On 2026-11-02 at 09:00:48, the scheduled job in my agent telemetry pipeline raised F3-SOURCE-DB_UNAVAILABLE with the reason active WAL present. My first question was whether the raw archive had been lost. It had not. The next 7 ticks of the job all exited with rc=0, and the archive count rose monotonically from 10231 to 10346 over that stretch. The source database was fine. The check that read it had made a decision from the wrong evidence.

What the check was doing

The pipeline keeps a SQLite database of agent sessions, and a freshness check reads the newest ended_at from it to decide whether the data is current. The check is meant to be a passive observer. Its rejection message says so directly: "read-only observer will not create or mutate SQLite sidecars". To keep that promise it opens the database with an immutable=1 URI, so the read does not create a -wal or -shm file of its own.

That choice has a cost the code accepts on purpose. An immutable read does not pick up frames that are still sitting in the write-ahead log, so when the WAL holds frames, the value it returns can be stale. The check therefore refused to read whenever a -wal file sat beside the database, and reported DB_UNAVAILABLE instead of a freshness verdict.

The refusal was keyed to the file's existence. Any -wal file, of any size, meant "unavailable".

The reproduction

On 2026-11-06 at 07:35 I reproduced the condition. Reading the database with the macOS system CLI, /usr/bin/sqlite3, and doing nothing but reading, left a 0-byte -wal file and a -shm file next to it. That is enough to trip a rule that only asks whether the file exists.

I have not looked into why the CLI leaves those files behind after a read, and I am not going to explain SQLite internals I did not check. What I can say is narrower: on this machine, with this database, a read through that binary leaves an empty -wal file, and my check treated an empty file the same way it treated a log full of frames.

This is not the first time a read of mine tripped this check. In Fixing a Python fromisoformat Error Unmasked a Stale Alarm That Had to Fire, a direct query I typed during an investigation, through Python's sqlite3 with a mode=ro URI, left the same pair of sidecar files and produced the same DB_UNAVAILABLE (active WAL present). That time I changed the caller: I deleted the files by hand and switched my direct queries to immutable=1. The rule stayed as it was. This time the trigger was a different client, the system CLI, and fixing my own habits again would only have covered the next client I happened to think of. So this time the rule changed.

What I could not pin down

I do not know what opened the database on 2026-11-02. The reproduction shows one way an empty -wal file appears, not that a CLI read is what happened that morning. The record is also short on the recovery: after the weekly build at 09:30, the check was normal again from the next tick. I did not confirm what that build did to the sidecar files.

The fix: size, not existence

The new rule looks at how many bytes are in the -wal file. A 0-byte file means there are no frames to apply, so this check now lets an empty WAL through. Only a file larger than 0 bytes still blocks the read. This is the relevant part of _read_ended_at after the change:

        try:
            wal_bytes = Path(str(path) + "-wal").stat().st_size
        except FileNotFoundError:
            wal_bytes = 0
        if wal_bytes > 0:
            return None, "active WAL present; read-only observer will not create or mutate SQLite sidecars"
        uri = path.as_uri() + "?mode=ro&immutable=1"

A missing file and an empty file now take the same path. A WAL that still holds frames keeps the old behaviour, because reading past it with immutable=1 is exactly the case where the answer could be stale. I wanted to drop the false alarm without giving up the reason the alarm existed.

Two tests, one for each side of the line

The first regression test builds a small database, puts an empty -wal and a zeroed -shm of 32768 bytes beside it, and expects a FRESH verdict. It also snapshots the directory before the call and asserts the snapshot is unchanged afterwards, so the observer still does not create or modify any files while it reads. I confirmed this test failed before the fix.

def test_zero_byte_wal_left_by_cli_read_is_readable(tmp_path):
    home = _home(tmp_path)
    db = _make_db(tmp_path / "wal.db", ["2026-10-08T12:00:00Z"])
    Path(str(db) + "-wal").write_bytes(b"")
    Path(str(db) + "-shm").write_bytes(b"\0" * 32768)
    before = _tree_state(tmp_path)
    assert _freshness(db, home)["status"] == "FRESH"
    assert _tree_state(tmp_path) == before

The second test pins the other side. A -wal file with some bytes in it must still produce DB_UNAVAILABLE, with the same reason text as before. Without it, a later change could relax the rule to "never block" and the first test would still pass.

def test_nonempty_wal_stays_unavailable(tmp_path):
    home = _home(tmp_path)
    db = _make_db(tmp_path / "wal.db", ["2026-10-08T12:00:00Z"])
    Path(str(db) + "-wal").write_bytes(b"frame")
    report = _freshness(db, home)
    assert report["status"] == "DB_UNAVAILABLE"
    assert "active WAL present" in report["reason"]

The full suite passed with 147 tests, and the shared telemetry environment check passed. I also ran a CLI query against a copy of the production database and then ran the check against that copy. It produced no alert output.

The next weekly report

The weekly report generated after the fix shows the check reading the source normally. Its ingest health section reports source freshness as FRESH, with source_data_age_seconds=1923.098247 and the newest ended_at at 2026-11-09T00:00:03.108000Z. The latest ingest log line in the same report is 2026-11-09 09:00:34 rc=0 source_readable=4636 archive=10443 prev_archive=10428, and the F3 line reads normal. That is a single report after the fix, so it tells me the fix did not break the normal path. It does not tell me the check has stopped producing false alarms in general.

What is still open

I still do not know what opened the database on 2026-11-02. If it was something other than a CLI read, there may be another way to produce an empty -wal that I have not seen yet.
The alert writes a flag file. I deleted it by hand on 2026-11-06. Nothing clears that flag automatically when the condition goes away, so a false alarm stays raised until a person removes it. That path still does not exist.
The size test assumes that a -wal file with 0 bytes holds nothing an immutable read would miss. I relied on that, and on the reproduction on this machine. I did not check it against SQLite's documentation.
A non-empty WAL still blocks the read. If a writer leaves frames in the log for a long stretch, the check will report DB_UNAVAILABLE for that whole stretch, and the record does not show whether that happens in practice.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

My Weekly AI Audit Filed 25 Reports. No Automation Acted on a Grade.

John — Wed, 11 Nov 2026 00:00:04 +0000

Originally published on hexisteme notes.

On November 2 I stopped a scheduled job that had tried to grade how I use my AI tooling every Monday at 06:00 since May. It had filed 25 reports by then. I looked for anything that acted on those grades and found no automation that did.

The job ran cross_family_audit.py 7 --sample 3 from launchd. It scanned every record in Claude Code session files modified within 7 days, sampled 3 sessions, and asked several model families to grade 5 capabilities as low, medium or high. The capabilities included task routing, multi-agent synthesis and async federation, which in plain terms means how much work runs unattended. I had added the job on May 15 as the first prescription for a low async federation grade.

The report went to a dated Markdown file. The last step of the job was /usr/bin/open -e "$OUT", so a text editor window opened on my Mac every Monday at 06:00. The final run used a prompt of 3,381 characters, 1 Anthropic Opus call with max_tokens 2000, and 1 Ollama cloud call.

What the decision record found

The audit came up as one decision in the same cleanup that retired my LLM council. Before changing anything I wrote down what the job did.

Consumers: I searched my scripts, hooks, skills, rules and agents, plus the py, sh, json and yaml files across the fleet, for anything that read the report directory. The record says 0 pieces of code did. That was too strong, and the review of this post caught it. My dashboard registers the report directory, summarizes each report from its head, and can show one in the browser. What I can still say is narrower: I found no automation that used a grade.
Reads: the dashboard keeps read timestamps, and 1 of the 25 reports has one, the report dated July 13. A morning brief on September 21 listed the job as unread.
Coverage: the script is named for comparing model families, and 4 were configured. The reports from May 18 through June 8 each say 0 / 4 and hold no grades, only a note that no family was available and that SDKs and API keys needed setup. June 15 says 2 / 4. June 22 says 1 / 4, and that one answer was a connection error counted as a response. Everything from June 29 to November 2 says 2 / 4, with Anthropic and Ollama answering.
Cadence: my design note said monthly with a 30-day window. The schedule said weekly with a 7-day window.
Description: the dashboard card called it a code quality audit that finds rule violations and dead code, and it listed a dependency on a server the job had stopped using after October 28. The job graded usage capabilities.

The audit graded samples with the text removed

The sample was meant to show the reviewers real conversations. sample_conversations keeps the last 3 sessions its file traversal reached, which are not guaranteed to be the newest, and the first 5 records of each. render_samples turns those records into prompt text:

if isinstance(content, list):
    content = " | ".join(
        str(b.get("type") if isinstance(b, dict) else b)
        for b in content
    )
content = str(content)[:200]

When a message's content is a list of blocks, this keeps the block type names and drops the text, so a text block and a tool call reach the reviewer as text | tool_use. Content stored as a plain string survives, cut at 200 characters.

The November 2 report says as much in its opening lines. The Anthropic reviewer started with a premise: of the 3 conversation samples, only 1 showed content, and the other 2 were metadata (/clear, mode, attachment), so the assessment rested mostly on aggregate signals. The tool had described its own input problem, and no automation was there to act on it. That was the last report it wrote.

The grades moved for reasons the input could not see

Here is the Anthropic reviewer's summary table for the last three reports:

Capability	October 19	October 26	November 2
Multi-agent synthesis	low	low	medium
Async federation	medium	low	medium

The October 26 report justified its low async grade by saying sessions had async work but there was no schedule-based operating signal. That report was written by a launchd job, and several launchd jobs were running. The extractor does not inspect launchd state, so launchd reached the reviewers only through text that mentioned it, such as a conversation sample.

On November 2 the same capability went up. The Ollama family graded it high and cited a conversation sample containing a launchd setup for unattended runs on Sunday at 20:00. The Anthropic reviewer gave it medium and cited the launchd design it saw. Multi-agent synthesis split in that report too: medium from Anthropic, low from Ollama.

I read that as the grade following whichever conversation landed in the sample. That is an inference from the citations. I did not diff the weekly inputs, and the grades were never compared against an outside measure, so I can't say which grade was right in any given week.

Why I stopped it instead of fixing it

My first draft was short: stop the schedule and keep the code and reports. The alternative was to fix render_samples and add the two missing keys. I rejected that because I could find nothing that acted on the grades, so better input would only improve a report with no automation behind it.

This decision was one of the 4 in the blind A/B I wrote up in I Retired My LLM Council After It Lost a Blind A/B 4 to 0. Two verifiers agreed on these corrections to my draft:

It said Google and OpenAI were missing keys and packages. The packages were installed. Only the keys were missing.
open -e leaves no read record, the dashboard surface policy only surfaces failures, and the September 21 brief had named the job as a follow-up input. So my read count could not show whether a person used that window. Only I could answer that.
Stopping ends the weekly series. extract_signals(since) has no end-time argument, so the existing extractor cannot replay a bounded past week.
The morning brief script imports this audit module, so the module has to stay even with the schedule gone.
A monthly run without the popup had not been evaluated.

The second correction changed what I could claim. The read count covers the dashboard. "Nobody read them" was an assumption from a record that could not have seen the popup.

The corrected plan had 2 options: stop and archive, or reduce to monthly with a 30-day window and fix render_samples. It put the choice on one question to me: had I ever read that Monday window and acted on it? If not, stop. I approved the recommended stop.

What changed

launchctl bootout removed the job, and launchctl list | grep -c code-audit returned 0. The plist went to a backup directory.
All scripts and the 25 reports stay, about 100KB, with no expiry.
The dashboard card is marked retired, with a corrected description and the stale dependency removed.
I deleted the job's secret file. I did not revoke the key, because the morning brief uses the same one.
The morning brief ran on November 3 and November 4, and launchd shows its last exit status as 0.

The restart condition is written down: when I need a usage diagnosis, fix render_samples and re-register the job monthly. A manual run with a 30-day window stays available.

What this does not show

"No automation" is the result of searches made for this decision and this post, and a search can miss a reader, as the first one did. It says nothing about me reading the popup on a given Monday, and I cannot reconstruct that now. The one timestamp is what the dashboard recorded, which leaves out any read that happened through the popup.

I also can't call the audit worthless. Its grades were never compared against an outside measure, so I don't know how good or bad they were. My claim is narrower: across 25 reports, I found no automation acting on a grade, the dashboard has 1 read, and the input had a defect the last report described.

Stopping has its own cost. The series ended on November 2, so there is no before and after for this decision, and restarting the existing extractor would not backfill the gap on its own. If I restart the job, its first run would have repaired samples, so those grades would not be comparable with the old ones either.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Your First In-App Purchase Has to Ride With an App Version in App Store Connect

John — Tue, 10 Nov 2026 00:00:06 +0000

Originally published on hexisteme notes.

On October 29 I went to submit the 1.0 of one of my apps, and that version carried the app's first in-app purchase, a non-consumable product. My normal release path submits the version, so on this release it would have left the product behind. I did the setup through the API, got both items into a single submission through a short web flow with two traps in it, and then checked the result through the API.

My record ends at WAITING_FOR_REVIEW for both items, so I can't tell you how the review went.

What Apple says

Apple's help page Overview for configuring In-App Purchases states the rule directly:

If you're submitting your first In-App Purchase, you must submit it with a new version of your app.

The product screen in App Store Connect shows its own wording of the same rule: "Your first In-App Purchase must be submitted with a new app version".

The rule is about the first one, so everything below applies only to the first.

The API page Managing In-App Purchases settles the automation question in an Important note:

Submit your first In-App Purchase together with an app binary submission through appstoreconnect.apple.com. The review-submission API calls below don’t apply to that first submission. For subsequent In-App Purchases, submit using the API endpoints below without an associated app binary submission.

So the documented route for a first purchase is the website. The API flow the note points to for later purchases is POST /v1/reviewSubmissions, a reviewSubmissionItems item with an inAppPurchaseVersion relationship, then submitted: true. The same page marks POST /v1/inAppPurchaseSubmissions deprecated as of 4.4.1 and sends new integrations to that review submissions workflow.

I followed the note and never sent those calls for a first purchase, so I have no API error to show you. One warning about how I checked: when I read the page through a scraper on November 4, the markdown had dropped the Important note, and I nearly wrote that Apple's API docs didn't mention the first purchase. The page's raw doc data has it. If a claim hangs on an Apple doc page, read the note boxes on the page itself.

Why my automation didn't carry the product

My normal release path is fastlane submit_review and a 3-step reviewSubmissions script. Both submit the version. If I had used either one here, the purchase would not have gone in. A version approved without its product leaves one of two endings: a buy button that gets rejected, or a product waiting for the next version. I didn't run it that way, so those are the outcomes I was steering around and not something I observed.

One read also failed in a way that is easy to over-interpret. I asked the items endpoint to include the product:

GET /v1/reviewSubmissions/{id}/items?include=inAppPurchaseV2
-> 400 'inAppPurchaseV2' is not a valid relationship name

That 400 says the name is not accepted on this read. It says nothing about submitting a first purchase, and I don't use it as evidence for that. Apple's note is the evidence. It does mean this include can't show you the product inside a submission, which comes back when you verify the result.

Two traps in the web flow

Both traps involve the Add for Review button, which appears on the version screen and on the product screen.

The product button opens a menu

Add for Review on the product screen doesn't add the product directly. It opens a menu with two entries:

"iOS submission draft (1)"
"Create new submission"

Nothing happens unless you choose the existing draft. "Create new submission" would split the product from the version, which is the separation Apple's rule says not to make, so I did not take that entry.

The draft dialog covers the product button

Add for Review on the version screen first opens a "submission draft" dialog, and that dialog covers the Add for Review button on the product screen. My first click on the product button returned OK and had no effect, because the dialog was covering it. After I closed the dialog and clicked again, the menu appeared, and a MutationObserver caught its text.

I didn't check why a click on a covered button comes back OK. The practical result is that the OK told me nothing about whether the menu had opened.

A warning you can clear ahead of time

After the version's Add for Review, a warning said content rights info must be set in app info. contentRightsDeclaration is an app-level value, so a version metadata push doesn't fill it. Apps that have already shipped have it set, so you only hit this on a first app. It takes one API call, so I list it with the API work below, where it gets done before the web step starts.

The order that worked

The rule I kept: do everything the API can do first. That shrinks the web part to making the draft, putting the product in the draft, confirming two items, and submitting.

The API part uses key auth with a JWT of at most 1200 seconds. I wrote about that JWT expiry hard cap in Three App Store Connect API Hard Limits That Only Bite at a Fast Release Cadence. The calls:

Attach the build: PATCH /v1/appStoreVersions/{ver}/relationships/build with {data:{type:"builds",id}}, which returns 204.
Set the release type: PATCH /v1/appStoreVersions/{ver} with releaseType as MANUAL or AFTER_APPROVAL.
Set content rights: PATCH /v1/apps/{app} with contentRightsDeclaration as DOES_NOT_USE_THIRD_PARTY_CONTENT or USES_THIRD_PARTY_CONTENT.
Check the product: GET /v2/inAppPurchases/{id} should return READY_TO_SUBMIT. By this point the review screenshot, notes, price, and localizations were done.

Then the web part, in this order:

On the version screen, press Add for Review and keep the attached build. A "submission draft" appears.
Close that dialog.
On the product screen, press Add for Review, the button outside any dialog.
In the menu, pick the iOS submission draft, not Create new submission.
The draft dialog must show 2 items ready, the version and the product. Press Submit for Review, and the screen reports "2 items submitted".

The version goes first so the product's menu has an existing draft to choose. The close step is there because leaving the dialog open is what made my first click do nothing.

Confirming it through the API

A successful click is not evidence. I had just seen one return OK and change nothing, so I check the item count on screen and through the API, and I don't submit a draft without the product. After submitting, I ran these reads:

GET /v1/reviewSubmissions?filter[app]=<appId>  -> state WAITING_FOR_REVIEW
GET /v1/reviewSubmissions/{id}/items           -> 2 items
GET /v2/inAppPurchases/{id}                    -> WAITING_FOR_REVIEW

In the items response, the version item has an appStoreVersion relationship and the product item shows no relationship. So the product's item doesn't identify itself as a purchase there, and the include from earlier returns a 400. The count comes from that response and the product's state comes from its own endpoint, so I run all of these reads.

If you set releaseType to MANUAL, a person presses release after approval.

What I haven't established

The review outcome. My record stops at WAITING_FOR_REVIEW for both items, so nothing here shows that this submission passes review.
Other product types. I only ran the non-consumable case, on one app.
Why the screens behave this way. I recorded what they did on October 29 and didn't look into how they work.

This procedure is the one I know works up to the review queue. The next thing to check is how the review ends, and whether a first subscription needs the same path, which I haven't run.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Playwright's new_page() Pulled Chrome to the Front. A Background Tab Alone Didn't Fix It.

John — Mon, 09 Nov 2026 00:00:01 +0000

Originally published on hexisteme notes.

A blog publishing job ran every 15 minutes on the Mac I work on, and each run opened a browser tab through Playwright. Any of those runs could pull the browser over the app I was using. A stats job at 09:20 could do the same. I have no count of how many runs actually took the screen.

The browsers themselves made this hard to avoid. There were two, both headed and always on, both driven over CDP by scheduled jobs: Chrome for Testing on port 9333 and Google Chrome on port 9334. launchd kept both alive with KeepAlive, so closing one didn't help. It relaunched within 30 seconds and its window came to the front. On October 30, at 18:54 and 21:05, both browsers exited normally (exit 0) and relaunched. My notes record the exit code and not the reason.

That made two separate ways onto my screen: a job opening a tab, and a browser coming back after an exit. The fix has one layer for each, and the second layer is needed because the first one un-hides the browser.

Everything here is Mac-only. open, launchd and NSRunningApplication are macOS pieces, and every result below comes from one Mac. The tab layer is Playwright and CDP, but I only ran it there too.

What I measured

The first row comes from a probe on October 29.

What ran	What happened
`context.new_page()` on a headed browser	The browser came to the front and took the focus
CDP `Target.createTarget` with `background: true`	Focus stayed where it was, but a hidden browser got un-hidden
`open -j`, which launches hidden	No effect, because Chrome shows itself when it opens its first window
`open -g` on a relaunched Google Chrome with an existing profile	The browser came to the front anyway
`bring_to_front()` in one other job	Called on purpose, because that job needs the clipboard

The two open rows ruled out a launch flag as the whole answer. -g is the flag meant to keep a launched app in the background, and it didn't hold for that relaunch. I can't say whether the existing profile is the cause or only the condition I saw it in, because I didn't check why.

The second row shaped the design. background: true solves the focus problem, and a background tab by itself leaves a hidden browser visible again.

The last row was never a bug. That job pulls the browser forward deliberately, and I decided to leave it that way.

Layer one: open tabs behind the app

BackgroundTabs replaces new_page() with the CDP call and hands everything else to the real context.

class BackgroundTabs:
    def __init__(self, context):
        self._context = context
        self._cdp = None

    def new_page(self):
        if self._cdp is None:
            self._cdp = self._context.browser.new_browser_cdp_session()
        with self._context.expect_page() as opened:
            self._cdp.send('Target.createTarget', {'url': 'about:blank', 'background': True})
        return opened.value

    def __getattr__(self, name):
        return getattr(self._context, name)

Reading it from the top:

The CDP session is browser-level. It's created on the first tab and reused after that.
expect_page() wraps the send, so the caller gets a Playwright page back, the same kind of object new_page() returns.
The tab starts at about:blank, so the caller still has to navigate it.
__getattr__ forwards every other attribute to the wrapped context.

I wrapped the context instead of editing the browser module because a deployment review pins that module's bytes. Session pages and the re-login tab go through the wrapper too, since any path left on new_page() would still take the focus.

If your browser isn't hidden and you only mind the focus change on new tabs, this layer is all my table calls for. It does nothing for the relaunch path.

Layer two: a launch wrapper that keeps hiding

launchd started each browser through a shell wrapper. This is its launch line:

open -n -g -a "$APP" --args --user-data-dir=... --remote-debugging-port=... --remote-debugging-address=127.0.0.1

-g stays on that line, but the wrapper doesn't depend on it. What keeps the browser out of the way is a hide loop that starts after launch:

For the first 20 seconds, it hides unconditionally.
After that, every 2 seconds, it hides only when the browser is not the frontmost app.

The frontmost check is what leaves me a way in. If I switch to the browser with Cmd+Tab or the Dock, it's frontmost, so the loop leaves it visible. By the same rule, a browser un-hidden by a background tab is visible but not frontmost, so the next pass hides it again. That's how the two layers fit together.

The hide is NSRunningApplication.hide, called through the JXA ObjC bridge with osascript -l JavaScript. It's not an Apple Event, so there's no automation permission prompt.

The rest of the wrapper keeps launchd's view of the process correct:

The shell waits until the browser exits, so launchd sees the browser's exit and restarts it.
If launchd ends the shell, the browser ends too.
If a browser with the same profile is already running, the shell waits on that one rather than starting another.

What I checked

A one-hour watch from 18:57 to 19:57 covered 3 publish runs. Tabs opened in the background each time, and either browser came to the front 0 times.
On 2 temporary profiles, screenshots, typing and clicking inside the page all worked while the browser was hidden. When the wrapper got a TERM, the browser stopped with it.
On the real browser, I simulated Cmd+Q with a TERM. The relaunch came up with hidden=true and active=false, the frontmost app stayed where it was, and the login persisted.

Each path to the screen maps to a different check:

Path to the screen	Layer that handles it	Check
A job opens a tab	`BackgroundTabs`	One-hour watch
A browser relaunches	Launch wrapper	Simulated Cmd+Q on the real browser
A background tab un-hides a hidden browser	Hide loop	No direct measurement, only the rule

The temporary-profile check sits outside that table. It's there because a hidden browser is only useful if the jobs can still work in its pages.

What this doesn't show

The evidence is one hour, 3 runs and one Mac. The publishing job was paused at 21:37 that evening by my decision, so this hour is the only watch on record. Some specific gaps follow from that:

A 15-minute schedule should fit more than 3 runs into an hour. My record says 3 and doesn't say why.
The stats job ran at 09:20, outside the watch, so the watch says nothing about it.
The watch counted browsers coming to the front. It didn't measure how long a browser stays visible between a background tab un-hiding it and the next hide pass.
By the frontmost rule, a browser pulled forward by the bring_to_front() job would stay there, the same as if I had switched to it. I have no measurement of that case. It would also trip the falsifier below as written, so any such event has to be checked against that job first.
I don't know why the browsers exited on October 30, or why that relaunch ignored -g.

The falsifier is specific: either browser comes to the front with no human switching to it, or a hidden browser produces empty screenshots or timeouts. Before calling this settled I'd want a longer watch that includes a 09:20 stats run and a run of the bring_to_front() job, because nothing in that hour's record shows either one.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

I Retired My LLM Council After It Lost a Blind A/B 4 to 0

John — Sun, 08 Nov 2026 00:00:06 +0000

Originally published on hexisteme notes.

On November 3 I shut down an LLM council I had built myself: launchd jobs booted out, MCP server unregistered, hooks removed. The reason was a pre-registered blind A/B against one fresh-context subagent, which ended 0 wins, 4 losses and 0 ties for the council across four real decisions.

The score is the short part. Most of what I want on record is the procedure: how the two arms were blinded, why I replaced the judge after the experiment had started and what I disclosed about that, how I stopped at 4 of a planned 10 and logged it as a deviation, and the fact that the winning arm was the more expensive one. n=4 says little about councils in general, and I'll mark where it stops.

What was being tested

The server ran 8 historical-figure personas, all on claude-sonnet-5. The "debate" was one call writing claims and rebuttals together, and the problem statement was cut to 400 characters. I also had a second home-built council layer, but the A/B tested only the 8-persona server.

A re-evaluation on October 27 had already removed most of its reasons to exist. Multi-angle generation, task decomposition and arithmetic support were now absorbed by current frontier models and by Claude Code's built-in subagents and workflows. Two older results pointed the same way:

A blind review on May 25 scored a single Opus run 5.92 out of 7, against 3.41 for council quick and 2.91 for council deep.
A cross-vendor utility audit on August 22 sorted about 60 cases where a check changed a decision. It classified about 85% under fresh context, an adversarial brief and independent execution, and about 1 case as specific to a model family, which was access to a capability and not a verification finding. The audit had no matched control, so this was a classification and not a proven cause.

One result pointed the other way. A human blind review on June 21 found 3-4 of 6 questions in the council's draft critique valid, so the critique function itself was alive. That was the one function left to test, so the experiment asked one thing: does the council's critique of a draft produce more valid points than an independent critique pass?

Codex reviewed the plan twice independently and agreed with it. A third vendor was out of quota and did not review it.

The design, fixed on October 27

Sample: real load-bearing decisions only, no invented test problems. Target n=10, deadline December 31.
Input, identical for both arms: the question, Claude's draft (conclusion, evidence, assumptions) and evidence file paths. No self-assessment and no reasoning trace.
Arm A: the council at default depth, all personas on claude-sonnet-5.
Arm B: one fresh-context subagent on sonnet. Same model grade, to remove a capability confound.

Arm B's brief asked for "the most plausible ways this draft is wrong, up to 5, each one line of claim + one line of how to check. Do not write what you agree with."

The decision rule, excluding ties:

Council wins 8/10 or more: keep it and promote it to arm A of the critique pass.
Council wins 6-7/10: hold and extend to n=20.
Council wins 5/10 or fewer: retire the server.
Secondary rule: even if the council wins, do not keep it if its wrong rate is 2x arm B or worse.

How the blinding worked

A council output carries persona names and a debating style, so it can't be scored blind as it stands. Both raw outputs went through the same sonnet extractor, which kept at most 5 items, removed persona names and style, and reduced each item to one line of claim and one line of check.

The two lists were then randomly assigned the labels X and Y, with the seed stored in the ledger. The mapping was revealed only after scoring.

Each list came out at 5 items. I have no figure for what that cap removed from either arm's raw output.

Replacing the judge after the experiment started

The original judge was me, alone. I had ruled out LLM judges because an earlier experiment put LLM-judge agreement at a kappa of about 0.

The first decision, D1, broke that plan. Its items were infrastructure facts, launchd and the like, and I could not judge them without reading code. My scoring would have turned into gut feel or into following an LLM anyway. Claude could not take over either, because it had read D1's raw output before normalization and was not blind.

So I amended the protocol on October 28. Zero scores were recorded at that point, but the timing was not clean, and the amendment says so: it came after seeing a first Codex verification result for D1, X n=3 and Y n=3. The X/Y mapping was still undisclosed to me. The decision rule did not change.

The new judges were two verifiers that never see raw output or the mapping: Codex in read-only mode and a fresh sonnet subagent. They got the same brief and ran the check commands. Each item received one of three labels:

n: true and not covered by the draft
k: true and already covered by the draft
w: the core factual assertion is wrong

An item is confirmed only where both verifiers agree. Disagreements are undecided, and they leave both the win count and the wrong-rate denominator.

This reversed the no-LLM-judges rule I had started with, so the amendment carries its own falsifier: if pooled Cohen kappa is below 0.4 after 10 decisions, the experiment ends as unjudgeable.

Results

These are confirmed items only. A is the council and B is the critique pass.

Decision	Topic	A new	A wrong	A undecided	B new	B undecided	Verifier agreement
D1, October 28	retire file bus + wake job	1	0	4	3	1	0.5
D2, October 28	move two scheduled jobs off the council's env	2	1	2	5	0	0.8
D3, November 2	drop a weekly audit job	2	1	0	4	1	0.9
D4, November 3	adopt settings from a biweekly scan	1	2	1	4	1	0.9

B won every round. The wrong rate over confirmed items was 4/13 = 0.31 for the council and 0/17 = 0 for the critique pass.

Pooled kappa ended at 0.599, excluding abstentions. It was 0.107 on D1 alone, 0.346 after D2 and 0.534 after D3, so it started under the 0.4 line and climbed over it. The falsifier did not trigger, with one qualification: it was written for 10 decisions and I stopped at 4, so 0.599 is a reading at n=4 and the checkpoint the protocol named was never reached.

Here is what confirmed items looked like:

Arm B on D2, confirmed new: the draft's check, that an output file exists, can't catch a failed migration, because the job writes an error string into the file. Running it once would also overwrite that day's existing output, since the file name is date-fixed and written without an existence check.
Arm B on D4, confirmed new: the draft assumed unattended runs ignore Claude Code settings. One scheduled job actually ran Claude with skip-permissions.
Arm A on D2, confirmed wrong: that a path on the external SSD was a worse new single point of failure.
Arm A on D4, confirmed wrong, two items: that a 71% figure had no source in the question or the draft, and that the figure was unconfirmed.

D1 is the weakest round. 4 of the council's 5 items stayed undecided and verifier agreement was 0.5, so that loss rests on few confirmed items. It doesn't change the outcome band: had D1 gone to the council, its ceiling would have been 7/10, still short of keep.

Why the critique pass did better, I did not test. The design compares two arms and can't say which part of the council lost the points.

The winner cost more

Decision	A time (s)	B time (s)	B tokens
D1	95.5	388	131,676
D2	56.4	254	118,593
D3	60.1	234	130,127
D4	64.3	448	138,861

Arm B was slower in every round. The council's token count exists only for D1: 13,076, API metered, against 131,676 for arm B. A's tokens were not reported on the later quick runs, so I have no token comparison for D2 through D4. The arm I kept is the slower and heavier one, and the win was not cheap.

The experiment's own machinery sat on top of that. Normalization and verification added about 94-105k and 124-152k tokens per round, plus one Codex run.

Stopping at n=4

On November 3, after seeing the results, I stopped at n=4, before the target of 10 and before the deadline. That is a deviation from the pre-registration and it is recorded as one. The decision rule itself did not change.

The reasoning:

"Keep" (8/10 or more) was already unreachable at D3.
At 0-4, winning all 6 remaining decisions gives 6/10, which is "hold, extend to n=20". That was the ceiling.
If the arms were equal (p=0.5), 6 straight wins has probability 1/64, about 1.6%.

The rule had not produced "retire" by itself. The retirement at n=4 was my decision, made with the results in hand, and that arithmetic is the justification I wrote down.

The ledger's tally tool still says incomplete, because n is below 10. The written early-stop section is the record of why.

Other recorded deviations

On D3, Claude's grep during a brief exposure check surfaced parts of three items.
On D4, the arm B subagent's read of a local config file went outside the intended access.

Verifiers' access to forbidden paths was audited, but unevenly: Codex's logs were grepped, while the sonnet subagent self-reported. I wrote about why a reviewer's reach matters more than what its prompt leaves out in An Empty Prompt Is Not a Blind Review.

After the result

The server was retired on November 3 with final backups taken, and the repository was kept. Retirement also turned up something unrelated to the A/B: the backup job had been silently skipping since October 2, because it looked for a Docker container that wasn't running. The last dump was from October 1.

I kept the independent critique procedure and the pre-call controls and ledger.

What this does not show

n=4 is not a statistical claim about councils in general.
All four decisions were one person's infrastructure decisions.
Both arms used the same model grade, so this says nothing about councils that mix models.
Arm A ran at default depth, and "quick" on later runs, so it doesn't cover a deeper council setting.
I did not check whether the server's 400-character cut applied to the input in these runs. If it did, "same input" is weaker than it reads.

The retirement has its own falsifier. If a critique pass misses 2 or more defects on important decisions that later surface, and the council is reproduced catching them on the same input, I reconsider.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

My YouTube Shorts Pipeline's First Five Verdicts All Said Unjudgeable

John — Fri, 06 Nov 2026 00:00:06 +0000

Originally published on hexisteme notes.

The first five outcome verdicts from my YouTube Shorts pipeline came back, and all five say unjudgeable. Four of the videos have no prediction on record. The fifth has a prediction and a measurement that differ by about 10x on views, and it still can't be judged, because the baseline file it needs was never built.

I had made the prediction mandatory on one channel: from October 25, upload on channel A is refused without one. The verdict needs a baseline as well, and nothing made that mandatory on any channel. The input I required and the input the verdict needed were different, and this note is about that difference.

The five verdicts

This is the state of the records as of 2026-11-04. The window runs 7 days, from publish day D0 to D6 in Pacific dates.

Window start	Channel	Views D0-D6	Engaged/views	Prediction	Label
10-13	A	505	22.2%	none	`unjudgeable:no_prediction`
10-22	B	81	46.9%	none	`unjudgeable:no_prediction`
10-23	A	189	9.5%	none	`unjudgeable:no_prediction`
10-24	C	1,238	35.5%	views 120, engaged 25%	`unjudgeable:no_baseline`
10-24	B	347	36.3%	none	`unjudgeable:no_prediction`

Five of five are unjudgeable. Two of the no_prediction rows are channel A videos published before the requirement took effect. The other two are channel B videos, which the requirement doesn't cover. That leaves one row, the 10-24 video on channel C, where a prediction existed and the sweep had something to compare. The requirement doesn't cover channel C either, so nothing demanded that prediction.

Why the loop existed

The pipeline was good at "the file is valid" and had no loop back from results: a ledger of 49 rows and 2 predictions. A design pass found the premises were worse than I had proposed. Two findings matter here:

The 2 prediction rows belonged to a commerce-review schema, and all 7 metrics were unknown. Comparable predictions: 0. No step required one.
The baseline of "engaged 45%, 760 views per video" was a lifetime channel total. That is a different unit from a 7-day window, so it had to be re-measured on the same window.

The first finding turned into a gate. The second turned into a planned manual run.

What I built on 2026-10-24

predict-outcome records engaged view rate and D7 views, bound to the final render's sha256, only before publish. On channel A, from October 25, upload is refused for a video with no prediction. The other channels are untouched.
judge-outcomes is a daily launchd sweep with no model call, because the job is arithmetic. It collects at D+10 because of analytics lag. It labels a video above, within or below baseline, or unjudgeable with a reason: no_data, no_prediction, no_baseline, or public_time_unverified for a schedule it can't verify.
Engaged/views is a proxy, labeled as one, because YouTube Studio's stayed-to-watch versus swiped-away number isn't in the API.

The design also planned the piece this note turns on, a one-time manual same-window baseline re-measurement: take at least 10 prior channel A videos, measure them on the same D0-D6 window, and write that channel's baseline file. Baselines load per channel.

The one prediction pair

The 10-24 video on channel C is the only row with a prediction. The agent predicted 120 views and 25% engaged. The window measured 1,238 views and 35.5% engaged. That is an error of +1,118 views, about 10x, and +10.5 points on engaged rate.

The prediction states its basis: a channel lifetime average of ~110 views per video, 23% engaged, and September videos' lifetime views of 12-278, with the video placed near the channel average. So the agent predicted from the lifetime number the design had just said was the wrong unit. The 760 and 45% in the design were channel A's lifetime figures, and these are channel C's. They have the same problem: they describe a channel's whole lifetime, and the prediction is about a 7-day window.

No gate looked at that row, because channel C is outside the requirement. The gate on channel A wouldn't have stopped it either: it checks that a prediction exists and doesn't read the basis. I haven't checked why the agent picked that number, or whether it would have used a same-window baseline if one had existed.

What the label refused to say

The sweep didn't call the 1,238 views "above baseline", and it didn't call the video a win. It wrote unjudgeable:no_baseline.

The forecast error is a separate calculation and it's on the record: +1,118 views. The label answers a different question, how the video did against its channel's baseline, and there's no failed-forecast label even when a baseline exists. So the miss tells me the forecast was off and says nothing yet about the video. If an ordinary video on that channel collects something close to 1,238 in the same window, this one was ordinary. If an ordinary video collects something close to 120, this one beat the channel. Only the same-window number separates them.

This is a different failure from the one in My dev.to Gate Couldn't Measure, and the Suite Called It a Regression. In that note a live check never completed its measurement. In this one the views are measured, 1,238 of them, and the missing piece is the reference to compare them with.

The input I required and the input the verdict needed

On channel A, the upload gate asks one thing: does a prediction exist for this video? An above, within or below label needs the same-window baseline too. The sweep does notice a missing baseline, at D+10, after the video is already out. In the design it was a one-time manual run. As of 2026-11-04 it was planned and never run, and the baseline file does not exist.

Both requirements came out of the same design pass. The prediction got a refusal on channel A's upload path, so a video under the requirement couldn't get past it. The baseline got a line in a plan, so the pipeline could upload, predict and sweep without it, and no step failed.

The result is a growing set of predictions the sweep can't place. Five more predictions are on file for later videos, six rows in total counting the first, 3 on channel A and 3 on channel C, with one video re-recorded after a re-render with the same values. Several of them say the same-window baseline file was not created. Each one with measured data will also come out no_baseline until its channel has that file, and a video with no view data ends as no_data instead. The planned run only covers channel A, so channel C's predictions would stay without a baseline even after it happens.

The verdicts are honest, and the sweep runs daily and reports this. Neither of those made the re-measurement happen. Fixing the Proxy Deleted the Objective ends up asking for a severed path to report as a distinct state from an early one. Here I already have the distinct state, and no_baseline names the missing input exactly. Nothing in the pipeline turns that label into the run.

What I would check first

None of this is built. These are the checks I would run against this loop:

List every reason the judge can return unjudgeable, and write next to each one the step that guarantees the input. Here no_prediction has the upload gate on one channel and no_baseline has a line in a design document. I'd want the same line for no_data and public_time_unverified.
Treat the one-time manual run as a dependency of the loop, and have something fail when its output file is missing. A refusal at upload is the direct version, and it would also stop publishing until the run happened, so a human has to make that call.
Check what the gate can't see. It confirms a prediction exists, and the first pair shows a prediction can rest on a basis the design had already rejected.

What this doesn't show

This is five verdicts and one prediction pair. The design says an "n=3 verdict is a resource decision, not a statistical claim", and one pair is smaller than that. The later rows haven't reached a verdict.
The +10.5 points is a gap in a proxy. The records I used for this note don't include a Studio value for that video.
At least 10 prior videos is the design's plan for the baseline. I don't know yet whether that is enough to separate a good video from a bad forecast.

The next step is the re-measurement on channel A, and a decision on whether channel C gets one too. The 10-24 pair won't change by itself either way, because the sweep skips a video it has already judged.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Python Reader I Measured Never Opened 34,557 Files. 310 Tests Still Needed Them.

John — Thu, 05 Nov 2026 00:00:07 +0000

Originally published on hexisteme notes.

On 2026-10-31 I quarantined 34,557 HTML files in a content-collection system because the reader I had measured never opened them. The collection round that ran next finished complete and a re-trace passed. Then I compared the full test suite to its sealed baseline and found 310 failures beyond it.

All of the files went back the same day, and that was cheap only because I had moved them instead of deleting them. The retry that came after is still in progress. As of 2026-11-04 its storage gate is red, three of its four nightly runs ended restored, and the step that actually deletes anything has never run.

132.8GiB and no alarm

The system's report/ directory had grown to 132.8GiB with no alarm. Jobs running every 15 minutes left a new folder each round, and nothing cleaned up after them.

The first pass was a reference-based pruner. It keeps anything cited in the database, the state or the source, and it sits under a budget gate of 140GiB. It took the directory from 132.8 to 115.1GiB by reclaiming 556 unreferenced units (10.7GiB) and cutting a 9.86GB SQLite file to 2.30GB. Collection runs were still 57.4GiB, and almost all of that was referenced, so a reference check had nothing left to take.

Measuring one reader

The second attempt ran the same day, 16:50 to 17:40 KST, and started from a plain question: isn't this data unnecessary once it's on the blog? The numbers seemed to say yes.

Of 60,492 tasks, 626 (1%) had published.
Of 46,643 collected HTML files (52.4GiB), the collection-state validator opened 1,920 (2.2GiB).
The rest were the same ~2,100 videos re-fetched every 15 minutes, 33,434 history entries of them.

So the candidate set became HTML older than 7 days that the validator doesn't open: 34,557 files, 37.8GiB. At 07:52Z a release tool moved them to quarantine. Moved, not deleted. Afterward the collection round finished complete and a re-trace passed.

Both results were accurate, and neither covered the code that was about to fail. I've written before about checks that report accurately on what they examined and say nothing about what they could not see. This was another case of it.

The full suite disagreed

Comparing the full test suite to the sealed baseline (v20, 31 failures) showed 310 failures beyond the baseline. The causes were ValueError: YouTube page body unavailable, 312 times, and FileNotFoundError on runs/*.html.

Publish replay, re-judging blocked batches and joining past provenance all reopen old capture bodies. None of those appeared in what I had measured, because I had measured one reader.

That was the error. A read set tells you what one program opened. I took its complement and treated it as the files nobody needs, when the complement only describes the validator. One reader's read set was not every reader's, and a delete list has to be computed against all of them.

Recovery cost a rename

I moved all 34,580 quarantined files back, 0 missing, and re-ran the 908 failed tests: 0 failures beyond the baseline. Then I removed the tool, its launchd job, the gate branch and the line in the agent instructions, and locked the lesson with a regression test: a superseded capture cited only by history keeps its whole unit.

Because the move was a quarantine, recovery cost a rename and no re-crawl. The wrong candidate list would have been the same mistake either way. The quarantine changed what it cost to find out.

My record has two counts that don't line up with the earlier ones, and it explains neither. The rollback moved 34,580 files against the 34,557 I had quarantined, and the re-run covered 908 failed tests against 310 failures beyond the baseline. The results I relied on were the 0 missing and the 0 beyond baseline.

Third attempt: pins from running the readers

The third attempt ran 19:30 to 20:40 KST and began with an approved change of policy:

Delete raw HTML for finished items and keep only receipts.
Keep raw for in-progress items.
Give up regenerating old published posts from raw.

The last item is a real loss, accepted on purpose.

I also found the operational reader behind the second failure. Two blocked publishes traced to publication queue replay, which reopens raw files for finished posts: 26 raw files, up to 28 days old. "Finished" did not mean "never read again", so the policy alone could not define the delete list.

The pin list, the files that have to stay, was built by actually running the readers and recording what they open, instead of reading the code. The recorder had two traps of its own.

Trap one: the read that didn't match

Path reads were captured with an audit hook. One module reads like this:

os.open(name, dir_fd=parent_fd)

With the audit hook alone, the collection-state validator appeared to read 0 raw files. That is the same validator I had measured opening 1,920 files that afternoon. I checked the reason on Python 3.13.0 with a small script: the audit open event carries the relative name 'capture.html', so matching on absolute paths misses it.

The fix was an os.open wrapper plus fcntl F_GETPATH to recover the full path. In the final list the validator's pins include 1,898 raw files.

Trap two: reads collected at exit

Collecting test read records in atexit lost the main pytest process's reads entirely. The regression test for this uses a script ending in os._exit(1). My record stops at the symptom and the test. I haven't confirmed the exact path by which those records went missing, so I won't guess at it here.

Both traps undercounted, and in this design an undercount is the dangerous direction: every read the recorder misses is one fewer pin and one more candidate for deletion.

The list that came out

Reader	Pins	Of which raw
Collection-state validator	21,867	1,898
Publication queue replay	4,160	26
Backlog in-progress	3,355	not broken out
Publish-waiting	58	not broken out
Full test suite	498	15

I applied it as one batch: 32,955 files, 36.6GiB, moved to a quarantine directory in 123 seconds. The self-check passed, and the publish planner passed for both pending publishes after quarantine.

What runs now

A tripwire job runs every 15 minutes and watches logs for reads of missing files.
A nightly release job recomputes pins, restores any held file that became pinned, deletes batches past 7 days, quarantines new candidates and self-checks.
The storage gate fails on a stopped, refused or restored job, and on any tripwire restore within 7 days.
Bytes in quarantine are excluded from the budget and shown separately.

What I'd carry into another cleanup

This comes from one system, so treat it as what I would try first:

Move before you delete, on a fixed timer, so a wrong list costs a rename.
Compute candidates against every reader you can run, and count the test suite as a reader. Mine held 15 raw pins of its own, and it was the check that caught the failure.
Before trusting a recorder's zero, run it against a reader whose count you already have.
Compare the whole suite to a baseline after the move. The checks closest to the change passed here.

Where it stands on 2026-11-04

The budget gate prints report/ at 49.1GiB of 140GiB, with 41.0GiB in quarantine awaiting deletion. The first number looks like success, but quarantined bytes are excluded from it, and those 41.0GiB have not been reclaimed.

The tripwire showed tripped [] in the checks I read on 2026-11-04.

The nightly job is where it isn't working. Of four runs since the first batch:

2026-10-31, 20:20Z: ended restored
2026-11-01: ended restored
2026-11-02: held
2026-11-03: ended restored

All three restored runs carry the same error: pin source collection failed: CaptureReleaseError: collection state stayed busy. That turned the storage gate red today, PASS -> REGRESSION. The gate is doing what I built it to do, since it fails on a restored job. The failure underneath is not resolved at the time of writing.

Nothing has reached the 7-day delete yet. The expired lists are empty in all receipts, so the deletion step has not run for real. Everything that has gone right so far was a move, and a move can be undone. The step that can't be undone is still ahead of me.

The limits I'm carrying into that step:

I don't know why collection state stayed busy on three of four runs.
The pin list covers the readers I ran. A reader I haven't listed, or a path those runs didn't take, isn't in it, which is the second attempt's mistake made smaller.
After a real delete the tripwire can still report a read of a missing file, but there will be no quarantined copy left to restore.

The next thing to check is the first nightly run with a non-empty expired list, and what the tripwire reports after it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

My dev.to Gate Couldn't Measure, and the Suite Called It a Regression

John — Wed, 28 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

On October 23 the daily drift report for my gate suite contained this line (translated from Korean):

check_offer_subject_alignment.py: PASS → REGRESSION  ← a fix you'd made broke again.

That note is the loudest thing the drift watcher can say. It is reserved for the case where something I had already fixed has stopped working. The same gate had printed it on October 18, gone back to PASS on October 21, and printed it again on the 23rd. Before that day's fix, the drift log already held ten verdict changes for this one gate, going back to September 7.

When I opened it, the live data the gate checks was nowhere near the failure it guards against. What had failed was the gate's ability to look. The suite had no verdict for that, so it used the loudest one it had.

The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*

What the gate checks

The gate is one of 22 scripts in a suite I run over my publishing pipeline. This one checks, against live data, that a link I put in some dev.to posts sits on the posts it belongs on (the ones tagged agents) rather than drifting onto unrelated ones. It reads articles/me/published, then makes one comments?a_id= call per commented article: about 100 live API calls per run. A launchd job runs the suite once a day; on October 23 the run landed at 00:19 UTC.

The gate ends with one of three exit codes:

0 ALIGNED: the link sits where it should
2 REVERSED: the link has drifted onto the wrong posts, which is a real finding
1 UNMEASURABLE: it could not read the live data

Inside the gate those three meanings are distinct. The runner only saw two of them.

The runner knew one kind of non-zero

The suite runner turns each gate's exit code into a verdict. Before the fix, the decision was this (comments mine):

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED          # the runner's own timeout
    if not declared_reason:
        return REGRESSION       # any other undeclared non-zero lands here
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

declared_reason comes from a file, gate_human_blocks.json, where I can declare that a gate is red because it is waiting on a person. A declared gate that fails reads BLOCKED instead of REGRESSION, and a declared gate that passes reads STALE_BLOCK, so exemptions can't outlive their reason. This gate had no declaration. Exit 1 and exit 2 went down the same branch: "I couldn't read dev.to at 00:19" and "the link has drifted onto the wrong posts" both came out as REGRESSION, and the watcher put its broken-fix note on both.

How I worked out that the 00:19 run never measured

The drift log stores verdicts only. It keeps neither the exit code nor the stderr of each scheduled run, so I could not look up what happened at 00:19. I had to reason from a side effect.

The gate caches its last successful live measurement and reuses it for 24 hours. It writes the cache only right after a successful fetch. If loading the API key or the fetch raises an error it expects, such as a network error or a timeout, it prints UNMEASURABLE to stderr and exits 1 before the write. Any other exception in that span also exits 1, without that line, and also before the write.

At 13:32 UTC I ran the gate by hand, without the flag that forces a refetch. It reported its data as 0.0 hours old: it had found no cache young enough to reuse and had fetched everything live. As far as my session logs and the scheduler list show, nothing else had run the gate in between. The daily launchd job is the only scheduler that calls it, and no session of mine had run it since the previous day's scheduled run. That leaves two ways the 00:19 run could have produced a verdict from data:

It fetched live successfully, whatever it found. Then it would have written a cache about 13 hours old by 13:32, and the afternoon run would have reused it instead of fetching. It didn't.
It reused an older cache. Any cache still young enough at 00:19 had to come from the previous day's scheduled run, and the verdict is computed from the cached data alone. That data had produced PASS the day before, so reusing it would have produced PASS again, not REGRESSION.

Neither fits. What fits is a run that failed somewhere before the cache write, before it had any data. The fetch from dev.to is the only network step on that path, so the likeliest reading is that the gate could not read dev.to at 00:19. The logs can't tell me which error it was, or whether the gate caught it.

Here is what the afternoon run measured:

76.5% (26 of 34) of agents posts carry the link, against 32.0% (32 of 100) of the other posts
a second, comment-based measure pointed the same way: 58.8% against 34.0%

That is the alignment the gate exists to protect, by a wide margin. At 00:19 the gate never got to look, and the suite reported that as the gate finding a problem.

Why exit 1 couldn't become "unmeasurable"

The obvious patch is one line in the runner: read exit 1 as unmeasurable. That would have broken other gates. Exit codes are a protocol between the gates and the runner, and in this suite 1 already meant something else. check_reader_turn.py, for one, returns EXIT_MISSING_TURN = 1 when it finds a real defect. Remapping 1 suite-wide would have quietly downgraded that gate's real failures. Python also exits 1 on an uncaught exception, so 1 is what a crashing gate produces by default.

The suite already had a convention. It just wasn't written down where the runner could enforce it. Seven other gates declared EXIT_UNMEASURABLE = 3. This gate declared the same constant with a different value, 1, and it was the only one that did.

The fix

Three small changes:

The runner documents EXIT_UNMEASURABLE = 3 as the suite-wide convention and returns a new verdict, UNMEASURABLE, for exit 3. A human-block declaration does not turn it into BLOCKED. A measurement gap is not a failure waiting on a person, and letting a declaration absorb it would hide a dark gate behind a reason written for something else. Exit 2 still returns REGRESSION, because a gate that found a bad state should still alarm.
The drift watcher reports a transition into UNMEASURABLE with its own note ("couldn't measure at all; neither a pass nor a regression; check live access and rate limits") and never with the broken-fix note. In both the suite report and the watcher, UNMEASURABLE sorts just below REGRESSION and CRASHED, and the suite report still prints the tail of the gate's output for it, as it does for those two.
The gate changes its EXIT_UNMEASURABLE from 1 to 3, joining the other seven.

The runner now reads:

EXIT_UNMEASURABLE = 3  # suite-wide: "I could not measure", not "I found a problem"

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED
    if exit_code == EXIT_UNMEASURABLE:
        return UNMEASURABLE     # a declared human block does not convert this
    if not declared_reason:
        return REGRESSION
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

Three tests keep it from drifting back:

the verdict table gets rows for exit 3 with and without a declared reason, and both must be UNMEASURABLE
a guard test walks every gate the runner discovers and fails if any of them declares EXIT_UNMEASURABLE with a value other than 3, which is the exact drift that caused this
a watcher test asserts that a PASS → UNMEASURABLE transition is worded as a measurement problem and carries no broken-fix wording

The guard test is short:

def test_every_gate_declaring_unmeasurable_uses_the_shared_exit_code():
    offenders = []
    for gate in discover_gate_scripts(REPO_TOOLS):
        text = gate.read_text(encoding="utf-8")
        m = re.search(r"^EXIT_UNMEASURABLE\s*=\s*(\d+)", text, re.MULTILINE)
        if m and m.group(1) != "3":
            offenders.append(f"{gate.name}={m.group(1)}")
    assert not offenders

The tools test suite passed, 113 tests. I re-ran the drift watcher. It recorded REGRESSION → PASS for the gate and the false alarm cleared.

A check has three outcomes, not two

A check can look and find everything fine, look and find a problem, or fail to look at all. A runner that only has words for the first two folds the third into one of them.

Fold it into the first and a check that cannot run reports something that looks like a result. In A Measurement That Never Ran Became a Verdict, every read failed and the script still printed a normal-looking verdict built entirely out of failed reads; part of that fix was a guard that refuses to turn a failed read into a zero. In The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., a nightly job's step failed on its first scheduled run while the job's own heartbeat kept reporting healthy.

Fold it into the second and you get what I had: a gate that cries regression on every transient outage. An alarm that is wrong often enough gets read as noise, and a real reversal printed on the same line, with the same note, gets read the same way. The exit code is also where Exit 0, Empty stdout: the Quota Died on stderr goes wrong, from the other side. There, a CLI worker hit its usage cap and exited 0 anyway, so a dispatcher that only checks the exit code would log a completed task. In that case the process sent the wrong code. In my suite the gate sent the code it meant, and the runner read it with the wrong dictionary.

Test runners already draw this line. pytest has a separate exit code for "no tests were collected", distinct from the one for failing tests: a run that tested nothing is not reported as a run where tests failed. A gate suite needs the same vocabulary. The measurement gap gets a verdict loud enough to see, because the gate went dark and someone should check live access, and quiet enough not to pass for a regression.

Applying this to your own gates

Pick one exit code for "could not measure" and document it in the runner, not only in each gate. Avoid 1: Python exits 1 on an uncaught exception, so a crashing gate would read as unmeasurable.
Give the runner a verdict for it that is neither PASS nor your regression verdict, and sort it near the top of the report so a dark gate is still seen.
Don't let exemption mechanisms such as human-block declarations or skip lists absorb it.
Keep "found a bad state" on its own code, and keep it loud.
Test the convention across every gate the runner discovers, not just the one that broke.
Record the exit code and the stderr tail next to each scheduled verdict. My drift log doesn't, which is why the diagnosis above rests on a cache side effect instead of a log line.

What this does not prove

The 00:19 diagnosis is inferred from the cache and from what runs the gate. The drift log keeps verdicts only, so the specific cause (DNS, a rate limit, a timeout) is not pinned.
If the 00:19 failure was an exception the gate doesn't catch, it exited 1 without printing UNMEASURABLE, and the same failure would still read REGRESSION after this fix. Only the errors the gate catches now become UNMEASURABLE.
The cache argument covers the October 23 run alone. I have no exit codes for the earlier entries in the drift log, and some of those REGRESSION verdicts may have been real reversals.
The suite is now quieter on measurement gaps, on purpose. A gate that is chronically unmeasurable shows UNMEASURABLE instead of REGRESSION: still visible, but it no longer screams. A gate that should hard-fail when it cannot measure must not use exit 3.
The guard test reads only the declared EXIT_UNMEASURABLE constant. A gate that used 3 for a real failure without declaring that constant would now be read as unmeasurable, and the test would not notice. A grep of the current gates found none that does, but nothing stops a future one.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Before I Compared the Models, I Had to Decide What Not to Count

John — Tue, 27 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run several models across a fleet of agent CLIs, and I wanted a plain question answered: on my own work, which ones re-edit the same file more, which ones burn more validation runs, which ones error on tool calls more often. I already log every turn, so the measurement looked like a GROUP BY.

It wasn't. The query was an afternoon. The honesty was the week. Almost everything I did after the first GROUP BY was a decision about what not to count, because every shortcut I could take would have produced a confident number that was wrong. This is that list.

The average of two roles is not a number about either one

Each turn in my logs belongs to a role: a main thread, where a model works interactively on a long task, or a sidechain, a short one-shot the harness spins off. These are not two samples of the same thing. A main thread for one model had a median re-edit rate near 0.44; its sidechains sat at 0. Pool them and the model's "re-edit rate" becomes an artifact of how many sidechains happened to run that week — a number that moves when the workload mix moves, not when the model changes.

So the unit of comparison is model × role, never model alone. Every table is stratified by role, and I do not compare a main cell to a sidechain cell even for the same model: the session length and structure differ too much to read the gap as anything but confounding.

Different regimes are different treatments, so I don't pool them

The same model name covers more than one thing over time. One of mine ran under a pre-launch configuration, then a relaunch, with a blocked window between. Those are not one population with more data; they are distinct treatments. Averaging across them hides exactly the shift you built the lab to see. I keep them as separate rows — model/pre and model/relaunch never get summed — with epoch boundaries pinned to calendar dates so the split is reproducible and not drawn to flatter a result.

A floor for the word "comparable"

Two thresholds, declared before I looked at the outcomes:

n ≥ 5 to appear as a comparable cell. Below that I print NOT_COMPARABLE, not the value. One cell had exactly one observation. Its median is a real number and a useless one; setting it beside a cell with 2,781 observations invites a comparison the sample cannot support.
n ≥ 20 on both sides before I'll compute a confidence interval for a difference.

The point of writing the floors down first is that otherwise you discover them afterward, at whichever value makes the cell you like look solid.

The rows I refused to count

871 of my threads — 57 main, 814 sidechain — could not be attributed to any model. They were synthetic harness activity with zero tool and token behavior. The tempting move is to treat "unknown" as "zero" and fold them in. That single substitution would have shifted every rate and every denominator, because missing is not the same as zero and an unattributed turn is not a well-behaved one.

They appear in exactly one place: a reference line that says excluded, 871, here is where they went. They enter no average. A measurement that silently drops what it can't classify is the one that lies first, so I made the drop loud and countable instead.

When a confidence interval of [0, 0] is not confidence

Most of these per-turn metrics are zero most of the time. A model does not re-edit on most turns; it does not error on most turns. When I bootstrapped a median difference between two such cells, the interval frequently came back [0, 0].

That looks like a precise result. It is the opposite. When the data is almost all ties at zero, the median of every resample is zero, so the interval collapses to a point — not because the estimate is sharp but because the statistic has nothing to vary over. I label those collapses as a coverage limitation of a median bootstrap under heavy ties, not as a tight finding. A degenerate interval and a precise one print the same two characters; only the sample behind them tells you which you have.

The tokens that look additive but aren't

I wanted a total-output comparison, so I summed output tokens per turn. That one is genuinely additive. The trap was next to it: the stored input-token and cache-read counts are the context re-sent on every turn, counted again each time — a running sum with the same tokens in it over and over, not a count of distinct tokens. Summing those to compare "how much each model read" would have rewarded whichever model took more turns. They are logged; they are not a total. Only the output sum answers the question I actually asked.

The metric is not the thing it is named after

Two labels in my own schema promised more than they measure, and I wrote the gap down next to each:

same_file_reedit_rate counts a file touched again. It does not separate iterating toward a result from reworking a mistake. The name suggests a quality signal; the measurement is a frequency. If I ever make it a primary metric, that limitation gets pre-registered with it.
completion_proxy reads the shape of a thread's last line. It is a heuristic for "finished," not a judgment that the work was done, and I had to censor threads whose snapshot was cut off mid-turn so a truncation wouldn't get scored as an abandonment.

The measurer is inside the measurement

The sessions in which I built and audited this pipeline are in the corpus it measures. Those turns land in one project's stratum and distort its numbers — the tool is watching itself work. I can't retroactively remove the observer from the observed, so the limitation is named in the report and the next version adds a stratum that excludes self-measurement. Naming it is not fixing it; it's refusing to let the contamination pass as a finding.

Association, not cause

Models were never randomly assigned to tasks. Harder work, different projects, different weeks, changing dispatch policy — all of it is confounded with which model ran. So the whole thing is graded association-only and exploratory. There are no p-values; the comparison answers what co-occurred, not what caused what, and any sentence that drifts toward "model X is better" is one the design cannot support.

What this does not prove

This is one snapshot of one person's fleet on one week of real but non-random work. It is not a benchmark, and it does not rank models on quality.
The role split removes one confounder (session structure) and leaves the rest — task difficulty, project, timing, and policy changes are all still tangled with model assignment.
The sequence metrics are computed inside a model's own epoch, so work that crosses a model switch — one model's edit validated by another — is not charged to either.
"Thread" and "session" are not the same count; one session spans several files, and the attribution unit here is the per-turn epoch, not the session.
Every threshold (n ≥ 5, n ≥ 20, the purity cutoff for proxy threads) is a choice. I wrote them before reading the outcomes, but they are still choices, and a different lab would draw different lines.

The deliverable here was never the ranking table. It was the list above — the specific numbers I declined to print, and the reason each one would have been a lie. A comparison is mostly the rows you refuse to count.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Same-Size Old Image Passed My Upload Check. Codex Found the Race.

John — Mon, 26 Oct 2026 00:00:08 +0000

Originally published on hexisteme notes.

My blog publisher draws a chart image, pastes the post into a hosted blog editor, and clicks publish on a schedule. The editor does not always keep the image I paste as is. In my probes it re-uploaded some pasted images to Kakao's CDN (blog.kakaocdn.net) and rewrote the src, left others at their original URL, and dropped an image that returned 404 from the post without an error. The re-upload is asynchronous. So before the publish click, the publisher checks the editor: the right number of images, in order, rehosted on the CDN, each with the width and height of the PNG I generated.

That last condition was the whole identity check. Codex, which I use as a second reviewer, showed that an older chart with the same dimensions satisfied it, inside a window between two steps of the publisher.

What the check compared

The publish sequence runs in this order: write the body to the clipboard, press the paste key, confirm the pasted text (PASTE_VERIFIED), wait for the rehosted images to pass, log BODY_IMAGES_VERIFIED, write an ATTEMPTED delivery marker, click publish. If the image check fails, the run stops with an error before the marker, and the next scheduled slot retries the same body.

The image check exists because the text check alone passed even when images were lost; that was a first-round Codex finding. But the image check compared metadata: the src prefix, whether rehosting had finished, the dimensions. None of those is the picture.

The second Codex round found two ways this went wrong. A missing image could report complete=true with naturalWidth=0, and the check accepted it; I fixed that by also requiring the decoded natural size. The second finding was the one this note is about. Correcting a single historical value in the chart data changed the PNG's hash but left the body text and the image dimensions identical. If an earlier run's body, carrying the earlier chart already on the CDN, landed in the paste between the clipboard write and the paste keystroke, the text check and the size check would both pass. Codex also noted that my publish-gate tests expected ok from an arbitrary CDN address and matching sizes alone, so they could not catch either defect.

The first fix did not close it

My first repair for that finding required the editor to contain zero images before the paste. That removes leftovers sitting in the editor before the run starts. It does nothing about the window after the clipboard write.

The third Codex round said so. It ran read-only and reproduced the bypass with an in-memory mock of the browser. With the editor confirmed empty, replacing the paste with the earlier body between clipboard.write and the paste key still went BODY_IMAGES_VERIFIED → ATTEMPTED → PUBLISH_CLICK. Its summary: the size comparison does not bind the image content. Five of the seven second-round findings were now blocked; the two that were not were this same defect, once in the publisher and once in the tests. The tests I had added checked a broken image and a non-empty editor before the paste, but nothing replayed the swap after the clipboard write, so the false pass had no test that would go red. It was the second round in a row to report this defect.

The recorded reproductions used mocks and did not observe a real publication. The point is that the check could not have told me if one had.

Compare what the CDN serves

The repair is compare_rehosted_images. Once the editor state looks right, the publisher downloads each rehosted image from the CDN, in order, and compares it with the source PNG. The comparison is SHA-256 first. If the bytes differ, it decodes both files and requires the same size and identical RGBA pixels. A failed download returns fetch, a mismatch returns content, and both stop the run before the ATTEMPTED marker.

I checked the assumption that makes this workable against the real service. I downloaded the CDN copy of the chart, and its SHA-256 matched the source PNG byte for byte (prefix 113b23bd). A same-size PNG with different content is blocked as content. When I reverted the fix, three of the new tests failed. The full suite finished at 1,720 passed.

The regression test that models the post-paste image state, test_rehosted_image_content_is_bound_after_paste, has two cases. A fake browser shows a rehosted image only after the clipboard write, and that image reports the expected dimensions in both cases. If the downloaded file is the fresh chart, the result contains BODY_IMAGES_VERIFIED:1. If it is a PNG with a different shade, the result is ERR:body_image_unverified(content), no publish button is clicked, and no delivery marker is written.

What a hash check still let through

The fourth Codex round, scoped to this one finding, confirmed the original bypass was blocked and the test gap was filled. It then found two new holes in the comparison itself.

An APNG whose first frame matches (P1). When the bytes differ, the fallback decodes the image. An animated PNG whose first frame equals the source passed the pixel comparison. The fix rejects any multi-frame file before comparing pixels.
Redirects that leave the CDN (P2). The download followed a redirect off the CDN and accepted a final URL outside it. The fix refuses a redirect target outside Kakao's CDN before requesting it, and checks the final URL after the response.

For both, I wrote the failing test first, then the fix. I did not run a fifth Codex round; those two fixes rest on their tests.

A side finding about the size itself

The dimensions did not even stay fixed during the work. When I injected the draft into the real blog theme on a 390px screen, the body column was 310px wide, so the 1080px PNG shrank to 0.287x and its axis labels rendered at about 8px. I moved the chart to an 800px canvas, and the smallest text at 310px became 10.9px. The first injection into the real editor recorded the rehosted chart at 1080×1670; after the canvas change, the next recorded 800×1580 with BODY_IMAGES_VERIFIED:1. Neither injection published.

Dimensions are a layout choice. In Codex's reproduction, a corrected chart kept the exact dimensions of the old one. They were never going to tell one day's chart from another's.

What this does not prove

A lossy platform would stop every run. The pixel fallback requires exact equality. If the platform starts re-encoding images in a way that changes the decoded RGBA pixels, each such publish will stop at content. On the day I measured, the CDN copy was byte-identical; that is one observation of one service, not a guarantee.
It costs a request per image. Each body image now means one CDN download, with a 15-second timeout and a 4 MB cap. A failed download stops the run before the marker, and the next slot retries.
The race was mocked, not seen. The fix rests on tests plus one live measurement. The first scheduled publish with this path on was the next check, not part of this record.
A hash binds bytes, not correctness. It shows the CDN serves the picture I generated. It does not show the chart is right, and the APNG and redirect findings show that a content check has bypasses of its own to enumerate.

The lesson I took: equal dimensions are not identity. When a third party rehosts your asset, download what it serves and compare content hashes, then list what the comparison still accepts. It is the same rule as verifying the output surface: tests verify the payload you build, not what the other system renders or, here, serves.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Fixing a Python fromisoformat Error Unmasked a Stale Alarm That Had to Fire

John — Sun, 25 Oct 2026 00:00:05 +0000

Originally published on hexisteme notes.

At 21:00 on October 9, and again at 09:00 the next morning, the snapshot job in my agent telemetry pipeline raised this alert:

F3-SOURCE-CHECK-ERROR: argument --since: not an ISO-8601 timestamp: '1970-01-01T00:00:00Z'

The alert's wording pointed at suspected loss of the raw archive. Nothing had been lost. The archive copy was working, and its newest file was from 08:59 that morning. What failed was a source-freshness check inside the same job. The parse error was the easy part; what it had been covering up was the real lesson.

Python 3.9 and a trailing Z

Under cron's PATH, python3 resolved to /usr/bin/python3, which is 3.9.6. Before 3.11, datetime.fromisoformat does not accept a trailing Z, and the job passes --since 1970-01-01T00:00:00Z. My interactive shell runs 3.13, so nothing reproduced by hand. The argument was not the only casualty: 540 of the 541 ended_at values in the database also end in Z, so under 3.9 the parser would have read none of them.

The fix is a small wrapper, _fromisoformat(), that rewrites a trailing Z or z to +00:00 before parsing; both the database and CLI parsers use it. The test that matters, test_cron_interpreter_reads_z_suffix_timestamps, runs the module under the cron interpreter (PAMSL_PY, defaulting to /usr/bin/python3), not the Python running pytest. It reproduced the error before the fix and passed after. The suite passed at 143 on 3.13, and the cron command, run verbatim, exited 0.

An earlier note, The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., covers how a bare python3 resolves to Apple's system Python under launchd's PATH while a fail-soft step keeps the heartbeat green. That failure was silent. This one was loud, named the wrong problem, and sat on top of a second one.

What the parse error was covering

The job's alert function writes each message into a single flag file, replacing whatever was there. Once the parser worked, the next run would put this in its place:

F3-SOURCE-STALE source_data_age_seconds=771145.565 newest_ended_at=2026-10-01T11:46:44.435000Z: Claude record files newer than newest ended_at exist; DB lags source

771,145 seconds is just under nine days. I read it as a real alarm and left the flag in place while I looked for the cause.

There was no broken build. The database is rebuilt only by the weekly report job, Mondays at 09:30. The October 5 build exited 0 and loaded all 541 top-level files then in the archive. The 76 files not in the database (617 archived minus 541 loaded) all belonged to sessions that started after that build. The database was exactly as current as a weekly design promises.

The newest session inside it, though, had ended on October 1 at 11:46 UTC. Between September 20 and October 5 there were only four Claude Code sessions, because most of my agent work had moved to Codex (see Our LLM Telemetry Stayed Green While the Work Moved to Codex). So the October 5 build came after a quiet gap of 3 days, 12 hours and 43 minutes.

Why that alarm had to fire

The rule was: STALE if the newest ended_at is more than 8 days old and Claude record files newer than it exist. The build runs every 7 days. On build day the newest loaded record is already as old as the quiet gap before it, and it ages all week. If that gap exceeds one day, the record crosses 8 days before the next build, and any work done since supplies the newer files. The STALE condition is then certain to come true before the next build. It stays true for roughly the gap minus one day, and whether a scheduled check catches it depends on how that window lines up with the check times.

This time the STALE window ran from 20:46 KST on October 9 until the build at 09:30 KST on October 12. Both failed runs fell inside it. The parse error had been hiding an alarm that would have fired on both of those runs and kept firing until Monday.

The second condition makes it worse. With no new activity, the same rule reports INACTIVE and stays quiet. What tripped it was going back to work.

Measuring the wait instead of the newest record

The Codex note describes the opposite failure in this pipeline: a staleness check keyed to when the database file was last rewritten, which a weekly rebuild always satisfies, so it stayed green while the newest session inside was ten days old. That note closed by recommending a freshness check on the gap between the newest ingested session and the report date. The checker here did exactly that, and test_db_file_mtime_is_not_freshness keeps a freshly rewritten file with old data from reading as FRESH.

Newest-record age answers "when did the data last move?" In a store loaded in weekly batches, that depends on when I happened to stop working before the last batch. The question I wanted answered was "has anything waited too long to be loaded?"

So the definition changed. The check takes the record files modified after the newest ended_at (plus a five-minute grace for that session's own last write) and measures how long the oldest of them has waited: unreflected_wait_seconds. Those files are a proxy for what the database has not loaded yet. When the newest ended_at is more than 8 days old, STALE now also requires that wait to exceed 8 days. If such files exist but none has waited that long, the status is AWAITING_BUILD, which does not alert. A quiet gap before a build no longer trips the alarm; a file has to sit newer than the loaded data for longer than the threshold. The alert line also prints unreflected_wait_seconds=.

Three existing tests were rewritten to the new meaning, and a new one replays October 10: newest ended_at on October 1, first file after the build at 00:58 UTC on October 5, source age over 8 days, expected AWAITING_BUILD. The suite passed at 145, and the shared telemetry environment check passed. The cron command, run verbatim under 3.9, exited 0 with no alert: AWAITING_BUILD, 299 record files waiting, the oldest for 5.37 days, newest-record age 8.94 days. Those are record files, not sessions; one session spans several files. With both causes resolved, I deleted the alert flag.

A read-only query that wasn't

I caused one more failure during the investigation. I queried the database directly with Python's sqlite3 and a file:...?mode=ro URI. The database runs in WAL mode, and that read left a 0-byte -wal file and a -shm file beside it. The checker refuses to read while a -wal file exists, so the next check reported DB_UNAVAILABLE (active WAL present). I confirmed no process had the database open and deleted both files.

On a copy, mode=ro created both sidecar files and immutable=1 created neither. The checker already opens with immutable=1, and a test verifies that its read of a WAL-mode database leaves the directory unchanged; my hand-typed query skipped that. Direct queries now use ?immutable=1. This is what happened on this machine with this database. I have not checked it against SQLite's documentation, so it is not a general claim about SQLite.

What this does not prove

The evidence stops on October 10. The record documents one cron run of the new rule against real data, and it does not show what the check printed after the October 12 build.
The new rule is quieter on purpose. A new record file can now wait up to 8 days to be loaded with no alert. In the alert mode the cron job uses, AWAITING_BUILD prints nothing; the wait is still recorded in the JSON report.
The wait is measured from file modification time, which the module itself calls a proxy: the last write to a file, with its contents unread.
In the code, the link between the 8-day threshold and the 7-day build is a comment beside the constant. If the build cadence changes, the threshold has to change with it by hand.
_fromisoformat() handles a trailing Z or z and nothing else, and the job still runs on the 3.9 system interpreter. Whatever else differs between 3.9 and 3.13 will surface either in a test that runs the module under that interpreter or in the scheduled run itself.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Qwen3-TTS Silently Dropped My Instruction, and My A/B Never Noticed

John — Sat, 24 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

While evaluating Qwen3-TTS for narration, I ran a small A/B to see whether an instruction could raise the speaking rate. It couldn't, I wrote that down, and speed control moved to post-processing. The next day a different experiment turned up a line of code that made half of that A/B meaningless: on the smaller model, the instruction never reached the model. The call returned normal audio, the transcript was perfect, and nothing in the measurements looked wrong.

Before comparing settings of a control knob, show that the knob moves the output at all. Otherwise the comparison measures nothing.

The A/B

The goal was fast narration, around 230 words per minute. The setup was Qwen3-TTS's CustomVoice models through the qwen_tts 0.1.1 package on torch with the MPS backend, at two sizes: 0.6B and 1.7B. generate_custom_voice has no speed or rate parameter. Its only lever is instruct, a free-text instruction. I tried two instructions on both model sizes, one asking for roughly 230 wpm and one asking for auctioneer-fast delivery.

Rate was measured over the voiced part of an 18-word sentence. The 0.6B model came in at 139.1 and 131.8 wpm for the two instructions, and the 1.7B at 140.6 and 126.3. The matched no-instruction runs on the same sentence came in at 129.9 wpm for the 0.6B and 128.3 for the 1.7B. The decision record summarised this as a gain of 0–8% over a 126–130 baseline; against the matched baselines, the largest change was 140.6 against 128.3, or +9.6%. Word error rate stayed at 0%. The decision record concluded that instruction could not get to 230 wpm and pointed to a post-processing time-stretch or the other engine's speed setting instead.

The same record chose the 0.6B as the default model. Its peak memory footprint of 3.22 GiB fit a rule requiring 4 GiB of available memory, and the 1.7B's 5.6 GiB did not.

The line that discarded it

A voice-cloning smoke test the next day recorded this as a side finding. In qwen_tts 0.1.1, generate_custom_voice contains:

if self.model.tts_model_size in "0b6": # for 0b6 model, instruct is not supported
    instruct = None

The 0.6B CustomVoice checkpoint's config carries the size tag 0b6. For that model the instruction becomes None before tokenisation and generation proceeds as if none had been passed. The 1.7B checkpoint carries a different tag, so its instructions went through.

Nothing in the installed 0.1.1 package announces the drop. The branch raises no exception and emits no warning or log line. The wrapper file that holds the branch has no logging calls at all, and the two warning_once calls in the model file concern gradient checkpointing. The docstring describes instruct as optional, "If None, treated as empty (no instruction)", and says nothing about model size. Passing an instruction to the 0.6B model is accepted, has no effect, and returns perfectly good audio.

So the two 0.6B "instructed" runs were really two more runs without an instruction. One of them landed at 139.1 wpm, above its matched no-instruction run at 129.9, with no instruction behind it.

The pattern matches Why My Correct Config Value Was Being Ignored: a correct value set in one place gets overruled before it reaches the code that uses it, and nothing throws.

What the A/B still supports

The 0.6B half did not test the instruction's effect. The other half did receive the instruction, and the 1.7B runs reached 140.6 and 126.3 wpm. That pair still supports the narrow claim that these two prompts did not get the 1.7B anywhere near 230 wpm. It rests on one run per prompt, though, and the 0.6B pair shows that a run without any instruction on the same sentence can land at 139.1. The models differ, so this does not show the 1.7B instruction did nothing. It does show that no run in the A/B demonstrated the knob changing the output.

Nothing in the A/B flagged the problem. There was no error, the WER was 0%, and the small rise in wpm looked like a weak effect when one input was in fact missing. The record looked like a finished experiment.

Speed control went to a per-segment ffmpeg atempo stage that rejects factors outside 0.5–2.0. A calibration run set the factor: Qwen3's raw speech ran at about 131 wpm against an effective 183 for the existing engine, so matching it took factors of 1.41 to 1.76 depending on the speaker.

Checking the knob on the clone model

The smoke test that found the line also asked the question the A/B had skipped, this time about tone on the clone model. Qwen3's Base model, which does the cloning, has no instruct argument on its public clone method, and issue #25 says Base does not support instruction. The smoke test passed instruction ids through a non-public path anyway and measured the result. With an "excited" instruction the clone came out at an F0 mean/standard deviation of 237/59 Hz and 6.80 syllables per second. With "calm" it came out at 234/56 Hz and 6.46. There was no error, and these single runs did not show meaningful tone separation.

The reference clip, by contrast, moved the output a lot. Cloning Korean from a calm reference gave 221/52 Hz at 5.99 syllables/s, and an excited reference gave 271/86 Hz at 6.42. Into English, the same two references gave 221/42 and 217/41, so tone transfer across languages was weak.

So tone control goes through a bank of per-tone reference clips, not instructions. On my own recordings the Qwen tone bank spread F0 by 23.3 Hz across five tones and rate by 1.94 syllables/s. Its rank correlations against my own five-tone reading were 0.7 for F0 and 0.6 for rate. With n=5, that shows direction only.

The same check caught a knob that worked on one voice and not another. VoxCPM2's bracketed style prefix separated tones on the stand-in speaker, but on my voice the five tones spread F0 by only 7.3 Hz, with a rank correlation of -0.41. A knob checked on a stand-in has to be checked again on the configuration that ships.

The instrument needs the same check

The Base model's built-in speaker encoder scored the clones at 0.987–0.994 cosine similarity and a different, male preset speaker at 0.946. The clones were above that control, but on a narrow scale. Re-measured with WavLM-SV, which belongs to neither TTS system, the different speaker scored 0.275 and the same speaker across other tone references scored 0.977–0.992. The negative control that tested the wrong field makes a related point about checks: a check that flips on cue still has to be shown to react to the property it guards.

The rule I take from it

Before running an A/B on a control knob:

Run the knob at its two extremes and measure a number that should move, such as wpm for speed or F0 for tone. If the extremes don't separate by more than repeat runs vary, stop.
For an optional argument, trace its path to the model input. A library can discard it without raising an error.
Calibrate the measuring instrument against a control known to differ.
Repeat the check on the model size, speaker and mode you will ship, not only the one you prototyped on.

What this does not settle

The clone numbers come from one run per combination with seed 7 fixed, and the speed A/B from one synthesis per setting, so they show direction only. The stand-in reference clips were generated by Qwen's own CustomVoice model, which gives Qwen Base a favourable distribution to reproduce. That bias is why the timbre-similarity ranking had to be measured again on my own recordings. The blind listening test on those recordings had one listener and 15 Korean clips. The Qwen tone bank got 2 of 5 tones right, against a chance level of 1 in 5. Serene, calm and warm were mixed up with each other, and my own reading has nearly the same F0 for those three, so part of that limit probably comes from the reference performance. That is an estimate, not a measurement.

The dropped instruction is observed behaviour in qwen_tts 0.1.1. The package comment says instruction is not supported for the 0.6B model, and I did not test whether that checkpoint would respond if the wrapper passed an instruction through. Later releases may handle this differently. The 1.7B speed result still rests on two runs.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.