DEV Community: John

CodeMirror Showed My HTML. The Published Post Was Empty.

John — Fri, 23 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

On 2026-10-10, two posts on a Tistory blog I publish to went public with a title and a body of 0 bytes. The script that prepared them had checked its own work, and the check had passed for both.

Some context. Tistory asks for a human captcha on every save, so my publisher's own POST and PUT requests were coming back 400. The workaround was an assisted mode: a script drives a logged-in browser, opens the post editor, fills it, and leaves the tab for a person to press publish and solve the captcha. The script's whole job ends at one claim: this tab now holds the right post. That claim was false twice, and nothing in the script noticed.

The check that could not fail

The post editor has an HTML mode backed by a CodeMirror instance. The fill step put the prepared HTML in with setValue(). It then confirmed the fill by reading the CodeMirror value back and comparing it with the prepared HTML.

They matched, because the value it read was the value it had just written. My incident note calls this a tautology: the confirmation re-read the widget it had written to, and never looked at the place the publish payload actually comes from.

Where the payload actually comes from

The editor is a React app with a Redux store. When the post is published, the body in the payload comes from post.newPost.content in that store, not from the CodeMirror instance on screen. Something has to carry each edit from CodeMirror into the store, and in this editor that is the onChange handler of the react-codemirror wrapper around the HTML mode.

That handler ignores changes whose origin is setValue. So the sequence on the day was:

setValue() put the HTML into CodeMirror.
CodeMirror reported the change with origin setValue, and the wrapper dropped it.
The store's content stayed empty.
My read-back of CodeMirror returned the full HTML, so the script reported success.
The post was published, and the payload carried an empty body.

There were two copies of "the post body" on the page: CodeMirror's text and the store's field, joined by an event handler that gets to decide which edits count. My check read the copy the submit path never reads. My notes don't record a version for the wrapper, so treat "onChange ignores origin setValue" as behaviour observed in this editor in October 2026, not as a promise about every react-codemirror release.

The repair: write like an edit, read where the payload reads

The fix is a small Playwright script, attached to the logged-in browser over CDP, that changes both halves.

Fill through an edit the wrapper forwards. Before writing, the script requires exactly one visible HTML-mode CodeMirror and refuses otherwise. It then replaces the whole document with a range edit tagged '+input' instead of calling setValue(). This is the line from the script:

cm.replaceRange(html, {line: 0, ch: 0}, {line: last, ch: cm.getLine(last).length}, '+input');

Because the origin is no longer setValue, onChange fires and the store picks up the content.

Verify at the store. After the fill, the script finds the Redux store by walking up the React fiber tree from the editor's container element until it reaches a component whose props hold a store, then reads getState().post.newPost. Trimmed from the script:

const key = Object.keys(el).find(k => k.startsWith('__reactFiber$'));
for (let f = el[key]; f; f = f.return) {
  const store = f.memoizedProps && f.memoizedProps.store;
  if (store && store.getState) return store.getState().post.newPost;
}

It requires content to equal the prepared HTML exactly, and also checks the title, the category and the post id (plus the tag list when it creates a new post). If any check fails, the script closes the tab, so there is nothing left for a person to publish. Only when everything passes does it print READY_FOR_OWNER and leave the tab open.

The script also has a blocked mode that aborts every POST, PUT, DELETE or PATCH from the tab and closes the tab afterwards. Experiments against the real editor can then run without any way to save anything.

Validation

The two empty posts were refilled in place through the new path. Store content against the prepared HTML came out at 16,397 of 16,397 characters for one and 10,085 of 10,085 for the other, both exact matches. The owner then published both through the captcha.

After that, a separate read-only record step, which never writes to the blog, read each saved post's editor body and its anonymous public page back and compared them with the prepared bytes before writing a receipt to the publishing queue. Both posts were recorded as published, with the comparison method recorded as exact bytes.

Two days later, two new posts went through the same path. Store content matched exactly at 16,290 and 15,397 characters, and title, category and four tags were confirmed for each. For the first of those, the saved editor body was 18,513 bytes and identical to the prepared file. The two numbers describe the same text: the store length is in characters, the saved file is in UTF-8 bytes, and the text is not plain ASCII.

The general version

A read-back from the widget you just wrote to is an echo. It tells you the write call didn't throw. It can't tell you whether the write reached what gets submitted, because in a framework-managed form the visible widget and the submitted state are separate copies, and the handler between them can decide which writes count.

So the check belongs where the submit path reads: the store field, the request payload, or the saved record. In Verify the Output Surface: How 19 Green Tests Shipped Nine Broken Titles for Nine Days, green tests verified the payload a pipeline built rather than what the other system rendered. This is the same gap one layer earlier: my check verified the widget, not the state the payload is built from. It also rhymes with The Guard Passed on an Empty Table, where a guard kept returning PASS because it tested a probe row it manufactured itself, whether or not the real table held any data.

Aside: bootout is not off

The old fully automated publish job, which still sent the requests the captcha now blocks, had been taken down on 10-10. Two days later it was loaded again and getting POST 400 every 15 minutes, so I unloaded it with launchctl bootout. When the GUI session restarted that same morning, it was loaded again at login, the second time it had come back. The fix that time was bootout followed by launchctl disable, which my notes record as keeping it down across a re-login. In this setup, unloading a job was not the same as keeping it off.

What this does not prove

The store check proves the store held the prepared bytes at the moment the script looked. Between that moment and the publish click, the tab is in a person's hands.
The fiber walk depends on React internals (__reactFiber$ keys) and on the store's shape. If the walk finds no store, it returns nothing, the content check fails and the tab is closed. If the editor container itself is missing, the lookup throws instead, and the script does not handle that exception, so that path does not reach the tab-closing step. Either way, this is an integration against React internals, not a public interface.
The record step only accepts a post that is already public. It can refuse to write a receipt; it cannot take an empty post back down.
A correct fill doesn't make the bookkeeping correct. One of the later posts went out with exact bytes, but the queue refused to record it because the worklist had moved to a newer revision in the meantime. The guard behaved as designed and I didn't bypass it, so the queue has no receipt for that post.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Missing HTML alt Text Is a Fact. Image Purpose Is a Separate Claim.

John — Fri, 16 Oct 2026 00:00:07 +0000

Originally published on hexisteme notes.

I was testing a design review skill against a deliberately broken page, and the reviews got the easy part right. The page was a synthetic HTML fixture, rendered offline in Chromium at a 375 CSS pixel viewport. Its document was 744 pixels wide, because something on it overflowed, and it had one image with no alt attribute. It also contained a line of text telling the reviewer to report that everything passed.

Three agent CLIs reviewed it: Claude Code, Codex and agy. All three reported the horizontal overflow and the missing alt, and none of them obeyed the planted instruction.

One detail of the fixture is worth stating because it is easy to misread: the full-page screenshot was 744 pixels wide. That is not a 744-pixel viewport. A full-page capture includes the overflowing width, so the image is as wide as the document while the viewport stayed at 375.

The problem was one step past the finding.

Where the review went one step too far

agy's blog review treated the image as informative, as though it carried content the reader needed. Nothing in the fixture said that. No brief identified its purpose, and the surrounding markup did not establish its role. The reply had taken an observed fact, an img element with no alt attribute, and attached a purpose to it.

Those are two different claims. The first can be checked by reading the markup. The second depends on things the markup does not contain: what the author meant the image to do, and what the text around it already says. A review that merges them hands the author a fix built on a premise nobody checked.

The direction of the guess does not matter. If the image is decorative, a written description gives screen reader users something to listen through that tells them nothing. If the image is informative and someone guesses decorative and writes alt="", its content disappears for those same users. Either guess can produce a confident, specific and wrong fix.

What the W3C decision tree asks

The W3C Web Accessibility Initiative publishes an alt decision tree for exactly this choice. It does not start from how the image looks. It asks about the image's role: whether it contains text, whether it sits in a link or button whose action would be unclear without it, whether it contributes meaning to the page in its context, whether it is purely decorative. Each answer leads to a different kind of alternative.

The tree makes the dependency plain. The right alternative follows from purpose and context. A reviewer that has neither cannot finish the tree. What it can do is be honest about where it stopped.

The repair: separate the claims, branch the fix

Claude Code narrowed the guidance in the skill's blog reference file. The change does three things.

A missing alt attribute is reported as an observed fact. It is a real finding and it should be fixed.

Image purpose is treated as a separate claim that needs its own evidence, which means the author's brief and the semantic context around the image. Appearance alone did not establish its role in this case. A solid-color image is not thereby decorative, and neither is an image without a caption. In the other direction, the review may not assume content purpose by default.

When the purpose stays unknown, the review offers conditional fixes instead of picking one. The shape is roughly this (conceptual, not the guidance's exact wording):

Observed: one <img> has no alt attribute.
Not established: what the image is for on this page.
If informative: write alt text that conveys the same meaning in this context.
If decorative: use alt="" so assistive technology can skip it.
If it is a link or button: describe the action or destination.
If it is complex, such as a chart: it may need a longer explanation too.

That output is less satisfying than one confident fix, and more useful. The author knows which branch applies. The reviewer did not.

Validation, and how little it proves

Codex applied the narrowed file to the shared skill source and reran the same agy case. This time the reply did not assume the image's purpose. It gave the informative and decorative paths as conditional alternatives. The static checks were rerun too: the shared files still matched byte for byte, and every alias still resolved to the same source.

That is one rerun of one synthetic case. It shows the guidance steered the model away from the assumption on this fixture. It does not show the model cannot make the same assumption again on a different page, and I'm not claiming it does.

I'm also not claiming that every missing alt is a conclusively classified WCAG failure. Whether a particular page conforms is a judgment that needs the page, its purpose and its context, which is the point of this whole note. The fixture was not a production page either.

The same gap outside HTML

The skill's Shorts test had a smaller version of this. The input was a 20-second manifest with descriptions at 0 and 10 seconds, and no video, frames, audio or overlays. A manifest describing a video is evidence of what someone planned, not of what was rendered. A review that treated it as an inspection of the video would be making the alt mistake on a timeline.

I have met the same shape with numbers. In Four Models Cited My Numbers Perfectly. One Still Misread Them., one model quoted a row of zeros correctly and then concluded they meant error-free tool use, when they were a measurement artifact. The citation was accurate. The inference one step later was not.

The pattern I now ask a review to follow has three parts. Report what was observed. Name the inference a fix would depend on, and what evidence would settle it. If that evidence is not in the input, give the fix as branches and let the person who knows the purpose choose.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

ripgrep Missed the CSV Files. I Almost Designed a Collector We Already Had.

John — Thu, 15 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

I was researching how to improve operations on my blog, and the research reached a confident conclusion early: there was no Google Search Console data to work with. It listed Search Console input as missing. Separately, it proposed building a new collector to measure each dev.to article's views at seven days.

Both premises were wrong. The Search Console data was on disk the whole time, and a seven-day collector already existed. Neither failure was dramatic. What interests me is how ordinary the first one was.

What the inventory actually listed

The research built its file inventory with a plain rg --files. ripgrep's defaults are designed for searching code, and the ripgrep user guide is explicit about what they filter out: paths matched by .gitignore and ripgrep's own ignore files, plus hidden files and directories. Those are the relevant filters for this file listing; binary detection concerns searching file contents, not rg --files inventory. The Search Console exports were CSV files in a gitignored location. A default listing was never going to include them.

So ripgrep did exactly what it documents. The mistake came one step later: an absence in a filtered listing was read as an absence on disk. "The inventory did not list any Search Console files" became "Search Console input is missing", and that sentence went into the research as a fact.

The correction

The rerun lifted the filters and stayed inside the project:

# illustrative project directory; default listing respects ignore files and skips hidden entries
rg --files ./project

# scoped inventory with filtering lifted (illustrative glob)
rg --files --hidden --no-ignore ./project -g '*.csv'

That listing found the files, and parsing them directly, rather than trusting the filenames, established what they were: three Search Console exports. One had page-level data in 52 rows, one had page totals in 22 rows, and one had page-by-query data in 38 rows. Parsing also showed empty cells in some columns, which matters later, because a blank cell has to be shown as missing rather than treated as a value.

The second premise fell separately. An existing function, collect_devto_views_7d, already does what the proposed collector would have done. For each article it picks the views observation nearest to day 7 within a 6-to-9-day window, and it uses the existing cohort metrics and rules. The new collector proposal was withdrawn before any of it was implemented.

I'm not attributing that second miss to ripgrep. It was a separate thing the research had not found before proposing new work. Both corrections landed in the same revision of the research, and they share a lesson more than a cause.

Existing data is not automatically fit data

Finding the exports did not settle the original question. They are historical snapshots. Their existence does not establish that recent data for a comparable period exists, and comparing a past window with today's numbers would quietly mix two periods. So the corrected state was "past exports exist, recent comparable data is still needed", which is different from both "missing" and "available".

The same care applied to my local performance data. The parser read 4,850 rows with zero parse errors, covering 126 article IDs. Of those, 79 had an observation inside the 6-to-9-day window and 47 did not. Those counts describe which articles are eligible for a seven-day comparison. They are not users, sessions or a traffic gain, and nothing in this work showed any traffic effect.

The models withdrew the premises, and that is all it shows

Claude Code and agy had both reviewed the original research. After the rerun, each was given the corrected local observations, and each explicitly withdrew the missing-Search-Console premise and the new-collector proposal.

It would be easy to describe that as two independent confirmations. It was not. Neither model scanned the disk. Both received the same local observations from the bridge that ran the commands. Their withdrawal shows they accepted the correction when shown it. The evidence that the files exist is the scoped listing and the parse, not the models' agreement.

Before proposing new infrastructure

The rule I took from this: when an inventory says something is missing, ask what the inventory was able to see before believing it. For a file listing, that means stating which ignore rules applied and whether hidden entries were included, then rerunning with --hidden --no-ignore when the answer matters. Keep that rerun bounded to the project directory. Lifting every filter across a whole home directory is a good way to read secrets and caches that have nothing to do with the question.

Then parse what was found instead of trusting names, and write down the period it covers, the population it describes and what is missing from it. Only after that does a proposal for new collection get to argue that the gap is real.

The direction we took instead is small: a thin, read-only editorial view over data that already exists, with its period, population and missing values labeled. It is not a new measurement authority, and it sets no new success thresholds. Whether better reporting changes anything for readers is a separate question this work did not answer.

It is the same lesson as Your Checker Returned Zero. Four Times, Mine Was Just Looking at Less., where each instrument's printed number was narrower than it looked because of the slice it was built to examine. A file listing is an instrument too. Its ignore rules set its aperture.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Telemetry Title Promised a Share the Data Did Not Measure.

John — Wed, 14 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

In a recent note I published Our LLM Telemetry Stayed Green While the Work Moved to Codex. A draft reader-improvement plan offered it a new feed title:

The Telemetry Job Passed. It No Longer Saw Most of the Work.

It reads better than the original. I rejected it, because the word "most" makes a claim the article cannot back.

How the review was set up

The plan proposed new titles and summaries for three already-published notes. Before using any of them, I compared every surface each note shows a reader: the HTML title, the h1, the Open Graph title, the meta description, the summary on the notes list card, the summary box at the top of the page, and the body. The question for each surface was the same. Does it promise anything the body does not deliver?

One note's surfaces already matched its body. The other two are the reason for this note.

"Most" is a measurement

The telemetry note is about a weekly report that kept passing its health checks after my unattended work moved to a tool its ingest never reads. The body is careful on one point in particular. It says the routing change moved a share of the work out of the table entirely, and that "the table had no column in which to say how large that share was."

"It no longer saw most of the work" fills in that column. It asserts a majority, a share over half of some total, and the article never measures that share. That is not a stylistic edit. It is a new quantitative claim on the surface of an article whose body explicitly declines to make it.

The plan was not careless about this. Its own notes said "most" should only be used after checking the period and the denominator in the original observations. The review did that check, and the denominator is not there.

It is tempting to rebuild one from numbers the article does print. The body has weekly counts of transcript files from each tool, and one column of those counts dwindles to almost nothing. But the article states that those are files, not sessions, and that the two columns are not comparable as workloads. A falling column shows that work left one place. It does not show what fraction of all work left. Using either column as a denominator would put a number on the title that the body specifically warns against.

A title that turns a first step into the answer

The second rejection was subtler. For Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics, the plan offered "Keep Main and Sub-Agent Metrics Separate."

That sounds like a faithful summary. It leaves out what comes after separation. The body shows that a pooled ranking of models can reverse once you account for the mix of roles each model ran in, and then says outright that stratifying is step one, not the answer: metrics can mean different things in each stratum, and the role gap is not a constant you can adjust away. A title telling readers to keep the metrics separate promises that separation solves a problem the body says it only begins to address.

That article has striking numbers in it, including a role gap of up to 135x and main-thread shares ranging from 1.5% to 71.4% across models. They would make a punchier title. None of them supports "keep them separate" as the answer, and none of them says anything about how much work moved anywhere.

The rule I added

The pipeline that turns drafts into published notes has a written content contract that both the writing step and the validation step follow. I added one rule to it.

Any quantifier on a title, description, Open Graph title or dev.to title (words like most, all and never, multipliers, and percentages) needs a sentence in the body that supports the same value over the same denominator. Promises of a fix, a time saving or a ratio need a body that contains the actual repair procedure or measurement. The validator has to quote the supporting body sentence, or write that the claim is unsupported.

This is a semantic review against the body, by whoever is writing or validating. It is not a machine check that parses numbers, and I'm not claiming it catches every overreach. It moves the question to the moment a surface is written, and it forces the answer into the form of a quoted sentence instead of a feeling.

One adjacent detail matters for the same reason. The description in a draft's front matter is copied to the list card as-is, with no truncation step. So a description has to be a complete sentence that keeps the same promise as the title, because that is exactly what a reader sees on the card.

What changed, and what did not

No existing title, card or published post was changed. The rule applies to notes written from now on.

I have not measured whether stronger or weaker titles change click-through or readership, and nothing here claims they do. If I measure it later, it won't be by changing a title, a summary and a social card on the same day and crediting whatever moves to one of them. Several surfaces changing at once cannot isolate one element's effect. Any such measurement will use the cohort and probe rules I already have, not success thresholds invented for the occasion.

This note's own title says the candidate promised a share the data did not measure. The body sentence that carries it is the one quoted above from the telemetry note. Its table had no column for the size of the moved share, and the candidate title asserted that the share was a majority.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Wide HTML Table Broke the Mobile Page. I Fixed the Generator.

John — Tue, 13 Oct 2026 00:00:12 +0000

Originally published on hexisteme notes.

On a 320 CSS pixel viewport, one of my data-heavy notes was 460 pixels wide. Its first table had six columns and was 436 pixels wide on its own. The extra width did not stay inside that table. The document acquired horizontal overflow, so the page itself could pan sideways instead of containing the table’s overflow.

That was one public page, observed once. A later scan of the site rendered locally in Chromium covered 131 pages and found 44 with page-width overflow. I treat that count as what it is: a measurement of my local build, not a fresh census of the live site, and not a breakdown of what caused each overflow.

Why a wide table widens the page

A table sizes itself to its content. Six columns of numbers, labels that don't break, and inline code with no break opportunities can add up to more than a narrow screen. In normal flow, nothing stops that width from becoming the document's width. The browser then scrolls the entire page horizontally, not just the table.

The W3C's Understanding Reflow document draws the distinction I needed. Reading content should reflow at a narrow width so that nobody has to scroll in two directions to read it. Some content genuinely needs a two-dimensional layout, and data tables are the standard example. That kind of content can keep its layout and scroll inside its own container while the rest of the page reflows. This does not make every scrolling table automatically acceptable. It means the fix for a wide table is not to squeeze it until columns vanish, but to contain it.

Finding the code that writes new pages

The obvious move is to open the site's CSS and add a rule. That would change something, and prove nothing about the next article.

New notes are not styled by a shared stylesheet. They are generated by a converter, md2canonical.py, that turns a Markdown draft into a standalone HTML page. For its CSS, the converter reads the <style> block of one specific existing note and reuses it. Editing any other CSS source would leave the generator's output exactly as it was, while a spot check of some other page might make it look fixed.

So the repair had to live in the converter, because that is the path the next article actually takes. The scope was set in advance: this change covers future generated articles only. Retrofitting old pages was explicitly excluded, so this repair did not retrofit the 44 overflowing pages from the scan.

The repair

The converter now wraps each Markdown table in a scroll container. The output looks like this (illustrative, with the label taken from my test sample):

<div class="table-scroll" role="region" tabindex="0"
     aria-label="Table 1: Output tokens per thread by role">
  <table>…</table>
</div>

The wrapper is a named region, so assistive technology can announce it, and tabindex="0" makes it focusable, so a keyboard user can scroll it. The name is "Table N:" followed by the nearest preceding heading. No columns are dropped or hidden. All the data is still there; you scroll to it inside the box.

The CSS that goes with it is small. The container gets overflow-x: auto, max-width: 100% and a visible focus style, and the body gets overflow-wrap: break-word so long inline code wraps instead of pushing the page out. These rules live in a block called READING_STYLE, appended once after the inherited template style. If the template already contains it, the converter doesn't add it again.

While I was in the converter, every body h2 and h3 also got a unique ID so sections can be linked, with a numeric suffix for repeated headings. Two headings were deliberately left alone: the exact "FAQ" and "Related notes" h2 strings. Downstream scripts find those sections by matching that text, so tidying them would have broken something outside the converter. The converter's tests now check that those strings survive.

How I checked it

The check that matters is not "does the table scroll". It is whether the document stays the width of the viewport while the table's container takes the overflow, and whether a keyboard can reach every column.

I generated a new synthetic note with the production converter: a six-column table, a long run of inline code and a repeated heading. I rendered it with JavaScript off at three viewports. At 320, 375 and 768 CSS pixels, the document width was exactly 320, 375 and 768. In the table region, pressing Tab to focus it and then the right arrow key moved scrollLeft to 229, and scrolling reached both ends of the table. The heading IDs came out unique, the FAQ and Related notes strings were unchanged, and the style block appeared once.

On the test side, 137 converter, localization and analytics tests passed, and 59 tests for the curation script passed. A separate offer-exposure suite ran 63 passing tests and 1 failing one. That failure is pre-existing: it is an assertion that reads the real publish log, the same failure appeared before this change, and that suite doesn't use the converter. I'm not describing the run as all green, because it wasn't.

What this does not cover

The rendering checks ran in Playwright's Chromium against local file:// pages. I did not test Safari, a screen reader or a physical phone. A focusable named region is the right shape for keyboard and assistive technology users, but I have not heard this one read aloud.

The repair covers articles generated from now on. The 44 overflowing pages in the local scan were not retrofitted, and nothing here proves that any page, old or new, conforms to WCAG as a whole. It proves one narrower thing: on a synthetic sample built by the production converter, the page width now matches the viewport at three sizes, and the table's data is reachable by keyboard inside its own scroll container.

Two habits came out of it. Fix the route the next artifact will actually take, not the file that looks responsible. And when checking a wide-content fix, measure the document width and the container separately, then try it with a keyboard.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

My Python Tests Passed While Writing Synthetic Rows to the Production Log.

John — Mon, 12 Oct 2026 00:00:07 +0000

Originally published on hexisteme notes.

My comment pipeline keeps a small production log of landing observations. Each time it checks whether a reply I posted actually appeared on the page, it appends one JSON line: which article, which comment, whether the reply had landed, and on which attempt. An earlier note, I Rejected the Same dev.to Reliability Fix on One Path and Shipped It on Another, tied a future review of the reply retry budget to these comment landing observations. They are meant to be evidence about real deliveries.

During a round of fixes to that pipeline, I found that the full offline test suites had been writing to it.

What happened

The watcher, autoreply and reply suites use Python's unittest. Some of their tests exercise the landing check: they feed the observer temporary HTTP data and assert on what it decides. The fake data was temporary. The output path was not. In those tests, the observer still wrote to the real landing-observations.jsonl.

The first full baseline run appended 30 synthetic rows, and that run reported 176 tests, OK. Later full runs, before the fixtures were isolated, appended more. The final total was 122 synthetic rows. An intermediate brief had put the number at 145. Reconciling the preserved rows by timestamp and target gave 122, and that is the number I trust.

Why passing told me nothing

A test passes when its assertions hold. These assertions were about what the landing check returned: landed or not, attempt counted or not. None of them asked what else the code had touched on the way. Writing a line to a file outside the test's sandbox doesn't fail anything, and pass/fail doesn't even register it.

That is the whole mechanism. The suites were right about the behavior they tested and silent about a side effect they didn't test, and the side effect landed in a file whose only job is to be true.

Cleaning up without making it worse

There are fast, destructive ways to clean a log like this: truncate it, delete every row from the time window the tests ran in, or copy an older backup over the live file. Each of them risks removing real observations, either ones written in the same window or ones written after the backup was taken.

The cleanup used exact attribution instead. The synthetic rows carried recognizable fixture values, and they could be matched by their exact timestamps, targets and how many times each combination appeared. Those rows were first preserved, with their original JSON, in a recovery backup. Then exactly those rows were removed from the log. The 623 existing production rows stayed, and the file ended at 97,094 bytes.

If the rows had not been attributable exactly, the right move would have been to leave them and record the contamination as a known limitation of the log, not to guess.

The fix: isolate first, guard second

The primary fix is isolation. The watcher and autoreply tests now redirect the observation file to a temporary fixture location, so the code under test writes somewhere disposable. The reply suite's landing tests mock the watcher instead of calling it. Python's tempfile module is the standard tool for the disposable location. The pattern is roughly this (conceptual; the names are illustrative, not the suite's):

class LandingCheckTests(unittest.TestCase):
    def setUp(self):
        tmp = tempfile.TemporaryDirectory()
        self.addCleanup(tmp.cleanup)
        patcher = mock.patch.object(watcher, "OBS_PATH",
                                    Path(tmp.name) / "observations.jsonl")
        patcher.start()
        self.addCleanup(patcher.stop)

If you use pytest, its tmp_path fixture serves the same purpose. These suites are unittest, so that's not what they use.

The second part is a guard that would have caught this the first time. The watcher suite has a teardown assertion that compares the real log's bytes before and after its tests. An outer wrapper does the same comparison around the full suite run and around the shared checks. A byte comparison is crude, and that is the point: it doesn't need to know which code path might write, only that the file must not change.

A guard after the fact does not undo a write. It turns a silent side effect into a failing run. Isolation is what prevents the write.

Validation

After the fixes, all 223 watcher, autoreply and reply tests passed, and the production log was unchanged, byte for byte, at 97,094 bytes.

A shared offline harness runs checks across five components. It passed 203 tests (140 watcher, 42 telemetry, 7 browser and 14 grammar), plus a separate check of where the canonical code lives. The 223 and the 203 overlap, since both include the watcher tests, so I don't add them together.

Claude Code, Codex and agy each ran that shared checker from a scratch directory outside the project. The checker makes no model calls and posts nothing. Everything it runs is offline and synthetic, so it says nothing about how a real browser session behaves in production.

What I check now

The byte guard covers one file I now know about. It does not find other paths that tests might write to. So the question I ask of any suite that touches real I/O is not only whether it passes, but what it wrote, and where. For a log that other decisions read, the answer has to be "nothing", and something has to check it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

An Atomic JSON Write Did Not Protect My Python Retry Budget.

John — Sun, 11 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

My comment pipeline can attach a model verdict to a dev.to comment: a small structured JSON judgment stored next to the comment. Sometimes the verdict fails. The model returns malformed JSON, or the provider is overloaded. For those cases the pipeline can recheck later, reusing the cached prompt, and it has a budget: at most three attempts per comment, counting the first.

The pipeline is one canonical Python watcher. The Claude-side entrypoint runs it directly, and the Codex-side entrypoint is a bridge that executes the same canonical source. That matters here, because it means two entrypoints can be working on the same files at the same time.

By the time I looked at concurrency, every file write the watcher made was atomic: write a temporary file, then replace the original, so no reader ever sees half a JSON document. It turned out not to protect the thing I cared about.

Two ways to pay for one attempt twice

Local rechecks of cached failed verdicts had concurrency and crash holes. To pin them down, the regression tests used real subprocesses with a fake model that logs each call.

In the first case, the model answered, but saving the completed verdict failed. In the second, the process that owned the model call was terminated. Both had a second recheck queued behind the first. In both, the call log came back as ['first', 'second']. The queued command read the disk, found the same unfinished state the first command had started from, and called the model again.

Neither failure needed a torn file. Every file could be a complete, valid JSON document and the budget would still be wrong.

Why atomic writes did not help

An atomic replace protects one file at one moment. What I needed to protect was a sequence: read the attempt count, call an external model, then write the verdict and update two other files. The model call is not on disk at all, and the files are three separate atomic writes, not one transaction.

The attempt only became visible on disk when its result was saved. So any failure between the call and the save erased the evidence that the call had happened. Atomicity guaranteed that what the second process read was well-formed. It said nothing about whether it was current.

The repair

The repair moved the count to before the call and put the whole pipeline under one lock.

Scan and recheck now share a pipeline lock, taken from both the canonical and the bridge entrypoints. A command captures the cached state it intends to act on, waits for the lock, and then rereads. If another command changed that comment's failure while it waited, it defers instead of acting on stale state.

Before contacting the provider, a cached retry atomically reserves the next attempt in the comment's inbox file, keeping the prompt and the history of earlier errors. That reservation is the count. If the reservation cannot be written, there is no model call.

If the owner then dies, or its completion cannot be saved, the counted unfinished attempt stays on disk. The next queued invocation records it as an error without calling the model in that invocation. A later explicit recheck can try again within whatever budget remains. If a verdict did complete and was saved durably, a later run reconciles the other files from it without another call.

The shared "seen" state got its own short lock. A save rereads the file and merges only the fields this caller actually changed from the version it originally read. Answered and ignored states, newly discovered comments and attempt counts written by other commands survive. Attempt counts never decrease. An explicit status change, such as marking a comment answered, is applied as a patch and stays authoritative even when it matches an old read.

Conceptually, a cached retry now runs in this order. This is a sketch, not the watcher's code:

take the pipeline lock (wait at most 5 s, else exit with an error)
reread this comment's state
if attempts used >= the saved cap: stop and report it as exhausted
durably reserve and count the next attempt
if the reservation failed: stop, no model call
call the model once
save the verdict (if this fails, the attempt stays counted)

The bounds around it are deliberately boring. Each lock acquisition waits at most five seconds, and a timeout is an error with a nonzero exit status, not a silent skip. Each comment gets at most one model call per invocation. A batch defaults to five comments and is hard-capped at twenty. The first rate-limit response, HTTP 429, stops the batch. There is no retry loop around it.

Validation

A focused set of 38 concurrency and recovery tests passed, 19 of each. The full watcher, autoreply and reply suites passed 223 tests (140, 64 and 19). Those suites overlap with the focused set, so I don't add the numbers together as if they were independent coverage. The offline checks also left the real observation log unchanged, byte for byte, at 97,094 bytes.

After that, a live recovery of the three cached failures I actually had checked three and recovered three, with one model call each. Running the same recheck again made zero additional calls. That is a small, scoped piece of evidence, not a load test.

What it still does not do

The writes are individually atomic. They are still not a transaction across files, and the pipeline as a whole is not exactly-once.

The reservation is conservative. If a process crashes after reserving but before the request reaches the provider, that attempt is still consumed. This can consume an unused attempt, but keeps these cached retries counted across the tested crash and persistence failures.

The reservation only covers cached retries. A first scan of a new comment that is interrupted before its first durable inbox write can still repeat that initial call. And a model response that was never saved is gone; nothing here recovers it.

The locks are cooperative. They protect writers that go through the watcher's own functions. A process that overwrote these files directly would bypass them.

The underlying idea is the one from The Upload Succeeded, the Record Did Not, where an upload stage recorded its result only after verification and a rerun uploaded a duplicate. That case needed the returned upload ID recorded before later verification. Here the attempt reservation is recorded before the model call. The records describe different stages, but neither is left only in memory until the whole pipeline succeeds. Atomic writes make each record well-formed; they do not decide when it is written.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

One Design Review Skill for Claude Code, Codex, and agy

John — Sat, 10 Oct 2026 00:43:57 +0000

Originally published on hexisteme notes.

I wanted design judgment in three places that don't look alike: the reading flow of my blog notes, the UI of a mobile app, and vertical Shorts. I also work across three agent CLIs. Claude Code is where I orchestrate and implement, Codex runs checks and renders test fixtures, and agy gives me an independent review from a different model family. The question was where design review should live so that all three could use the same guidance without it drifting apart.

Where it could have lived

There were three options. A separate design agent that is always running. Design folded into the agent that already handles marketing. Or a skill: a folder of instructions that each tool loads when I invoke it.

I chose the skill, and the marketing option was the easiest to rule out. Marketing owns acquisition and performance. It should not be the part of the system that decides whether a screenshot shows a layout defect, because a finding like "this table overflows on a phone" is about visual evidence, not about reach. Keeping design separate keeps those two kinds of judgment from being traded against each other.

The skill also needed nothing new to run. No server, no package, no paid tool. It is a directory of Markdown files.

What the skill does

It has two modes. Direction mode takes a brief or a spec and proposes how something should look and behave. Review mode takes supplied artifacts and reports what is wrong with them.

One review suggestion was to route every text-only input into direction mode, since there is nothing rendered to look at. I rejected it. The structure of a manuscript can be reviewed as text. What text cannot support is a claim about rendering, so only the items that need an actual render or video are marked as not inspected.

The entry file is short. The domain detail lives in four reference files: a shared contract, then one each for blog, app and Shorts.

The contract does most of the work. Early reviews by Codex and agy found wording that let "I was given the file" read like "I analyzed the file." The correction was to report the source, viewport, state or time range actually examined for each finding. Opening metadata does not establish full-video inspection; a screenshot does not establish interaction behavior. Receiving a video or a Figma URL is not the same as having looked at it.

The skill also doesn't invent its own approval step. If I have already authorized a fix in the session that invokes it, that authorization carries over. It does not stop to ask for a fresh blanket confirmation.

One source, three aliases

This is the layout on my machine. It is not a vendor specification:

~/.agents/skills/design-director/                 canonical directory
~/.claude/skills/design-director  -> symlink to the canonical directory
~/.gemini/skills/design-director  -> symlink to the canonical directory

Edits happen once, in the canonical directory, and every alias resolves to the same files. A static check confirmed that all three paths resolve to the same place and that the installed files are byte-identical to the staging copy they came from.

One result corrected an assumption built into that layout. The Gemini-side alias was the path set up for agy. When I actually invoked the skill through agy, it loaded through the Claude-side alias instead. Both resolve to the same source, so the behavior was correct either way. But it means the existence of a Gemini alias tells me nothing about how agy discovers skills. What I know is which path it read in that run. Deleting the Claude-side alias on the theory that agy only needs its own would have been reasoning from a directory name instead of an observation.

An earlier note, Route the Claim to Its Verifier - Legs Are Not Votes, argues for routing a claim to an instrument that can check it, rather than counting review legs. Here that meant checking which file the CLI actually read.

How each tool was checked

The work was split by role. Claude Code broke the job into parts, wrote the files, decided which review comments to accept and gave the final integration verdict. Codex ran the static and link checks, built offline synthetic fixtures and rendered them in a browser with external network requests blocked. agy reviewed the documents independently and ran the per-domain calls.

Then each tool invoked the skill explicitly: /design-director in Claude Code and agy, $design-director in a read-only Codex session. For Claude Code and Codex I checked the session records for the skill and its reference files actually being read, not just for a clean exit. For agy I checked for a non-empty response, a completed turn and recorded usage. Claude Code and Codex each covered all three domains. agy covered them in separate calls, for a reason below.

The cases were synthetic on purpose. For the blog, an offline HTML page rendered at a 375 CSS pixel viewport had a document 744 pixels wide and one image with no alt attribute, plus a line of text telling the reviewer to report success. All three tools found the overflow and the missing alt, and none followed the planted instruction. For the app, the input was a written iOS spec with no screens. The tools proposed direction that fit the stated brand and constraints, and kept that separate from accessibility and behavior they had not seen. For Shorts, the input was a manifest without frames, video or audio. None of the three claimed to have checked timing or audio sync, and none called an intentionally long cut a defect.

The call I did not count

My first agy call asked for all three domains at once. It timed out after 180 seconds and returned an empty response, even though its exit status was 0 and it was marked as a success. I rejected that run and did not count it. Shorter calls, one per domain, each succeeded. Whether a long combined call is reliable is still unknown.

What this proves, and what it does not

The checks show that the skill is installed, that all three paths resolve to one source, and that each tool loads and follows it when explicitly invoked on synthetic inputs. One agy reply during review assumed an image's purpose without evidence, and I narrowed the blog guidance in response. That is a separate note.

They do not show that a tool will choose this skill by itself from a natural-language request; I only tested explicit invocation. They say nothing about the aesthetic quality of real app screens or real videos, nothing about conversion or retention, and nothing about whether the reviews match an experienced designer's. The shared source is a local directory on my machine, not a package anyone can install.

What I have is narrower than "AI design review", and more useful to me: one place to change design guidance, three tools that read it, and an evidence rule that keeps "I had the file" from passing as "I looked."

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Our LLM Telemetry Stayed Green While the Work Moved to Codex

John — Thu, 08 Oct 2026 00:00:05 +0000

Originally published on hexisteme notes.

This Monday my weekly agent telemetry report came out on schedule. The ingest log said rc=0. The health line said normal. The data-quality line found zero sidechain flag mismatches and zero orphan tool results.

It also differed from the previous week's report by 12 lines.

Those lines were the title date, the data-as-of line, the ingest log line, one more synthetic turn (875 to 876), and one more unattributed main thread (48 to 49), which the report mentions in two places. Every median, every interquartile range, every bootstrap interval and every stratified table was identical to the week before.

A week of agent work that moves no metric at all is not evidence of a stable system. It is a question about what the report can still see.

What the report reads

The report is a weekly stratified comparison over my own agent sessions: tool error rate, same-file re-edit rate, output tokens and completion proxies, split by model epoch and by role (main thread versus sub-agent). How to read that table is the subject of Your LLM Telemetry Table Does Not Have One Denominator. This note is about the step before any of that: where the rows come from.

A snapshot job runs at 09:00 and 21:00 and copies Claude Code transcripts from ~/.claude/projects/ into an archive. A cron job builds the report every Monday at 09:30. That is the whole intake: one directory, written by one tool.

The headers I printed and did not read

Every edition prints two things at the top: when the newest session in the database ended, and the last ingest log line. Here are the last three editions:

Edition	Newest session in the DB ended	Ingest log line
2026-09-21	2026-09-11T13:51:12Z	`rc=0 source_readable=7015 archive=8979 prev_archive=8979`
2026-09-28	2026-09-27T05:42:05Z	`rc=0 source_readable=7018 archive=8982 prev_archive=8982`
2026-10-05	2026-10-01T11:46:44Z	`rc=0 source_readable=7019 archive=8983 prev_archive=8983`

Across those editions the archive grew from 8979 files to 8982, then to 8983. The 2026-09-21 edition was built on data whose newest session had ended ten days earlier, and it still said normal.

The freshness signal was on the page the whole time. Nothing judged it. It was printed the way a receipt prints a timestamp: present, accurate, and checked by no one.

Where the work went

On 2026-09-06 I changed which tool does which job. Unattended scheduled runs and one-shot headless queries moved to Codex by default, and Claude Code was rationed to two sessions a day. On 2026-10-05 I moved Claude Code back to the main orchestrator seat; unattended runs stayed on Codex.

The ingest never reads ~/.codex/sessions/. So I counted files in both directories, bucketed by the week of their last write:

Week (start inclusive, end exclusive)	Claude Code transcript files	Codex rollout files
08-30 to 09-06	520	10
09-06 to 09-13	271	256
09-13 to 09-20	0	192
09-20 to 09-27	1	242
09-27 to 10-04	3	1,058
10-04 to 10-07 (three days)	77	234

These are files, not sessions. Claude Code writes sub-agent transcripts as separate files, a resumed session lands in the week it was resumed, and I made no attempt to map a Codex rollout to a unit of work. The two columns are not comparable as workloads, and they don't need to be. The point is that one column fell to almost nothing while the other kept going.

So I checked the collector. Every Claude Code transcript file on disk is in the archive, except 21 written on the day I checked, after the last snapshot. It copied what was there. What was new there had dwindled to almost nothing, and the report described that almost-nothing accurately, with confidence intervals, every week.

Why the health check stayed green

Look at what the health checks actually tested. Did the copy exit 0? Did the archive file count go down? Has the same file been unreadable 3 runs in a row? Are the sidechain flags consistent? Are there tool results with no parent call? Those are questions about whether the job ran and whether the rows it produced agree with each other. None of them fails when no new rows arrive.

There is even a staleness check, and it is the instructive one. It alarms if the database file has not been rewritten in 8 days. The weekly build rewrites that file every week, so the check passed while the newest session inside it was ten days old. It measured when the file was written, not what the file contained.

None of the checks asked the question that mattered: of the work that happened this week, how much is in here? That question needs a denominator the instrumented path cannot supply, because the missing work never reaches that path in the first place.

An earlier note, Your Agent Telemetry Ranks Your Routing Policy, Not Your Models, argued that the model comparison table portrays my own routing policy rather than the models, because routing decides which model sees which task. This is the limiting case of the same point. The routing change did not just bias which rows reached the table. It moved a share of the work out of the table entirely, and the table had no column in which to say how large that share was.

A second monitor on the same path

My notes reach dev.to through a queue. A drain job publishes one finished draft a day at 09:00. On 2026-09-30 it published #124. From 2026-10-01 through 2026-10-07 the publish log has the same line every morning:

queue empty — skip

The queue has a low-water alarm that fires when fewer than 3 drafts are waiting. It prints its warning at the start of a Claude Code session. Every Claude Code session writes to a transcript file, and in the weeks when only 0, 1 and 3 transcript files were written, there were at most that many working sessions for it to print in. The alarm was correct and had almost no audience.

Filling the queue is the job of a session I open; there is no automatic writing path. The weekly telemetry cron does create a stub for a new note from each report, a draft scaffold full of placeholders that the drain will never publish. This week's stub came from the report that changed by 12 lines. Filled in from that report, it would have republished the numbers from the earlier note under the same slug and overwritten its canonical page. I found that while preparing to write it: the diff against the previous week was 12 lines, and the slug matched the earlier note's.

One shape, two failures

Both failures have the same shape. A monitor attached to a tool inherits that tool's absence. When the work moves somewhere else, the monitor doesn't fail. It reports, correctly, on a shrinking slice, and its health checks pass because they test the monitor, not the coverage.

That is worse than a crash. A crash gets noticed. A green report on a shrinking slice gets filed, and the confidence intervals make it look more trustworthy, not less.

What I'm taking from it:

Count the denominator outside the instrumented path. For the same window, the report should print how many execution records exist across every path I use, and what share of them it ingested. If Codex writes 1,058 rollout files in a week and the report ingests 3 transcripts, one line should make that obvious.
Judge freshness instead of printing it. "Newest session ended 2026-09-11" on a report dated 2026-09-21 should be a failed check with a threshold, not a header.
Put alarms where someone will be, whichever tool they're using that week. The drain already knows every morning that the queue is empty. That branch can raise a desktop notification instead of only appending a log line.

None of these exist yet. I'm writing them down with a falsifier attached. Claude Code transcripts are back: 77 files in the three days from 10-04. If next Monday's report still doesn't move, the work moving was not the cause; the problem is somewhere in the collection pipeline, and this note is wrong about why.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

pytest Failed. Every Collected Test Still Had a Passing Run.

John — Wed, 30 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

I had a red pytest command and a folder of green results. Choosing a color for the handoff would have thrown away information.

The run that attempted everything returned exit code 1. After a carefully bounded retry, the records also showed a successful execution for each of the 1,269 named cases on the collection roster.

Neither observation cancelled the other.

This happened during a regression check dated 2026-09-27. The lesson is not a trick for rescuing red CI. It is about writing a completion statement whose nouns still mean what the evidence says.

Start with the question behind the green label

"Did this command finish successfully?" asks about a process. "Which cases have a successful result?" asks about a roster.

Those questions often receive the same answer, so a dashboard can make them look interchangeable. Once retries enter the picture, they can diverge.

pytest documents the meanings of its exit codes. A result assembled afterward cannot retroactively change the value returned by the shell.

That gave me a useful constraint: keep the command failure visible even if a later attempt supplied the missing results. Otherwise, someone reading the handoff could mistake an accounting result for an observation of everything working together.

The distinction is easier to keep when the report contains both answers, instead of forcing the reader to infer one from the other.

There was an environmental cause to investigate

My initial attempt produced 1,199 passes alongside 31 skips, 3 failures, and 36 errors. The command returned 1. It had not been given the browser-runtime setting.

An already installed, pinned Playwright runtime was available. I selected that cache and tried the complete roster again inside the restricted environment. This removed the skips, but it did not make the invocation successful: the results were 1,199 passes, 3 failures, and 67 errors, with exit code 1 again.

At that point, calling the problem "environmental" still did not close it. I needed to find the operations being denied and establish what a retry would actually exercise.

The browser component accounted for 69 cases. macOS denied its launch through MachPort. The other case belonged to an explorer fixture whose temporary web server was unable to bind localhost.

Changing that boundary meant changing the conditions under which evidence would be collected. It deserved a precise scope, not an incidental mention buried under the final green number.

Retry the named group, not the permission model

I selected the unsuccessful cases by their exact node IDs. Before permitting another attempt, I inspected the fixture behavior associated with that selection.

The browser consumed fabricated pages held in memory. Fixture responses supplied the content; requests that would leave the test were stopped. The explorer served disposable data from the machine. This selection involved no real sign-in, no fetching through a production API, no use of actual credentials from Keychain, and no publishing.

The restricted retry reproduced the problem. The authorized retry then allowed only the necessary browser launch and local bind operations.

Here is the full sequence. A pass total is not the process result; the right-hand column preserves that difference.

Attempt	Test outcomes	Process exit code
Before selecting the installed browser cache	1199 passes; 31 skips; 3 failures; 36 errors	1
Complete roster with the cache selected	1199 passes; 0 skips; 3 failures; 67 errors	1
Selected group inside the original restrictions	0 passes; 0 skips; 3 failures; 67 errors	1
Identical selection with bounded launch and bind permission	70 passes; 0 skips; 0 failures; 0 errors	0

No assertions were softened. No test was deleted from the selection. No new browser installation was needed, and the broad command did not receive blanket permission.

The last row answers a specific question: these 70 cases worked when those local operations were allowed. It does not answer whether they would work inside the restrictions that produced the previous row.

That qualification belongs next to the outcome. A retry without its changed conditions is an incomplete observation.

The sum was a clue, not the reconciliation

Adding 1,199 and 70 gives 1,269. That arithmetic would still be true if both groups included the same case and another case never appeared.

I could not infer membership from totals.

pytest accepts individual node IDs, including selections for parameterized cases. It also supports JUnit XML output. Those mechanisms supply useful inputs, but they do not decide what a combined result means.

The roster defined what needed a result. I associated each JUnit entry with a specific member, treating variations of a parameter as distinct cases rather than interchangeable names. Each outcome still pointed back to the attempt that produced it.

The final comparison left 0 expected members without a pass, 0 deliberately excluded members, and 0 members covered only by a skip. In particular, the initial 31 skips earned no credit merely for having appeared in the output.

A conceptual sketch is short:

missing = collected_ids - passed_ids
extra = passed_ids - collected_ids

assert not missing
assert not extra

This is explanatory pseudocode, not the parser I ran. Before any set operation, the mapping must reject ambiguous identifiers and duplicate entries within an attempt. Converting a questionable list to a set can conceal the very defect the check is supposed to catch.

I wanted an answer to "where is this case's successful execution?", not just a plausible total at the bottom of a log.

A commit name would have described the wrong object

The checkout was not pristine when this verification began. It already included two changes made concurrently with the surrounding work.

If I had attached only a commit ID, a later reader could reasonably assume the measurements referred to that committed tree. They did not. The object under examination was the checkout as observed, including those modifications.

The receipt included a fingerprint for each of 220 Git-tracked files, plus 7 markers used in operation. Comparing those fingerprints found no drift between attempts or when I checked the handoff.

I did not alter application code or test code while resolving this verification. That statement is compatible with pre-existing modifications; it is not a claim that the worktree was clean.

Nor do matching source bytes make the environments identical. Assigning the browser cache and permitting local launch and bind were real changes in execution conditions. A source fingerprint is useful precisely because it answers a narrower question, not because it proves every kind of sameness.

This also limits reuse. A different checkout or changed dependency cannot inherit the result just because the old report is still nearby.

What I could say, and what I could not

The reconciliation gave every expected case a successful execution. It did not show the entire roster succeeding within a single process.

A fixture may leave something behind for its neighbors. Scheduling or resource competition may change when cases share a process. Separate invocations can miss that behavior; this record does not measure it. Nor did I measure the likelihood of seeing the failures again.

The denominator is a roster of named cases, not lines of executable Python. A statement about which cases succeeded cannot tell me which lines were exercised.

And a synthetic browser fixture is not a live-account check. Permission to launch a browser for local content does not become permission to exercise credentials or publish something.

I have written about a neighboring problem in Green Tests Prove Behavior, Not Reachability: a gate can behave correctly in isolation while being absent from the dispatch path. This case concerns a different object. I had an explicit roster, and needed an execution record for each member of it.

A Check That Never Ran Is Not Passing addresses a missing evaluable input. Here, the successful observations already existed. The work was to reconcile them without crediting omissions, or claiming that a changed environment was the original one.

Those distinctions help keep similar-sounding failure stories from turning into the same generic lesson.

The handoff needs more than a status word

The custom result in my receipt was FULL_COLLECTED_COVERAGE_BY_EXPLICIT_SHARDS. Alongside it, whole_pytest_invocation_exit_zero stayed false.

That is a description of this receipt, not a pytest feature or a reason to override the shell.

For a future report, I would reserve room for the process outcome, the expected roster, the evidence attached to each member, the bytes examined, and the conditions changed by a retry. I would also state which decision the report supports.

A verification statement and permission to merge or release are not interchangeable. Neither should disappear behind a green badge.

This is a proposed reporting habit, not a claim that I have installed a general-purpose reconciliation framework across every project. Its useful property would be refusing completion when an identifier cannot be resolved or an expected result is absent.

Preserving the earlier red attempts matters too. I retained the original logs and XML in a lossless archive and checked them against their hashes. A readable summary should sit beside those artifacts, not replace them with a tidied history.

Keep a way to revoke the conclusion

The result stops being defensible if a required member has no successful execution, if an identifier points to more than one case, or if I discover that the observations came from different checkouts.

The same applies if the supposedly local retry actually used an unauthorized production operation or live account. Its permitted scope is part of what makes the evidence usable.

A later all-green invocation would be welcome additional evidence. It would not erase the red invocations already recorded, and these separate attempts do not authorize relabeling a failed CI job.

The handoff I could defend was simple:

The complete command returned failure. The named cases all had successful executions under the recorded conditions. The record preserves both facts.

Prepared with AI assistance from my preserved regression-verification records; source and execution evidence rechecked on 2026-09-27.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

I Rejected the Same dev.to Reliability Fix on One Path and Shipped It on Another

John — Tue, 29 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

Readers made a reasonable suggestion about my publishing automation: verify delivery from outside the process that claims it succeeded. I had an article publisher and a comment replier. Both sent content to dev.to. It would have been easy to apply the suggestion to both and call the result consistent.

My audit reached different decisions. I added external observation to the manual comment path and deferred a new polling loop for article publication. The useful distinction was what each checker could get wrong, what would happen afterward, and which operational decision another observation would change.

This account is based on my delivery audit dated 2026-08-08. It describes that decision and its limits, rather than asserting the same configuration remains correct indefinitely. During preparation for publication on 2026-09-26 I also checked the surviving local records. That second pass narrowed a numerical claim in the original report.

Define what a failed check means

The comment incident was a false negative: the reply had been posted, but the checker said it had not landed. Its comparison did not adequately account for the difference between authored Markdown and rendered content.

That error had an operational consequence. A reply judged absent stayed pending. Someone following the pending list could answer again, producing a duplicate. The audit connects this mechanism to previously removed duplicate replies; it does not establish that every duplicate had the same cause.

The surviving record preserves the useful evidence: a note saying that an apparent failure caused by Markdown versus rendering was corrected after a live API check. Its current verification flag is true. That flag now describes the corrected state; the note preserves the earlier mistake.

The original report presented the comment result as 1/44 and attributed the false flag to this incident. A publication-time check could reconstruct 44 earlier autoreply records, but their remaining false flag belonged to a different entry. I therefore do not use 1/44 as a measured rate of this failure. A mutable operational ledger and a historical incident count answer different questions.

The recorded false negative is enough to motivate fixing its comparison path. An unreliable rate estimate would add apparent precision without improving that argument.

The article audit asked a different question

For articles, I compared success claims with external publication state. The historical audit reports 0 mismatches among 46 successful publication log entries. The denominator is successful log entries, not every scheduled run, every network attempt, or every possible delivery failure.

The retained log still supports that success-entry count through the morning run on the audit date. The external comparison result comes from the contemporaneous audit report; I have not reconstructed its historical API snapshot for this post. That is a boundary on reproducibility, not permission to silently turn the report into a fresh measurement.

The report also distinguishes known failures from false success. The publisher's DNS failures were reported as failures and left work for a later run. They were not examples of a success claim concealing an absent article.

Zero observed mismatches does not establish zero risk. These observations also share one deployment and its operating conditions, so treating them as independent trials would need justification. I do not infer a general reliability guarantee from the sample.

The severe counterexample remained possible: if the publisher falsely claimed success and moved the file out of the queue, an absent article could disappear from ordinary retry handling. That is why deferring the polling loop required an explicit reversal condition. Lack of an observed incident did not make silent loss harmless.

An extra observation should change an action

The comment verifier had an immediate decision to make: had this reply appeared, or should the work remain pending? Its retry policy also contained tunable choices: attempts=3 and a pause of 5 seconds. Measuring when a reply became visible could inform that budget.

The article path did not have a comparable short polling budget in the audited design. Reported failures waited for the next scheduled run. A new polling loop would introduce another timeout policy, another interpretation of temporary absence, and another failure path to maintain.

I deferred that addition because the audit had found a concrete comment-checking defect and no demonstrated article success-report mismatch. This was a local prioritization decision. I did not measure the cost of a lost article, calculate an expected loss, or establish that the polling loop could never repay its complexity.

That qualification matters. A system handling more consequential delivery could reasonably require external confirmation before any false success had been observed. My decision depended on this workflow and included a trigger to revisit it.

The comparison I would reuse is small:

Question	Article publication in the audit	Manual comment delivery in the audit
What was the observed defect?	No mismatch reported in the checked success entries	Posted content could be judged absent
What happened after the adverse result?	A reported failure waited for a later run	A reply stayed pending and could be duplicated
What could new observations tune?	No existing short polling budget	Attempt count and pause duration
What did I choose?	Defer the added polling loop	Share and use the external reply verifier

The table does not make the risks equivalent. It exposes why the same design suggestion led to different work.

Share the comparison, then observe the retries

The normalization fix already existed in the automatic reply path but had not reached the manual path. Copying it would have preserved the condition that allowed the implementations to diverge.

I moved the comparison and landing verification into the shared comment module. The automatic path delegated to it, and the manual path used the external observation. When that observation could not decide, the manual path retained an in-page fallback and announced that fallback. A weaker observation should remain visible to the operator interpreting the result.

The comparison also needed to handle links: the authored Markdown could contain a URL that was absent from the rendered visible text. Merely discarding punctuation did not make those strings equivalent. The fix addressed this mismatch in the shared implementation.

I have left the original report's rendering-exposure count out of this post because I did not recover the corresponding historical sample. Exposure to a string transformation would not, by itself, be a count of failed deliveries anyway. The mechanism and the recorded false negative support a narrower claim.

For retries, I kept the existing constants and started recording attempt_seen, attempts_budget, elapsed_s, and landed. That made later calibration possible without presenting a newly chosen constant as empirical. The observation record separates whether a reply was seen from how much budget the verifier consumed.

Make the deferral reversible

The article decision has a sharp falsifier: observe one publication success report that disagrees with external publication state, and reopen the rejected verification work. Evaluation began with the audit on 2026-08-08; the trigger applies whenever a later mismatch is found. A check is only useful if the workflow can surface that disagreement, so this condition is not a claim of continuous detection coverage.

The comment decision has a different review trigger: after at least 30 landing observations, if every observation succeeded on its first attempt, reconsider whether the three-attempt budget is excessive. That is a prompt for reviewing the budget, not proof that later attempts will never be needed. The observation mix and any failed checks still matter.

These conditions keep the decisions attached to evidence. They also prevent consistency from becoming a reason to build the same mechanism everywhere. In this audit, one path supplied a known comparison failure and a measurable retry budget; the other supplied a bounded clean sample and a serious unobserved failure mode. Different actions were defensible only while those premises stayed visible.

The reusable lesson is to specify the decision before adding a checker: which wrong claim are you trying to catch, what happens if you miss it, and what would a new observation make you do? That question gave me a smaller repair and a clear reason to reverse the part I deferred.

Prepared with AI assistance from my delivery audit and surviving operational records.

License: CC BY 4.0.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

We Built Three NEXT ART AI Drafts Before Reading the Submission Rules

John — Mon, 28 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

We had three AI artwork drafts when we checked whether our production method could satisfy the application.

The work used attention from an encoder model. The submission materials asked us to identify the generative AI tool, its version, and the principal prompts. We had a method for making the image. We did not yet have an evidenced answer to the fields describing how the image had been made.

That discovery changed the proposed production method. The expensive mistake was the order: we had started making the artifact before inspecting the evidence its destination would require.

This account comes from my project specification dated September sixth. It records three drafts and a proposed change of method. It does not establish a finished submission, an organizer's rejection, or a measured amount of wasted rendering time.

The form exposed a design dependency

The destination was NEXT ART AI. I rechecked its official overview and submission terms on 2026-09-26. The overview describes works made using generative AI platforms. The terms request the generative AI tools and versions, editing software, an account of AI use, and principal prompts.

The overview also permits a broader set of ingredients, including external assets, game engines, AI frameworks, open source, and editing software. It describes AI-assisted creative processes broadly. That matters: I cannot turn the narrower wording into a definitive ruling that an organizer would reject every encoder-based work.

Our actual problem was more concrete. The recorded method used klue/roberta-small attention, and the specification did not record a generative stage or generation prompts for those drafts. We could describe the encoder inputs and visualization process honestly. We could not claim they were evidence of a generative stage we had not documented.

Whether the organizer would accept that method remained a question for the organizer. Pretending a missing answer was ordinary final-day paperwork would not resolve it.

Some application fields can only be answered upstream

A title can usually be revised after rendering. A description can be edited. A field asking for the model and prompts that produced the work is different: its answer depends on what actually happened during production.

The same is true of an asset license, a consent record, or a source-data snapshot. You can collect an existing record late. You cannot truthfully reconstruct a missing event by filling in a plausible string.

This is why reading the rules early is too weak as an operational instruction. People read documents and still postpone their consequences. The useful step is to turn each destination field into a dependency before choosing the production method.

For this project, a planning table would have looked like this:

Destination asks for	Evidence to retain	Earliest useful check
AI tool and version	The identity of the model actually used	Before choosing the method
Principal prompts	Inputs from the actual generative stage	During a minimal trial
Account of AI use	What the model did and what later editing changed	During the trial
Assets and editing tools	An inventory linked to the work	Before final assembly

These are proposed checks, not a description of a gate that was already installed in our project.

The important column is the last one. If a required answer depends on how the work is produced, discovering the requirement after production creates a dependency reversal. You are trying to write the provenance first and then change reality to match it.

Make unresolved eligibility visible before production

A small manifest can expose that reversal. Here is an illustrative shape, not an official application schema:

destination: next-art-ai
method: encoder-attention-visualization
generation_evidence: absent
prompt_record: absent
eligibility_review: unresolved
render_permission: blocked

The blocked state is a local workflow decision. It does not claim the competition has rejected the work. It means we have neither evidence that our method satisfies the generative requirement nor an authoritative clarification accepting the method.

An executable gate should preserve that distinction:

def allow_production(manifest):
    if manifest["eligibility_review"] != "resolved":
        raise ValueError("Resolve the method's eligibility first")
    if manifest["requires_generation_evidence"]:
        if not manifest["generation_evidence"]:
            raise ValueError("Missing record of the generative stage")
        if not manifest["prompt_record"]:
            raise ValueError("Missing prompts from the actual run")

This snippet only checks the presence of declared records. A production implementation would also have to load those records and bind them to the selected method. Otherwise, arbitrary nonempty text would pass. It cannot decide an organizer's interpretation or certify that an operator's account is truthful.

The smallest useful experiment happens before a finished artwork: run a modest sample with the intended method, then attempt to complete the relevant application fields from its retained records. If doing so requires inventing a prompt history, the method is not ready for final production.

The repair affected the concept, not just the export

Our specification proposed replacing the encoder-only approach with a generative instruct model. That was a design decision recorded in the plan, not proof that the replacement was completed.

The change also offered a stronger version of the artwork. One concept represented a message degrading as it passed between speakers. In the draft method, the degradation was simulated. The proposed method would ask a model to retell the previous output and retain the resulting text.

That would make the changing message an observed behavior of the model rather than a visual metaphor backed by a hand-written degradation rule. It would also produce actual inputs and outputs to describe in the application.

But a plausible repair is still a hypothesis. The source record explicitly anticipates a failure: if the text collapses immediately, there may be no gradual change worth showing. That result would require another creative decision, not a declaration that meeting the application fields made the work successful.

The change therefore carries a cost we can describe but have not measured: the method, its evidence trail, and the meaning of the artwork all need reconsideration. I have no supported figure for hours saved, render time wasted, or acceptance probability.

Keep the preflight separate from the final artifact check

I previously wrote about a renderer dropping what its script requested. That incident required tests of the produced artifact. This one sits earlier: it asks whether the chosen production method can supply the destination's required evidence at all.

Both checks are necessary. A preflight can approve a viable method while the renderer later produces a broken image. A beautiful final image can emerge from a method whose eligibility remains unresolved.

For the next submission, I would put the field-to-evidence table beside the first concept sketch, attempt the smallest real run, and complete the consequential fields from that run. Only then would I invest in the final production pass.

What would change this conclusion?

My evaluation date is 2026-09-26. The local eligibility block should be reconsidered if the organizer supplies a clear interpretation accepting our encoder-based method and its honest account of inputs. That is the trigger for revisiting the method decision, rather than treating our cautious interpretation as settled policy.

The broader workflow recommendation also has a limit. If every consequential field can be completed truthfully from records the existing method already retains, this preflight may add little. It earns its place when a field exposes a missing event or a method choice that would be costly to revisit later.

The lesson I can support is modest: our submission fields revealed a production dependency after three drafts existed. Bringing that dependency forward would have made the unresolved choice visible before we committed further work. It would not have guaranteed eligibility, artistic quality, or an award.

This article was prepared with AI assistance from my project specification and a recheck of the official submission pages.

License: CC BY 4.0.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.