Eight Frames Said Fail. Twenty-Four Said Pass.

Originally published on hexisteme notes.

On 2026-08-14, a screen-metric experiment in my YouTube Shorts pipeline ended undeterminable. Not because the metric misbehaved, but because a threshold I had pre-registered depended on a number I had never pinned down. Once I fixed that, a second, quieter version of the same mistake was already waiting one layer downstream — inside the rule I wrote to fix the first one.

What the threshold couldn't decide

The pre-registration set a single bar: motion median has to be at least 50% of the original for a clip to pass. It said nothing about how many frames that median gets computed from.

Measured across 8 frames, the clip's motion median came out at 47.5% — below the bar, falsifier fires, clip fails. Measured across 24, it came out at 56.5% — above the bar, falsifier doesn't fire, clip passes. Same clip, same threshold, same metric definition — only a sampling parameter the pre-registration never fixed, and that alone flipped the verdict's sign.

A threshold with an unpinned measurement isn't wrong, it's undefined: pass on one run, fail on the next, both honestly computed. (This isn't the underpowered-sample problem from Perfect Instrumentation Won't Save an Underpowered Experiment — that piece has a real, fixed measurement and not enough of it; this one never had a fixed measurement at all.)

The rule I wrote right there in the retro

I wrote the fix into that experiment's retro (report/simreveal-axisB-official-render-20260814.md, §4-3), on the spot. (The passages quoted in this section are translated from the Korean original.)

Rule starting with the next pre-registration: a rate prediction has to pin its
measurement parameters — sample count, window, sampling rule — alongside its
threshold. A threshold without a pinned measurement can't adjudicate anything.

Same commit, next pre-registration document (report/prereg-simreveal-axisB-replication-montecarlo-20260815.md). Its §2 followed the rule exactly: window fixed to [0, 5.5s], sample count committed to report both 8 and 24, sampling rule spelled out. It said so directly:

The reason axis B ended undeterminable on motion median on 08-14 was that the prediction fixed only the threshold (50%) and never fixed the sample count (§4-3). Enacting that lesson here for the first time.

Less than a day after I wrote the rule, the person who wrote it followed it. So far, a pipeline working exactly as designed.

Counting the corpus, a day later

On 2026-08-15 I was doing unrelated work — an audit forcing machine-readable front matter onto every pre-registration document — and for the first time counted the whole corpus instead of one document. report/prereg-*.md came to 3 files. Only 1, the Monte Carlo replication above, had pinned the rule's three required slots (sample count, measurement window, sampling rule) in prose at registration.

1/3. I read that as "the rule didn't survive 24 hours" and wrote it into the retro — a cousin of an older counting mistake, The Count Read in the Thousands. It Was Thirty-Seven Things, Recounted., though that total was real and just uninterpreted. This one was about to be flat-out wrong.

That reading was wrong

I wrote the sentence before checking the thing I should have checked before writing it. Then I checked when the other two documents were actually born:

$ git log --diff-filter=A --format='%h %ad' --date=short -- report/prereg-*.md
03da812 2026-08-15   # the commit where the rule was written = the montecarlo replication's birth commit
670cdff 2026-08-14   # ink-coverage
b3478d8 2026-08-14   # sim-graph-reveal

Both of the other two were created before the rule existed. They were never in scope. Counting them as violations meant counting retroactive non-application as non-compliance. The correct denominator isn't 3, it's 1 — compliance rate 1/1, 100%.

Two wrong readings, one missing definition

The lesson isn't "the rule died." It's that the same data got read two opposite ways, on the same day, by the same person:

reading	fraction	conclusion	why it's wrong
count the numerator alone	1 compliant case	the rule is alive	only one case was ever put to the test
extend the denominator back past the rule's birth	1/3	the rule is dead	it puts out-of-scope documents in the denominator

Both are wrong, for the same reason: the denominator was never defined.

(A different flavor of the same word: The Gate Only Logged When It Fired. I Replayed 116,022 Candidate Stop Points to Find the Rest. is about a denominator never written down anywhere, reconstructed from retained logs; this one was never missing, just scoped wrong.)

The corrected 1/1 = 100% isn't comforting either — maximum reassurance and minimum evidence at once, since the rule has only ever been tested once. That one test case had already declared, in its own text, that it was "enacting this lesson here for the first time," so the story of the rule working was complete inside a single document, written, followed, and later re-read by the same person. "Wrote the rule" and "followed the rule" described one data point, not the whole corpus.

One clause at the outset would have prevented all three readings: had the fraction been defined the moment the rule was written, it would have read 1/1 from day one, and the denominator being 1 would have been the signal itself — don't adjudicate yet. The discovery wasn't a designed audit either, just luck: the unrelated front-matter task happened to require counting the corpus start to finish.

The fix, and the same failure inside the fix

I moved the rule out of prose into something machine-checkable: schema: prereg/v1 front matter on line 1 of every pre-registration document, plus a checker (shorts-factory prereg, in src/shorts_factory/experiment_prereg.py) scanning the corpus for missing slots, unevidenced exemptions, and undeclared amendments. I transcribed front matter retroactively into all 3 documents; where a slot hadn't been pinned at registration time, I left it null — backfilling with a post-hoc value would have been a lie about what was pinned when.

That fix reproduced the exact failure it was fixing. Nothing calls the checker: no git hook, no CI workflow, no scheduler in the repository runs this subcommand.

# the checker exists
.venv/bin/python -m shorts_factory prereg   # or: agent-youtube prereg

# but nothing wired to the repo calls it (confirmed 2026-08-15)
grep -rl "prereg" .claude/hooks/ .codex/hooks.json   # no results

The thing I built to fix "a rule written in prose gets forgotten" landed exactly where it can be forgotten, for the same reason: nobody calls it automatically. I nearly got that claim wrong too — counting "automation surfaces," I almost stopped after git hooks and CI. Actually counting found 4 active hooks (session_start_state.sh, pre_upload_channel_check.sh, post_render_verify.sh, post_edit_module_tests.sh) plus SessionStart/PreToolUse/PostToolUse registrations in .codex/hooks.json — none call this checker. "There's no automation" isn't a sentence I get to write without naming what I checked.

What the same command prints today

This is where the essay's own honesty gets tested. Running the same commands against the repository as of 2026-08-15, the day of the audit:

$ ls report/prereg-*.md | wc -l
3
$ grep -c '^measurement_spec:' report/prereg-*.md
report/prereg-ink-coverage-20260814.md:1
report/prereg-sim-graph-reveal-20260814.md:1
report/prereg-simreveal-axisB-replication-montecarlo-20260815.md:1

All 3 documents now carry measurement_spec:. That is not "the rule recovered to 3/3" — it's the value after the retroactive transcription above, a product of the audit work, not of the rule being followed; the field didn't exist before that work ran. "1/3" was counted before the transcription, while the rule existed only in prose; "3/3" is counted after. The two can't answer the same question — "is the rule being followed?" — because having the slot doesn't mean compliance. It means missing slots are now visible as null instead of silently absent (sim-graph-reveal's sample_count still is).

A follow-up check, dated 2026-08-31: the corpus gained one document since that audit. report/prereg-publish-window-20260818.md was born in commit c12f93f on 2026-08-19 — four days after the rule, so it belongs in the denominator. Its front matter pins all three required slots (sample_count, measurement_window, sampling_rule) at registration, plus three amendments, each with a dated, non-silent what: entry. That makes the corrected rate 2/2, not 1/1 — still short of the 3-or-more this piece's second falsifier actually asks for, but the first new data point since the rule's birth, and it points toward compliance.

What generalizes

• Cite a rule's health as a fraction, not the last compliant instance: outputs applied to, divided by outputs applicable to. A rule whose denominator can't be enumerated shipped without anyone observing it.
• The denominator's boundary is the rule's birth — outputs made earlier don't belong in it, or retroactive non-application reads as violation (my mistake here). Recoverable with git log --diff-filter=A or an equivalent creation check.
• A denominator must be what the author marked, not what files exist, to be countable. Here, the prereg-*.md filename convention was that mark.
• If the denominator is 1 or 2, don't adjudicate — write "not enough signal yet." n/n = 100% is maximally reassuring and near-zero information at once; keeping the denominator in view when citing a fraction removes most of this trap.
• Building an enforcement mechanism and having enforcement run are different pieces of work. Stopping at the checker just adds a command a human must remember. If you can't name a caller — hook, CI job, schedule — don't write "closed."
• Count every automation surface before asserting automation is absent: git hooks, CI, agent-specific hook registries (.claude/hooks/, .codex/hooks.json, or equivalents), schedulers.

Falsifier

This piece's claim is not "rules get forgotten." That conclusion isn't supported by this incident's evidence — I wrote it into an early draft and falsified it myself, above, where the 1/3 reading turned out wrong. The rule got one chance to apply, and that chance was followed.

What's left is narrower: a compliance rate with an undefined denominator reads in whichever direction you're primed to read, and an n=1 compliance rate reassures without informing. The sample here is one rule, one repository, one observation.

Two ways this could be shown wrong:

• If counting compliant instances alone — without a defined denominator — ever correctly judges a rule's health, the prescription above is overkill.
• This incident's real test hasn't happened yet: does the compliance rate hold once 3 or more further rule-applicable outputs pile up after the rule's birth? One has arrived so far (above), still compliant — short of the bar, but pointed the right way. If the rate drops before that bar is reached, "a rule doesn't survive without enforcement" finally has evidence in this repository. Until then, I'm not entitled to write that sentence.

More notes at hexisteme.github.io/notes.