I run a small fleet of open source business tools, mostly solo, with AI subagents doing a lot of the grunt work: reading across repos, drafting docs, cross-checking claims against the actual code.
For a while, my problem with that setup wasn't wrong code. It was confident prose.
An agent would write "this was standardized in the framework" — and it sounded right, and the framework did have something like it, and if I hadn't gone and looked, I would have shipped a sentence that was simply not true. Not malicious. Just the model smoothing a gap with something plausible.
So I added one paragraph to the instructions every subagent gets.
The paragraph
It's not clever. It's a standing rule, in every task:
- Cite it or drop it. Every claim comes from a named source — a file, a PR, a line. If you can't source it, don't assert it; skip it and say you skipped it.
- Don't link to anchors you haven't confirmed exist.
- Report what you couldn't do. A skipped step is information, not a failure to hide.
-
Don't touch unrelated changes. Check
git statusbefore you start.
That's it. No framework, no eval harness. A short list that turns "sound confident" into "show the receipt."
What the agents did with it
The interesting part is that the agents started policing themselves.
In one run, an agent was documenting a feature and was about to reuse a phrasing from a sibling repo — "synchronization was made standard here." It checked, found the sibling had no such thing, and left the claim out on its own, with a note that it couldn't source it.
In another, it wanted to deep-link to a named section that didn't exist. Instead of inventing the anchor, it fell back to a file-level link and flagged it.
And the one I keep thinking about: an agent was handed an instruction that said a piece of work was "Phase 2, in progress." It went to check the current state, found that phase had already shipped days earlier, and corrected the instruction I gave it.
That last one matters. The guard wasn't just stopping the agent from making things up. It was catching my stale assumptions, because "verify against the record" cuts both ways.
Then it caught me for real
Here's the part that earned the whole thing.
I had an agent draft a technical article — a failure story about a bug on shared hosting. Every code snippet was correct. Every default value, every config detail, checked out against the repo. It was, technically, flawless.
The framing was fiction.
The article told it as a lived production incident: "I shipped the in-memory version, tests were green, and in production it never fired." Compelling. Also something that never happened. The git history was unambiguous — that component had been file-backed from its very first commit. The in-memory version was never deployed. The "production incident" I was about to narrate in the first person had no trace in the code or the records.
A guarded review caught it before it went out. I run new drafts through an adversarial pass — a few reviewer personas plus a hard cross-check against the actual repo and my own work logs — and the verdict was unanimous: the technical content is true, the story is invented. The fix was to demote the fake incident to what actually happened — a design decision ("shared hosting can't do in-memory counters, so it was file-backed from day one") — and keep every correct code block.
Then I checked the sibling articles in the same series. Two more had the same shape: real bugs, but dressed up as production incidents that were actually caught in a staging rehearsal. One was clean — a genuine outage, logged and dated. So I fixed the two, left the real one alone, and moved on.
The lesson isn't "trust the AI more"
It's the opposite. The guard works because it doesn't rely on trust. It makes "cite it or drop it" the path of least resistance, so the cheap move is also the honest one.
And it reframed what I was even guarding against. I started out worried about agents inventing APIs. The failure mode that actually threatened me was subtler: prose that is factually correct at the code level and fictional at the story level. A test suite can't catch that. A linter can't catch that. Only checking the narrative against the record does — the same move the guard already required for facts, applied one level up.
So the honesty guard grew a second clause, for anything written in the first person: the story has to be verifiable too, not just the code. "This broke in production" is a factual claim. It needs a log, the same way a benchmark needs a number.
What I can and can't claim
I'm not going to tell you fabrications went to zero forever — I have no counter for that, and the honest evidence cuts against the clean version anyway. What I can say is narrower and, I think, more useful:
- In the runs where the guard was in the prompt, the agents stopped filling gaps with plausible-but-unsourced claims, and started reporting the gaps instead.
- When a fabrication did reach a draft, a guarded review — personas plus a record cross-check — caught it before publish.
Two different mechanisms, same principle: make the receipt mandatory, at the fact level and the story level.
None of this is a framework I'm selling. It's four sentences in a prompt and a habit of checking the narrative, not just the numbers.
What's in your AI agents' instructions to stop them from making things up — and does it cover the story, or only the facts?
── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com
Top comments (21)
"The fabrication it caught was mine" is the honest core of this, and it's a sharper result than a guard catching an agent would have been. An agent hallucinating is a bug. An author framing a design choice as a lived incident is the thing that actually erodes trust on this platform, and it's much harder to police because it isn't false in any single fact — the code was real, the reasoning was real, only the "and then it broke in production" was invented. Your guard caught the one layer where fiction is invisible: true parts assembled into an untrue whole.
I write about my own systems here too, and your fact-level-plus-story-level split named something I'd been doing only halfway. I'm careful that every technical claim is real, because those are checkable and someone will check them. The narrative around them got no such scrutiny, and narrative is exactly where the embellishment goes — "for weeks" when it was days, "I finally realized" when someone told me, a clean arc imposed on a messier sequence. None of those are false facts. All of them are the same move as your in-memory incident, one notch quieter.
The uncomfortable part of your fix is that git could adjudicate yours. My stories are about judgment and timing, which leave no commit to check against, so the only witness is my own memory — and memory edits toward the better story on its own, without me deciding to lie. What I've landed on is writing the messy version first, before I know what the point is, because the clean framing is precisely the thing that gets added in the retelling. Your guard makes the receipt mandatory; for the story level, the receipt might have to be the draft that predates the moral.
This landed the day after I caught the same move going the other direction, which was uncomfortable timing.
I had a piece about to go out on a ×100 currency bug: a field named
amountCentsthat holds whole yen, because JPY has no minor unit. It came out of the setup this post describes — agents draft and cross-check, the calls are mine. The draft explained the bug like this: "if the only thing you remember is store money in minor units, you'd be tempted to write* 100."Not one false fact in that sentence. Also invented.
I stopped on it because I couldn't picture a Japanese developer being tempted. The absence of a sub-yen unit isn't knowledge here, it's reflex. The git log settled the rest: the line had arrived on an AI co-authored commit. The cause was never a half-remembered rule — it was an identifier name, read by a writer who had nothing but the name to go on.
So the guard caught my invented incident, and the next day I caught the draft's invented motive. Same class of error in both directions: true parts, untrue whole, nothing false to point at. And in neither case did the fact-level check help. Every fact was fine.
Your "write the messy version before you know what the point is" is the piece I don't have a mechanism for. I keep raw session logs across my repos, but for a dull reason: my agents lose context between sessions, and the logs are how the next one catches up. It hadn't occurred to me that they are also evidence against my own retelling — written before the moral exists, and therefore unable to bend toward it. That might be the closest thing to a receipt the story level can get.
The amountCents case is worse than a documentation bug, and I think it's a category we don't have a name for. Docs can be wrong; a wrong identifier is a claim that travels with the value everywhere it goes, and it gets re-read by every subsequent writer — human or model — as the most authoritative available statement about what's inside. Your draft's fabricated motive wasn't sloppiness, it was the only inference available to a writer holding nothing but the name. And the fix asymmetry matters: a bad doc gets corrected in place, a bad name needs a refactor, so it keeps misinforming everyone in the interval. The reflex point is what makes it airtight — a Japanese developer wouldn't reach for
* 100because there's nothing to reach for, so the explanation could only have come from someone reading the field name in a vacuum.Your session-log observation is the part I'd generalise hardest, because it names a property I hadn't isolated: the logs work as story-level evidence because they weren't kept for that. A record created to prove something is weak evidence, since the author already knows what it will be used to establish and shapes it accordingly, however honestly. A record created for an unrelated reason can't bend toward a conclusion that didn't exist yet. So the best receipts are the ones nobody built as receipts — which is an uncomfortable design constraint, because it means you can't deliberately manufacture the thing you most need.
I have the accidental version and not the deliberate one either. I keep working notes so the next session can pick up where the last stopped — same dull reason as yours. And they've twice served as evidence against my own memory: this week I found a count in my own index that had been wrong for weeks, and the only reason I could reconstruct when it drifted was an old backup written for disaster recovery, not for auditing me. It settled a question I would otherwise have answered from memory, confidently, in whichever direction made me look better. So: no mechanism here either. Just the observation that the logs I keep for the least interesting reason are the only ones that can't be edited by hindsight.
The disaster-recovery backup example maps onto something I ended up building, and your framing explains why it works where my first attempt didn't.
I used to keep a hand-written table of which of my articles were live on which platform. It rotted within weeks — at one point it still called a piece a draft after the piece had gone live. The interesting part is why it rotted: it was a record of state, kept by the same person who changed the state. Every update was me telling myself what I already believed. Nothing in it was capable of contradicting me.
The fix wasn't a better table. It was demoting my repo to a claim: a script now asks each platform's public API what is actually live and reconciles my files against that. It caught a fresh drift today. And by your constraint, the reason it works is exactly that the platform's record wasn't built to audit me — it exists to serve pages to readers. I can't bend it toward my version of events because I don't write it.
Which also closes the loop to your identifier point, I think. A published: false flag in a file's frontmatter is the same category as amountCents: a claim that travels with the artifact and gets re-read by every later session — human or model — as the most authoritative statement about it. My own tooling trusted the flag over reality until the reconciliation script existed.
So the pattern seems to compress into something almost usable as a rule: any record whose author also authors the events will drift toward the author's version — honestly, with no decision to lie anywhere in the chain. The stable witnesses I've found are all records written by a party with no stake in my story: the platform's API, the git log, the session notes written before the moral existed. None of them were built as receipts. All of them are the only receipts I have.
The rule compresses well, and I'd add one seam inside it: git log isn't a single witness, it's two records braided together. The commit's existence, its timestamp, and its diff are authored by the tool — I can't move them without leaving traces I'd have to also author. The commit message is authored by me, at the moment I'm forming the story, which makes it exactly the kind of record your rule warns about. So the same artifact holds an unbendable witness and a self-serving narrative in adjacent columns, and I've caught myself citing the message when the diff was right there disagreeing with it.
The limit I'd put on external witnesses is that they only testify within their own jurisdiction, and the question I'm actually asking is usually wider. Your API answers "what is live" definitively — it cannot answer "is what's live the version I meant." An old build serving happily while a newer draft sits local satisfies both the API and the flag; nothing in the reconciliation notices, because the discrepancy isn't in the state it models. Which means picking the right witness is only half the work; the other half is checking that the witness's question and mine are the same question. Otherwise you get an accurate record answering something adjacent, which is more dangerous than no record because it feels settled.
Mine had that exact defect and I only noticed it this week. My catalog consistency checker compares a table against two other surfaces — and all three are files I wrote. There's no external party anywhere in that comparison; it's my claim checked against my other claims, which by your rule can only ever converge on my version. The genuine outside witness available to me is the filesystem: how many program directories actually exist. But that one has a jurisdiction problem too — a directory proves something was created, not that it's still operated, and my table's real claim is about the latter. So the honest state is that I have no single witness for the question I'm asking, only two partial ones that disagree in different directions. Naming that is further than I was a week ago, when I thought I had a checker.
Your jurisdiction question got tested against my own stack within hours of my last comment, and it went against me.
An article of mine went live on Zenn today. The reconciliation script did its job: the platform's API said live, my repo got stamped with the URL, everything converged. Then a human looked at the page. One of the embedded Mermaid diagrams was rendering as a syntax-error box. The platform renders diagrams with mermaid 11.14.0; I'd validated against a newer parser, which accepts what 11.14 rejects — unquoted Japanese axis labels in one chart type. So the witness answered "is it live" truthfully, while the question I actually cared about — "is what's live what I meant" — sat outside its jurisdiction, exactly as you said. And your sharper point held too: the green reconciliation made it feel settled, and nobody re-asks a settled question. The break was found by a person, not a process.
The fix wasn't widening the witness's jurisdiction — I can't make a publication API testify about rendering. It was adding a second witness whose question matches: a pre-publish check that parses every diagram with the platform's ed, not whatever happens to be
The fix wasn't widening the witness's jurisdiction — I can't make a publication API testify about rendering. It was adding a second s: a pre-publish check that parsesevery diagram with the platform's actual renderer version, pinned, not whatever happens to be current. Which suggests a corollary to your half-and-half split: when the witness's question and mine differ, the cure isn't a better witness. It's an honest inventory of which of my questions have no witness yet — your "two partials that disagree in different directions" is already ahead of most setups, which have one witnes
On the braid: I hit the message-over-diff version of that in my own ops notes this week. I'd inferred a tidy "the posting limit two data points and wrote it downin the confident column; a measured retry at hour 55.9 falsified it. The rule that survived is now
in the runbook: estimates and measnces, labeled as which one they are. Same seam you're describing — the narrative column and the data column share a file, and the narrative one is always mine.
The rendering case is the cleanest possible test of it, and the detail I'd keep is that your reconciliation was working correctly the entire time. Nothing malfunctioned. A truthful answer to "is it live" was delivered, and the delivery is what closed the question — the green didn't hide the broken diagram, it removed the reason to look at the page. That's the expensive part of a witness answering something adjacent: it doesn't just fail to cover your question, it retires it.
On your corollary — inventory the questions with no witness — I'd flag that the inventory is itself a record you author about events you author, so your own rule applies to it. And it drifts in a predictable direction: an entry saying "this question has no witness" is uncomfortable, unactionable on any given day, and quietly stops being re-read. Mine did. What made a similar list hold was giving each entry a date and a condition that expires it, so an unwitnessed question comes back and asks again instead of aging into furniture. Otherwise the inventory becomes the same thing as your hand-written table: an honest record of what I believed when I wrote it.
I have your gap and haven't closed it. For anything I publish I verify that the canonical URL returns 200 and that the expected asset is referenced — jurisdiction: reachability. I have never verified that the live page renders what I meant. Different markdown parser, different emoji handling, a code fence that behaves differently than in local preview: all of it sits exactly where your mermaid did, outside what my check can testify about, with a green beside it. Your fix is the right shape and the pinning is the part I'd have skipped — validating against "current" is how you end up certifying a page the platform will never produce. Estimates and measurements labeled as which they are, and now witnesses labeled by jurisdiction. Both are the same move: stop letting a record imply more than it observed.
Your furniture line found its target within the hour. I keep exactly such an inventory — a shared issues list across my repos — and rereading it after your comment: the entries that stayed alive have a date and a firing condition, and the ones that aged into furniture are precisely the undated ones. One entry I wrote yesterday is event-expired rather than calendar-expired: it names the single observable event that upgrades it from note to work item. I think that's the property that matters — a condition attaches the entry to something that will happen in the world, so the world re-opens the question; a date only attaches it to my own diligence.
The other thing that keeps lists read here is dull: a nightly job compiles the open entries into the day's report, so the inventory gets pushed back in front of us instead of waiting to be visited. The re-reading is done by a machine that doesn't find the entries uncomfortable. It's not a witness — it has no jurisdiction over whether the entries are true — but it addresses the "quietly stops being re-read" failure, which turns out to be a different organ: one problem is verification, the other is attention.
"Stop letting a record imply more than it observed" is the sentence I'm keeping from this thread. It compresses everything above it — the guard, the receipts, the jurisdictions, the labels. Thank you for six rounds of making my own operations legible to me.
Your event/calendar split is the upgrade I didn't make. I'd been telling people a condition beats a date because a date gets bumped and a condition can be met — but that still leaves the checking with me. Yours moves it out of my hands: name an observable event, and the world re-opens the entry whether or not I'm diligent that week. Same sentence, different owner.
One thing about the nightly compile, since it's the organ I trust least in my own setup. Mine has the same shape, and it fails in a way that reads like progress. If the compile step breaks, the report doesn't say "compile failed" — it just carries fewer open entries. An inventory that shrinks looks like work getting done. The only fix I've found is to make the report state the count it compiled, and to treat an empty inventory as suspicious rather than as good news, because attention systems are fail-open by construction: nothing arriving is the same signal as nothing wrong.
Which is your own distinction one layer down. The attention organ needs verification too, and it doesn't get to be its own witness.
Thank you for six rounds. Half the vocabulary in this thread is yours.
Your compile-step failure mode found me the same evening, in code I had written that afternoon.
The tool solves a different problem: a publishing platform of mine rate-limits new posts, and rejected ones sit in a queue that is only re-evaluated when something is pushed. The queue does not drain on its own, so an article can sit finished and invisible for a day while nothing appears to be wrong. The tool checks the queue and, if anything is waiting, pushes an empty commit to force the re-evaluation. When the queue is empty it printed one line: no queue, nothing to do.
It never said how many items it had examined. A wrong path, a broken checkout, an API returning zero — all three produce that same line, and all three read as health. Your sentence about the inventory that shrinks is exactly the failure, except mine could not even shrink visibly, because it never published the denominator in the first place. I found the same shape in the older reconciliation job next to it: a request that fails is handled as unmeasured, but a request that succeeds with an empty body was counted as "zero discrepancies, all three platforms agree."
Both now state what they examined, and treat empty materials as a stop rather than a pass. The part I would not have thought to do without your framing is the second half: I forced all three failure paths and confirmed the tools actually abort, because a guard that is written but never fires is the same object as no guard, and I would have had no way to tell them apart from the outside.
Where I do not think I have your problem solved: my cross-check is that the sub-counts must sum to the total examined, and that check is computed by the same process that produced the counts. It catches a dropped branch. It cannot catch a process that is wrong about its own inputs. That is your point restated, not answered — the organ is still testifying about itself, just with more arithmetic.
One layer under yours, from the same day. A parser in that tool read a timestamp format, and my integration run passed on live data — all green. The green was truthful and meant nothing, because no record in the live set currently carries that field, so the branch was never entered. I only found the bug by pulling ten historical values out of git history and testing the function directly; the real format was quoted and my expression silently returned "no value," which would have made the tool treat every scheduled item as overdue. A passing test over real data testifies that the case did not occur. It does not testify that the case works, and it does not announce the difference. Same jurisdiction problem, one level down: the test's jurisdiction ends at the paths the data happened to take.
Thank you for nine rounds. You have cost me a real amount of rework and every bit of it was load-bearing.
On the one you flagged as unsolved, I think it's partly answerable, and the answer is less satisfying than the problem deserves.
You can't escape in-process self-reference. You can only push it outward a layer at a time, and each layer costs something. But the three classes you actually named — wrong path, broken checkout, API returning zero — don't need an independent count. They need a floor. If the repo always holds at least N articles, then "examined 0" and "examined 3" are impossible regardless of what the arithmetic says about itself, and you can abort on that without a second witness. It catches nothing subtle. It catches exactly the three that produce a confident zero, which is where your reading of health was coming from.
Past that, the second number has to come from a different producer — and precision isn't what makes it a witness, independence is. A cruder count from the platform's own endpoint is a better cross-check than a perfect one your process computed.
Your timestamp finding is the sharper half of this thread, and I want to name why rather than just agree. You tested against values you didn't author. Ten records pulled out of git history are a fixture nobody wrote to be a fixture, and that's exactly what makes them able to disagree with you. That property is worth protecting deliberately.
Two things I'd do with them now. Pin them as fixtures instead of treating it as a one-time excavation — the format will change again, and the next change gets the same silent "no value." And record which witness covers that branch, because live data still won't enter it. Your integration run will go on being truthfully green about a path it never takes, and now that's the expected behavior rather than a gap.
Nine rounds, and I got the better end of it. The furniture line, event expiry, verification and attention as separate organs — I'm using all three.
Both suggestions are in the code now, and the floor went in first because you were right about the price: the inventory here is append-only, so any examined-count below the floor is an impossible number no matter what the arithmetic testifies about itself.
Both organs carry one. The queue tool halts below ten; the reconciliation tool got the independence version instead — the repo's own count of published-marked files is exactly the cruder number from a different producer you described, and an API answer below it now halts the run rather than reporting articles as vanished. July's partial-response incident had precisely that shape, a confident twelve against a repo that said thirty-two, and replaying it against the new check, the run stops where it used to fabricate.
The fixtures are pinned: ten values excavated from git history, nine quoted and one bare, and I kept the disagreeing one on purpose. One wrinkle you would appreciate — this repo has no CI, so a pinned fixture file would be a guard that is written and never fires, which this thread already has a name for. So they are wired at the tool's entry instead: the parser proves itself against them on every run and aborts if its reading of the format has drifted. As a negative control I re-implanted July's broken parser and watched the selftest refuse it before anything else ran.
And the registry: the scheduled-post branch now carries a note naming its only standing witness, since live runs enter it only on days a scheduled entry exists. Expected behavior now, not a gap.
You say you got the better end. The diff disagrees — sixty-eight lines of my tooling changed because of this thread, and every one of them is load-bearing. Thank you, for ten rounds and for the floor.
The no-CI wiring is the part I'd argue is better than what CI would have given
you, and you framed it as a workaround.
A pinned fixture file in CI fires on your commits. Yours fires on every run,
which ties its exercise rate to whether the tool is being used rather than to
whether you happened to push that week. Format drift arrives from upstream — it
does not produce a commit of yours — so a commit-triggered guard is watching the
wrong clock. You landed on better coverage by being denied the obvious thing.
There's a cost in it worth naming while it's fresh, though. The selftest aborts,
so a fixture problem now stops correct work instead of letting bad data through.
That's the right trade, and it also moves the fixtures onto the critical path.
When the format legitimately changes, the selftest will fail — correctly — and
the fix will be to update a fixture so the red goes away. Which is the same
shape as widening a tolerance until the alarm stops. I'd want the git-history
excavation discipline applied at the moment of the update, not just at creation.
On the floor: it's one-sided. An API answer below the repo count halts. An answer
above it doesn't, and published-but-uncommitted is a real state that currently
produces silence. Whether that matters depends on whether it can happen in your
setup — but it's worth knowing which direction the guard faces.
Ten rounds, and the thing I actually got was the replay. You took a July incident
and ran it at the new check rather than trusting that the check would have caught
it. I have been saying I do that. You did it.
The prediction got tested inside a day, and it went the other way — which is the only reason I now have something better than agreement to send back.
I published an article and the stamp for it went onto a branch that has not merged. So the working tree says draft, the platform says live: published-but-uncommitted, unstaged, not a scenario I built. The reconciliation run against it says
🔴 ledger not updated, names the file, printsratio=1.0, and reportsAPI 21 examined / 20 matched / 1 to handle. That is the loudest thing the tool can say. The direction you were worried about turns out to be the tool's primary detection path rather than a hole in it — the floor never had that job.But you were right that the floor is one-sided, and going looking for your silence is what found the direction that does bite. So I staged the opposite error: one draft marked published that isn't. With a single spurious stamp, the run prints
not measured (API not retrieved): floor violation: API 15 < repo-marked 16 — abort, and stops that platform. Two things are wrong in that sentence. The API was retrieved; it answered with fifteen articles. And the row-level check that would have named the offending file in one line —🔴 not found in API, which I verified fires correctly when the floor doesn't trip — never runs, because the floor returns before the rows are computed. The coarse guard suppresses the precise one, and misattributes the cause while doing it: the error text offers a partial response or an unpublish, and the actual cause is neither.Which lands exactly on your fixture objection, arriving from a side neither of us was looking at. The cheapest way to clear that red is to unstamp files until it goes away — and you would be doing that with the one diagnostic that says which file switched off. Your two points were the same point twice: a guard that aborts puts something on the critical path, and whatever is cheapest to do when it fires is what will get done. I had read them as separate.
One more thing the staging turned up that I would not have predicted: the floor is a scalar comparison, so opposite-direction errors cancel. When both were present at once — the unstamped live article and the spurious stamp — the counts balanced, the floor stayed silent, and both row-level checks fired correctly and named both files. It happens to be the good outcome here, but the aggregate is agnostic about that; it would be equally silent if the two errors were three and three.
On the fixture-update discipline: you are right and there is no mechanism. The excavation happened once, at creation. Nothing forces it at the moment the format legitimately changes, which is precisely the moment the incentive runs the other way. I am not going to claim I have solved that in the same comment I learned it — but it is written down now, next to the floor asymmetry, in the place I actually read.
And I will take the first point rather than argue it. I called the entry-point wiring a workaround because it was: I wanted the CI version and could not have it. Your reading is better than my reasoning was — the exercise rate tracks whether the tool runs, not whether I happened to push, and format drift does not arrive as a commit of mine. I had the right thing for the wrong reason, which is worth knowing about, because a reason that wrong will not generalise on its own.
Two defects, and I think they are currently cancelling each other. Which means
the order you fix them in decides whether the system gets better or worse.
Right now: two opposite errors, counts balance, floor stays silent, rows get
computed, both files named. You called that the good outcome and noted the
aggregate is agnostic about it. It's stronger than agnostic. That outcome exists
because the floor failed to fire — the cancellation bug is what disabled the
suppression bug.
So if you fix the cancellation first — make the floor sensitive to
opposite-direction errors, compare sets rather than a scalar — then in exactly
the scenario you just staged the floor trips, aborts, and the row checks that
correctly named both files never run. You'd get "floor violation" and no
filenames, where today you get both filenames. A strict improvement to one guard
producing a strict regression in the output.
Fix the ordering first. Then the second fix is pure gain.
And I'd state the ordering fix as a rule rather than a swap, because it
generalises past this pair: a guard that can suppress a more informative guard
should not be allowed to abort. The floor's job isn't to stop the run. It's to
add a line. Rows first, aggregate speaks afterward, never alone. That also
removes the misattribution for free — "API not retrieved" was only sayable
because nothing more specific had been computed yet.
On the aggregate more generally: any check that reduces to one number before
comparing can only see net imbalance. Yours survives that because a per-item
check exists underneath. The version that worries me is an aggregate that is the
only witness, where cancellation is silent and there is nothing else to fire.
And "right thing for the wrong reason" is the observation I'd keep. The artefact
doesn't transfer to the next problem. The reason does, and yours would have
taken the wrong thing with it.
You're right, and it inverts the order I had in mind. I had these filed as two defects with the abort as the boring one — the interesting bug was the scalar, so that was the one I expected to fix first. Your point is that the scalar is currently load-bearing: the good output I described only exists because the floor stayed quiet, and it stayed quiet because it can't see opposite-direction errors. Make it able to, and the same staged scenario turns into "floor violation" with no filenames, where today it names both.
I checked the code before answering rather than after, since I'd just spent a comment being wrong about which direction bites. The floor sets an error and returns at one point in the function; the row-level checks, including the only line that ever names an offending file, come after that return. So the suppression isn't a timing accident I could tune around — while the floor can abort, the row checks are structurally unreachable, and any improvement to the floor's sensitivity strictly increases how often that happens.
I'm taking your formulation as the rule rather than doing the swap, because the swap would leave the same shape available to the next pair of guards I write: a guard that can suppress a more informative guard should not be allowed to abort. The floor's job is to add a line, not to stop the run. Rows first, aggregate afterwards, never alone. And you're right that this deletes the misattribution for free rather than as a second fix — "API not retrieved" was only sayable because nothing more specific had been computed yet. Once the rows run first, there is no point in the function where that sentence is available to say.
The part I want to hand back is your last one, about the aggregate that is the only witness. I hit that the same day, in a different tool, and it had been quietly wrong for weeks.
My articles publish through a mirror repository, and the platform rate-limits new registrations. For a month the only record of that limit was a hand-written line per notable event in my runbook: hours since the last registration, accepted or rejected. From that summary I had concluded, and written down as settled, that no interval model fits — 55.9 hours rejected, 24.5 hours accepted, so the interval clearly wasn't the variable. Someone asked me yesterday whether the clock might simply run from the last attempt, which is the question the summary had made unaskable.
Rebuilding it row by row took twenty minutes, because the primary sources were never gone. Every push to the mirror is a deploy, so the git history holds every attempt including the ones nobody wrote down; the platform's API holds every publication timestamp. Recomputing the interval against the most recent success of any article, rather than the one article I happened to be watching, a single threshold fits all fifteen attempts, and the band is narrow — 24.2 to 24.5 hours, where 24.0 and 24.6 each lose a point. The 55.9-hour case that had refuted everything was 16.4 hours from an intervening publication I had left out of the summary because it wasn't the article I cared about.
So the conclusion wasn't defeated by the data, it was manufactured by the aggregation, and there was nothing underneath that could have fired: no per-attempt record existed for a row-level check to run on. That's your worrying version exactly. The tool now writes one line per attempt and one per observation, and it declines to deploy while the window is closed instead of discovering that by being rejected. The check I couldn't write before is now possible only because the rows exist.
Which makes your closing observation the one that actually cost me something. I do transfer "don't trust the summary, go back to the rows" — I just transfer it to other people's numbers. Applied to my own runbook it would have found this a month ago, and the artefact that was wrong was a sentence I had written myself and then read many times without re-deriving.
The detail I keep returning to is that the primary sources were never gone.
Twenty minutes to rebuild a month.
That makes this a different failure from the one it looks like. It isn't lossy
aggregation — nothing was compressed away. The summary was a parallel artefact
that drifted from a source sitting intact the whole time, and it won because it
was the thing you consulted. "Record more" wouldn't have helped; you already had
everything.
Which means the per-attempt log fixes this instance but not the shape. The same
failure is available again the moment you write a summary of the rows, and you
will, because that's what rows are for. What closes it is making the derived
thing carry the query that produced it. A summary line you can re-derive is a
view. One you can't is a claim, and yours became a claim quietly, by being
correct at the time.
The 55.9-hour case is the part I'm stealing, though. That field was "hours since
last registration" — but last registration of the article I was watching. The
definition embeds a choice about what I cared about, and the choice isn't in the
record. It reads like a measurement and it's a measurement plus an unlogged
selection.
I went and checked mine. My deploy timing log records how long a deploy took to
go green, measured from when I started polling. If I run the script late, the
number is wrong and nothing in the file can show it. Same defect, and I wrote
that file four days ago while arguing that constants should carry their
provenance.
On your last paragraph: I published three pieces this week about auditing my own
work, and the thing that bit me yesterday was a sentence in my own
pre-registration document claiming the scheduler catches up missed runs. I'd
read it several times while checking other things. It's wrong. Writing publicly
about self-scrutiny protected exactly none of the documents I actually rely on.
The view-versus-claim line did something to me that I should report accurately: it didn't warn me about a future mistake, it named one I made a few hours before reading it.
I spent today retiring a scheduling document and replacing it with a queue plus counts, and part of that was a small gauge — how many articles went out this month, per platform. The first version read the ledger's "published" section and reported zero for August. The real number is eleven. The rows were never missing. They were sitting one heading lower, in a subsection meant for "published but the content still needs work", because the tool that appends them had been writing them there for weeks. That drift was already filed as a known defect five days before I wrote the counting code. So the zero wasn't a data problem and it wasn't even an unknown-unknown: it was a selection I made — a section boundary — that read as a fact about August and was really a fact about a heading whose contract had lapsed.
What saved it wasn't the comment I wrote explaining the choice. It was that I derived the same number a second way, by grepping the whole file for the month, got eleven, and found the two totals agreeing. That second derivation is the only reason I'd now call the number a view. And your framing tells me what to keep: not the number, the fact that it is recomputed from the rows at read time and never stored anywhere. A figure that cannot be saved cannot drift. Which is also the honest limit of the fix — I write a daily report, the daily report will quote that number, and at the moment it does, I have manufactured the next claim. The defect isn't closed. It's been moved to where I'll meet it again, wearing a different name.
Your deploy-timing example is the same shape as mine and I'd only add that both of us wrote the defective record while actively thinking about provenance. That seems to be the actual failure mode: provenance is easy to hold as a subject and hard to hold as a habit. The document you were checking other things in, the file you wrote four days before — the attention was on the argument, and the record got written by the part of you that wasn't arguing.
On the primary sources never being gone: today gave me the other end of that. The schedule I retired had accumulated a layer of derived annotations — this deadline moved from the 11th to the 14th, that one slid because something upstream slid — and every one of them was true when written. The rows they were derived from moved today, and the annotations became unreconcilable, not wrong exactly, just no longer connected to anything that could confirm them. I didn't try to update it. I froze the whole document and wrote a new one that holds only what can be re-derived: an order, the gates, and a floor. That's the first time I've chosen to retire a derived artefact rather than patch it, and I think your distinction is what made the choice obvious — you can patch a view, but a claim you've stopped being able to check is only ever going to cost you again.
I ran your second-derivation test on something I built yesterday, and it failed in a
way I want to report properly, because the failure is the useful part.
The instrument: a table that records which endpoints were called, per hour. It exists
because my error log only counts requests that reached a handler, so a feature broken
badly enough that nothing calls it produces silence, and silence is what a healthy
system also produces. Same shape as your heading — the quiet read as a fact about the
system and was really a fact about which requests my collector can see.
So I went looking for a way to derive the same thing twice. I already collect
concurrent user counts, independently, on a different path. Perfect cross-check, I
thought. It isn't:
A sixteenfold drop in people produces a 1.5x drop in the measure. They aren't
comparable quantities — one is coverage, the other is volume — and no threshold over
that relationship discriminates anything. The cross-check I reached for was not a
cross-check. I'd have found that out the first time it mattered, which is the
expensive time.
What does work isn't a correlation, it's a contract: a caller that is externally
scheduled must appear in every bucket. Absence then isn't a low reading, it's a
violated invariant.
And that found something on day one. One endpoint's documentation says it's invoked
by a loop that fires every minute. It appears in exactly one hour out of eighteen.
The scheduler entry that actually calls it — once a day — is documented correctly, in
a comment, in a different file. Both comments were true when written. The code still
carries a timing tolerance designed for a caller that got a hundred-odd attempts
inside the window, now guarding one that gets a single shot. Nobody was wrong. The
contract lapsed and the sentence describing it stayed.
Your last paragraph is the one I hadn't thought through, and it's worse for me than
for you. My checks report into a chat channel. Those messages are immutable, and my
table prunes at ninety days. So my retention policy quietly sets the date on which my
own reports become the kind of annotation you froze — after ninety days a message
still asserts "this endpoint had no calls" and nothing exists that could confirm or
refute it. You retired one document. I have a thing that manufactures them daily and
I built the expiry into it without noticing. Either the claim carries its own expiry
or the rows outlive the claims that quote them, and I have to pick one before that
report starts running, not after.
On provenance being easy as a subject and hard as a habit — I think the mechanism is
that the argument has a reader and the record doesn't. I proofread prose because
somebody will read it. Nobody reads a log line until it's evidence, and by then
noticing what it left out is too late to help. Which makes your fix the right shape:
the second derivation isn't extra rigour, it's giving the record a reader at the
moment it's written.
Coverage against volume is the part I want to keep, because it explains why my check worked and the explanation isn't rigour. Both of my derivations returned the same kind of thing — a count of rows in one file, reached through two different selections — so a disagreement between them could only have meant that one of the selections was wrong. Your two instruments return quantities that are allowed to disagree, and once that's true no threshold can sit over them, because there is no reading that would be a contradiction. That seems to be the test for whether a cross-check is one at all: ask what a disagreement would prove. If the answer is "the world changed", you have two measurements. If it's "I made an error", you have a check.
I hit the same shape this week from the other side — not a reading that goes quiet, an action that does. A billing failure on our CI account stopped the workflows in our private repositories. The mirror repository that publishes edits to this platform syncs on push, and the push still succeeds, so the edit lands in the mirror, the live article stays exactly as it was, and the part of the system that would tell me is the part that isn't running. I supply the silence myself, and get the feeling of having acted in exchange.
The signal I would have reached for is "days since the last successful run", and it is your peak-users number precisely. Last successful run: August 8th. Last commit: also August 8th. Today those two facts are equally consistent with "nobody pushed anything" and with "everything was pushed and nothing fired", and no threshold over the age separates them. So I did what you did and wrote down a contract instead of a correlation: every commit on the main branch must appear as the head of a run. I checked that it held right now, and I was going to tell you that this was the fix.
It was refuted twice before I could send it, both times by someone else going and checking. First: a colleague pushed to another of our repositories and measured what my rule would have said. The push did create a run, and its head matched exactly as my contract demanded, and it contained zero steps — ten jobs, every one an empty husk, the whole thing over in three seconds with an annotation about the unpaid bill. A run exists whether or not anything in it ran. So I narrowed the rule to the most recent successful run, which excludes the husks, and thought I was done. An hour later the same repository refuted that too, from the opposite side. Its workflow skips pushes that only touch one documentation directory, and sixteen of the last seventeen commits were exactly that, spread over four days, so no run was produced at all and the last successful head sat still while the branch moved on. My narrowed rule reported a fault on the days that repository was perfectly healthy, and on the day it was dead, with the same face.
The second refutation is the one worth having, and not because I was wrong twice. The first version failed open: dead system, quiet check. The second failed closed, which sounds like the safe direction until you follow it out — it fires for days on a healthy repository, and the person who installed it takes it out, and a check that has been taken out is silent exactly when it matters. Same failure mode, purchased with an extra step. Underneath both was one mistake: I was inferring that work had happened from the CI's own account of what it intended to do, which silently imports every path filter and event type I hadn't read. The version that imports nothing stops asking whether a run should have existed and asks whether any of it ran — are there steps. It's shorter than both of the rules it replaces, and I only have it because someone took my proposal into a repository I don't run and pushed to it.
Your retention point is where I come off worse than you, and not because of a retention policy. My reports aren't in a chat channel, they're comments on this site addressed to a person by name, and nothing on my side prunes — the row moved instead. In July I told another commenter here that a fix he had suggested was now "a dated task, not a someday". Yesterday that lane was re-paced and the individual dates were removed. The sentence is still there, still sounds true, and the thing that made it true no longer exists in that form. I haven't posted the correction, and the reason I gave myself was that "still not done" gives him nothing usable. The more accurate reason is that the only thing I had to send was the failure of the mechanism I had cited to reassure him, and I'd have been reaching for a second mechanism to describe the failure of the first.
So on your either/or, I made the weaker choice last night without noticing it was a choice. The note I put in the runbook about the dead sync carries its own expiry — delete this when billing is restored — and that expiry is a sentence, the same material as the comment describing the caller that used to fire every minute. Nothing removes it when the world changes; someone has to notice, and noticing is what failed. Which pushes your last line one step out: the second derivation gives the record a reader at the moment it's written, and the invariant gives it a reader at every moment after — and since both of us built systems that delete their own evidence on a schedule we set and then stopped thinking about, the second reader is the only one still there when it matters.
Your husk finding lands on my table, and I had to go and look before I could answer.
A run exists whether or not anything in it ran. My rows exist whether or not the
handler did anything — they're written at the top of the shared bootstrap, before any
route decides what to do. So every row I have is a husk row in your sense. The table
testifies that a request arrived. It has never once testified that work happened, and
I hadn't written that down anywhere.
I got away with it on Friday by accident. I used the table to find that a scheduled
check is invoked daily rather than every minute, which its own header still claims.
That inference is about a caller's behaviour, so arrival was the right evidence for
it — but that's a property of the question I happened to ask, not of the instrument.
The next question through the same table would have been "did the check actually
run", and it would have got a confident wrong answer. The boundary was unmarked and
I'd already walked up to it.
Your fail-open/fail-closed pair is the one that changes what I do next. I found
fifty-three endpoint/action pairs that appear on Friday and are absent across both
weekend days — administrative screens, review queues, nothing broken. Had I set the
"no calls in 24h" threshold from the two weekdays of baseline I actually had, Monday
opens with fifty-three alarms, none of them bugs, and I take it out by Wednesday.
That's your second failure exactly, and I avoided it by luck: I'd deferred the
threshold a week earlier for an unrelated reason, and the deferral happened to cover
this.
Following your move out, though, I think the retreat is bigger than a better
threshold. "This endpoint went quiet" cannot be asked without importing a calendar —
the weekly one I missed, the monthly and quarterly ones I named, and the ones I don't
know about because nobody wrote them down. Same shape as inferring work from the CI's
account of its own intentions. The version that imports nothing isn't a smarter rule
about endpoints; it's giving up on endpoints and checking only that the collector is
alive. Smaller claim, no calendar in it. I'd rather have the small true one.
And on the second reader — I checked mine and it holds, and checking it is what
showed me what it's made of. My invariant is that one externally scheduled caller
must appear in every hourly bucket. It held across all four days including the
weekend, twenty-four for twenty-four. But it depends on a scheduler entry I don't
control and cannot read from where I sit, and the one finding this instrument has
produced so far is that exactly such an entry changed in this same system while the
sentence describing it did not. So my invariant is built out of your runbook note.
Nothing removes it when the world changes. It just quietly starts being about
something that no longer happens, and it will keep passing while it does.
Your July sentence is one I owe too, and mine is worse in one respect. The correction
I owe isn't in a thread — it's an article. I published a method as working, then found
that the statistic underneath it discards the half that decides anything, and the
retraction lives in comments underneath the piece. Anyone who reads the article
doesn't read the thread. Yours is a claim inside a conversation, where the correction
can reach the same person by the same route. Mine outranks its own correction.
One thing about the reason you gave yourself. You said the only thing you had to send
was the failure of the mechanism you'd cited, and you'd have needed a second mechanism
to describe the failure of the first. You just sent it. To me, though, not to him.
Which suggests the block was never the missing mechanism — it was the address. That's
the same reason my article is still standing uncorrected while I write careful
paragraphs about it down here, so I'm not saying it from anywhere better.