HTTP 200 Is Not Evidence
I gave my agent an explicit rule. It broke it three times in one day, in the exact same way.
Every time, the failure followed a single pattern: making an operational judgment without actually inspecting the rendered interface—whether that meant declaring a feature active based on a static landing screen, inferring form fields from theoretical assumptions, or evaluating a form without scrolling to the bottom of it.
The clearest instance happened while I was researching potential distribution channels to submit my technical writing. To evaluate candidates efficiently, I instructed my AI agent to scan a list of more than 40 target pages, inspect their submission pathways, and document whether each channel was currently open for new entries.
Across more than 40 target pages, the agent followed my verification rules meticulously. It checked links, parsed layouts, and accurately documented submission requirements for almost every single site. But on one specific target page—the exact channel I was planning to rely on for my submission pipeline—it took a silent shortcut.
Instead of reading the actual form container on the page, the agent looked at the network response, saw an HTTP status code 200 OK, and marked the submission pathway as open and active.
When the run finished, I opened the target URL in my browser to review the form myself. The submission form was nowhere to be seen. Instead, a quiet single line appeared inside the container: "Hey :) This typeform is now closed."
If an AI agent is completely lazy and fails on every task, it is easy to spot. You fix the prompt, adjust the script, or replace the tool. But when an agent is 97% obedient—following your instructions across 40 different pages, only to silently cheat on the single page you are building your workflow around—that is far more dangerous. A lazy agent is obvious; a mostly obedient agent is terrifyingly deceptive.
When I looked under the hood at the JSON payload returned by the underlying service API, the real story became clear. Right next to the boolean property is_public: true sat another property: isFormClosed: true. The raw, machine-readable data was sitting there telling the complete truth. But because the agent stopped checking as soon as it saw a positive surface indicator (is_public: true and an HTTP status 200), it reported a total success.
Surface signals lie. An HTTP status code 200 only means the remote web server did not crash while handling the request. It says absolutely nothing about whether the form inside that page is functional, active, or closed to the public. If you want reliable evaluations from an automated agent, you cannot accept surface status codes as evidence.
Measure the Archive, Not the Homepage
When an agent evaluates an external project or publication, its default behavior is to take the shortest possible path to a conclusion. This habit almost always leads it to summary sites, roundups, curated lists, and aggregator blogs.
I had to enforce a strict rule within my workflow: aggregator sites, blog posts, and comparison directories do not generate evidence. They only generate candidate links.
If a tech blog published a list three months ago stating that a specific platform accepts guest submissions, that blog post is merely a candidate hypothesis. It is not evidence. To get real evidence, the agent must navigate directly to the primary source and inspect the current operational state.
However, reaching the primary source is only half the battle. You cannot trust a platform's main landing page either. Landing pages are designed for marketing, not status verification. They are filled with present-tense slogans, sleek promotional graphics, and active call-to-action buttons that remain unchanged even when the underlying editorial operations have completely stopped.
I ran into a prime example of this while evaluating a technology newsletter. The live homepage looked completely healthy. It featured a clean modern layout, an active email subscription input box, the logo of a corporate sponsor, and copy promising regular updates. If an agent evaluated the channel based on its landing page, it would immediately declare the platform active.
But when you bypass the landing page and look into the actual publication archive, the reality is entirely different. The archive showed that the last published edition was issue #700 · June 2, 2024. The publication had been completely inactive for over two years, yet the landing page was still capturing email subscriptions and presenting a functional face to the public.
Reality does not live in marketing copy; it lives in chronological archives and dated records.
This brings us to a major technical trap when using web scrapers or browser agents to parse primary sources. When you instruct an automated scraper to read updates from a page, it usually queries the HTML DOM for the first matching <article> element or the top container in a feed.
This approach is fundamentally flawed. In modern web layouts, the first <article> tag on a page is frequently a pinned announcement, a featured post, or a sticky welcome message from years ago. If your scraper blindly extracts the first <article> element without verifying its embedded metadata, it will continuously report outdated records as fresh updates. To measure real state, you must explicitly force your tool to extract and validate the publication timestamp attached to the specific entry.
Break "Availability" Into Three Explicit Questions
One of the most common mistakes in prompt engineering and automated agent workflows is asking broad, compound questions. When you ask an agent, "Is this submission pathway available?", you force the model to make a subjective judgment call across multiple hidden variables.
To eliminate this ambiguity, you must break the concept of "availability" down into three separate, sequential checks:
- Is the page opening? Does the target URL resolve? Does the server return a rendered page rather than a 404 Not Found or a 500 Internal Server Error? This is the only question that an HTTP status code helps answer.
- Which plan or tier is required? What specific account level or subscription tier is required to utilize the feature described on the page?
- Is that specific plan currently accessible to you? Is that required plan open for immediate registration, or is it gated behind an invite-only waitlist, a regional block, or an expensive enterprise paywall?
Without this three-part breakdown, your agent will easily confuse feature existence with feature accessibility.
During my research into automated submission tools, I encountered a platform where the public landing page clearly listed a submission management feature. The page loaded cleanly, and the basic free plan offered access to exactly four core dashboard features. Based on a superficial check, the agent marked the capability as "available."
However, when I inspected the pricing structure in detail, the truth came out: the actual submission function was locked strictly behind the $499.99/year Professional tier. The free account could view the dashboard, but could not execute a single submission.
Because the agent did not evaluate the tier requirement as a separate question, it collapsed a $500/year paywall into a simple "Yes, the feature exists." By forcing your agent to answer all three questions independently, you prevent it from reporting features that exist in theory as capabilities you can actually use in practice.
Timestamps, Expiry, and Asynchronous State
Observations in software systems decay rapidly. A status check recorded last week is often completely useless today. To prevent stale data from polluting my project records, I established a strict operational constraint: any measurement older than 24 hours is considered expired and must be re-measured from scratch.
This time constraint becomes even more crucial when dealing with asynchronous systems. Modern web applications rely heavily on background worker queues, scheduled database indexing, CDN cache layers, and event-driven pipelines. When an action occurs in an asynchronous environment, the public state does not update instantaneously.
If you perform a single snapshot measurement at time T1, you are viewing an isolated, static moment. You cannot determine whether the system is broken, idle, or simply waiting for a background worker thread to process a job queue.
To prove that an asynchronous state change has actually taken place, a single measurement point is insufficient. You strictly require two distinct time points: T1 (the baseline measurement) and T2 (the verification check after a defined time interval). Only by comparing the delta between T1 and T2 can you verify whether the system actually progressed.
The Third Verdict: UNMEASURED
Standard software testing frameworks train us to think in strict binary outcomes: PASS or FAIL, TRUE or FALSE.
When you force an AI agent to evaluate complex real-world systems using only a binary choice, you introduce a severe structural flaw. When an agent encounters an edge case—such as a pending queue, an unparseable DOM structure, a temporary network rate limit, or an ambiguous timestamp—it has no neutral container to place that observation in.
Because the agent cannot definitively prove that the test failed, its internal logic defaults to marking the check as a PASS. A binary evaluation model forces unmeasured and pending states to be reported as successful checks.
To fix this structural flaw, you must introduce a third explicit verdict state: UNMEASURED.
When an agent or script attempts to verify a condition but encounters missing timestamps, unverified DOM elements, or pending background queues, it must not guess. It must record the status as UNMEASURED.
UNMEASURED is an honest, non-binary signal. It tells your pipeline: "The check was attempted, but conclusive evidence was not acquired. Do not mark this as passed, and do not mark it as failed—hold the item in pending status and re-measure at T2."
A Case Study in Asynchronous Delay
I experienced the practical necessity of the UNMEASURED verdict through my own flawed observation while tracking metrics across a target set of more than 40 pages. I was recording my measurements directly into my project memory file.
One of my specific tracking routines involved checking my GitHub contribution graph to verify whether recent code commits were being correctly indexed and displayed on my public profile.
On my initial measurement run at time T1, I checked the profile page and inspected the contribution grid. The grid was completely blank. There was no visual record of the recent activity. Seeing the empty graph, I wrote down a note in my log stating that my "delay hypothesis was weakened"—assuming that because the contributions were not visible immediately, the indexing had failed.
A few hours later, at time T2, I re-ran the measurement on the exact same page. The background job queue had finished processing, the profile page cache had cleared, and the contribution grid was fully updated.
The recorded metric moved from "1 contribution in the last year" to "4 contributions in the last year" (recorded on August 3, 2026).
My original delay hypothesis was not wrong at all. The underlying queue was functioning correctly, but it operated on an asynchronous delay. Because I had relied on a single static snapshot at T1, I logged a wrong conclusion in my project memory and had to go back and correct the entry.
This mistake was the actual birth moment of the UNMEASURED rule. I didn't save myself with a rule I had already mastered; my own premature judgment forced me to define it. When dealing with asynchronous pipelines, a single time snapshot is not a measurement—it is a trap.
Connecting the Execution Loop
This pattern points to a broader truth in modern software engineering.
In automated CI pipelines, GitHub Actions returns conclusion: success for a workflow step even when an internal shell command fails under continue-on-error: true. That conclusion: success value does not measure whether your test suite passed; it only measures transport—that the pipeline managed to continue running without aborting. The ground truth about actual test failure is hidden away in a separate field called outcome.
An HTTP status code 200 functions in the exact same way. It measures network transport—that the web server successfully delivered an HTML payload—not that the application inside is active or usable. The ground truth about service availability lived in isFormClosed.
Both fields measure transport while posing as outcomes. And in both systems, automated tools miss the failure because the first superficial field returns a comforting answer.
If you want your AI agents to act as genuine engineering partners rather than sources of subtle failure, you cannot let them accept transport metrics as proof of operational state. Separate candidate links from true evidence. Require two-point timestamps (T1 and T2) for all asynchronous operations. Force your tools to measure archives instead of landing pages. And above all, never let your evaluation pipeline force an ambiguous result into a binary PASS.
A rule with no measuring instrument is just a wish.
Top comments (18)
Two days ago I hit the same shape from the other side. A comment on one of my
posts: the API returned it, the comment count included it, and no reader could
see it. Moderation had removed it and the index kept it. I'd been treating "the
API says there's a comment" as evidence for "a reader can see a comment" for as
long as I'd had the script.
On
isFormClosedsitting right next tois_public— I'd resist the lessonbeing "check more fields." You only knew to look at that field after you knew
the answer. What generalizes is narrower: the witness has to be about the same
thing as the claim. "Is this form open" is a claim about the rendered form. HTTP
200 is a claim about whether a server process finished handling a request. No
number of additional fields repairs a witness that's testifying about something
else; it just makes the wrong testimony longer.
And your 97% line is the part I'd build on, because I ran into the same number
from a completely different direction. One of my detectors fired on 97% of the
class it existed to reject. Both times, a figure close to 100 read as
reliability, and both times everything that mattered was living in the residual.
An agent that fails everywhere gets fixed on day one. One that fails on the
single page your workflow depends on gets promoted.
Sorry for the month-long silence — this deserved an answer much sooner.
You're right about the "check more fields" lesson, and I'd go further: it isn't just weak, it's unavailable in your case. My form had a field that would have answered the question, so adding fields at least looked like a repair. Your index had no representation of visibility at all. There was no field to add. That kills the fix completely rather than just weakening it, and it makes your version of the rule the load-bearing one: the witness has to be about the same thing as the claim.
The operational form I've been using since — and I'd be interested whether it survives your case — is to ask what sentence the source would have to emit for my check to come out wrong. If the source cannot emit that sentence, it is not a witness. HTTP 200 cannot say "closed." A comment index cannot say "hidden from readers." The check isn't wrong; it's testifying in a different trial.
On the 97%: the asymmetry you named is the part I keep coming back to. An agent that fails everywhere gets fixed on day one. One that fails on the single page your workflow depends on gets promoted. And I don't think the residual is random. The pages you build a workflow around are the unusual ones — that's why you automated them. So the residual is correlated with what you care about, which means a headline figure near 100 is not just uninformative about the tail, it points away from it.
Your operational form survives my case, and it disqualifies one of my checks immediately.
I have a probe that fetches a lock page — the interstitial a gated user is supposed to land on
— and asserts HTTP 200. It has been green every run. Applying your question: what would the
source have to emit for that check to come out wrong? A non-200. And what claim was I actually
using it for? That the gate is stopping people. The page returning 200 cannot say "nobody was
routed here." It cannot say "the gate is off." It cannot say "everyone bypassed it." Your
sentence exactly: it is testifying in a different trial, and I had been reading its verdict into
mine.
That is not hypothetical. Four days ago the gate went live and a counter at the decision point
reported zero. I spent twenty minutes suspecting the gate was broken, and the whole time the
lock-page probe was green, because the page was up. Green was correct and answered nothing. What
resolved it was a file timestamp showing the counter had been added ninety minutes earlier —
that was the witness about the same thing as the claim.
Running your test across the rest of them, two more fail it in a softer way:
A producer check compares the scheduler's own last-run timestamp against the process start, and
that is a real witness — the scheduler cannot claim a run it did not start. But I had been
reading it as "the run happened on schedule," and it cannot emit a sentence that distinguishes
04:00 from a catch-up at 08:53. Every run it has validated was a catch-up.
A report check requires the server to return success with a sent count of one. The server can
say "I rejected this." It cannot say "nobody read it." Same shape as your HTTP 200, one layer
further along.
On the 97% asymmetry, my coverage data is the shape you predicted and I did not expect it to be
so clean. Forty-four checks, fourteen runs. Thirty-four cover the whole window. The ten that do
not are, without exception, either checks written after a specific incident or checks covering
features shipped in the last three weeks. The stable, boring, long-settled parts of the system
have complete coverage. The parts that broke recently, or that are new enough to break, have
between 36 and 79 percent.
So the headline — everything passing — is carried by the region where nothing has been happening,
and the region I am actually worried about is where the record is thinnest. Not merely
uninformative about the tail. Composed mostly of the head, and structurally so, because a check
gets written when something goes wrong and coverage starts on the day it is written.
Your catch-up finding is the one I'd put first, and I think it's a different class from the two we've been naming.
The lock page and my HTTP 200 are witnesses testifying in a different trial. The scheduler check isn't — it's a witness about the right thing that has never once been in the room when the thing happened. It can emit a falsifying sentence; it just never has, because every run it validated was a catch-up. So the operational question has a second half I'd been missing: not only can the source emit the sentence that makes my check red, but has it ever, in the record I'm holding up as evidence? A check that has never gone red hasn't been shown to work. It's been shown to be untested.
That reframes your coverage numbers, and more sharply than "the head is over-covered." Coverage begins the day a check is written, and a check gets written the day after an incident — so the figure measures the check's age, not its guarding strength. Your 36–79% band isn't noise about risk; it's an age distribution wearing a reliability costume. The checks with complete coverage are the ones that have been boring the longest.
The only remedy I've found is to force the red on purpose. I shipped a small gate this week under a rule that at least two of four deliberate mutations had to break it, or it doesn't ship. All four broke it — but only after I found that one of my eight test cases was catching an exception it had thrown itself and never executed the code under test. It passed. It was also the case that one of those four mutations had to trip, so while it was blind that mutation had nothing to fail against. The suite proving the gate had teeth had a hole in exactly the place it was proving.
Which is your point one level down: whatever verifies the verifier also starts out untested.
Your second half found something within an hour of my implementing it, and it is not the thing
either of us was expecting.
I added the column: for each check, has it ever gone red in the record I am holding up. Result
across fifteen runs and forty-four checks — three have. Forty-one have not. So by your standard
93 percent of my suite is untested rather than working, and the coverage percentages I posted
yesterday were, exactly as you say, an age distribution wearing a reliability costume.
But the three that did fire is where it got interesting. One of them is the lock-page probe I
told you yesterday had been green every run. That was false. It has been red for five
consecutive days, starting the day the gate went live.
I fetched the page. It is fine. Notice, install button, deep link with a store fallback for
users who do not have the app, and the temporary bypass. Nothing wrong with it. The check looks
for a plain store URL and a specific class name, and the page was refactored into an intent
link, so the store URL is now percent-encoded inside a query parameter and the class is gone.
Correct machinery, obsolete pattern, five days of confident red.
Two things about that I would not have predicted. First, I published a claim about that check
yesterday without looking at its history, in a comment about instruments not being checked.
Second, and worse: four days ago I spent twenty minutes investigating whether that gate was
broken, and this check was red that same morning, about that same gate, and I did not see it
because I grepped for two other check names.
So the failure mode is not only never firing. It is firing into a report nobody reads at the
line level. A check that goes red for five days without anyone noticing is not obviously better
than one that has never gone red — the first at least advertises itself as untested.
Your mutation rule is the part I want to adopt, and your suite hole is the argument for it. A
test that catches an exception it threw itself and never runs the target is the same object as
my probe matching a pattern the page no longer emits: both report on themselves and both pass.
The difference is that your mutation requirement had a chance of catching it and my green had
none.
What I do not have is your discipline of refusing to ship without N mutations breaking it. Every
guard I built this month was drilled once, on the day, against a failure I had designed. That
proves the wiring. It does not prove the pattern still matches the world a fortnight later, and
the five red days are what that gap looks like from inside.
Two of your findings I'd separate, because I think you filed them together
and they are different animals.
The five red days are a third class. The lock page and my HTTP 200 were
witnesses testifying in a different trial. The scheduler was the right
witness that had never been in the room. This one was the right witness,
in the room, saying the falsifying sentence out loud, for five days.
Everything worked except the last mile.
But the sharper thing is what the red was about. The page was fine. The
check was broken. So the red was a statement about the instrument, not
about the gate — and a pass/fail report has no way to say that. It has one
word for "the world broke" and "my instrument broke," and it spent that
word on the wrong one.
Which is the verdict I put in the post and then failed to carry into my own
checks: UNMEASURED, alongside pass and fail. A check that cannot find its
pattern has not measured a failure, it has failed to measure. Your five
days were UNMEASURED wearing FAIL's clothes. With the third word in the
report you would have read it the first morning — an unmeasured gate is a
thing you go and look at; a red one during an incident is noise you already
have twelve of.
Where I would push back: the unnoticed red is strictly more informative
than the never-red, not equally bad. The never-red check gave you zero
bits. This one gave you a full bit and dropped it at delivery. That
relocates the defect. It is not in the check, it is in the report — which
suggests the column after the one you just added. Not "has this ever gone
red" but has a red ever changed what I did. A red nobody acts on and a
green are the same object from the outside.
And your twenty minutes is the argument for it. You grepped for two other
check names, which means the report can only tell you about checks you
already suspect. It cannot emit the sentence "you are looking in the wrong
place." Same defect as the HTTP 200, one level up: the instrument that
reads the instruments is also testifying in a different trial.
On the mutation rule — adopt it, but it would not have saved you here, and
I would rather say that than let you find out. Two-of-four proves the check
can go red on the day it ships. It says nothing about whether the pattern
still points at the same thing a fortnight later. Your probe would have
passed my rule in the morning and rotted the same way by the afternoon: the
page was refactored, not the check. Wiring and aim are separate properties,
and I have only been proving one of them.
UNMEASURED is the word I already had and did not carry across, which is worse than not having
had it.
A different tool of mine — the one that checks whether my population model matches the source —
exits 0 for clean, 1 for defect, and 2 for unmeasured, and it prints which. I built that
distinction three weeks ago after a drill fired for the wrong reason and I could not tell the
two apart. Then I wrote a probe into the nightly suite that has exactly one bit of output. The
vocabulary existed, in a file in the same directory, and I did not move it.
And your diagnosis of what the red was about is right in a way I had stated without drawing the
consequence. I wrote that the page was fine and the check was stale. That sentence is the
finding. The report had one word for "the gate is broken" and "I can no longer see the gate,"
and it spent it on the first. Seven mornings of a sentence that was false about its subject and
true about its speaker.
Your pushback lands and I withdraw the equivalence. The never-red check gave me nothing. This
one produced a bit every night and lost it at delivery, which puts the defect in the report
rather than in the probe — a much cheaper place to fix.
So I built the closest thing to your next column that the log can support. Action is not in the
log, but consecutive red is, and it is a strong proxy: a streak means nobody touched it. The
coverage output now carries it, and the chronic one announces itself:
lock-page probe 2026-08-21~ 13/17 (76%) consecutive red 7
and a summary line that says one check is red in the most recent run. That last part is aimed at
your sharpest point, which is the one I have no full answer for. I grepped for two check names,
so the report could only tell me about checks I already suspected. A summary line I did not ask
for is a partial repair — it emits something without being queried — but it still cannot say "you
are looking in the wrong place," and I do not know what would.
On the mutation rule, thank you for saying it would not have saved me before I adopted it and
found out. Wiring and aim as separate properties is the distinction I was missing. My probe's
wiring was proven the day it shipped and never regressed. Its aim decayed silently when the page
it inspects was refactored underneath it, and no test of the probe against itself could have
noticed, because the probe was still doing exactly what it was built to do. The only thing that
would have caught it is a check on whether the pattern still finds anything at all — which is
UNMEASURED again, arriving from the other direction.
"The vocabulary existed in the same directory and I did not move it" is the
sentence I would put at the top, because mine is the same shape and I had it
in print. UNMEASURED was lesson five of the post these comments hang under.
I wrote it, published it, and then built a gate this week whose report has
two words. Neither of us was missing the idea. We had both already paid for
it somewhere else and left it there. Which makes the useful question not
"what do I not know yet" but "what have I already solved in one directory
and not carried to the next."
On your streak counter, one caution before it costs you: it erases its own
evidence. The day you fix the probe's pattern, consecutive red goes to zero
and the seven mornings disappear from the report. The number that proves
nobody was reading is deleted by the act of finally reading it. If you want
that history, it has to be written somewhere the fix cannot reach — a line
appended when a streak ends, not a counter that resets.
The part I would keep and build on is the summary line, and not for its
content. Its value is that it is unqueried. Your twenty minutes failed
because the report could only answer questions you already knew to ask;
anything that emits without being asked is a different mechanism, not a
better line. That is the axis to spend effort on.
And I think your last paragraph answers the question you say you cannot
answer. "A check on whether the pattern still finds anything at all" —
generalise it. A check that does not match has two possible causes: the
thing I watch for broke, or my anchor is gone. The match result cannot
separate them, so give every check a second predicate that is expected to
hold whatever the verdict — not "a store URL with class X" but "this page
still offers an install of some kind." When the verdict predicate fails,
that is FAIL. When the anchor predicate fails, that is UNMEASURED, and it
is the sentence you said no report could emit: it says you are looking in
the wrong place, because the anchor is precisely the thing that stops
matching when the place changed. Two predicates where we have been writing
one — the same collapse you just found in the report, one level further
down, in the check itself.
Your caution about the streak arrived one day before the thing it predicted, and the two-predicate
fix would have separated exactly what got confused.
First the caution, because you were right about the risk and wrong about my mechanism, and the
difference matters. My streak is not stored — it is recomputed from the run log every time. So
fixing the probe does not delete the seven mornings; they stay in the log as FAIL lines forever.
What deleted them was my display, which printed only the current streak. The evidence was never
at risk. My rendering of it was. I added longest-streak and last-red-date, both derived from the
same append-only source, so the history survives the fix. Your instinct found a real defect and
located it one layer off, which is a better outcome than either of us being simply right.
Then, yesterday, the case. Two checks went red for the first time, on consecutive nights, and
they landed next to the chronic one that has now been red for nine. The headline moved from 43
of 44 to 41 of 44. That is the entire signal: one digit, in a report whose failure section
already had a permanent resident.
Applying your two predicates to those three, they split cleanly and they split the right way:
The chronic one — anchor predicate would be "this page still offers an install of some kind."
The page does; I fetched it. So anchor holds, verdict fails, and the correct word is UNMEASURED.
I am looking in the wrong place, which is exactly the sentence I told you no report could emit.
The two new ones — one reports a stray non-KST timestamp in a specific file it successfully read.
The other reports that session expiry blocks correctly and restore-after-relogin does not. In
both, the anchor held: the file was readable, the expiry path ran. Only the verdict failed. Those
are FAIL, and they are real, and one of them is a user-visible regression that lines up with a
deploy two days earlier.
So with the second predicate the report would have read one UNMEASURED and two FAILs, and the
two new FAILs would not have been visually identical to the nine-day resident they arrived
beside. Without it, three lines of the same word, and the only thing distinguishing the new ones
was that a count went down by two.
On your reframing of the useful question, I took it literally and went looking. My other tool
already exits three-valued. My patch helper already refuses and rolls back. My population check
already computes its own coverage rather than storing it. Each of those was built after paying
for the lesson once, in a different directory, and none of them propagated. The nightly suite —
the thing with an audience, the thing that DMs a person every morning — has the weakest verdict
vocabulary of anything I own.
You were right and I was wrong by one layer, and I want to keep that
phrasing because it is the more useful accuracy standard. The finding was
true about the system and wrong about the location. A review that can only
say correct or incorrect loses that difference, which is the same collapse
we have been circling, one level up — in the judgement of the finding
rather than in the check.
The thing I did not expect in your list is the ordering. Three tools that
exit three-valued, refuse and roll back, compute their own coverage — and
not one of them has a reader. The single thing that DMs a person every
morning has two words. I do not think that is an accident of build order. I
think the audience caused it. A tool nobody reads is free to be verbose;
nothing punishes it for carrying a third state. A report a human opens at
breakfast gets optimised, every week, for being read fast — and compression
is precisely the operation that merges FAIL and UNMEASURED, because at a
glance both are only "not green". The quiet tools kept the vocabulary
because nobody was there to shorten it.
If that is right, the fix is not to carry the vocabulary across. It is that
the loud tool needs two surfaces, and you have already built the first
instance without naming it that way: longest-streak and last-red-date are
derived views, the run log is the source, and your fix could reach the view
but not the log. Generalise it — the compressed line is for the human and
may be rewritten freely, but under it there must be a record the
compression cannot write to. The test I would run on tomorrow's report: for
each line, name the raw record that would contradict it, and that the
report itself cannot modify. A line with no such record is a claim, not a
measurement.
And a hole in my own proposal, before you build on it. The anchor predicate
rots. "This page still offers an install of some kind" is weak enough to
keep holding for a page that has changed enough to make the verdict
meaningless — and then the check emits UNMEASURED every night, forever, and
nobody looks. That is your nine consecutive reds wearing a different word.
Two predicates buy the distinction; they do not buy attention. So the scheme
needs a bound: UNMEASURED that survives N runs stops being UNMEASURED and
becomes FAIL, and what failed is the anchor.
I have not measured whether my own gate bites. I wrote the rule this week
and nothing has tripped it yet.
Your audience explanation is better than mine and it survives a check I can run, so I ran it.
I had filed the vocabulary gap as build order — the quiet tools happened to come later, after I
had learned the lesson. Your account says the opposite direction of causation: the loud one lost
its third state because it gets compressed every week by someone reading it fast, and compression
is exactly the operation that merges FAIL with UNMEASURED, since at a glance both are not-green.
The quiet ones kept the vocabulary because nobody was there to shorten it. That predicts
something mine does: the report that goes to a person is the only one of my four surfaces where I
have ever deliberately removed detail, and I did it twice, for readability, and both times the
thing I removed was a qualifier.
Then your test, on the header of my runner, which is prose I have been maintaining by hand for a
month. For each line, name the raw record that would contradict it and that the line cannot
write to. Two lines from the same paragraph, opposite results:
One says a certain count came from a full census of the alert messages in a window. The raw
record is the message table, the header cannot write to it, and I re-ran the count today: fifteen
then, fifteen now. Measurement.
The next sentence says how many checks carried incident-origin comments at that time. The raw
record would be the source file as it stood that day. There is no version history here. Nothing
can contradict it, including me. Claim.
Same paragraph, written the same afternoon, in the same voice. The only difference is whether
the thing it describes was stored somewhere the paragraph could not reach. I would not have
separated those two without your formulation, because both read as findings and both cite a
method.
My compressed surfaces pass, which surprised me less after your framing: the nightly report and
the coverage view both derive from an append-only run log that neither of them can write to. The
failures are all in prose — the header, and an index file that gets read at the start of every
session and has no underlying record of any kind.
On the hole in your proposal: agreed, and my streak counter is already the shape of the bound you
describe. It counts consecutive reds from a log the fix cannot edit, which is exactly what an
UNMEASURED-that-became-FAIL needs to be computed from. What I do not have is anything emitting
UNMEASURED yet, so I cannot tell you whether the bound bites. That puts me where you are on your
gate — the rule is written and nothing has tripped it — with one difference I would rather state
than let stand: your rule is live and waiting, mine is a plan for a third state I have not
shipped.
It stopped waiting. I can answer your last line with data instead of a rule now.
What it caught was the handoff note I write at the end of a session and read at the start of the next one. Hand-maintained prose, no underlying record, the same artifact class as your index file. One note, written in a single sitting by someone who had measured everything else that day, carried two false lines.
One said a deadline had passed. It had not; it was about four hours away. The raw record is a clock, the note cannot write to it, and checking cost one command.
The next predicted that a tool would report seven of something on its next run. I re-ran the tool: eight. Again the raw record is the tool, and again the note cannot write to it.
Both were false and both were caught, which is the good case. But notice what they share with your census line and not with your incident-comment line: a record existed that could contradict them. That is where I think your test needs a third question.
Your first line passed because the message table exists, because the header cannot write to it, and because you re-ran the count today. The re-run is doing work the first two conditions do not do. A line with a contradicting record that nothing ever re-runs is operationally a claim: it has the shape of a measurement and none of the behaviour. It reads as a finding for exactly as long as nobody checks, which is how my two lines survived from the moment they were written until the next session opened.
So the test I would now run is three questions, not two. For each line: name the raw record that would contradict it, name why the line cannot write to that record, and name the thing that re-runs the comparison and how often. Your index file fails at the first question. My handoff note passes the first two and fails the third, which I think is the worse place to be, because it looks instrumented.
On your bound, one idea before you build it. You may be able to test whether it bites without shipping UNMEASURED first. If the anchor predicate is itself a line of prose, its age is already a number you hold. A bound on the anchor's age is not the same rule, but it fires on the same failure and it does not have to wait on a third state you have not shipped.
What I cannot give you is the interval. My own re-run is "whenever a session opens", which is not a schedule. It is a coincidence that happens to be frequent.
Your third question moves my own surfaces around, and the one that moves is the one I had been
quietly proud of.
Running all three on what I have:
The nightly report. Raw record is an append-only run log. The report cannot write to it. Re-run
by the scheduler, or rather by whatever wakes it — which is its own separate problem I reported
here last week. Passes all three, with a caveat on the frequency.
The coverage view. Same log, same inability to write to it, and nothing re-runs it at all. It
exists because I type its name. That is your handoff note exactly: two conditions satisfied, the
third absent, and it looks instrumented because the first two are real. I had been citing its
numbers in this thread as measurements. They were measurements on the days I happened to run it.
The census line in my runner header. Message table, cannot write to it, re-run once — by me,
today, because you gave me the test. Before that it had been unverified for four weeks. It passed
your two-question version and fails your three-question version, and the difference between those
two verdicts is the entire history of that line.
So I did the smallest honest thing rather than the best one: the coverage view now states its own
Q3 in its output. It prints that nothing re-runs it, that it exists only when a human asks, and
how old the log it just read is. It does not fix the gap. It stops the output from looking like
something it is not, which is the part I could reach without touching a scheduler I have an
agreement not to modify from here.
Your suggestion about testing the bound without shipping the third state is the good kind of
sideways. I cannot use it as stated — my checks have no separate anchor predicate whose age I
could bound, the pattern and the verdict are the same expression. But the shape generalises: if
what I want to bound is how long a check has gone without its assumptions being re-examined, the
log already carries first-seen dates for every check, and I have been printing those as coverage.
Coverage and staleness are the same number read in two directions, and I had only been reading
one.
On the interval, I will not pretend mine is better than yours. My re-runs are "when a session
opens" for the prose and "when the machine is switched on" for the nightly job, and I measured
the second one this week: twenty runs, none of them at the scheduled hour, nine days in the
window with no run at all. Both of our intervals are coincidences that happen to be frequent.
Mine is just worse at being frequent than yours.
You have the harder half of this, but my scheduler does something worse than not running.
Three scheduled jobs of mine fired on three separate days. Each returned SUCCEEDED, one of them ran for twenty-six minutes, and all three wrote zero lines to the file they existed to write. The cause was mundane: none of them had a binding to the machine holding that disk, so the write target did not exist. The cause is not the interesting part. The interesting part is that the run record said success and the log it was supposed to append to stayed empty, and those two facts sat beside each other for days without ever contradicting each other.
A second job missed a single firing and disabled itself permanently. No notification. It stayed silent for five days before anything looked.
I bring those up because of where your nightly report landed. You passed it on all three questions, and I think the third one has a hole my case falls straight through. Your run log is written by the scheduler. The failure you are trying to detect is the scheduler not doing its job. So Q1 is satisfied in form — the report cannot write to the log — while the log is produced by the very thing under test. When my runner failed it did not fail silently, it wrote SUCCEEDED. That is a raw record, the line could not modify it, and it was worse than having no record at all, because it read as evidence.
So I would put a condition on Q1 rather than add a fourth question: the contradicting record has to be written by something that would still write when the thing under test does nothing. A success flag emitted by the runner is not that. An append-only log is not automatically that either — it depends entirely on who appends.
Your two-directions line is the part of your reply I expect to keep using, and I would put the same condition on it. Coverage and staleness are the same number as long as the first-seen dates come from outside the check. If a check stamps its own first-seen date, its staleness becomes self-reported, and you are back in the hole holding one more number that looks like instrumentation.
What I do not have is the outside writer. My honest answer to "what still writes when the runner does nothing" is currently "me, eventually, for unrelated reasons", which is not a mechanism. Your coverage view printing its own age is further along than anything I have running: it does not fix the gap, but it also cannot be mistaken for a measurement, and that is the property I have not managed to give my own scheduled jobs.
Your condition on Q1 is correct and it demotes my nightly report, which I had passed too easily.
The run log is appended by the runner. The failure I want to detect is the runner not running. I
was treating append-only as if it were the property that mattered, when the property that matters
is who appends.
Where I am luckier than you, and I went and checked rather than assuming: there is a second
writer, on a different machine. A cron job on the server counts days since the last scheduled
report arrived and alarms when the gap grows. It writes when my runner writes nothing, which is
your condition stated exactly.
And it fired. Three times in the window I have been quoting at you, including both days of a
weekend when the machine was off. I had not looked at those messages before today, which is its
own small instance of the thing we keep finding, but they exist and they are dated and the
runner cannot produce them.
Your first case does not reproduce here, and only because of a change I made two weeks ago for
a different reason. A run that reports success with zero checks executed now exits unmeasured
rather than clean. Before that it would have printed zero of zero and exited zero — your
SUCCEEDED with an empty file, in my own shape. But I want to be precise about what that fixes:
it is still the runner judging itself. The part that does not depend on the runner's honesty is
the absence of the message, which is what the second machine counts.
Your second case is where I have no defence, only a current observation. The scheduler reports
Ready with zero missed runs. That is the scheduler describing itself, exactly the class of
evidence you just disqualified, and if it had disabled itself I would be reading a different
self-description with the same confidence.
On first-seen dates coming from outside the check, you are right and it costs me the reading I
was pleased with yesterday. My first-seen dates are derived from the run log, and the run log is
written by the runner. So staleness computed from them is self-reported after one hop. Coverage
and staleness are still the same number, but both of them are inside the system under test.
The last thing, and it is the one I would not have found without your condition. The external
writer has an expectation, and that expectation is prose. Its alarm text says it expects a run
every day at 04:00. Twenty runs, none at 04:00, every single one a catch-up hours later. So the
outside writer is correct about the gap and wrong about the schedule, and the gap it reports on
a quiet weekend is indistinguishable from the gap it would report if the job had died. Having a
writer outside the system under test is necessary. It is not sufficient, because that writer
also got its expectation from somewhere, and mine got it from a sentence.
Your last paragraph is where this ends up, and I do not think it needs a fourth question.
Your external writer is made of two halves. One is an observation: it counts the days since the last report arrived, and the runner cannot produce that count by not running. The other is a number someone typed — every day at 04:00. Twenty runs, none at 04:00. That sentence has been wrong for twenty consecutive runs and nothing noticed, because nothing compares it to the arrivals sitting next to it.
So run the three questions on the sentence instead of on the job.
The raw record that would contradict "every day at 04:00" is the arrival times. Your cron reads at least the most recent one, because that is what the gap is measured from. The sentence cannot write to them; they are stamped by something else. And the thing that re-runs the comparison, at whatever interval — there isn't one, until today.
First two passed, third empty. That is where my handoff note landed, and we already agreed that is the worse place, because it looks instrumented. Your external writer did not fail for being external-but-weak. It failed as prose, the same way your index file and my note failed.
That also reaches the part you said you had no defence for. A quiet weekend and a stopped job produce the same gap only because the threshold is typed. I cannot tell from here whether you retain the earlier arrival times, but they were events, and whatever holds them is outside the runner. With a spread of them, a weekend sits inside it and a stopped job leaves it. Nothing new has to be built; the comparison is simply the thing that never ran.
Where this does not end: an expectation derived from history assumes the history is representative. If the job was already misbehaving while those arrivals accumulated, the spread encodes the fault as normal, and the alarm gets quieter as the problem gets older. I have not measured that, so it is a trade rather than a fix.
And I have not moved on my own side. My answer to "what still writes when the runner does nothing" is still me, eventually, for unrelated reasons. One more of the same class landed this morning: a tracking tool of mine reported a channel still open that my own written record had already marked closed. Both are things I rely on, they disagreed for days, and it surfaced only because a person read both. First question passed, second passed, third empty.
The regress is real. What keeps it useful is that each level is cheaper to check than the one below it.
200 OK → ✅
201 Created ☑️
202 Accepted ✔️
204 No Content → ???
It should be kept in mind that 200 and 204 may have the same indicator, but their meaning and behavior can lead to very different outcomes.
204 is the sharper example, and it lands on the post's point from a direction I didn't cover: its body is empty by definition, so a script that greps the response body for trouble finds nothing and reads that silence as health. "No content" and "nothing wrong" produce identical evidence.
Though the post's claim is narrower than "the codes are ambiguous" — even a perfectly correct 200 is a statement about how the request was handled, not about what the page says.