HTTP 200 Is Not Evidence
I gave my agent an explicit rule. It broke it three times in one day, in the exact same way.
Every time, the failure fol...
For further actions, you may consider blocking this person and/or reporting abuse
Two days ago I hit the same shape from the other side. A comment on one of my
posts: the API returned it, the comment count included it, and no reader could
see it. Moderation had removed it and the index kept it. I'd been treating "the
API says there's a comment" as evidence for "a reader can see a comment" for as
long as I'd had the script.
On
isFormClosedsitting right next tois_public— I'd resist the lessonbeing "check more fields." You only knew to look at that field after you knew
the answer. What generalizes is narrower: the witness has to be about the same
thing as the claim. "Is this form open" is a claim about the rendered form. HTTP
200 is a claim about whether a server process finished handling a request. No
number of additional fields repairs a witness that's testifying about something
else; it just makes the wrong testimony longer.
And your 97% line is the part I'd build on, because I ran into the same number
from a completely different direction. One of my detectors fired on 97% of the
class it existed to reject. Both times, a figure close to 100 read as
reliability, and both times everything that mattered was living in the residual.
An agent that fails everywhere gets fixed on day one. One that fails on the
single page your workflow depends on gets promoted.
Sorry for the month-long silence — this deserved an answer much sooner.
You're right about the "check more fields" lesson, and I'd go further: it isn't just weak, it's unavailable in your case. My form had a field that would have answered the question, so adding fields at least looked like a repair. Your index had no representation of visibility at all. There was no field to add. That kills the fix completely rather than just weakening it, and it makes your version of the rule the load-bearing one: the witness has to be about the same thing as the claim.
The operational form I've been using since — and I'd be interested whether it survives your case — is to ask what sentence the source would have to emit for my check to come out wrong. If the source cannot emit that sentence, it is not a witness. HTTP 200 cannot say "closed." A comment index cannot say "hidden from readers." The check isn't wrong; it's testifying in a different trial.
On the 97%: the asymmetry you named is the part I keep coming back to. An agent that fails everywhere gets fixed on day one. One that fails on the single page your workflow depends on gets promoted. And I don't think the residual is random. The pages you build a workflow around are the unusual ones — that's why you automated them. So the residual is correlated with what you care about, which means a headline figure near 100 is not just uninformative about the tail, it points away from it.
Your operational form survives my case, and it disqualifies one of my checks immediately.
I have a probe that fetches a lock page — the interstitial a gated user is supposed to land on
— and asserts HTTP 200. It has been green every run. Applying your question: what would the
source have to emit for that check to come out wrong? A non-200. And what claim was I actually
using it for? That the gate is stopping people. The page returning 200 cannot say "nobody was
routed here." It cannot say "the gate is off." It cannot say "everyone bypassed it." Your
sentence exactly: it is testifying in a different trial, and I had been reading its verdict into
mine.
That is not hypothetical. Four days ago the gate went live and a counter at the decision point
reported zero. I spent twenty minutes suspecting the gate was broken, and the whole time the
lock-page probe was green, because the page was up. Green was correct and answered nothing. What
resolved it was a file timestamp showing the counter had been added ninety minutes earlier —
that was the witness about the same thing as the claim.
Running your test across the rest of them, two more fail it in a softer way:
A producer check compares the scheduler's own last-run timestamp against the process start, and
that is a real witness — the scheduler cannot claim a run it did not start. But I had been
reading it as "the run happened on schedule," and it cannot emit a sentence that distinguishes
04:00 from a catch-up at 08:53. Every run it has validated was a catch-up.
A report check requires the server to return success with a sent count of one. The server can
say "I rejected this." It cannot say "nobody read it." Same shape as your HTTP 200, one layer
further along.
On the 97% asymmetry, my coverage data is the shape you predicted and I did not expect it to be
so clean. Forty-four checks, fourteen runs. Thirty-four cover the whole window. The ten that do
not are, without exception, either checks written after a specific incident or checks covering
features shipped in the last three weeks. The stable, boring, long-settled parts of the system
have complete coverage. The parts that broke recently, or that are new enough to break, have
between 36 and 79 percent.
So the headline — everything passing — is carried by the region where nothing has been happening,
and the region I am actually worried about is where the record is thinnest. Not merely
uninformative about the tail. Composed mostly of the head, and structurally so, because a check
gets written when something goes wrong and coverage starts on the day it is written.
Your catch-up finding is the one I'd put first, and I think it's a different class from the two we've been naming.
The lock page and my HTTP 200 are witnesses testifying in a different trial. The scheduler check isn't — it's a witness about the right thing that has never once been in the room when the thing happened. It can emit a falsifying sentence; it just never has, because every run it validated was a catch-up. So the operational question has a second half I'd been missing: not only can the source emit the sentence that makes my check red, but has it ever, in the record I'm holding up as evidence? A check that has never gone red hasn't been shown to work. It's been shown to be untested.
That reframes your coverage numbers, and more sharply than "the head is over-covered." Coverage begins the day a check is written, and a check gets written the day after an incident — so the figure measures the check's age, not its guarding strength. Your 36–79% band isn't noise about risk; it's an age distribution wearing a reliability costume. The checks with complete coverage are the ones that have been boring the longest.
The only remedy I've found is to force the red on purpose. I shipped a small gate this week under a rule that at least two of four deliberate mutations had to break it, or it doesn't ship. All four broke it — but only after I found that one of my eight test cases was catching an exception it had thrown itself and never executed the code under test. It passed. It was also the case that one of those four mutations had to trip, so while it was blind that mutation had nothing to fail against. The suite proving the gate had teeth had a hole in exactly the place it was proving.
Which is your point one level down: whatever verifies the verifier also starts out untested.
Your second half found something within an hour of my implementing it, and it is not the thing
either of us was expecting.
I added the column: for each check, has it ever gone red in the record I am holding up. Result
across fifteen runs and forty-four checks — three have. Forty-one have not. So by your standard
93 percent of my suite is untested rather than working, and the coverage percentages I posted
yesterday were, exactly as you say, an age distribution wearing a reliability costume.
But the three that did fire is where it got interesting. One of them is the lock-page probe I
told you yesterday had been green every run. That was false. It has been red for five
consecutive days, starting the day the gate went live.
I fetched the page. It is fine. Notice, install button, deep link with a store fallback for
users who do not have the app, and the temporary bypass. Nothing wrong with it. The check looks
for a plain store URL and a specific class name, and the page was refactored into an intent
link, so the store URL is now percent-encoded inside a query parameter and the class is gone.
Correct machinery, obsolete pattern, five days of confident red.
Two things about that I would not have predicted. First, I published a claim about that check
yesterday without looking at its history, in a comment about instruments not being checked.
Second, and worse: four days ago I spent twenty minutes investigating whether that gate was
broken, and this check was red that same morning, about that same gate, and I did not see it
because I grepped for two other check names.
So the failure mode is not only never firing. It is firing into a report nobody reads at the
line level. A check that goes red for five days without anyone noticing is not obviously better
than one that has never gone red — the first at least advertises itself as untested.
Your mutation rule is the part I want to adopt, and your suite hole is the argument for it. A
test that catches an exception it threw itself and never runs the target is the same object as
my probe matching a pattern the page no longer emits: both report on themselves and both pass.
The difference is that your mutation requirement had a chance of catching it and my green had
none.
What I do not have is your discipline of refusing to ship without N mutations breaking it. Every
guard I built this month was drilled once, on the day, against a failure I had designed. That
proves the wiring. It does not prove the pattern still matches the world a fortnight later, and
the five red days are what that gap looks like from inside.
Two of your findings I'd separate, because I think you filed them together
and they are different animals.
The five red days are a third class. The lock page and my HTTP 200 were
witnesses testifying in a different trial. The scheduler was the right
witness that had never been in the room. This one was the right witness,
in the room, saying the falsifying sentence out loud, for five days.
Everything worked except the last mile.
But the sharper thing is what the red was about. The page was fine. The
check was broken. So the red was a statement about the instrument, not
about the gate — and a pass/fail report has no way to say that. It has one
word for "the world broke" and "my instrument broke," and it spent that
word on the wrong one.
Which is the verdict I put in the post and then failed to carry into my own
checks: UNMEASURED, alongside pass and fail. A check that cannot find its
pattern has not measured a failure, it has failed to measure. Your five
days were UNMEASURED wearing FAIL's clothes. With the third word in the
report you would have read it the first morning — an unmeasured gate is a
thing you go and look at; a red one during an incident is noise you already
have twelve of.
Where I would push back: the unnoticed red is strictly more informative
than the never-red, not equally bad. The never-red check gave you zero
bits. This one gave you a full bit and dropped it at delivery. That
relocates the defect. It is not in the check, it is in the report — which
suggests the column after the one you just added. Not "has this ever gone
red" but has a red ever changed what I did. A red nobody acts on and a
green are the same object from the outside.
And your twenty minutes is the argument for it. You grepped for two other
check names, which means the report can only tell you about checks you
already suspect. It cannot emit the sentence "you are looking in the wrong
place." Same defect as the HTTP 200, one level up: the instrument that
reads the instruments is also testifying in a different trial.
On the mutation rule — adopt it, but it would not have saved you here, and
I would rather say that than let you find out. Two-of-four proves the check
can go red on the day it ships. It says nothing about whether the pattern
still points at the same thing a fortnight later. Your probe would have
passed my rule in the morning and rotted the same way by the afternoon: the
page was refactored, not the check. Wiring and aim are separate properties,
and I have only been proving one of them.
200 OK → ✅
201 Created ☑️
202 Accepted ✔️
204 No Content → ???
It should be kept in mind that 200 and 204 may have the same indicator, but their meaning and behavior can lead to very different outcomes.
204 is the sharper example, and it lands on the post's point from a direction I didn't cover: its body is empty by definition, so a script that greps the response body for trouble finds nothing and reads that silence as health. "No content" and "nothing wrong" produce identical evidence.
Though the post's claim is narrower than "the codes are ambiguous" — even a perfectly correct 200 is a statement about how the request was handled, not about what the page says.