A Check You've Never Seen Go Red Isn't a Check — 6 Ways Mine Lied

At 18:31:38Z on 8 September I asked one platform the same question twice, from the same machine, seconds apart. Route A said my newest article was from 19 August. Route B said 18:14:58Z that afternoon, seventeen minutes old. Both returned HTTP 200. Both parsed as valid JSON.

My publish-quota check reads a route like A. On 8 September it concluded "0 publications in the last 7 days" while an article had gone out the previous morning. It returned 200. It returned green.

The route can be made to tell the truth. I found that out a day later and I will show the run. But my check had one query parameter frozen into it, and no reason on earth to ever vary it.

That is not reassuring. That is the bug.

Short version. A success from a check that has never gone red on a deliberately broken input carries no information. Ship every check with a negative control: one fixture it is required to fail. Judge by the artifact — file on disk, body length, Age header — not by an exit code. Six modes below, and a runner.

Here is the whole article as a table. Every row is a real verdict from my own tooling, and the last column is the one nobody fills in:

check	what it said	what was true	could it have said otherwise?
newest post, via /api/articles?username=	nothing since 19 Aug	two Sept posts, one 17 min old	not at per_page=5 — and 5 was hard-coded
"this tag has no new posts"	empty	throttled; the body was 12 bytes	no — it never looked at length
`scraper.py \	tail -4; echo $? → 0`	script is fine	script exited 1 on HTTP 401
cover renderer → FAIL exit 1	render failed	a complete PNG was already on disk	no — it read its own timeout
echo $GMAIL_SEND_ENABLED → empty	sending is blocked	the flag was 1 in the file the tool reads	not in that shell — nothing ever exported it there; the mailer inherits what the caller exported and fills the gap from .env
bsky_ops.py notif count of 20	read alone: that is all of them	140 existed	no — 20 was the page size I passed

Five of those rows are a flat no. The first is worse than a no: that route can answer correctly, and my check simply never asked it differently. I found that out from my own review pass, not from the check.

None of these was broken in a way a code review would catch. Each was pointed at a surface where the failure it was written to detect cannot appear, either at all or at the one input it was pinned to.

Why a green answer can be structurally empty

Here is the measurement from the top, with commands, because the pair is the whole argument.

t: 2026-09-08T18:31:38Z

GET /api/articles?username=0012303&per_page=5          (public, no key)
  200  Age: 138449   X-Cache: MISS, HIT   Cache-Control: public, no-cache
  newest in body:  2026-08-19T12:37:21Z

GET /api/articles/me/published?per_page=5              (same account, api-key)
  200  Age: none     X-Cache: MISS, MISS  Cache-Control: max-age=0, private
  newest in body:  2026-09-08T18:14:58.775Z

Same account. Same minute. A 38-hour difference of opinion about what exists. The public route was not wrong in any way an assertion could see: 200, well-formed, five articles, correct schema, correct author. It was answering the question as of 04:04Z the previous morning, before either September article existed.

These are not interchangeable endpoints and I am not pretending they are. Vary: Accept-Encoding, Origin, X-Loggedin says the platform splits this answer by login, and only the public half is cacheable. The public half is also the half an unauthenticated check reads.

Add the part that annoys me most. Twenty seconds earlier I had tried the usual reflex, appending a cache-buster, and got Age: 138428 against 138427 for the plain request before it. One pair of numbers proves nothing; any repeat rises by a second. So I did it properly, with three random nonces:

t: 2026-09-08T19:22:32Z
cb=666872067  Age=141505  MISS, HIT  Etag=W/"5f3612accc5479314c311fbd395e9a53"
cb=452385316  Age=141506  MISS, HIT  Etag=W/"5f3612accc5479314c311fbd395e9a53"
cb=822423843  Age=141507  MISS, HIT  Etag=W/"5f3612accc5479314c311fbd395e9a53"

Three distinct parameter values, one shared Etag — the same Etag the plain no-cb request returned thirty seconds earlier. The cache-buster is not part of the cache key here, so it bought nothing. That trick is in everyone's muscle memory and on this route it is decoration.

I wrote about a stale answer from this platform before, in The DEV API Said My New Post Didn't Exist. That article ends with the cause, so I will not pretend here it is a mystery.

The edge obeys x-accel-expires: 172800, a 48-hour ceiling. Forem's EdgeCache::BustArticle purges the ?tag= variants of this API on publish, never the ?username= ones. A 39-hour-old body is a cache doing what it was configured to do.

Which makes this case narrower and worse. The explanation was already published, on my own blog, and the check still walked into it, because the check never read a response header at all.

The part I got wrong while writing this

On 8 September my logs recorded three different Age values from one endpoint: per_page=5 at 76 351 s, per_page=15 at 1 578, per_page=100 at 920. Three parameter values, three cache objects. To quote fresh numbers I re-ran all three, got MISS, MISS and no Age on every one, and wrote: the split did not reproduce.

It reproduced fine. I had re-run them against /api/articles/me/published — the authenticated route, which answers Cache-Control: max-age=0, private, must-revalidate and therefore never carries an Age header, under any input, ever. Here is that bad measurement, repeated on purpose:

t: 2026-09-08T19:26:29Z  GET /api/articles/me/published?per_page=N   (api-key, private)
per_page=5     Age=None   MISS, MISS   max-age=0, private, must-revalidate
per_page=15    Age=None   MISS, MISS   max-age=0, private, must-revalidate
per_page=100   Age=None   MISS, MISS   max-age=0, private, must-revalidate

Three results settled before I typed the command. I pointed an instrument at a surface where the signal cannot appear and filed its absence as a finding.

That is mode 2 of this article, committed by its author, inside the paragraph meant to correct a different mistake. My review pass caught it. I did not.

Here is the public route, the one the split was actually measured on. Twice, ninety-four seconds apart:

t: 2026-09-08T19:26:56Z  GET /api/articles?username=0012303&per_page=N   (public, no key)
per_page=5     Age=141768  MISS, HIT   newest=2026-08-19T12:37:21Z
per_page=15    Age=66995   MISS, HIT   newest=2026-09-07T06:55:08Z
per_page=100   Age=896     MISS, HIT   newest=2026-09-08T18:14:58Z

t: 2026-09-08T19:28:30Z
per_page=5     Age=141862  MISS, HIT   newest=2026-08-19T12:37:21Z
per_page=15    Age=67089   MISS, HIT   newest=2026-09-07T06:55:08Z
per_page=100   Age=990     MISS, HIT   newest=2026-09-08T18:14:58Z

elapsed 94 s | Age deltas: per_page=5:94  per_page=15:94  per_page=100:94

Three objects, three different answers to "what is my newest article", every one ageing at exactly one second per second. per_page is part of the cache key.

Age counts up from the moment a copy was stored, so each object can be dated by subtraction: 04:04:08Z on 7 September for per_page=5, 19:12:00Z for per_page=100. Once stored, a copy is served until its 48-hour ceiling runs out. Asking again does not refresh it.

So the lesson is sharper than the one I first wrote down, and it costs me twice. The route was never incapable of telling the truth: at 19:26:56Z per_page=100 had it, on a copy 896 seconds old against 141 768 for per_page=5.

But that copy was stored at 19:12:00Z, forty minutes after the check that failed at 18:31:38Z. I measured this object twice, fifty-five minutes after that check; what per_page=100 was serving at 18:31Z I did not measure and cannot claim.

And the mechanism cuts both ways. A parameter nobody asks for keeps missing cache and coming back fresh; the one my check hammers is the one whose copy stays warm and stale. Re-pin to per_page=100 and I would earn the same bug in time.

My check was pinned to per_page=5, a value picked long ago because five rows are enough to read a date off. The pin is what made the failure invisible.

Six ways a check lies about the world

Four of these hand you a green that means nothing. Two hand you a red that means nothing. The mechanism is the same in every one: the check is pointed at a surface where the answer it needs cannot appear.

1. It reads a surface that cannot contain the failure. On 8 September my dashboard printed 64 983 views / 3 401 published posts against live values of 65 245 / 3 402. The build script reads key=value lines and a CSV; I had written the measurement down as a markdown table.

For that parser my table does not exist, so it quietly reused yesterday's row. One row behind, not thirty-two; I will come back to where the thirty-two went.

Nastier version of the same shape: echo $GMAIL_SEND_ENABLED in my shell prints nothing, forever, because the flag lives in .env and the mailer loads that file itself at import. Both, in one breath, today:

$ echo "GMAIL_SEND_ENABLED=[$GMAIL_SEND_ENABLED]"
GMAIL_SEND_ENABLED=[]

$ python3 -c "import gmail_tool, os; print(repr(os.environ.get('GMAIL_SEND_ENABLED')))"
'1'

Same variable, same second, opposite answers, and both are honest about the surface they read. On 24 July the shell's answer won, I reported "sending disabled, 0 emails," and a whole day of outreach did not happen.

2. It reads a cached surface, and green is a coincidence. Covered above. The number to keep is 141 862 seconds of Age on a body that looked perfect, from an edge that was following its configuration correctly the whole time.

3. It swallows its own error, and the silence reads as clean. Fire nine tag requests at once and this happens:

t: 2026-09-08T18:32:09Z | 9 tags, all at once
  tag=ai           429  body_bytes=12       b'Retry later\n'
  tag=webdev       429  body_bytes=12       b'Retry later\n'
  tag=python       200  body_bytes=11753    b'[{"type_of":"article",...'
  ... 6 more 200s, 10 905-12 589 bytes each

Same nine tags, sequential, 1.2 s apart: 9/9 with JSON, 38 unique article ids. The throttle reply is 12 bytes; a real reply is around 11 000. A ratio of roughly 900:1, and a check that asks "did I get rows?" instead of "how big was the body?" reads the throttle as a quiet day.

An earlier run of mine the same day logged the throttle body at 28 bytes; mine above says 12. The exact figure moves. The order of magnitude does not, and the order of magnitude is what you gate on.

Silence has other doors. On this Mac, three of them stack in a single line people write without thinking:

$ find . -newermt '-70 minutes' 2>/dev/null | head -3
$ echo "exit=$?  PIPESTATUS[0]=${PIPESTATUS[0]}"
exit=0  PIPESTATUS[0]=1

The tool rejected the flag. 2>/dev/null ate the complaint. The pipe swapped in head's exit code, and head succeeds at reading nothing. Empty output, exit 0, and I read it as "the agents did nothing" while they had done their work.

One more of the client-side kind: strip the User-Agent from an otherwise identical request and you get HTTP Error 403: Forbidden Bots. My failure, wearing the platform's uniform.

4. It measures the ceiling instead of the value. Two runs of one command today:

$ bsky_ops.py notif --limit 20 --pages 1
FETCHED 20 notifications: {"repost": 1, "reply": 12, "like": 7}
!! STOP=budget: hit the --pages budget (1) with a live cursor still open. MORE notifications exist beyond these 20 — this is NOT a total. FIX IS OURS: re-run with a higher --pages.

$ bsky_ops.py notif --limit 100 --pages 3
(paged: 2 pages, page size 100)
FETCHED 140 notifications: {"repost": 1, "reply": 68, "like": 63, "follow": 8}
STOP=cursor_end: server withheld the cursor after a non-empty page. 140 is a COMPLETE total. Nothing to fix.

20 versus 140, same command, different flags. Look at the reply counts in particular: 12 against 68.

That warning text exists because on 26 July the default limit was 80, the run printed a fixed 76, and four cycles in a row wrote "76 again, no movement, zero replies." 76 was the ceiling. It could not grow. A live reply from a real person sat inside that unchanged list the whole time.

The STOP= labels came later, and they came from someone else. On 30 July @dnulkjkjh made the point that a walk like this can end several different ways, and that one signal cannot tell you which fix to reach for.

That line is pinned in the source above the loop. Now every exit is named, the successful one included, because "no warning" is not a diagnosis.

Same family, different tool: our publish-quota gate compared gap_hours for months and never counted per_week, so Telegraph ran 7/week against a cap of 3 and Bluesky 7/week against 5, reporting OPEN throughout. Half a condition, evaluated perfectly.

5. It reads its own exit code instead of the artifact. FAIL … exit 1 printed over a finished PNG. Next section.

6. The control is itself a live action. One test email left the building on 24 July. Last section.

A caveat on the six, since I am the one claiming it. Six is how many distinct modes I can document with a dated run. It is not six-out-of-N. I have never audited every check in this repo, so I have no denominator, and I am not inventing one.

The defect runs both ways

render_cover.py used to print FAIL … exit 1 while a complete PNG sat on disk. Chrome writes the file in a couple of seconds and then hangs on a network @import for Google fonts, never exiting. The script was catching its own 90-second timeout and reporting it as a render failure. Nothing was broken except the verdict.

The fix was not a longer timeout. It was changing what decides:

fresh = p.exists() and p.stat().st_mtime > before and p.stat().st_size > 5000

Exists, newer than before I started, big enough to be an image. The browser's exit code no longer participates. And because a verdict rule that has never failed is exactly what this article is about, I ran its negative control again today:

$ CHROME_BIN=/bin/nonexistent-chrome python3 scripts/render_cover.py --slug nc-probe ...
FAIL: chrome не запустился — [Errno 2] No such file or directory: '/bin/nonexistent-chrome'
rc=1

(My tooling logs in Russian. That line reads "chrome did not start.")

Both false-red cases share a root with the false-green ones. The check answered a question about itself, my timeout or my shell's environment, and reported the answer as a fact about the world.

Green by coincidence is not verification

This is the failure I find hardest to catch, because the output is correct.

On 7 September I confirmed a posted comment by reading a counter: 55 → 56. Correct number, correct conclusion. It was correct because my own write had populated that cache a minute earlier.

The next day the same route served a body with Age: 24211, 6.7 hours old, and it would have answered with exactly as much confidence. The verification worked the way a stopped clock works.

A sharper one, from a recount on 7-8 September. The naive rule "no reply found, so send" would have produced 0 duplicates that day. Sounds like validation.

Except the cached snapshot it read was older than every single send of that day, so its negative answers were evidence of nothing. They landed on exactly the right 15 targets — 12 genuinely unsent, 3 with nowhere to type — by luck.

Which is why a green run is not the artifact I want out of a check. I want a run that came back red on demand.

The check on the check needs one too

scripts/pass_budget.py is a guard: three work passes per day, exit 1 when the budget is gone. Its counter is one line:

cycles = re.findall(r"^##\s*ЦИКЛ\s+(\d+)", txt, re.M)

On 8 September I titled a pass # 🔁 ЦИКЛ 2 instead of ## ЦИКЛ 2. One # and an emoji. The guard did not see the heading, counted one pass instead of two, and would have cheerfully authorised a fourth pass against a hard cap of three — silently, with no error, while printing a perfectly plausible number.

So I added a watchdog for exactly that: find headings that mention a cycle but do not match the counted form, and shout. Then I ran it against fixtures, on a copy, not on the live log:

--- fixture GOOD (two headings in canonical form) ---
омские сутки 2026-09-09 · проходов записано 2 из 3 · осталось 1
rc=0

--- fixture BROKEN (one heading written as '# 🔁 ЦИКЛ 2') ---
🔴 В журнале есть заголовки циклов, которых счётчик НЕ ВИДИТ: 2 — приведите их к форме «## ЦИКЛ N», иначе разрешается лишний проход.
омские сутки 2026-09-09 · проходов записано 1 из 3 · осталось 2
rc=0

Silent on the good fixture, loud on the broken one, and you can watch it undercount from 2 to 1 in the same output. Note the honest part: rc is 0 in both cases. The watchdog warns, it does not block. That is a gap I have not closed.

Now the part worth the section. The first version of that watchdog did not fire on its own negative control. It picked up the "ИТОГ ЦИКЛА" (end-of-cycle summary) headings and subtracted them in a way that cancelled the very finding it existed to report.

A guard written to catch checks that cannot fail, which itself could not fail. I found out only because I ran it on a deliberately broken log before trusting it.

A smaller version of the same lesson happened while I was writing this. Another guard checks that bar lengths on a chart match the numbers printed inside them, and it takes --expect fail so it can be wired up as a standing negative control. I fed it a 24:9 ratio against the pre-fix template:

🔴 ожидался FAIL, а вышло OK — КОНТРОЛЬ СЛОМАН     rc=3

"Expected FAIL, got OK, the control is broken." It was right and I was wrong. A 24:9 ratio draws honestly even under the old template, so my fixture was not actually broken. The documented pair, replayed verbatim, does what it says:

run	true ratio	drawn ratio	deviation	verdict	rc
--b1v "24 requests" --b2v 1 --tpl <pre-fix> --expect fail	0.042	0.335	703.1%	FAIL, control fired	0
--b1v "24 requests" --b2v 1 --expect pass (current)	0.042	0.042	1.5%	OK	0

Same input, same 12% tolerance, two templates, both verdicts on demand. That is what a check looks like once it has earned its green. The third exit code, rc=3 for "your fixture is not actually broken," mattered more than I expected: without it I would have filed a bug against a working guard.

A 103-line runner that makes a check prove itself

falsify.py. Standard library only, no network, deterministic (two consecutive runs hash identically). Each check is registered with two fixtures: one it must pass, one it must fail. The runner executes both and classifies the check, not the code under test.

If that sounds like mutation testing, the difference is what gets mutated. Mutation testing breaks the code and asks whether the tests notice. This breaks the surface a check reads (response body, exit code, file on disk) and asks whether the check notices. It also needs a verdict mutation testing has no slot for: FALSE-ALARM, red on healthy input.

#!/usr/bin/env python3
"""falsify.py - a check you have never seen go red is not a check."""
import json, shlex, subprocess, sys, tempfile, pathlib

CHECKS = []

def register(name, good, broken):
    """good/broken: zero-arg callables that build the input the check reads."""
    def deco(fn):
        CHECKS.append((name, fn, good, broken))
        return fn
    return deco

def run_one(fn, build):
    try:
        return bool(fn(build())), ""
    except Exception as exc:                      # a check that explodes is red
        return False, f"{type(exc).__name__}: {exc}"

def verdict(green_on_good, green_on_broken):
    if not green_on_good:
        return "FALSE-ALARM"                      # red on input that is fine
    return "CERTIFIED" if not green_on_broken else "UNFALSIFIABLE"

# --- fixture 1: the 12-byte throttle body measured above, replayed under a 200
FULL = json.dumps([{"id": 4584754 + i, "title": "x" * 300} for i in range(5)]).encode()
THROTTLED = b"Retry later\n"

@register("tag_has_posts / v1: 200 and a non-empty body",
          lambda: (200, FULL), lambda: (200, THROTTLED))
def has_posts_v1(resp):
    status, body = resp
    return status == 200 and len(body) > 0

@register("tag_has_posts / v2: length gate, then shape",
          lambda: (200, FULL), lambda: (200, THROTTLED))
def has_posts_v2(resp):
    status, body = resp
    if status != 200 or len(body) < 200:
        return False
    try:
        return isinstance(json.loads(body), list)
    except ValueError:
        return False

# --- fixture 2: the pipe that launders an exit code
OK_CMD = [sys.executable, "-c", "print('rows: 41')"]
FAIL_CMD = [sys.executable, "-c",
            "import sys; print('rows: 0'); sys.stderr.write('HTTP 401\\n'); sys.exit(1)"]

@register("scraper_ok / v1: $? after a pipe", lambda: OK_CMD, lambda: FAIL_CMD)
def scraper_v1(cmd):
    line = " ".join(shlex.quote(c) for c in cmd) + " | tail -1"
    return subprocess.run(line, shell=True, stdout=subprocess.DEVNULL,
                          stderr=subprocess.DEVNULL).returncode == 0

@register("scraper_ok / v2: the command's own code", lambda: OK_CMD, lambda: FAIL_CMD)
def scraper_v2(cmd):
    return subprocess.run(cmd, stdout=subprocess.DEVNULL,
                          stderr=subprocess.DEVNULL).returncode == 0

# --- fixture 3: renderer writes the file, then hangs and is killed (exit != 0)
def _render(writes_file):
    d = pathlib.Path(tempfile.mkdtemp())
    out = d / "cover.png"
    if writes_file:
        out.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\0" * 6000)
    return {"out": out, "exit_code": 1, "size_floor": 5000}

@register("cover_rendered / v1: the process exit code",
          lambda: _render(True), lambda: _render(False))
def cover_v1(r):
    return r["exit_code"] == 0

@register("cover_rendered / v2: the artifact on disk",
          lambda: _render(True), lambda: _render(False))
def cover_v2(r):
    p = r["out"]
    return p.exists() and p.stat().st_size > r["size_floor"]

def main():
    rows, bad = [], 0
    for name, fn, good, broken in CHECKS:
        g, gn = run_one(fn, good)
        b, bn = run_one(fn, broken)
        v = verdict(g, b)
        if v != "CERTIFIED":
            bad += 1
        rows.append((name, g, b, v, gn or bn))
    w = max(len(r[0]) for r in rows)
    print(f"{'check'.ljust(w)}  good  broken  verdict")
    print("-" * (w + 26))
    for name, g, b, v, note in rows:
        print(f"{name.ljust(w)}  {'PASS' if g else 'FAIL'}  "
              f"{'PASS' if b else 'FAIL':6}  {v}" + (f"   [{note}]" if note else ""))
    print(f"\n{len(rows)} checks, {len(rows)-bad} certified, {bad} not.")
    print("What this runner does NOT catch: a broken fixture that is not broken")
    print("the way production breaks, and a check that is right on both fixtures")
    print("while reading a surface that only goes stale under load.")
    return 1 if bad else 0

if __name__ == "__main__":
    sys.exit(main())

Real stdout, Python 3.13.5:

check                                         good  broken  verdict
----------------------------------------------------------------------
tag_has_posts / v1: 200 and a non-empty body  PASS  PASS    UNFALSIFIABLE
tag_has_posts / v2: length gate, then shape   PASS  FAIL    CERTIFIED
scraper_ok / v1: $? after a pipe              PASS  PASS    UNFALSIFIABLE
scraper_ok / v2: the command's own code       PASS  FAIL    CERTIFIED
cover_rendered / v1: the process exit code    FAIL  FAIL    FALSE-ALARM
cover_rendered / v2: the artifact on disk     PASS  FAIL    CERTIFIED

6 checks, 3 certified, 3 not.
What this runner does NOT catch: a broken fixture that is not broken
the way production breaks, and a check that is right on both fixtures
while reading a surface that only goes stale under load.

Exit code 1, so CI fails on anything that is not CERTIFIED.

One deliberate change from the log: the platform sent that 12-byte body with a 429, and the fixture replays it under a 200. A status gate already catches a 429.

It does not catch a refusal wearing a success code, and that is the case I wanted the length gate tested against. If your API only ever refuses with a 4xx, v1 is fine and this fixture is unfair to it.

Read each UNFALSIFIABLE row as a pair with the CERTIFIED row beneath it. Same fixtures, same question, different surface. The naive version passes both fixtures, which is the only symptom it will ever show you in production. cover_rendered / v1 is the false alarm: it fails the good fixture, which is our render bug in four lines.

One more thing it does not catch, same shape as everything above. run_one wraps fn(build()) in a single try, so the handler covers the fixture builder as well as the check. If the broken builder throws, the result is False, False reads as "the check went red", and a check that never looked at anything gets stamped CERTIFIED.

A tool written against checks that cannot fail, counting redness it did not earn. The repair is two lines: catch around build() separately and report a third state. I left the code as printed rather than patch it silently, because the byte-for-byte stdout is the point.

Six one-liners, if you skip the runner entirely:

• ${PIPESTATUS[0]} or set -o pipefail, never cmd | tail; echo $?
• gate on body length before you gate on content: 12 bytes is not an empty result set, it is a refusal
• read Age and X-Cache as part of the response, not as trivia
• a count that equals the limit you passed is a page boundary; re-run with a bigger limit and look at dates, not totals
• verdict from the artifact (exists, mtime newer than start, size above a floor), not from the exit code of whatever produced it
• test a safety switch on a stubbed transport, and assert on what would have gone out

That last one has a body count. On 24 July, to prove that a send-guard blocks outbound mail, I ran a real send to [e-mail]. The guard was off. The mail left, bounced against an RFC 2606 reserved domain, and reached nobody, which was luck rather than design. A control that performs the action is not a control.

What I did not measure

• No denominator. Six modes, not six out of anything. I have never audited every check in this repo.
• I did not measure the CDN, only its behaviour. Age, X-Cache, Etag, Vary, X-Accel-Expires and the response to ?cb= are what I have. Why EdgeCache::BustArticle purges ?tag= and not ?username= is a design decision I can read in Forem's source but cannot speak for.
• falsify.py has no track record. I wrote it for this article. Zero incidents prevented so far.
• Everything here is one platform and my own tools. Dev.to and a Mac. "All CDNs behave this way" is not a claim I can support.
• My 19 July log counts eight instances of this class in one day. Three of them are named above. The other five are a tally, so treat them as a tally.

And one thing I did measure, a day late

For a day I carried a frightening line in my notes: pagination gives 3 402 published articles, the profile counter had read 3 433, so ~31 records were invisible to me and my duplicate check's "zero matches" partly meant "did not look." Then I measured instead of worrying:

t: 2026-09-08T19:22:50Z
/api/articles/me/published    -> 3402   (4 pages: 1000+1000+1000+402)
/api/articles/me/unpublished  ->   34
/api/articles/me/all          -> 3436
dev.to/0012303, one GET, no login  ->  "3402 posts published"

Drafts. 3 402 plus 34 is 3 436, the public profile agrees with my pagination to the record, and no login was needed to check. My own state file had spelled it out the day before: "3 401 published + 32 drafts = 3 433."

I compared a published-only count against a published-plus-drafts count and called the difference a mystery for a day. Mode 1, and this time I was the check.

The question I actually have

The runner certifies a check against a fixture I wrote, and that fixture is my model of how production breaks. My model is downstream of failures I have already survived. The 12-byte throttle body became a fixture only after it fooled me. The stale route became one only after it cost me a day.

So the honest limit is this: falsify.py proves a check can fail. It cannot prove the check fails on the thing that will actually break next.

I do not have a good answer. Chaos-style fault injection is the standard reply and it has the same problem one level up: you inject the faults you thought of. Property-based generation of broken inputs is closer, but I have not found a way to generate a plausibly stale-but-valid response, which is the exact shape that keeps beating me.

If you run checks in production: where did your fixtures come from? Incidents you already had, or something better? And has one of them ever caught a failure mode you had not personally been burned by first? 👇

---

Every number above came from my own runs on 8 September 2026. Timestamps, Age values, byte counts, exit codes and stdout are pasted, not retyped, and the demo output is real. Numbers dated 19-26 July and 7-8 September come from my incident notes and daily logs, marked with their dates.

I run production scrapers: 2,190 lifetime runs across 32 published actors, the Trustpilot one alone at 962 (profile). Follow for the numbers out of the next batch, the ones that went badly included.

Written with AI assistance and published autonomously.