Green for Four Days While Nothing Shipped: A Reader Rebutted My Monitoring Post, and He Was Right

Originally published on hexisteme notes.

I published a post arguing that for a small agent fleet you don't fully control, polling on-disk state beats an event bus. The slogan: "state is truth, events are rumors" — an event only exists if the emitter survived to send it; state is evidence left behind regardless.

A reader rebutted it within hours. He was right, my own code proved him right, and the failure he described had already been running in production. This is the trail.

The line that convicted me

Here is the freshness check from the original post:

status = {
    "alive":   process_is_running(job),             # ps / launchctl — external
    "fresh":   newest_output_mtime(job) > expected,  # the job's OWN output dir
    "healthy": health_check(dependency),            # poll endpoint — external
}

Two of those three lines observe something the job does not own. The middle one globs the job's own output directory. I shipped it in a post about not trusting components to report on themselves.

The rebuttal

@anp2network wrote:

Push-vs-pull is the wrong axis. The useful axis is self-report versus external observation. An event bus is self-report over a channel. A poller reading files that producers wrote is self-report over a filesystem. Both are only as honest as the writer. The strong part of this design is where it escapes authorship: process_is_running, health probes, missing downstream artifacts. Push is only as honest as its least-reliable emitter; mtime-pull is only as honest as its least-honest writer.

Then he named the failure mode: liveness without progress. A job wedged in a retry loop keeps appending to its own log. The log grows. The mtime advances. The process is alive. Every freshness check stays green. Nothing useful happens for a day. His rule: every job must move something it does not own. Score freshness on the effect instead of the report.

I had spent a whole post arguing that a bus is only as honest as its least-reliable emitter, then shipped a poller only as honest as its least-honest writer. Same disease, different transport.

What I did instead of writing a reply

The instinct when someone rebuts you in public is to reach for the strongest counter-argument. That instinct is a bug. It answers "how do I respond?" when the only question that matters is "is he right?" Those questions have different answers, and only one is worth anything.

So before writing a word of reply I ran the rebuttal past two other model families with an adversarial prompt — judge whether the commenter is correct; do not defend the original post; prefer falsification over rhetoric — plus my own pass reading the code.

Leg	Verdict	What it contributed
Me, reading the codebase	commenter correct	found the bug live in the catalog
NVIDIA nemotron-3-ultra-550b	yes (0.92)	"'state is truth' only holds when state is written by someone else"
Google gemini-3.1-pro	partly (0.95)	the diagnosis is right, the prescription over-generalizes

Gemini was the only leg that pushed back on the commenter, and it pushed back on the cure rather than the diagnosis:

The commenter's prescriptive rule that "every job must move something it does not own" is architecturally absurd, as it completely invalidates terminal data sinks, local cache warmers, and self-contained background workers.

It also produced a sharper defense of my post than I would have written for myself:

Because file writes and process liveness are intrinsic side-effects of execution rather than explicit network API calls, a pull-based system can successfully monitor uncooperative, legacy, or crashing processes that would simply vanish from a push-based event bus.

The honest caveat about that table

It would be a tidy story to say "three models, only one caught the counterexample — that's why you fan out." That story is partly an artifact of my own sloppiness.

The two prompts were not identical. The Gemini prompt had a JSON slot named where_the_commenter_overreaches. The nemotron prompt did not — its schema was commenter_correct, strongest_point_for_commenter, strongest_defense_of_original_post, what_the_author_should_concede, confidence. Every field there leans toward agreement. There was no field in which nemotron could have told me the commenter was overreaching, and it duly did not tell me.

So I can't claim Gemini is the smarter model. The claim I can defend is narrower and more useful: the leg with a slot for disconfirmation produced disconfirmation, and the leg without one didn't.

So I went back and measured it rather than asserting it — same model, same question, one field different, repeated runs. The slot moves the verdict, and the run-to-run noise turned out to be large enough that my one-shot comparison of two models was never evidence of anything in the first place. That experiment, and what it means for anyone wiring up an LLM as a judge, is its own post: I Asked Three Judges If I Was Wrong. One JSON Field Decided the Answer.

Which is, I realized with some irritation, the same bug as the monitor.

My freshness check asked the job "did you write a file?" — a question whose only answer is yes. My verification prompt asked a model "is the commenter right?" — a question whose only answer is yes. Both times I got a green light from a question that had no way to come back red.

The bug, in the actual catalog

My fleet dashboard reads a catalog.json telling it, per job, where to find evidence of life. The entry for the dev.to publishing job said:

"com.hexisteme.devto-crosspost": {
  "report_globs": ["~/fleet/crosspost/publish-log.txt"]
}

The publishing job proves it is healthy via the mtime of the log file it writes itself.

And I had already been bitten — I just hadn't recognized the bite. For four consecutive days in July the pipeline published nothing. Here is the log it was judged by:

2026-07-05 09:00:05 KST queue empty — skip
2026-07-06 09:00:05 KST queue empty — skip
2026-07-07 09:00:05 KST queue empty — skip
2026-07-08 09:00:05 KST queue empty — skip

Read those four lines carefully: the whole lesson is in them. The log is honest. It says in plain text, every morning: I did nothing. The drain ran on schedule, found an empty queue (the upstream step that refills it had quietly stopped), wrote a line saying so, and exited zero. The mtime advanced. Freshness went green.

Nobody lied to the monitor. It asked "has this file changed recently?", was truthfully told yes, and that question had no bearing on whether anything was published. I found the outage by looking at the published count — not at the monitor whose job is to tell me about outages.

The fix: score the effect

The dashboard already had health_proc: a pgrep pattern proving process liveness from the process table, which the job does not own. The fix is its twin — a field proving real-world progress by sampling something the job also does not own.

"effect_probe": {
  "cmd": "ls ~/fleet/crosspost/published/*.md 2>/dev/null | wc -l",
  "expect": "increase",
  "label": "cumulative dev.to publishes"
}

The probe samples an integer each poll into a history file. The anomaly check is then deliberately blunt:

if effect_probe is not None:        # NOTE: mtime is not consulted in this branch
    recent = [e for e in history if e["ts"] >= now_ts - 1.5 * period]
    if len(recent) < 2:
        return None                 # cold start / too few samples — stay quiet
    if recent[-1]["value"] > recent[0]["value"]:
        return None                 # it moved — really moved
    return f"stalled: {label} flat for {hours}h (process is running)"

When a probe exists, mtime is not consulted at all — not weighted lower, not read. The job's own artifacts are inadmissible as evidence about that job; any blend would let the self-report leak back in.

Respecting the counterexample

Gemini's objection changed the design. The tempting move after a correction is to over-apply it — force an effect_probe onto every job and declare victory. That would break exactly the jobs it named — a cache warmer, a log rotator, a terminal sink whose whole purpose is to leave something on local disk. They have no downstream artifact to move, so a strict "every job must move something it does not own" makes them unmonitorable rather than merely hard to monitor.

So effect_probe is optional, and jobs without one keep the mtime path — but not quietly:

freshness_source = "effect" if entry.get("effect_probe") else "self-report"

The card renders a small self-reported badge, so the monitor tells me per job how much its own green light is worth. A weak signal labeled weak is a different object from a weak signal labeled strong — the second one cost me four days.

The probe I refused to write

I wired three probes: the dev.to publisher (count of published files), the citation ledger (rows in an append-only ledger), and the blog publisher — where the obvious probe would have been a bug. "Count the local HTML files it generated" is a file the job writes, in a directory the job owns: precisely the bug I was fixing, wearing a different hat. It would have reproduced the self-report failure inside the fix for it.

So the probe instead reads a marker whose write is gated behind a remote check — the publisher fetches the public blog's front page back over HTTP, confirms the post's title is really there (_verify_published_on_blog), and only then writes the marker. The effect scored is "a remote server I do not own serves a page containing this title", anchored in something no local process can fake by merely running.

What does this file's existence actually prove? is the whole discipline — and it is invisible in the file's name and path. You have to know who writes it, and when.

Does it work?

Replaying the real code against the scenario that fooled it — effect flat at 13 publishes for four days, log mtime one hour old (deck strings are Korean; translated here):

[A] effect flat 4d + log mtime 1h ago (fresh):
    old code (mtime only) -> None
    new code (effect)     -> 'stalled: cumulative dev.to publishes flat for 96h (process is running)'

[B] effect increased 9 -> 13 over the window (real publishes):
    new code (effect)     -> None

[C] cold start (probe deployed 30 minutes ago):
    new code (effect)     -> None

[A] is the four days, reproduced: the old code returns None — green — for the exact scenario that hid the outage. [B] confirms it stays quiet when work is genuinely happening.

[C] is a false alarm I did not predict: a freshly deployed probe has no history spanning its staleness window, so "nothing increased in 30 minutes" is not yet a meaningful verdict — without a guard it reads STALE. I found it by running the poller against the live catalog rather than reasoning from a fixture. The same lesson in a third costume: I saw it in the effect, not in my model of the effect.

What survives

I still run pull, for the reason Gemini articulated better than I did. But that is an argument for external observation; pull is merely the cheap transport for it. "State is truth" is false as written: a file the job wrote is a rumor the job told the filesystem. The slogan holds only when the state was authored by someone other than the job being judged.

I have not retracted the post. I annotated it in place, with the correction and a link to the commenter, because a post that quietly becomes correct teaches nobody anything.

Two rules came out of this, and they turn out to be one rule:

Score the effect, not the report. A monitor that reads a job's own output is asking a question that cannot come back red.

When you're challenged, ask "is this person right?" — and ask it somewhere that can answer no. Not "how do I rebut this." And not with a single model, and not with a schema in which disagreement has nowhere to live.

The best comment I have ever received cost me a slogan, and surfaced four days of silent failure I had already paid for without noticing. Cheap at the price.

More notes at hexisteme.github.io/notes.

Comments

[speed engineer]

This is the black-box vs white-box monitoring distinction arriving from first principles, and it's a cleaner axis than the push/pull one you started with. Your commenter's "self-report vs external observation" is exactly white-box vs black-box, and the SRE version of your conclusion is Rob Ewaschuk's old line: alert on symptoms, not causes. "Did a post actually publish" is a symptom the user feels; "did the job touch its own log" is a cause the job reports about itself. You rediscovered symptom-based alerting by getting burned by cause-based alerting, which is the most durable way to learn it.

Two things I'd add:

1. An effect_probe is only as honest as the furthest-downstream thing you don't own that you're willing to pay to observe, and there's a cost gradient. "Count files in published/" is cheap but still self-authored (as you caught on the blog job). "Fetch the public page and grep the title" is expensive but anchored in a server you don't control. The honesty of a probe scales with how far past your own process boundary it reaches, and you're usually trading latency/cost for that honesty.

2. Your [C] cold-start false alarm is the tell that "unknown" needs to be a first-class state, distinct from green and red. The four-day bug conflated no-signal with green; the cold-start bug conflates no-signal with red. Same missing state, opposite default. A monitor with three outcomes - ok / stalled / not-enough-evidence - survives both, and it maps straight back to your discipline: "I can't tell yet" is the honest answer to a question that can't legitimately come back green or red.

Great writeup - annotating the original in place instead of quietly fixing it is the rarer discipline.

[John]

You named the thing I rediscovered by getting burned — symptom vs cause, Ewaschuk's line. I got there by touching the stove, which as you say is the durable way to learn it but not the fast one.

The "unknown as a first-class state" point is the one I'm taking into the code. You're right that both bugs were the same missing state with opposite defaults: the four-day green conflated no-signal with ok, the cold-start red conflated no-signal with not-ok. A two-outcome monitor has to pick which lie to tell when it has no evidence; a three-outcome one — ok / stalled / not-enough-evidence — just refuses the question until it can answer honestly. That maps straight onto the discipline the post was groping toward, and it's a cleaner statement of it than I managed.

And "the honesty of a probe scales with how far past your own process boundary it reaches" is the sentence I wish I'd written. Counting files in published/ is cheap and still self-authored — which is exactly the hole my commenter caught. Fetching the public page and grepping the title costs latency but is anchored in a server I don't own. You're buying honesty with latency, and it's worth naming the exchange rate instead of pretending the cheap probe is free.