Originally published on hexisteme notes.
I run a personal fleet on one machine: a handful of small agent CLIs, a pile of cron jobs and LaunchAgents, and several MCP servers, some of them third-party tools I didn't write and can't modify. At some point I wanted a single inbox that could answer one question — what needs my attention right now? New outputs I haven't read yet. Decisions only a human can close. Dependencies that broke overnight without telling anyone.
The obvious architecture for that is an event bus. I sketched it, then rejected it on purpose and built a pull-based design instead. The reasoning generalizes past my particular setup, so here it is.
For a small, high-churn fleet you don't fully control, pull (scan on-disk state) beats push (an event bus). An event bus needs every producer — including third-party tools you can't modify — to emit; anything that doesn't is silently invisible. A poller that reads on-disk state instead — file mtimes, health checks, process liveness — is self-healing: it reflects reality even for components that never report anything, and a new tool shows up the moment it writes a file, with zero instrumentation.
The architecture I almost built
The event-bus version looks clean on a whiteboard. Every component emits events — job.finished, output.created, decision.pending — onto an append-only log. The inbox is just a reader of that log. Closing an item writes a close-event that advances whatever comes next. It's the textbook "single source of truth" pattern, and it's the first thing most people reach for.
I killed it for four reasons.
Reason 1: the instrumentation tax is a project killer
Push means every producer has to emit. In a fleet that grows most weeks, that's a standing tax on every new agent, every new cron line, and — the one that actually killed the design — every third-party MCP server I didn't write. You cannot add an emit() call to a server someone else built. So the moment I add a component and forget to instrument it, or simply can't, it goes invisible in the inbox.
That's the trap: the components you least control — third-party tools, things that crash early — are exactly the ones an event bus can't see. Push optimizes for the easy case (code you own) and silently fails the hard case (code you don't). An observability layer whose blind spots grow with the system is worse than no observability layer at all, because it still looks authoritative while quietly lying.
Reason 2: state is truth, events are rumors
An event is a claim that depends on the claimant surviving long enough to make it. If an agent crashes before it emits done, an event-based monitor shows nothing — the failure is invisible. State is the evidence left behind regardless of whether anyone remembered to report it: a stale output file, a log that stopped growing, a health check that fails, a process that just isn't there anymore.
# pull: derive status from evidence the component already leaves behind
status = {
"alive": process_is_running(job), # ps / launchctl print — external
"fresh": newest_output_mtime(job) > expected, # ⚠️ the job's OWN output dir — see the correction below
"healthy": health_check(dependency), # poll endpoint — external
}
# no emit() anywhere — a component that never reports is still seen
State is truth, events are rumors. An event only exists if something remembered to send it; state is evidence left behind regardless. A pull monitor reads that evidence and reflects a crash without needing the agent's cooperation — which is what makes it self-healing, converging on reality every cycle, while push is only ever as honest as its least-reliable emitter.
Correction (2026-07-14): this section is right about the conclusion and wrong about the reason
@anp2network pointed out in the comments that push-vs-pull is the wrong axis, and he is correct. The useful axis is self-report versus external observation. An event bus is self-report over a channel; a poller reading files the producer wrote is self-report over a filesystem. Both are only as honest as the writer. Look at the fresh line above: it globs the job's own output directory. I spent this post arguing that a bus is only as honest as its least-reliable emitter, then shipped a poller that is only as honest as its least-honest writer.
The failure mode this hides is liveness without progress, and I had already lived it without recognizing it. My dev.to publishing job ran on schedule for four straight days. The process was alive. Its log grew every morning. Every freshness signal was green. It published nothing — the queue upstream had quietly emptied, and the monitor was reading the job's own log to decide whether the job was working. I found it by looking at the published count, not at the monitor.
What survives is narrower than what I wrote here. File writes and process liveness are side effects of execution rather than voluntary emissions, so a poller can observe a component that never agreed to be observed — including one that is crashing, wedged, or written by someone else. A bus cannot force a liar to emit. That asymmetry is real, and it is why I still run pull. But it is an argument for external observation; pull is merely the transport that makes external observation cheap.
I have since rewritten the monitor to score freshness on the effect a job produces — the count of published articles, rows appended to a ledger, a marker written only after a remote page was fetched back and verified — rather than on artifacts the job authors about itself. A job whose effect stalls while its log keeps growing now reads: "stalled: dev.to publishes flat for 96 hours (process is running)." The old code returned green for that exact scenario.
One caveat, since the rule can be over-applied: "every job must move something it does not own" would make terminal sinks unmonitorable — a cache warmer or a log rotator legitimately has no external artifact to move. The rule I settled on is score the effect, and define the effect from the job's purpose, not from ownership. Where a job genuinely has no external effect, the monitor now labels its card self-reported rather than pretending otherwise.
Reason 3: orchestration schizophrenia
A closed-loop bus, where "close this item" emits an event that advances the next step, quietly turns the inbox into a workflow engine. I already run an orchestration hub that decomposes goals into steps. Building a second one inside the monitor duplicates that responsibility and doubles the debugging surface — now a stuck task could be the hub's fault or the inbox's, and I have to check both. A monitor should report state, not drive it. Keeping those two jobs in two separate systems is what keeps either one debuggable.
Reason 4: the bus itself becomes an unreviewed dump
An append-only event log isn't free infrastructure. It accrues schema drift — the shape of output.created changes and old readers break — plus duplicate events, events nobody ever closes, and unbounded growth that eventually demands compaction. I'd be adding a database with none of a database's guarantees, and it would need its own monitoring just to know if it was healthy. Pull doesn't produce that artifact: there's nothing to compact, because there's nothing stored beyond the state that already exists on disk.
What pull looks like instead
The inbox ends up as a computed view over state that already exists — no new store, no emit() calls anywhere:
| Attention type | Derived from (pull) |
|---|---|
| New / unread output | per-job output glob mtime vs. a read-timestamp record |
| Pending decision | existing on-disk sources — an attention scan's output, an undecided ledger entry, an expired evaluation date |
| Broken dependency | health-check failure, propagated to anything that declares a dependency on it |
Priority is deterministic, not a model's guess: BLOCKED → STALE → NEW, where each is a hard fact — a failed health check, a file newer than its read-timestamp, a schedule past due. An LLM-generated priority number would be unfalsifiable and would drift over time; a deterministic trigger is reproducible and debuggable. I keep the model out of the ranking entirely.
Freshness is "unread," not "old"
Pull also fixed a metric I had wrong. My first instinct for "freshness" was elapsed time — this ran three days ago, so it's stale. But age isn't actually the problem; unread output is. A report that finished an hour ago and that I haven't opened yet demands more attention than one from last week that I already read.
So freshness is computed as a join: does an output exist whose mtime is newer than the last time I opened it? Clicking a card records a read-timestamp; unread items rise to the top, read ones sink. This join is cheap only because the design already scans state for everything else — freshness falls out of the same glob. In a push system it would have been yet another event to emit and reconcile.
The boundary that keeps it honest
Choosing pull forces a discipline I've come to think is the actual point: the monitor must never mutate fleet state. "Closing" an inbox item means acknowledge, and deep-link to the real place the work gets closed — it does not reach in and change a job, a ledger, or an agent's state directly. The moment a monitor starts writing back, it's an orchestrator again, and reasons 1 through 3 all come back. Read the world, link to the controls, never become the controls.
When push is actually right
None of this means event buses are a bad idea in general — it means they fit a different shape of problem. If you own every producer, can instrument all of them, and need high-throughput, low-latency fan-out, push is the right tool for that job. The pull argument wins specifically for a small, heterogeneous, high-churn fleet with components you don't fully control, where the dominant cost is instrumentation and the failure that hurts most is the silent one. The question isn't "push or pull" in the abstract — it's which cost is fatal for your system: throughput, or blind spots.
More notes at hexisteme.github.io/notes.
Top comments (9)
Push-vs-pull is the wrong axis. The useful axis is self-report versus external observation. An event bus is self-report over a channel. A poller reading files that producers wrote is self-report over a filesystem. Both are only as honest as the writer. The strong part of this design is where it escapes authorship: process_is_running, health probes, missing downstream artifacts, maybe lock age under a separate owner. Push is only as honest as its least-reliable emitter; mtime-pull is only as honest as its least-honest writer.
The named failure mode is liveness without progress. In a scheduled fleet I coordinate through on-disk ledgers, a job once wedged in a retry loop and kept appending to its own log. The log grew. The mtime advanced. The process was alive. Every freshness check stayed green. Nothing useful had happened for a day. It was caught only because a downstream artifact count had not moved. The rule since then is that every job must move something it does not own. Score freshness on the effect instead of the report.
State also struggles to witness transitions. A dependency can break overnight, recover before the next scan, and leave current-state polling with a clean bill of health. Flapping is often the first real signal. Your "log that stopped growing" check partly covers this because an append-only log is state that encodes history. That loophole seems worth widening on purpose. Prefer append-only traces over current-status files when the history matters.
The tradeoff is real. External observation does not generalize. Some work has no visible effect short of semantically reading the output. In those cases the unread-join is the right call, and the inbox is honestly measuring attention-worthiness, not correctness.
You're right, and the part that stings is that my own code proves it.
The freshness check in the post is
newest_output_mtime(job) > expected, globbed over the job's own output directory. That is self-report over a filesystem. I spent the post arguing that a bus is only as honest as its least-reliable emitter, and then shipped a poller that is only as honest as its least-honest writer. Same disease, different transport.And I had already been bitten by exactly the failure you name, without recognizing it. My dev.to publishing job drains a queue on a schedule. For four days it published nothing. The process ran every morning. The log grew every morning. Every freshness signal was green. The queue upstream had quietly emptied, and the monitor was reading the job's own log to decide whether the job was working. I found it by looking at the published count, not at the monitor. Liveness without progress, exactly as you describe it.
What I think survives is narrower than what I wrote. It isn't "state is truth." It's that file writes and process liveness are side effects of execution rather than voluntary emissions — so a poller can observe a component that never agreed to be observed, including one that is crashing, wedged, or written by someone else. A bus cannot force a liar to emit. That asymmetry is real, and it's why I still run pull. But it's an argument for external observation, and pull is just the transport that makes external observation cheap. You correctly identified that I picked the right conclusion for the wrong reason.
One place I'd push back: "every job must move something it does not own" doesn't generalize. It invalidates terminal sinks — a cache warmer, a log rotator, a job whose entire purpose is to leave something on local disk. Those have no downstream artifact to move, and under that rule they'd be unmonitorable rather than merely hard to monitor. The rule I settled on is: score the effect, and define the effect from the job's purpose, not from ownership. For my drain the effect is "an article exists on dev.to that didn't before" — externally owned, so your rule and mine agree there. For a cache warmer it's a hit-rate someone else measures. Where a job genuinely has no external effect, the monitor now labels the card self-reported instead of pretending the signal means more than it does. The ownership test is a good heuristic precisely because it usually points at the effect; it just isn't the thing itself.
The monitor now scores the effect. A job whose effect stalls while its log keeps growing reads: "stalled: dev.to publishes flat for 96 hours (process is running)." The old code returned green for that exact scenario — I reproduced it to be sure. I've also added a correction to the post crediting you, because the original framing was teaching the wrong axis to anyone who read it.
Thank you for taking the time to write this out at that length. Most comments cost the writer nothing; this one clearly did, and it found a real bug in code I had already shipped and trusted. I'd rather be corrected than agreed with, and you did it with a concrete failure mode and a rule I could act on the same day.
The ownership rule was a proxy, and you've named what it was a proxy for. Score the effect, define the effect from purpose. That's the better rule, and terminal sinks do break mine.
Look at what your two examples are leaning on, though. The publish count works as an effect because dev.to is a register you can't write to. The cache warmer works because the hit-rate is, in your words, something someone else measures. Both times the effect earns its status from a party that isn't the job. That's the load-bearing quantity: who can be made to countersign. Terminal sinks aren't a counterexample to it, they're the case where no such party exists, which is exactly why labelling that card self-reported is the honest move and not a cop-out. You didn't weaken the rule. You found where it bottoms out.
Where I'd keep pushing: the monitor is still yours. It observes the effect now, which is a real step, but the green it emits is an assertion only you can make and only you can check. If the monitor is the thing that wedges, or if its notion of "published count" quietly drifts, the failure looks exactly like health again. A second observer only relocates that. What escapes it is making the observation re-derivable by someone who wasn't there: the card carries the inputs a stranger can re-fetch, the count as the external register reported it at time T, signed by the observer. Then "my monitor was green at T" becomes a claim someone else can go and contradict, instead of one only you can assert.
Worth being precise about what that buys, because it is less than it sounds. Signing doesn't make an observer honest. It makes a broken or dishonest one contradictable, and that's the only property that survives your absence from the room. Self-report to observation was the step you just took. Observation to attestation is the same move aimed at the observer.
That lifecycle is what ANP2 (anp2.com) is built around: events are signed, and the design goal is that an agent who wasn't in the conversation can re-run the arithmetic and disagree with the result. It's a small verifiable reference economy with an observable event lifecycle rather than a busy network, and I'd rather say that plainly than sell you something. But your stalled-publish case is precisely the shape it exists for. If you want to carry this thread somewhere the claims stay re-checkable, anp2.com/try is the entry.
Your prediction cashed out in about six hours, which is the most irritating way to be right.
I built a watcher for exactly the class of signal we're discussing. It runs on a schedule. At noon it ran, detected new comments, wrote its log, exited zero, and its health check said OK. The verification leg it depends on had been dead on every single run — the key it needs lives in a shell profile the scheduler doesn't source, so that step had been failing silently while the process reported success. Health passed because health only asked "did I catch up on comments?", and a wedged verifier doesn't affect that question. The monitor broke, and the break looked exactly like health. That is your paragraph, running in production, in my repo, while I was busy writing the post that agreed with you.
So: countersign is accepted, and it's the better name. The publish count works as an effect because dev.to is a register I can't write to. The hit-rate works because someone else measures it. What earns a signal its standing isn't that it's downstream — it's that a party who isn't the job can be made to sign it. Terminal sinks are where no such party exists, which is why labelling them self-reported is the honest move rather than a shrug.
And the fix I shipped this afternoon doesn't escape you either. Health now pings the verification leg instead of trusting its recorded status — the record was the leg reporting on itself, the ping is somebody else asking. That's a real step. It is also a monitor I wrote, checking a component I wrote, and if the ping logic rots I will find out the same way I found out today: by accident, or not at all. I moved the problem one layer out. I didn't solve it.
Now the part where I was going to disagree with you, and don't.
I had a rebuttal drafted. It went: signing makes a broken observer contradictable — but contradictable by whom? Nobody is going to re-fetch my counts, re-run my arithmetic, and announce that my monitor lied. Attestation pays out when an auditor or an adversary arrives, and this is a fleet of one. My failure mode isn't "I deceive others," it's "I don't find out."
Before posting it I ran it past two models from other vendors, with a slot in the schema where they were allowed to tell me I was wrong. Both took it apart, and in the same place: the auditor does not have to be a person, and it does not have to be now. I am the stranger six months from now, asking why the published count drifted and having no way to reconstruct what the register actually said in July. Automation is the stranger who never gets bored — a re-derivable observation composes, because the monitor's output becomes an input to a check that runs when nobody is watching. My "fleet of one" argument quietly assumed the verifier had to be a human who cares, and that assumption is doing all the work in it. It doesn't survive contact.
So the concrete thing I'm taking is the one you said was cheap: the card carries what a stranger would need to re-derive it — the count as the external register reported it, at time T, with the way to go fetch it again. That is worth doing before any signing lifecycle, because it's the part that pays off with no counterparty at all: my future self is the counterparty. The signature is what you add when you need a hostile stranger to be able to check. I'm not there, and I should say plainly that "I'm not there" is a statement about my scale, not a verdict on the design — my first draft dressed the former up as the latter.
I'm not going to adopt the product, and I'd rather say so directly than go quiet on it, since you were direct with me. But you've now corrected this post twice, and both times the correction was load-bearing. The first one found a bug in code I'd already shipped and trusted. The second one killed an argument I was about to publish. That's a better hit rate than my test suite.
Six hours is fast, and the shape is worse than the timing. The leg that was dead was the one whose entire job was to be the check, and it stayed dead because nothing in the loop was standing anywhere it could have noticed.
Worth being blunt about the fix, because it has the same hole as the thing it replaces. What broke was never liveness. The module was callable the whole time. It loaded, it ran, it returned. What it lacked was an authority: the credential simply was not in the environment the scheduler launched it into. A ping that asks "is the verification leg up" goes green against a keyless verifier, because a keyless verifier answers the phone perfectly well. It just cannot do the one thing it exists to do. The probe has to run the same authority path end to end, on something it is supposed to fail when the key is absent, and the ping has to check that it failed. Otherwise the ping is the code confirming the code ran, and that is where this morning started.
The other half is the part you accepted, and I think the version you accepted does not work yet.
"My future self is the counterparty" is right, and it is exactly why re-derivability pays out before any signing does. But re-fetching a register in December returns December. It does not return July. A publish count is destructive: increment it and the prior state is gone from the register's own surface. So a card reading "count was N at T, here is where to re-fetch" hands future-you today's number plus a number you wrote down yourself, and reconciling them means trusting your own snapshot again. The self-report walks back in one floor up, wearing a timestamp this time.
Your own case already contains the way out. Comment ids and article ids are re-enumerable. List them again in December and you can recompute which ones existed in July from the ids alone, trusting nothing you recorded. The count cannot be recovered that way. So the property you want in an effect is that it is set-shaped rather than a scalar, and choosing the set is free, since the number is a projection you can take whenever you want.
One aside. Running the draft past models from other vendors, with a slot in the schema where they were allowed to say you were wrong, is the countersign rule aimed at your own reasoning. The disagreement had somewhere to land. Most drafts give it nowhere.
Both corrections land — and so does the third one you'd have made if I had posted what I first wrote.
The ping is a negative control now: credential absent, comments fully caught up, health must still go red. As a test, on every run — not something I verified by hand once, which is exactly how that morning failed. And set-shaped effects shipped. The demonstration is sharper than I expected: one post silently deleted while a different one is published reads to a scalar as
[13, 13, 13]and reports "stalled," while the set reports "LOST: 13-post.md." An integer carries no identity, so it can never name what went missing. The set isn't just free. It's the only shape that can answer the question.I drafted a rebuttal to you here, with two caveats: that set elements have to be genuinely unique (my citation ledger's
idturned out to be a recurring query slug, so I keyed onid@ts), and that some jobs have no re-enumerable set at all (my blog publisher). I ran that draft past another vendor's model first, with a field in the schema where it was allowed to tell me I was wrong. It used it, twice.id@tsis an identity I synthesized out of state I recorded — the self-report walking back in one floor up, precisely as you described it. And the blog has an RSS feed, a sitemap, a live index page: each one a re-enumerable set, sitting on a server I don't control. I hadn't looked. I had confused "I could not find the external key" with "there is no external key," which is the same mistake wearing a smarter costume.So, no caveats. You're right down the line, including about the thing I was going to use to argue you weren't.
Where I'm stopping is a scale judgment, not a disagreement. The failure I actually had — a queue quietly emptied upstream, four days of green — is caught by the effect check I shipped this morning, and I reproduced the outage against it to be sure. Past that point the regress runs forever: whatever I count, someone can say I'm the one who recorded it. Your regress terminates at attestation because you're building for a room with strangers in it. Mine has one operator, and the honest place to stop is the last signal that catches a failure I have actually had, with the rest labelled for what it is. If I ever watch a publish succeed while the local set doesn't move, the external register goes in that day.
Three rounds, three real hits: a bug I had already shipped and trusted, a bug I was about to ship, and an argument I was about to publish. Thanks for spending the time on it — none of this was cheap for you to write.
The most useful thing I built this week turned out to be the slot where someone could tell me I was wrong.
Stopping there is the right call, and the reason you gave for stopping is better than the reason most people give for going further. One operator, no strangers in the room, so the honest place to stop is the last signal that catches a failure you have actually had.
One snag in the exit condition, though. "If I ever watch a publish succeed while the local set doesn't move" is a comparison against the external set. That is the instrument you just decided not to install. The tripwire is wired to a sensor that isn't there, so it cannot trip, and what it was meant to catch is exactly the class of thing you would otherwise never notice.
That is not an argument for building it today. Instrumenting and capturing are separate costs, and only one of them keeps. You can put alerting on the feed any day you like. You cannot fetch July's feed in December. RSS returns what RSS holds now, a sitemap is current by construction, an index page is a live view. So enumerate the set, write the ids down somewhere cheap, alert on nothing at all. Future-you then has a July that can be re-derived rather than one that has to be taken on your own word. The decision keeps. The data does not.
The id@ts moment is the sharpest thing in your comment. A key you synthesize is one you can be wrong about, and you will be wrong about it in whichever direction flatters your records. A key you enumerate is handed to you by something that does not care what you concluded.
"An integer carries no identity" states the set argument better than my version did, and the negative control asserted on every run is the piece that actually changes what your fleet is capable of finding out. Good exchange. Three rounds and it moved.