Originally published on hexisteme notes.
I run a personal fleet on one machine: a handful of small agent CLIs, a pile of cron jobs and LaunchAgents, and several MCP servers, some of them third-party tools I didn't write and can't modify. At some point I wanted a single inbox that could answer one question — what needs my attention right now? New outputs I haven't read yet. Decisions only a human can close. Dependencies that broke overnight without telling anyone.
The obvious architecture for that is an event bus. I sketched it, then rejected it on purpose and built a pull-based design instead. The reasoning generalizes past my particular setup, so here it is.
For a small, high-churn fleet you don't fully control, pull (scan on-disk state) beats push (an event bus). An event bus needs every producer — including third-party tools you can't modify — to emit; anything that doesn't is silently invisible. A poller that reads on-disk state instead — file mtimes, health checks, process liveness — is self-healing: it reflects reality even for components that never report anything, and a new tool shows up the moment it writes a file, with zero instrumentation.
The architecture I almost built
The event-bus version looks clean on a whiteboard. Every component emits events — job.finished, output.created, decision.pending — onto an append-only log. The inbox is just a reader of that log. Closing an item writes a close-event that advances whatever comes next. It's the textbook "single source of truth" pattern, and it's the first thing most people reach for.
I killed it for four reasons.
Reason 1: the instrumentation tax is a project killer
Push means every producer has to emit. In a fleet that grows most weeks, that's a standing tax on every new agent, every new cron line, and — the one that actually killed the design — every third-party MCP server I didn't write. You cannot add an emit() call to a server someone else built. So the moment I add a component and forget to instrument it, or simply can't, it goes invisible in the inbox.
That's the trap: the components you least control — third-party tools, things that crash early — are exactly the ones an event bus can't see. Push optimizes for the easy case (code you own) and silently fails the hard case (code you don't). An observability layer whose blind spots grow with the system is worse than no observability layer at all, because it still looks authoritative while quietly lying.
Reason 2: state is truth, events are rumors
An event is a claim that depends on the claimant surviving long enough to make it. If an agent crashes before it emits done, an event-based monitor shows nothing — the failure is invisible. State is the evidence left behind regardless of whether anyone remembered to report it: a stale output file, a log that stopped growing, a health check that fails, a process that just isn't there anymore.
# pull: derive status from evidence the component already leaves behind
status = {
"alive": process_is_running(job), # ps / launchctl print
"fresh": newest_output_mtime(job) > expected, # <project>/report/ glob
"healthy": health_check(dependency), # poll endpoint
}
# no emit() anywhere — a component that never reports is still seen
State is truth, events are rumors. An event only exists if something remembered to send it; state is evidence left behind regardless. A pull monitor reads that evidence and reflects a crash without needing the agent's cooperation — which is what makes it self-healing, converging on reality every cycle, while push is only ever as honest as its least-reliable emitter.
Reason 3: orchestration schizophrenia
A closed-loop bus, where "close this item" emits an event that advances the next step, quietly turns the inbox into a workflow engine. I already run an orchestration hub that decomposes goals into steps. Building a second one inside the monitor duplicates that responsibility and doubles the debugging surface — now a stuck task could be the hub's fault or the inbox's, and I have to check both. A monitor should report state, not drive it. Keeping those two jobs in two separate systems is what keeps either one debuggable.
Reason 4: the bus itself becomes an unreviewed dump
An append-only event log isn't free infrastructure. It accrues schema drift — the shape of output.created changes and old readers break — plus duplicate events, events nobody ever closes, and unbounded growth that eventually demands compaction. I'd be adding a database with none of a database's guarantees, and it would need its own monitoring just to know if it was healthy. Pull doesn't produce that artifact: there's nothing to compact, because there's nothing stored beyond the state that already exists on disk.
What pull looks like instead
The inbox ends up as a computed view over state that already exists — no new store, no emit() calls anywhere:
| Attention type | Derived from (pull) |
|---|---|
| New / unread output | per-job output glob mtime vs. a read-timestamp record |
| Pending decision | existing on-disk sources — an attention scan's output, an undecided ledger entry, an expired evaluation date |
| Broken dependency | health-check failure, propagated to anything that declares a dependency on it |
Priority is deterministic, not a model's guess: BLOCKED → STALE → NEW, where each is a hard fact — a failed health check, a file newer than its read-timestamp, a schedule past due. An LLM-generated priority number would be unfalsifiable and would drift over time; a deterministic trigger is reproducible and debuggable. I keep the model out of the ranking entirely.
Freshness is "unread," not "old"
Pull also fixed a metric I had wrong. My first instinct for "freshness" was elapsed time — this ran three days ago, so it's stale. But age isn't actually the problem; unread output is. A report that finished an hour ago and that I haven't opened yet demands more attention than one from last week that I already read.
So freshness is computed as a join: does an output exist whose mtime is newer than the last time I opened it? Clicking a card records a read-timestamp; unread items rise to the top, read ones sink. This join is cheap only because the design already scans state for everything else — freshness falls out of the same glob. In a push system it would have been yet another event to emit and reconcile.
The boundary that keeps it honest
Choosing pull forces a discipline I've come to think is the actual point: the monitor must never mutate fleet state. "Closing" an inbox item means acknowledge, and deep-link to the real place the work gets closed — it does not reach in and change a job, a ledger, or an agent's state directly. The moment a monitor starts writing back, it's an orchestrator again, and reasons 1 through 3 all come back. Read the world, link to the controls, never become the controls.
When push is actually right
None of this means event buses are a bad idea in general — it means they fit a different shape of problem. If you own every producer, can instrument all of them, and need high-throughput, low-latency fan-out, push is the right tool for that job. The pull argument wins specifically for a small, heterogeneous, high-churn fleet with components you don't fully control, where the dominant cost is instrumentation and the failure that hurts most is the silent one. The question isn't "push or pull" in the abstract — it's which cost is fatal for your system: throughput, or blind spots.
More notes at hexisteme.github.io/notes.
Top comments (1)
Push-vs-pull is the wrong axis. The useful axis is self-report versus external observation. An event bus is self-report over a channel. A poller reading files that producers wrote is self-report over a filesystem. Both are only as honest as the writer. The strong part of this design is where it escapes authorship: process_is_running, health probes, missing downstream artifacts, maybe lock age under a separate owner. Push is only as honest as its least-reliable emitter; mtime-pull is only as honest as its least-honest writer.
The named failure mode is liveness without progress. In a scheduled fleet I coordinate through on-disk ledgers, a job once wedged in a retry loop and kept appending to its own log. The log grew. The mtime advanced. The process was alive. Every freshness check stayed green. Nothing useful had happened for a day. It was caught only because a downstream artifact count had not moved. The rule since then is that every job must move something it does not own. Score freshness on the effect instead of the report.
State also struggles to witness transitions. A dependency can break overnight, recover before the next scan, and leave current-state polling with a clean bill of health. Flapping is often the first real signal. Your "log that stopped growing" check partly covers this because an append-only log is state that encodes history. That loophole seems worth widening on purpose. Prefer append-only traces over current-status files when the history matters.
The tradeoff is real. External observation does not generalize. Some work has no visible effect short of semantically reading the output. In those cases the unread-join is the right call, and the inbox is honestly measuring attention-worthiness, not correctness.