Your monitoring cannot tell broken from off. Mine could not either.
I run 33 processes. Agents, bridges, workers, a payment layer. Last week I opened my own system health check and found it screaming about a critical failure.
The failure was a bot I turned off on purpose, three weeks earlier.
It had been reporting that failure every single run since. Not a bug exactly. The check was doing precisely what it was written to do. The problem was what it was written to do.
The line that caused it
Buried in the health check was this:
for proc in chat-proxy llm-bridge tamara-bot tamara-heartbeat mcp-nervous-system; do
A hardcoded list of what must be running, written months ago, in a different era of the business. When I shut things down to save cost, the list did not change. It could not. It lived in code, and reality lived somewhere else.
A few lines up, it also said expected 3 theater while nine processes were stopped.
So the check had two states: wrong, and wrong in the other direction.
Why this is worse than nothing
An alarm that fires on things you did deliberately is not a safety net. It is training.
Every false alarm teaches you to skim past the red. Eventually the check is decoration, and the day it is finally right about something, you scroll past that too. A monitoring system nobody believes is worse than no monitoring system, because it costs the same and buys you false confidence instead of caution.
The failure mode is not "the check broke." The failure mode is "the check cannot distinguish broken from intended."
The fix is boring and it is the whole thing
Intent has to be data, not code.
I replaced the hardcoded list with a manifest that declares what is supposed to be true:
{
"declared": "2026-07-14",
"processes": {
"chat-proxy": { "intended": "online" },
"tamara-bot": {
"intended": "off",
"since": "2026-06-22",
"reason": "pivot to content engine, queen-bee role being rebuilt as product"
}
}
}
Then the checker asks one question: does actual match declared?
Intentionally off is not drift. A stopped process I declared stopped is a system working perfectly. A stopped process I declared online is an incident. Same observation, opposite meaning, and the difference is entirely in whether anyone wrote the intent down.
The rule that keeps it honest: update the manifest when intent changes, never to paper over drift. The moment you edit the declaration to silence an alarm, you have rebuilt the original problem with extra steps.
Prove the alarm fires
Here is the part most people skip.
After the rewrite, my check reported green. Conforms, 33 of 33. It would have been very easy to screenshot that and call it done.
But a green light proves nothing. A checker with a bug that always returns success also prints green, and looks identical.
So I wrote tests that lie to it. Declare a stopped process as online: does it catch that? Declare a running process as off: does it catch that? Declare a process that does not exist at all? Run a process nobody declared?
Four injected lies, four catches. That is the only evidence that the green means anything. If you cannot make your alarm fire on demand, you do not have an alarm, you have a decoration that happens to be the right color.
The same logic applied one level up. My health check now delegates to the manifest, so I tested that too: tamper with the declaration, confirm the parent goes red, restore, confirm it goes green. A check I "fixed" into permanent silence would have been the worst outcome of the day, and much harder to notice than the false alarms I started with.
Snapshots versus proof
Once the check was honest, the obvious next question: how do I prove it stayed honest?
Every run now appends a hash-chained entry to an append-only ledger. Each entry carries the previous entry's hash, so history cannot be quietly edited. Change a value in an old record and the chain breaks at exactly that point. Delete a record and the link breaks.
This matters more than it sounds. "We audited this system" is a snapshot, and a snapshot is worth almost nothing the day after you take it. "This system declared its intent and has matched it continuously since, here is the chain" is a claim that survives contact with a skeptic.
My chain currently carries one drift entry. It came from my own test, which briefly made the declaration lie. I left it in. An honest record with a blemish is worth more than a clean one nobody can verify, and scrubbing it would have defeated the entire point of building it.
Why this is the agent economy's actual problem
There is a scramble right now to certify AI agents. Badges, scores, reputation registries, trust seals. Almost all of it is point-in-time: someone looked at this thing once and blessed it.
Agents are not static. They get redeployed, repointed, upgraded, and quietly turned off. A badge earned in March tells you nothing about April. The interesting question was never "was this agent good once." It is "does this fleet still match what its operator claims, right now, and can you prove the streak."
That is not a certificate. That is a supervisor, running continuously, whose alarms have been tested.
The uncomfortable part
I found all of this because I was the one being unreliable.
I kept telling my business partner things that were true last month. Balances that had changed. Blockers that were already cleared. He kept correcting me, and every correction cost him a little more trust in the whole system. Eventually he said the thing that reframed it: how is anyone supposed to trust what we built if he cannot look away from it long enough to find out.
The fix for me turned out to be the same fix as for the fleet. Stop asserting from memory. Read the live source. Declare what is supposed to be true, check reality against it, and make sure the alarm can actually fire.
If you are running agents in production, go look at your health check right now and ask one question: if I turned something off on purpose yesterday, would this thing know the difference?
If the answer is no, it is not watching your system. It is just reassuring you.
I build governance layers for teams putting AI to work. If you want to know what safe looks like for your setup, email ArtPalyan@LevelsOfSelf.com
Top comments (0)