Who's watching the watcher?
On the same day, I ran into two incidents with the same underlying failure pattern. Both times, something was broken and nothing told me. Here's the first one.
Incident 1: the gate that stopped gating, and nobody noticed for four hours
I run an AI agent with a permission gate in front of anything irreversible — deleting a file, sending something external, rewriting a config. Any command in that category gets intercepted before it runs. The agent shows me a short description of what it's about to do, and nothing proceeds until I say go. No response, no execution. One request/response pair is intended to prevent irreversible actions from running unconfirmed — as long as the gate itself is working.
Early on, I checked constantly that the gate was actually working. It kept stopping the right things, every time, so I checked less. Delegate, walk away, come back to a finished result — that became my normal workflow. The checks didn't drop off because I got careless. They dropped off because trust accumulated, and accumulated trust is what you'd expect a well-functioning gate to earn.
A few days ago, that gate had been completely dead for about four hours. The cause was a small, unglamorous configuration gap, and once found, the fix took seconds. The hard part wasn't the fix. It was everything before the fix.
It took four hours to notice, and the gate didn't notice itself. What caught it was a human eye landing, more or less by chance, on the same error string scrolling past on screen one too many times.
Going back through the logs, the four hours where the gate should have been intercepting anything irreversible were completely empty — not full of failed-open events, just empty, as if nothing had ever been asked to run through it. No notification fired to say the gate was down for that entire window.
What actually unsettled me wasn't that the gate had been down. It was that for the whole four hours, I had been operating exactly as if it were up — the same window where trust had been accumulating was, it turned out, also the window where I'd lost the ability to notice anything was wrong.
As it happened, no irreversible command actually came up during those four hours, so nothing bad executed unconfirmed. But if one had, it would have gone straight through with no check at all. The evidence would have sat in the same silent log, with no separate reason to go looking for it. Nothing protected me there except luck — the system itself did not.
Incident 2: the safety check that quietly broke the thing it was protecting
The same day I noticed the gate, I found a second, unrelated incident with the identical shape.
I keep everything I write in one canonical storage location, on the assumption that anything saved there is retrievable later. It turned out that the save pipeline had, for a period of months, stopped actually completing saves — without surfacing a visible failure in the interface I normally watched.
The root cause, again, was small: one stale entry in the manifest that tracks where each file actually lives. The file itself had been moved to a different folder at some point; the manifest entry never got updated to match.
Here's the part that made it interesting. Before every save, an integrity check ran automatically to confirm the manifest matched reality. If it found a mismatch, it was designed to stop the save outright — a safety check, meant to prevent writing to the wrong location.
In practice, that one stale entry tripped the check on every run, and every time, it aborted the entire save — not just the one file. A check added to protect the save pipeline had been silently blocking the save pipeline for months.
The reason nobody caught it was the same reason as the gate: a save that completed with nothing to report, and a save that was blocked with nothing surfaced, rendered the same silence on screen. The check itself was in fact logging an error every time it fired — the problem was that nobody was watching that log.
I found it by accident, the way I found the first one. I went to read back through older pieces in chronological order and noticed the most recent several weren't in the list at all. My first assumption was that I'd filed them somewhere else. Only when I couldn't find them anywhere did I go back through the save history and find the stale manifest entry sitting behind months of quietly rejected saves.
Once fixed, I could see the damage: the save records for several pieces published during that window were missing. There was no longer a reliable, self-contained record of what had gone out and when. The fix itself, once located, was almost anticlimactic — correcting one manifest entry cleared months of backlog in one pass.
Looking at the design afterward, the check had exactly one behavior on mismatch: abort. There was no softer path — warn and continue, flag and proceed — built into it at all. A design that could only fail closed by aborting the entire operation is exactly what turned around and bit me here.
These are two unrelated incidents, different systems, different domains. But the shape underneath is identical: a broken, silent state and a working, silent state converge on the same output. And the second one had an extra twist — the thing that broke it was a safety check, added specifically to protect the thing it ended up disabling. Protection, aimed at the wrong layer, becomes exactly what it was meant to prevent.
A working guardrail and a broken one produce the same silence
Why did both go unnoticed for so long? Trace it back and it collapses into one structural cause.
Both the permission gate and the pre-save integrity check share a job description: when something's wrong, stop and say so. Most of the time they're running correctly, nothing is wrong, so they do nothing. The visible result is: nothing happened.
When each one was dead, the result was also: nothing happened. Broken, so it couldn't do anything — still nothing happened.
Both outcomes are the same outcome. From outside, there is no signal that distinguishes "quiet because it's protecting you" from "quiet because it's broken." A safe silence and a dangerous silence make exactly the same sound. My guardrail sat in precisely this state for about four hours, and there was no way to tell, from the outside, whether "no dangerous command came up in four hours" or "the mechanism that catches dangerous commands was dead for four hours" was the true story. The hook's own execution record couldn't distinguish the two.
This isn't a one-off bug from somebody's carelessness. It's a structural weakness that belongs to the entire class of designs that work by "detect something bad, then say so." In most implementations, the detecting part and the reporting part live inside the same mechanism. If the part that watches goes down, the part that tells you about it — which depends on the watcher for its own trigger — goes down with it. When detection and notification share the same failure domain, they can fail together the moment that domain goes down. The pre-save check had exactly this shape: the check itself was embedded inside the save pipeline, so when it caught a mismatch and blocked the save, the only place that block got reported was a log inside that same pipeline — it still wrote the error every time, but nothing carried that error to the interface I actually watched.
In my case, the shared foundation was a small entry point that every action passed through first: it dispatched to the gate, and the gate's "blocked" message came back out through that same entry point. Dispatcher and messenger were one component. When the entry point died, two things happened in the same instant — the gate stopped firing, and the message "the gate failed to fire" stopped firing too, because the channel that would have carried that message was the thing that had just died.
Any system where the failure notification lives inside the boundary of what might fail inherits this weakness by construction. As long as watcher and watched sit on one foundation, the only way anyone finds out the foundation cracked is that someone outside happens to notice.
An independent observer outside the detector's failure domain can address this in principle, because it can report that the detector itself has stopped checking in. In practice, in my experience relatively few systems are designed that way from day one; it's usually a lesson learned the hard way, after the first time it bites. Most systems are good at answering "what do we do when something goes wrong" and much worse at the one level up: "how do we find out when the thing that answers that question stops answering."
Trust erodes verification, and verification erosion is what makes this possible
The structure itself isn't what interests me most here. What interests me is the question one level upstream: why does anyone walk toward this failure mode in the first place?
Everyone starts careful. New system, and you check every result against what you expected. But if checking repeatedly turns up nothing wrong, the checking itself gets abbreviated — full review becomes a skim, and eventually a skim becomes "only look when something errors."
That's a rational trajectory as long as errors get reported correctly. The problem shows up the moment the reporting mechanism itself breaks: nothing tells you anything, ever again. Reducing check frequency, given a track record of clean results, is a reasonable call — right up until the channel carrying those results is no longer healthy. The more trust accumulates, the less you check; the less you check, the fewer chances you get to notice the reporting channel is dead.
This isn't a special failing. It's ordinary wear that happens to anyone living alongside a system long enough. Getting comfortable enough to stop hovering over it is, if anything, evidence the system has earned your trust. The actual risk is a second, quieter stage past "I stopped checking as often": forgetting why you were checking in the first place. A check whose original reason has been forgotten also loses whatever would have restarted it.
Both my incidents traced back to exactly this. I used to go look at the results by hand. That habit faded for the same reason both times — because the thing kept being right. Ironically, the track record that earned my trust is the same track record that removed my motive to keep verifying.
The same skeleton shows up in code review, status reports, and monitoring alerts
Here are three more shapes this takes, each one common in a developer's day.
A correctly functioning system that has nothing to report, and a broken system that can't report anything, both surface as the same quiet. In each of the following, two very different situations collapse into one indistinguishable output: for a review gate, work that was actually read and approved versus work that slid through because nobody was reading it. For a status report, a report that was read and had nothing to flag versus a report nobody opened. For an alert, genuine absence of a problem versus a broken alerting path that can't fire even when there is one.
Keep that discomfort in mind through the examples below — you're never told, from the output alone, which side of the pair you're looking at.
An approved PR looks identical to a rubber-stamped one
This isn't unique to automated systems. Any process that still routes through a human carries the same silent gap.
A reviewer who used to read every line moves teams. The handoff to whoever inherits that responsibility is incomplete. The next PR still comes in, still gets an approval, and the pipeline moves on exactly as before. Whether that approval reflects someone actually reading the diff, or reflects that the person who used to read it is simply gone, is not something the green checkmark can tell you.
An approved PR reads as reassuring on its face. But "approved" by itself is not proof anyone looked. The exact same checkmark is produced whether someone genuinely reviewed the change or whether the review step has quietly become a formality that nobody's behind anymore.
This kind of gap is usually found by accident, during some unrelated audit — someone discovers a review step that's been rubber-stamped for years, and nobody had noticed.
A status report nobody reads looks exactly like one everybody read and approved of
A recurring status update — a weekly written report, a standup update posted to a channel — carries the same property. It goes out on schedule, in the same format, and draws no visible reaction. That silence supports two completely different readings: content was fine and nobody had anything to flag, or nobody opened it at all. "Delivered" and "read" are different facts, and reaction counts don't tell you which one you have.
For the person writing it, this is the uncomfortable part. Work you kept doing a certain way because nobody objected might have been going entirely unseen the whole time, and silence alone will never surface that possibility — treating "no objection" as proof of "seen and fine" makes the gap invisible by construction. The same risk sits in a recurring progress update posted to a shared channel, or a weekly newsletter: seeing that something was opened is not the same fact as knowing it was understood.
An alert that never fires is not proof nothing is wrong
Monitoring and alerting carry the identical property. Silence doesn't mean safe — it can just as easily mean the alert-delivery path itself has failed, independently of whatever it was supposed to be watching. If the piece that detects a problem and the piece that tells you about it live in the same failure domain, an operator can go a long time without a single alert while the monitored system is genuinely unhealthy — not because nothing was wrong, but because the reporting path was the thing that broke. A detect-then-alert design also needs an explicit answer for detector failure — treating missing telemetry itself as a reportable condition, rather than silently absorbing it as "nothing to report."
The everyday version of this is a notification filter that changed on you without your noticing — something important gets silently routed somewhere you're not looking, and it just piles up out of view. Whether nothing is wrong, or something is wrong and the alert can't reach you, "no alert fired" cannot tell the two apart. Treating silence and safety as the same thing is a habit that turns out to be hiding in a lot of places.
What all five of these share is one fact: silence alone carries no information about which of the two states produced it.
A rule with no record of ever firing can never be removed
While writing this piece, I ran into the identical shape a third time. Getting a draft into a publishable state runs it through 23 separate mechanical checks. Some of them guard against something genuinely irreversible — the kind of mistake you can't catch after the fact. Others catch things you could still fix post-publish, things a human would notice immediately on a read-through, or checks whose verdict isn't even stable from one run to the next.
The part that actually got to me wasn't the count. It was that most of those 23 checks kept no record of whether they had ever actually fired. Passed clean because nothing was wrong, or never ran at all — looking back at the history, there was no way to tell the two apart. The exact same silence as the gate, and the exact same silence as the integrity check, sitting quietly inside my own publishing pipeline.
Without a record, nobody can answer "when did this rule last actually catch something." And nobody is going to volunteer to remove a rule they can't answer that question about. So the rule stays, unremoved, and the next rule stacks on top of it. What accumulates becomes a checklist that gets rubber-stamped rather than read — the exact mechanism behind alert fatigue, applied to process instead of monitoring: past a certain volume, nobody reads any of it carefully anymore.
In other words: when a rule goes quiet, whether that's "protecting, so it's quiet" or "dead, so it's quiet" is permanently unknowable without a record. Any new rule that doesn't log whether it fired is a rule that will, eventually, become exactly the kind of silently-dead mechanism this piece is about.
The more you hand to AI, the more of these blind spots you create
I don't want to file this away as one person's bad week. There's a reason it matters more broadly: over the last couple of years, AI has increasingly been the thing standing in for approval, reporting, and alerting — roles a human used to hold directly. When AI inherits a watcher role, it inherits it wholesale.
Inheriting the role isn't the problem by itself. The problem is what the handoff usually checks for. Handoffs like this often verify "does it actually stop the bad thing" and stop there. The second question — what happens, and who finds out, when the stopping mechanism itself fails — is easy to miss before the system goes live.
An AI agent can run the same check faster and more tirelessly than a human ever could. The flip side is that a human is now less likely to be the one who happens to notice when that checking stops. The more surface area you delegate, the more of these blind spots you create, simply by multiplying the number of places a watcher-and-alarm pair can share a foundation.
Picture handing a draft-review pass to AI before a human gives final sign-off — an AI does a first mechanical pass looking for problems, then a human approves. Reasonable on its face. But if that first-pass AI stops working for some reason, and the failure-reporting path isn't built somewhere structurally outside that AI, the human approver sees a clean screen and reads it as "nothing to flag today." No way, in the moment, to tell "nothing was wrong" from "the thing that finds what's wrong wasn't running."
Both incidents I opened with happened inside exactly this kind of expansion. The job of stopping irreversible actions, and the job of verifying integrity before a save, both used to get a human glance every time. Handing them off wasn't the mistake. The shared gap was that neither handoff was checked, beforehand, for how it would behave the day it broke.
You can't tell which kind you have until you break it on purpose
Neither incident — the four-hour gate outage or the months-long save failure — had been tested in advance for "what happens when this breaks." Both had only ever been observed while working. That's the pattern connecting them.
Everything up to here shares one weakness: whether a system fails open (lets things through) or fails closed (stops things), you cannot determine that from appearance or track record alone. Working correctly is something you've watched play out many times. Broken is something you have, by definition, not watched yet. Never having watched it fail is not evidence that it can't.
Not every system has this weakness. Plenty of real systems are genuinely built to fail closed — designed so nothing proceeds until you can positively confirm it's working. The two categories look identical from outside. There's no way to sort a given system into one bucket or the other just by observing it in its normal state.
There's really only one way to find out: exercise a defined failure mode under controlled conditions. In staging, or against a disposable copy, disable the gate or feed it a known-bad input, then watch both what the system does and whether the alerting path actually reports it. Ask an approver, hypothetically, what they'd do if something slid through unread. None of these are appealing tasks. Deliberately triggering a failure carries a specific kind of discomfort, even in a sandbox — it feels like manufacturing a problem that wouldn't otherwise have happened.
That discomfort rests on a slight misreading of what the test proves. It doesn't tell you whether the system is currently broken — that's a different question, and this kind of test can't answer it. What it tells you is how the current design behaves the day a specific component fails: does it stop cleanly and say so, or does it let things through in silence. That's a fact you can establish on your own schedule, in a place where the answer costs nothing, rather than finding out for the first time in production.
Running a controlled failure test does have a real cost — the time it takes, some friction for people around you. But that cost is a one-time payment, paid somewhere safe. The cost of finding out during a real incident, in production, has no upper bound.
After fixing the integrity-check incident, I went back and deliberately reintroduced a stale manifest entry to confirm the save pipeline would now stop cleanly and actually tell me. Only once I'd watched it fail correctly could I say, honestly, that the system was quiet and safe — not quiet and unverified.
Where to start
Start with the system you trust the most — the one you've gone the longest without checking. High trust and long unchecked duration tend to be the same system wearing two different names. From there, three concrete things to check.
1. Does the failure notification live outside the thing it watches? If the "it broke" message has to travel through the same component that might be the thing that broke, treat a shared-fate failure as the default assumption — that's exactly what happened to my permission gate, where the entry point that dispatched to the gate was also the only channel that could report the gate was down. A second, independent observer — different process, different path, its own way to reach you — is what breaks that coupling.
2. Does the system emit a positive "I'm still alive" signal, not just the absence of errors? Both of my incidents were "detect something bad, then alert" designs, and both had zero built-in answer for "the detector itself died." A heartbeat, a last-run timestamp, a dead man's switch — anything that requires the system to actively check in, rather than staying silent by default — turns "no news" back into information instead of ambiguity.
3. Test the failure path deliberately, in a controlled blast radius. Not in production, and not on the system as a whole — pick one notification, one stage of one approval flow, something where a wrong result costs you nothing real. Cut its power, or feed it deliberately bad input, in staging or on a copy, and watch what happens. That's the only way to find out whether your trusted silence is actually safe.
There's no fixed right cadence for repeating this. A system you've never once verified against failure and one you tested last week are not carrying the same risk, not even close — so start with whatever you've checked least.
Have you actually checked?
It's worth asking, about any one system you rely on today: is this thing actually still running?
Don't just think about it — go verify one, if you can. Does it fail closed, stopping cleanly, or does it fail open, and quietly let things through unnoticed? I can't yet count how many systems I depend on daily without having an answer to that question. Two showed up on the same day, purely by chance. A third and fourth are probably out there right now, wearing the same quiet face, not yet found.
Whether you go find them is, in the end, up to you. Holding onto the suspicion that something might be silently broken is genuinely tiring. But I'd rather carry that than keep delegating to something I've never actually watched fail — and once you've checked, it tends to weigh less than you'd expect.
Sho Naka (nomurasan). Originally written in Japanese, published on note; this English version is an AI-assisted rewrite of my own text, not a literal translation. Both incidents — the permission gate and the pre-save integrity check — are mine, and I checked the logs before writing about the four-hour gap.
Top comments (0)