Tuesday evening I counted something I had never counted before.
The number of times my own tooling interrupted me in one working day. Not other people. Not meetings. My own checks, guards, linters, and alerts, the ones I installed on purpose.
The count embarrassed me. Roughly once a minute, for hours.
And here is the part that actually stung. I could not remember reading most of them. My hands had learned to dismiss the interruptions before my eyes finished parsing them.
You know this reflex. It lives in your muscle memory too.
The yellow triangle in the build output you have not read since spring. The dashboard tab that is always open and never looked at. The pull request check that fails so reliably that the failure IS the expected state. The channel where the monitoring bot posts, which everyone muted in the same week they joined it.
We do not have a tooling problem. We have a listening problem.
The numbers are worse than the feeling
I went looking for data this week, because I wanted to know if I was unusually bad at this.
I am not. PagerDuty survey data cited in coverage this window says 41 percent of engineers openly admit to ignoring alerts because of overload. Engineers in larger orgs face somewhere between 500 and 1,200 alerts per day. A single incident can fan out into 50 or more pages across tools that do not know about each other.
Read that again. Four out of ten of us have stopped pretending we read the things we built to warn us.
And the prediction that made me wince. by the end of this year, 40 percent of enterprises are expected to carry what one report calls observability debt. Instruments installed, wired, paid for, and unread.
The monitoring industry keeps selling us more instruments. Nobody sells us the discipline of reading them.
What happens when nobody listens
There is a name for the mechanism, and once you know it you see it everywhere.
Normalization of deviance. One tolerated shortcut sets a precedent. Nothing explodes that day. The shortcut repeats, the team internalizes it, and within months nobody remembers it was ever a deviation. The broken thing has become the standard thing.
A muted alert channel is normalization of deviance. So is a linter rule the whole team suppresses inline. So is the CI job everyone re-runs until it goes green, because everyone knows it is flaky, and knowing has replaced fixing.
Every one of those started as a single justified exception. A deadline. A false positive. A Friday.
The dangerous part is not the individual exception. It is that each unread warning quietly teaches you to not read the next one. Attention is a budget, and noise spends it without asking.
I learned this on my own setup this week, in a way I will not fully unpack here because the deep dive deserves its own write up once the work is finished. The short honest version is that I measured my own automation for the first time in months, and the measurement disagreed with the story I had been telling myself about it. The instruments had been recording the whole time. I had built the panel and never once sat down to read it.
That sentence cost me some pride. I am sharing it because I suspect it describes your setup too.
The filter that cuts through all of it
The best thing I found this week did not come from a vendor. It came from the SRE community, in a thread where somebody asked how to know what deserves to page a human.
The top voted answer, from an engineer called PanTroglo, is one sentence.
If it is not actionable, and it is not directly impacting revenue, it is not an alert that should be paging anybody.
Sit with that. Actionable, and impacting something that matters. Two tests. Everything in your alerting that fails either test is not protection. It is noise wearing protection's uniform.
The same thread produced the second keeper, from a commenter who put it plainly. you need different levels. Log it, queue it, Slack it, page it. Not every failure deserves the same response.
Most of us run every check at the same volume. Everything pages, so nothing does.
What I actually changed
Three moves, all boring, all this week.
I started measuring the ignore rate. Not the alert count, the bypass count. When you override or dismiss a check more than you obey it, the check has stopped governing anything and started charging rent. The ratio tells the truth. Mine was high enough that I stopped arguing with it.
I started asking the actionability question of every interruption. Would a reasonable person DO something on seeing this, and does the thing it guards actually matter? Two nos means the check gets demoted or deleted. Not silenced. Demoted honestly, in the config, where the decision is visible.
And I accepted the hardest rule. an instrument you do not read is worse than no instrument, because it manufactures false confidence. The dashboard's existence feels like safety. Only the reading of it is safety.
The question for your team
Here is the audit that takes ninety seconds and no budget.
Open your alert channel. Find the last ten alerts. For each one, answer honestly. did anyone act on this?
If the answer is no ten times, you do not have monitoring. You have a diary that nobody reads, written by a robot that never sleeps.
The fix is not another tool. The fix is subtraction, tiers, and the uncomfortable habit of reading what you already collect. The teams that get this right are not the ones with the most instruments. They are the ones where every remaining instrument is believed.
Four out of ten of us have already admitted we walked away from the panel. The honest next step is either delete the instruments or start reading them.
I chose reading. It has been humbling.
Your turn
What is the one alert or check your team ignores every single day?
If this was useful
I work through this in public, the wins and the freezes both, mostly on LinkedIn and YouTube. If the real version of building in the open is useful to you, that is where it lives. Find me on X, GitHub, and the work at next8n.com.
Top comments (0)