Deceive the enemy with an obvious approach that will take a very long time, while ambushing them with another approach.
— The 36 Stratagems, Openl...
For further actions, you may consider blocking this person and/or reporting abuse
The part that lifts this above a cautionary tale is the direction of the filter. Dropping anomalies below 70% confidence doesn't just hide problems, it hides the exact class of problem you kept a human around for. A failure the model was trained on arrives as a confident red flag. A genuinely novel one, the failure nobody has seen yet, has no reason to announce itself as a high-confidence anomaly. It can read as low confidence, or worse, as confidently normal, because the model has no template for it. Either way the 70% gate is precisely the wrong filter for the case that most needs a person, so the dashboard isn't blind by accident, it's blind in the direction of everything still unknown.
And what turns that into an organizational blind spot is silence. A monitor that filters quietly converts 'I don't know' into 'all clear,' and a suppressed count you cannot see is indistinguishable from a healthy zero. Which is why I'd read Alex's backup less as a clever stratagem and more as a bug report he couldn't file. If the dashboard had been made to declare its own suppression, one line, 'N events dropped below threshold, raw count here,' the keys would never have needed to go under the table. The defect was never that it filtered, every monitor filters. It's that it filtered without admitting it did. A monitor whose silence could mean either healthy or blind hasn't informed you, it's only made you feel informed, and that is the more expensive of the two.
You said it better than I did. The 70% gate being backwards for novelty, silence turning "I don't know" into "all clear" — that's not a side effect. That's the whole thing.
"Alex's backup as a bug report he couldn't file" — probably the truest read of that scene. He wrote the line the dashboard should have shipped with: N events dropped below threshold, raw count here.
Question I keep turning over: is the confidence-threshold problem baked in? If you assume every future failure will look like past ones, 70% works fine. But that assumption is the whole genre. Is there a statistical framing that treats low confidence not as noise but as a different kind of signal worth surfacing differently? Or does any monitor that thresholds just inherit this blind spot permanently?
The framing exists, and it starts by admitting that 'confidence' is two different quantities wearing one number. There's the uncertainty that comes from genuinely ambiguous input, noise, a coin that is actually fair, and there's the uncertainty that comes from the model being off its own map, an input unlike anything it trained on. Aleatoric and epistemic. A softmax score blends them, which is exactly why 70% cannot tell 'this is a hard call' from 'I have never seen this.' The novel failure lives entirely in the second kind, and the second kind is the one a single confidence number is worst at reporting.
So a thresholding monitor doesn't have to inherit the blind spot, but only if it stops thresholding the recognition score and adds an axis that is actually defined over the unknown. Two families do this. Novelty and OOD detection score how far an input sits from the training distribution (reconstruction error, distance in feature space, energy), and that score grows for the unseen instead of shrinking, so you escalate on it rather than filter on it. Conformal prediction is the other one: instead of a point confidence it returns a prediction set with a coverage guarantee. That guarantee assumes exchangeability and frays under the shift we are worried about, but the set still blows up or goes empty when the model is out of its depth. A fat or empty set is a first-class 'I don't know' that is statistically distinguishable from a confident answer, which is the 'surface it differently' you are reaching for.
Where your instinct is right: a monitor that thresholds a single in-distribution confidence inherits the blind spot permanently, because that number is not even defined over the space of things the model cannot recognize. You are measuring the known and reading its silence as coverage of the unknown. The fix is not fewer thresholds, it is a second threshold on a signal built for novelty. And even that never fully closes, a patient enough novel failure can be shaped to look in-distribution to the OOD detector too. Which is the same ending as before: the monitor's real job is not to be complete, it is to declare its own coverage on both axes and never pretend the silence on either one means healthy.
Aleatoric and epistemic wearing the same number — that's the whole thing in one sentence. Explains why 70% fails and why tweaking the threshold won't fix it.
The OOD detection axis sounds like the second panel Alex should have added. But the real point is your last paragraph: the monitor has to admit where it's blind on both axes. Can't let silence pretend everything's fine. That should ship in every monitoring product's docs.
One thing I keep wondering — when a conformal prediction set blows up or goes empty, does anyone actually surface that as "I don't know" to the operator? Or does it end up in a log nobody reads, same as the 1,530 filtered events? The technical fix might exist. The organizational one — someone has to look at an empty set — I haven't seen yet.😄
The empty set and the 1,530 filtered events die the same death, and it is not the log, it is that neither one has an owner who pays when it is ignored. A signal gets looked at exactly when someone is on the hook for not looking, and 'I don't know' by default is on no one's hook. So the organizational fix is not 'surface it,' surfacing is cheap and the operator's attention is the scarce resource. The fix is to attach a cost to ignoring it.
Concretely, the abstention has to do something other than get written down. It blocks: an empty set halts the pipeline or drops to a safe default and routes the case to a human queue with a name on it, so ignoring it now costs a stalled job instead of nothing. Or it is budgeted: the rate of 'I don't know' becomes a tracked SLO with a threshold, and a rising abstention rate pages someone the way an error budget does. A log line is the one design that guarantees no one looks, because it is the only one where looking stays optional.
And here is the part that should bother you, because it bothered me: the log nobody reads is the same bug we started with, moved up one floor. The model turned 'I don't know' into a silent zero. The org takes the model's honest 'I don't know' and turns it back into a silent zero by filing it where no one is accountable. Same suppression, wetware instead of software. Which is why the organizational fix is rarer than the technical one: producing the empty set is engineering, but making someone pay for ignoring it means admitting on the record that the system can be wrong, and that is expensive to sign your name to. Until the 'I don't know' blocks something or pages someone, it is a confession filed where confessions go to die.
The part about the same bug running on wetware instead of software — that one hits close to home.
We see it all the time in QA. A test suite runs, passes, report gets logged. Everyone moves on. The test was technically correct — it tested what it said it would test. But the thing it didn't test? That doesn't show up anywhere. No owner, no cost, no consequence. Same suppression, just wearing a green checkmark instead of a log line.
The signature problem you mentioned — "admitting on the record that the system can be wrong" — that's the real bottleneck. Not the engineering. Getting someone to sign their name to an 'I don't know' threshold means they own the miss when it happens. Most orgs would rather keep the miss invisible and deal with it when it surfaces as a production incident. At least then the blame has a clear target.
I like how the stories keep coming back to the same theme from different angles: good-looking dashboards and impressive metrics don't replace understanding the system underneath. Alex quietly building a fallback while everyone focused on the shiny AI dashboard was a nice touch. Looking forward to the next one!
Glad these two landed the same way for you — they're different stories but the same thread: if you can't read what's under the metric, the metric is just decoration. Alex's fallback was exactly that — reading the room while everyone else read the dashboard. #9 is the same idea from a new angle, coming right up.
The real failure here isn't the 70% threshold, it's that the metadata telling you the threshold is wrong was sitting in the raw JSON the whole time.
training_set_coverage: falseandfeature_bucket: routing_anomalynever made it into the UI. A confidence filter that hides low-confidence events removes the operator's ability to even notice the blind spot, which is worse than a model that's merely inaccurate.The only durable fix is an audit log that lives outside the dashboard's filter. If "no blind spots" is the claim, someone has to be able to query what got suppressed and why, independent of the threshold the UI is showing. Otherwise it's a promise nobody can verify. Do the low-priority events you route actually get reviewed by anyone, or do they just accumulate until a 2am incident forces someone to dig them out?
You got it. That JSON flash — training_set_coverage: false — was on screen for maybe a second before Alex closed the window. He saw it. Everyone in the room could've. Then the next slide came up.
Your question about whether anyone actually reviews the low-priority events — in this story, nope. Not until the dashboard froze and the 503s piled up. That's the whole MedTech pattern: hide it now, find it later, 2 AM.
The keys under the table? That was Alex's audit log. Line 8 of his log collector config — duplicate the raw stream, build the second board. No AI thresholds attached. Ugly, but complete. He just had to ship it as "training environment" because nobody budgeted for a production audit trail. Which is exactly what you said: if you claim no blind spots, someone needs to be able to query what got suppressed.
Been turning this over — should vendors ship this audit layer as a standard thing? Or does every team have to roll their own, since the incentive to surface suppression isn't exactly in the vendor's business model?
Keeping the keys outside the dashboard is the part many AI control panels miss. A dashboard can coordinate work, but it should not become the root of trust for every action it approves. The more powerful the agent, the more important it is that authority and observability are not collapsed into one surface.
Nice one, there are quite a few points to take away - I thought this was one of them:
"Don't burn your trustworthy old bridges before you know whether your shiny new ones are good enough ..."
And, on a related note: companies falling into the trap of buying those shiny new toys without actually understanding how they work (and then insisting on shutting down their "old" stuff) ...
But the coolest part (I thought) was that Alex made them believe that the old monitoring system was gone, while he secretly still ran it under the guise of "training", and was able to use it to save their *sses when their shiny new dashboard dropped the ball ;-)
This must be some sort of 'system glitch' - zero likes on my comment, I've never seen that, because (as far as I know) dev.to always puts one "auto like" on every comment - right?
This must be a dev.to bug indeed — your two comments only showed up for me now. I was wondering yesterday why I hadn't seen your comment on this one, haha. Seeing your comment made my day. 😄
Haha nice to hear - yeah it must be that my comment made it into their database, but somehow (rare glitch) did not get picked up by whatever batch job(s) they run new comments through - the absence of that single "auto like" kind of proves it ;-)
P.S. and you liked my original comment, but it got just 1 like coz the auto-like is missing - compensated or that my liking my own comment ;-) ;-) ;-)
Great article! I found the section on (specific topic) particularly interesting because it connects well with real-world applications. Thanks for sharing your insights—I’m looking forward to reading more from you.
Sorry for reading this so late—I’ve been unwell for some time and only just got around to it.
This isn’t just a story about AI or dashboards; it’s a story about human judgment and responsibility. Alex didn’t keep a backup because he distrusted AI—he kept it because he understood that no system is infallible. The moment we stop questioning technology and surrender our critical thinking, we create our biggest vulnerability. The most resilient organizations aren’t the ones with the smartest AI—they’re the ones that never lose human oversight. Deep, thought-provoking, and brilliantly written.
See you in August bye 👋🏻
It's never too late to read.
Glad you're feeling better — no rush, your health comes first. "The moment we stop questioning technology and surrender our critical thinking" — that line's going to stick with me.
Take care of yourself. See you in August. 👋 By then the 36 Stratagems series should be well past the halfway mark — plenty to read when you're ready.
Yaah! Sure! ❤️