I once spent forty minutes of an outage staring at a wall of green dashboards while customers could not log in. Every health check was passing. Every service reported itself up. CPU, memory, disk, all nominal. By every metric we had chosen to watch, the system was perfectly healthy, and it was completely broken. That gap between what we measured and what was actually happening is the most dangerous blind spot in operations, and a green dashboard does nothing to warn you it exists.
The problem is that monitoring only sees what you thought to look for. We were checking whether each service was running, not whether the path a real user takes through them actually worked. The login service was up. The database it depended on was up. The token cache between them had silently started returning stale entries, and nothing on our board asked the one question that mattered: can a person actually sign in right now. We were measuring the parts and assuming the whole.
That is the difference between monitoring and observability that I wish someone had drilled into me earlier. Monitoring answers the questions you already knew to ask. Observability is whether, when something breaks in a way you never imagined, you have enough signal to figure out what. A dashboard of green checks is comforting precisely because it only reflects the failures you anticipated. The outage that hurts is almost always the one you did not.
What I do now is measure outcomes, not just components. The most valuable check we have is a synthetic user that logs in, does a real transaction, and logs out, every minute, exactly as a customer would. When that fails, I do not care that every underlying service claims to be fine. The thing we exist to provide is broken, and that is the only health that counts.
Green is not the goal. A working system is the goal, and those are not the same thing. Watch the outcome your users actually depend on, or you will keep discovering, forty minutes in, that all your reassuring numbers were measuring everything except the problem.
– Serguey Shinder
Top comments (0)