You've probably had this incident. Dashboards are all green. CPU is fine. Memory looks good. Pods aren't restarting. Databases are healthy. But customers can't log in, or payments won't process, or nothing's loading.
You check Prometheus. Nothing's firing. Everything says "we're fine."
Except you're not fine.
A healthy system is not the same as a working system.
The Blind Spot
Most monitoring setups measure what's happening inside the infrastructure.
CPU utilization. Memory consumption. Disk usage. Network throughput. Pod restarts. Request rates. Error counts.
These metrics matter. But they answer one question: How are our components behaving?
Customers are asking something different: Can I complete my task?
The gap between those two questions is where incidents hide.
Internal Metrics Show You The Engine
Think of a car dashboard showing engine temperature normal, fuel level normal, oil pressure normal, battery healthy.
Everything looks fine.
But the steering wheel is disconnected.
That's what a lot of monitoring does. We measure component health while assuming the customer journey works. Usually it does. Sometimes it doesn't.
The Scenarios You Learn From
Most teams adopt synthetic monitoring after a painful incident. The postmortem reads the same way every time:
"All services were healthy."
"Kubernetes showed no issues."
"Database latency was normal."
"But customers couldn't log in."
Or:
"But payments weren't processing."
Or:
"But they couldn't upload files."
The issue wasn't invisible. You just weren't measuring it.
What Gets Missed
Your API returns HTTP 200. Your authentication service is running. Your database is healthy. But the token validation fails because a certificate expired. Green dashboards. Users stuck.
Or a downstream dependency fails silently. Metrics show low latency, healthy containers, no restarts. Customers get incomplete results.
Or a DNS misconfiguration breaks resolution. Everything internal looks normal. Users see downtime.
Or a JavaScript bug on the frontend breaks the checkout flow. Your backend is fine. Your infrastructure is fine. Users can't complete transactions.
Blackbox Monitoring Actually Tests This
Blackbox monitoring doesn't care about implementation details. It behaves like a customer.
Instead of asking "Is the service running?" it asks "Can the user successfully log in? Make a payment? Upload a file? Finish a transaction?"
If the infrastructure is healthy but blackbox monitoring fails, you've found your incident.
Which Alert Matters More
CPU utilization exceeded 85%.
vs.
Customers cannot complete checkout.
The second one, obviously. Because customers don't buy CPU.
The whole point of observability isn't to monitor infrastructure. It's to protect business functions.
Use Both
This isn't a choice. Internal metrics and blackbox monitoring solve different problems.
Internal metrics help you understand why something failed. Which component is degraded. Where the bottleneck is. What engineers should investigate.
Blackbox monitoring tells you whether anyone cares yet. Are customers impacted? Can critical workflows succeed? Is the platform delivering value?
One explains the story. The other tells you if the story matters.
Real Example
Your streaming platform goes down.
Internally:
- Kubernetes healthy
- RabbitMQ healthy
- CPU normal
- Memory normal
- Databases healthy
Blackbox monitoring:
- Video playback success rate: 0%
Which alert wakes someone up? The playback failure. That's the closest thing to what your users actually experience.
The Danger
The most damaging outages happen when internal monitoring and customer experience tell different stories.
If you only measure what's happening inside your platform, you're seeing half the picture. Your pods are healthy. Your databases are fine. Your services are running.
Your users just can't do anything with them.
Top comments (0)