Imagine getting a message from a customer:
"Your application isn't working."
You open your monitoring dashboard.
Server uptime: 99.99%.
API response time: 120 milliseconds.
Error rate: 0.1%.
CPU usage: normal.
Database connections: healthy.
Everything is green.
So you respond:
"Our systems are working normally."
But the customer is still unable to complete their task.
And now you have an interesting problem.
Your monitoring system says everything is fine. Your user says nothing is fine.
Who is right?
I think this reveals something we don't discuss enough in software engineering.
A system can be technically healthy and still be completely unsuccessful.
We Measure What Is Easy to Measure
Software engineering has become remarkably good at collecting metrics.
We measure:
- uptime
- latency
- throughput
- error rates
- memory consumption
- CPU utilization
- database performance
- deployment frequency
- test coverage
These are important measurements.
They help engineers understand how software behaves.
But they don't necessarily tell us whether the software is achieving its purpose.
Consider an online shopping application.
The homepage loads.
The search API responds.
The product database works.
The checkout service is available.
But customers cannot complete their purchases because a confusing validation message prevents them from entering their delivery address.
From the infrastructure perspective, almost everything might look healthy.
From the customer perspective, the application is broken.
The system is available. The outcome is unavailable.
That's a very different failure.
The Problem With Green Dashboards
A green dashboard creates confidence.
Sometimes deserved confidence.
Sometimes false confidence.
Imagine a monitoring panel showing:
Service Health
API Uptime 99.99%
Avg. Latency 120ms
Error Rate 0.10%
CPU Usage 42%
Database Healthy
Overall Status: GREEN
Now imagine another dashboard:
User Outcomes
Checkout Completion 61%
Payment Retries High
Search Abandonment Rising
Failed User Journeys Rising
Overall Status: WARNING
Both dashboards can describe the same application.
Neither is necessarily wrong.
They're answering different questions.
The first asks:
"Is the infrastructure operating?"
The second asks:
"Can users accomplish what they came here to do?"
We need both.
The mistake is treating the first as proof of the second.
A Successful API Call Can Still Be a Failed Experience
This is one of the most interesting distinctions in modern software.
Consider an API:
@app.get("/recommendations")
def get_recommendations():
return {
"status": "success",
"recommendations": []
}
Technically, the API works.
It returns a valid response.
The HTTP status is 200.
No exception occurs.
Your monitoring system may count this as a successful request.
But if the user was expecting useful recommendations, the response might represent a product failure.
This doesn't mean an empty list is always wrong. Sometimes there genuinely are no suitable recommendations.
It means technical success and useful success are different concepts.
The application needs a way to distinguish them.
And that distinction cannot always be inferred from the HTTP status code.
This Gets More Complicated With AI
Now let's introduce an AI system.
Imagine a customer-support assistant.
It responds to every request.
Its average response time is excellent.
The model API rarely fails.
Token usage is within budget.
All technical indicators look good.
But customers keep receiving answers that are polite, grammatically correct, and completely unhelpful.
The system is functioning.
The model is responding.
The workflow is completing.
Yet the customer is frustrated.
This is why measuring an AI application only through latency, availability, and cost is insufficient.
We also need to understand things like:
- Did the response answer the question?
- Was it factually supported?
- Did the system follow the required process?
- Did the customer achieve their intended outcome?
- Was human intervention required?
- Did the interaction create another problem?
Some of these require carefully designed evaluation datasets, human review, or validated automated checks.
You cannot simply install a monitoring library and expect it to understand customer satisfaction.
Observability tells you what happened. Evaluation helps you determine whether it was good enough.
Both matter.
Sometimes Our Metrics Reward the Wrong Behavior
Here's an uncomfortable thought.
What if our measurements are not just missing problems?
What if they're encouraging them?
Imagine a development team being evaluated mainly on the number of features delivered.
Naturally, the team becomes very good at shipping features.
More releases.
More pull requests.
More completed tickets.
But perhaps the application becomes increasingly difficult to use.
Maybe customers only needed three improvements, while the team delivered thirty features.
The delivery metrics look impressive.
The product becomes worse.
The same thing can happen with AI.
If we reward a system for answering questions quickly, it may produce fast but incomplete responses.
If we optimize only for reducing human involvement, we may discourage necessary escalation.
If we optimize solely for low cost, we may sacrifice quality.
This is not an argument against metrics.
It's an argument for understanding what a metric actually represents.
When a measurement becomes a target, we need to be careful about what behavior it encourages.
The Missing Layer: Outcome Observability
I think software teams need to pay more attention to what we might call outcome observability.
Traditional observability helps us understand the internal state of a system through signals such as logs, metrics, and traces.
Outcome observability extends the question.
Instead of looking only at whether components operated correctly, we also look at whether important user journeys succeeded.
Consider an application that processes invoices.
We could measure:
Technical metrics
- API latency
- processing errors
- queue depth
- database availability
Workflow metrics
- invoices successfully processed
- invoices requiring manual correction
- duplicate submissions
- processing delays
Outcome metrics
- invoices correctly finalized
- time from submission to approval
- correction costs
- percentage of users completing the process without assistance
The three layers answer different questions.
Infrastructure
↓
Is the system operational?
Workflow
↓
Is the process functioning?
Outcome
↓
Is the user achieving the goal?
The key is to connect these layers instead of monitoring them separately.
The Question That Changes Everything
Before designing a dashboard, I think we should ask:
"What does success look like from the user's perspective?"
Not the developer's perspective.
Not the infrastructure team's perspective.
Not the product manager's perspective.
The user's.
If you're building a payment system, success isn't simply returning HTTP 200.
It's helping authorized customers complete payments correctly and safely.
If you're building an educational platform, success isn't simply counting videos played.
It's helping people learn.
If you're building an AI assistant, success isn't merely generating an answer.
It's helping someone accomplish a task reliably.
Once that outcome is clear, we can work backward.
What events would indicate success?
What events would indicate failure?
What should we measure?
Which measurements are merely proxies?
And what important failures might still remain invisible?
These questions are just as important as choosing a monitoring tool.
But Outcomes Are Harder to Measure
There is a reason engineering teams rely heavily on technical metrics.
They're relatively straightforward.
A server is available or it isn't.
A request takes 100 milliseconds or 500 milliseconds.
A test passes or fails.
Human outcomes are messier.
Did the user abandon checkout because of a bug?
Because the delivery price was too high?
Because they changed their mind?
Because the interface was confusing?
We cannot assume that every abandoned checkout represents a software failure.
A decline in conversion could also reflect pricing, seasonality, or changing customer demand.
That's why outcome metrics need interpretation.
Sometimes they require user research.
Sometimes controlled experiments.
Sometimes investigating individual journeys.
Sometimes the best question isn't:
"What does the dashboard say?"
But:
"What happened to the people represented by these numbers?"
A Better Way to Investigate Production Problems
Suppose a customer reports a failed transaction.
Instead of checking only whether the server was healthy, we can trace the journey:
User starts checkout
↓
Address validation
↓
Cart confirmation
↓
Payment authorization
↓
Order creation
↓
Confirmation delivered
Which stage failed?
Did the user receive a meaningful explanation?
Was the payment authorized but the order never created?
Was the order created but the confirmation never delivered?
Could the user safely retry?
These are workflow questions.
And they often reveal problems that infrastructure health metrics cannot.
This is one reason I believe systems thinking matters so much in software development.
The value of the application comes from the complete experience, not from individual components operating in isolation.
AI Makes the Measurement Problem More Urgent
AI-generated functionality introduces another challenge.
Traditional software often follows explicit logic.
AI systems can produce plausible outputs that are still incorrect.
A chatbot can confidently give the wrong answer.
A coding assistant can generate code that passes a superficial check but violates a requirement.
A document-processing system can extract the wrong number while returning a perfectly valid JSON object.
The request succeeds.
The output format is correct.
The result is wrong.
We need evaluation methods that go beyond asking whether the application ran successfully.
This may include reference-based tests, checking factual support, measuring task completion, and reviewing samples of real-world failures with appropriate privacy protections.
No single metric can capture the entire quality of a complex AI system.
The objective is to build enough evidence to make better decisions.
Maybe We Should Stop Celebrating Green So Quickly
I like dashboards.
I like monitoring.
I like measurable engineering.
But I'm becoming increasingly cautious about what a green status actually means.
Green should mean:
"The conditions we're monitoring are within acceptable limits."
It shouldn't automatically mean:
"Everything is fine."
Those are different statements.
A green dashboard is useful evidence.
It is not a complete description of reality.
And perhaps that's one of the most important lessons software engineers can learn as our systems become more complex.
We don't just need better software monitoring. We need better definitions of success.
The Real Question Isn't Whether Your System Works
It's whether your system works for the people it was built to serve.
A fast application that frustrates users isn't necessarily a good application.
A highly available service that produces unusable results isn't necessarily successful.
An AI model that responds to every request isn't necessarily helpful.
And a dashboard full of green indicators doesn't mean every important problem has been solved.
Sometimes the most dangerous failure is the one we never thought to measure.
So the next time every indicator is green, I think we should ask one more question:
"What could still be going wrong that this dashboard would never show us?"
That question might teach us more than another dozen monitoring widgets.
Have you ever worked on a system where all the technical metrics looked healthy, but users were still struggling? What did your dashboards miss?
Want More:
Head over to ReThynk AI to access our latest research, magazine articles, and developer resources. Click Here
Top comments (1)
We now need a better system and monitoring for audience satisfaction.