DEV Community: Sara

Your AI Agent Returned 200 OK but Failed the Task. Now What?

Sara — Tue, 01 Sep 2026 15:48:07 +0000

Your API returned 200 OK.

Latency looks normal.

No exceptions were thrown.

Every dashboard is green.

And your AI agent gave the user the wrong answer.

That’s one of the problems teams discover when AI moves from a demo into production: a technically successful request doesn't necessarily mean a successful task.

In traditional applications, failures usually leave evidence. A request times out. A dependency fails. CPU spikes. An exception gets thrown.

AI systems can fail much more quietly.

They can call the wrong tool, retrieve the wrong context, route a request to the wrong agent, silently recover from a failed step, or simply produce an incorrect answer that looks completely reasonable.

As Michael Tuszynski, Principal Architect at Presidio, put it during the What Nobody Tells You About Running AI in Production panel:

“Failure doesn't show up as an error. Failure shows up as a really well-formed answer.”

That changes what we need to monitor.

`200 OK` isn't task success

Consider a relatively simple multi-agent workflow:

User
  ↓
Orchestrator
  ↓
Specialized agent
  ↓
Retrieval
  ↓
Tool call
  ↓
Model
  ↓
Response

Every component in that chain can technically succeed while the overall task fails.

The orchestrator might choose the wrong agent.

The retrieval step might return irrelevant context.

A tool might execute successfully with the wrong parameters.

The model might then take all of that information and produce a polished response.

From the application's perspective:

status = 200
latency = 8.4s
error = false

From the user's perspective:

task_completed = false

That second signal is the one that matters.

Trace the trajectory, not just the response

Once an AI application starts using agents and tools, a prompt and response aren't enough to explain what happened.

Suppose a user asks:

Find a provider near me who accepts my insurance and has availability this week.

The application might need to:

1. Interpret the request
2. Determine the required specialty
3. Route it to a provider agent
4. Search provider data
5. Check insurance eligibility
6. Retrieve appointment availability
7. Reconcile the results
8. Generate the response

If the answer is wrong, knowing that the final model call took 2.8 seconds doesn't help much.

You need the trajectory.

Which agent was selected?
What context did it receive?
Which tools were called?
What arguments were passed?
Did something fail and get retried?
Did the agent recover?
Which model handled each step?

That's where tracing becomes useful.

Treat the complete user request as a trace and the individual actions as spans:

trace: provider_search

├── orchestrator
├── provider_agent
│   ├── retrieve_provider_data
│   ├── check_insurance
│   └── check_availability
├── response_synthesis
└── evaluation

Now, instead of trying to reproduce a bad response, you can reconstruct it.

This is also why instrumenting AI workflows with OpenTelemetry from the beginning makes sense. Models and agent frameworks will change. Your ability to understand what happened shouldn't depend on a particular framework.

Add the signal traditional monitoring doesn't have

Tracing tells you what happened.

It doesn't necessarily tell you whether the result was good.

That's where evals come in.

Kunal Pitale, an engineering leader focused on platform engineering at Atlantic Health, described this as the semantic layer missing from traditional monitoring.

A service can be available. The calls can succeed. The latency can be acceptable.

But was the answer correct?

Did the agent choose the right tool?

Did it actually complete what the user requested?

Production AI needs both operational telemetry and quality signals.

One useful approach is to maintain a set of known-good examples and run them whenever you change a model, prompt, skill, or tool.

Known-good task
      ↓
Run agent
      ↓
Evaluate result
      ↓
Compare with expected behavior

Then production failures can feed that evaluation set:

production failure
      ↓
inspect trace
      ↓
understand what went wrong
      ↓
add regression case
      ↓
change prompt/tool/model
      ↓
rerun evals

The same principle we've used for software regression testing starts applying to AI behavior.

Measure success at the task level

This also changes the metrics that matter.

Instead of stopping at:

request count
error rate
latency
token usage

you start asking:

task success rate
tool failure rate
retry rate
agent loop rate
evaluation score
tokens per successful task
cost per successful task

That last one is particularly interesting.

Imagine two agent configurations:

	Agent A	Agent B
Cost/request	$0.08	$0.12
Task success	62%	94%

Agent A looks cheaper if you're only looking at cost per request.

It looks very different when you ask what a successful outcome costs.

This is why cost, quality, and observability become difficult to separate in production AI systems.

What I'd instrument from day one

If I were putting an agent into production today, at minimum I'd want visibility into:

End-to-end traces across agents, models, and tools
Tool calls, arguments, failures, and retries
Model and prompt versions
Input and output tokens
Cost by agent and task
Evaluation scores
Task success

Most importantly, I'd want those signals correlated.

High token usage alone might not mean anything.

High token usage combined with repeated tool calls, increasing latency, and a falling evaluation score tells you something is going wrong.

Green dashboards aren't enough anymore

Getting an AI application to produce an answer is increasingly easy.

Operating one reliably is a different problem.

The production questions become:


- What did the agent actually do?
- Why did it make that decision?
- Which tools and models were involved?
- Did it accomplish what the user asked?
- How much did that successful task cost?
- Can I reconstruct what happened when it gets something wrong?

Because eventually your AI system will be wrong.

The dangerous failure isn't always an exception.

Sometimes it's a confident, beautifully formatted answer delivered while every traditional health metric remains green.

The goal of AI observability is to make that failure visible.

We're working on this problem at OpenObserve by correlating complete AI sessions with agent and tool traces, model calls, token usage, cost, latency, and agent behavior.

If you're building agents in production, I'd be curious: what signal has been most useful for catching failures that your traditional monitoring missed?

Your System Can't Tell You It's Down: Why Synthetic Monitoring Matters

Sara — Thu, 06 Aug 2026 15:52:52 +0000

Synthetic monitoring detects availability problems before users do by running automated checks against your applications from outside the system. Unlike logs, metrics, and traces, synthetic checks can detect failures even when your application is not receiving traffic.

Your observability stack tells you what happened inside your system.

But what happens when the system itself is unavailable?

A broken TLS certificate, expired domain, failed API endpoint, or broken checkout flow may produce little or no telemetry because requests never reach your application.

That is the gap synthetic monitoring solves.

What is synthetic monitoring?

Synthetic monitoring uses automated probes to continuously test applications, APIs, and user workflows.

Common examples:

Checking an API endpoint returns the expected response
Verifying TLS certificates before expiration
Testing TCP connectivity
Running a complete browser workflow like login or checkout

Instead of waiting for customers to report problems, teams detect failures proactively.

Synthetic monitoring without another tool

Many synthetic monitoring solutions create another observability silo.

You end up with one dashboard for logs, another for metrics, another for traces, and another for synthetic checks.

OpenObserve Synthetic Monitoring keeps check results alongside your existing telemetry.

Synthetic results can be queried, added to dashboards, and used for alerts alongside logs, metrics, and traces.

Test APIs and real user journeys

OpenObserve supports:

HTTP checks
TCP checks
TLS checks
SSH checks

For browser monitoring, checks run using Playwright.

Engineers can use existing Playwright scripts or record browser journeys and convert them into monitors.

The same workflows used for testing can also help detect production issues.

Monitor internal applications

Public synthetic checks cannot always reach internal systems.

Private Locations allow teams to run synthetic checks inside their own network using an outbound-only agent.

No inbound firewall rules.
No exposing internal endpoints.
No separate monitoring infrastructure.

Know why something failed

A failed check should provide more than "availability dropped."

Each run includes execution history and failure evidence so engineers can understand:

Which step failed
Where it failed
What the application returned
Screenshots of browser failures

Instead of only knowing that something broke, teams can quickly understand what happened.

Getting started

Synthetic Monitoring is available in beta on OpenObserve Cloud.

Create a check, choose your schedule and location, and start receiving results in the synthetics_results stream.

Learn more about Synthetic Monitoring at OpenObserve

What's New: Terraform Support, Kubernetes and AWS Automation, Bring Your Own Bucket, and UX Improvements

Sara — Tue, 19 May 2026 15:23:19 +0000

What's New in OpenObserve: Terraform Support, Kubernetes and AWS Automation, Bring Your Own Bucket, and UX Improvements

OpenObserve has shipped three major updates that help engineering teams automate observability, keep full control over telemetry data, and troubleshoot incidents faster.

In this release:

Terraform support for managing OpenObserve deployments and resources as code
Bring Your Own Bucket (BYOB) for Amazon S3 and Azure Blob Storage
UX and UI improvements for logs, distributed tracing, and root cause analysis

If you run observability on Kubernetes, AWS, Azure, or other cloud environments, these updates simplify deployment, improve governance, and streamline day-to-day troubleshooting.

Terraform Support for Observability as Code

OpenObserve now includes a Terraform provider that lets you manage observability resources using infrastructure as code.

Supported resources include:

Streams
Dashboards
Users and organizations
Retention policies
Indexed fields
Full-text search settings

OpenObserve also provides a Kubernetes Terraform module that deploys the platform using the official Helm chart. The module supports both single-node environments and production high-availability deployments with PostgreSQL, NATS, S3, and Ingress.

For AWS users, the module can optionally provision:

Amazon VPC
Amazon EKS
Amazon S3
IAM roles

This makes it possible to manage both the observability platform and its configuration through Terraform or OpenTofu.

Bring Your Own Bucket (BYOB) for Amazon S3 and Azure Blob Storage

Commercial OpenObserve Cloud customers can now connect their own Amazon S3 bucket or Azure Blob Storage container.

Telemetry data remains in your cloud account, region, and security boundary, while OpenObserve continues to handle ingestion, compaction, and querying.

Key benefits include:

Full ownership of logs, metrics, and traces
Data residency and compliance control
Better use of existing cloud storage commitments
No storage lock-in

UX and UI Improvements for Logs and Distributed Tracing

This release also includes several improvements to help engineers move from alert to root cause more quickly.

Highlights include:

Service Catalog
Span details directly in the flame graph
Better default log columns
Multi-stream log correlation
Smarter View Logs filters

These changes reduce the number of clicks required to investigate incidents and correlate logs and traces. Try it!

Get all the details, features, and how-tos:

This article is a summary of the latest OpenObserve release.

For screenshots, implementation details, and links to the Terraform provider and Kubernetes module, read the full announcement.

OpenObserve Just Raised $10M and Launched Observability 3.0 with New AI Capabilities

Sara — Wed, 29 Apr 2026 14:16:01 +0000

Today we’re announcing two things:

A $10M Series A
The launch of Observability 3.0

This funding accelerates a shift we’ve been building toward: Observability 3.0.

Observability is breaking under AI-scale systems.
More data. More tools. More noise.

Most teams are still:

1. Stitching together 6 – 15 tools
2. Sampling away critical data
3. Debugging incidents manually

That model doesn’t scale.

So we built something different.

Observability 3.0 is a shift from dashboards and alerts to systems that:

Correlate data automatically
Detect issues early
Help resolve incidents without manual digging

This includes:

AI SRE (autonomous incident analysis)
Anomaly detection (early warning signals)
LLM observability (visibility into AI systems)

All in a single platform. No fragmentation. No forced tradeoffs.

This is what the Series A is fueling.

👉 Full story, vision, and what we’re building next: https://na2.hubs.ly/H059Cq20