Your AI agent can return HTTP 200 and still be failing

Karan Sahu — Thu, 15 Oct 2026 19:31:56 +0000

An API returning HTTP 200 tells you that a request completed successfully. It doesn't tell you whether an AI agent actually did what it was supposed to do.

That distinction becomes important once agents start handling real workflows.

An agent might successfully call a tool, receive a valid response, and return an answer. From the infrastructure side, everything looks healthy.

But the agent may have selected the wrong tool, failed to complete the task, or produced an output that doesn't pass your application's validation rules.

Traditional uptime monitoring won't necessarily catch these failures.

The problem with measuring availability alone

Consider an agent responsible for validating structured documents.

Before a deployment, most documents pass validation. After a model or prompt update, malformed outputs become more frequent.

The API continues responding successfully, but application-level reliability has deteriorated.

To identify that regression, we need to record more than whether the API request succeeded.

At minimum, useful signals include:

Whether the operation succeeded or failed according to application-defined criteria
What category of failure occurred
Which model or prompt version was involved
Latency, token usage, and retry counts
Whether the system recovered after a change

The application needs to provide these outcomes. A monitoring system cannot reliably infer task correctness just from an HTTP status code.

A reproducible example

I've been working on ReliaMesh, an Apache-2.0 open-source project that explores this problem.

It accepts structured reliability events and uses deterministic detection rules to identify deterioration and recovery.

The repository includes a synthetic example with three stages:

Baseline: 50 successful validation events.
Regression: 50 validation failures associated with a different model-version label.
Recovery: 100 subsequent successful events.

The example checks that a failure-rate regression incident opens and subsequently resolves.

You can reproduce it locally using the installation instructions and the synthetic regression example.

No paid API account or blockchain connection is required.

How the detection works

Rather than sending prompts and responses to another LLM for evaluation, ReliaMesh works with explicitly reported outcomes.

For failure-rate regressions, it compares adjacent observation windows of 50 events.

A regression signal requires a failure-rate increase of at least 15 percentage points, along with separation between the corresponding Wilson confidence intervals.

The system also has rules for high absolute failure rates, latency, token usage, and retries.

This doesn't make the detector infallible. It provides an explainable, reproducible way to identify changes in the signals an application reports.

The detection rules are documented here.

What it doesn't solve yet

ReliaMesh is still an early release.

It doesn't automatically determine whether an LLM response is semantically correct. It doesn't establish root cause, and its detection thresholds haven't been calibrated against independent production workloads.

Cross-organization reliability intelligence is also not enabled.

Those are meaningful limitations, particularly when distinguishing a useful detection signal from a statistically validated reliability claim.

The larger question

As more applications depend on AI agents, infrastructure availability and task reliability need to be treated as separate measurements.

A model can be reachable, a tool can respond, and a workflow can still fail.

I'm interested in how other developers are measuring these failures in production.

Are you relying on application-specific validation, evaluation frameworks, tracing tools, or something else?

Project: https://github.com/PrefilerLabs/reliamesh