My AI Agent Refunded a Customer Twice, and Every Status Code Was 200

Monarch — Sat, 25 Jul 2026 19:43:54 +0000

Last weekend, during the Agents of SigNoz hackathon, I wanted to avoid building yet another self-healing agent or a demo that simply detected an error and retried the same workflow.

I was more interested in a harder class of failure: cases where an AI agent behaves incorrectly even though every service involved works exactly as expected.

So I built a deliberately flawed support agent that could refund ₹4,998 against a ₹2,499 duplicate charge. Nothing crashed. There was no exception, no 500, and no retry storm. Every service returned 200, and the agent confidently told the customer their problem had been solved.

On a traditional health dashboard, the entire system looked green while its behaviour was financially wrong.

This post is about what I built to catch that class of failure: Trace2Test, a control plane that compiles failed agent traces from SigNoz into deterministic, replayable regression tests.

It is also about the debugging detour that consumed three hours of a six-hour sprint.

The problem: agents fail semantically, not operationally

Some of the worst failures in an LLM agent do not look like errors.

The model may call the correct tool with the wrong arguments. It may call the right tool twice. It may correctly interpret the API schema while misunderstanding the business rule behind it.

HTTP-level observability says everything worked because, operationally, everything did work. The payments API successfully executed the refund it was asked to execute. The problem was that the agent should never have requested that refund in the first place.

My assumption going into the hackathon was that the trace is the most complete record of what the agent actually did.

A trace should therefore be more than evidence for a post-mortem. It should be something we can compile into a test.

The loop I wanted looked like this:

SigNoz detects a semantic failure → an incident opens automatically → the failed trace becomes a frozen recorded world → the same agent binary replays inside it deterministically for $0 → a candidate fix runs against that world with a real LLM before any customer sees it.

The demo system

The demo consists of three FastAPI services, all instrumented with the OpenTelemetry Python SDK and exporting OTLP telemetry to SigNoz.

I used Foundry to stand up the environment through a casting.yaml file. One command launched SigNoz and its MCP server locally.

The services were:

agent-gateway — the RefundBot agent, using gpt-4o-mini with tool calling
payments-svc — charge and refund operations, with refunds forced into dry_run mode
crm-svc — customer lookup

The telemetry contract is the heart of the system.

Every agent run emits spans with a predictable structure:

  agent.run                    root span: agent.id, prompt version, request JSON
  llm.exchange                 one per model call: request/response, tokens
  tool.payments.refund_charge  tool spans: tool.mode = live | dry_run | simulated
  eval.outcome                 verdict span: agent.outcome, attempted vs expected

The evaluator is deliberately boring.

It sums the refunds attempted by the agent and compares the result with the correct refund amount for the support ticket.

The buggy v1 prompt tells the agent to refund every charge involved in a duplicate-charge incident. As a result, the eval.outcome span contains:

agent.outcome        = semantic_failure
agent.eval.attempted = 499800
agent.eval.expected  = 249900

The values are stored in minor currency units.

That span becomes the ground truth for the rest of the system.

Detection: the alert that refused to fire

My first attempt used a metric-based alert.

I created a counter called agent.eval.failures, incremented it whenever an evaluation produced a semantic failure, and configured a threshold rule over its per-second rate.

Textbook observability.

It fired once during early testing and then went permanently silent.

The frustrating part was that every layer I inspected appeared to be working.

The metric existed in ClickHouse. A request to:

POST /api/v5/query_range

using the exact query from the alert rule returned a clean, non-zero spike.

The aggregation matched. The window matched. The data was there.

But SigNoz's alert evaluator continued printing:

alert.count=0

on every evaluation cycle.

I eventually inspected the time_series_v4 table and found the real problem.

The metric had 29 cumulative series and 6 delta series under the same metric name.

Partway through the sprint, I had changed the OTLP exporter to use delta temporality. The intention was to make rate alerts more resilient to service restarts.

Ironically, those restarts helped create the failure.

The services had restarted around a dozen times, and each restart created new series. My manual queries could still resolve the data, but the rule evaluator struggled with the mixed temporality under the same metric name.

At that point, I time-boxed the ClickHouse investigation and switched to something structurally simpler: alerting directly on the evaluation span.

SigNoz v5 threshold rules can query traces, so the rule became a count over matching spans:

"signal": "traces",
"aggregations": [{"expression": "count()"}],
"filter": {
    "expression": "name = 'eval.outcome' "
                  "AND agent.outcome = 'semantic_failure' "
                  "AND trace2test.mode NOT EXISTS"
},

No temporality. No series lifecycle. No counter resets. No metric-cardinality problem.

Just one question:

Did a semantic-failure span exist during the last five minutes?

The rule fired roughly 3.5 minutes after the next seeded failure and remained reliable after that.

I also learned two smaller provisioning lessons in the same debugging session.

First, the SigNoz v0.134 v2alpha1 rule API silently requires a notificationSettings block. Rules without it are rejected.

Second, provisioning scripts should upsert rules with PUT. My first implementation searched for a rule by name and happily returned the stale version instead of updating it.

The following clause also turned out to be essential:

trace2test.mode NOT EXISTS

Replay runs emit real telemetry.

My first successful end-to-end replay generated another semantic_failure span, triggered the alert again, and opened a second incident for the replay itself.

The system had successfully begun debugging its own debugging process.

I closed that feedback loop by propagating replay context through W3C baggage and excluding replay spans from the production detection rule.

The replay trick: swap the world, not the agent

When the alert fires, SigNoz sends a webhook to the Trace2Test control plane.

The control plane retrieves the full failed trace through the Trace API and compiles it into a case bundle containing:

the original agent request
every LLM request and response
every tool call and tool response
the expected evaluation verdict
a hash of the generated fixture

The key idea behind replay is to run the same agent binary while replacing the environment around it.

I used two seams.

1. Replay the recorded LLM exchanges

During deterministic replay, OPENAI_BASE_URL points to a local simulator.

The simulator serves the recorded model responses from the failed trace, in order.

The agent still believes it is communicating with an OpenAI-compatible API. Its application logic does not change, but no real model request is made.

2. Replay the recorded tool world

W3C baggage carries the following values through every service hop:

trace2test.case_id
trace2test.variant
trace2test.mode

The payments and CRM services inspect that baggage.

When the request belongs to a replay, they return the recorded responses instead of accessing real payment or customer systems.

A span processor copies the trace2test.* baggage values onto every generated span.

This means the replay is not hidden from observability. It appears in SigNoz as a first-class trace that can be compared directly with the original failure.

The waterfall structure remains visible, but the replay is tagged with:

variant = baseline

and every tool span records:

tool.mode = simulated

The baseline replay reproduced the double-refund failure in 591 ms for $0.00.

Verifying the fix

Reproducing the failure deterministically proves that the fixture is faithful.

The next step is verifying a candidate fix.

For verification, I replaced the buggy v1 prompt with a corrected v2 prompt.

The environment stayed frozen:

same support request
same customer
same charges
same tool responses
same expected refund amount

The only meaningful change was the prompt version.

The simulator switched into proxy mode, allowing the fixed agent to call a real gpt-4o-mini model while keeping the tool world deterministic.

I ran the candidate fix three times.

All three runs passed.

The total model cost was:

$0.001245

The complete golden path—

seed failure → detect alert → open incident → compile trace → reproduce failure → verify fix

—completed in 235 seconds.

The repository includes a make golden target that executes the entire sequence.

What I would tell my past self

Alert on spans for discrete semantic events

Metrics are useful for rates, saturation, resource usage, and long-running trends.

But when the thing you care about is an individual event such as “an agent evaluation failed,” the span already is that event.

Counting matching spans avoids the entire temporality and counter-lifecycle problem.

That is what worked for this system, and it is what I would reach for first when detecting semantic agent failures again.

Exclude replays from detection immediately

Do not wait until your replay system raises an incident against itself.

Replay traffic should be identifiable from the first version of the telemetry contract.

Propagate replay metadata through baggage, copy it onto spans, and exclude those spans from production alerts.

Query results and alert evaluation are different code paths

A query returning data does not prove that the alert evaluator sees the same result.

My manual query worked while the rule evaluator continued reporting:

alert.count=0

The evaluator logs were the actual source of truth.

I should have checked them much earlier.

Deterministic replay is a boundary decision

I achieved deterministic replay by recording at the network boundaries:

the LLM API
the tool HTTP APIs

That allowed the same agent implementation to run in production, baseline replay, and candidate verification.

The agent logic did not need separate test-only branches.

Recording at a finer level would have required mocking internal functions and likely allowed more nondeterminism to leak into the replay.

The useful abstraction was not “mock the agent.”

It was “replace the world around the agent.”

Wrap-up

A production agent failure should happen exactly once.

After that, it should become a regression test: compiled from its trace, replayable deterministically, cheap to execute, and ready to guard against the next prompt or model change.

Trace2Test was my attempt to build that loop.

SigNoz's existing pieces—OpenTelemetry-native traces, trace-based alerting, webhooks, and the Trace API—were enough to build the full workflow during a hackathon, despite the metric alert detour.

The result is a control plane that turns an observability artifact into an executable test case.

Code, telemetry contract, and the one-command golden path:

github.com/anima-regem/trace2test

3 : Procrastination slowing taking over me!

Monarch — Mon, 08 Aug 2022 19:39:55 +0000

Sheesh! Procrastination is getting the better of me. I have been lazy and barely do a CodeChef problem a day! I need to work on that. It might be because of the exams but I doubt that is it. I've been a ton more productive when I was alone, both here and at home. I'm doing simple CodeChef problems and at least a simple GoLang project a day. The hope is still strong #30DaysofCode.

Day 2 (06/08/2022) - The Procrastination

Monarch — Fri, 05 Aug 2022 19:37:00 +0000

Aah! Today procrastination got the good outta me! Still I'm fired up to make up for all the time I wasted. Did a simple code-chef problem and completed a one shot Go video. Will be doing some simple Go projects from tomorrow. And since university exams are coming up, time's getting a bit crammed, along with that, my flask skills pleading me to spare it at least a glance and the backend integration of college admin page. PS: A friend did almost all the UI work, who am I to complain :')

Here goes a new series, Code rant. 05/08/2022

Monarch — Thu, 04 Aug 2022 19:08:32 +0000

About Me

Honestly, I am one of the greatest procrastinators out there. I have had great ideas and plans that were ruined solely due to procrastination. This series is to document or record my journey. Currently I'm working on my basic coding skills with Go, learning a bit of devops and working on my Flask, React, Node and Svelte skills.

DEV Community: Monarch