DEV Community

Dhruv Mehta
Dhruv Mehta

Posted on

How I Traced My AI Agent's Decision Loop With OpenTelemetry and Signoz (and Caught It Calling the Same Tool Twice)

#agents #ai #debugging #monitoring

From a single JSON response to a full trace of every decision my AI agent made

I built a small AI agent, it answers questions by deciding on its own whether to use a calculator, search some notes, or check the time. The answers looked fine, but one query took eight seconds and another needed four tries to answer something a single tool call should've settled. All I had was a JSON response and a stopwatch. So I traced the whole decision loop with OpenTelemetry and SigNoz. Not just the request, every step inside it. In 30 minutes I could see exactly which iteration was slow, and caught the model calling my calculator with the same input three times in a row. Here's what I did.

The Agent

A ReAct-style loop: ask the model what to do, run a tool if it asks for one, feed the result back, repeat.

def run_agent(query: str) -> dict:
    transcript = [f"User question: {query}"]
    for iteration in range(1, MAX_ITERATIONS + 1):
        prompt = SYSTEM_INSTRUCTIONS + "\n\nConversation so far:\n" + "\n".join(transcript)
        action = _parse_action(_call_claude_api(prompt, iteration))

        if action["action"] == "final_answer":
            return {"answer": action["answer"]}

        result = TOOLS[action["tool"]](action["input"])
        transcript.append(f"Called {action['tool']}({action['input']!r}) -> {result}")
Enter fullscreen mode Exit fullscreen mode

Setup and Three Walls I Hit, in Order 

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
Enter fullscreen mode Exit fullscreen mode 
export OTEL_RESOURCE_ATTRIBUTES="service.name=ai-agent-demo"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
opentelemetry-instrument uvicorn app:app --port 8000
Enter fullscreen mode Exit fullscreen mode

PEP 668. opentelemetry-bootstrap failed with externally-managed-environment. It calls pip install internally, so --break-system-packages on the outer command does nothing. The real fix:

pip config set global.break-system-packages true
Enter fullscreen mode Exit fullscreen mode

Wrong model name. I'm on OpenRouter, and claude-3-5-sonnet-20241022 404'd, my account only had poolside/laguna-m.1:free.

ThinkingBlock has no .text. Some models return a reasoning block before the answer block. Fix: loop through content and grab the first block that actually has .text.

SigNoz Traces tab, filtered to service.name = ai-agent-demo

SigNoz Traces tab, filtered to service.name = ai-agent-demo

Instrumenting the Loop Itself

Auto instrumentation provides built-in tracing for FastAPI requests and outbound HTTP calls. However, it doesn't capture the application's internal decision making process, such as which tool was selected or how many reasoning iterations were executed. So I added manual instrumentation around that logic to generate custom traces and metrics.

with tracer.start_as_current_span("agent.llm_call") as span:
    span.set_attribute("agent.iteration", iteration)
    span.set_attribute("llm.duration_ms", elapsed_ms)
Enter fullscreen mode Exit fullscreen mode 
with tracer.start_as_current_span(f"agent.tool_call.{tool_name}") as span:
    span.set_attribute("tool.input", str(tool_input)[:300])
    span.set_attribute("tool.output", str(result)[:300])
Enter fullscreen mode Exit fullscreen mode

Spans nest by where you open them, so one request produces one root span with however many LLM/tool spans that specific query actually needed.

What the Traces Showed

"47 × 12, plus 8" 4 iterations. Called the calculator, got 572 right on the first try, then called it two more times with the identical input before finally answering. Correct the whole time. You'd never see this in the response, only in the trace.

"What's our cache TTL?" — the model sent the wrong action shape twice, got nudged, corrected itself on try three, answered on try four.

Click into any span and you get the attributes: iteration number, duration, prompt size, tool input/output. "Iteration 3 took 2.4s, prompt was 1,900 chars, called search_notes" that's a diagnosis. "8 seconds total" is just a number.

Multiple agent.tool_call.calculate spans, identical input, between LLM calls.

In the Image we see multiple agent.tool_call.calculate spans, identical input, between LLM calls.

What I Learned

  • Auto instrumentation gets you the network layer free, your own decision logic needs manual spans on purpose.
  • Agents redo work silently, the repeated calculator calls were the single most useful thing tracing caught, precisely because the answer was already correct.

Conclusion

The payoff is bigger: an agent's interesting behavior lives entirely inside the request, and a trace is the only place it's visible.

Top comments (2)

Collapse
 
tobarja profile image
Andrew Thompson •

That's nice, I guess...

Did you convince it to only call the calculator once? Why was the tool call shaped wrong?

Did you just prompt harder?

Collapse
 
dhruv_mehta_c8cf1a3eed42f profile image
Dhruv Mehta •

Repeated calls were probably the free-tier model not trusting its own correct answer. The bad JSON was it collapsing the tool name into the wrong field, likely from no few-shot example in the prompt. Didn't prompt harder, just let the retry-nudge handle it. Good ideas for a follow-up post though.