How I Traced My AI Agent's Decision Loop With OpenTelemetry and Signoz (and Caught It Calling the Same Tool Twice)

Dhruv Mehta — Sat, 18 Jul 2026 02:35:47 +0000

From a single JSON response to a full trace of every decision my AI agent made

I built a small AI agent, it answers questions by deciding on its own whether to use a calculator, search some notes, or check the time. The answers looked fine, but one query took eight seconds and another needed four tries to answer something a single tool call should've settled. All I had was a JSON response and a stopwatch. So I traced the whole decision loop with OpenTelemetry and SigNoz. Not just the request, every step inside it. In 30 minutes I could see exactly which iteration was slow, and caught the model calling my calculator with the same input three times in a row. Here's what I did.

The Agent

A ReAct-style loop: ask the model what to do, run a tool if it asks for one, feed the result back, repeat.

def run_agent(query: str) -> dict:
    transcript = [f"User question: {query}"]
    for iteration in range(1, MAX_ITERATIONS + 1):
        prompt = SYSTEM_INSTRUCTIONS + "\n\nConversation so far:\n" + "\n".join(transcript)
        action = _parse_action(_call_claude_api(prompt, iteration))

        if action["action"] == "final_answer":
            return {"answer": action["answer"]}

        result = TOOLS[action["tool"]](action["input"])
        transcript.append(f"Called {action['tool']}({action['input']!r}) -> {result}")

Setup and Three Walls I Hit, in Order

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_RESOURCE_ATTRIBUTES="service.name=ai-agent-demo"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
opentelemetry-instrument uvicorn app:app --port 8000

PEP 668. opentelemetry-bootstrap failed with externally-managed-environment. It calls pip install internally, so --break-system-packages on the outer command does nothing. The real fix:

pip config set global.break-system-packages true

Wrong model name. I'm on OpenRouter, and claude-3-5-sonnet-20241022 404'd, my account only had poolside/laguna-m.1:free.

ThinkingBlock has no .text. Some models return a reasoning block before the answer block. Fix: loop through content and grab the first block that actually has .text.

SigNoz Traces tab, filtered to service.name = ai-agent-demo

Instrumenting the Loop Itself

Auto instrumentation provides built-in tracing for FastAPI requests and outbound HTTP calls. However, it doesn't capture the application's internal decision making process, such as which tool was selected or how many reasoning iterations were executed. So I added manual instrumentation around that logic to generate custom traces and metrics.

with tracer.start_as_current_span("agent.llm_call") as span:
    span.set_attribute("agent.iteration", iteration)
    span.set_attribute("llm.duration_ms", elapsed_ms)

with tracer.start_as_current_span(f"agent.tool_call.{tool_name}") as span:
    span.set_attribute("tool.input", str(tool_input)[:300])
    span.set_attribute("tool.output", str(result)[:300])

Spans nest by where you open them, so one request produces one root span with however many LLM/tool spans that specific query actually needed.

What the Traces Showed

"47 × 12, plus 8" 4 iterations. Called the calculator, got 572 right on the first try, then called it two more times with the identical input before finally answering. Correct the whole time. You'd never see this in the response, only in the trace.

"What's our cache TTL?" — the model sent the wrong action shape twice, got nudged, corrected itself on try three, answered on try four.

Click into any span and you get the attributes: iteration number, duration, prompt size, tool input/output. "Iteration 3 took 2.4s, prompt was 1,900 chars, called search_notes" that's a diagnosis. "8 seconds total" is just a number.

In the Image we see multiple agent.tool_call.calculate spans, identical input, between LLM calls.

What I Learned

Auto instrumentation gets you the network layer free, your own decision logic needs manual spans on purpose.
Agents redo work silently, the repeated calculator calls were the single most useful thing tracing caught, precisely because the answer was already correct.

Conclusion

The payoff is bigger: an agent's interesting behavior lives entirely inside the request, and a trace is the only place it's visible.