How I Traced My AI Agent's Decision Loop With OpenTelemetry and Signoz (and Caught It Calling the Same Tool Twice)

#agents #ai #debugging #monitoring

From a single JSON response to a full trace of every decision my AI agent made

I built a small AI agent, it answers questions by deciding on its own whether to use a calculator, search some notes, or check the time. The answers looked fine, but one query took eight seconds and another needed four tries to answer something a single tool call should've settled. All I had was a JSON response and a stopwatch. So I traced the whole decision loop with OpenTelemetry and SigNoz. Not just the request, every step inside it. In 30 minutes I could see exactly which iteration was slow, and caught the model calling my calculator with the same input three times in a row. Here's what I did.

The Agent

A ReAct-style loop: ask the model what to do, run a tool if it asks for one, feed the result back, repeat.

def run_agent(query: str) -> dict:
    transcript = [f"User question: {query}"]
    for iteration in range(1, MAX_ITERATIONS + 1):
        prompt = SYSTEM_INSTRUCTIONS + "\n\nConversation so far:\n" + "\n".join(transcript)
        action = _parse_action(_call_claude_api(prompt, iteration))

        if action["action"] == "final_answer":
            return {"answer": action["answer"]}

        result = TOOLS[action["tool"]](action["input"])
        transcript.append(f"Called {action['tool']}({action['input']!r}) -> {result}")

Setup and Three Walls I Hit, in Order

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_RESOURCE_ATTRIBUTES="service.name=ai-agent-demo"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
opentelemetry-instrument uvicorn app:app --port 8000

PEP 668. opentelemetry-bootstrap failed with externally-managed-environment. It calls pip install internally, so --break-system-packages on the outer command does nothing. The real fix:

pip config set global.break-system-packages true

Wrong model name. I'm on OpenRouter, and claude-3-5-sonnet-20241022 404'd, my account only had poolside/laguna-m.1:free.

ThinkingBlock has no .text. Some models return a reasoning block before the answer block. Fix: loop through content and grab the first block that actually has .text.

(SigNoz Traces tab, filtered to service.name = ai-agent-demo — several traces, visibly different durations)

Instrumenting the Loop Itself

Auto-instrumentation covers FastAPI and outbound HTTP calls for free. The decision logic, which tool, how many iterations, so I wrapped it manually:

with tracer.start_as_current_span("agent.llm_call") as span:
    span.set_attribute("agent.iteration", iteration)
    span.set_attribute("llm.duration_ms", elapsed_ms)

with tracer.start_as_current_span(f"agent.tool_call.{tool_name}") as span:
    span.set_attribute("tool.input", str(tool_input)[:300])
    span.set_attribute("tool.output", str(result)[:300])

Spans nest by where you open them, so one request produces one root span with however many LLM/tool spans that specific query actually needed.

What the Traces Showed

"47 × 12, plus 8" 4 iterations. Called the calculator, got 572 right on the first try, then called it two more times with the identical input before finally answering. Correct the whole time. You'd never see this in the response, only in the trace.

(Waterfall: three agent.tool_call.calculate spans, identical input, between LLM calls)

"What's our cache TTL?" — the model sent the wrong action shape twice, got nudged, corrected itself on try three, answered on try four.

(Waterfall: two invalid attempts, then the successful search_notes call)

Click into any span and you get the attributes: iteration number, duration, prompt size, tool input/output. "Iteration 3 took 2.4s, prompt was 1,900 chars, called search_notes" that's a diagnosis. "8 seconds total" is just a number.

(Attributes panel for a single agent.llm_call span)

What I Learned

Auto-instrumentation gets you the network layer free, your own decision logic needs manual spans on purpose.
Agents redo work silently, the repeated calculator calls were the single most useful thing tracing caught, precisely because the answer was already correct.