DEV Community

Cover image for We Built a Flight Recorder for AI Coding Agents: Here's What SigNoz Taught Us About Watching Them Think
hussain jamal
hussain jamal

Posted on

We Built a Flight Recorder for AI Coding Agents: Here's What SigNoz Taught Us About Watching Them Think

#signoz #opentelemetry #ai #wemakedevs

We Built an AI Agent... Then Realized We Had No Idea What It Was Doing

Our AI agent was writing code, running tests, and opening pull requests, but I had absolutely no idea why some runs took 8 seconds while others took 45.

Was the LLM thinking too long?

Was it stuck retrying a failing shell command?

Was Docker slow?

I couldn't tell.

All I had was a loading spinner and a final result.

That gap between knowing an agent is doing something and actually knowing what it's doing is exactly why we built AXRAY, using SigNoz as our observability backbone.

This is the story of building it, the architecture behind it, and the deployment pitfalls that nearly broke everything.

🛑 Why This Needed Solving

Autonomous coding agents don't behave like traditional microservices.

A single agent turn might involve:

  • An LLM generating a multi-step plan
  • Searching the codebase
  • Reading files
  • Writing code
  • Running commands inside Docker
  • Executing tests
  • Creating Git commits
  • Opening pull requests

Every one of those actions has its own latency.

Every one can fail independently.

Traditional application logs flatten all of this into an unreadable wall of text.

You can't tell whether the agent was:

  • Thinking
  • Waiting
  • Retrying
  • Actually executing code

That distinction matters.

If latency comes from the LLM generating thousands of reasoning tokens, the solution is prompt optimization.

If latency comes from a hanging shell command, the solution is a timeout or sandbox fix.

Without separating those two, you're simply guessing.

What We Built

AXRAY instruments every agent turn using OpenTelemetry, following the official GenAI Semantic Conventions instead of inventing our own telemetry schema.

Examples include:

  • gen_ai.request.model
  • gen_ai.usage.input_tokens
  • gen_ai.usage.output_tokens

Every span is tagged with a simple phase:

  • llm
  • tool

From that we calculate two metrics for every agent turn.

Time-in-Brain

How long the LLM spent thinking, reasoning, and generating tokens.

Time-in-Environment

How long Docker actually spent executing commands.

Those two numbers unlock almost everything.

We also compute an overall execution efficiency score:

const efficiencyScore = Math.max(
  45,
  Math.min(98, 100 - (0.35 * brainPercent + 0.10 * envPercent))
);
Enter fullscreen mode Exit fullscreen mode

The first time I watched a real execution session, I noticed something surprising.

Almost 78% of the latency was inside the LLM.

Our Docker sandbox wasn't slow.

Our shell commands weren't slow.

The context window was simply too large.

That wasn't speculation.

It came directly from a SigNoz trace query against:

signoz_traces.signoz_index_v3

Where SigNoz Did the Heavy Lifting

SigNoz ended up powering three completely different layers of AXRAY.

1️⃣ OTLP Export for Raw Instrumentation

Every LLM request and every tool invocation exports spans through OTLP (:4318).

Once we mapped our attributes onto the official GenAI semantic conventions, everything started fitting naturally into the existing observability ecosystem.

No custom telemetry format required.

2️⃣ Direct ClickHouse Queries

We wanted sub-turn latency breakdowns that standard dashboards don't expose directly.

Because SigNoz stores traces inside ClickHouse, we could run custom SQL like this:

SELECT
    attributes_string['tool.name'] AS toolName,
    avg(durationNano) AS avgDurationNano,
    count() AS executionCount
FROM signoz_traces.signoz_index_v3
WHERE name = 'tool.call'
AND attributes_string['axray.session.id'] = 'sess_42'
GROUP BY toolName
ORDER BY avgDurationNano DESC;
Enter fullscreen mode Exit fullscreen mode

That query instantly showed which tools consumed the most execution time across an entire session.

3️⃣ SigNoz MCP Server

One feature we really wanted was live alert visibility inside AXRAY itself.

Instead of rebuilding an alerting system, we connected directly to SigNoz's MCP server.

Calling:

signoz_list_alerts
Enter fullscreen mode Exit fullscreen mode

through StreamableHTTPClientTransport gave us structured JSON containing active alert rules.

This meant:

  • token spikes
  • cost anomalies
  • latency alerts

could appear directly inside AXRAY's UI without duplicating any of SigNoz's alerting logic.

The Part That Broke

Here's the part I wish someone had warned me about.

Outdated Deployment Tutorials

Most tutorials still describe the classic:

  • docker-compose
  • install.sh

installation flow.

That isn't the recommended deployment anymore.

SigNoz has moved to Foundry, driven by an extremely small YAML manifest.

apiVersion: v1alpha1
kind: Installation

metadata:
  name: signoz

spec:
  deployment:
    flavor: compose
    mode: docker

  mcp:
    spec:
      enabled: true
Enter fullscreen mode Exit fullscreen mode

Deployment becomes a single command:

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

Foundry:

  • validates prerequisites
  • generates Docker Compose
  • starts the platform

It's actually simpler than the old approach.

I just lost an afternoon following outdated tutorials before discovering it.

Hardcoded UUIDs

The second bug was much sneakier.

I wrote a script that automatically imported dashboards and alert rules into SigNoz's Postgres metadata database.

Everything worked perfectly.

On my machine.

Why?

Because I had accidentally hardcoded my own:

  • org_id
  • user_id

Every fresh SigNoz installation generates completely different UUIDs.

That meant my setup script silently failed on a clean installation.

The fix was querying them dynamically.

function getOrgAndUser(container) {
  const orgId = runSql(
    container,
    "SELECT id FROM organizations LIMIT 1;"
  ).stdout.trim();

  const userId = runSql(
    container,
    "SELECT id FROM users LIMIT 1;"
  ).stdout.trim();

  return { orgId, userId };
}
Enter fullscreen mode Exit fullscreen mode

Tiny change.

Huge difference.

It's exactly the kind of bug that only appears when someone else runs your project.

What I'd Tell My Past Self

  • Test every setup script on a completely fresh machine.
  • Never hardcode IDs that can be generated dynamically.
  • Instrument "thinking" separately from "doing."
  • Use standard OpenTelemetry conventions whenever possible.
  • Production AI systems deserve production-grade observability.

Final Thoughts

AXRAY started with one simple idea:

"Let's add some logging to our AI agent."

It ended up becoming something much bigger.

A strong argument that AI agents should be treated exactly like any production backend:

  • Trace everything.
  • Measure everything.
  • Alert on everything.
  • Never trust a black box.

Using SigNoz's OpenTelemetry-native architecture meant we didn't have to invent our own telemetry format, and that decision paid off far more than we expected.

If you're building AI agents that combine LLM reasoning with real system execution, the very first metric I'd instrument is:

Time-in-Brain vs Time-in-Environment

It's inexpensive to add.

And it immediately answers the only question that matters when an agent feels slow:

Was it thinking… or was it stuck?

Built for

WeMakeDevs × SigNoz — Agents of SigNoz Hackathon

If you found this interesting, I'd love your feedback!

🔗 Links

*Thanks for reading! *

Top comments (0)