I Built a Crew of AI Agents That Review Code Like a Real Team — Then Watched Them Argue With SigNoz

My submission for the Agents of SigNoz Hackathon (Track: AI & Agent Observability)

The idea

Most "AI code review" demos are one LLM call with a clever prompt. That's fine, but it doesn't reflect how review actually works on a real team — different people care about different things. Someone obsesses over edge cases. Someone else nitpicks naming. Someone else only cares if it's going to be slow in production. And then someone has to actually make the call on whether the PR merges.

So I built that as a crew: a Logic Reviewer, a Style Reviewer, and a Performance Reviewer — three independent agents, each with a narrow system prompt that tells them to only look at their lane — followed by a Moderator agent that reads all three opinions and produces one final verdict, calling out disagreement when it happens.

The interesting engineering problem wasn't the prompting. It was: once you have four chained LLM calls, how do you actually know what's happening inside your own system?

Why observability, not just another agent demo

Once I had the crew working, I had zero visibility into it. Four sequential API calls, each with its own latency and token cost, and all I had was print() statements. If the moderator gave a weird verdict, I had no fast way to tell whether the logic reviewer hallucinated an issue, or the moderator just summarized badly. If a run felt slow, I couldn't tell which of the four agents was the bottleneck.

This is exactly the gap SigNoz is built for, so I instrumented every agent call with OpenTelemetry:

• Each specialist agent and the moderator run inside their own span (agent.logic_reviewer, agent.style_reviewer, agent.performance_reviewer, agent.moderator)
• All four are nested under one parent span, code_review_session, so a single review run shows up as one trace with four child spans
• Every span carries the attributes that actually matter for debugging an agent: model name, input/output token counts, latency, and a preview of what the agent actually said

What the traces showed me

Once this was wired up and running against a deliberately broken sample file (a function that divides by zero on empty input, an O(n²) duplicate checker, some PEP 8 violations), a few things became obvious immediately in SigNoz that were invisible before:

The performance reviewer was consistently the slowest agent, not because it needed more tokens, but because its system prompt asked it to reason about complexity line by line — visible directly in the latency attribute on its span, sitting noticeably higher than the other two specialists.

Token cost wasn't evenly distributed. I'd assumed the moderator would be the expensive step since it reads three reviews plus the code. In practice, the style reviewer's output was often the longest because it listed every naming violation individually — something I only noticed by comparing llm.output_tokens across spans instead of guessing.

Disagreement between agents is traceable, not just visible in the final text. When the logic reviewer flagged the division-by-zero bug as FAIL and the style reviewer gave a PASS on the same code (because from a pure naming/readability lens, it is fine), I could see both verdicts side by side in the trace before the moderator even ran — which made it easy to check the moderator was actually reconciling them, not just averaging.

What broke

The first version had a bug where the moderator span wasn't properly nested — it showed up as a sibling trace instead of a child span, because I'd called setup_telemetry() after already starting the tracer in one of the agent modules. Fixed by consolidating tracer setup into a single telemetry.py module imported once, at the top of main.py, before any agent code runs. A small thing, but it's exactly the kind of bug that's invisible until you're looking at a trace waterfall and asking "why is this span not a child of that one."

The second thing I learned the hard way: don't set span attributes after the LLM call inside a bare try without a finally or exception-aware except block — if the call throws, you lose the span entirely instead of getting an error trace. Wrapping it properly with span.record_exception() and span.set_status(Status(StatusCode.ERROR, ...)) means failed agent calls still show up in SigNoz as failed spans, which is far more useful than a silent crash.

What I'd add next

• Span events for each individual issue an agent finds, not just one span per agent call — that would let me trace "how many logic bugs get caught vs missed" over many runs, not just per-session token cost
• A security-focused specialist agent, run as a fourth reviewer
• Wiring this up to a real GitHub webhook so it reviews actual pull requests, not just a local sample file

Try it

The full project — four agents, the OpenTelemetry/SigNoz setup, and a deliberately buggy sample file to review — is up on GitHub. Clone it, drop in your Anthropic API key and SigNoz endpoint, and run python main.py to see your own crew's trace show up in SigNoz within seconds.

Building this reinforced something I didn't fully appreciate before: with multi-agent systems, the hard part stops being "can I get an LLM to do the task" and starts being "can I actually see what my agents are doing to each other." That's the whole pitch of agent-native observability, and it held up in practice.