Track 3: AI & Agent Observability — "Agents of SigNoz" Hackathon (WeMakeDevs × SigNoz, July 2026)
The Problem
Every team at a hackathon like this is going to show you an agent that's slow, or an agent that errors.
- We're showing you something worse—and far more common.
- A multi-agent system where nothing ever fails.
- No exception. No page. No red dashboard.
- And yet the system is still stuck.
Tasks bounce endlessly between an Executor and a Critic. Work is quietly repeated even after it's completed. Sometimes a task simply sits there because no agent picks it up. Meanwhile, LLM tokens continue to burn and costs keep increasing.
Traditional observability focuses on individual agent calls:
- Latency
- Errors
- Cost per call
It was never built to observe the thing that actually breaks multi-agent systems:
Coordination between agents.
A task endlessly bouncing between two agents doesn't look like a failure.
To most observability tools, everything still looks perfectly healthy.
What is AgentATC?
AgentATC is a real (not scripted) three-agent workflow consisting of a Planner, an Executor, and a Critic that collaborates to generate a competitive analysis report.
The workflow is straightforward:
- Planner breaks a user's goal into subtasks.
- Executor drafts each section using a search tool and an LLM.
- Critic evaluates every draft against a deterministic rubric. Rejected work loops back to the Executor with feedback, up to five iterations before escalation.
That workflow alone isn't what makes AgentATC interesting.
What makes it different is that every hand-off between agents becomes a first-class OpenTelemetry span.
Instead of trying to infer coordination after the fact from timestamps, AgentATC records coordination directly.
Every span such as:
- agent.execute
- agent.handoff
- agent.review
carries information about:
Which agent initiated the action
Which agent received it
Why the hand-off happened
Coordination becomes observable data rather than an invisible gap between API calls.
How the Swarm Coordinates
Every task is assigned:
One task_id
One root span (task.process)
One connected trace throughout the execution
This architectural decision is what keeps the entire workflow connected.
Without it, every HTTP request between the Orchestrator, Planner, Executor, and Critic would generate independent traces, fragmenting the execution into unrelated pieces.
Each subtask runs inside its own nested span while the Executor and Critic iterate until convergence.
Whenever the Critic rejects a draft,
agentatc_handoffs_total{reason="reject"}
is incremented.
When a task stalls, nothing throws an exception.
Nothing crashes.
Instead,
agentatc_task_last_update_timestamp
simply stops moving forward, providing the only indication that no agent is making progress anymore.
A Successful Execution
Submit the goal:
"Write a competitive analysis of three observability vendors: SigNoz, Datadog, and New Relic."
The Planner creates three subtasks.
Two of them—
SigNoz
New Relic
—converge immediately and pass the Critic on the first review.
That's the expected path.
It should look boring.
Successful coordination shouldn't generate excitement.
When Things Go Wrong
The Datadog subtask is intentionally different.
Its mock search tool can never retrieve pricing information, while the Critic's evaluation rubric requires pricing to approve the draft.
As a result, the Executor can never satisfy the Critic.
Every execution guarantees:
Five review iterations
Five rejections
One escalation
No randomness.
No hoping an LLM behaves the same during a live demo.
Because everything shares one trace ID, opening the trace reveals the entire history as one connected execution, showing every Executor ↔ Critic hand-off across all services.
Complete Transparency with SigNoz Dashboards and Copilot
The Swarm Health Overview dashboard tracks:
- Convergence rate
- Average approval iterations
- In-flight subtasks
- Redundant tool calls
The Thrash Leaderboard ranks subtasks by rejection count.
Datadog consistently appears at the top, making the coordination failure immediately visible.
Two production alerts help detect coordination failures early:
Task Thrashing
Task Stalled
Both alerts can automatically trigger the Copilot.
The Copilot uses SigNoz's MCP Server to investigate another agent's failure the way an on-call engineer would.
It retrieves telemetry using:
- signoz_search_traces
- signoz_get_trace_details
- signoz_search_logs
Most importantly, it is built to say "No data found" instead of fabricating an answer.
During one escalated execution it concluded:
"The Critic consistently rejected the submissions for the same reason: missing pricing information... this indicates a TOOLING GAP. The executor's data sources do not provide the required pricing information, leading to consistent rejections by the critic."
That distinction is the entire value proposition.
Not simply that a task failed five times.
But why it failed, backed by actual telemetry.
Why SigNoz Matters
We didn't bolt dashboards onto the project after the fact.
Traces, metrics, and logs all flow through SigNoz for real analytical work.
Coordination metrics are computed directly from raw telemetry in SigNoz's query layer instead of being pre-calculated inside application code.
While migrating dashboards to the Query Builder, we discovered that one PromQL query grouped results using a label that didn't actually exist.
The dashboard rendered successfully.
The data was wrong.
That wasn't a SigNoz issue.
It was an instrumentation bug in our own project that only became visible because we validated the dashboards against live telemetry instead of assuming a rendered graph must be correct.
Technology Stack
- Backend: FastAPI microservices (Orchestrator, Planner, Executor, Critic, Copilot)
- Frontend: React + Vite
- Observability: OpenTelemetry auto-instrumentation
- Transport: OTLP over gRPC
- Monitoring: Self-hosted SigNoz via Foundry
- Investigation: SigNoz MCP Server
Lessons Learned
The failure mode we set out to expose includes:
- Thrashing
- Stalled tasks
- Redundant work
These failures produce almost no useful signal in traditional monitoring systems.
Building the metrics to observe them forced us to think about coordination as telemetry, rather than treating it as a side effect of healthy service calls.
Going back to validate our dashboards against live data instead of trusting that they rendered correctly also uncovered a genuine instrumentation bug that would otherwise have shipped unnoticed.
Conclusion
We didn't build another AI agent demo.
We built the instrument panel for the failure mode every multi-agent team is about to discover the hard way.
The one where:
Everything looks healthy.
Nothing ever fails.
And the system is stuck anyway.
Top comments (0)