AgentATC

I Jithindranaath — Sat, 25 Jul 2026 22:54:59 +0000

Track 3: AI & Agent Observability — "Agents of SigNoz" Hackathon (WeMakeDevs × SigNoz, July 2026)

The Problem

Every team at a hackathon like this is going to show you an agent that's slow, or an agent that errors.

We're showing you something worse—and far more common.
A multi-agent system where nothing ever fails.
No exception. No page. No red dashboard.
And yet the system is still stuck.

Tasks bounce endlessly between an Executor and a Critic. Work is quietly repeated even after it's completed. Sometimes a task simply sits there because no agent picks it up. Meanwhile, LLM tokens continue to burn and costs keep increasing.

Traditional observability focuses on individual agent calls:

Latency
Errors
Cost per call

It was never built to observe the thing that actually breaks multi-agent systems:

Coordination between agents.

A task endlessly bouncing between two agents doesn't look like a failure.

To most observability tools, everything still looks perfectly healthy.

What is AgentATC?

AgentATC is a real (not scripted) three-agent workflow consisting of a Planner, an Executor, and a Critic that collaborates to generate a competitive analysis report.

The workflow is straightforward:

Planner breaks a user's goal into subtasks.
Executor drafts each section using a search tool and an LLM.
Critic evaluates every draft against a deterministic rubric. Rejected work loops back to the Executor with feedback, up to five iterations before escalation.

That workflow alone isn't what makes AgentATC interesting.

What makes it different is that every hand-off between agents becomes a first-class OpenTelemetry span.

Instead of trying to infer coordination after the fact from timestamps, AgentATC records coordination directly.

Every span such as:

agent.execute
agent.handoff
agent.review

carries information about:

Which agent initiated the action
Which agent received it
Why the hand-off happened

Coordination becomes observable data rather than an invisible gap between API calls.

How the Swarm Coordinates

Every task is assigned:

One task_id
One root span (task.process)
One connected trace throughout the execution

This architectural decision is what keeps the entire workflow connected.

Without it, every HTTP request between the Orchestrator, Planner, Executor, and Critic would generate independent traces, fragmenting the execution into unrelated pieces.

Each subtask runs inside its own nested span while the Executor and Critic iterate until convergence.

Whenever the Critic rejects a draft,

agentatc_handoffs_total{reason="reject"}

is incremented.

When a task stalls, nothing throws an exception.

Nothing crashes.

Instead,

agentatc_task_last_update_timestamp

simply stops moving forward, providing the only indication that no agent is making progress anymore.

A Successful Execution

Submit the goal:

"Write a competitive analysis of three observability vendors: SigNoz, Datadog, and New Relic."

The Planner creates three subtasks.

Two of them—

SigNoz
New Relic

—converge immediately and pass the Critic on the first review.

That's the expected path.

It should look boring.

Successful coordination shouldn't generate excitement.

When Things Go Wrong

The Datadog subtask is intentionally different.

Its mock search tool can never retrieve pricing information, while the Critic's evaluation rubric requires pricing to approve the draft.

As a result, the Executor can never satisfy the Critic.

Every execution guarantees:

Five review iterations
Five rejections
One escalation

No randomness.

No hoping an LLM behaves the same during a live demo.

Because everything shares one trace ID, opening the trace reveals the entire history as one connected execution, showing every Executor ↔ Critic hand-off across all services.

Complete Transparency with SigNoz Dashboards and Copilot

The Swarm Health Overview dashboard tracks:

Convergence rate
Average approval iterations
In-flight subtasks
Redundant tool calls

The Thrash Leaderboard ranks subtasks by rejection count.

Datadog consistently appears at the top, making the coordination failure immediately visible.

Two production alerts help detect coordination failures early:

Task Thrashing
Task Stalled

Both alerts can automatically trigger the Copilot.

The Copilot uses SigNoz's MCP Server to investigate another agent's failure the way an on-call engineer would.

It retrieves telemetry using:

signoz_search_traces
signoz_get_trace_details
signoz_search_logs

Most importantly, it is built to say "No data found" instead of fabricating an answer.

During one escalated execution it concluded:

"The Critic consistently rejected the submissions for the same reason: missing pricing information... this indicates a TOOLING GAP. The executor's data sources do not provide the required pricing information, leading to consistent rejections by the critic."

That distinction is the entire value proposition.

Not simply that a task failed five times.

But why it failed, backed by actual telemetry.

Why SigNoz Matters

We didn't bolt dashboards onto the project after the fact.

Traces, metrics, and logs all flow through SigNoz for real analytical work.

Coordination metrics are computed directly from raw telemetry in SigNoz's query layer instead of being pre-calculated inside application code.

While migrating dashboards to the Query Builder, we discovered that one PromQL query grouped results using a label that didn't actually exist.

The dashboard rendered successfully.

The data was wrong.

That wasn't a SigNoz issue.

It was an instrumentation bug in our own project that only became visible because we validated the dashboards against live telemetry instead of assuming a rendered graph must be correct.

Technology Stack

Backend: FastAPI microservices (Orchestrator, Planner, Executor, Critic, Copilot)
Frontend: React + Vite
Observability: OpenTelemetry auto-instrumentation
Transport: OTLP over gRPC
Monitoring: Self-hosted SigNoz via Foundry
Investigation: SigNoz MCP Server

Lessons Learned

The failure mode we set out to expose includes:

Thrashing
Stalled tasks
Redundant work

These failures produce almost no useful signal in traditional monitoring systems.

Building the metrics to observe them forced us to think about coordination as telemetry, rather than treating it as a side effect of healthy service calls.

Going back to validate our dashboards against live data instead of trusting that they rendered correctly also uncovered a genuine instrumentation bug that would otherwise have shipped unnoticed.

Conclusion

We didn't build another AI agent demo.

We built the instrument panel for the failure mode every multi-agent team is about to discover the hard way.

The one where:

Everything looks healthy.
Nothing ever fails.
And the system is stuck anyway.

DEV Community: I Jithindranaath