DEV Community

Cover image for The Dashboard Said My AI Team Finished. One Agent Never Even Started.
bilal-feroz
bilal-feroz

Posted on

The Dashboard Said My AI Team Finished. One Agent Never Even Started.

#opentelemetry #signoz #observability #ai

How SigNoz exposed a silent handoff failure inside a multi-agent AI workflow

My AI workflow reported that the task was complete.

The API returned 200 OK. The user interface showed every agent as completed. A final document was generated and delivered successfully.

There was only one problem.

The research agent had never returned any research.

The workflow silently caught a timeout, passed an empty response to the writing agent, and continued as though nothing had happened. The final output looked confident and polished, but it was based on incomplete context.

This was not a normal application crash. There was no obvious error page, failed request, or red status indicator.

From the outside, everything looked healthy.

SigNoz helped me see what had actually happened inside the workflow.

The Multi-Agent Workflow

The application uses several AI agents to complete one project:

  1. Lead Agent — understands the request and creates a plan
  2. Research Agent — collects relevant context and information
  3. Writing Agent — produces the first draft
  4. Review Agent — checks quality and accuracy
  5. Delivery Agent — prepares and exports the final result

The expected workflow is:

User Request
    ↓
Lead Agent
    ↓
Research Agent
    ↓
Writing Agent
    ↓
Review Agent
    ↓
Delivery Agent
Enter fullscreen mode Exit fullscreen mode

Each agent depends on the result of the previous agent.

If the research agent fails, the writing agent should not continue as though valid research was available. However, my initial implementation used fallback logic that allowed the workflow to continue.

try {
  researchContext = await runResearchAgent(task);
} catch (error) {
  console.error("Research agent failed", error);
  researchContext = [];
}
Enter fullscreen mode Exit fullscreen mode

This prevented the entire request from crashing, but it created a more dangerous problem.

The system returned a technically successful response even though an essential part of the workflow had failed.

Adding Observability with OpenTelemetry and SigNoz

I instrumented the workflow using OpenTelemetry and sent the telemetry data to SigNoz.

I created one parent span for the complete project run:

project.run
Enter fullscreen mode Exit fullscreen mode

Each agent execution became a child span:

lead.plan
researcher.search
writer.draft
reviewer.validate
delivery.export
Enter fullscreen mode Exit fullscreen mode

I also attached attributes that describe what happened during each step.

span.setAttributes({
  "agent.name": "researcher",
  "task.id": taskId,
  "handoff.from": "lead",
  "handoff.to": "researcher",
  "retry.count": retryCount,
  "documents.returned": documents.length,
  "fallback.used": fallbackUsed,
});
Enter fullscreen mode Exit fullscreen mode

For the writing and review agents, I tracked additional values:

span.setAttributes({
  "tokens.input": inputTokens,
  "tokens.output": outputTokens,
  "review.score": reviewScore,
  "output.approved": outputApproved,
});
Enter fullscreen mode Exit fullscreen mode

These attributes allowed me to investigate more than basic application uptime.

I could now see:

  • Which agent was running
  • How long each agent took
  • Whether a handoff succeeded
  • Whether fallback data was used
  • How many documents were returned
  • How many retries occurred
  • How many tokens each agent consumed
  • Whether the final output passed review

Creating the Failure

To test the workflow, I deliberately introduced a timeout into the research service.

The research agent was configured to stop waiting after [TIMEOUT VALUE] seconds.

const researchContext = await Promise.race([
  runResearchAgent(task),
  timeoutAfter([TIMEOUT VALUE]),
]);
Enter fullscreen mode Exit fullscreen mode

The research service did not respond within the allowed time.

The application caught the exception and continued with an empty research context.

The final API response still looked successful:

{
  "status": "completed",
  "httpStatus": 200,
  "documentGenerated": true
}
Enter fullscreen mode Exit fullscreen mode

However, the trace attributes told a different story:

workflow.status: completed
researcher.documents.returned: 0
researcher.retry.count: [REAL RETRY COUNT]
fallback.used: true
review.score: [REAL REVIEW SCORE]
Enter fullscreen mode Exit fullscreen mode

The workflow was technically completed but functionally degraded.

What the Trace Revealed

Inside SigNoz, the complete workflow appeared as one distributed trace.

The trace waterfall made the problem immediately visible.

project.run             [REAL TOTAL TIME]
├── lead.plan           [REAL TIME]
├── researcher.search   [REAL TIME] — timeout
├── writer.draft        [REAL TIME] — fallback context used
├── reviewer.validate   [REAL TIME] — low review score
└── delivery.export     [REAL TIME]
Enter fullscreen mode Exit fullscreen mode

The research span consumed most of the workflow time before timing out.

The writing agent then started with no research documents, but the span itself was still marked as successful because the fallback logic prevented an exception from propagating.

This showed me an important limitation in my original monitoring logic:

A successful span did not necessarily mean that the agent completed its intended responsibility.

The system needed to track semantic success, not only technical success.

Correlating the Trace with Logs

The trace showed me where the delay occurred. The correlated logs explained why the workflow continued.

The research span contained a log similar to:

Research provider timed out after [REAL TIMEOUT] seconds.
Continuing with empty research context.
Enter fullscreen mode Exit fullscreen mode

Another log from the writing agent showed:

Writing agent started with 0 research documents.
Fallback context enabled.
Enter fullscreen mode Exit fullscreen mode

Without trace and log correlation, these messages would have been difficult to connect.

A timeout log by itself does not explain which user request was affected. A completed API request does not explain that an upstream agent failed.

SigNoz connected the logs to the exact trace, task, and agent span.

Building a Useful Dashboard

I created a dashboard focused on AI-agent workflow health rather than only infrastructure health.

The dashboard included:

Agent Run Duration

This panel shows how long each agent takes to complete its work.

It makes unusually slow agents and timeout patterns easier to identify.

Failed or Degraded Handoffs

This tracks handoffs where:

handoff.status != "success"
Enter fullscreen mode Exit fullscreen mode

or:

fallback.used = true
Enter fullscreen mode Exit fullscreen mode

Token Usage by Agent

This shows whether one agent is consuming an unusually large portion of the workflow’s token budget.

Completed Runs Using Fallback Data

This was the most important panel.

A workflow should not be considered fully successful when it completed using missing or fallback context.

Creating an Alert for Silent Failures

Traditional alerts often focus on conditions such as:

HTTP status >= 500
Enter fullscreen mode Exit fullscreen mode

That would not detect this problem because the API returned 200 OK.

Instead, I created an alert for workflows that completed while using fallback data.

workflow.status = "completed"
AND fallback.used = true
Enter fullscreen mode Exit fullscreen mode

I named the alert:

Successful run with degraded agent handoff
Enter fullscreen mode Exit fullscreen mode

This alert captures a category of failure that normal uptime monitoring misses.

The infrastructure can be available, the API can return successfully, and the user can receive an output while the workflow is still logically incorrect.

Fixing the Workflow

The original logic treated missing research as an acceptable fallback.

if (researchFailed) {
  researchContext = [];
  continueWorkflow();
}
Enter fullscreen mode Exit fullscreen mode

I changed it so that an essential agent failure marks the complete workflow as degraded.

if (researchFailed || researchDocuments.length === 0) {
  span.setAttributes({
    "workflow.status": "degraded",
    "handoff.status": "failed",
    "fallback.used": false,
  });

  throw new Error(
    "Research agent failed. Drafting stopped to prevent unsupported output."
  );
}
Enter fullscreen mode Exit fullscreen mode

The revised behavior is now:

Research agent fails
        ↓
Workflow marked as degraded
        ↓
Writing agent does not start
        ↓
User receives a clear retry message
Enter fullscreen mode Exit fullscreen mode

I ran the same test again.

This time, SigNoz showed that the workflow stopped at the research span instead of producing an unsupported final document.

The application no longer confused technical completion with valid completion.

What I Learned

Before adding observability, I could see whether the application returned a response.

After adding SigNoz, I could see whether each agent actually fulfilled its responsibility.

That distinction matters in AI systems.

A normal application often fails through an exception, unavailable service, or unsuccessful HTTP request.

An AI workflow can fail more quietly:

  • An agent returns empty context
  • A retry produces duplicate work
  • A fallback hides an upstream failure
  • A reviewer approves an incomplete response
  • The workflow completes while violating its own assumptions

The most dangerous AI failures are not always crashes.

Sometimes every technical component appears healthy while the final result is still wrong.

SigNoz helped me observe the complete chain of decisions, handoffs, retries, logs, and outputs behind one AI-generated result.

The dashboard originally told me that my AI team had completed the task.

The trace showed me the truth: one of the most important agents had never completed its work.

Top comments (0)