DEV Community

Uzair
Uzair

Posted on

Observability for Multi-Agent Workloads: Cost Tracking & Fault Tolerance with SigNoz

#webdev #javascript #programming #productivity

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

As AI agents transition from simple single-prompt scripts into complex multi-step systems, observability becomes mission-critical. In production architectures, an unmonitored agent chaining successive LLM iterations, managing volatile context state, and invoking discrete microservices can silently burn thousands of dollars in API token consumption or crash under rate limits.

This technical guide details how we built and deployed the Agent Ops Command Center for Track 1 (AI & Agent Observability). Using Node.js, Express running on port 3000, OpenTelemetry, and SigNoz, we created a real-time control panel to track latency, token usage, sub-cent financial costs, and automated fault recovery.

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

Rather than manually configuring separate analytics layers, the architecture relies on a declarative deployment blueprint executed through Foundry. This wraps the underlying telemetry engine alongside a native Model Context Protocol (MCP) server instance to ensure the entire environment remains reproducible for verification.

Below is the structured installation configuration saved as casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: production-agent-sre
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    enabled: true
Enter fullscreen mode Exit fullscreen mode

To deploy the ecosystem, trigger the deployment compiler:

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

🧱 Real-World Gotcha: Port 3000 Ingestion & Proxy Handshakes

During initial deployment, the Express server spins up on port 3000 (process.env.PORT || 3000). When verifying local connectivity across virtualized cloud environments, testing the backend directly on port 3000 ensures the health and telemetry probes are operational:

curl -v http://localhost:3000/api/health
Enter fullscreen mode Exit fullscreen mode

The test suite returns a clean HTTP/1.1 200 OK along with runtime telemetry snapshot status. When routing through web previews, mapping the host variable explicitly grants direct access to the command center:

echo "https://3000-$WEB_HOST"
Enter fullscreen mode Exit fullscreen mode

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

To extract execution data cleanly without vendor lock-in, our Express backend (server.js) integrates directly with official OpenTelemetry GenAI Semantic Conventions (gen_ai.*). Every agent invocation is exported directly to SigNoz via OTLP/HTTP.

Below is the OpenTelemetry initialization and instrumentation setup used in Node.js:

const express = require('express');
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { BasicTracerProvider, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const app = express();
const port = process.env.PORT || 3000;
const serviceName = process.env.OTEL_SERVICE_NAME || 'agent-ops-command-center';
const otlpEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318';

// Configure OTLP Exporter targeting SigNoz
const exporter = new OTLPTraceExporter({
  url: `${otlpEndpoint.replace(/\/$/, '')}/v1/traces`
});

const provider = new BasicTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    'service.version': '1.0.0',
    'deployment.environment': process.env.NODE_ENV || 'development'
  })
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

const tracer = trace.getTracer(serviceName);
Enter fullscreen mode Exit fullscreen mode

🔑 Key Features & Telemetry Design

1. Granular Token & Cost Attribution

Every agent execution creates an agent.run span. We attach custom Semantic Conventions to track exact token consumption and sub-cent financial expenditure across providers (OpenAI, Gemini):

span.setAttribute('agent.name', agentName);
span.setAttribute('task.type', taskType);
span.setAttribute('gen_ai.system', providerName);
span.setAttribute('gen_ai.request.model', modelName);
span.setAttribute('gen_ai.usage.input_tokens', inputTokens);
span.setAttribute('gen_ai.usage.output_tokens', outputTokens);
span.setAttribute('gen_ai.usage.total_tokens', totalTokens);
span.setAttribute('agent.cost.usd', costUsd);
Enter fullscreen mode Exit fullscreen mode

2. Rate-Limit Resilience & Exponential Backoff

When API provider rate limits or quotas are hit (HTTP 429), the backend applies an exponential backoff mechanism. If the quota remains exhausted, the system automatically records a warning span in SigNoz and gracefully switches to a rate-limit fallback response instead of crashing:

// Catch rate limit errors and emit fallback trace to SigNoz
if (error.message.includes('Quota exceeded') || error.message.includes('429')) {
  span.setAttribute('gen_ai.request.model', `${modelName}-fallback`);
  span.setAttribute('gen_ai.usage.total_tokens', 150);
  span.setAttribute('agent.cost.usd', '0.000000');
  span.setStatus({ code: SpanStatusCode.OK });

  recordEvent('WARN', `Rate limit reached for ${providerName}. Used fallback.`, 'Agent run');
  span.end();

  return res.json({ run: fallbackRunData, live: true });
}
Enter fullscreen mode Exit fullscreen mode

📊 Telemetry Design: Custom ClickHouse SRE Queries

Using the ClickHouse datastore underneath SigNoz, we can query LLM telemetry directly through the SigNoz Query Builder to monitor live system metrics.

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

Calculates financial accumulation across distinct model interactions:

SELECT
    attributes_string['gen_ai.request.model'] as Model,
    SUM(attributes_int['gen_ai.usage.input_tokens']) as TotalInputTokens,
    SUM(attributes_int['gen_ai.usage.output_tokens']) as TotalOutputTokens,
    SUM(toFloat64(attributes_string['agent.cost.usd'])) as TotalSpendUSD
FROM signoz_traces.signoz_index_v2
WHERE name = 'agent.run' AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY Model
ORDER BY TotalSpendUSD DESC
Enter fullscreen mode Exit fullscreen mode

🔄 Panel 2: Agent High-Latency & Error Tracking

Identifies agent runs experiencing high latency or execution errors:

SELECT
    attributes_string['agent.name'] as AgentName,
    COUNT(*) as TotalRuns,
    AVG(durationNano) / 1000000 as AvgLatencyMs
FROM signoz_traces.signoz_index_v2
WHERE name = 'agent.run' AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY AgentName
HAVING AvgLatencyMs > 250
Enter fullscreen mode Exit fullscreen mode

🧠 Core Engineering Takeaways

  1. Port 3000 Standardization: Aligning application routing and container proxy variables on port 3000 ensures clean local evaluation and deployment health checks.
  2. Resilient Fallback Loops: Gracefully capturing rate limits inside OpenTelemetry spans prevents application downtime while preserving full trace visibility in SigNoz.
  3. Declarative Reproducibility: Deploying through Foundry's casting.yaml ensures the underlying SigNoz telemetry collector stays reproducible for hackathon verification.

Top comments (0)