Cracking the Black Box: How I Caught a Runaway AI Agent Using Self-Hosted SigNoz and OpenTelemetry

Architecture Deep Dive: Defeating the AI Agent 'Black Box' with OpenTelemetry and SigNoz

Autonomous AI agents are incredibly powerful engineering solutions—until they hit an unhandled data schema mutation and spin out into a recursive execution loop. In production architectures, an unmonitored agent chaining successive LLM iterations, managing volatile context state, and invoking discrete microservices can become a massive liability, silently burning thousands of dollars in API token consumption.

If your operational visibility ends at the perimeter of the network proxy, you are operating blindly. This analytical guide details how to build a production-hardened observability foundation by self-hosting SigNoz via Foundry, configuring OTLP pipelines using standard semantic conventions, and assembling a custom ClickHouse-backed telemetry dashboard to detect cost runaway and agent loops dynamically.

---

🛠️ Infrastructure Assembly: Declarative Deployment via Foundry

Rather than manually configuring separate analytics layers, the architecture relies on a declarative deployment blueprint executed through Foundry. This wraps the underlying telemetry engine alongside a native Model Context Protocol (MCP) server instance to ensure the entire environment remains reproducible for verification.

Below is the structured installation configuration saved as casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: production-agent-sre
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    enabled: true

To deploy the ecosystem, trigger the deployment compiler:

foundryctl cast -f casting.yaml

🧱 Real-World Gotcha: Mitigating Web Preview Proxy Handshakes

During initial initialization, the container cluster successfully spun up all primary components (ClickHouse cluster, OTLP ingestion layer, and frontend gateway). However, accessing the dashboard via standard virtualized port configurations resulted in standard proxy handshaking timeouts.

To isolate the failure point, low-level testing directly on the local routing layer was used:

curl -v http://localhost:8080

The test suite immediately returned a clean HTTP/1.1 200 OK followed by the raw UI document structure, indicating a completely functional application layer. The underlying network proxy environment variables were caching dead routing handles. The workaround requires mapping the host address explicitly by querying the cluster's environment variables:

echo "https://8080-$WEB_HOST"

Pasting the output token directly into a clean browser instance completely bypassed the virtual network constraints, granting access to the SigNoz control console.

---

📡 Pipeline Architecture: End-to-End Tracer Instrumentation

To extract execution data cleanly without vendor lock-in, the agent framework integrates directly with official OpenTelemetry GenAI Semantic Conventions (gen_ai.*). This registers every microservice step—from LLM generation metrics to discrete utility routing—as structured OTLP data payloads.

Below is the initialization code for a Python-based multi-agent execution thread:

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Bind TracerProvider to the local SigNoz gRPC ingestion socket
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
telemetry_tracer = trace.get_tracer("agent-core-apm")

def execute_agent_pipeline(user_input, tracking_session_id):
    # Enclose the parent context inside a distinct trace scope
    with telemetry_tracer.start_as_current_span("orchestrate_agent_workflow") as master_span:
        master_span.set_attribute("gen_ai.workflow.name", "Enterprise-Data-Orchestrator")
        master_span.set_attribute("gen_ai.conversation.id", tracking_session_id)

        # Track individual LLM interaction sequences
        with telemetry_tracer.start_as_current_span("llm_chat_transaction") as inference_span:
            inference_span.set_attribute("gen_ai.system", "Anthropic")
            inference_span.set_attribute("gen_ai.request.model", "claude-3-5-sonnet")

            # Record downstream tools and structural database interactions
            with telemetry_tracer.start_as_current_span("dispatch_utility_tool") as service_span:
                service_span.set_attribute("gen_ai.tool.name", "query_analytical_db")
                # Emulate an active runtime schema violation to verify pipeline response
                service_span.set_status(trace.StatusCode.ERROR, "Unexpected data structure returned from storage layer")

---

📊 Telemetry Design: Custom ClickHouse SRE Query Dashboards

Standard system analytics checking generic memory metrics and HTTP statuses fail to surface internal logic errors within AI pipelines. By utilizing the raw performance of the ClickHouse datastore underneath SigNoz, we can engineer custom dashboards to track execution logic directly through the SigNoz Query Builder.

📈 Panel 1: Real-Time Token Spend & Cost Tracking Metrics

Calculates financial accumulation across distinct customer conversations to surface anomalous cost metrics:

SELECT
    attributes_string['gen_ai.conversation.id'] as TargetSession,
    SUM(attributes_int['gen_ai.usage.prompt_tokens']) as AggregatedInputTokens,
    SUM(attributes_int['gen_ai.usage.completion_tokens']) as AggregatedOutputTokens,
    (SUM(attributes_int['gen_ai.usage.prompt_tokens']) * 0.000003) + (SUM(attributes_int['gen_ai.usage.completion_tokens']) * 0.000015) as FinancialExposureUSD
FROM signoz_traces.signoz_index_v2
WHERE name = 'llm_chat_transaction' AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY TargetSession
ORDER BY FinancialExposureUSD DESC

🔄 Panel 2: Agent Recursion Loops & Storm Counter

Identifies broken logic configurations by flagging single transactions that issue more than ten nested tools inside a short temporal window:

SELECT
    attributes_string['gen_ai.conversation.id'] as TargetSession,
    COUNT(*) as DetectedToolExecutions
FROM signoz_traces.signoz_index_v2
WHERE name = 'dispatch_utility_tool' AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY TargetSession
HAVING DetectedToolExecutions > 10

🔍 Panel 3: Runtime Error Analysis Matrix

Categorizes and aggregates explicit platform exceptions to identify if issues stem from third-party vendor slowdowns or local code exceptions:

SELECT
    attributes_string['error.type'] as ExceptionType,
    COUNT(*) as AggregateIncidents
FROM signoz_traces.signoz_index_v2
WHERE response_status_code = 'STATUS_CODE_ERROR'
GROUP BY ExceptionType
ORDER BY AggregateIncidents DESC

---

🧠 Core Engineering Takeaways

1. Context State Expands Multiplicatively: As multi-step reasoning models operate, prompt structures grow larger. OpenTelemetry tracing exposes this inflation clearly before total context limit saturation occurs.
2. Infrastructure-as-Code Prevents Divergence: Deploying via Foundry's casting.yaml ensures analytical pipelines remain completely consistent across isolated environments.
3. Semantic Standardization Eliminates Rewrite Overhead: Utilizing uniform standard semantic attributes ensures your tracking metrics adapt perfectly if you transition from Python routines to production Node.js frameworks.

By moving away from unstructured console outputs and anchoring our tracking to OpenTelemetry standards with SigNoz, we convert unpredictable AI processes into fully transparent, debuggable software architectures.