Tanmay Hathile

Posted on Jul 16

Debugging the Ghost in the Machine: Building a Self-Healing SRE Agent with OpenTelemetry

#agents #ai #monitoring #sre

AI agents are awesome right up until they silently eat it in production, leaving you with zero logs, empty traces, and a cloud bill that makes you want to cry. Over the last 48 hours, I’ve been building an Autonomous Self-Healing SRE Agent Pipeline. The goal? Instrument it with OpenTelemetry so it doesn’t just fix its own mess, but reports the ROI of that fix in real-time.

Here’s the raw, unvarnished blueprint—complete with the architectural headaches that almost cost me my entire weekend.

The Setup

We wanted something practical: an agent that catches its own crashes (like a choked cache or a dropped database connection) and tries to fix itself.

Our app is a FastAPI service with a /execute-task endpoint. To keep it real—and stressful—we threw in a hardcoded 15% failure rate to simulate resource exhaustion. When it crashes, our middleware catches the exception, spawns an async "self-healing" loop, and logs the outcome to our telemetry backend.

Network Topology

Before we got to the fun stuff, we had to get the plumbing right. If you’ve ever had services fight over the same local port, you know the drill.

Component	Port	Purpose
FastAPI Agent	`8081`	Our main pipeline.
SigNoz UI	`8080`	Visualization dashboard.
OTel Collector	`4317`	gRPC ingest.

Note: Everything defaults to 8080. It took me a solid 30 minutes of checking logs before I realized my app and the SigNoz UI were fighting for the same port. I eventually just moved the agent to 8081 and called it a day.

The Math: $9.94 per Heal

We wanted to track the value of our automation. We based our ROI on a $60/hr SRE cost, a 10-minute manual fix time, and a 3.6-second auto-repair. After factoring in some SLA risk mitigation, every successful heal nets us exactly $9.94.

Seeing that data finally hit the dashboard? It felt like winning the lottery.

The Implementation

import os
import random
import time
from fastapi import FastAPI, HTTPException
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider, Status, StatusCode
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Setup OTel Providers
resource = Resource.create(attributes={"service.name": "sre-agent-pipeline"})

# Tracer setup
trace_provider = TracerProvider(resource=resource)
otlp_trace_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))
trace.set_tracer_provider(trace_provider)
tracer = trace.get_tracer("sre.agent.tracer")

# Metrics setup
otlp_metric_exporter = OTLPMetricExporter(endpoint="http://localhost:4317", insecure=True)
reader = PeriodicExportingMetricReader(otlp_metric_exporter, export_interval_millis=5000)
metric_provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(metric_provider)

meter = metrics.get_meter("sre.agent.metrics")
heals_counter = meter.create_counter(name="pipeline_heals_total", unit="1")
money_saved_counter = meter.create_counter(name="pipeline_money_saved_total", unit="USD")

app = FastAPI()

@app.post("/execute-task")
async def execute_task():
    with tracer.start_as_current_span("ExecuteAgentTask") as span:
        # Simulate a crash
        if random.random() < 0.15:
            with tracer.start_as_current_span("SelfHealingLoop") as heal_span:
                # Trigger recovery...
                heals_counter.add(1, {"type": "cache_purge"})
                money_saved_counter.add(9.94, {"currency": "USD"})
                return {"status": "remediated"}
        return {"status": "success"}

The "I Spent Way Too Long on This" Section

1. The OTel SDK Deprecation Trap

I spent way too long wondering why my imports were failing, only to realize the status submodule was deleted in recent updates. Classic. If you're using opentelemetry-sdk >= 1.27.x, stop trying to import from opentelemetry.sdk.trace.status.

The fix:

# Change this:
from opentelemetry.sdk.trace.status import Status 

# To this:
from opentelemetry.sdk.trace import Status, StatusCode

2. The Port Collision

As I mentioned, don't let your FastAPI app try to claim port 8080. If you're on a workstation, set it explicitly:

$env:APP_PORT="8081"
python main.py

Final Thoughts

Building autonomous systems is wild, but if you can't see what they're doing, you're just adding a "black box" to your production stack. By forcing our agent to log its own recovery, we turned an invisible maintenance cycle into a tracked asset.

Reproducibility: Everything is locked down in casting.yaml and its lock file.
Transparency: Yes, I used AI to brainstorm some of the architectural patterns and clear up my dependency errors—hackathon life is hard enough without fighting syntax errors alone at 3 AM.

If you run into similar OTel issues, check the official OTel Python docs. Happy coding, and may your build never fail on a Sunday night.

DEV Community