DEV Community: Rahul Chandra Padamuttam

OmniSRE: Closed-Loop AI Observability and Autonomous Self-Healing

Rahul Chandra Padamuttam — Sat, 25 Jul 2026 07:15:27 +0000

How I Built an Autonomous AI SRE Agent with SigNoz (and the Telegram Kill Switch That Keeps It Safe)

AI & Agent Observability | WeMakeDevs x Agents of SigNoz Hackathon

It was late at night, and I was watching my terminal with a mixture of excitement and genuine dread.

I had just injected simulated database chaos into my own service and handed an AI agent full, unchecked access to my Docker socket. The agent had 2 minutes to read the SigNoz traces, call Google Gemini, diagnose the root cause, and send an approval request to my Telegram before I would call it a failure.

No human was going to fix this. Either the agent worked, or the container stayed broken.

That's when I understood the real problem with modern observability. SigNoz was already showing me everything: P99 latency spiking to 3,420ms, HTTP 500 errors flooding in above 40%. The dashboard was perfect. But a dashboard can only tell you what is on fire. It cannot put the fire out.

If Large Language Models are smart enough to write code, shouldn't they be smart enough to read telemetry, find the root cause, and restart the container themselves?

I built OmniSRE to answer exactly that question. But the moment I gave my AI agent the keys to my Docker socket, I realized that an autonomous agent with unchecked infrastructure access is genuinely terrifying. One bad LLM response, one stale approval message—and it starts restarting production containers on its own.

I needed a kill switch. So I built one into Telegram.

In this post, I'll walk you through the exact code — from deploying a self-hosted SigNoz stack, to wiring OpenTelemetry traces into a Gemini-powered root-cause engine, to the timestamp-guarded Telegram gate that keeps the whole system from going rogue.

(Alternatively, Click Here to Watch the 3-Minute Live Demo)

The Problem in One Sentence

Modern observability tells you what broke. OmniSRE tells the AI why it broke, then fixes it—with your approval.

Phase	Traditional On-Call	OmniSRE
Detection	SigNoz Alert fires	SigNoz Alert fires
Diagnosis	Engineer wakes up, reads logs	Gemini reads ClickHouse logs autonomously
Authorization	Engineer decides	Telegram HITL gate — YOU decide
Remediation	SSH + manual restart	Docker socket heal in < 40 seconds
MTTR	15–45 minutes	Under 40 seconds

How I Deployed the Observability Stack

I wanted a completely reproducible observability environment. Instead of manually wiring up ClickHouse and the OpenTelemetry Collector, I used SigNoz Foundry CLI to spin up the entire self-hosted stack in under 2 minutes.

Why self-hosted SigNoz over a managed cloud tool? Two reasons: full ClickHouse query access (critical for the agent to pull raw logs programmatically), and zero data egress costs. The agent's entire log-extraction pipeline depends on direct /api/v1/query_range access—which you only get with self-hosted.

First, define the deployment manifest:

# casting.yaml - SigNoz Foundry Deployment Specification
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose
  mcp:
    spec:
      enabled: true

Then deploy the entire stack with a single command:

# Install Foundry and cast the infrastructure stack
curl -fsSL https://signoz.io/foundry.sh | bash
export PATH="$HOME/.local/bin:$PATH"

foundryctl cast -f casting.yaml

Within moments, six isolated core containers are live:

Port 3301 — SigNoz UI & Query Engine
Port 4317/4318 — OTLP gRPC/HTTP ingestion endpoints

The Architecture: Every Container Has One Job

OmniSRE is designed with a strict multi-container isolation model. Telemetry flows upstream into SigNoz. Remediation commands flow downstream via host-level socket bindings. The SRE Agent is completely decoupled from the application it monitors.

Key Design Principle: If buggy_service crashes and burns, omnisre_agent stays alive to fix it. They share zero runtime dependencies.

Here is the critical docker-compose.yml section showing the socket mount:

# config/docker-compose.yml
version: '3.8'

services:
  buggy_service:
    build:
      context: ..
      dockerfile: docker/Dockerfile.buggy
    container_name: buggy_service
    ports:
      - "8000:8000"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318
      - DB_POOL_SIZE=10

  omnisre_agent:
    build:
      context: ..
      dockerfile: docker/Dockerfile.agent
    container_name: omnisre_agent
    ports:
      - "8001:8001"
    volumes:
      # This single line gives the agent surgical control over host containers
      - /var/run/docker.sock:/var/run/docker.sock
      - ../.env:/app/.env
    environment:
      - SIGNOZ_ENDPOINT=http://host.docker.internal:3301
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
      - GEMINI_API_KEY=${GEMINI_API_KEY}

Setting the Trap: Auto-Instrumentation & Chaos Injection

To give the AI real, structured data to reason about, I instrumented the /checkout endpoint with OpenTelemetry SDKs and wired up explicit error span tagging. When chaos fires, the spans going into ClickHouse are not generic—they carry precise failure context.

# app/buggy_service.py
from fastapi import FastAPI, HTTPException
from opentelemetry import trace
from opentelemetry.trace import StatusCode

app = FastAPI(title="Buggy E-Commerce Checkout")
chaos_mode_enabled = False

@app.get("/checkout")
def checkout():
    global chaos_mode_enabled
    if chaos_mode_enabled:
        current_span = trace.get_current_span()
        if current_span.is_recording():
            current_span.set_status(StatusCode.ERROR, "Chaos mode active: Simulated 500 Outage")
            current_span.set_attribute("http.status_code", 500)

        raise HTTPException(status_code=500, detail="500 Internal Server Error: Connection pool exhausted.")
    return {"status": "success", "latency_ms": 25}

@app.post("/chaos/inject")
def inject_chaos():
    global chaos_mode_enabled
    chaos_mode_enabled = True
    return {"status": "chaos_injected", "message": "Simulated DB pool failure active."}

One curl -X POST http://localhost:8000/chaos/inject later, the SigNoz Dashboard lights up:

P99 latency spikes to 3,420 ms
HTTP 500 error rate climbs above 40%
SigNoz Alert Manager fires a webhook to omnisre_agent:8001

Every chaos-mode /checkout request is fully indexed as a named error span in ClickHouse, giving the AI structured, auditable evidence to reason against.

Wiring Up SigNoz: The Central Nervous System

This is where SigNoz sits at the core of OmniSRE. After traces start flowing in, I configured four things inside the SigNoz UI:

Metrics Panels (Query Builder): Four custom panels on a single dashboard track the incident lifecycle in real time — a P99 latency time-series, an HTTP 500 rate panel using signoz_calls_total filtered by status_code=500, a pie chart showing the live 200/500 traffic health ratio, and a recovery curve showing 200 OK requests restoring after healing.

Alert Rule: I created a threshold alert on P99 latency in the SigNoz Alerts UI. When p99 > 2000ms for 2 consecutive minutes, SigNoz fires:

Alert Name: High Checkout Latency
Condition:  p99_duration_ms > 2000
For:        2 minutes
Labels:     severity=critical, service=buggy_service

Webhook Channel: In SigNoz → Settings → Alert Channels, I configured a Webhook notification channel pointing to http://host.docker.internal:8001/webhook/signoz. The moment the alert fires, SigNoz POSTs a JSON payload directly to the SRE agent's receiver endpoint — no polling, no cron jobs.

This is the key integration point: SigNoz is not just a dashboard in this system. It is the event source that kicks off the entire autonomous remediation pipeline. Without the alert channel, nothing moves.

The Debugging Nightmare Nobody Talks About

This was the hardest part of the entire build, and I guarantee you will hit this wall too.

Everything worked perfectly in my local Python environment. But the moment I containerized the agent, the observability pipeline went blind. The agent received the webhook, then crashed trying to query ClickHouse logs.

The error log: Connection Refused: localhost:3301

The Insight: localhost inside a Docker container is the container itself—not your host machine. SigNoz was running fine on host:3301, but the agent was knocking on its own door.

To fix this resiliently across any deployment environment, I built an automatic fallback engine:

# agent/investigator.py
import requests

class SigNozInvestigator:
    # Ordered priority matrix for cross-container bridge resolution
    FALLBACK_ENDPOINTS = [
        "http://localhost:3301",
        "http://host.docker.internal:3301",
        "http://signoz-frontend:3301"
    ]

    def fetch_real_signoz_logs(self, query_payload: dict) -> dict:
        for base_url in self.FALLBACK_ENDPOINTS:
            try:
                url = f"{base_url}/api/v1/query_range"
                response = requests.post(url, json=query_payload, timeout=3)
                if response.status_code == 200:
                    return response.json()
            except requests.exceptions.ConnectionError:
                continue
        raise ConnectionError("Failed to reach SigNoz ClickHouse backend across all bridges.")

The agent now auto-resolves the correct network bridge every single time. No manual configuration required.

SigNoz features used in this project: Distributed Tracing (Trace Explorer), ClickHouse Log Querying (/api/v1/query_range), Custom Metrics Dashboards (Query Builder), Threshold Alert Rules, and Webhook Alert Channels — five features running as one connected pipeline.

The Gemini Prompt: How the AI Actually Thinks

Here is the exact prompt that is sent to gemini-1.5-flash via LiteLLM during a live incident. This is the core "brain" of the entire system:

# agent/investigator.py — the exact prompt sent to Gemini
prompt = f"""
You are an autonomous Site Reliability Engineer (OmniSRE).
An alert has been triggered: {alert_payload}

Context gathered from observability tools:
{context}  # <-- Real ClickHouse log data injected here

Determine the root cause and the required action.
Respond with ONLY raw JSON, with no markdown formatting or backticks.
It must contain:
- "root_cause": string, your hypothesis
- "action": string, either "RESTART_SERVICE", "SCALE_UP", or "NO_ACTION"
- "target_service": string, the name of the service to act on (e.g., "buggy_service")
"""

And here is a real example of the structured JSON response Gemini returns:

{
  "root_cause": "The /checkout endpoint is experiencing connection pool exhaustion. ClickHouse logs show a sustained spike of StatusCode.ERROR spans over the last 15 minutes, consistent with DB_POOL_SIZE being undersized for current traffic volume.",
  "action": "RESTART_SERVICE",
  "target_service": "buggy_service"
}

Why this prompt works: Forcing a strict JSON schema eliminates ambiguous, prose-based responses entirely. The agent's downstream healer.py can then deterministically parse action and target_service without any fragile string matching.

The Telegram Kill Switch

Having an AI that can diagnose a problem is amazing. Having an AI that can unilaterally modify .env files and execute container restarts is a recipe for a career-ending incident.

I needed a "Financial Kill Switch" for my infrastructure. So I built one into Telegram.

When SigNoz fires the alert, the pipeline executes like this:

SigNoz Alert → Webhook → Gemini RCA → Telegram HITL Gate → (YES) → Docker Heal
                                                          ↓
                                                     (NO / Timeout)
                                                          ↓
                                                  Abort. Log. Done.

The agent halts completely at the Telegram gate and sends a formatted diagnostic card showing exactly what it found and exactly what it plans to do. You reply YES or you walk away.

To prevent the agent from reading stale YES messages from previous incidents—a real bug I hit during testing that caused phantom auto-restarts—I implemented timestamp-filtered polling:

# agent/notifier.py
import time, requests

def wait_for_human_approval(bot_token: str, chat_id: str, alert_start_time: float) -> bool:
    """Polls Telegram updates and strictly filters against alert trigger timestamp."""
    timeout = 120
    start_poll = time.time()

    while time.time() - start_poll < timeout:
        updates = requests.get(f"https://api.telegram.org/bot{bot_token}/getUpdates").json()
        for result in updates.get("result", []):
            msg = result.get("message", {})
            msg_date = msg.get("date", 0)
            text = msg.get("text", "").strip().upper()

            # Only process messages sent AFTER the incident started
            if msg_date >= (alert_start_time - 5) and text == "YES":
                return True
        time.sleep(3)
    return False

Once authorized, healer.py programmatically mutates the .env config and restarts the container via the Docker socket:

# agent/healer.py
import docker

def apply_remediation_and_restart(service_name: str, key: str, value: str):
    # Step 1: Mutate the configuration parameter
    with open(".env", "r") as f:
        lines = f.readlines()
    with open(".env", "w") as f:
        for line in lines:
            f.write(f"{key}={value}\n" if line.startswith(f"{key}=") else line)

    # Step 2: Execute the Docker restart via host socket binding
    client = docker.from_env()
    container = client.containers.get(service_name)
    container.restart()
    return True

The Results: Before vs. After

Metric	During Chaos	After Self-Healing
P99 Latency	3,420 ms	840 ms
HTTP 500 Rate	> 40%	0%
HTTP 200 Rate	< 60%	100%
MTTR	N/A (no human awake)	< 40 seconds

3 Things I Learned That Nobody Tells You

The biggest takeaway from this build: context quality is everything. By explicitly tagging OpenTelemetry spans with StatusCode.ERROR and http.status_code, I gave Gemini real, structured facts instead of ambiguous log strings. The quality of the root-cause analysis was night and day compared to my first attempt with raw log text.

The second thing genuinely surprised me: always timestamp your HITL guardrails. During testing, the agent was auto-restarting containers the instant it received a new alert — it turned out it was finding a YES I'd sent 10 minutes earlier for a completely different incident. A 5-second timestamp offset filter fixed it. Obvious in hindsight, two hours wasted finding it.

The last one I'll carry into every future project: observability is not optional for autonomous agents, it's the foundation they run on. Without SigNoz bridging raw OTLP traces into queryable ClickHouse, the agent would have nothing factual to feed Gemini. The AI is only as trustworthy as the data underneath it.

If I rebuilt this today, I'd replace the Telegram polling loop with SigNoz's own webhook payload as the authorization token — a cryptographically verifiable approval chain instead of plain-text YES matching. I'd also expose a /status endpoint on the agent so SigNoz dashboards can show the current remediation state in real time, closing the feedback loop entirely inside the observability platform.

Try It Yourself in 5 Minutes

Clone the repo: git clone https://github.com/rahulchandra2004/omnisre-signoz-hackathon.git
Create .env with your Gemini API key and Telegram Bot Token
Run foundryctl cast -f casting.yaml to spin up SigNoz
Run docker-compose -f config/docker-compose.yml up --build -d
Hit curl -X POST http://localhost:8000/chaos/inject
Watch your Telegram light up with a live incident diagnosis

What's Next

The next version of OmniSRE would swap Docker socket bindings for Kubernetes CRDs and kubectl scale deployments, making it cluster-native and production-grade. I also want to split the single agent into a network of specialized LLMs — one for querying, one for diagnosis, one for remediation — rather than one Gemini call doing all three jobs. The feature I most wish I'd shipped: an automated rollback engine that watches post-healing metrics and reverts the .env mutation if recovery stalls.

AI Usage Disclosure

In compliance with hackathon transparency guidelines, Gemini and GitHub Copilot were used during development for code refactoring and documentation layout. At runtime, gemini-1.5-flash is natively integrated via LiteLLM for log triage, hypothesis generation, and root-cause analysis during live incidents.

Check out the full source code, clone it, and try injecting chaos yourself: OmniSRE GitHub Repository

Have you ever built an autonomous remediation system? How do you handle the Human-in-the-Loop problem at scale? Let me know in the comments—I'd love to hear your approach.

Built by Rahul Chandra Padamuttam for the Agents of SigNoz Hackathon.

Visualizing Deep Trace Hierarchies: Self-Hosting SigNoz via Foundry on Windows

Rahul Chandra Padamuttam — Mon, 13 Jul 2026 12:19:17 +0000

If you’ve ever had a night of going through a disorganized pile of flat text logs which had you searching for the reason a single API endpoint performance slowed to a crawl, you know the issue. Distributed systems require distributed tracing but often by the time you get that cloud-based observability platform up and running you end up dealing with issues of setup friction or unexpected costs. In this piece I’ll be talking about how I was able to get around the cloud completely and self-host a full local SigNoz backend on Windows with Foundry which in turn allowed me to trace out complex microservice call stacks with microsecond precision at no cost.

Why Local Observability Matters

While working on modern applications, it's becoming increasingly important to not only know which calls are failing but to identify the sources of latency in the whole call graph. Terminal logs are nice and all, but the moment you have to deal with something more involved than a simple syntax error you're on your own.

I set up this zero-cost sandbox to be able to generate some interesting structured telemetry and to be able to play around with OTel ingestion pipelines without incurring costs and without having to faff around with a staging environment. By running the 12-container SigNoz backend that I deployed via Foundry, I've turned my Windows machine into a self-contained observability playground.

This is how I got it to work, what gotchas I had to deal with, and what the data looks like behind the scenes.

Setting Up the Pipeline & Showing the Work

Setting up a distributed observability platform involves a number of networking gotchas in a local environment. If you launch a load-generator (such as the one provided by the OpenTelemetry project as telemetrygen) in a detached container, it needs to learn how to route the traffic to the SigNoz OTLP receiver running on the host machine.

Here’s how I fixed that on Windows using good old Docker’s --add-host flag to bind the host’s internal IP address in the container’s network namespace. Here’s a PowerShell command I’ve used to start generating some synthetic telemetry and forcing a complex 15-level deep execution tree:

docker run --rm --add-host host.docker.internal:host-gateway ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest traces --otlp-endpoint host.docker.internal:4317 --otlp-insecure --rate 5 --duration 300s --child-spans 15

Once the signal generator started sending messages over port 4317, the local ClickHouse started indexing information in real-time. Here is what the whole observability stack looked like while working with the actual request flow:

Step 1: Working with the Traces Explorer and Waterfall View
Upon launching the application, the initial window that appeared to me was the Traces tab. First of all, I was struck by how easy it is to get an overall idea of all distributed transactions using the timeline view. It is possible to sort transactions using the service names or particular operations:

Opening any one of the transactions shows the power of distributed tracing. Instead of seeing one simple line of the log, the custom --child-spans 15 parameter actually prompted OpenTelemetry to generate a nicely nested tree of execution spans.

This perspective allows you to identify performance issues at a glance. The parent span is waiting for a cascading staircase of child microservices. On the right-side, attributes pane, you can see the critical OpenTelemetry semantic conventions discovered by SigNoz - including the trace kind Server, status code Unset, and the network routing endpoint:

{
  "attributes": {
    "network.peer.address": "1.2.3.4",
    "service.peer.name": "telemetrygen-client"
  },
  "duration_nano": 123000,
  "kind_string": "Server",
  "name": "okey-dokey-9"
}

Step 2: Tracking Application Metrics & Logs
In order to observe the interaction between the components, it is essential to correlate multiple telemetry sources. To see the connection between this stream of information and the current state of the system, I proceeded to the APM dashboard:

The graphs were simple to read, clearly showing our load metrics. By looking at the graphs, I could see that our application worked stably at a certain level, handling 1.75 ops/s with low P99 latency that rarely peaked above 1.85 ms.

I proceeded to the Logs Explorer tab to see the actual requests. The generator sends rather simple-looking messages to the backend in the format of JSON arrays:

{
  "timestamp": "2026-07-12 19:10:02.250",
  "body": "the message",
  "attributes": {
    "app": "server",
    "service.name": "telemetrygen"
  }
}

The Log Details interface parsed these JSON attributes accurately, indexing the custom app: server and transaction tags dynamically right alongside the string message:

Step 3: Building Dashboards and Alerts Out of the Box

To conclude the exploration, I attempted to create a custom Dashboard with a specific query from scratch. The native Dashboard builder was employed, and the UI Query Builder allowed me to track the chosen internal metric over the 30-minute period:

Panel Type: Time Series
Metric Filter Selection: gen
Aggregation: Avg within time series (Every Auto Seconds)
Cross-Series Aggregation: Avg by Everything (no breakdown)

Next, I wanted to turn this monitoring graph into an active safety net by configuring a custom alert rule:

Using the Query Builder interface, create a condition for which you want to be alerted. For example, “Alert me if the error rate is more than 5%”. Then, by analyzing data stream A, set a value of critical threshold near zero. Thus, in case our containerized environment encounters any errors or network issues, the defined rule will react to it, instead of forcing our developers to watch the dashboard in real-time.

What I Learned & Key Takeaways

The process of installing and configuring the local observability pipeline was educational for me, as I've got several crucial takeaways from the activity:

The telemetry data is decoupled: one of the first insights I had while working on the project was that sending traces does not automatically make them available as logs and application metrics. These are three separate data streams handled by OpenTelemetry, and they should be correlated in the unified interface for proper analysis (in this case, the ClickHouse database).
Visualization is the ultimate productivity booster: having a visual representation of my request processing flow in the form of the layered waterfall makes analyzing my application's behavior significantly more intuitive than poring over the raw text logs in the terminal.
The local development loop is critical: having an observability setup that I can experiment with at will without fear of impacting production systems and wasting precious cloud resources on unneeded expensive instances is invaluable. With this approach, I can stress-test my own application to find what metrics thresholds are needed to trigger alerts.

Conclusion

By building this local observability sandbox I have gained valuable insight into the workings of a full-stack observability system. Now I am looking forward to applying my understanding of OpenTelemetry to my hackathon projects so that all components are observable, understandable, and high-performing.

You can view my complete deployment manifests, including the environment-specific casting.yaml files, in my public GitHub repository.