Muhammad Lutfi Muzaki

Posted on Aug 8

Stop Chasing Symptoms: How We Built an Autonomous Root Cause Analysis Engine in Rust 🦀

#rust #devops #observability #ai

It’s 2:15 AM. Your phone buzzes aggressively. 🚨

You jump out of bed, open your laptop with half-closed eyes, and join an emergency incident response call. Your team’s Slack channel is exploding:

⚠️ [ALERT] Payment API 500 Error Rate > 15%
⚠️ [ALERT] Redis Latency Timeout (>5000ms)
⚠️ [ALERT] Node-04 CPU Saturation (98%)

You spend the next 2 hours manually connecting the dots: querying Prometheus metrics, scrolling through endless Loki logs, cross-referencing Tempo traces, and checking recent ArgoCD deployments.

Eventually, you uncover the truth: Deployment #218, pushed right before midnight, introduced a subtle memory leak that triggered GC pressure, spiked CPU, starved the Redis connection pool, and knocked down the Payment API.

Sounds familiar? 😅

💥 The Problem: Observability Shows Symptoms, Not Causes

Modern observability tools like Grafana, Prometheus, Loki, and Jaeger are fantastic at collecting metrics, logs, and traces. But they suffer from one fundamental design limitation:

They tell you WHAT is breaking, but leave you to figure out WHY it broke.

When a microservice fails in Kubernetes, it triggers a domino effect (cascading failure):

Deployment #218 (Memory Leak)
       │
       ▼
Garbage Collection Pressure
       │
       ▼
CPU Saturation (98%)
       │
       ▼
Redis Connection Timeout
       │
       ▼
API Gateway Retry Storm
       │
       ▼
Payment Service Down (HTTP 500)

Traditional alerting floods you with alerts for the bottom 4 nodes (the symptoms), leaving SREs and DevOps engineers stuck sifting through noise during high-stakes outages.

💡 Introducing IRCAE: Autonomous Root Cause Engine

To solve this, we are building IRCAE (Intelligent Root Cause Analysis Engine)—an open-source, enterprise-grade platform designed to turn raw telemetry into autonomous causal reasoning.

Instead of asking SREs to correlate telemetry manually, IRCAE automatically answers: "Why did the system fail?" in less than 10 seconds.

🌟 Key Highlights

🚀 Written in Rust (Axum + Tokio): Built for high-throughput, near-bare-metal performance with zero garbage collection pauses.
🕸️ Dynamic Multi-Layer Knowledge Graph: Automatically maps service dependencies, Kubernetes pods, nodes, git commits, and cloud infrastructure.
🧮 Mathematical Causal Inference (SCM & Bayesian Networks): Deterministic, hallucination-free causal algorithms (PyTorch Geometric GNN / TGN).
📝 Explainable AI (XAI): LLMs are only used at the very last step to translate structured mathematical proofs into human-readable incident post-mortems!

⚙️ How IRCAE Works Under the Hood

IRCAE processes millions of telemetry events per minute through a clean 4-stage pipeline:

1. TELEMETRY INGESTION (Prometheus, Loki, OTel, K8s, Git)
                     │
                     ▼
2. TOPOLOGY GRAPH DISCOVERY (Service & Infra Dependency Graph)
                     │
                     ▼
3. CAUSAL REASONING ENGINE (Structural Causal Models & DBN)
                     │
                     ▼
4. EVIDENCE RANKING & POST-MORTEM GENERATION (< 10 seconds)

1️⃣ Telemetry Ingestion & Correlation

IRCAE ingests metrics (Prometheus/VictoriaMetrics), logs (Loki/Elastic), traces (Jaeger/OTel), and infrastructure events (Kubernetes API, ArgoCD, GitHub webhooks) into a synchronized temporal sliding window.

2️⃣ Dynamic Topology Discovery

Using trace headers and Kubernetes metadata, IRCAE constructs a dynamic graph:

Nodes: Services, Pods, Nodes, Commit SHAs, Database Instances.
Edges: CALLS, RUNS_ON, DEPLOYED_BY, DEPENDS_ON.

3️⃣ Hallucination-Free Causal Reasoning

Unlike "AI Ops" tools that throw raw logs directly at an LLM (leading to wild hallucinations), IRCAE relies on strict mathematical models:

Structural Causal Models (SCM): Formulates variables as $Y = f(X, U)$.
Dynamic Bayesian Networks: Computes $P(\text{RootCause} \mid \text{ObservedAnomalies})$.

4️⃣ Ranked Evidence Output

IRCAE outputs ranked hypotheses with concrete confidence scores and supporting evidence:

{
  "incident_id": "inc-2026-0807-001",
  "confidence_score": 0.965,
  "primary_root_cause": {
    "type": "DEPLOYMENT_MEMORY_LEAK",
    "target_entity": "deployment/payment-service",
    "commit_sha": "8f2a1c9b"
  },
  "evidence": [
    "Deployment v2.1.8 occurred at 14:00 UTC",
    "Pod memory increased by +420%",
    "Redis connection pool exhausted at 14:03 UTC"
  ]
}

⚡ Quick Start: Analyzing an Incident via REST API

Because IRCAE is written in Rust, running an analysis is lightning fast:

curl -X POST http://localhost:8080/api/v1/incidents/analyze \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Payment Gateway Timeout",
    "events": [
      {
        "id": "ev-101",
        "source_system": "KubernetesAPI",
        "event_type": "Deployment",
        "entity_id": "deployment/payment-service",
        "timestamp": "2026-08-07T00:00:00Z",
        "anomaly_score": 0.95
      },
      {
        "id": "ev-102",
        "source_system": "Prometheus",
        "event_type": "MetricAnomaly",
        "entity_id": "pod/payment-pod-1",
        "timestamp": "2026-08-07T00:01:00Z",
        "anomaly_score": 0.75
      }
    ]
  }'

🤝 What's Next & How to Get Involved

Observability needs a paradigm shift from passive dashboards to autonomous root cause reasoning.

We are actively developing IRCAE as an Apache-2.0 open-source project, and we’d love your feedback, contributions, and ideas!

⭐️ GitHub Repo: muhammadlutfimuzaki/ircae (give us a star if you like the concept!)
💬 Drop a comment below: How does your team currently handle cascading microservice failures during on-call incidents?

Happy coding & zero-downtime shipping! 🚀🦀

DEV Community