Dev Hajare

Posted on Aug 16

Agentic AI for Production Support: Moving from Alerts to Intelligent Incident Resolution

#ai #agentaichallenge #llm #rag

Production support today is still highly dependent on engineers.

An alert comes in → engineer checks logs → searches previous incidents → identifies possible RCA → validates the solution → takes action.

While working on this problem, I started exploring a simple question:

Can Agentic AI reduce this dependency without giving AI uncontrolled access to production?

This is the idea behind my Agentic Production Support project.

From Alert to AI-Assisted RCA

The basic flow I am building is:

Incident → Clean Layer → RAG → AI Agent → RCA → Validation → Human Approval → Action

Instead of sending every incident directly to an LLM, the system first prepares and controls the context.

RAG (Retrieval-Augmented Generation) searches historical incidents and knowledge to provide relevant context.

The AI agent then uses this context to generate a possible Root Cause Analysis (RCA) and recommended action.

But there is an important challenge here.

Production logs can contain:

The LLM doesn't need most of this information to understand that a service failed because a disk became full.

So I introduced a Model Egress Governance Layer.

Production Incident
↓
Clean / Sanitization Layer
↓
RAG + Embeddings
↓
AI Agent
↓
RCA + Validation
↓
Human-in-the-Loop
↓
Action

The principle is simple:

Give AI the technical context it needs, but not the identity of your production environment.

The same control should apply before LLM calls, embeddings, RAG indexing, retrieval queries and AI observability.

For high-risk information such as passwords, API keys or authorization tokens, my approach is fail closed:

Sensitive information still detected → Block the model call.

I don't see Agentic AI replacing production engineers.

I see it reducing the repetitive work around:

Incident Detection → Correlation → Knowledge Retrieval → RCA → Validation → Recommended Action

while keeping governance and Human-in-the-Loop (HITL) controls around high-impact actions.

This project is still evolving, and I am building it openly so that the architecture can be tested, challenged and improved.

The implementation includes Python, LLM integration, RAG, embeddings, AI agents, incident management, RCA and AI governance.

If you are working on AIOps, Agentic AI, SRE automation, LLMOps, RAG or AI Governance, I would be interested in your feedback and contributions.