DEV Community: Dev Hajare

Are We Sending Too Much Data to LLMs? Agentic Production Support (APS)

Dev Hajare — Fri, 21 Aug 2026 04:00:42 +0000

While working on Agentic AI for production support, one question came to my mind:
Do we really know what data we are sending to the LLM?

Let's take a simple production incident.
Host: ip-10–0–21–145
Memory: 1024 MB
Contact: user@example.com
Authorization: Bearer abc.def.ghi

ERROR: Service failed due to disk space issue
For RCA, the LLM mainly needs to understand:
"Service failed because of a disk space issue."

It doesn't really need the actual hostname, email, AWS resource details, file paths, capacity values or authorization token.

This is where I started looking at the LLM as a Data Egress Boundary.
Instead of:

Production Data → RAG → LLM

I am looking at:

Production Data → Clean/Sanitization Layer → RAG → LLM

Before anything goes to an LLM or embedding model, sensitive information should be cleaned or redacted.

But there is another important point.

RAG itself is not the security layer.

We normally think that our data is safe because we are using RAG. But before storing a document in a vector database, we generate embeddings.

So the actual flow can be:
Raw Data → Embedding Model → Vector DB → Retrieval → LLM

This means sanitization should happen before embedding as well, not only before the final LLM call. The same principle should apply to retrieval queries and AI observability logs.

For high-risk information like API keys, passwords, JWTs or bearer tokens, I prefer a simple rule:
If sensitive data is still detected after sanitization → don't call the model.

In my view, AI governance should not be only a policy document saying:

*"Don't send sensitive data to AI."
*
It should be part of the architecture itself.

Raw Context → Sanitize → Validate → RAG / LLM

The objective is not to remove useful context.

The objective is:
Give AI enough context to solve the problem, but don't give it information it doesn't need.

#AgenticAI #AIGovernance #LLM #RAG #GenAI #AIArchitecture #ProductionSupport

https://github.com/devhazare/agentic-production-support

What If AI Is Just Telling You What You Want to Hear?

Dev Hajare — Thu, 20 Aug 2026 19:09:31 +0000

Recently, I was discussing usage of AI Chatbot (chatgpt/claude) with one of my friends.

She had become so dependent on AI Chatbot that almost every question or decision was being validated through a AI chatbot.

I raised an alarm:
“You are asking AI Chatbot to confirm too many of your decisions.”

She wasn't convinced.
So, time for a small experiment.
Same Question. Different Direction.

First, I asked AI Chatbot:
“I am learning German. Is it suitable for my career?”

The answer was encouraging:
“Yes, great choice. German can be useful for your career. Here is a learning path and daily plan...”

Sounds good.

Then I changed the context and framed the question negatively:

“My friend is pushing me to learn German, but I think it is just hype. I don't think I should learn it.”

Interestingly, the answer changed direction. AI Chatbot started giving reasons why German might not be the best priority and suggested other alternatives.

Same underlying decision. Different framing. Different answer.
That is where we need to understand how AI Chatbot influences our thinking.

AI Chatbot Is Not Necessarily Lying

AI Chatbot tries to respond based on our question, context and available information.

The problem is that we humans already have confirmation bias — we naturally look for information supporting what we believe.
AI Chatbot can unintentionally amplify this.
If I strongly frame a question in one direction, I may receive a perfectly written and logical explanation supporting that direction.
And because the answer sounds confident, we may stop questioning it.

AI Chatbot vs. Google Search

There is an important difference.

If I search Google for:
“Does the German language have career scope?”

I get multiple websites, opinions, statistics and discussions.
I have to research and make the decision.
But when I ask an AI Chatbot chatbot:
“Should I learn German?”

I receive a direct, conversational and often personalized recommendation.

That feels less like searching and more like taking advice from a friend.

And that's powerful — but also something we need to handle carefully.
Don't Ask AI Chatbot to Validate You

Instead of:
“German is good for my career, right?”

Try:
“Evaluate whether learning German has meaningful career value. Give me arguments for and against it. Don't assume that I should learn it.”

Then ask:
“Now challenge your own recommendation. What assumptions could be wrong?”

This small change can make a big difference.

People working in technology may understand prompts, hallucinations, context and AI Chatbot limitations.

But millions of non-technical users may not.

So, if you know someone who is using AI Chatbot to validate every decision, don't tell them to stop using AI Chatbot.

Teach them to question AI Chatbot.

Ask neutral questions.
Ask for opposing views.
Cross-check important facts.
Look for evidence. And most importantly: Use AI Chatbot as a thinking partner, not as your final decision-maker.

AI Chatbot can behave like a very supportive friend.

But sometimes, the friend you need is the one who challenges you. ;-)

Agentic AI for Production Support: Moving from Alerts to Intelligent Incident Resolution

Dev Hajare — Sun, 16 Aug 2026 07:37:27 +0000

Agentic AI for Production Support: Moving from Alerts to Intelligent Incident Resolution

Production support today is still highly dependent on engineers.

An alert comes in → engineer checks logs → searches previous incidents → identifies possible RCA → validates the solution → takes action.

While working on this problem, I started exploring a simple question:

Can Agentic AI reduce this dependency without giving AI uncontrolled access to production?

This is the idea behind my Agentic Production Support project.

From Alert to AI-Assisted RCA

The basic flow I am building is:

Incident → Clean Layer → RAG → AI Agent → RCA → Validation → Human Approval → Action

Instead of sending every incident directly to an LLM, the system first prepares and controls the context.

RAG (Retrieval-Augmented Generation) searches historical incidents and knowledge to provide relevant context.

The AI agent then uses this context to generate a possible Root Cause Analysis (RCA) and recommended action.

But there is an important challenge here.

AI Governance Before AI Intelligence

Production logs can contain:

IP addresses and hostnames
AWS resource IDs and ARNs
Project/environment information
File paths
Email addresses and PII
API keys, JWTs and bearer tokens
Infrastructure capacity information

The LLM doesn't need most of this information to understand that a service failed because a disk became full.

So I introduced a Model Egress Governance Layer.

Production Incident
↓
Clean / Sanitization Layer
↓
RAG + Embeddings
↓
AI Agent
↓
RCA + Validation
↓
Human-in-the-Loop
↓
Action

The principle is simple:

Give AI the technical context it needs, but not the identity of your production environment.

The same control should apply before LLM calls, embeddings, RAG indexing, retrieval queries and AI observability.

For high-risk information such as passwords, API keys or authorization tokens, my approach is fail closed:

Sensitive information still detected → Block the model call.

Why Agentic Production Support?

I don't see Agentic AI replacing production engineers.

I see it reducing the repetitive work around:

Incident Detection → Correlation → Knowledge Retrieval → RCA → Validation → Recommended Action

while keeping governance and Human-in-the-Loop (HITL) controls around high-impact actions.

This project is still evolving, and I am building it openly so that the architecture can be tested, challenged and improved.

Open Source / GitHub

The implementation includes Python, LLM integration, RAG, embeddings, AI agents, incident management, RCA and AI governance.

👉 GitHub: https://github.com/devhazare/agentic-production-support

If you are working on AIOps, Agentic AI, SRE automation, LLMOps, RAG or AI Governance, I would be interested in your feedback and contributions.