DEV Community: Siddharth Pandey

Why RAG Alone Cannot Understand Infrastructure

Siddharth Pandey — Mon, 11 May 2026 12:04:13 +0000

Every AI tooling discussion eventually reaches the same sentence:

“Just use RAG.”

At this point, RAG is basically the duct tape of AI architecture.

Missing context?
RAG.

Bad answers?
RAG.

AI hallucinating production infrastructure?
Apparently also RAG.

To be clear: Retrieval-Augmented Generation is genuinely useful.

It improves grounding.
It reduces hallucinations.
It helps AI systems access external knowledge.

But there’s a growing misconception in AI engineering right now:

More retrieval does not automatically create system understanding.

And infrastructure problems are usually system understanding problems.

The Problem Is Not Missing Text

Most infrastructure failures in AI coding assistants are not happening because documentation is missing.

The problem is that infrastructure knowledge is relational, fragmented, and operational.

For example, imagine asking an AI assistant:

“Can this DynamoDB access pattern use an existing index?”

That answer depends on:

actual GSIs
partition keys
sort keys
deployed schema state
access patterns
environment-specific infrastructure
workload constraints

That is not just a “retrieve the right paragraph” problem.

A vector database cannot magically infer operational relationships from semantically similar chunks of text.

Infrastructure is not a Wikipedia article.

It is a topology.

The “Looks Correct” Problem

This is where AI gets dangerous.

The generated output often looks completely reasonable.

Example:

await ordersTable.query({
  IndexName: "customerId-createdAt-index",
});

Looks valid.

Compiles.

Very professional-looking.

Tiny problem: the index does not exist.

The AI retrieved enough surrounding context to generate something statistically plausible.

But plausible infrastructure is not the same thing as real infrastructure.

This is the core limitation of relying entirely on semantic retrieval.

RAG helps models retrieve information.

Infrastructure awareness requires understanding relationships.

Those are different problems.

Infrastructure Is a Graph Problem

Modern systems are deeply interconnected.

A single API endpoint might depend on:

DynamoDB tables
GSIs
SQS queues
Lambda functions
deployment environments
feature flags
event consumers
schema versions

Now imagine asking:

“What breaks if I change this schema?”

That is not a chunk retrieval problem.

That is dependency analysis.

The AI needs to understand:

who consumes the schema
where it is used
which systems depend on it
whether environments differ
whether migrations already exist

Most RAG systems fundamentally operate on semantic similarity.

Infrastructure problems often require deterministic relationship mapping.

Very different category of problem.

Bigger Context Windows Do Not Solve This Either

A lot of people assume larger context windows will eventually solve infrastructure awareness.

I do not think that is true.

A 2 million token context window filled with:

Terraform
CloudFormation
CDK
schemas
docs
configs
deployment files

still does not automatically produce reliable topology understanding.

Because the issue is not just visibility.

It is interpretation.

Even humans struggle with this.

Every engineering organization has:

infrastructure nobody wants to touch
queues nobody fully understands
“temporary” resources that became permanent
databases held together by historical accidents and organizational fear

Giving AI more tokens to read does not automatically create operational intelligence.

Otherwise every engineer who read all the docs would already fully understand production.

Which is obviously not how reality works.

Semantic Similarity vs Deterministic Context

This distinction matters a lot.

RAG systems answer questions like:

“What information is semantically related to this query?”

Infrastructure-aware systems need to answer questions like:

Which indexes actually exist?
Which services consume this queue?
Which schema version is deployed?
Which environment contains this resource?
Which APIs depend on this table?
Which resources are connected?

Those answers should not come from probabilistic guessing.

They should come from deterministic infrastructure context.

That is the missing layer most AI coding systems still lack.

This Is One of the Reasons I Started Building Infrawise

One of the reasons I started working on Infrawise was this exact gap.

The goal is not:

“put infrastructure docs into embeddings.”

The goal is to make infrastructure relationships explicit enough that AI systems stop guessing.

Things like:

DynamoDB index awareness
schema relationships
infrastructure mapping
static analysis
AI-consumable infrastructure context
deterministic extraction pipelines

Not just “more retrieval.”

Because honestly, some infrastructure mistakes are too expensive to leave to semantic similarity.

The Future Is Not Just Better Retrieval

RAG is useful.

It will absolutely remain part of AI systems.

But infrastructure-aware AI requires more than retrieval.

It requires:

topology understanding
dependency mapping
deterministic infrastructure state
operational context
relationship awareness

The next generation of AI coding assistants will not just retrieve infrastructure knowledge.

They will need to understand infrastructure reality.

And that is a much harder problem than embedding documents into a vector database.

Your AI Assistant Knows Your Code But Not Your Architecture

Siddharth Pandey — Sun, 10 May 2026 11:03:22 +0000

AI coding assistants are getting dangerously good.

They can refactor code, generate APIs, write SQL queries, explain regex you wrote at 2 AM during a production incident, and confidently suggest changes that look correct.

And that’s exactly the problem.

Because your AI assistant knows your code.

It does not know your architecture.

There’s a huge difference between:

“I read your files”

and

“I understand your system.”

Right now, most AI tools are basically that one engineer who joined last week, skimmed the repo for 20 minutes, and immediately started suggesting database changes in production.

The Confidence Problem

Let’s say you ask your AI assistant:

“Generate a DynamoDB query for customer orders.”

It happily gives you this:

const result = await dynamo.query({
  TableName: "Orders",
  IndexName: "customerId-createdAt-index",
  KeyConditionExpression: "customerId = :customerId",
});

Looks clean.

Very professional.

Tiny issue though.

That GSI does not exist.

The assistant saw patterns in code and statistically guessed what probably exists.

Which is honestly terrifying when you think about it.

Because infrastructure is not autocomplete-friendly.

Your Architecture Lives Outside the Code

Modern systems are scattered across:

Terraform
CloudFormation
CDK
databases
queues
topics
IAM policies
deployment pipelines
cloud consoles
“temporary” scripts from 2022 that somehow became production-critical

Meanwhile the AI assistant is sitting there reading:

await ordersTable.query(...)

and internally going:

“I have absolutely no idea what this system actually looks like, but statistically speaking… vibes.”

That’s how you end up with:

PostgreSQL joins on non-indexed columns
Event flow assumptions based entirely on naming conventions
DynamoDB scans quietly disguised as “optimizations”
APIs generated against schemas that were deleted three migrations ago
Infrastructure suggestions that look plausible but do not match deployed reality

The code compiles.

The architecture cries silently in the background.

Inference Is Not Infrastructure Awareness

This is the core issue.

Today’s AI coding assistants are really good at inference.

But infrastructure requires deterministic understanding.

There’s a massive difference between:

“This probably exists” and
“This definitely exists in production”

Humans already struggle with this.

Every company has at least one production resource that:

nobody fully understands
nobody wants to touch
somehow costs thousands per month
and is protected by pure organizational fear

Now imagine an AI confidently modifying systems like that.

Why Bigger Context Windows Won’t Solve This

A lot of tooling right now is focused on:

larger context windows
better embeddings
repository RAG
more docs ingestion

Helpful? Yes.

Sufficient? No.

Because infrastructure problems are relationship problems.

Questions like:

Which services consume this queue?
Which indexes support this access pattern?
Which schemas are actually deployed?
Which Lambda depends on this topic?
Which environment even has this resource?

These are topology questions.

Not autocomplete questions.

A 2 million token context window still cannot magically understand infrastructure relationships if the information itself is fragmented or ambiguous.

More tokens do not automatically create system awareness.

This Is Why I Started Building Infrawise

This problem is one of the reasons I started working on Infrawise.

The idea is simple:

Instead of forcing AI assistants to guess infrastructure,
give them deterministic infrastructure context.

Things like:

DynamoDB index awareness
schema relationships
infrastructure mapping
static analysis
infrastructure-aware context for AI systems
MCP integration paths

Not “AI magic.”

Just explicit system understanding.

Because honestly, hallucinated code is annoying.

Hallucinated infrastructure is how people accidentally discover new billing tiers on AWS.

The Next Generation of AI Coding Tools

The current generation of AI assistants understands syntax.

The next generation will need to understand systems.

Not just:

functions
files
classes
frameworks

But:

architecture
runtime dependencies
infrastructure topology
deployment reality
operational constraints

Because software does not run inside VS Code.

It runs inside infrastructure.

And infrastructure is where bad assumptions become expensive.