DEV Community: Tushar Vashishth

AI Solved the Problem Perfectly. There Was Just One Problem.

Tushar Vashishth — Tue, 08 Sep 2026 10:59:46 +0000

It wasn't the problem we actually had.

![AI solving the wrong problem]

There is something slightly funny about working with AI today.

We have incredibly capable models. They can write code, analyze data, summarize documents, reason through problems, call tools, and even take actions on our behalf.

So naturally, when something doesn't work, our first instinct is often:

“Maybe we need a better model.”

Then we try a bigger model.

Still not working?

“Maybe we need an agent.”

Still messy?

“Let's add RAG.”

Still not quite there?

“Maybe we need more tools, memory, or a more complicated pipeline.”

And before we know it, we've built a very sophisticated system...

for the wrong problem. 😅

The AI system can be technically correct and still be wrong

This is something I've been thinking about more while learning about AI-native systems.

Imagine asking an AI system:

“Help me reduce customer support response time.”

And it comes back with a beautifully designed solution for improving customer acquisition.

The solution might be:

logically sound
technically impressive
well researched
perfectly implemented

But it doesn't matter.

It solved a different problem.

This distinction is easy to miss because modern AI is extremely good at producing convincing outputs.

A bad answer used to look bad.

Now, a bad answer can look professional.

That's a much more interesting engineering problem.

Intelligence isn't the same as direction

A highly capable model still needs the right direction.

Think about a very smart employee.

You don't just give them access to every company document, every software tool, and the freedom to make decisions and say:

“Go do something useful.”

You first explain:

what we're trying to achieve,
what information matters,
what they're allowed to do,
what tools they should use,
and how we'll know whether the work was successful.

AI systems need similar thinking.

The model is only one component.

The surrounding system determines how that intelligence is actually used.

This is where things get interesting

When building AI applications, it's tempting to start with technology.

For example:

“Which model should we use?”

But perhaps a better starting point is:

“What problem are we actually trying to solve?”

Then we can work backwards.

Do we need external information?

→ Maybe we need retrieval.

Does the system actually need to take multiple steps or interact with tools?

→ Maybe we need an agent.

What kind of reasoning or generation is required?

→ Now we can think about the appropriate model.

And once the system produces an answer or takes an action:

→ How do we know it worked?

That's where evaluation, verification and proof become important.

More components don't automatically mean a better system

This is probably one of the easiest traps to fall into with modern AI.

Because there are so many interesting technologies available, we can start adding them simply because we can.

Need a chatbot?

Add RAG.

Need RAG?

Add a vector database.

Need more flexibility?

Add an agent.

Need the agent to do more?

Add tools.

Need better reasoning?

Use a bigger model.

Need reliability?

Add another layer.

Eventually, the architecture looks incredibly sophisticated.

But complexity should have a reason.

Every component should solve a problem.

Otherwise, we're just making the system harder to understand, maintain and debug.

The model isn't always the problem

Suppose an AI application is giving poor answers.

There are many possible reasons.

Maybe the model doesn't have the information it needs.

Maybe retrieval is returning irrelevant context.

Maybe the context is too large or poorly structured.

Maybe the agent is choosing the wrong tool.

Maybe we're using an unnecessarily expensive model for a simple task.

Maybe our evaluation process isn't catching failures.

Or maybe...

we simply misunderstood the original problem.

Replacing the model might fix none of these.

This is why I think AI engineering is becoming less about “which model are you using?” and more about how the entire system is designed around the problem.

A useful mental model: Retrieve → Agents → Models → Proof

One framework I've been exploring through RAMP is:

Retrieve → Agents → Models → Proof

I like this because it encourages thinking beyond the model itself.

Retrieve

Give the system the information it actually needs.

The goal isn't simply to retrieve more information.

It's to retrieve relevant information.

Agents

If the problem genuinely requires actions, decisions, or multiple steps, then agents can become useful.

But not every problem needs an agent.

Sometimes a simple workflow is better.

Models

Choose the model based on the actual requirement.

The biggest or newest model isn't automatically the right choice.

Sometimes speed matters more.

Sometimes cost matters.

Sometimes reasoning capability matters.

Sometimes a smaller model is perfectly sufficient.

Proof

Finally:

How do we know the system worked?

This is arguably one of the most important questions as AI systems become more autonomous.

An answer isn't automatically trustworthy just because it sounds convincing.

We need ways to evaluate, verify and measure system behavior.

The real goal isn't “make AI smarter”

Of course, better models matter.

Model capabilities have improved dramatically, and they will continue to improve.

But there's another layer to the problem.

A smarter model can produce a better answer to the wrong question.

And that's still the wrong answer.

So perhaps the goal shouldn't simply be:

Make AI smarter.

It should be:

Build systems that use intelligence in the right direction.

That means understanding the problem first, choosing the right architecture second, and selecting the technology that actually supports it.

One question I'm trying to ask more often

Whenever I see a new AI technique, framework or model, instead of immediately asking:

“How can I use this?”

I'm trying to ask:

“What problem would this actually solve?”

That small change in perspective can make a surprisingly big difference.

Because the AI ecosystem is moving extremely fast.

There will always be another model.

Another framework.

Another agent architecture.

Another tool.

Another technique that everyone is talking about.

We probably don't need to use all of them.

We need to understand why we'd use them.

The takeaway

AI can be incredibly smart.

It can reason.

It can retrieve information.

It can use tools.

It can write code.

It can take actions.

But none of that guarantees that we're solving the right problem.

So before asking:

“Which model should I use?”

Maybe ask:

“What exactly am I trying to solve?”

Because sometimes the best AI solution isn't a bigger model.

It's realizing that the problem was defined incorrectly in the first place.

And honestly...

“Cool. But that's not what I asked.”

might become one of the most important debugging messages in AI engineering. 😄

I'm currently exploring these ideas as part of my RAMP learning journey, and this shift—from thinking primarily about models to thinking about the complete AI system—is one of the perspectives I've found particularly useful.

From Learning the Stack to Understanding the System

Tushar Vashishth — Tue, 25 Aug 2026 13:30:40 +0000

Why AI-native development may require a different way of thinking about technology.

The way we learned software for years
LAMP → MEAN → MERN → other stacks.
Learning the stack gave developers a practical mental model for building applications.
Then AI changed the shape of the application
We're no longer just choosing frontend, backend and database technologies. A modern AI application may involve retrieval, agents, multiple models, tools, memory, evaluation, and human intervention.
The problem with thinking only in terms of tools
Models change quickly. Frameworks change quickly. Tools come and go. If your understanding is tied too closely to one particular stack, it can become outdated quickly.
So what should we understand instead?
This is where we introduce the four RAMP areas naturally:

Retrieve — How does the system get the information it needs?
Agents — How does it decide, act, and interact with tools?
Models — Which intelligence is appropriate for each task?
Proof — How do we know the result is reliable?

RAMP isn't another programming stack
This distinction is important. It's a way of organizing the major pieces of an AI-native system, rather than telling developers which programming language or framework to use.
What this could mean for developers
Instead of only asking “What technologies do I know?”, developers may increasingly need to ask “Can I design and reason about an AI system?”
What happens to MERN, MEAN, LAMP, etc.?
Don't say they are obsolete. They still solve the application-stack problem. RAMP is addressing a different layer.
The bigger question
Could RAMP become a common vocabulary for describing AI-native engineering, in the same way that stack names gave developers a shorthand for web development?

Then finish with something conversational:

I'm still exploring this idea, but I think there's an interesting shift happening here.

We're not necessarily moving away from stacks.

We're adding a new layer of thinking on top of them.

Most RAG Problems Don’t Start With the LLM

Tushar Vashishth — Fri, 21 Aug 2026 10:46:30 +0000

If you've worked with RAG, you've probably seen this:

The answer is wrong.

So you change the prompt.
Try another model.
Increase the context window.
Maybe even switch to a bigger LLM.

And somehow...

the answer is still wrong.

I've started to think that we sometimes look at the wrong part of the system.

The problem may have started before the LLM ever saw the question.

First, look at the whole flow

A simple RAG system can be thought of as:

User Question
      ↓
   Retrieval
      ↓
    Context
      ↓
     LLM
      ↓
    Answer

The LLM is only one part of this.

Before it generates anything, the system has already made several decisions:

What information should be searched?
How were the documents split?
Which results are relevant?
How many results should be returned?
Should some results be filtered out?
Which results should appear first?
What finally goes into the model's context?

So when the final answer is bad, the model isn't necessarily where things went wrong.

Here's a simple example

Imagine you're building an internal support assistant.

Someone asks:

“What's our refund policy for prepaid orders?”

The company has the answer somewhere in its documentation.

Your LLM is capable of understanding the policy.

But your retrieval system returns these:

Refunds are available for eligible purchases.

Customers can contact support regarding refunds.

Refund requests are reviewed within 5 business days.

Prepaid orders are processed immediately.

Refunds may be issued to the original payment method.

Everything looks relevant.

But there's one problem:

The actual rule for prepaid orders wasn't retrieved.

Now the LLM has incomplete information.

It might still produce a very confident answer.

And we might say:

“The LLM hallucinated.”

Maybe.

But the problem actually started earlier.

We gave the model the wrong context.

Where can things go wrong?

There are a few common places.

1. The document was split badly

A policy might say:

Customers can request a refund within 30 days.

And immediately after:

Prepaid orders are subject to different conditions.

If those two statements are split into different chunks, retrieval might find one without the other.

The model gets half the story.

2. The right document wasn't retrieved

This is probably the easiest one to overlook.

The answer may be sitting inside your knowledge base.

But your search never brings it back.

So you:

change the prompt → change the model → change the temperature → try again

...while the correct document is still sitting in the database.

Sometimes the model isn't failing.

The system simply didn't give it the information it needed.

3. Similar doesn't always mean useful

Suppose your system retrieves the five most similar chunks.

That sounds reasonable.

But imagine all five are about refunds while the one chunk containing the actual prepaid-order exception is ranked sixth.

You've technically retrieved relevant information.

But you haven't retrieved the right information.

That's where things like better search, filtering and re-ranking start becoming important.

4. Old information can look very relevant

Imagine your knowledge base contains:

2022 Refund Policy
2023 Refund Policy
2024 Refund Policy
2025 Refund Policy

A question about the 2025 policy might still retrieve the 2023 document because the wording is almost identical.

This is where metadata can matter:

date → source → department → document type → version

Sometimes the system needs more than semantic similarity to find the right answer.

So... should we just use a bigger model?

Not necessarily.

A bigger model can certainly help with reasoning and generation.

But it can't reliably answer from information it never received.

Think about it this way:

Bad context
     ↓
Good model
     ↓
Bad / unreliable answer

versus:

Good context
     ↓
Appropriate model
     ↓
Much better chance of a useful answer

That's why I'm starting to look at RAG less as:

“Search + LLM”

and more as:

“Getting the right information to the right model at the right time.”

What I would check first

If a RAG application is giving poor answers, I'd be tempted to check the retrieval pipeline before immediately replacing the LLM.

A simple checklist:

1. Retrieval
Did we actually find the right information?

2. Chunking
Did we split the information in a useful way?

3. Ranking
Did the useful result make it near the top?

4. Metadata
Are we filtering by things like date, source or version?

5. Context
Is the model actually receiving enough of the right information?

6. Generation
Only then — is the model struggling to turn that context into a good answer?

The Chapter 2 material I've been going through goes quite deep into these areas, including embeddings, vector databases, chunking, hybrid search, re-ranking and retrieval evaluation.

The bigger takeaway

The more I work through this, the more I think:

Building a good RAG system isn't just about choosing a smart model.

It's about building a good path between:

the question → the information → the model → the answer.

If the information going into that path is wrong, incomplete or badly ranked, throwing a smarter model at the end of it may not solve much.

And that's probably the part of RAG I find most interesting right now.

What has caused more trouble in your RAG projects?

Retrieval or generation?

I'd genuinely be interested to hear what others have run into.

Your ML model isn't the whole AI system anymore

Tushar Vashishth — Thu, 20 Aug 2026 07:59:49 +0000

For a long time, when we talked about an ML project, the model was usually at the center of everything.

Get the data.
Train the model.
Improve the accuracy.
Deploy it.
Monitor it.

Pretty straightforward.

But AI applications are starting to feel different.

Imagine you build a system that answers questions about your company's internal data.

Having a good model is important, but that's only the beginning.

Where does the information come from?

What if the answer isn't in one document?

What if the system needs to search, think through a few steps, use a tool, and then come back with an answer?

What if a smaller model is better for one part of the job and a larger one is better for another?

And probably the hardest question:

How do you know the answer is actually good?

Suddenly, the model isn't doing everything.

There's a whole system around it: information, different models, actions, checks, and people when needed.

I think this is one of the biggest changes in how we should think about building AI applications.

We're not just building a model anymore.

We're building a system that knows how to use intelligence.

We're getting really good at adding AI to things.

Tushar Vashishth — Wed, 19 Aug 2026 13:33:15 +0000

Add an LLM here.
Add a copilot there.
Put an agent in the workflow.

Done. “AI-native.” 😅

But building around AI feels like a different problem.

Where does the right information come from?
Which model should handle the job?
What should an agent actually do?
And probably the biggest one — how do we know the result is good?

I've been looking at an idea from AiDOOS around this called RAMP — Retrieve, Agents, Models, Proof.

Still early, but I think there's something worth exploring here.

Maybe being AI-native is less about adding AI...

and more about rethinking how the whole system works.