The Hard Part of AI Engineering Isn’t the Model

Touma Asakura — Sun, 27 Sep 2026 19:58:26 +0000

When I first started looking seriously at AI engineering, I thought the main problem would be choosing the right model.

GPT, Claude, Gemini, open-source models, context windows, benchmarks, reasoning ability…

But after building with them for a while, I started noticing something.

The model is often not the hardest part.

The difficult part is everything around it.

You can have an extremely capable model and still build a terrible AI system.

For example, imagine an AI support agent.

At first it looks simple:

User → LLM → Answer

Then you actually try to put it into production.

Now it needs to read documentation, search previous conversations, call APIs, understand permissions, remember context, handle tool failures, retry requests, validate outputs, and sometimes admit that it doesn’t know what to do.

Suddenly it looks more like:

User
 ↓
Context / Retrieval
 ↓
LLM
 ↓
Validation
 ↓
Tools
 ↓
Verification
 ↓
Response

The LLM is just one component in the middle.

And unlike most components in traditional software, it is probabilistic.

If I write:

result = add(10, 20)

I expect 30.

But if I ask a model:

result = model("What action should I take?")

I am not getting a guaranteed result.

I’m getting a prediction.

That difference seems small until the model is allowed to do something real.

Imagine:

Model decides → refund customer
Model decides → delete record
Model decides → send email
Model decides → execute code

Now a hallucination is no longer just a weird sentence.

It becomes a system failure.

That’s why I think one of the most important patterns in AI engineering is:

Model proposes
System validates
Tool executes
System verifies

The model should be able to reason.

But reasoning and authority should not be the same thing.

Another thing I’ve started thinking about differently is prompt engineering.

People talk a lot about writing the perfect prompt.

I’m not sure that’s the biggest problem anymore.

A perfect prompt with the wrong context is still useless.

If a model needs five documents to answer a question, the real engineering challenge is finding the correct five documents out of thousands.

So the problem becomes less:

“How should I phrase the instruction?”

and more:

“What information should the model see right now?”

That is a much deeper problem.

The same thing happens with memory.

Giving an agent “memory” sounds impressive until you ask basic questions:

What should it remember?

For how long?

What happens when information becomes outdated?

What if two memories contradict each other?

When should something be forgotten?

At that point, memory stops being a cool AI feature and starts becoming a serious data engineering problem.

And then there is debugging.

With normal software, if something fails, I can usually inspect the code path.

With an AI system, the failure could come from almost anywhere.

Maybe retrieval returned the wrong document.

Maybe the prompt was ambiguous.

Maybe a tool failed.

Maybe the model misunderstood the tool output.

Maybe old memory polluted the context.

Maybe the model just made a bad decision.

So logs like:

Request failed

are not enough anymore.

We need to see the whole chain:

User input
Retrieved context
Model output
Tool calls
Tool responses
Validation
Retries
Final answer

That’s where AI engineering starts feeling less like “using an API” and more like actual systems engineering.

And weirdly, a lot of the solutions are not new.

Distributed systems already taught us about retries and failures.

Security taught us about permissions and trust boundaries.

Databases taught us about state and consistency.

Software architecture taught us not to give one component control over everything.

AI just adds a new kind of component:

something extremely useful that can also be confidently wrong.

Maybe that is the real challenge.

Not building a system where the AI is always correct.

But building a system that remains reliable when it isn’t.

That, to me, is where AI engineering becomes genuinely interesting.

Curious how others see this.

When you build with LLMs, what causes more problems: the model itself, or the system around it?

DEV Community: Touma Asakura

The Hard Part of AI Engineering Isn’t the Model