A few years ago, if you wanted to build an intelligent chatbot, most of your effort went into getting the computer to understand people. Intent classification, entity extraction, stemming, confidence scores, dialogue trees, fallback logic. You annotated thousands of examples, hand-crafted regular expressions, and tuned thresholds until the system stopped misreading "cancel my order" as a request to place one.
The hardest problem was not deciding what the system should do. It was figuring out what the user actually meant.
Today, that problem is largely solved.
Large Language Models have made natural language one of the easiest parts of the stack. We rarely train intent classifiers anymore. We do not spend weeks annotating data or crafting brittle pattern matches. We hand a request to an LLM and, more often than not, it understands.
So the bottleneck moved. The question stopped being:
How do I make the computer understand English?
It became:
Now that it understands, what should it do — and how do I make it do that reliably?
That second question is where most of the difficulty now lives.
The engineering shifted up the stack
The challenges are no longer primarily about language processing. They are about systems engineering:
- agent orchestration
- tool calling
- memory and state
- permission models
- sandboxed execution
- observability
- evaluation
- reliability
- cost control
- governance
- human approval
Ironically, software engineering has become more important, not less.
Calling an LLM API is easy. Building a system that can reliably interact with dozens of services, recover from failures, execute code safely, respect permissions, remain observable in production and scale to thousands of users is anything but. Most of that difficulty is invisible in a demo. A demo needs one happy path to work once. A product needs every path to fail safely, ten thousand times a day.
It is worth being concrete about why these problems are difficult, because "orchestration is hard" is the kind of statement everyone agrees with and no one learns from. Consider four of them.
Sandboxed execution
The moment an agent can run code — even if it is only doing arithmetic, transforming data or generating a report — you have given an untrusted author the ability to execute arbitrary logic inside your infrastructure. That author is the model, influenced by whoever or whatever supplies its context.
The naive implementation is to run generated code inside the application process, or inside a long-lived container reused across requests. Both approaches are quietly dangerous. Code written by the model may be able to read environment variables, access credentials, make outbound network calls, consume unbounded CPU or memory, inspect files it should not see, or leave state behind that leaks into the next execution.
Doing this properly means treating every execution as hostile. You need real isolation — separate namespaces, a stronger sandbox, a microVM, or technologies such as gVisor or Firecracker rather than a shared runtime. You need no ambient credentials inside the execution environment, controlled network egress so generated code cannot simply phone home, hard limits on CPU, memory, time and filesystem usage, and an ephemeral environment that is destroyed after each run.
While building Jhansi, this was the shift that became impossible to ignore. What initially looked like "let the model execute some Python" quickly became a runtime problem. The interesting questions were no longer about prompting. They were about process isolation, sandbox lifecycle, resource limits, auditability, and what an execution environment should be allowed to see.
The difficult part was never running the code. It was running code you did not write, cannot fully predict and should not trust — safely, on every request.
Permission models and tool calling
When you give an agent tools, the model decides which tool to call and what arguments to supply. That decision is non-deterministic, and — more importantly — it can be influenced by input you do not control.
The naive implementation is to register every tool with application-level credentials and allow the model to choose freely. That creates a classic confused-deputy problem: a prompt injection hidden inside a document, email or web page may persuade the agent to call a destructive tool on the attacker's behalf, using the application's authority.
The model is not malicious. It is obedient. And obedience is the vulnerability.
Getting this right pushes you towards least privilege at the action level rather than the application level. The agent should not hold long-lived credentials. High-blast-radius actions should require explicit authorisation, policy checks, or a human in the loop. Every tool call should be attributable, reviewable and revocable — the system should be able to say who authorised an action, whose permissions were used, what data was exposed, what the model was attempting to do, whether it could have been prevented, and whether it can be reversed.
At that point, you are no longer writing prompt logic. You are designing an authorisation system that happens to have a language model sitting in the middle of it.
Observability
With deterministic software, when something fails you inspect the logs or read the stack trace. With an agent, "why did it do that?" may span a non-deterministic model decision, retrieved context, accumulated memory, multiple tool calls, external API responses, retries, intermediate outputs and a final action. A stack trace captures almost none of that.
To debug an agent in production, you need traces that record the complete decision-and-tool-call chain: token and cost accounting for each step, plus the prompt, retrieved context, tool arguments, tool outputs and policy decisions that shaped the run. Ideally you also need the ability to replay the workflow against the same context. Without that, you are staring at an outcome with no reliable way to reconstruct the sequence of information that produced it. The context may already be gone, the model may produce a different answer if you run it again, and the downstream service may have changed.
Debugging shifts from read the error and find the faulty line to reconstruct a decision the system made from a chain of state, context and external events. That is a different engineering discipline, and many teams only discover they need it after the first serious production incident.
Evaluation
Traditional software has unit tests. Agentic systems need something broader, because a workflow that succeeds today may fail tomorrow even when your own source code has not changed. The model may have been updated, a prompt may have drifted, a retrieval source may now contain different information, a downstream tool may return a new response shape, or the model may simply choose a different sequence of actions for the same request.
A conventional test suite can verify that the code runs. It cannot always verify that the system still makes good decisions.
Teams therefore need evaluation suites built around representative tasks and expected outcomes — measuring not just task success, but factual accuracy, policy compliance, tool-selection quality, latency, token usage, monetary cost, consistency, safety and escalation behaviour. You also need regression datasets built from real failures. Every production incident should become an evaluation case. Every unexpected tool call should become a scenario. Every prompt injection that almost worked should become a permanent test.
The question changes from does the code compile? to does the system still behave well? That sounds like a small distinction. It is not. It changes what testing means.
We have become obsessed with prompts
Prompt engineering has its place. A carefully designed prompt can improve reasoning, constrain output and reduce ambiguity. But prompts are increasingly becoming an implementation detail. Models continue to improve, reasoning becomes stronger, context windows grow, and tool-use capabilities become more standardised. A prompt advantage is also easy to copy.
The competitive advantage is unlikely to come from discovering a clever phrase that every competitor can reproduce tomorrow. It will come from designing better systems around the intelligence.
The companies that win will not necessarily have the smartest model. They will build the best products around it — products that are reliable, secure, observable, affordable, governable, scalable and useful. The model is increasingly a commodity input. What you build around it is not.
We have been here before
In the early days of the web, one of the hard problems was making a page render correctly. Frameworks made that easier. Then the challenge moved to server-side applications, then to APIs, then to distributed systems, then to cloud infrastructure, deployment automation, observability and platform engineering. Each new abstraction removed one category of work and exposed another. The work did not disappear. It moved.
AI feels like the next version of that transition. LLMs removed much of the effort required to interpret natural language, but that did not eliminate engineering — it made a new class of products possible, and those products still have to be secured, operated, tested and scaled. Every generation of tooling creates the impression that engineering is about to become unnecessary. In practice, better abstractions usually raise the level at which engineering happens.
We no longer spend as much time asking whether a machine can understand a sentence. Now we have to decide what authority it should have, how it should act, what happens when it is wrong, and how the rest of the system should contain the consequences. Those are not smaller problems. They are bigger ones.
If anything, LLMs increased the amount of systems work available, because they made previously impossible products feasible — and everything that becomes possible eventually has to be made reliable. The engineers who thrive over the next decade will not simply know how to use AI. They will know how to architect systems around it, integrate it with real platforms, constrain its authority, evaluate its behaviour, operate it safely, recover when it fails, and turn intelligence into something customers can trust.
The question is no longer:
Can AI understand the user?
It is:
Can we build systems that make that understanding useful — reliably, safely and at scale?
Language is no longer the bottleneck. Systems are.
Top comments (0)