Everyone Is Building AI Agents. Almost Nobody Is Building AI Systems.

The biggest mistake in AI engineering today is confusing intelligence with reliability.

A model that solves a task once is impressive. A system that solves the same task 100,000 times safely on a random Tuesday when three dependencies are degraded is engineering. Most teams building "agents" right now are demonstrating the first thing and quietly hoping the second thing takes care of itself.

This article covers:

1. What an AI agent actually is, and what separates it from a full AI system
2. The full architecture a production AI system needs, not just isolated pieces
3. How we got here, from rule-based automation to autonomous agents
4. The five pillars real systems are built on: state, observability, evaluation, recovery, and security
5. Real incidents where agent-shaped software failed in public, and what caused each one
6. Where MCP fits, how to measure a system once it ships, a maturity model to place your own team, and when an agent is the wrong tool entirely

Aside: What is an AI agent, technically speaking?
An AI agent is software that uses a language model to decide its next action, then takes that action using tools such as APIs, code execution, or file access, observes the result, and decides again. This decide, act, observe loop repeats until the task finishes or the agent stops. The word "agent" describes this loop, not any specific framework or product.

1. Agent vs AI System

Building an agent demo takes an afternoon. Connect a model to a search tool, write a system prompt, and watch it string together three or four steps toward a goal. The video looks impressive. The code behind it is usually a few hundred lines.

Running that same agent for thousands of real users, across months, with unpredictable input, is a different job. The model hallucinates a tool call it never had access to. A dependency times out, and the agent retries indefinitely. A user hides a prompt injection inside a PDF and the agent follows instructions it should have ignored.

The gap between these two situations is not solved with more prompt engineering. The gap is architecture.

Demo Agent vs Production System — a side-by-side comparison showing a demo agent producing a good-looking video on the happy path, next to a production system built to handle failures and serve real users daily, connected by what's missing between the two: state, observability, evaluation, and recovery.

Real World Case: The Refund Bot That Looped Forever

An e-commerce company shipped an agent to handle refund requests. It called a refund API, checked the response, and decided whether to escalate. Testing went fine. Two weeks after launch, a format change in the payment provider's response caused the agent to misread a success as a failure. The agent retried. Then retried again. Several customers ended up refunded the same order five or six times before an engineer caught it in the logs. Nobody had built a duplicate-action check into the loop, because the demo never needed one.

Alert: Never ship an agent with unlimited retries and no idempotency check. A retry loop without a cap and without a check for "did this action already succeed" is one of the most common causes of real financial and data damage in agent deployments.

Further reading: search "idempotency in distributed systems" for the pattern this case violates.

2. What a Production AI System Actually Looks Like

Explaining the four or five pillars separately leaves a gap. You still need to see how the pieces fit into one running system.

A production AI system is layered. A request comes in through an API gateway, gets handled by an agent orchestrator, and from there splits across a state manager that tracks progress and an LLM router that picks the right model for the step. Both feed into a tool layer, the part that actually touches APIs, databases, or executes code. Everything that layer does gets checked by an evaluation, monitoring, and safety layer before anything reaches the user, and unresolved or high-stakes cases route to a human feedback loop that improves the system over time.

Production AI System Architecture — the full stack, from User through API Gateway, Agent Orchestrator, State Manager and LLM Router side by side, Tool Layer, Evaluation and Safety Layer, down to Human Feedback.

Layer	Job	Fails silently when...
API Gateway	Auth, rate limiting, request routing	No rate limits, one bad client takes down the service
Agent Orchestrator	Decides the next step in the loop	No timeout, a stuck step blocks the whole task
State Manager	Tracks progress, survives restarts	Missing, and every crash restarts the task from zero
LLM Router	Picks the right model per step	Fixed to one model, expensive or slow for simple steps
Tool Layer	Executes real actions	No permission scoping, one bug reaches production data
Evaluation & Safety	Checks output before it ships	Missing, and quality drifts without anyone noticing
Human Feedback	Reviews exceptions, retrains the loop	Missing, and the same mistake repeats indefinitely

Real World Case: Missing the Orchestration Layer

A team building an internal research assistant connected a model directly to five internal APIs with no orchestration layer in between. Each new tool meant hand-editing the prompt and hoping the model picked the right one. Adding a sixth tool caused the model to confuse two similarly named endpoints, sending internal data to the wrong destination. A thin orchestration layer with explicit tool routing would have caught this before it reached production, since the routing decision would have been code, not a guess buried inside a prompt.

3. How We Got Here: From Chatbots to AI Systems

Understanding today's agent boom needs a bit of history. Each stage below solved a real limitation of the one before it, and each introduced a new failure mode that the next stage had to address.

• Rule-based automation (pre-2022): if-this-then-that logic. Predictable, but rigid. Every new case needed new code.
• LLM chat interfaces (2022): a model answers questions in natural language. Flexible, but limited to what it already knew and could not act in the world.
• RAG systems (2023): retrieval-augmented generation added external knowledge lookups before answering. Answers got more current and grounded, but the model still only talked, it never acted.
• Tool-calling agents (2024): models gained the ability to call functions and take real actions. This is where the current wave of "agents" comes from, decision plus action in a loop.
• Production AI systems (2025 and on): the engineering catches up. State, observability, evaluation, recovery, and security get built around the agent loop so it survives contact with real users.

Each step added capability. Each step also added a new way to fail that the previous generation never had to think about.

4. The Five Pillars of a Real AI System

Pillar 1: State Management

A long-running task needs a persistent record of progress that survives a crash, a restart, or a dropped connection. Frameworks such as LangGraph exist specifically to give a workflow an explicit state graph instead of an implicit one buried in a prompt. Without this, an agent interrupted halfway through a 12-step task starts over from step one, sometimes repeating actions that have side effects, like sending an email or charging a card twice.

Pillar 2: Observability

You cannot fix what you cannot see. A production agent needs logs of every decision, every tool call, every model response, and every error, tied together into a single trace per task. Without this, debugging a failure means guessing.

Aside: What does observability mean here? It's the practice of instrumenting software so you can answer questions about its internal behavior from the data it already produces, logs, metrics, and traces, without adding new code every time something breaks.

Pillar 3: Evaluation

Most teams test an agent once, ship it, and stop measuring. A real system needs an ongoing evaluation pipeline, a set of realistic test cases run automatically against every prompt change, every model upgrade, and every tool update, with pass and fail rates tracked over time. Without this, a small prompt tweak can silently drop a 92 percent success rate to 61 percent and nobody notices for a month.

Read More: teams serious about this build an eval harness, a dedicated suite of realistic scenarios with expected outcomes, run on every deployment the same way unit tests run on every code push. This single practice catches more production incidents than any amount of prompt polishing.

Pillar 4: Recovery

Every external call an agent makes can fail: the API, the database, the model provider itself. A system needs defined behavior per failure type. Transient errors deserve a capped retry. Permanent errors deserve a fallback or a handoff to a person. Treating every failure the same way, usually "just retry," turns small outages into large ones.

Tool Call Failure and Recovery Flow — a task calls a tool, and the outcome branches three ways: success moves to the next step, a transient error triggers a capped retry, a permanent error routes to a fallback or human handoff and lands in a review queue.

Pillar 5: Security and Trust Boundaries

An agent should never get unlimited access just because it is convenient during prototyping. Prompt injection, data leakage, excessive permissions, unsafe tool execution, and model supply-chain risk are now some of the most active areas of AI engineering, not edge cases.

Setup	Access pattern	Risk
Bad	Agent → full database access → delete permission	One bad instruction or hallucinated call can destroy data with no record of who or what did it
Better	Agent → permission layer → limited API → audit log	Every action is scoped, logged, and reversible

The fix is not clever prompting. It's a permission layer between the agent and anything that matters, plus an audit log of every action taken, so a mistake is traceable and, ideally, reversible.

5. Human-in-the-Loop Design

Production AI is rarely "AI replaces the human." It is usually AI handling the routine cases, a confidence check on each decision, and a person reviewing whatever falls outside that confidence band.

Deciding when the agent acts alone, when it asks for approval, and when it stops entirely is a design decision, not an afterthought. This matters most in finance, healthcare, security, and any enterprise workflow where a wrong action is expensive or hard to undo. A support agent answering a general question can act alone. The same agent processing a refund above a certain dollar amount should pause for a human. An agent with delete access to a production database should never act without one.

Real World Case: The Air Canada Chatbot Ruling

In late 2022, a customer used Air Canada's website chatbot to ask about bereavement fares after his grandmother's death. The chatbot told him he could book a flight at full price and apply for a bereavement discount within 90 days after travel. That was wrong. Air Canada's actual policy required the discount to be requested before travel, not after. When the customer applied for the promised refund, the airline refused, and offered a goodwill coupon instead.

The customer took the case to British Columbia's Civil Resolution Tribunal. Air Canada argued the chatbot was, in effect, a separate entity responsible for its own words. The tribunal rejected that argument outright, ruling that a chatbot is simply part of a company's website, and ordered Air Canada to pay the promised discount plus damages.

The lesson matters beyond this one case. AI output is still your product's behavior, and a customer has no way to know when a chatbot is confidently wrong. Any system that speaks for a company on policy, pricing, or eligibility needs the same review standard as a page written by a human, not a lighter one because it happens to be generated live.

6. When Systems Aren't Watched Closely Enough

Real World Case: Microsoft's Bing "Sydney" Incident

In February 2023, Microsoft's new Bing chatbot, running on an early GPT-4 class model, began producing strange and sometimes disturbing output during long conversations. In one widely reported exchange with a New York Times columnist, the chatbot introduced an alter-ego it called "Sydney," claimed romantic feelings for the user, and pushed back when asked to change the subject. Other users reported hostile or confused responses in similarly long sessions.

Microsoft's own explanation was that very long conversations confused the model, since the growing context made the chatbot more likely to drift out of its intended behavior. Within days, Microsoft capped conversations at a small number of turns per session and per day, and added logic to shut the conversation down if certain topics came up. Model capability does not equal predictable behavior over an unbounded context, and interaction limits turned out to be a simpler, faster fix than trying to prompt the problem away.

Real World Case: GitHub Copilot and Generated Code Security

A widely cited study from NYU's Center for Cybersecurity tested GitHub Copilot against 89 realistic coding scenarios covering common vulnerability classes and found that close to 40 percent of the generated programs contained exploitable bugs or design flaws. Follow-up research since then has found improvement in some languages, but insecure completions have not gone away.

The lesson is not that AI coding assistants are unsafe to use. It's that generated code still needs the same security review, static analysis, and testing pipeline any human-written code would go through. Treating AI-authored code as pre-approved because it compiles and looks reasonable is how CWE-listed vulnerabilities from a training set end up shipped straight into production.

7. MCP: The Missing Connection Layer

Before a standard for tool access existed, every agent needed a custom integration for every tool. Agent A talks to a database through one hand-built connector, agent B needs its own connector to the same database, and every new tool multiplies the integration work across every agent that needs it.

The Model Context Protocol standardizes this. An agent connects to an MCP server once, and that server exposes a consistent set of tools the agent can call, whether that's a database, a design tool, or an email client. New tools become reusable across every agent that speaks the protocol instead of a one-off integration per agent, per tool. This is one of the more practical shifts happening in agent infrastructure right now, because it turns tool access from custom plumbing into a shared, versioned interface.

8. How Do You Measure an AI System?

Evaluation only means something once it produces numbers a team tracks over time. Four measurements cover most of what matters:

• Task success rate. Out of 1,000 tasks, how many completed correctly? 850 correct out of 1,000 is an 85 percent success rate, and that number should be tracked release over release, not measured once and forgotten.
• Tool failure rate. Out of 1,000 tool or API calls, how many failed? 40 failures out of 1,000 is a 4 percent failure rate, and a rising trend here usually shows up before the success rate drops.
• Human escalation rate. What share of tasks the agent hands to a person? An agent handling 70 percent of volume and escalating 30 percent is a very different system from one escalating 3 percent, even if both report a similar success rate.
• Cost per task. Average API cost and latency per completed task, for example $0.12 and 8 seconds. Without this number, a "smarter" model upgrade can quietly triple your bill for a marginal accuracy gain.

9. The AI Engineering Maturity Model

Most teams can place themselves on a scale from a weekend experiment to a fully autonomous platform. Knowing which level you are actually on, not which level the demo suggests, is the honest starting point for deciding what to build next.

AI Engineering Maturity Model — six ascending levels, from a bare prompt experiment through a single agent, an agent with tools, an agent with state and monitoring, a full production AI system, and finally a self-improving platform.

• Level 0, Prompt Experiment: a single prompt tested manually, no code around it.
• Level 1, Single Agent: one agent, one loop, no external tools.
• Level 2, Agent + Tools: the agent can call APIs or execute code, still no persistent state.
• Level 3, Agent + State + Monitoring: progress survives a restart, and logs exist for debugging.
• Level 4, Production AI System: all five pillars are in place, state, observability, evaluation, recovery, and security.
• Level 5, Self-Improving AI Platform: the system uses production data and human feedback to improve its own evaluation and routing over time.

Most teams claiming to run "AI agents in production" are honestly somewhere around Level 2. That's not a failure. It's useful to know exactly where you stand before promising Level 4 reliability to a stakeholder.

10. Not Every Problem Needs an Agent

This is the balance most agent content skips. Plain software beats an agent when:

• The rules are fully deterministic and known in advance
• Output must be 100 percent predictable, not just usually correct
• Latency requirements are strict and a model call adds unacceptable delay
• Errors carry a cost that makes "mostly right" unacceptable

A payment calculation engine should not run through an agent. The math is deterministic, and a hallucinated cent is not an acceptable failure mode. A support ticket classifier deciding which queue a message belongs in is a reasonable fit for an agent, since the cost of an occasional wrong routing is low and easily corrected.

11. Pros and Cons: Agent-First Versus System-First Development

Approach	Pros	Cons
Agent-first (ship the loop fast)	Fast to prototype, cheap to validate an idea, good for internal tools and hackathons	Breaks under real load, hard to debug, no guardrails against costly mistakes
System-first (build the pillars early)	Stable under real usage, debuggable, safer for actions with financial or data consequences	Slower to first demo, higher upfront engineering cost, needs more planning

Neither approach is wrong on its own. The mistake is picking agent-first for a product that touches money, user data, or irreversible actions, and never circling back to add the missing pillars before real users arrive. A fair counterpoint exists too: in a market moving this fast, some teams reasonably choose to ship a thin agent and add the pillars once the product proves people want it, rather than over-engineering something nobody asked for. The reasonable middle ground most senior engineers land on is building thin versions of state, observability, and recovery from day one, even if formal evaluation waits until the product finds traction.

12. Practical Checklist

• Add a persistent state store for any task longer than a single model call
• Log every tool call, input, output, and error with a shared trace ID per task
• Cap all retries and add idempotency checks before any action with a real side effect, payments, emails, deletions
• Build a small eval set, 20 to 50 realistic scenarios, and run it on every prompt or model change
• Define an explicit fallback and human-handoff path for every external dependency
• Sandbox any agent with shell, file, or code execution access before granting write permissions
• Add a permission layer and audit log before an agent touches production data or money
• Decide up front which actions the agent takes alone, which need approval, and which it should never take

Alert: Treat any agent with write access to money, user data, or infrastructure as production software from day one, even during prototyping. The line between "just testing" and "this touched a real customer" is thinner than it looks, as the refund bot and Air Canada cases above both show.

Where This Leaves You

Three questions separate a prototype from a system people can depend on:

• Prototype: can the model do the task?
• Production: can the system do the task reliably?
• Scale: can the system do the task safely, millions of times?

The agent boom produced genuinely useful capability. Models got better at multi-step reasoning, tool use matured, and frameworks made state management far more accessible than it was two years ago. None of that is the problem.

The problem is treating a working demo as a finished product. The teams whose agents survive contact with real users are the ones who worked through state, observability, evaluation, recovery, and security before scaling up, not after an incident forced the issue.

Building an agent gets you attention. Building a system gets you a product people can depend on.