Designing Smart AI Agents: Architecture Patterns That Survive Production

A practical field guide to agent topologies, state design, and the failure modes that separate a demo from a system that survives 90 days in production.

Six months ago, a logistics company in Dubai flew me in to look at their "autonomous customer operations" pilot. The vendor demo was stunning. An agent quoted delivery timelines, resolved address discrepancies, and flagged high-risk shipments for human review. In the controlled demo, it resolved 94% of test cases without a human in the loop.

I asked for the production numbers. A pause. Then the CTO pulled up a dashboard. On real traffic — about 4,000 tickets a day, messy addresses, late tracking feeds, angry customers — the same agent resolved 11% of cases, and it hallucinated a delivery promise onto at least a dozen of the rest. Some of those promises cost the company real money in refunds and re-shipping.

The demo was not fake. The model was fine. The problem was that nobody had designed the system's architecture. They had pointed a capable model at a prompt, wrapped it in a loop, and called it an agent.

This article is the pattern language I wish that vendor had used: the topologies, the state design, the tool contracts, and the failure modes that decide whether an agent survives production. By the end you will be able to look at any agent project, name the pattern it is using, and — more importantly — say whether it is the right one.

First, Kill the Word "Agent" — Talk About Topology

"Agent" is a marketing word. "Topology" is an engineering word. A topology is the shape of your system: how many reasoning loops exist, how they talk to each other, who owns the state, and who decides what happens next. When a production agent collapses, it is almost never the model's fault. It is a topology that did not match the task.

Every agent architecture, no matter how clever the slideware, is one of six topologies. Learn to spot them, because each has a cost profile, a failure mode, and a narrow range of tasks it is genuinely good at.

1. The Single Loop

One model, one context window, a set of tools, a while loop. This is the default and the workhorse. The system prompt holds the goal, the context window holds working state, tools give it hands, and budget counters stop it from running forever.

Cost: lowest. Latency: lowest. Good for: narrow, well-scoped tasks — balance lookups, form extraction, single-domain Q&A with tools.

2. The Router

A small, fast model classifies the request and dispatches it to one of several specialized handlers. A ticket-triaging router sends payment disputes to a refund workflow, delivery questions to a tracking tool, and everything else to a general agent.

Cost: low. Latency: adds one cheap call. Good for: high-volume traffic where most requests are one of a few known shapes. This is the most underrated pattern in production, and the one I reach for first.

3. Orchestrator–Worker

One orchestrator decomposes a task into subtasks and hands each to a worker agent (or a plain function, or a search job). Workers return results; the orchestrator synthesizes. This is what people actually mean when they say "multi-agent," and most of the time it is one orchestrator with several specialized workers.

Cost: medium. Latency: medium. Good for: report generation, research, code review — tasks with a natural breakdown.

4. Hierarchical

Agents manage agents. A lead orchestrator spawns sub-orchestrators, each managing its own workers. This is how you scale orchestrator–worker to genuinely huge tasks, and it is also where complexity and cost start to compound.

Cost: high. Latency: high. Good for: enterprise research pipelines with thousands of documents. Usually a mistake for anything pattern 3 handles.

5. Peer Team

Multiple agents with equal standing converse or work in parallel toward a shared goal — the classic CrewAI and AutoGen picture: a researcher, a writer, and a critic arguing over a document until they agree.

Cost: high — every peer turn is a full model call and coordination overhead is real. Latency: high. Good for: creative drafting and debate-style tasks. Bad for: anything with a deadline and a strict budget.

6. The State Machine (Workflow)

No loop at all. A directed graph of steps — query, validate, charge, confirm — where each step is deterministic or model-assisted. LangGraph's graph model and n8n's node model are this pattern wearing graph paper.

Cost: lowest per step. Latency: predictable. Good for: anything that is 80% a known process with a few fuzzy decision points. Most "agent" use cases are secretly this, and it is the most honest pattern in the list.

Here is the quick reference I put in front of clients:

Pattern	Loop?	Cost	Failure mode	Best for
Single loop	yes	low	context creep	narrow, scoped tasks
Router	no	low	bad classifier	high-volume triage
Orchestrator–worker	yes	medium	handoff context loss	research, reports
Hierarchical	yes	high	exponential cost	huge decompositions
Peer team	yes	high	coordination chatter	drafting, debate
State machine	no	low	rigid on exceptions	known processes

The most useful question I ask before writing any code: is this task a process with a few judgment calls, or an open-ended goal with unknown steps? Process → state machine. Open-ended → single loop or orchestrator–worker. Almost never a peer team on day one.

State: The Part Everyone Forgets

Now the part that kills more production agents than any topology choice: state.

An agent's state is everything it carries between steps — the task definition, what it has already tried, what it has ruled out, the results of tool calls, and the budget it has left. If state lives only in the model's context window, you have a memory problem: context windows are bounded, noisy, and easy to poison. If state lives in your database, you have an engineering problem: every step needs a save, a load, and a version.

Here is the rule I now enforce with clients. Working state (what is on the model's desk right now) goes in the context window, trimmed ruthlessly. Durable state (what this task has accomplished, across retries and restarts) goes in a store — Postgres for structured task state, a vector store for retrieved knowledge, Redis for ephemeral job state. Every step is a pure function of durable state plus the model's decision. That one discipline — "state in the store, not in the prompt" — fixed more agent projects than any model upgrade I have ever shipped.

Concretely, a task row in Postgres looks like this:

CREATE TABLE agent_tasks (
  id            uuid PRIMARY KEY,
  pattern       text NOT NULL,             -- which topology
  goal          text NOT NULL,
  status        text NOT NULL DEFAULT 'queued',
  step_count    int  NOT NULL DEFAULT 0,
  tool_calls    jsonb NOT NULL DEFAULT '[]',
  result        jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

Every tool call is appended to tool_calls. If the process crashes, a worker picks up the row and replays from step_count. That is the entire secret of "reliable" agents: they are just jobs that can resume.

Tool Design: Descriptions Are Contracts

I keep saying tools are the agent's hands, but the part people get wrong is the description. The model reads your tool description and decides whether to use the tool. Write a lazy description and the model will misuse it in production, every single time.

Treat the description as a contract with three clauses:

1. What it does. "Fetches the current available balance for a verified account."
2. When to use it. "Call this when the customer asks about money they have or owe. Do not call it for transaction history — that is get_transactions."
3. What it returns. "Returns {balance: number}. Returns an error object if the account is not verified."

One more rule: validate inputs server-side before execution. The model's arguments are model output — they can be wrong, and in adversarial inputs they can be malicious. A SQL injection string smuggled through a tool argument is not a joke; it is a Tuesday.

A Working Orchestrator–Worker, Minimal

Here is the smallest orchestrator–worker I would ship, with the state discipline above. No framework — just Postgres, a queue, and two model calls per task.

import json
from typing import Any

def orchestrator(task: dict) -> str:
    plan = llm_call(
        "You are a research lead. Split this task into 3-5 subtasks "
        "that can be executed independently. Return JSON.",
        task["goal"],
    )
    subtasks = json.loads(plan)["subtasks"]

    results = []
    for sub in subtasks:
        results.append(worker(sub))           # worker may call tools
        save_task_state(task["id"], results)  # durable state every step

    return llm_call(
        "You are a synthesis editor. Combine these subtask results "
        "into one coherent answer for the original task.",
        json.dumps({"goal": task["goal"], "results": results}),
    )

def worker(subtask: dict) -> Any:
    # deterministic routing: one tool call, one model pass
    return run_tool(subtask["tool"], subtask["args"])

def save_task_state(task_id: str, results: list) -> None:
    # UPDATE agent_tasks SET tool_calls = $1 WHERE id = $2
    pass

Run this against real traffic and you will find the handoffs — the exact spots where context gets lost and tasks stall. That is the point: you want your failures in the handoff layer, because handoffs are cheap to instrument and cheap to fix. A hallucinated subtask decomposition, by contrast, is expensive to catch and expensive to repair.

Production Reality: The Failure Modes That Actually Hurt

After a year of shipping these systems across fintech, logistics, and support clients, here is my honest list of what breaks, ranked by how much it hurts:

1. Silent overreach. The agent does something you never authorized, confidently. It sends the email, applies the discount, closes the ticket. Fix: a permission layer — read-only tools are free; mutating tools require approval or a hard policy.
2. Context creep. Every step appends to the prompt, so by step 9 the model is reading a wall of its own noise. Fix: trim aggressively, summarize old steps, or move state to the store.
3. Handoff loss. In orchestrator–worker, the orchestrator re-sums what workers already spent tokens producing. Fix: have workers return structured JSON, not prose, and let the orchestrator only assemble.
4. Cost explosion. Peer teams and hierarchical patterns burn tokens at 5–10x a single loop. I costed a research task last month: single loop $0.18, orchestrator–worker $0.61, peer team $2.90 — for roughly the same output quality. Measure cost per resolved task, not per run.
5. Silent degradation. The model slowly stops calling tools and starts guessing from training data. Fix: monitor tool-call rate per pattern and alert when it drops below a threshold.

And the biggest one, which is not technical at all: the demo/test gap. Your evaluation set was curated by the same person who built the system. Measure success on held-out production traffic from week one, or you will discover your own 11% version in a client call, like the logistics company did.

When NOT to Use a Pattern

• Single loop, but your task is a known process? Use a state machine. You are paying for open-ended reasoning you do not need.
• Orchestrator–worker, but your subtasks cannot run independently? You have invented a slower single loop. Each worker waits on the previous one and the orchestrator just adds a hop.
• Peer team, but there is no genuine disagreement or complementarity between the roles? You are paying for theater.
• Any pattern, but a deterministic script would do? It would. This year I told a client their "agent" for parsing an invoice format that never changes was really a 40-line parser. The parser cost a fraction of a cent per run. The agent cost four cents and failed 3% of the time. I saved them a recurring bill by not building the architecture.

The Checklist I Use Before Shipping Any Agent

• [ ] The topology is named and justified against the task, not assumed ("multi-agent" is not an answer)
• [ ] Process-like tasks use a state machine; only open-ended tasks get a loop
• [ ] Durable state lives in a store; the context window holds only what the current step needs
• [ ] Tasks can resume from a crash (step_count plus a tool-call log)
• [ ] Tool descriptions specify what, when, and what they return
• [ ] Tool inputs are validated server-side before execution
• [ ] Mutating tools have a permission layer
• [ ] Cost per resolved task is measured and budgeted
• [ ] Tool-call rate is monitored as a health signal
• [ ] Evaluation runs on held-out production traffic from day one

What Survives Production

Back to the logistics company in Dubai. We rebuilt the pilot as a router plus a state machine: a cheap classifier sent tickets to one of five deterministic workflows, and only the genuinely fuzzy cases reached a single-loop agent with strict budgets and a permission layer. Resolution climbed from 11% to 78% in the first month — not because the model got better, but because the shape of the system finally matched the shape of the work.

The model was never the problem. The topology was.

Start by naming the pattern you are actually building. If you cannot name it, you do not have an architecture — you have a prompt with a cost. Draw the graph, put the state in the store, write the tool contracts, and treat the checklist above as your last step before production.

---

*Gulshan Yad