AI Agents Explained for Backend Engineers

Introduction

AI agents are not intelligent. They are persistent.

An agent is a loop: the LLM thinks, picks a tool, executes it, reads the result, and thinks again. It continues until the task completes, a stop condition fires, or your budget runs out.

There is no built-in planning module. No memory of past failures unless you add it. No self-correction unless you engineer it. It is a while-loop wrapped around an API call.

That is what makes agents powerful and dangerous. They keep trying. They burn tokens. They call the wrong tool twice. They loop forever if you do not set exit conditions.

Why This Matters

Backend engineers know that unbounded loops are production incidents waiting to happen. Agents are unbounded loops with a non-deterministic decision function.

A runaway agent is not a research curiosity. It is a line item on your cloud bill and a reliability risk for downstream systems. Persistence without guardrails is an expensive infinite loop.

Prerequisites

This article assumes you have read Blog 001, Blog 002, and Blog 003. You should understand LLM inference, RAG for grounding, and MCP for standardized tool access.

The Problem

Teams ship agents by wrapping an LLM in a ReAct-style prompt and calling it done. Production breaks when:

1. No iteration cap: Agent calls tools indefinitely.
2. No token budget: A single user request costs dollars in API fees.
3. No timeout: Hung tool calls block the loop.
4. No idempotency: Retried tool calls double-charge or duplicate writes.
5. No human gate for irreversible actions (refunds, deletes, deployments).

Understanding the Core Concept

The agent loop

while not done:
    response = llm(messages, tools)
    if response.has_tool_call:
        result = execute_tool(response.tool_call)
        messages.append(result)
    else:
        return response.text

The LLM decides what to do next based on conversation history and tool results. Quality depends on prompts, tools, and guardrails, not on hidden reasoning.

Agent vs single LLM call

Aspect	Single LLM call	Agent loop
Latency	One round trip	Multiple round trips
Cost	Fixed per call	Scales with iterations
Capability	Text in, text out	Can act on external systems
Failure modes	Bad output	Bad output plus bad actions
Observability	Simple	Requires per-step tracing

Use an agent when the task requires multiple steps, external data, or actions. Use a single call when retrieval plus generation suffices.

Harness responsibilities

The harness (your code around the model) must provide:

• Termination policy: max iterations, max tokens, wall-clock timeout
• Tool dispatch: schema validation, auth, retries
• State management: conversation history, scratchpad, episodic memory
• Error handling: surface tool errors to the model or escalate
• Human-in-the-loop: approval for high-risk operations

How It Works Internally (High Level)

1. User submits a goal.
2. Harness assembles system prompt, tool catalog, and user message.
3. LLM returns either a final answer or a tool call.
4. If tool call: harness validates, executes via MCP or direct API, appends result.
5. Loop until final answer or stop condition.
6. Harness logs full trace for debugging and billing.

Step-by-Step Example

Task: "Find the latest error rate for service checkout and post a summary to #incidents."

1. Iteration 1: Model calls metrics_query(service="checkout", window="1h").
2. Harness executes, returns error_rate: 4.2%.
3. Iteration 2: Model calls slack_post(channel="#incidents", message="...").
4. Harness checks approval policy, posts message.
5. Iteration 3: Model returns "Posted summary to #incidents."
6. Harness terminates. Total: 3 LLM calls, 2 tool calls.

Without a max iteration limit, a confused model might query metrics repeatedly.

Architecture

Exit conditions on the loop are not optional.

Python Example

Minimal agent harness with iteration cap and token budget.

"""
Minimal agent harness with guardrails.
"""
import json
from dataclasses import dataclass, field

MAX_ITERATIONS = 5
MAX_TOTAL_CHARS = 8000

@dataclass
class AgentState:
    messages: list[dict] = field(default_factory=list)
    iterations: int = 0
    total_chars: int = 0

def mock_llm(messages: list[dict], tools: list[dict]) -> dict:
    last = messages[-1]["content"] if messages else ""
    if "error rate" in last.lower() and not any("metrics" in str(m) for m in messages):
        return {
            "type": "tool_call",
            "name": "metrics_query",
            "arguments": {"service": "checkout", "window": "1h"},
        }
    if any("error_rate" in str(m) for m in messages):
        return {"type": "text", "content": "Checkout error rate is elevated at 4.2%."}
    return {"type": "text", "content": "I need more information."}

TOOLS = {
    "metrics_query": lambda args: {"error_rate": "4.2%", "service": args["service"]},
}

def run_agent(user_message: str) -> str:
    state = AgentState(messages=[{"role": "user", "content": user_message}])

    while state.iterations < MAX_ITERATIONS:
        state.iterations += 1
        response = mock_llm(state.messages, list(TOOLS.keys()))

        if response["type"] == "tool_call":
            name = response["name"]
            result = TOOLS[name](response["arguments"])
            state.messages.append({"role": "tool", "content": json.dumps(result)})
            state.total_chars += len(json.dumps(result))
        else:
            return response["content"]

        if state.total_chars > MAX_TOTAL_CHARS:
            return "Error: token budget exceeded"

    return "Error: max iterations exceeded"

if __name__ == "__main__":
    print(run_agent("What is the checkout error rate?"))

Replace mock_llm with your provider client. Add structured logging per iteration.

Real-World Applications

• Code agents that read files, run tests, and open pull requests
• Support agents that query CRM, search docs (RAG), and draft replies
• Data agents that generate SQL, execute read-only queries, and summarize
• DevOps agents that inspect logs and trigger approved runbooks

Performance Considerations

• Latency: Each iteration adds a full LLM round trip plus tool execution time.
• Cost: Token usage grows with history length. Summarize or prune old tool results.
• Concurrency: Multiple agents sharing tools need per-tenant rate limits.
• Reliability: Tool failures should be explicit in context, not swallowed.

Common Mistakes

1. No max iterations or timeout.
2. Giving agents write access without approval workflows.
3. Passing entire tool outputs into context without truncation.
4. Assuming the model will self-correct after repeated failures.
5. No distributed trace ID across LLM and tool calls.

Interview Questions

Q1: What is an AI agent in production terms?

A: A loop where an LLM repeatedly decides to call tools or return a final answer until a harness stop condition fires.

Q2: Why are agents expensive?

A: Each iteration is a full LLM inference call plus tool execution, with growing conversation history.

Q3: What guardrails are mandatory?

A: Max iterations, timeout, token budget, schema-validated tool calls, and human approval for irreversible actions.

Q4: Agent vs RAG: when to use which?

A: RAG grounds answers in documents. Agents take multi-step actions. Many systems use both.

Q5: What is the harness?

A: Your application code that manages the loop, tools, state, policies, and observability around the LLM.

Q6: How do you debug a bad agent outcome?

A: Inspect the per-iteration trace: which tools were called, with what arguments, and what the model saw at each step.

Termination Policies

Define explicit exit conditions in code, not in prompts alone:

Stop reason	Trigger	User experience
success	Model returns final answer	Normal completion
max_iterations	Loop count exceeded	Partial result plus explanation
budget_exceeded	Token or cost cap hit	Graceful degradation message
timeout	Wall clock limit	Retry suggestion
human_required	High-risk tool pending	Escalation UI
tool_failure_limit	N consecutive tool errors	Stop and log incident

Prompts that say "stop when done" are insufficient. Models do not reliably self-terminate under ambiguity.

State management options

Ephemeral: Full history in memory per request. Simple, no cross-session memory.

Session store: Redis or DB keyed by session ID. Required for multi-turn agents.

Scratchpad: Separate channel for intermediate reasoning and tool JSON, trimmed before user display.

Summarized memory: Compress turns older than K into a rolling summary to save context (Blog 012).

Cost model for agents

Approximate cost per task:

cost ≈ sum(iteration_i prompt_tokens + completion_tokens) × price_per_token

A five-iteration agent with 4K context per iteration is not five times a single 4K call if history grows each loop. Cap tool result size aggressively.

ReAct and Tool-Calling Patterns

ReAct interleaves reasoning text with tool calls in the transcript. The model emits natural language planning, then a tool invocation, then observes results.

Production tip: separate user-visible messages from internal tool traces. Users do not need raw JSON blobs; they need summaries.

Parallel vs sequential tools

Some APIs allow parallel tool calls. Rules:

• Parallelize only independent reads
• Serialize writes that touch the same resource
• Define merge order when results conflict

Human-in-the-loop placement

Insert approval gates before:

• Financial transactions
• Data deletion
• External emails
• Production config changes

Return a pending state to the UI instead of blocking inside the model loop indefinitely.

Testing Agents

Unit test tool dispatch without the LLM:

• Given mocked model output tool_call X, assert handler X runs with validated args.

Integration test with frozen model responses (record/replay) for deterministic CI.

Eval in staging with real models on golden tasks measuring success rate, average iterations, and cost per task.

Observability for Agent Traces

Structure logs as an ordered trace:

{
  "trace_id": "abc",
  "steps": [
    {"type": "llm", "tokens_in": 1200, "tokens_out": 45},
    {"type": "tool", "name": "search_docs", "latency_ms": 120},
    {"type": "llm", "tokens_in": 1400, "tokens_out": 200}
  ],
  "outcome": "success",
  "total_cost_usd": 0.004
}

Dashboards: success rate, avg steps, cost per task, tool error rate.

When Not to Use an Agent

Use workflow code when:

• Steps are fixed (always fetch invoice, then email)
• Logic is deterministic if data is known
• Latency budget is tight

Use agents when:

• Path depends on intermediate results
• Tool set is large and choice is context-dependent
• Exploration is required (within guardrails)

Over-agentifying simple pipelines adds cost and failure modes without benefit.

Incident Response Playbook

Symptom: Token spend 10x overnight.

Check: Agent traces for loops calling same tool, growing history, missing max_iterations.

Mitigation: Lower iteration cap globally, disable expensive tools via feature flag, hotfix harness.

Symptom: Wrong production data mutated.

Check: Write tools missing approval gate, idempotency, or staging environment separation.

Mitigation: Freeze write tools, require human approval, audit last 24h tool calls.

Capacity Planning

Estimate peak concurrent agents as:

concurrent_agents × avg_iterations × avg_tokens_per_call

Compare to provider TPM/RPM limits. Queue or shed load before hard failures.

Comparison: Workflow Engine vs Agent

Workflow engine	Agent
Fixed DAG	Dynamic tool choice
Predictable cost	Variable cost
Easy testing	Requires eval traces
Best for known processes	Best for exploratory tasks

Use Temporal or Step Functions when steps are known. Use agents when the path is data-dependent. Many products need both: workflow invokes agent only on fallback branch.

Reference Appendix: Production FAQ

How do I know this is working in production?

Instrument the layer this article describes before changing models or prompts. Compare p50 and p95 latency, error rate, and task-specific quality scores week over week. AI regressions are subtle: flat aggregate uptime can hide wrong answers.

What is the first config change to try?

Reduce variability before increasing capability. Lower temperature for factual paths, shrink retrieval top-K, tighten context budgets, add output validation. Complexity is not a substitute for measurement.

What belongs in an on-call runbook?

Symptom, dashboard link, rollback lever (model version, feature flag, index snapshot), owner team, and customer communication template. LLM incidents need content rollback, not only service restart.

How do I explain tradeoffs to product managers?

Use dollars and seconds: cost per successful task, p95 time to first token, accuracy on golden set. Avoid debating model intelligence; debate measurable user outcomes and failure tolerance.

When should we retrain, re-index, or rewrite prompts?

Re-index when documents change. Rewrite prompts when behavior spec changes. Retrain or fine-tune when prompt plus RAG cannot meet format or tone requirements after eval iteration. Default order: prompt, RAG, fine-tune.

What is the common rollback path?

Keep previous model version, previous index snapshot, and previous prompt template addressable by version id for at least seven days. Rollback should be one feature flag or deploy revert, not a fire drill.

How does this interact with the rest of the handbook?

This topic is one layer in a stack. Read prerequisites listed in frontmatter. When debugging end-to-end failures, walk the request path from ingress through retrieval, inference, and output validation before concluding the model is wrong.

Summary

Agents are persistent loops, not autonomous minds. Engineering value is in the harness: termination, budgets, tool governance, and observability. Make agents stop safely before you make them smarter.

Further Reading

• ReAct paper (Reasoning and Acting)
• MCP specification for tool standardization (Blog 003)
• OpenAI function calling and tool use documentation

Next in Series

Blog 005: Fine-Tuning LLMs: Behavior vs Knowledge and When to Use It