Your AI agent works on your laptop. You ran it a hundred times in development. It gives reasonable answers. Now someone wants to run it in production.
Here is the part of the AI agent tutorial nobody writes: what happens when the model returns something unexpected at 2am, when your observability dashboard costs $400/month, and when your test suite passes but your agent still does the wrong thing.
This is the production checklist I wish I had when I started shipping Python AI agents.
The three things that break in production (and how to test them before they do)
AI agents fail in patterns that don't appear in development:
- The model changes its output format — your code expects JSON, gets markdown
- Tool calls succeed but return garbage — function executes, result isn't what the agent expects
- Multi-step reasoning drifts — step 1 is fine, step 4 accumulates enough context noise to produce a wrong answer
Unit tests catch none of these. Standard mocks catch none of these. The testing patterns that matter for AI agents are different from the ones that matter for web apps.
Testing AI agents: what actually works
The core insight: you're not testing code, you're testing behavior under uncertainty.
This changes what you assert:
# Wrong: testing the exact response
def test_summarizer():
result = agent.run("Summarize this document")
assert result == "The document discusses..." # Fragile — breaks on any model update
# Right: testing behavioral properties
def test_summarizer():
result = agent.run("Summarize this document")
assert len(result) < len(ORIGINAL_DOCUMENT) # Compression happened
assert all(key_term in result for key_term in REQUIRED_TERMS) # Coverage check
assert not contains_pii(result) # Safety property
Pattern 1: Property-based assertions over exact string matching
Use pytest with property assertions rather than snapshot tests for LLM outputs:
import pytest
def contains_valid_json(text: str) -> bool:
import json
try:
json.loads(text)
return True
except json.JSONDecodeError:
return False
def test_structured_output_agent():
"""Agent must return parseable JSON with required keys."""
result = structured_agent.run("Extract entities from: 'Alice called Bob on Tuesday'")
assert contains_valid_json(result), "Output must be valid JSON"
parsed = json.loads(result)
assert "people" in parsed, "Must identify people"
assert "Alice" in parsed["people"] and "Bob" in parsed["people"]
assert "date_references" in parsed, "Must identify temporal references"
Pattern 2: Mock the LLM call, not the agent
The mistake most developers make: mocking at the wrong layer.
# Wrong: mock so deep you're not testing anything
with patch("my_agent.llm_client") as mock_llm:
mock_llm.return_value = "mocked response"
result = agent.run("Do something")
assert result == "mocked response" # You tested nothing
# Right: mock the API call, test the agent's behavior with controlled inputs
from unittest.mock import patch
import json
CONTROLLED_LLM_RESPONSE = json.dumps({
"action": "search",
"query": "Python async patterns",
"confidence": 0.92
})
def test_agent_routes_to_search_tool():
"""Agent should invoke search tool for information requests."""
with patch("anthropic.Anthropic.messages.create") as mock_create:
mock_create.return_value = make_mock_response(CONTROLLED_LLM_RESPONSE)
result = agent.run("What are the best async patterns for Python?")
# Verify routing, not content
assert mock_create.called
assert agent.last_tool_used == "search"
assert "Python async" in agent.last_search_query
The make_mock_response helper (and the fixtures for it) are exactly the kind of thing that's worth packaging — I put all of mine in a starter kit after rewriting them three times across different projects.
Pattern 3: Test tool call sequences, not just final outputs
Multi-step agents need sequence-level testing:
def test_research_agent_tool_sequence():
"""Research agent must search before summarizing — no hallucinated summaries."""
tool_calls = []
def capture_tool(tool_name, **kwargs):
tool_calls.append(tool_name)
return MOCK_TOOL_RESPONSES[tool_name]
with patch_tool_dispatcher(capture_tool):
result = research_agent.run("What happened at the MCP Dev Summit?")
# Sequence assertion: search must precede summarize
assert "web_search" in tool_calls, "Agent must search before answering"
assert tool_calls.index("web_search") < tool_calls.index("summarize"), \
"Search must happen before summarization"
assert len(tool_calls) <= 5, "Runaway tool calls indicate reasoning failure"
Observability without the SaaS contract
Most production observability recommendations end with "set up LangSmith" or "connect to Datadog." Both cost money and add external dependencies. Here's what you can run for $0 that covers 90% of what you need in production.
What you actually need to observe
Before setting up any tooling, decide what questions you need to answer:
| Question | Signal | Implementation |
|---|---|---|
| Is the agent producing correct outputs? | Output property checks logged | Structured logs + assert-on-write |
| Where is latency coming from? | Span timing per tool call | Python time.perf_counter() + structured logs |
| What inputs produce failures? | Input/output pairs at failure | Log on exception with context |
| Is cost growing unexpectedly? | Token counts per session | Log token usage from API response |
| Are tool calls succeeding? | Tool call success/failure rate | Decorator-level logging |
Structured logging without a SaaS
import logging
import json
import time
from functools import wraps
from typing import Any, Callable
# Single logger, JSON format for grep-ability
logging.basicConfig(
format='%(message)s',
level=logging.INFO,
handlers=[
logging.StreamHandler(),
logging.FileHandler('/var/log/agent/agent.jsonl') # One file per service
]
)
logger = logging.getLogger("agent")
def log_event(event_type: str, **kwargs):
"""Structured log event for agent observability."""
logger.info(json.dumps({
"ts": time.time(),
"event": event_type,
**kwargs
}))
def trace_tool_call(func: Callable) -> Callable:
"""Decorator: log every tool call with timing and success/failure."""
@wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
tool_name = func.__name__
try:
result = func(*args, **kwargs)
duration_ms = (time.perf_counter() - start) * 1000
log_event("tool_call",
tool=tool_name,
status="success",
duration_ms=round(duration_ms, 2),
input_preview=str(args[0])[:100] if args else None
)
return result
except Exception as e:
duration_ms = (time.perf_counter() - start) * 1000
log_event("tool_call",
tool=tool_name,
status="error",
error=str(e),
duration_ms=round(duration_ms, 2)
)
raise
return wrapper
# Usage
@trace_tool_call
def web_search(query: str) -> str:
# your search implementation
...
This gives you every tool call, timing, and failure reason in a queryable JSONL file. grep "error" agent.jsonl | jq . is surprisingly far when you're debugging production issues at 2am.
Token cost tracking
def log_llm_call(response, prompt_context: str = ""):
"""Extract and log token usage from Anthropic response."""
usage = response.usage
log_event("llm_call",
input_tokens=usage.input_tokens,
output_tokens=usage.output_tokens,
total_tokens=usage.input_tokens + usage.output_tokens,
# Anthropic Sonnet pricing: $3/$15 per 1M in/out
estimated_cost_usd=round(
(usage.input_tokens * 0.000003) + (usage.output_tokens * 0.000015),
6
),
context_preview=prompt_context[:50] if prompt_context else None
)
A daily cat agent.jsonl | grep llm_call | jq '.estimated_cost_usd' | awk '{sum+=$1} END {print sum}' gives you your daily spend without a dashboard.
What to alert on
With structured logs, you can write simple health checks:
# Simple health check script — run as a cron every 5 minutes
import json
import sys
from pathlib import Path
LOG_FILE = Path("/var/log/agent/agent.jsonl")
RECENT_MINUTES = 5
def check_agent_health():
cutoff = time.time() - (RECENT_MINUTES * 60)
recent_events = [
json.loads(line)
for line in LOG_FILE.read_text().splitlines()
if json.loads(line).get("ts", 0) > cutoff
]
errors = [e for e in recent_events if e.get("status") == "error"]
if errors:
error_rate = len(errors) / max(len(recent_events), 1)
if error_rate > 0.1: # 10% error rate
print(f"ALERT: {error_rate:.0%} error rate in last {RECENT_MINUTES}m")
sys.exit(1)
print(f"OK: {len(recent_events)} events, {len(errors)} errors in last {RECENT_MINUTES}m")
check_agent_health()
The deployment checklist
Before putting an AI agent in production:
Testing coverage:
- [ ] Property-based assertions for all LLM outputs (not exact string matching)
- [ ] Tool call sequence tests for multi-step agents
- [ ] Adversarial inputs: empty input, very long input, non-English input, injection attempts
- [ ] Retry behavior: what happens when the LLM returns malformed JSON?
- [ ] Cost ceiling test: does the agent ever loop indefinitely?
Observability:
- [ ] Structured logging on every LLM call (tokens, latency, context preview)
- [ ] Structured logging on every tool call (timing, success/failure)
- [ ] Log input/output pairs at failure time (for post-mortem debugging)
- [ ] Health check script or endpoint with error rate check
Operational:
- [ ] Max token budget per session (prevents runaway cost)
- [ ] Timeout on tool calls (prevents hanging on external services)
- [ ] Graceful degradation: what does the agent return when a tool fails?
- [ ] Rollback plan: how do you quickly revert if the model update breaks behavior?
What I packaged up
After building these patterns across several projects, I packaged the reusable pieces:
Pytest for AI Agents Starter Kit — The mock fixtures, property assertion helpers, and tool call sequence testers as drop-in pytest modules. Includes the conftest.py patterns, make_mock_response() for Anthropic and OpenAI, and a set of reusable property checkers for common LLM output patterns ($49 on Gumroad).
Python Agent Observability Toolkit — The structured logging setup, token cost tracker, and health check script as production-ready Python files. Drop the agent_logger.py into your project, add the @trace_tool_call decorator to your tools, and you're instrumented in under 10 minutes ($49 on Gumroad).
Both are one-time purchases. No ongoing SaaS subscription, no vendor lock-in, no telemetry sent anywhere.
The part nobody talks about
The hardest part of shipping AI agents to production isn't the LLM call. It's the surrounding infrastructure: tests that actually catch regressions, observability that tells you what broke, and cost controls that prevent surprises.
Most AI agent tutorials skip this because it's less exciting than the model call. But it's what separates a demo from a product.
The patterns above are enough to get you to a production-worthy baseline. The testing toolkit handles the fixtures so you're not rewriting them for every project. The observability toolkit handles the structured logging so you're not building it from scratch.
Build the agent. Then build the safety net around it.
If you've been following the Testing Without the Subscription Tax series (pytest fixtures → Hypothesis → async testing → pytest for AI agents), this is the capstone. The full testing stack from unit tests to production observability, without a subscription.
Have a production AI agent pattern I didn't cover? Drop it in the comments.
Top comments (0)