Why your AI agent works in the notebook and breaks in production

#devops #ai #machinelearning #automation

Every team I talk to has the same story.

The LangChain prototype works. The demo
impresses the stakeholders. Then someone
asks when it ships -- and the real work
starts.

Deployment pipelines. Security review.
Observability. Governance. Six months of
infrastructure your team has to build
before a single user sees the agent.

Here is the part nobody talks about:

AI agents show 63% variation in execution
paths for identical inputs. Your unit tests
are not broken. Unit testing just does not
work for something that behaves differently
every single run.

Traditional DevOps was built for
deterministic systems. Agents are not
deterministic.

What you actually need:

BEHAVIORAL TESTING not unit testing
Test what the agent does across 100 runs
not whether the function returns the
expected value once.
COMPOUND RELIABILITY monitoring
10 agents at 95% reliability each =
60% system reliability overall.
Nobody's monitoring that.
AGENT IDENTITY
Every agent needs an ID, an owner, a
version history. Right now most teams
are running anonymous scripts in
production with no audit trail.
GOVERNANCE before launch not after
SOC 2 review takes 3-6 months if you
bolt it on. It takes 0 days if it is
built in.
COST ATTRIBUTION per agent per run
Not per session. Sessions are the wrong
unit in multi-agent systems. You need
token cost, tool call cost, and hop cost
attributed to each specific agent.