I've spent 9 years deploying services on Kubernetes. Last year, my team started deploying AI agents to production. That's when I realized everything I knew about safe deployments was suddenly inadequate.
The Moment I Knew Something Was Wrong
It was a Thursday afternoon. We pushed a "minor update" to a customer-facing AI agent — a two-word change in the system prompt. No code changes. No dependency updates. The container image tag didn't even change.
Within an hour, the agent started hallucinating product features that didn't exist. Customer complaints spiked. We rolled back, but "rolling back" a prompt change in our setup meant SSH-ing into a config repo, reverting a YAML file, and waiting for the CI pipeline to redeploy — a 15-minute scramble that felt like an eternity.
The worst part? Our monitoring didn't catch it. Grafana showed green across the board — HTTP 200s, latency within SLA, CPU and memory nominal. Every signal we trusted for years told us everything was fine, while the agent was confidently lying to customers.
That's when it hit me: we were deploying AI agents like microservices, and it was fundamentally wrong.
The 60/70 Problem
I started talking to other platform engineers. The pattern was eerily consistent.
At a recent meetup, I asked: "How do you deploy your agents?" The most common answers were "GitHub Actions to K8s" and "docker compose is enough." A ZenML survey confirmed this — 60-70% of teams deploy agents like any other service.
Then I asked: "How do you know if a new version is better than the old one?" Silence. Nervous laughter. "We... watch Slack for complaints?"
This isn't a tooling gap. It's a conceptual gap. We're applying a deployment paradigm designed for deterministic, stateless services to something fundamentally different — and the cracks are showing. A Cleanlab survey of 1,837 practitioners found that 70% of regulated enterprises rebuild their agent stack every three months. Not iterate — rebuild.
What Makes Agents Different (It's Not What You Think)
When I first encountered this problem, I assumed it was about AI being "harder" in some vague sense. But after months of digging, I found the differences are structural and specific. There are exactly four things that break your existing deployment playbook.
1. Four Layers Change Simultaneously
A microservice has one version identity: the container image tag. Change the code, bump the tag, deploy. Simple.
An agent's behavior depends on four interdependent layers:
- Prompt / system context — the instructions that define the agent's personality and rules
- Model version — the LLM being called (GPT-4o vs Claude Sonnet vs a fine-tuned variant)
- Tool configurations — which external tools the agent can call and their API versions
- Memory / state — accumulated conversation history and learned patterns
Change any one of these, and behavior shifts unpredictably. Change two simultaneously, and you have a combinatorial explosion of possible behaviors that no amount of pre-deployment testing can cover.
Here's the kicker: your Git history won't tell you what changed. A prompt update is a config change, not a code change. A model version swap is an environment variable change. A tool API upgrade happens upstream, outside your control entirely. Your deployment pipeline sees "nothing changed" while the agent's behavior has transformed.
Practitioners report that tool versioning causes 60% of production agent failures and model drift causes 40%. These are failure modes that don't exist in traditional software deployment.
2. You Can't Unit Test Non-Determinism
If I deploy a REST API, I can assert: "Given input X, output should be Y." If the test passes, I have reasonable confidence the service works.
With an agent, the same input can produce different outputs every time. The agent might call different tools, follow different reasoning paths, or generate slightly different phrasings — and all of these might be "correct." Traditional pass/fail assertions are meaningless.
Anthropic's engineering team put it clearly in their agent evaluation guide: you need multiple trials per task, with code-based, model-based, and human graders. It's a fundamentally different testing paradigm — closer to A/B testing in product development than to unit testing in software engineering.
But here's the operational problem: if your evaluation takes 20 minutes to set up, developers won't test. The AWS DevOps Agent team discovered this firsthand. You need evaluation infrastructure that's as easy as kubectl apply — integrated into the deployment pipeline, not bolted on after the fact.
3. Cost Is a Deployment Variable
When I deploy a microservice, I know roughly what it costs per request — CPU time is predictable, memory is bounded. I can capacity plan.
An agent's cost per request varies wildly. A simple query might consume 500 tokens. A complex multi-step task might trigger 15 tool calls and consume 50,000 tokens — a 100x difference. One startup spent €5,000 in compute to determine optimal email send times — a task solvable with 50 lines of business logic.
This means a "successful" deployment — one where the agent works correctly — can still bankrupt you if the new version's reasoning patterns are more token-hungry. Cost needs to be a first-class rollback signal, not an afterthought in your monthly cloud bill review.
4. Rollback Is Structurally Harder
Rolling back a microservice means pointing traffic to the previous container. Done.
Rolling back a stateful agent means undoing actions it already took. If the agent sent emails, created Jira tickets, updated a CRM, or modified a database — those side effects persist. You can revert the container, but you can't unsend the email.
This makes the prevention of bad deployments — catching problems during canary before they reach 100% of traffic — far more critical for agents than for traditional services.
The Missing Layer
Here's the thing: excellent tools exist for every other part of the agent lifecycle.
Building agents? LangGraph, CrewAI, OpenAI Agents SDK — mature frameworks with active communities.
Serving models? vLLM, Ray Serve, BentoML — production-hardened inference engines.
Observing agents? Langfuse, Arize Phoenix, LangSmith — comprehensive tracing and evaluation.
Running on Kubernetes? Kagent is bringing agent CRDs to the CNCF ecosystem.
But between "I built an agent" and "it's reliably running in production with safe, continuous updates" — there's nothing. No open-source tool treats agents as first-class deployable units with evaluation-gated progressive delivery.
The agent framework authors know deployment is hard. They explicitly punt on it — LangGraph requires a commercial license for production deployment, CrewAI offers a managed cloud, OpenAI's SDK has no hosting story at all. They assume you'll solve deployment yourself.
Meanwhile, Argo Rollouts provides beautiful progressive delivery for Kubernetes workloads — canary deployments, blue-green switches, automated analysis. But it doesn't understand agent health. It can check HTTP error rates, but it can't check hallucination rates. It can measure latency, but not cost-per-task. It has no concept of prompt versions or model versions.
What the Right Solution Looks Like
After living with this problem for months, I believe the right approach has three principles:
1. Composite version tracking. An agent's version isn't just a container tag — it's the combination of prompt + model + tools + code. The deployment system needs to track all four layers as a single versioned entity. When you do a canary rollout, you need to know: "stable is running prompt v1 with Claude 3.5, canary is running prompt v2 with Claude 4."
2. Evaluation-gated progressive delivery. Instead of checking HTTP 200 rates during canary, check agent-specific quality signals: hallucination rate, tool call success rate, cost-per-task, task completion rate. If quality degrades at 5% traffic, auto-rollback before it hits 100%.
3. Framework-agnostic, Kubernetes-native. It shouldn't matter if your agent is built with LangGraph, CrewAI, or custom Python. Once it's a container, the deployment infrastructure should work the same. And it should be built on top of proven Kubernetes primitives — not another proprietary platform.
Introducing AgentRoll
This is why I started building AgentRoll — an open-source Kubernetes operator that brings evaluation-gated progressive delivery to AI agent deployments.
Instead of writing raw Argo Rollouts YAML and figuring out AnalysisTemplates from scratch, you declare an AgentDeployment:
apiVersion: agentroll.dev/v1alpha1
kind: AgentDeployment
metadata:
name: customer-support-agent
spec:
container:
image: myregistry/support-agent:v2.1.0
agentMeta:
promptVersion: "abc123"
modelVersion: "claude-sonnet-4-20250514"
toolDependencies:
- name: crm-mcp-server
version: ">=1.2.0"
rollout:
strategy: canary
steps:
- setWeight: 5
pause: { duration: "5m" }
analysis: { templateRef: agent-quality-check }
- setWeight: 20
pause: { duration: "10m" }
analysis: { templateRef: agent-quality-check }
- setWeight: 100
Under the hood, AgentRoll:
- Creates an Argo Rollout (not a raw Deployment) with properly translated canary steps
- Manages AnalysisTemplates with agent-specific quality checks — or lets you bring your own
- Tracks the composite version (prompt + model + image tag) as labels on every pod
- Provides opinionated defaults while allowing full customization at every layer
It's early — we're in alpha. But the core loop works: AgentDeployment → Argo Rollout with canary strategy → evaluation-gated promotion → automatic rollback on failure.
What's Next
The roadmap is straightforward:
- Real evaluation metrics: Replace placeholder analysis with Langfuse/Prometheus integration for actual hallucination rate, tool success rate, and cost-per-task tracking
- Cost-aware scaling: KEDA-based autoscaling using queue depth instead of CPU (agents are I/O bound, not CPU bound)
- MCP tool lifecycle: Manage MCP tool server versions alongside agent versions
- Multi-agent coordination: Coordinated canary deployments across dependent agent networks
If you're a platform engineer deploying AI agents and feeling the pain I described — I'd love to hear your story. The gap between "agents work in my notebook" and "agents are reliably serving production traffic" is real, and I believe it's solvable with the right infrastructure primitives.
Star the repo if this resonates. Open an issue if you have a use case I haven't considered. The best infrastructure is built by the people who feel the pain.
Max Li is a software infrastructure engineer with 9 years of experience in Kubernetes, Helm, Terraform, and cloud-native systems. He's building AgentRoll because he believes AI agents deserve the same deployment rigor as microservices.
Top comments (0)