The uncomfortable truth about AI agents
By the time most teams reach this stage, they’ve already built:
• a multi-step workflow
• a supervisor + worker setup
• integration with tools and APIs
And yet, the system still fails in production.
Not because the model is weak.
But because the system is non-deterministic.
⸻
Where reliability actually breaks
In real deployments, failures don’t come from “bad reasoning”.
They come from:
• malformed outputs (invalid JSON, missing fields)
• inconsistent decisions across steps
• uncontrolled retries and loops
• unsafe or duplicated side effects
You can’t patch these with better prompts.
You need contracts, validation, and control layers.
⸻
From probabilistic outputs → deterministic contracts
The first shift is simple but critical:
Treat every model output as untrusted input
Instead of accepting free-form text, define strict schemas using
Pydantic or
PydanticAI.
⸻
Example: Root Cause Contract
class RootCause(BaseModel):
service: str
confidence: float
error_type: Literal["OOM", "MemoryLeak", "Config", "Network"]
evidence: list[str]
next_steps: list[str]
This does three things:
1. Forces the model into a structured format
2. Enables automatic validation
3. Creates a stable interface between system components
⸻
What this looks like in practice
A production pipeline becomes:
LLM Output → Schema Validation → Accept / Reject → Retry / Escalate
This is no longer “AI responding”.
It’s a controlled data pipeline.
⸻
The self-healing loop
Validation is only half the system.
The real reliability comes from how you handle failure.
⸻
Controlled retry pattern
1. Generate output
2. Validate against schema
3. Capture validation error
4. Feed error back into model
5. Retry with constraints
6. Stop after N attempts
⸻
Example failure feedback
Instead of:
“Try again”
You send:
“Field confidence must be a float between 0 and 1.
error_type must be one of [OOM, MemoryLeak, Config, Network].
Fix the JSON.”
This transforms the model into a self-correcting system.
⸻
Why Gemma 4 fits this model well
With Gemma 4, this loop becomes practical at scale.
Because:
• thinking mode improves structured reasoning
• MoE architecture reduces cost per retry
• long context allows passing validation history
• tool calling aligns with structured outputs
This is critical.
Self-healing systems require multiple attempts.
Cost-efficient inference makes that viable.
⸻
Guardrails are not optional
Without guardrails, your system will eventually:
• loop indefinitely
• call the wrong tools
• execute unsafe actions
⸻
Minimum guardrail layer
You should implement:
Step limits
• Hard cap on number of node executionsError classification
• Retry: timeouts, rate limits
• Fail: schema errors, auth issuesCircuit breakers
• Stop calling failing dependenciesHuman-in-the-loop
• Required for destructive actions
⸻
Visualizing guardrails in the system
Think of your system as:
State Machine
↓
Validation Layer
↓
Guardrails
↓
Execution
Each layer reduces uncertainty.
⸻
Going beyond validation: adaptive systems with DSPy
Validation ensures correctness.
But how do you improve the system over time?
⸻
Enter DSPy
DSPy treats your pipeline as a program:
• inputs → outputs
• defined signatures
• measurable metrics
It allows you to:
• run evaluation datasets
• measure output quality
• optimize prompts automatically
⸻
What this unlocks
Instead of manual tuning:
• the system detects failures
• adjusts prompts / examples
• improves over time
This is the missing layer in most agent systems.
⸻
Combining everything: the deterministic stack
A production-ready Gemma 4 system looks like:
State Graph (LangGraph)
↓
Supervisor (Gemma 4 thinking mode)
↓
Workers (task-specific agents)
↓
Pydantic Validation
↓
Guardrails
↓
DSPy Evaluation + Optimization
Each layer solves a specific failure mode.
⸻
Real-world application: autonomous DevOps agent
Example workflow:
Trace
• collect logs, metrics, events
RootCause
• detect anomalies (OOMKilled, memory leaks)
Plan
• decide corrective action
Fix
• restart pods, scale services, or open PR
Verify
• confirm system recovery
⸻
Why this works
Because:
• every step is validated
• every action is controlled
• every failure is recoverable
This is not an “AI agent”.
It’s a deterministic system with AI inside it.
⸻
Practical implementation stack
If you’re building this today:
• Model: Gemma 4 (26B MoE)
• Orchestration: LangGraph
• Validation: Pydantic / PydanticAI
• Guardrails: custom + middleware
• Evaluation: DSPy
⸻
Resources
Core
• https://github.com/google-deepmind/gemma
• https://github.com/google/gemma_pytorch
Orchestration
• https://github.com/langchain-ai/langgraph
• https://github.com/langchain-ai/langgraph-example
Validation & Guardrails
• https://github.com/pydantic/pydantic-ai
• https://github.com/jagreehal/pydantic-ai-guardrails
Evaluation & Optimization
• https://github.com/stanfordnlp/dspy
• https://github.com/Scale3-Labs/dspy-examples
Real-world systems
• https://github.com/qicesun/SRE-Agent-App
⸻
Final perspective
Most teams are still chasing:
• better prompts
• better models
• better outputs
That’s not where reliability comes from.
⸻
Reliability comes from:
• explicit state
• strict contracts
• controlled execution
• continuous evaluation
Top comments (2)
Has Gemma 4 been able to things with this harness that it was not able to do without it, and if so what?
Yes with a harness, gemma 4 can be much better at agentic/tool-use and reproducible evaluation than when you just prompt it directly.