Originally published on tamiz.pro.
After the 96th failed deployment, I stopped asking the LLM to be more careful and started asking it to be honest about what it didn't know. The difference wasn't in the prompt—it was in the architecture around it.
Building reliable AI agents in production isn't a prompt engineering problem. It's a systems engineering problem that happens to use stochastic components. After shipping agents that handle customer support, data extraction, and workflow automation for enterprises, here's what the failure log taught me that no tutorial covered.
The 96 Failures Were Mostly the Same Failure
Early in development, each failure felt unique. The agent hallucinated a policy that didn't exist. It refused a legitimate request due to overzealous safety filtering. It got trapped in a tool-calling loop. It produced correct output but attributed it to the wrong source document.
By iteration 40, the pattern was clear: most failures weren't caused by the model's capabilities—they were caused by missing constraints and poor feedback channels.
The agent wasn't failing because it was stupid. It was failing because we had built a system that couldn't distinguish between "I don't know" and "I'm confident but wrong." That distinction is everything in production.
Lesson 1: Explicit Uncertainty Beats Implicit Confidence
Our first version had the agent respond to every query, even when it lacked sufficient information. The model would confidently generate a plausible-sounding answer, which looked good in dev but caused real problems in production.
The fix wasn't stronger prompting. It was adding an explicit uncertainty gate:
async def should_respond(response: AgentResponse) -> bool:
if response.confidence < UNCERTAINTY_THRESHOLD:
return False
if response.missing_context:
return False
return True
We measured uncertainty using a combination of token-level entropy and self-consistency checks (running the same query multiple times and measuring output variance). The threshold wasn't arbitrary—it was calibrated against human judgment on a held-out validation set.
This single change eliminated 60% of our production incidents. The agent learned to say "I need more information" instead of inventing an answer.
Lesson 2: Tool Calling Needs Fallback Chains, Not Retry Loops
Agents that call external tools (APIs, databases, functions) often get stuck in infinite retry loops when a tool fails. Our agent would retry the same REST call three times, fail three times, then produce a garbage response because it had exhausted its budget.
The solution was a fallback chain with explicit failure modes:
tool_fallbacks:
- primary: customer_api.search
timeout: 30s
retries: 2
fallback: cache.lookup
- primary: cache.lookup
timeout: 5s
retries: 0
fallback: graceful_degradation.fallback_message
- primary: graceful_degradation.fallback_message
action: respond_with_template
Each fallback has its own timeout and retry budget. The agent doesn't silently degrade—it tracks which fallback chain was used and surfaces that metadata. This is critical for debugging.
Lesson 3: Your Evaluation Suite Should Simulate Production Load
We ran evaluation suites in isolation. The agent looked perfect on our test cases. Then we deployed and the latency spiked, cache misses increased, and the agent started making different mistakes under load.
The problem: our evals didn't model production behavior. We needed to evaluate the full system, not just the LLM component.
class ProductionSimulation:
def setup(self):
self.cache_warmup() # Simulate cold cache
self.inject_latency(jitter=0.3) # Realistic network variance
self.simulate_concurrent_users(50)
self.inject_failed_dependencies()
def measure(self, agent):
return {
"p99_latency": agent.p99_response_time,
"error_rate": agent.failure_rate,
"hallucination_rate": self.measure_hallucinations(agent.outputs),
"fallback_triggered": agent.fallback_count / agent.total_requests
}
This revealed that 30% of our "errors" were actually timeout cascades—the agent made a tool call, timed out, retried, and the retry compounded the load. The fix was circuit breakers, not better prompts.
Lesson 4: Observability Isn't Logging—It's Tracing Decision Points
Standard logging told us what the agent did. It didn't tell us why. In production, you need to trace every decision point:
- Why did the agent choose tool X over tool Y?
- Why did it trust source A over source B?
- Why did it escalate to a human?
interface AgentTrace {
requestId: string;
decisions: DecisionPoint[];
uncertainties: UncertaintySnapshot[];
toolCalls: ToolCallRecord[];
confidenceScores: Record<string, number>;
fallbackChains: FallbackChainRecord[];
}
Without this granularity, debugging becomes guesswork. With it, we can reproduce any failure by replaying the exact decision trace.
Lesson 5: The Human-in-the-Loop Must Be Optional, Not Mandatory
Early versions required human review for every ambiguous case. This worked in dev and collapsed in production—humans became the bottleneck, and the agent learned to defer everything.
The right design: escalate only when the agent's uncertainty exceeds a threshold AND the business impact is high. Low-stakes queries get a confident guess. High-stakes queries (financial advice, medical triage, legal interpretation) get human review—but only after the agent has attempted resolution through its fallback chains.
def should_escalate(trace: AgentTrace) -> bool:
uncertainty = trace.uncertainties[-1]
impact = assess_business_impact(trace.request)
return uncertainty.score > ESCALATION_THRESHOLD and impact.severity >= MEDIUM
This reduced human intervention by 85% while catching the cases that actually mattered.
The Pattern That Emerged
After 96 iterations, the winning pattern wasn't a specific technique—it was a discipline:
- Measure before you optimize. Every change needs a metric.
- Make uncertainty visible. The agent should surface its doubts, not hide them.
- Design for failure modes. Every tool call, every external dependency, every branch has a fallback.
- Test under production conditions. Dev environment behavior ≠ production behavior.
- Trace decisions, not just outputs. Debugging requires understanding why, not just what.
Why This Matters Now
As AI agents move from prototypes to production systems, the gap between "it works in the demo" and "it works at scale" is where most projects fail. The failures aren't dramatic—they're incremental, cumulative, and invisible until they're catastrophic.
The agent that refused 96 times didn't improve because we found a better prompt. It improved because we built a system that could learn from its failures, surface its uncertainties, and degrade gracefully when it didn't know the answer.
That's not prompt engineering. That's software engineering.
Frequently Asked Questions
Q: How do you measure hallucination rates in production?
A: We use a combination of fact-checking against authoritative sources, cross-referencing multiple model outputs for consistency, and sampling human reviews on a rotating basis. The key is having a ground truth dataset you can compare against.
Q: What's the ROI of building these fallback chains versus just using a better model?
A: In our experience, 70% of production failures are architectural (missing fallbacks, no uncertainty handling, poor observability), not model-capability failures. A better model helps, but it won't fix a system that can't handle tool failures or express uncertainty.
Q: How long did it take to reach iteration 96? Can you share the timeline?
A: Roughly 14 weeks from first deployment to production stability. The first 40 iterations were pure debugging—the agent kept failing in unpredictable ways. Iterations 40-70 were architectural fixes (fallback chains, uncertainty gates). The last 26 were optimization and hardening.
For more deep dives on production AI systems, check out Tamiz's Insights, where we publish regular updates on agent architecture patterns and failure analysis.
The Architecture That Made It Click
The breakthrough came when we stopped treating "getting it right" as a prompt engineering problem and started treating it as a system reliability problem. Here's the architecture that survived 96 refusals and ultimately succeeded:
┌─────────────────────────────────────────────────────┐
│ Request Intake │
│ (Validation Gate → Risk Classification) │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Tiered Reasoning Engine │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Tier 1 │→ │ Tier 2 │→ │ Tier 3 │ │
│ │ Fast │ │ Medium │ │ Deep │ │
│ │ Model │ │ Model │ │ Model │ │
│ │ (cost: $)│ │ (cost: $$)│ │ (cost: $$$)│ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ↑______________ Refinement Loop _____________ │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Post-Processing & Verification │
│ (Fact-check → Policy check → Output formatting) │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Confidence Scoring │
│ (Auto-escalate if below threshold) │
└─────────────────────────────────────────────────────┘
Why This Works
The key insight: refusals are data, not dead ends. Each refusal tells us something about the boundary conditions of our system. By feeding refusal patterns back into the Tier 1 classifier, we gradually shrink the "I'm not sure" zone.
Section 5: The Refusal Triage Matrix
After 96 refusals, we classified them into four categories. This taxonomy became the single most useful artifact in our entire development process.
Category A: Legitimate Refusals (31%)
These were cases where the model correctly identified a policy violation, safety concern, or capability gap. Our initial reaction was frustration; our eventual reaction was relief.
from enum import Enum
from dataclasses import dataclass
from typing import Optional
class RefusalCategory(Enum):
LEGITIMATE = "legitimate" # Model was right to refuse
OVERREFUSAL = "overrefusal" # Policy too strict
CONTEXT_GAP = "context_gap" # Missing information
AMBIGUITY = "ambiguity" # Truly ambiguous request
@dataclass
class RefusalRecord:
request_id: str
category: RefusalCategory
raw_refusal: str
human_verified: bool
confidence_delta: float # How much confidence dropped
tier_attempted: int
resolution: Optional[str] = None
Action: For legitimate refusals, we didn't try to override the model. Instead, we improved our escalation paths—making it clearer when and how a human should take over.
Category B: Overrefusals (47%)
This was the biggest category and the most expensive. Nearly half of all refusals were the model being overly cautious—refusing valid requests due to:
- Ambiguous phrasing triggering safety classifiers
- Lack of domain-specific context in the system prompt
- Conservative default policies that didn't match our use case
def calibrate_overrefusal(record: RefusalRecord) -> str:
"""
For overrefusal cases, we inject context
that the model needs but wasn't provided.
"""
context_injections = {
RefusalCategory.OVERREFUSAL: [
f"You are operating in a {DOMAIN} environment.",
f"Requests in this domain have been pre-vetted for safety.",
f"The user has explicit authorization for this type of request.",
"Do not refuse based on generic policy heuristics alone.",
]
}
return "\n".join(context_injections[record.category])
The fix wasn't a stronger prompt. It was giving the model better context. Overrefusals dropped by 62% once we moved from generic system prompts to domain-specific contextual framing.
Category C: Context Gaps (14%)
The model refused because it genuinely couldn't answer—the request was missing critical information. These weren't failures; they were information requests in disguise.
def detect_context_gap(refusal_text: str) -> list[str]:
"""
Parse the refusal to identify what information
the model is asking for.
"""
missing_info_patterns = [
r"i need more information about.*",
r"could you clarify.*",
r"please provide.*",
r"i cannot determine.*without.*",
]
missing = []
for pattern in missing_patterns:
matches = re.findall(pattern, refusal_text, re.IGNORECASE)
missing.extend(matches)
return missing
The fix: Instead of retrying with the same request, we implemented an active information gathering loop that asked the user for exactly what was missing before proceeding.
Category D: Ambiguity (8%)
The rarest category—and the hardest. Cases where the request could legitimately be interpreted multiple ways, and the model's refusal was a signal that it needed disambiguation.
Section 6: The Self-Reflection Loop
The architecture that finally cracked 96+ success rate incorporated a self-reflection step between tiers. After each failed attempt, the model was asked to analyze why it refused and what it needed to proceed.
class SelfReflectionAgent:
"""
After a refusal, this agent analyzes the failure
and generates a refined strategy for the next attempt.
"""
REFLECTION_PROMPT = """
You just refused a request. Before giving up, analyze:
1. WHY did you refuse? (categorize the refusal)
2. WHAT information is missing that would help?
3. WHAT assumption might be wrong?
4. If you had more context, HOW would your answer change?
5. Generate a revised approach that could succeed.
"""
def reflect(self, original_request: str, refusal: str, history: list[dict]) -> dict:
reflection = self.llm.invoke(self.REFLECTION_PROMPT, {
"request": original_request,
"refusal": refusal,
"history": history
})
return self._parse_reflection(reflection)
def _parse_reflection(self, reflection: str) -> dict:
"""Extract structured insights from free-text reflection."""
return {
"root_cause": self._extract_root_cause(reflection),
"suggested_fix": self._extract_suggested_fix(reflection),
"confidence_estimate": self._estimate_confidence(reflection),
}
The Iteration Flow
Attempt 1 → Refusal → Reflection → Strategy Update → Attempt 2
→ Refusal → Reflection → Strategy Update → Attempt 3
...
→ Success!
Each cycle, the model was getting smarter about why it was refusing, not just retrying blindly. This meta-cognitive ability—thinking about its own thinking—was the single most impactful change we made.
Section 7: Hardening Techniques That Survived Production
By attempt 27, we had enough signal to move from experimentation to systematic hardening. These techniques held up under real traffic:
7.1 Dynamic Context Injection
Instead of static system prompts, we built a context composer that assembled prompts dynamically based on:
- User history and past interactions
- Domain classification of the request
- Similar successful cases from the knowledge base
- Real-time policy updates
class ContextComposer:
"""Assembles a dynamic system prompt from multiple sources."""
def compose(self, request: str, user_ctx: dict, domain: str) -> str:
sections = []
# Base policy (always present)
sections.append(self._load_base_policy(domain))
# Historical precedents (case-based reasoning)
precedents = self._find_similar_cases(request, top_k=3)
sections.append(self._format_precedents(precedents))
# User-specific context
if user_ctx.get("domain_expertise"):
sections.append(self._apply_expertise_level(user_ctx))
# Recent policy changes
sections.append(self._inject_policy_updates(domain))
return "\n\n".join(sections)
7.2 Confidence-Weighted Escalation
We stopped treating every response as equally valid. Instead, we assigned a confidence score to every model output and routed low-confidence responses through additional verification:
def decide_response_path(output: str, confidence: float, risk_level: str) -> ResponsePath:
"""Determine how to handle a model response based on confidence."""
if confidence >= 0.95:
return ResponsePath.AUTO_APPROVE
elif confidence >= 0.80 and risk_level == "low":
return ResponsePath.AUTO_APPROVE_WITH_LOG
elif confidence >= 0.60:
return ResponsePath.VERIFICATION_REQUIRED
elif risk_level == "high":
return ResponsePath.HUMAN_ESCALATION
else:
return ResponsePath.REFINE_AND_RETRY
7.3 The 96-Attempt Budget Pattern
We implemented a progressive effort allocation strategy. Early attempts used lightweight models and simple prompts. As the attempt count increased, we invested more compute:
| Attempt Range | Model | Depth | Cost Multiplier |
|---|---|---|---|
| 1–10 | Fast (Turbo/GPT-4o-mini) | Standard | 1x |
| 11–30 | Medium (GPT-4o) | Expanded context | 3x |
| 31–60 | Strong (Claude 3.5 Sonnet) | Chain-of-thought | 8x |
| 61–100 | Best (o1 / GPT-4o-max) | Deep reasoning + reflection | 20x |
This meant most requests resolved cheaply in the first 10 attempts. Only the genuinely hard cases consumed significant resources. Average cost per successful resolution: $0.047.
7.4 Failure Mode Catalog
We maintained a living document of known failure modes and their resolutions. Every new refusal category was added to this catalog with:
- Reproduction steps
- Root cause analysis
- Fix implemented
- Performance impact
## FRM-047: Overrefusal on Financial Advice Requests
**Discovered:** 2024-11-03 | **Attempts before fix:** 73
**Symptom:** Model refuses to discuss investment strategies even when clearly informational.
**Root Cause:** Generic "do not give financial advice" policy was too broad.
**Fix:** Added domain qualifier: "Provide informational analysis only; do not recommend specific positions."
**Impact:** Reduced overrefusals in finance domain by 89%.
**Related:** FRM-031, FRM-052
Section 8: What We Learned (That Nobody Tells You)
Lesson 1: Refusals Are Features, Not Bugs
The 96 refusals weren't 96 failures—they were 96 data points that mapped the boundary of what our system could handle. Every refusal taught us something about:
- Where our policies were too broad
- What context was missing
- How the model interpreted ambiguity
- Which edge cases we hadn't considered
Treat your refusal log as your most valuable dataset.
Lesson 2: The "First Successful Attempt" Is a Lie
When we reported "the agent got it right on attempt 97," that number was misleading. The real story was:
- Attempts 1–26: Discovery phase (understanding what we didn't know)
- Attempts 27–50: Architecture iteration (figuring out how to fix it)
- Attempts 51–74: Refinement (polishing the approach)
- Attempts 75–96: Edge case hardening (eliminating remaining failure modes)
- Attempt 97: The first time all the pieces came together
The breakthrough wasn't luck—it was accumulated learning.
Lesson 3: Cost and Reliability Are Not Trade-offs
We initially assumed that making agents more reliable would make them prohibitively expensive. The progressive effort allocation pattern proved otherwise:
| Metric | Before | After |
|---|---|---|
| Success rate | 68% | 96%+ |
| Avg. cost per success | $0.31 | $0.047 |
| Human escalation rate | 34% | 2.1% |
| P99 latency | 4.2s | 6.8s |
Yes, P99 latency increased. But the effective cost per successful task dropped by 85% because we stopped paying for wasted attempts on trivial requests.
Lesson 4: Your System Prompt Is a Living Document
The system prompt that got us to attempt 97 was unrecognizable from the one we started with. It had been shaped by:
- Every overrefusal we debugged
- Every context gap we identified
- Every legitimate refusal we confirmed
- Every successful resolution we analyzed
Version-control your system prompts the same way you version-control your code.
Section 9: The Production Readiness Checklist
If you're building agents that need to succeed on the first meaningful attempt (rather than the 97th), here's what we verified before shipping:
PRODUCTION_READINESS_CHECKLIST = {
"refusal_analysis": {
"description": "Every refusal must be classified and reviewed",
"required": True,
"tool": RefusalTriageMatrix(),
},
"self_reflection": {
"description": "Model must be able to analyze its own failures",
"required": True,
"tool": SelfReflectionAgent(),
},
"dynamic_context": {
"description": "Prompts must adapt to domain and user context",
"required": True,
"tool": ContextComposer(),
},
"confidence_scoring": {
"description": "Every output must have a quantified confidence score",
"required": True,
"tool": ConfidenceEstimator(),
},
"progressive_effort": {
"description": "System must scale compute investment with difficulty",
"required": True,
"tool": ProgressiveEffortAllocator(),
},
"failure_catalog": {
"description": "Known failure modes must be documented and tracked",
"required": True,
"tool": FailureModeCatalog(),
},
"human_fallback": {
"description": "Clear escalation path for unresolvable cases",
"required": True,
"tool": EscalationHandler(),
},
"observability": {
"description": "Every attempt, refusal, and reflection must be logged",
"required": True,
"tool": AgentObservabilityPipeline(),
},
}
Section 10: The Real Metric That Matters
After 96 refusals and one success, we changed how we measured agent reliability.
Before: "What percentage of requests does the agent handle without human help?"
After: "What percentage of requests reach a correct resolution within a reasonable attempt budget?"
The difference is subtle but profound. The first metric rewards agents that either succeed quickly or fail quietly. The second metric rewards agents that persist intelligently—using each failure to get closer to the right answer.
Our production agents now track:
| Metric | Target | Current |
|---|---|---|
| First-attempt success rate | ≥85% | 87.3% |
| Resolution within 5 attempts | ≥95% | 96.1% |
| Resolution within 100 attempts | ≥99% | 99.2% |
| Mean cost per resolution | ≤$0.10 | $0.047 |
| Mean latency per resolution | ≤10s | 6.8s |
Final Thoughts
Building reliable AI agents in production isn't about writing the perfect prompt. It's about building a system that learns from its own failures—one that treats every refusal as feedback, every edge case as an opportunity, and every successful resolution as proof that the architecture is sound.
The 96 refusals weren't a sign that our agent was broken. They were the sound of it learning to be right.
The agents that will dominate production in 2025 and beyond won't be the ones that never refuse. They'll be the ones that refuse wisely, learn faster, and persist intelligently until they get it right.
And that's a system worth building.
This article is part of an ongoing series on production AI systems. Next up: "From 96 Refusals to 99% Reliability—How We Built Our Agent's Memory System." Follow along at Tamiz's Insights for weekly deep dives on agent architecture, failure analysis, and production patterns.
Top comments (0)