I watched a resume tailoring agent run 50 perfect demos. Every test resume produced clean, accurate output. On the first real user submission, the model fabricated a degree, invented a job title, and added a company that didn't exist. The demo never caught it because the demo data was curated. Production data is never curated.
The problem wasn't the model. It was the contract.
Most teams treat the LLM as a black box. Feed it text, get text back, move on. When it fails in production, they blame the model. But the model is doing exactly what you asked. The fault is in the system prompt you never read, the schema you never validated, and the fallback you never built.
Here's what I've learned building production AI pipelines that don't fall apart when real users show up.
The demo trap
Every LLM demo looks good because it runs on clean inputs. The resume tailor I built exposed this immediately. In testing, the model correctly extracted work history, education, and skills from every sample. Then a real upload arrived with a nonstandard format, and the model filled in the gaps by inventing data.
Why? Because the schema said "education" was required. The LLM chose to satisfy the schema rather than return null. It wasn't malicious. It was following instructions.
This is the first thing you need to understand about production LLMs: they will satisfy the schema you give them, even if it means lying. The fix isn't to prompt harder. It's to design a schema that lets the model say "I don't know" without breaking the pipeline.
When I built a job scoring pipeline processing 10,000 listings daily, every extraction function included a confidence field and a boolean flag for insufficient data. The model was allowed to skip a record. That single change eliminated fabricated data from the pipeline.
Read the system prompt (the one you didn't write)
Every model provider ships a default system prompt. If you haven't read yours, you're guessing.
The first thing I do with any new LLM integration is audit the default system prompt. I found one that included "be helpful and thorough." That sounds harmless until you realize "thorough" means the model will guess when it doesn't have enough information. In a production pipeline, guessing is worse than failing.
The fix was explicit. I added a confidence threshold to every function call. If the model couldn't hit 0.8 confidence, the record was skipped instead of getting fabricated data.
const extractionConfig = {
functions: [{
name: "extract_job_details",
parameters: {
type: "object",
properties: {
title: { type: "string" },
company: { type: "string" },
confidence: { type: "number", minimum: 0, maximum: 1 },
has_insufficient_data: { type: "boolean" }
},
required: ["confidence", "has_insufficient_data"]
}
}]
}
The model must be allowed to decline. If there's no path to say "I can't do this," it will fabricate. Every production system I build now includes this pattern.
Validate the output, not the intent
Most teams validate the input to an LLM call. Almost none validate the output. That's backward.
The LLM's output is the only thing that touches your downstream system. If it's malformed, the entire pipeline breaks. I built a resume tailoring system where the output was validated against a strict JSON schema before it ever reached the document generator.
The key pattern is conditional presence flags. Every optional field has a corresponding boolean guard. The model must set the guard to true before it can fill in the field. If the guard is false, the field is omitted entirely. No fabricated data can leak through.
const resumeOutputSchema = {
type: "object",
properties: {
has_work_experience: { type: "boolean" },
work_experience: {
type: "array",
items: { /* company, role, duration */ }
},
has_education: { type: "boolean" },
education: {
type: "array",
items: { /* institution, degree, year */ }
}
},
required: ["has_work_experience", "has_education"],
allOf: [
{
if: { properties: { has_work_experience: { const: true } } },
then: { required: ["work_experience"] }
},
{
if: { properties: { has_education: { const: true } } },
then: { required: ["education"] }
}
]
}
This eliminated hallucinations in the resume pipeline. The model could set a guard to false, and the system would simply skip that section. No invented data, no downstream corruption.
Build for failure, not success
A production AI agent needs a failure chain, not a success path.
The most common mistake I see is a single retry loop. If the LLM fails, try again. Maybe three times. That's not a fallback chain. That's hope.
Real fallback chains look like this:
- Try the primary model with the full prompt and strict schema
- If confidence is low, try a simpler prompt with fewer constraints
- If the simpler prompt fails, fall back to a deterministic rule-based extractor
- If nothing works, surface the error to a human with enough context to fix it
I built this into a document analysis pipeline for legal documents. The primary model (GPT-4o) attempted full extraction. If confidence dropped below threshold, the system fell back to a simpler model with reduced scope. If that also failed, it flagged the document for manual review.
The breakdown: 87% of documents handled by the primary model. 11% by the fallback. 2% needed human review. That 2% was the difference between a reliable system and a constant fire drill.
The loop that doesn't break
Agent loops are where most production systems die. The agent gets stuck in a cycle, burning tokens and producing nothing useful.
The fix is a dead man's switch. Every loop iteration must increment a counter. If the counter exceeds a threshold, the agent must stop and return whatever it has, even if incomplete. This prevents the silent budget drain that kills so many production AI systems.
I apply this to every agent loop I build now. The agent can iterate up to N times. After that, it must produce a partial result and explain what it couldn't complete. Bounded loops, always.
async function runAgent(task, maxIterations = 5) {
let result = null;
for (let i = 0; i < maxIterations; i++) {
result = await agentStep(task, result);
if (result.status === "complete") break;
if (result.status === "stuck") {
result.partial = true;
break;
}
}
return result;
}
The loop is bounded. The agent must produce something. It cannot spin forever.
If your team is wrestling with AI agents that work in demos but fail when real users show up, that's the kind of thing I help with. I build production AI pipelines that don't fall apart, and I'm happy to compare notes on what's breaking in your system.
Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.
Top comments (1)
Excellent insights into production AI. The emphasis on schema validation, confidence scoring, and fallback strategies highlights what it takes to build reliable AI systems beyond demos.