You didn't ship a broken AI pipeline — you shipped a pipeline where the AI sounds completely certain even when it's completely wrong, and you had no check to tell the difference.
The Problem
You connect GPT-4 to your production workflow. Lead classification. Contact enrichment. Automated summaries. You test it on 10 samples. It works perfectly. You ship it.
Three weeks later you discover a contact was enriched with a fabricated job title. A sales rep sent a personalized email using that title. It reached the prospect. Your credibility took the hit.
Or maybe your lead classifier started routing enterprise accounts into the wrong sales bucket. Nobody noticed for two weeks because the pipeline kept running confidently, producing output that looked exactly like correct output — same format, same structure, same confident tone.
This is the core problem with LLM outputs in production: the model does not know when it is wrong. It produces a hallucinated job title with the same confidence as a verified one. There is no error state. There is no warning. There is just output — and if you have no check, you consume it as truth.
Why It Happens
LLMs are probabilistic, not deterministic. The model predicts the most likely next token given the context — it does not verify claims against a source of truth before responding.
If you set temperature to 0 expecting deterministic accuracy, that does not fix hallucination. Temperature 0 makes the output repeatable — the same wrong answer every time, consistently.
If you add a system prompt instruction like "Do not hallucinate. Only return factual information," the model will comply — in the sense that it will continue producing output in the same confident format. The instruction changes tone, not accuracy.
The deeper issue is architectural: most pipelines have no output contract. You call the LLM, receive a string or a JSON blob, and pass it to the next step. There is no schema the output is validated against. There is no confidence gate that routes low-confidence outputs to review. There is no cross-reference check against a known source. The pipeline assumes every output is correct by design.
That assumption breaks at scale, on edge cases, on non-English inputs, on ambiguous company names, on truncated context — on exactly the inputs that never appeared in your 10-sample test set.
The Defense Pattern
You do not need to eliminate LLM errors — that is not achievable. You need to catch them before they propagate downstream. Here is the four-layer defense pattern:
Layer 1: Schema Validation
Enforce a structured output contract using JSON mode or function calling. Validate required field presence and field type constraints before the output leaves the LLM layer.
import Ajv from 'ajv';
const ajv = new Ajv();
const schema = {
type: 'object',
required: ['company', 'job_title', 'industry'],
properties: {
company: { type: 'string', minLength: 1 },
job_title: { type: 'string', minLength: 1 },
industry: { type: 'string', enum: ['SaaS', 'Fintech', 'Healthcare', 'Ecommerce', 'Other'] }
}
};
const validate = ajv.compile(schema);
function validateLLMOutput(rawOutput) {
let parsed;
try {
parsed = JSON.parse(rawOutput);
} catch (e) {
return { valid: false, error: 'JSON parse failure', raw: rawOutput };
}
const valid = validate(parsed);
if (!valid) {
return { valid: false, errors: validate.errors, raw: rawOutput };
}
return { valid: true, data: parsed };
}
If the output fails schema validation, route it to your retry queue — not downstream.
Layer 2: Confidence Gating via Self-Consistency
If your model exposes logprobs (GPT-4 API does), use them as a confidence gate. Without logprobs, use a self-consistency check: call the model three times with temperature > 0 and check if the outputs agree.
async function selfConsistencyCheck(prompt, n = 3) {
const responses = await Promise.all(
Array(n).fill(null).map(() =>
callLLM(prompt, { temperature: 0.3 })
)
);
// For classification outputs, check majority agreement
const counts = {};
for (const r of responses) {
const key = r.industry; // the field you want to gate
counts[key] = (counts[key] || 0) + 1;
}
const majority = Object.entries(counts).sort((a, b) => b[1] - a[1])[0];
const confidence = majority[1] / n;
return {
value: majority[0],
confidence,
routeToReview: confidence < 0.67 // less than 2/3 agreement
};
}
If confidence falls below your threshold, route to a human review queue instead of propagating downstream.
Layer 3: Cross-Reference Validation
For enrichment outputs, validate against a known source before writing to your CRM or database. If the output is a company name, check it against your existing records. If the output is a numeric value (revenue range, headcount), apply a sanity gate for known valid ranges.
Layer 4: Retry with Stricter Prompt
If an output fails schema validation on the first attempt, retry with a more constrained prompt before escalating to human review:
async function enrichWithRetry(contact, maxRetries = 2) {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
const prompt = attempt === 0
? buildStandardPrompt(contact)
: buildStrictPrompt(contact); // narrower output constraints, explicit enum values
const raw = await callLLM(prompt);
const result = validateLLMOutput(raw);
if (result.valid) return result.data;
if (attempt === maxRetries) {
routeToHumanReview(contact, result.errors);
return null;
}
}
}
Using Apify to Build Your Validation Test Suite
Before shipping a new LLM enrichment or classification pipeline, you need a validation test set — a collection of inputs with known correct outputs that you can run your pipeline against and measure accuracy.
If you do not have labeled data internally, you can scrape public sources to build one. For lead enrichment, the LinkedIn Job Scraper on Apify can pull structured job listing data — company name, title, industry — that you can use as a ground-truth comparison set.
For classification pipelines, the Google SERP Scraper can pull company and industry data from search results that you validate your LLM classifications against.
A 20-item validation test suite costs $0 to build on Apify's free tier. Run your pipeline against it before every deployment. If your validation error rate on the test set crosses a threshold you set, block the deployment and fix the prompt first.
This is not a substitute for production monitoring — it is a pre-ship gate that catches systematic prompt failures before they reach real data.
Monitoring in Production
Once you ship, instrument three metrics:
Schema violation rate — percentage of LLM outputs that fail your schema validator. If this spikes, your prompt has drifted from the output contract.
Human review queue rate — percentage of outputs routed to manual review due to low confidence. A rising rate may indicate a new class of inputs the model handles poorly.
Downstream null/error rate — percentage of downstream pipeline steps that receive null, error, or unexpected inputs from the LLM layer. This is your lagging indicator: if it spikes, something upstream broke.
Set alerts on all three. Your schema violation rate alert is your fastest signal — it fires before downstream errors accumulate.
The Cost Frame
$0 to build a 20-item validation test suite using public data and Apify's free tier.
A few hours to add a schema validator, a confidence gate, and a human review queue routing path.
The alternative: silent data corruption in production. Hallucinated job titles in your CRM. Misclassified leads in the wrong sales bucket for two weeks. A personalized email with the wrong information reaching a prospect.
The validation layer is not expensive to build. What is expensive is the absence of one — and it does not send you an alert when it happens.
Related: If your pipeline also calls external APIs (enrichment APIs, search APIs, classification endpoints), check your rate limit handling — the same production incident pattern applies. How to Handle API Rate Limits Before They Break Your Production Integration
Top comments (0)