Building a Production-Grade AI Fact-Checker: Patterns, Pipelines, and the Question Every AI Engineer Must Answer

Ajay Mahadeven — Wed, 22 Apr 2026 14:10:55 +0000

A research project from Economic Data Sciences

Before we begin

This is not a tutorial about calling an AI API. You can find that in ten minutes on YouTube.

This is about what happens after you've called the API and realised that's the easy part — about the architecture decisions that separate a prototype from a system you can reason about, trust, and maintain. About the one question that keeps coming up in every serious AI project we build at Economic Data Sciences:

Should this input go to the AI model at all?

The code is public. The app is live. Follow along.

What we built

A full-stack AI fact-checker. You submit a claim — text, PDF, CSV, DOCX, or Markdown — and get back a verdict (TRUE / FALSE / DISPUTED / UNVERIFIABLE), a confidence score, reasoning, and cited sources.

The full interface — five input types, a rate limit counter, and a "How to use" link for first-time visitors.

Under the hood:

Two-stage AI pipeline — a guardrail classifier runs before the analyzer
Cloud provider abstraction — one interface, three providers (Azure AI Foundry, AWS Bedrock, GCP Vertex AI), swappable via a single env var
Model rotation — primary, round-robin, or fallback strategies across a model pool
File upload support — documents extracted, analyzed, deleted. No retention.
MCP server — the entire pipeline exposed as a callable tool for Claude Desktop and Claude Code
DB-level caching, token tracking, rate limiting, and a monthly spending guard

Not familiar with the app? There's a How to Use page built for non-technical users.

The How to Use page — plain language, step-by-step, with live result mock-ups so users know what to expect before they try.

The question that shapes everything

Here is a claim: "Neil Armstrong walked on the moon in 1969."

Here is another input: "How long do dolphins live?"

A naive implementation sends both to the AI analyzer. The first is fact-checkable. The second is a question. They require completely different handling — but they look identical at the API boundary. Both are strings. Both come from a user.

This is the question every AI engineer eventually faces: not all inputs are equal, and treating them as if they are costs you money, time, and trust.

Stage 1: The Guardrail Classifier

Before the fact-check analyzer ever runs, every input goes through a classifier. One AI call. One purpose: is this a verifiable true/false claim?

// src/server/ai/classifier.ts

const SYSTEM_PROMPT = `You are a strict claim validator for a fact-checking application.

Your only job is to decide whether a given input is a verifiable 
true/false claim about the real world.

A VALID claim:
- Can be verified as true or false using evidence
- Makes a factual assertion about the real world

An INVALID claim falls into one of these categories:
- INFORMATIONAL: asking for information
- OPINION: subjective, no factual answer
- MATH: a mathematical expression
- IRRELEVANT: unrelated task
- HARMFUL: dangerous, hateful, or abusive content`;

If the classifier returns INVALID, the pipeline stops. No second AI call. No DB write for analysis. The user gets a clear, structured rejection.

The guardrail classifier in action - "How long do dolphins live?" is a question, not a claimPipeline stops here. No analyzer call made.

This is not a UX feature. It is an architectural gate. It exists because:

Cost — the analyzer call costs 10x more tokens than the classifier. Never pay for it on garbage input.
Quality — the analyzer produces worse results on non-claims. A garbage verdict is worse than no verdict.
Security — harmful content is rejected before it reaches the more capable model.

The classifier is cheap, fast (temperature 0, max 100 tokens), and wrong occasionally — but wrong cheaply. The asymmetry is intentional.

The pipeline in full

rawClaim
  │
  ▼
Normalize → SHA256 hash
  │
  ▼
DB cache check → HIT: return immediately, zero AI calls
  │ MISS
  ▼
Spending guard → throws if monthly USD cap reached
  │
  ▼
Guardrail Classifier (AI Call 1)
  ├─ INVALID → return rejection + store as training data
  └─ VALID
       │
       ▼
  Fact-Check Analyzer (AI Call 2)
       │
       ▼
  Store: Submission → Claim → AnalysisResult → Sources
       │
       ▼
  Return: verdict + confidence + reasoning + sources

Two AI calls maximum. Often zero — the cache handles the rest. This is not an accident; it is a constraint we enforced from the first line of code.

Here's what a successful result looks like:

FALSE · 95% confidence. The claim, the verdict,the confidence bar, the reasoning, three cited sources with credibility ratings, a timestamp, and thumbs up/down feedback. Everything surfaced from two AI calls.

The spending guard

// src/server/pipeline/spending-guard.ts

export async function enforceSpendingLimit() {
  if (process.env.NODE_ENV === "development") return;

  const [classifierSpend, analyzerSpend] = await Promise.all([
    db.classifierResult.aggregate({ _sum: { inputTokens, outputTokens } }),
    db.analysisResult.aggregate({ _sum: { inputTokens, outputTokens } }),
  ]);

  const totalUsd = calculateUsd(classifierSpend, analyzerSpend);
  if (totalUsd >= MONTHLY_SPEND_LIMIT_USD)
    throw new Error("Monthly spend limit reached");
}

Every token from every AI call is stored in the database. The spending guard queries the real numbers before every AI call. Not estimates. Not request counts. Actual token costs. If you've spent $5 this month, the gate closes until the month resets.

This pattern is not optional in production AI systems. Without it, a single misbehaving client or a bug in your rate limiter can route hundreds of dollars out of your account overnight.

Provider abstraction: the architecture decision that ages well

The app started on Azure. The provider abstraction was designed from day one to support AWS Bedrock and GCP Vertex AI as well — the adapters are built and the interface is identical across all three. Swapping providers requires changing a single environment variable, nothing else in the pipeline changes.

// src/server/ai/providers/types.ts

interface CloudProvider {
  complete(request: AIRequest): Promise<AIResponse>;
}

One interface. The concrete implementation is a factory decision made once at startup:

// src/server/ai/providers/index.ts

export function getProvider(): CloudProvider {
  switch (env.CLOUD_PROVIDER) {
    case "azure": return new RotatingProvider(new AzureProvider());
    case "aws":   return new RotatingProvider(new AWSProvider());
    case "gcp":   return new RotatingProvider(new GCPProvider());
  }
}

The RotatingProvider wrapper implements three strategies:

Strategy	Behaviour
`primary`	Always use the first model in the pool
`round-robin`	Cycle through models, one per call
`fallback`	Try the first; on error, try the next

Every AIResponse carries modelUsed — stored to the DB alongside the result. When you rotate to a new model and your quality metrics shift, you know exactly what changed.

File uploads: when the document is the source of truth

The original pipeline assumes the claim is about the world. File uploads flip that assumption entirely — the analyzer should look only at the uploaded document. General world knowledge is irrelevant and potentially harmful.

The PDF upload tab — drag-drop zone, claim input, two separate inputs: the document and the claim about it.

This required rethinking two pipeline stages.

The classifier. A document assertion like "the monthly retainer is $12,500" would be rejected by the text classifier as unverifiable from general knowledge — a correct assessment in the wrong context. The fix was a separate system prompt for file mode:

const FILE_SYSTEM_PROMPT = `You are a claim validator. The user has uploaded 
a document and is making a claim about its contents.

Mark as VALID if the input makes any factual assertion, even one specific to 
a contract, report, or dataset.

Mark as INVALID only if: it is a question, purely subjective, 
irrelevant, or harmful.`;

Same output schema. Same Zod validation. Same DB storage. Different intent.

The sources. The text analyzer returns cited web URLs. The file analyzer has nothing to cite — the document itself is the evidence. We considered fabricating placeholder URLs to satisfy the schema. We didn't:

const FILE_SOURCE_INSTRUCTION =
  `For the sources array: return an empty array — do not invent URLs.
   Your reasoning already explains what in the document supports the verdict.`;

The Zod schema was relaxed to min(0). The result card hides the sources section when empty. The reasoning carries the document evidence. No fabrication, no silent failures.

CSV aggregation. A claim about totals — "total revenue is $62,375" — cannot be verified against a truncated row sample. The CSV extractor computes full column aggregates across every row and appends them below the table. The model receives both the sample and the computed totals. Claims about aggregates are verifiable even on large files.

The CSV pipeline — test-sales.csv uploaded, claim checked against computed column totals. FALSE at 100%, confidence: column summary shows 1,935 units sold, not 5,000. No sources section — the data is the evidence.

Blob lifecycle. Files touch Azure Blob Storage for seconds:

Upload → Download (Azure SDK) → Extract → Delete → Analyze

Submission.fileUrl is stored as null after deletion. Nothing persists after processing.

A real production bug: pdf-parse and the test fixture trap

When we deployed to Vercel, PDF uploads failed immediately:

ENOENT: no such file or directory,
open './test/data/05-versions-space.pdf'

The cause: pdf-parse's index.js entry point contains a debug block that reads a test fixture file at import time. In a local Node.js environment the path resolves. Inside Next.js's webpack bundler it doesn't — the file doesn't exist in the build output.

The fix was two lines:

// Import the lib path directly — bypasses the test runner in index.js
const { default: pdfParse } = await import("pdf-parse/lib/pdf-parse.js");

// next.config.js — tell webpack not to bundle this package at all
const config = {
  serverExternalPackages: ["pdf-parse"],
};

This class of bug — a package that behaves differently when bundled versus run natively — is common enough in Node.js that it's worth knowing the pattern. serverExternalPackages in Next.js is specifically designed for it.

Prompt versioning: the discipline that makes AI systems debuggable

Every AI call stores two fields:

modelVersion:  "gpt-4.1"
promptVersion: "classifier-v1.0"

When you change a system prompt, you need to know which results were produced by which version. Without this, you cannot measure whether a change improved or degraded quality.

Input type	Classifier version	Analyzer version
Text	`classifier-v1.0`	`analyzer-v1.0`
PDF	`classifier-v1.1-file`	`analyzer-v2.0-pdf`
CSV	`classifier-v1.1-file`	`analyzer-v2.0-csv`
DOCX	`classifier-v1.1-file`	`analyzer-v2.0-docx`
MD	`classifier-v1.1-file`	`analyzer-v2.0-md`

When the prompt changes, the version increments. Every result in the DB is forever linked to the prompt that produced it. The classifier results are especially valuable — every VALID and INVALID decision is labeled training data for the next iteration.

The MCP server: the pipeline as a tool

The entire pipeline is exposed as a single MCP tool callable from Claude Desktop or Claude Code:

{
  "mcpServers": {
    "fact-check-analyzer": {
      "command": "npx",
      "args": [
        "tsx",
        "--env-file=/path/to/.env",
        "--tsconfig=/path/to/tsconfig.json",
        "/path/to/src/mcp/index.ts"
      ]
    }
  }
}

After that, in any Claude conversation:

"Is it true that the Great Wall of China is visible from space?"
→ Claude calls fact_check_claim, runs the full pipeline, returns FALSE · 95% confidence.

The MCP server is not a wrapper around the app. It is the app — same pipeline, same DB, same spending guard. The interface changed; the logic did not.

The architectural parallel: when AI is not the answer

This project was built as research at Economic Data Sciences. The direct application is a larger platform where AI models work alongside deterministic decision engines.

The central design challenge there — and the one this project was built to study — is this: not every user prompt belongs in front of a language model.

Some inputs contain signal that a deterministic system can process faster, cheaper, and more reliably than an AI model. The job of the classifier layer is to detect that signal and route accordingly — before the expensive call, not after.

In the fact-checker, that line is drawn at "is this a verifiable claim?" — a relatively simple binary. In a production decision-support system, the same question is harder: does this prompt contain intent that a structured reasoning engine should handle? Is the user expressing a constraint, an objective, a preference — something that maps to a computational model rather than language generation?

The architectural answer is the same in both cases:

Run a lightweight classifier on every input, before the main operation
Make the routing decision explicit and auditable — store it
Never skip the gate to save one round trip — that round trip is the system's immune system
Keep the classifier prompt narrow and fast — one job, doesn't need to be smart

The fact-checker is the simplified, public version of that pattern. The gate is binary. The routing is binary. But the lesson — that a cheap classifier protecting an expensive operation is not overhead, it is architecture — transfers directly to systems where the routing decisions are far more complex and the stakes far higher.

What we'd tell ourselves at the start

- Never trust AI output shape — always validate with Zod
- AI calls are not database calls — slow, expensive, non-deterministic
- Cache aggressively — never pay the AI twice for the same question
- Always cap max_tokens — every call, every time
- Version your prompts — so you can measure what changed
- The classifier is not optional — it is the cheapest line of defence you have
- serverExternalPackages exists for a reason — know when to use it

Try it yourself

Live app: fact-check-analyzer.vercel.app
How to use: fact-check-analyzer.vercel.app/how-to-use
Code: github.com/ajaymahadeven/fact-check-analyzer

A few claims worth trying:

Claim	Expected
"The Great Wall of China is visible from space	FALSE
with the naked eye"
"Humans use only 10% of their brain"	FALSE
"Finland has more lakes than any other country"	TRUE
Upload `test-contract.pdf` → "The contract	TRUE
expires on 31 December 2025"
Upload `test-sales.csv` → "Total units sold	FALSE
across all months exceeds 5,000"

Test files are in src/tests/uploads/ with expected verdicts documented.

This project is a research initiative of Economic Data Sciences, exploring production patterns for AI-augmented decision systems. Questions, disagreements, and pull requests welcome.

DEV Community: Ajay Mahadeven