I built a prompt pipeline that grades itself with an independent AI judge, not its own output

jamil fayad — Mon, 12 Oct 2026 12:45:04 +0000

Most prompt-engineering setups either hope the model complies with formatting/compliance requirements, or let the same model that generated the output also score it. Both fail quietly. I wanted a pipeline that catches that before a human ever sees the result.

The problem

Two failure modes show up constantly in LLM-generated content: the model "forgets" to include a required section (compliance language, a caveat, a citation), and when you ask the same model to grade its own output, it's a biased judge of its own work — it rates its own mistakes more charitably than an independent reviewer would.

What I built

A sequential pipeline: user input → MCP-grounded research → DSPy reasoning/generation → deterministic validation → independent evaluation → optional closed-loop verification.

MCP Research Module — injects live domain knowledge into context before generation starts, instead of relying on the model's frozen training data
DSPy.ai for the actual reasoning/generation step — declarative, optimizable prompt pipelines instead of hand-tuned prompt strings
Deterministic QA Linter — a rule-based hard gate (zero LLM calls) that catches placeholder gaps, duplicated sections, contradictions, and fabricated citations before anything reaches evaluation
Independent Evaluation via DeepEval/G-Eval — an external judge model scores the output, specifically to avoid the model grading its own work
Iterative Enhancer — generate → critique → refine, with a bounded iteration budget and early-stopping once quality targets are hit, so it doesn't loop forever chasing marginal gains
Multi-provider routing across OpenAI, Anthropic, Mistral, Qwen, Cerebras, OpenRouter, Groq, and Gemini with priority-fallback, so a single provider outage doesn't take the pipeline down

215 offline tests run with zero API keys required, and the whole thing ships through GitLab CI/CD with SAST, secret detection, bandit, and pip-audit scanning.

Key design decisions

Deterministic gate before the AI judge, not instead of it: cheap, fast, zero-cost checks (duplication, placeholders, missing required sections) get caught by a linter before spending an API call on evaluation. The expensive independent-judge step only runs on output that already passed the free checks.

External judge over self-scoring: explicitly rejected having the generator model rate its own output — avoids the obvious bias problem, at the cost of an extra model call per evaluation.

Bounded iteration, not infinite refinement: the critique→refine loop has a hard budget and early-stopping. Early versions without this would occasionally chase diminishing returns on borderline outputs.

What's next

Expanding the enterprise-safeguard sections (the linter currently enforces Security & Compliance, Scalability & Observability, and Governance & Audit blocks on production-bound prompts) and tightening the closed-loop verification trigger conditions.

Portfolio: jamilfayad.com

DEV Community: jamil fayad

I built a prompt pipeline that grades itself with an independent AI judge, not its own output

The problem

What I built

Key design decisions

What's next