Most prompt-engineering setups either hope the model complies with formatting/compliance requirements, or let the same model that generated the output also score it. Both fail quietly. I wanted a pipeline that catches that before a human ever sees the result.
The problem
Two failure modes show up constantly in LLM-generated content: the model "forgets" to include a required section (compliance language, a caveat, a citation), and when you ask the same model to grade its own output, it's a biased judge of its own work — it rates its own mistakes more charitably than an independent reviewer would.
What I built
A sequential pipeline: user input → MCP-grounded research → DSPy reasoning/generation → deterministic validation → independent evaluation → optional closed-loop verification.
- MCP Research Module — injects live domain knowledge into context before generation starts, instead of relying on the model's frozen training data
- DSPy.ai for the actual reasoning/generation step — declarative, optimizable prompt pipelines instead of hand-tuned prompt strings
- Deterministic QA Linter — a rule-based hard gate (zero LLM calls) that catches placeholder gaps, duplicated sections, contradictions, and fabricated citations before anything reaches evaluation
- Independent Evaluation via DeepEval/G-Eval — an external judge model scores the output, specifically to avoid the model grading its own work
- Iterative Enhancer — generate → critique → refine, with a bounded iteration budget and early-stopping once quality targets are hit, so it doesn't loop forever chasing marginal gains
- Multi-provider routing across OpenAI, Anthropic, Mistral, Qwen, Cerebras, OpenRouter, Groq, and Gemini with priority-fallback, so a single provider outage doesn't take the pipeline down
215 offline tests run with zero API keys required, and the whole thing ships through GitLab CI/CD with SAST, secret detection, bandit, and pip-audit scanning.
Key design decisions
Deterministic gate before the AI judge, not instead of it: cheap, fast, zero-cost checks (duplication, placeholders, missing required sections) get caught by a linter before spending an API call on evaluation. The expensive independent-judge step only runs on output that already passed the free checks.
External judge over self-scoring: explicitly rejected having the generator model rate its own output — avoids the obvious bias problem, at the cost of an extra model call per evaluation.
Bounded iteration, not infinite refinement: the critique→refine loop has a hard budget and early-stopping. Early versions without this would occasionally chase diminishing returns on borderline outputs.
What's next
Expanding the enterprise-safeguard sections (the linter currently enforces Security & Compliance, Scalability & Observability, and Governance & Audit blocks on production-bound prompts) and tightening the closed-loop verification trigger conditions.
Portfolio: jamilfayad.com
Top comments (0)