I built a RAG evaluation tool that catches failures RAGAS misses

Abdullah Ahmad — Mon, 17 Aug 2026 14:33:21 +0000

I spent 6 weeks building RAG Sentinel – an open-source evaluation tool for RAG systems.

The problem: RAGAS scored my RAG system 0.91. But it was still hallucinating citations and ignoring half the retrieved context.

RAGAS doesn't answer the questions I actually cared about:

→ Does the answer actually USE the retrieved documents?
→ Do the citations exist in the source documents?
→ Does the answer contradict its own sources?
→ Is the system's confidence actually accurate?

So I built 5 custom metrics to measure exactly these.

What I Built

RAG Sentinel is a production-ready evaluation platform that goes beyond RAGAS with:

🔹 Retrieval-Generation Alignment – Does the answer actually use the retrieved docs? RAGAS faithfulness can score 0.9 even if half the context is ignored.

🔹 Citation Accuracy – Do citations actually exist in the source documents? Or are they hallucinated?

🔹 Context Contradiction – Does the answer contradict its own sources? Standard metrics don't check for this.

🔹 Confidence Calibration – When the system says "90% sure," is it actually right 90% of the time?

🔹 Production Monitoring Dashboard – Track quality trends. Detect degradation before users notice.

Tech Stack

Layer	Technology
Backend	FastAPI (Python)
Frontend	React (Netlify)
LLM	Groq (openai/gpt-oss-120b)
Deployment	Railway (backend) + Netlify (frontend)

The Hard Parts

The code took ~2 weeks. Getting it deployed took ~4.

CORS preflight errors: 3 days
PostgreSQL misconfiguration: 2 days
LLM API deprecated mid-project: Rewrote the evaluation pipeline entirely

That's where I learned the real difference between "works on my laptop" and "someone can actually use it."

What I Learned

🔸 Building systems > building chatbots. Anyone can call an API. Building a monitoring platform is harder but way more valuable.

🔸 Deployment is 50% of the work. The code is the easy part. Making it work in production is where the real learning happens.

🔸 Ship. Then iterate. I deployed with basic features, added more as I went. If I'd waited for "perfection," I'd still be coding.

Links

🔗Linked In:https://www.linkedin.com/feed/update/urn:li:activity:7495058202019938307/
🔗 Live Demo: https://rag-sentinel.netlify.app

🔗 GitHub: https://github.com/Abdullah-Ahmad-2025/rag-sentinel

Built solo. 1st-year BS AI student. 6 weeks.