Sub-Millisecond RAG Faithfulness Evaluation with agentic-rag-faithfulness

Evaluating whether a Retrieval-Augmented Generation (RAG) system actually sticks to retrieved context usually requires calling an expensive LLM judge (like GPT-4), adding hundreds of milliseconds of latency and recurring API costs.

agentic-rag-faithfulness is an ultra-fast, local evaluation library that calculates RAG faithfulness scores in under 1 millisecond with support for both English and Arabic text.

seotarek / agentic-rag-faithfulness

A lightweight, sub-millisecond evaluation library to measure RAG faithfulness and detect LLM hallucinations against retrieved context documents in Arabic and English.

Agentic RAG Faithfulness Evaluator 🎯🧠

A lightweight, sub-millisecond evaluation library to measure RAG faithfulness and detect LLM hallucinations against retrieved context documents in Arabic and English.

Developed by Tarek Mohamed (@seotarek)

View on GitHub

---

⚡ Why Local & Sub-Millisecond?

• Real-Time Guardrails: Can be run directly inside production serving pipelines before streaming answers to end users.
• Zero API Costs: No external LLM judge tokens required for evaluation.
• Cross-Lingual Support: Specifically calibrated for complex Arabic morphology as well as standard English text.

---

💻 Quick Code Example

from rag_faithfulness import FaithfulnessEvaluator

evaluator = FaithfulnessEvaluator()

context = "Valorex is an ultra-fast native desktop browser built in Rust."
response = "Valorex is written in Python and uses Electron."

score = evaluator.evaluate(context=context, response=response)
print(f"Faithfulness Score: {score.score:.2f} (Hallucination Detected: {score.is_hallucination})")
# Output: Faithfulness Score: 0.12 (Hallucination Detected: True)

Star the project and view benchmarks on GitHub:
👉 https://github.com/seotarek/agentic-rag-faithfulness