fast-rag-auditor: zero-config RAG retrieval & chunking audit (free, open-source)
A zero-config CLI that audits the two things that silently break RAG: chunking quality and retrieval speed/recall.
The problem
Most RAG failures aren't in the LLM — they're in how you split and retrieve text. Chunks that are too small lose context; chunks that are too large dilute the embedding; hidden near-duplicates bloat your index; and slow top-k queries kill UX. By the time you notice in production, it's expensive.
The tool
fast-rag-auditor is a tiny, dependency-free Python CLI from Autonomous Studio. Point it at a folder of files or a live URL:
pip install git+https://github.com/3km-collab/fast-rag-auditor.git
fast-rag-auditor ./docs
fast-rag-auditor https://en.wikipedia.org/wiki/Retrieval-augmented_generation --chunk-size 300
It prints an instant report:
- Chunk size distribution (tiny / small / good / large)
- Self-retrieval recall@k — does each chunk return itself as a top-k match?
- Near-duplicate ratio — redundant content wasting tokens
- Retrieval latency (ms/query)
-
Health score (0–100) + actionable warnings, with
--fail-underfor CI gates
Everything runs locally with deterministic hashed embeddings — no API keys, no vector DB, no config.
Free vs. production
The auditor is the free entry point. When you're ready for production, the RAG Production Automation Kit adds real embeddings, hybrid search, ingestion connectors, evaluation, and Docker deployment.
| Capability | fast-rag-auditor (free) | RAG Production Automation Kit (pro) |
|---|---|---|
| Chunk-size audit | ✅ | ✅ |
| Recall@k proxy | ✅ | ✅ real embeddings |
| Duplicate detection | ✅ | ✅ auto-dedupe |
| Latency benchmark | ✅ local | ✅ real vector DB |
| Ingestion / hybrid / eval / deploy | – | ✅ |
Get started
- ⭐ Repo: https://github.com/3km-collab/fast-rag-auditor
- 📦 Pro kit: https://autonomousstudio.gumroad.com/l/xzxbdn
- 🛒 Store: https://autonomousstudio.gumroad.com
MIT licensed. Feedback and stars are welcome.
Top comments (0)