Stop Guessing Your RAG Hyperparameters: Why I Built a Local-First Benchmarking Framework

Building a "Hello World" Retrieval-Augmented Generation (RAG) app takes about 5 minutes. But taking that pipeline to production and ensuring it consistently gives the right answers? That takes months.

Every time I tweaked a hyperparameter—changing the retrieval method, adding a reranker, or adjusting the Top-K value—I found myself playing a guessing game. Did this change actually improve the output, or did it just break a different edge case?

I needed a way to measure the impact of these changes systematically, without relying on expensive cloud observability platforms. That’s why I built Muffakir.

The Problem with RAG Development

When evaluating RAG pipelines, developers face three major blind spots:

1. Lack of Visibility: You see the final answer, but you don't easily see the exact context retrieved, the generated queries, or the step-by-step latency.
2. Experiment Chaos: Running 50 trials with different prompt templates and rerankers usually ends up in messy Jupyter notebooks or scattered logs.
3. Reproducibility Issues: If a configuration works today, can you reproduce the exact same result tomorrow if the underlying documents change?

Enter Muffakir

Muffakir is an open-source, local-first RAG optimization framework. It acts as your control center for building, benchmarking, and comparing RAG pipelines on your own data.

Instead of writing custom evaluation scripts for every project, Muffakir provides a unified interface to tune your system and understand exactly why a specific configuration won.

Key Features Under the Hood:

• ComposerUI: A dedicated local dashboard that lets you define your search space. You can swap out retrievers, test different rerankers, change Top-K limits, and modify prompt templates interactively.
• Granular Execution Traces: For every single trial, Muffakir records the full execution path. You get complete visibility into latency, cost, answer quality, retrieved context, and the exact queries generated by the LLM.
• Reproducible Trials: Stop guessing. Muffakir saves configurations and checkpoints, ensuring you can replicate exact test conditions.
• Temporal Benchmarking Ready: The architecture is designed to handle complex edge cases, such as evaluating how your RAG system reacts when source documents are superseded, corrected, or withdrawn over time.

How It Works

Getting started takes less than a minute. You can install the framework and spin up the ComposerUI directly from your terminal:

# Install Muffakir with standard dependencies
pip install "Muffakir[standard]"

# Launch the UI and start experimenting locally
muffakir serve --open

---

If you are building LLM applications and want to stop flying blind when optimizing your RAG pipelines, I’d love for you to give it a spin!

Check out the repo and give it a star ⭐️: Mohamed28112003/Muffakir

I'm actively looking for feedback, feature requests, and open-source contributors. What do you currently use to evaluate your RAG pipelines? Let me know in the comments!