DEV Community

Cover image for Comparing RAGs, Part 2: the benchmark
Serhiy Kucherenko
Serhiy Kucherenko

Posted on

Comparing RAGs, Part 2: the benchmark

In Part 1, I laid out why "better" RAG isn't about raw accuracy: it's about whether your
data stays inside your own infrastructure or ends up as a standing index on someone else's. Six approaches got tested
head-to-head against the same 10-question golden set, payments-rag's own hand-rolled build, openai-file-search,
Google NotebookLM, and three library builds (Haystack, LlamaIndex, LangChain/LangGraph). Here's how each was
actually built, and what happened when they ran.

Development

The source code lies in a GitHub repo.
But the inner works across all six implementations can be summarized in the following sections.

System Design

System design: one test client drives all six systems into a shared adapter, judge and score

This project consists of the following components:

  1. Corpus: the raw PDFs (three SEPA/ISO 20022 rulebooks), the input every system indexes.
  2. Indexing (one-time, per system): chunk the corpus, embed each chunk, store as vectors. payments-rag/Haystack/LlamaIndex/LangChain all do this themselves; openai-file-search and NotebookLM do it inside the vendor's own managed service instead.
  3. Embedding: turns text (a question, or a chunk) into a vector. The same OpenAI model across every system except NotebookLM, whose embedding step is entirely internal/opaque.
  4. Retrieval: given a question's embedding, find the closest stored chunks. pgvector similarity search for payments-rag; each framework's own equivalent for the rest.
  5. Generation: question + retrieved chunks → an LLM → the answer. Claude for payments-rag, gpt-4o for most framework builds and openai-file-search, Gemini for NotebookLM.
  6. Adapter (comparison-only, not part of any system's real architecture): a thin wrapper per system normalizing each one's very different calling convention into one common shape: question in, {answer, contexts, cost, latency} out. This is what makes the next layer possible across six otherwise-incompatible systems.
  7. Judge/Eval (offline, never per-query): the normalized output gets scored two ways: RAGAS ( faithfulness/relevancy/precision/recall) and a cross-model judge (correctness vs. ground truth). Runs after the fact, on the fixed 10-question golden set.
  8. Client vs. pipeline test client: a real user's question only ever touches payments-rag's own Embedder → Retriever → Generator chain and never reaches eval. The pipeline test client (the golden-set harness) is the only thing that reaches Adapter. It's also the only thing that talks to the other five systems.
  9. Reaching the other five systems: four of them (openai-file-search, Haystack, LlamaIndex, LangChain) have a real API, so the test client calls them directly, same as it calls into payments-rag. NotebookLM doesn't, and it's reached through a Browser instead.

Indexing and Retrieving Text

1. payments-rag

payments-rag sequence: embed, pgvector search, Claude answer

2. openai-file-search

openai-file-search sequence: one-time upload to a persistent OpenAI vector store, then per-question retrieval inside the API

3. Google NotebookLM

NotebookLM sequence: manual upload and manual copy-paste per question

4. Haystack

Haystack sequence: PyPDF load, split, embed into in-memory store, retrieve top 5, generate

Haystack bug: one malformed 12,000-word 'sentence' exceeds the embedding size limit and the chunk is silently dropped

Haystack's sentence-based chunker treated a malformed PDF-extracted block as one giant "sentence," pushing it past
OpenAI's embedding size limit. As a consequence, Haystack silently dropped that chunk instead of raising an error,
quietly losing a third of the index. Fixed by switching to Haystack's own word-based chunking default.

5. LlamaIndex

LlamaIndex sequence: pypdf load, sentence split, VectorStoreIndex, query engine

LlamaIndex bug: raw PDF metadata and binary noise leaked into extracted text, outranking real answers

LlamaIndex's default PDF reader leaked raw PDF metadata into extracted text and mangled some pages into unreadable
binary noise, both silently, without an error thrown. The noise sometimes outranked the real answer during retrieval.
Fixed by loading PDFs with pypdf directly, the same library Haystack already used cleanly.

6. LangChain / LangGraph

LangChain/LangGraph sequence: pypdf load, recursive splitter, retrieve and generate nodes in a compiled graph

Results

Results: judge score, RAGAS metrics, cost and latency for all six systems

LlamaIndex is the best in both judge and faithfulness scores.

OpenAI file search cost $0.3526 for the same ten questions, which is the most expensive system in the comparison
by a wide margin: over 8x pricier than the next-priciest option (Haystack, $0.0413), and roughly 13x pricier than
payments-rag's own hand-rolled approach ($0.0276). Likely because it stuffs more retrieved context into every request
than expected.

NotebookLM costs nothing in dollars, but it required 11 min of setup work of clicking
before the first question could even be asked. Meaning, it just moves the cost from money to time.

NotebookLM ties LlamaIndex for the highest answer-relevancy of all six, despite its other RAGAS metrics being
unusable. LangChain, the most popular framework by mindshare, placed second-worst, just above Haystack.
Haystack's remaining weakness is a retrieval bias toward one source document on standard-SCT questions, not a
grounding failure.

Conclusion

Some numbers worth pulling forward before the caveats:

  • own built payments-rag placed 4th of 6 on judge-scored accuracy (84.8), behind LlamaIndex (96.5), NotebookLM (92.5), and openai-file-search (90.2).
  • it was also the cheapest ($0.0276) and fastest (2.68s) system in the whole comparison. For instance, OpenAI markup caused it to cost about 8-13x, while NotebookLM took 11 minutes to set up.
  • silent bugs may creep simply because of how the systems are wired up as it happened to LlamaIndex and Haystack. In this case it was about clean corpus and a sane chunk size.

The obvious flaw of this stage is a small set of questions (or maybe topics of questions) of a golden set.
Additionally, the compared options could be expanded dimension-wise by combining different models
(both embedding, llm, judge).

However, this intel brings some clarity on why we might end up having RAGs everywhere as well as concerns about both
correctness and security. Building a working and useful RAG is one effort. Building one at scale is a different thing.
And building one for privacy is yet another problem. The only way to fully avoid exposure is self-hosting all
the layers: the embedding, the generation models, and the retrieval layer. payments-rag's own hand-rolled setup still
sends chunks to OpenAI and Anthropic at embed- and generation-time (see the exposure diagram in Part 1).

Out of Scope

  • Bigger and more diverse golden sets, additional metrics beyond RAGAS/judge
  • Testing every embedding/LLM/judge model combination for full experimental cleanliness: this run fixed one model set throughout
  • NotebookLM's paid Enterprise tier: investigated, doesn't change the conclusion (still no way to ask it a question programmatically)

Also, that was made on Python. I'd imagine someone doing it, say, on C++ or Rust (whatever is better) would make it
faster.

Sources

Top comments (0)