Serhiy Kucherenko

Posted on Sep 5

Comparing RAGs, Part 1: the framework

In the past, I built my own RAG. The idea was to implement and debug concepts rather than read documentation. So far, a RAG is supposed to address this issue for a person or a company:

AS an engineer,
I WANT TO get accurate data on need
SO THAT I get the exact answer, with a citation, as fast as possible

This is the problem a RAG tries to solve. Since this is an engine that also includes a database, safety and scalability concerns apply. There are multiple options, and the problem might be: which option is the best?

Hypothesis

We could categorize solutions by usage and integration:

Manual personal interaction services Google's like NotebookLM
API services like OpenAI's file_search
Libraries like Haystack
Custom solution

Typically, if a system needs scale (imagine lots of people constantly surfing through documentation which could be indexed in a RAG), the choice lies between building an own solution using a library and a cloud solution from another company.

Warning: a claim like "no need for expertise because we have AI" means we have no way to verify if a clue detail was
missed (described at Judge Strictly Avoid Hallucinations).

Safety and Politics as concerns

Recent worldwide events have shown that depending on another company is not only a security vulnerability but also a political one. This matters because while it might be easier and work better initially to just dump everything into a cloud solution of another enterprise, this creates a single point of failure in many ways. Also tends to be quite expensive in the long run.

Maintaining an own solution is not cheap either: you will have an internal service with a team maintaining it, and with the pace the AI tech progresses, you risk falling behind already in a few months if not weeks.

Define "better"

So... which solution would work better, and how a RAG would benefit us?

Before comparing anything, it's worth being precise about what "better" even means here. It isn't rather who answers questions most accurately, as we will end up just comparing frontier models and context windows on a ten-question set. The real question is integration: can this run inside a system where data is private?

The Judge

The most helpful way is to use another LLM as a judge with strict scoring, to avoid any false positives, as these are the most dangerous silent killers of confidence. So a gpt-4o with text-embedding-3-small will do just fine as a different vendor from the Claude-generated answers this project produces, which avoids grading its own homework.

RAGAS

We will also use [RAGAS] https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) 0.4.3 for evaluation. RAGAS was picked as the scoring method for the comparison itself specifically because it is a library, in-process, introduces no new recipient of corpus data, and has a documented telemetry kill switch.

This standard quartet of metrics will do:

Faithfulness: does the answer only say things the retrieved chunks actually support?
ResponseRelevancy: does the answer address the question asked, not something adjacent?
LLMContextPrecisionWithoutReference: of what got retrieved, how much was actually relevant?
LLMContextRecall: of what should have been retrieved, how much actually was?

Note: even the scoring library itself shipped broken. A plain pip install ragas crashes on import with ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai', caused by RAGAS's own code imports a path that newer versions of one of its own dependencies removed. Real, still-open bug on RAGAS's side
(issue #2753, fix still unmerged), not something this project did wrong. Worked around by pinning an older langchain-community.

Issue: Transient vs. Persistent Exposure

That's a second axis worth noting. Transient means a request goes out, a vendor's model processes it, and nothing about your corpus survives on their infrastructure afterward as a queryable asset.

Persistent means the opposite: your corpus (or an embedding of it) gets stored on someone else's infrastructure as a standing index that outlives the call that created it and keeps being a breach target indefinitely. OpenAI's file_search and Google's NotebookLM both fall into the second category: uploading your documents creates a managed index that exists independently of any single question you ask. That exposure means your private data may be breached or even secretly used as training by host companies.

Note: a RAG already accepts some transient exposure, as we will end up sending a chunk (100-1000 words, which is a typical size of a chunk).

sequenceDiagram
    participant You as Your infrastructure
    participant Embed as Embedding model (cloud API)
    participant LLM as Answering LLM (cloud API)
    Note over You, LLM: Embed-time (chunk text leaves your infra)
    You ->> Embed: chunk text (index + query)
    Embed -->> You: embedding vector
    Note over You: stored/searched locally (Postgres/pgvector)
    Note over You, LLM: Generation-time (chunk text leaves your infra again)
    You ->> LLM: chunk text (as prompt context)
    LLM -->> You: answer

The only way to avoid exposing your data completely is to host the embedding, the answering model, and the judging model yourself, at a cost of having weaker models and continuous maintenance.

Issue: Integration

Another issue is how easy it is to integrate a RAG into your knowledge base. Typically, a team has a separate ticketing platform, data, and a logging storage. Also, it has a company-wide context, dependencies on other teams, documentation, alerts. That presents a set of challenges when trying to find a necessary set of pieces of info. As well as an issue with integrating such a solution as RAG and maintaining the relevance of its data.

Besides, each team will have their unique and individual stuff like specified above. Meaning, having a single same corpus for all teams is not possible.

Note: As of August 2026 NotebookLM only works by hand, through its own web page, there's no way to script it or plug it into anything else
even on the paid enterprise tier. Google added an API in 2025, but it still can't be asked a question through it. Regardless, given the pace AI tech evolves, it is still included in this comparison to keep an eye on.

The six systems compared, and how each was queried

payments-rag: the project's own production path, Claude-generated answers over its own pgvector retriever.
openai-file-search: OpenAI Responses API, model="gpt-4o", tools=[{"type": "file_search", "vector_store_ids": [...]}], corpus PDFs uploaded once to a named OpenAI vector store
NotebookLM: queried through its web UI, answers and citation filenames captured by hand
Haystack: a RAG pipeline built with the deepset library instead of hand-rolling it with the same corpus and models. It is the framework that does the orchestration.

Note: Haystack ships with anonymous, opt-out telemetry (on by default), though it sends only component types (e.g., which retriever/store you used), no personal data. To turn it off: HAYSTACK_TELEMETRY_ENABLED=False.

LlamaIndex: the same idea, built with a different library, included to see whether "framework vs. framework" matters as much as "framework vs. hand-rolled"
LangChain/LangGraph: a RAG pipeline built with LangChain's components (loader, splitter, vector store, retriever), with LangGraph doing the orchestration instead of a hand-rolled function or the other frameworks' own pipeline objects

The comparison was based on answers of the same 10-question golden set that was used throughout the development of payments-rag.

Part 2 covers how each of these was actually built, and what broke along the way.

Publishing Monday.

DEV Community