These days, you hear about RAG pretty much everywhere. Open GitHub, spend a couple of minutes browsing, and you'll inevitably stumble upon yet another "AI-powered knowledge base" that can ingest PDFs, dump text into a vector database, and confidently answer questions.
It all sounds fairly straightforward: take some text, split it into chunks, turn those chunks into vectors, connect a large language model (LLM), and you're done.
At least until your first query returns something completely different from what you expected.
That's exactly what I ran into when I started building Dalil, a search engine for Russian-language religious documents, for a client.
The original task was pretty specific: let users ask questions in natural Russian and find the relevant religious ruling among thousands of documents. Not just by matching keywords, but by meaning.
Sounds like another simple RAG project, right?
Well, I quickly discovered that there's quite a distance between "Hey, it actually finds something!" and "I can trust these results."
Where It All Started
First, a quick explanation of RAG.
RAG stands for Retrieval-Augmented Generation. In simple terms, the system searches for relevant documents and then passes them to a language model, which generates an answer based on the retrieved information.
This allows the LLM to rely on external sources rather than just whatever it learned during training.
In my case, I was primarily interested in the first part: retrieval.
I started with a dataset of religious rulings in JSON format. Each record contained a title, question, answer, footnote, link to the original source, and category information.
The first thing I did was write a small Python pipeline to convert the original JSON into JSONL, a format where each document occupies a separate line.
Why? So I could process records one at a time instead of loading everything into memory. No need to make the RAM suffer unnecessarily.
This stage also required some text cleanup, since the dataset contained both Russian and Arabic.
Not every record was encoded properly, so I added text validation. The script removed unwanted control characters, handled problematic Unicode sequences, and sent records that couldn't be processed into a separate file rather than letting them interfere with everything else.
I know, not exactly the most exciting engineering achievement.
But these are usually the little things you remember when a script crashes halfway through processing a dataset.
Next came chunking.
Chunking means splitting documents into smaller pieces that can be indexed separately. In many RAG examples, text is divided by character or token count, with some overlap between chunks, before being passed to an embedding model.
But my records already had a fairly convenient structure: title, question, answer, and footnote.
So instead of splitting the text arbitrarily, I decided to use that structure.
Each record became two chunks:
- The first contained the title and question.
- The second contained the answer and footnote.
This allowed the system to search independently through question formulations and the answers themselves.
Whenever a relevant chunk was found, both chunks could be reunited using the original record ID.
That mattered because I didn't want users receiving random paragraphs stripped of their context.
Turning Text Into Numbers
Once the corpus was ready, the next challenge was getting the system to compare texts by meaning.
That's where embeddings come in.
Embeddings are numerical representations of text in a multidimensional space. Roughly speaking, an embedding model turns a sentence into a long array of numbers, where semantically similar texts tend to end up closer together.
For indexing, I used OpenAI's text-embedding-3-large with 3,072-dimensional vectors, and Qdrant for storage.
Each chunk received its own vector, text, and metadata.
I also generated point IDs deterministically from the original record ID and chunk number. This meant that reimporting the same records didn't require generating new identifiers for the same chunks.
The pipeline looked roughly like this:
JSON → JSONL → Chunks → Embeddings → Qdrant
On the user's side, the process worked in reverse.
A user entered a question, the system converted it into an embedding, and Qdrant returned the closest vectors.
The early results looked promising.
The search could find documents that used completely different wording but discussed similar subjects.
And that's where the first real problem appeared.
Semantic similarity doesn't necessarily mean a document answers the question.
Two texts might discuss the same religious topic, use the same terminology, and even describe similar actions, while differing in circumstances, conditions, or the actual subject of the ruling.
In an ordinary search engine, that's annoying.
In a search engine dealing with religious or legal rulings, the consequences can be considerably more serious.
Why I Decided Not to Generate Answers With an LLM
At this point, the obvious next step would have been to connect a language model, pass it several retrieved documents, and ask it to produce a nicely worded answer with citations.
That's the standard RAG approach.
But I deliberately decided against using generation in the user-facing search experience.
The problem is that an LLM can produce a perfectly convincing summary while overlooking an important condition or combining several different rulings into one answer.
And when an answer sounds confident enough, users might not even notice that the original meaning has changed.
So in Dalil, the system doesn't generate answers to users' questions.
Instead, it retrieves existing documents, ranks them, and returns the original source.
What started as a RAG project effectively became a semantic search engine with hybrid ranking.
Technically, that description is more accurate than calling it RAG, but RAG is still where the whole project began.
Of course, removing generation didn't solve the main problem.
Now I had to make the retrieval itself genuinely reliable.
When Vector Search Alone Wasn't Enough
The initial implementation relied on dense embeddings.
A query was converted into a vector, and Qdrant retrieved the nearest documents using cosine similarity.
For a prototype, this worked reasonably well.
But in a specialized corpus, certain words and concepts matter not only because of their general meaning, but because of their exact terminology.
Vector models are good at capturing semantic similarity, but sometimes they interpret it a little too freely.
Keyword search, on the other hand, can identify precise wording but struggles when users phrase the same question differently.
Eventually, I moved to hybrid search.
Instead of relying on a single retrieval mechanism, I used two:
- Dense retrieval — semantic search using OpenAI embeddings.
- Sparse retrieval — lexical retrieval using BM25 through FastEmbed.
The next challenge was combining their results.
For that, I used Reciprocal Rank Fusion, or RRF.
The idea is fairly simple: RRF merges several ranked result lists by taking each document's position in those lists into account.
A document that ranks well across multiple retrieval methods gains an advantage.
In practice, this helped the system consider both the meaning of a query and exact matches for specialized terminology.
But then I ran into a rather funny problem.
It turned out I'd been treating the RRF score as a measure of relevance.
Except that's not what RRF is designed to measure.
RRF evaluates a document's position in the combined ranking, not the probability that it actually answers the query.
Those are two completely different things.
Imagine a swimming competition with three contestants, none of whom can swim.
One of them will still finish first.
That doesn't make them an Olympic champion.
I was seeing essentially the same behavior in my search results.
So I had to separate ranking from relevance validation.
First, I introduced an additional cosine similarity check using the dense vectors.
This allowed me to evaluate semantic similarity independently of the document's position in the RRF ranking.
For regular queries, I used a threshold of 0.32, while shorter queries used 0.20.
The lower threshold for short queries was necessary because one- or two-word searches often produced lower similarity scores, and an overly strict filter could eliminate useful results.
But that filter alone wasn't enough.
Which brought me to the next stage: reranking.
Reranking: A Second Exam for Search Results
After hybrid retrieval, Qdrant returns a list of candidate documents.
But who says those candidates are already in the best possible order?
That's where I introduced Cohere Rerank through its API.
Unlike an embedding model, which independently converts queries and documents into vectors, a reranker evaluates the relationship between a query and each candidate document more directly.
It receives the query and candidate text, then assigns a relevance score.
The architecture now looked like this:
Query → Normalization → Dense/Sparse Retrieval → RRF → Cohere Rerank → Results
I also introduced another relevance threshold after reranking.
If the best candidate received a score that was too low, the system returned an empty result set.
And yes, an empty result is not always an error.
Sometimes the correct behavior of a search engine is to admit that it couldn't find a suitable document.
Especially when the alternative is confidently showing users something that doesn't actually answer their question.
Of course, thresholds like 0.32, 0.20, and 0.09 aren't universal constants of search quality.
They're specific to this particular retrieval pipeline and need to be validated against queries from its own domain.
A Small Dictionary Turned Out to Be Useful Too
Another challenge was handling the way people actually formulate questions.
People don't write like neatly organized datasets.
They make typos, use informal expressions, and choose different words for the same concepts.
So I added a normalization stage before retrieval.
The system cleans up unnecessary whitespace, uses a dictionary of religious synonyms, applies lemmatization through pymorphy3, and performs fuzzy matching with rapidfuzz.
One more thing I learned during development: not every search problem requires another neural network.
Sometimes a decent dictionary and a few lines of Python are enough.
Not nearly as impressive on paper, but it works :)
What Happens if Cohere Stops Responding?
Once you introduce an external API into your retrieval pipeline, you've also introduced another potential point of failure.
The reranker might become slow, return an error, or stop responding altogether.
But making the entire search engine unavailable just because one external service has decided to take a break would be a strange design choice.
So I implemented graceful degradation.
The idea is that the system should continue functioning, even at reduced quality, when an optional component becomes unavailable.
For Cohere, I set a three-second timeout.
I also added a circuit breaker. After repeated failures, the system temporarily stops calling the problematic service and later attempts to restore normal operation.
If the reranker is unavailable, the system can still return results from Qdrant's hybrid retrieval.
I also added reranking result caching and separate operational statuses:
- Successful request
- Timeout
- Error
- Open circuit breaker
- Skipped request
In other words, I started monitoring not only what the search engine returned, but also the state of the retrieval pipeline at the time of each request.
From a Few Thousand Documents to More Than 10,000
As the project evolved, the document corpus grew as well.
Initially, the system worked with several thousand records.
Later, I moved to an updated corpus containing more than 10,000 documents with revised translations.
At first glance, this sounds simple enough: generate new embeddings, upload them to Qdrant, and carry on.
But by that point, the project already had public document pages, categories, internal relationships between records, and persistent URLs.
And the data was stored in two separate systems.
Qdrant handled the search index, while PostgreSQL stored the canonical records used by the public pages and application.
That turned a seemingly straightforward dataset update into a full migration.
I created a new collection called records_ru_v2, keeping the previous collection as a fallback.
Existing slugs were preserved, while new documents received additional URLs.
I also prepared a rollback mechanism using the previous Qdrant collection and a PostgreSQL backup.
So this was no longer just a matter of uploading a few thousand extra vectors.
It required updating the search index, document database, and public-facing application without breaking their consistency.
Why I Started Measuring Search Quality Separately
As the system became more complex, it became obvious that simply logging a query and its top result wasn't enough.
For example, when a user gets no results, it's important to understand why.
Was there genuinely no relevant document?
Did the dense similarity filter eliminate the candidates?
Did the reranker assign a low score?
Or did Cohere simply fail to respond in time?
So I expanded the search analytics to capture additional information:
- Normalized query
- Cosine similarity
- Reranker score
- Reranker execution status
One particularly important change was stopping the use of RRF's best_score as a quality metric.
Ranking and relevance are not the same thing.
Mix them up, and you can spend a long time celebrating beautiful numbers while your users keep finding the wrong documents.
I also introduced a separate offline evaluation mechanism to test the retrieval pipeline against a set of test queries.
This is probably one of the most important parts of the entire development process.
Because replacing one embedding model with another or adding another ranking stage is relatively straightforward.
What's much harder is answering a simple question objectively:
Did the search actually get better?
What I Learned From Building It
When I first started working on Dalil, I didn't expect document search to require so many additional engineering decisions.
The search system now uses a corpus of over 10,000 documents, structure-aware chunking, vector and lexical retrieval through Qdrant, RRF for result fusion, additional relevance filtering, and Cohere reranking.
The application itself runs on FastAPI, PostgreSQL, Next.js, and a Telegram Mini App.
At first, I thought of RAG as a fairly well-defined task: prepare documents, generate embeddings, and retrieve the nearest vectors.
Over time, I realized that was only the beginning.
You have to account for language-specific details, distinguish semantic similarity from actual relevance, reduce false positives, handle external service failures, preserve links to original sources, and find ways to evaluate your own changes.
And perhaps the most important conclusion I came to is this:
A good search engine shouldn't just know how to find answers. It should also know when to return nothing.
You can make an LLM sound extremely convincing.
Search is a different challenge. It doesn't need to sound convincing. It needs to be accurate.
And that's what ended up taking far more of my time than the original RAG prototype.
If you'd like to see how it works with real questions, you can try the project at Dalil.
Top comments (0)