Why My RAG App Uses BM25, Vectors, Parent-Child Chunks, and Chat Memory Together
PROJECT REPOSITORY: Open
Most RAG projects start with a very confident idea:
“Let us split the PDF, embed the chunks, store them in a vector database, and ask questions.”
Beautiful. Elegant. Wrong often enough to be annoying.
That was also my first idea while building Docflow, a multi-user document question-answering app where users upload PDFs/images and chat with their own files. The application processes documents, stores chunks in Qdrant, retrieves relevant context, and sends that context to an LLM.
The first version looked simple:
And honestly, for a simple scenario with three clean paragraphs, this works nicely. Unfortunately, my documents were not three clean paragraphs. They were college lecture notes, policy PDFs, invoice files, scanned documents full of headers, footers, tables, page numbers, short codes, and lines like:
Company Confidential
Employee Benefits Guide
Page 7 of 42
Very official. Also very noisy.
So the retrieval system had to grow up a little.
Final retrieval design:
This blog explains why each piece exists.
The Project In One Minute
Docflow is a full-stack RAG application.
Users can:
- register and log in
- upload PDFs/images
- manage uploaded files
- create multiple chats
- ask questions over only their own documents
- continue old chats with memory
- delete files from storage and index
The system uses:
- FastAPI for backend APIs
- React for UI
- Celery + Redis for async document processing
- S3/MinIO for raw file storage
- Qdrant for vector search
- BM25 for keyword retrieval
- Groq LLM for final answers
- SQLite/PostgreSQL for users, files, chats, and messages
The important part: retrieval is user-scoped. A user should never retrieve another user’s document chunks. That is not a feature. That is a lawsuit warming up.
Problem With The First Naive Idea: Just Embed Chunks And Search Qdrant
The first plan was simple:
- Extract text from a PDF.
- Split it into chunks.
- Convert chunks into embeddings.
- Store vectors in Qdrant.
- Embed the user query.
- Retrieve nearest chunks by cosine similarity.
- Send those chunks to the LLM.
This is the classic semantic search flow. Sentence Transformers describes semantic search as embedding both the query and corpus into the same vector space, then finding the closest embeddings by semantic similarity: Sentence Transformers Semantic Search.
That already gives a decent system.
Vector search is good at meaning.
If the document says:
Employees can request paid leave after completing the probation period.
And the user asks:
When am I eligible for paid time off?
A vector model may understand that paid leave and paid time off are related.
Might.
That word is carrying a lot of emotional damage.
Semantic search can struggle with:
- acronyms:
PTO,SLA,KYC,GST - policy codes:
HR-204,SEC-17,INV-009 - exact clause names
- invoice numbers
- section titles
- short technical queries
- rare terms that are important
BM25: Because Exact Words Still Matter
To fix this, I added BM25.
BM25 is a classic lexical ranking method. It scores documents based on query term matches, term frequency, inverse document frequency, and document length.
Real Example
Suppose the uploaded document contains:
Payment terms: Net 30. The customer must complete payment within 30 calendar days from the invoice date.
Late payments may incur a 2% monthly service charge.
User asks:
What are the Net 30 payment terms and late fee?
Vector search may retrieve semantically similar billing chunks.
BM25 strongly boosts chunks containing:
Net
30
payment
late
fee
That gives better recall for technical phrases, policy names, codes, and numbers.
So now we have:
Vector search -> meaning
BM25 search -> exact terms
Combining Rankings With RRF
Now we have two ranked lists:
Vector results:
1. chunk A
2. chunk B
3. chunk C
BM25 results:
1. chunk C
2. chunk D
3. chunk A
How do we combine them?
Bad idea:
final_score = vector_score + bm25_score
Why bad?
Because vector similarity and BM25 score are not on the same scale. Adding them directly is like adding kilograms and degrees Celsius.
So I used Reciprocal Rank Fusion, or RRF.
RRF combines rankings using rank positions instead of raw scores: Reciprocal Rank Fusion paper.
The scoring idea:
score(document) = sum over rankings: 1 / (k + rank)
This gives a nice behavior:
- If a chunk ranks high in both vector and BM25, it rises.
- If a chunk ranks high in only one method, it can still survive.
- We do not care about incompatible score scales.
Simple. Effective.
Why Parent-Child Chunking Is Better
The next problem: what size should chunks be?
Small chunks are great for retrieval:
"Late payments may incur a 2% monthly service charge."
This is focused and searchable.
But small chunks are bad for final answering because the LLM may miss surrounding context:
Which invoice does this apply to?
Is the 2% charge monthly or one-time?
Does the 30-day period start from invoice date or delivery date?
Are there exceptions?
Large chunks are better for context, but worse for search. A giant chunk may contain the right answer buried under three pages of policy text, one approval matrix, and a footer reminding you that the document is confidential. Very searchable. Obviously.
So I use parent-child chunking.
Parent chunk: larger block, sent to LLM
Child chunk: smaller block, used for retrieval
Structure:
parent: abcd1234-p0
child: abcd1234-p0-c0
child: abcd1234-p0-c1
child: abcd1234-p0-c2
Indexing stores both:
{
"id": chunk["id"],
"text": chunk["text"],
"type": "child",
"parent_id": "abcd1234-p0",
"user_id": user_id,
"file_id": file_id,
"job_id": job_id,
"filename": filename,
}
Search only looks at children:
FieldCondition(key="type", match=MatchValue(value="child"))
Then after RRF, the system fetches parent chunks:
parent_ids = [hit["parent_id"] for hit in fused_hits]
parents = retrieve_by_ids(parent_ids)
This gives the best of both worlds:
Child chunks -> precise search
Parent chunks -> useful LLM context
This matters a lot for real documents, where a rule, exception, effective date, and approval condition may be spread across nearby lines. Sending only the tiny matching sentence can make the LLM answer like it skimmed the policy during lunch and decided that was enough.
Answering Follow-Up Questions
The user does not always ask beautiful standalone questions.
They ask:
What is the company policy for remote work?
Then:
Who needs to approve it?
Humans understand it. A stateless API does not.
So each user can create multiple chats. Each chat has messages stored in the database:
chats
messages
When a new message arrives, Docflow loads recent messages:
history = _recent_chat_history(chat_id, user_id, limit=12)
Then it builds a retrieval query using recent context, so that the retrieval layer sees:
user: What is the company policy for remote work?
assistant: ...
user: Who needs to approve it?
Now the search query has enough context to understand that it probably means remote work.
Important design choice:
Chat history helps understand the question. It is not treated as factual source material.
That distinction matters. The LLM should not invent facts from conversation history. The uploaded documents remain the authority.
Grounded Answers And Fallback Behavior
The system prompt tells the model:
Use only the current user's retrieved document context for factual answers.
Use conversation history only to understand follow-up references.
If the answer is not in the context, say:
"I could not find this information in the provided documents."
If the user asks for an example and no example appears in the context,
say that no source example was found before giving a clearly labeled general example.
This is there because RAG systems should not pretend the document said something it did not say.
For example, if the document explains the expense reimbursement policy but gives no sample reimbursement scenario, and the user asks:
Give an example for it.
A good answer is:
I could not find an example in the provided documents.
General example:
If an employee spends ₹1,200 on approved client travel and uploads the receipt within the required period, the finance team may reimburse the amount after manager approval.
That is much better than:
The document says...
when the document absolutely did not say it. Classic LLM confidence.
How The Final Retrieval Flow Looks
Here is the final flow:
1. User asks a question
2. API loads recent chat history
3. API creates retrieval query
4. Query is embedded
5. Qdrant finds semantic child chunk matches
6. BM25 finds exact keyword child chunk matches
7. RRF combines both rankings
8. Best child chunks are mapped to parent chunks
9. Parent chunks become LLM context
10. LLM answers with sources
11. Chat history is saved
Takeaway
The main lesson from building Docflow:
RAG quality depends more on retrieval design than on just calling a powerful LLM.
Embeddings are useful, but they are not magic. BM25 is old, but old does not mean useless. Parent-child chunking sounds fancy, but it solves a very real context problem. Chat memory is necessary because users ask follow-up questions like normal humans. Grounded fallback behavior is needed because LLMs are very comfortable saying things with full confidence and zero evidence.
Final design:
Vectors for meaning
BM25 for exact terms
RRF for rank fusion
Child chunks for retrieval
Parent chunks for context
Chat history for follow-ups
Source-grounded prompting for honesty
A RAG app is not impressive because it uses embeddings. Everyone and their grandma can do that now.
It becomes impressive when it handles the messy parts:
- noisy PDFs
- exact terminology
- multi-user isolation
- follow-up questions
- missing source evidence
- explainable retrieval decisions
That is where the engineering starts.
Top comments (0)