A basic Retrieval-Augmented Generation (RAG) demo is surprisingly small:
- Embed some documents.
- Retrieve the closest chunks.
- Add them to a prompt.
- Ask an LLM to generate an answer.
But when I turned that flow into an API, the LLM call became the least interesting part.
I needed to process PDFs without blocking requests, combine semantic and keyword search, rerank noisy results, preserve source metadata, cache answers, and secure the API.
So I built a PDF question-answering backend with:
- FastAPI
- Qdrant
- PostgreSQL
- Redis
- PyMuPDF
- LiteLLM
- FastEmbed
- A cross-encoder reranker
This article focuses on the most interesting part: the path from a user’s question to a grounded answer.
The architecture in one minute
The application has two main workflows.
Document ingestion
When a client uploads a PDF, the API:
- Validates the file type and size
- Creates a document record in PostgreSQL
- Returns a document ID immediately
- Extracts text in a background task
- Splits the text into chunks
- Generates dense and sparse vectors
- Stores the vectors and metadata in Qdrant
Question answering
When a question arrives, the API:
- Checks Redis for a cached response
- Generates dense and sparse query representations
- Runs both searches in Qdrant
- Combines the rankings with reciprocal rank fusion
- Reranks the best candidates with a cross-encoder
- Sends the top chunks to an LLM
- Returns the answer with its sources
Here is the complete query flow:
┌─────────────────┐
│ User question │
└────────┬────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Dense embedding│ │ Sparse vector │
└───────┬────────┘ └───────┬────────┘
│ │
└───────────┬────────────┘
▼
┌───────────────────┐
│ Qdrant + RRF │
│ 20 candidates │
└─────────┬─────────┘
▼
┌───────────────────┐
│ Cross-encoder │
│ Top 5 chunks │
└─────────┬─────────┘
▼
┌───────────────────┐
│ Grounded prompt │
└─────────┬─────────┘
▼
┌───────────────────┐
│ LLM answer │
└───────────────────┘
Why I used two retrieval methods
Dense embeddings are good at retrieving text by meaning.
For example, a semantic search system may recognize that these sentences are related:
"How are API credentials invalidated?"
"How can I revoke an access key?"
The wording is different, but the intent is similar.
Technical documents also contain exact lexical signals:
- Error codes
- Function names
- Version numbers
- Product names
- Abbreviations
- Configuration keys
A semantic model may not always preserve the importance of an identifier such as:
ERR_AUTH_0042
Sparse retrieval helps with those exact words and identifiers.
Instead of choosing between semantic and lexical retrieval, I store both representations for every chunk:
PointStruct(
id=point_id,
vector={
"dense": dense_vector,
"sparse": SparseVector(
indices=sparse_vector["indices"],
values=sparse_vector["values"],
),
},
payload={
"text": chunk_text,
"source": filename,
"document_id": str(document_id),
"page_number": page_number,
"chunk_index": chunk_index,
},
)
Each Qdrant point contains:
- A dense vector
- A sparse vector
- The original chunk text
- The source filename
- The document ID
- The page number
- The chunk index
Keeping provenance next to the vectors makes it possible to return useful sources with each answer.
Combining both searches with RRF
Dense and sparse searches produce different score scales.
Adding their raw scores directly would require normalization and tuning. Instead, I use reciprocal rank fusion, or RRF.
RRF focuses on where a result appears in each ranked list rather than directly comparing the original scores.
The hybrid query looks like this:
response = await qdrant_client.query_points(
collection_name="embeddings",
prefetch=[
Prefetch(
query=dense_query_vector,
using="dense",
limit=limit * 4,
),
Prefetch(
query=SparseVector(
indices=sparse_query["indices"],
values=sparse_query["values"],
),
using="sparse",
limit=limit * 4,
),
],
query=FusionQuery(fusion=Fusion.RRF),
limit=limit,
with_payload=True,
)
Qdrant executes the dense and sparse searches and then fuses their rankings.
This allows a chunk to rank well because it:
- Matches the meaning of the question
- Contains important exact terms
- Or performs reasonably well in both searches
Hybrid retrieval is not automatically better for every dataset. Its value depends on the documents, query patterns, embedding models, and search configuration. It still needs evaluation against real questions.
Retrieve broadly, then rerank narrowly
Initial retrieval needs to be fast enough to search the full collection.
It does not always need to produce the final ordering.
My pipeline retrieves 20 candidates and sends them to a cross-encoder:
pairs = [
(query, candidate["text"])
for candidate in candidates
]
scores = reranker.predict(pairs)
The candidates are sorted using those scores:
reranked = sorted(
zip(candidates, scores),
key=lambda item: item<span class="footnote-wrapper">[1](1)</span>,
reverse=True,
)
top_chunks = [
candidate
for candidate, score in reranked[:5]
]
Unlike independent vector embeddings, a cross-encoder examines the question and candidate together.
That can produce a more precise relevance score, but it is also more computationally expensive. This is why I use it only after the initial retrieval stage.
The pipeline narrows the context like a funnel:
Hybrid retrieval ████████████████████ 20 candidates
Cross-encoder output █████ 5 chunks
LLM context █████ 5 chunks
These bars show the candidate counts configured in the code. They are not benchmark results or accuracy measurements.
Grounding the final answer
After reranking, the five best chunks are joined into a context block.
The prompt tells the model to use only that context:
system_prompt = (
"Answer the question using only the provided context. "
"If the answer is not present in the context, say that "
"the available documents do not contain enough information."
)
user_prompt = f"""
Context:
{context}
Question:
{question}
"""
This instruction establishes a clear contract:
- Retrieved chunks provide the evidence.
- The LLM turns that evidence into an answer.
- Missing evidence should produce an explicit limitation.
A prompt cannot guarantee factual correctness. If retrieval returns irrelevant chunks, the generator still receives poor evidence.
That is why I think of RAG quality as a chain:
Document quality
×
Chunk quality
×
Retrieval quality
×
Reranking quality
×
Generation quality
=
Final answer quality
A strong LLM cannot fully compensate for a weak retrieval pipeline.
Keeping PDF processing outside the request
PDF ingestion includes several expensive operations:
- Parsing the file
- Splitting the text
- Generating embeddings
- Writing vectors to Qdrant
- Writing records to PostgreSQL
I did not want the upload request to remain open during that work.
The endpoint creates the document record and schedules processing as a FastAPI background task:
background_tasks.add_task(
process_document,
document_id,
pdf_bytes,
file.filename,
)
return {
"document_id": str(document_id),
"filename": file.filename,
"processing_status": "processing",
}
The client receives a response immediately and can check the status later:
GET /documents/{document_id}
The document moves through states such as:
processing → completed
↘ failed
This is enough for a prototype, but an in-process background task is not a durable job queue.
If the API process stops, accepted work may be interrupted.
For a more dependable version, I would move ingestion to a dedicated worker system with:
- Automatic retries
- Idempotent jobs
- Concurrency controls
- Failed-job recovery
- Cleanup for partial writes
- Dead-letter handling
Caching is also a correctness problem
Completed answers are cached in Redis for 24 hours.
The current cache key is based on the question:
digest = hashlib.sha256(
question.encode("utf-8")
).hexdigest()
cache_key = f"rag:{digest}"
This is simple, but incomplete.
The same question can produce a different answer when any of these change:
- The indexed documents
- Retrieval filters
- The embedding model
- The reranking model
- The generation model
- The system prompt
- The tenant or user scope
A safer cache key would include those dependencies:
cache_input = {
"question": normalized_question,
"corpus_revision": corpus_revision,
"filters": filters,
"embedding_version": embedding_version,
"reranker_version": reranker_version,
"prompt_version": prompt_version,
"generation_model": generation_model,
}
serialized = json.dumps(
cache_input,
sort_keys=True,
)
digest = hashlib.sha256(
serialized.encode("utf-8")
).hexdigest()
Caching is not just a performance optimization. A stale cache can return an answer that no longer reflects the current knowledge base.
The backend around RAG still matters
The retrieval pipeline is only one part of the service.
The API also includes:
- Cryptographically generated API keys
- SHA-256 key hashing
- Revocable credentials
- Admin-secret validation
- Per-caller rate limits
- Document status tracking
- Structured logging
- Redis response caching
- PostgreSQL operational records
The full system looks more like a backend platform than a single AI function:
┌─────────────┐
│ API client │
└──────┬──────┘
│
┌──────▼──────┐
│ FastAPI │
└──────┬──────┘
┌─────────────────┼─────────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ PostgreSQL │ │ Redis │ │ Qdrant │
│ Status/Auth │ │ Cache │ │ Retrieval │
└─────────────┘ └─────────────┘ └──────┬──────┘
│
┌──────────▼──────────┐
│ Reranker and LLM │
└─────────────────────┘
The LLM may be the most visible component, but most reliability problems live around it.
What I would improve next
The next version would focus on measurement and failure recovery.
Durable ingestion
I would replace in-process background tasks with a proper worker queue.
Idempotent writes
Stable point IDs would make retries safer and reduce duplicate chunks.
Reconciliation
Qdrant and PostgreSQL cannot share a transaction. A reconciliation process should detect and repair partial ingestion.
Better cache versioning
Cache keys should include corpus, model, filter, and prompt versions.
Retrieval evaluation
I would build a small evaluation dataset containing:
- A representative question
- The expected source document
- The expected chunk or passage
- The important facts the answer should contain
Then I would compare:
- Dense-only retrieval
- Sparse-only retrieval
- Hybrid retrieval
- Hybrid retrieval with reranking
Useful retrieval metrics would include:
- Recall at K
- Mean reciprocal rank
- Context relevance
- Source coverage
I would also measure latency for each pipeline stage.
Until those experiments exist, I would avoid claiming that one configuration is faster or more accurate than another.
Final takeaway
The most useful lesson from this project was that RAG is not one model call.
It is a chain of systems:
Ingestion
→ Chunking
→ Embedding
→ Retrieval
→ Fusion
→ Reranking
→ Prompt construction
→ Generation
→ Caching
→ Evaluation
My current pipeline uses dense and sparse retrieval to find a broad candidate set, reciprocal rank fusion to combine the rankings, and a cross-encoder to select the final context.
The LLM comes last.
That is exactly why the LLM was the easy part.
If you are building something similar, I would be interested to hear how you handle:
- Hybrid retrieval
- Reranking
- Cache invalidation
- Durable document ingestion
- Retrieval evaluation
Source code: https://github.com/abuhurayraniloy/RAGEval
If this walkthrough was useful, consider leaving a comment or reaction.
Top comments (0)