Large language models can explain concepts and generate useful responses, but they do not automatically know the contents of a private PDF uploaded by a user.
A document question-answering application needs a way to retrieve relevant information from that document and provide it to the language model before generating an answer.
That is the idea behind Retrieval-Augmented Generation (RAG).
In my PDF Chatbot project, I worked with a stack involving FastAPI, LangChain, FAISS, PostgreSQL, SQLAlchemy, Streamlit, and Docker. The application combines document retrieval with conversational question answering and persistent chat history.
This article explains the architecture and some of the engineering decisions involved.
1. What is RAG?
RAG combines two main operations:
- Retrieval: Find information relevant to a user's question.
- Generation: Give that information to a language model so it can formulate an answer.
Instead of relying entirely on information learned during model training, the application retrieves relevant passages from the uploaded documents.
A typical workflow is:
PDF upload → Text extraction → Chunking → Embeddings → Vector search → Answer generation
The quality of the final answer depends on more than the language model. If retrieval misses the relevant passage, the model may not have enough information to answer correctly.
2. Preparing documents for retrieval
A PDF is not automatically suitable for semantic search. Its text needs to be extracted and organized into manageable sections.
Text extraction and chunking
The application processes uploaded PDFs and divides their content into smaller text chunks.
Chunking is useful because embedding and retrieval systems generally work better with manageable pieces of text than with an entire long document treated as one unit.
However, chunk size involves a trade-off:
- Very small chunks may lose important context.
- Very large chunks may contain irrelevant information and make retrieval less precise.
The right strategy depends on the document structure and the questions users are likely to ask.
Generating embeddings
An embedding model converts text into numerical vectors that represent semantic information.
In my project, local ONNX-based embeddings are used as part of the retrieval pipeline. Using a local embedding model can reduce dependence on an external embedding API, although model quality and deployment resource requirements still matter.
Searching with FAISS
FAISS provides vector similarity search. The application can use it to retrieve document chunks that are semantically related to a question.
This is different from ordinary keyword matching: a semantic search system can find related passages even when the wording of the question differs from the original text.
FAISS handles vector search; it is not itself the language model that generates the final answer.
3. Generating answers from retrieved context
After retrieving relevant chunks, the application passes the available context and the user's question to the language model.
A useful instruction is to answer using the supplied context and acknowledge when the retrieved material does not contain enough information.
This reduces the risk of unsupported answers, but it cannot guarantee that hallucinations will never occur. Retrieval quality, prompt design, and answer validation all affect reliability.
My project also uses history-aware retrieval so that follow-up questions can be rewritten using previous conversation turns.
For example:
- First question: "What does the document say about authentication?"
- Follow-up: "What are its main limitations?"
The second question is ambiguous by itself. Conversation history can help the retrieval system understand that "its" refers to the authentication approach discussed previously.
4. Designing persistent conversation storage
A chatbot becomes more useful when users can return to earlier conversations instead of losing their history whenever the interface reloads.
I used SQLAlchemy ORM models to organize the application's relational data into three main entities.
| Model | Responsibility |
|---|---|
Document |
Stores document metadata, including the filename, page count, chunk count, and creation time |
Conversation |
Associates a conversation with a document and stores its title and creation time |
Message |
Stores each message's role, content, source information, and creation time |
The relationships follow a straightforward structure:
Document → Conversations → Messages
A document can have multiple conversations, and each conversation can contain multiple messages. Foreign keys maintain the associations between these records, while SQLAlchemy's ORM relationships make it easier to work with related objects in Python.
The Message model also includes a JSON field for source information, allowing retrieved source details to be stored alongside the response.
5. Why separate vector search from relational storage?
The application has two different data responsibilities.
FAISS supports similarity search over embedded document chunks.
PostgreSQL stores structured application information such as document metadata, conversation records, and messages.
Keeping these responsibilities separate makes the design easier to understand. Relational data can be queried and managed through SQLAlchemy, while vector search is handled by the retrieval component.
The two systems still need consistent lifecycle management. For example, if a document is deleted or reprocessed, the application must handle its associated retrieval data and metadata appropriately.
6. Managing the application layers
The project uses FastAPI for backend API functionality and Streamlit for the interactive interface.
FastAPI provides a boundary for operations such as uploading documents and asking questions. Streamlit gives users a convenient way to interact with the application without building a separate JavaScript frontend.
SQLAlchemy sessions also require careful handling. A session should have a clear lifecycle, and database changes that need to persist must be committed. Closing a session alone does not guarantee that pending changes are committed.
Docker provides a consistent packaging approach for running the application and its dependencies, while deployment configuration must still account for environment variables, database connectivity, and persistent storage.
7. Lessons from building a RAG application
This project reinforced several important ideas:
- Retrieval quality is just as important as answer generation.
- Chunking affects both relevance and context.
- Conversation history can help interpret follow-up questions.
- Source references improve traceability, but do not independently prove an answer is correct.
- Persistent chat history needs a clear relational data model.
- Database sessions and application resources need explicit lifecycle management.
Conclusion
Building a PDF chatbot involves more than connecting a language model to a file. It requires document processing, semantic retrieval, context-aware generation, persistent storage, and a user interface that brings the components together.
RAG provides a practical foundation for document-grounded question answering, but reliable results still require careful testing of retrieval relevance, answer correctness, and failure cases.
Technologies: FastAPI, LangChain, FAISS, PostgreSQL, SQLAlchemy, Streamlit, ONNX embeddings, and Docker.
Live Demo:PDF Chatbot with RAG
Project: Github
Further reading: LangChain learning resources · SQLAlchemy ORM documentation
Top comments (0)