DEV Community

Cover image for Building Document Intelligence: A Document Q&A API with FastAPI, Qdrant, and RAG
Yagnesh
Yagnesh

Posted on

Building Document Intelligence: A Document Q&A API with FastAPI, Qdrant, and RAG

#ai #programming #beginners #opensource

Introduction

Imagine having multiple financial documents—PDFs, spreadsheets, scanned receipts, and reports—and needing to answer a specific question about their contents.

You could manually open each file, search for relevant information, extract the necessary figures, and organize the results.

But what if you could upload these documents, index their contents, and ask questions directly?

That is the problem I wanted to explore with Document Intelligence, an AI-powered document-question-answering API built with Python.

The project combines document parsing, OCR, embeddings, hybrid retrieval, and retrieval-augmented generation (RAG) into a single backend workflow.

🔗 GitHub Repository:

https://github.com/yagnesh97/document-intelligence

What Problem Does It Solve?

Financial information is often distributed across different document formats.

For example:

  • A financial report may be available as a PDF.
  • A spreadsheet may contain revenue and expense information.
  • A scanned document may contain information that cannot be extracted using normal text extraction.
  • A presentation may contain financial summaries across multiple slides.

Finding relevant information across these documents can require manually opening files and searching through their contents.

Document Intelligence provides a workflow to:

  1. Upload documents.
  2. Extract their text and metadata.
  3. Split the extracted content into smaller chunks.
  4. Generate embeddings and index the chunks.
  5. Retrieve relevant document content using a question.
  6. Generate an answer using the retrieved context.

The objective is to make uploaded financial material searchable and enable question answering grounded in the retrieved document content.

What Can It Process?

Document Intelligence supports the following formats:

Category Supported formats
Text .txt
CSV .csv
Excel .xls, .xlsx
PDF .pdf
Word .docx
PowerPoint .pptx
Images .png, .jpg, .jpeg, .tif, .tiff, .bmp, .webp, .gif

Different document formats require different extraction strategies.

For example, extracting text from an Excel spreadsheet is different from extracting text from a scanned PDF.

The project uses format-specific parsers and dedicated PDF analysis and OCR paths to handle these differences.

Architecture

The application is built around a FastAPI API, a Celery ingestion worker, document storage, MongoDB, and Qdrant.

Architecture diagram

The main components are:

Component Responsibility
FastAPI HTTP API and request handling
Celery Asynchronous document ingestion
Redis Task queue and Celery backend
MongoDB Document records, extracted content, metadata, and chunks
MinIO AIStor / S3 Original document storage
Qdrant Dense and sparse vector storage and hybrid retrieval
Sentence Transformers Dense embedding generation
FastEmbed Sparse embedding generation
OpenAI / Gemini Question answering
PyMuPDF / OCRmyPDF / Tesseract PDF extraction and OCR

The architecture separates document ingestion from question answering.

Document processing happens asynchronously through Celery, while search and chat requests are handled synchronously by the API.

Document Processing Workflow

1. Uploading documents

The API exposes:

POST /files

The endpoint accepts multipart file uploads.

Before processing, it validates:

  • Number of uploaded files
  • Individual file size
  • Total request size
  • Supported file extensions

After validation, the API creates a document record in MongoDB.

The original file is stored in S3-compatible object storage, and the document ID is submitted to Celery for background processing.

The upload response includes the document ID and task ID.

This allows the client to track the document without waiting for the entire ingestion process to finish.

2. Extracting document content

Once the Celery worker receives the document ID, it retrieves the original file and selects a parser based on the filename extension.

For common office documents:

  • Text files are processed using text extraction.
  • CSV files are parsed for their content.
  • Excel files include worksheet text.
  • Word documents include document and table content.
  • PowerPoint files include slide content.

PDF processing requires additional analysis.

3. Handling different types of PDFs

One of the interesting parts of the project is its PDF processing strategy.

Not every PDF contains extractable text.

Some PDFs contain native text, some contain scanned images, and others combine both.

Document Intelligence analyzes PDFs and classifies them into:

  • Native-text PDFs
  • Image-based PDFs
  • Hybrid PDFs

Native-text PDFs

These are processed using PyMuPDF text extraction.

Image-based PDFs

These are processed using OCRmyPDF, followed by text extraction.

Hybrid PDFs

These are analyzed page by page.

Pages that require OCR are processed using PyMuPDF OCR, while pages with usable native text can continue through normal extraction.

This avoids treating every PDF as an image or assuming that every PDF contains extractable text.

4. OCR for scanned documents

Scanned documents require OCR because their contents are stored as images rather than selectable text.

The project uses:

  • Tesseract for standalone image OCR
  • OCRmyPDF for image-based PDF processing
  • PyMuPDF for PDF analysis and extraction
  • OpenCV and Pillow for image processing

For standalone images, the OCR pipeline applies grayscale conversion and enlargement before extracting text.

OCR details are also retained in document metadata.

The goal is to make scanned financial documents available to the same downstream indexing and retrieval workflow as other supported documents.

5. Chunking extracted content

After extraction, the document content is split into paragraph-sized chunks.

The current implementation uses blank-line paragraph boundaries.

Each chunk is stored in MongoDB with its document association and chunk information.

This makes it possible to retrieve relevant portions of a document instead of passing an entire document to the language model.

One current limitation is that chunking does not yet support configurable chunk sizes or overlap.

Building the Retrieval Pipeline

Extracting text is only the first part of the workflow.

The next challenge is finding the relevant information when a user asks a question.

Document Intelligence uses dense and sparse embeddings.

Dense embeddings

The project uses:

BAAI/bge-m3

Dense embeddings represent the semantic meaning of document chunks.

This helps retrieve content that is conceptually related to a query, even when the exact wording differs.

Sparse embeddings

The project uses:

Qdrant/bm25

Sparse retrieval helps identify content based on important terms and keyword relationships.

This is particularly useful when questions contain specific financial terminology, names, or figures.

Hybrid retrieval

The project combines dense and sparse retrieval using Qdrant reciprocal-rank fusion.

The retrieval workflow is:

  1. Embed the user query in both dense and sparse spaces.
  2. Retrieve candidate document chunks.
  3. Combine the results using reciprocal-rank fusion.
  4. Optionally rerank the retrieved candidates using a CrossEncoder.
  5. Return the most relevant chunks.

The optional reranker uses:

BAAI/bge-reranker-v2-m3

This additional step helps refine the candidate results before they are used for question answering.

Retrieval-Augmented Question Answering

The final stage is answering questions using retrieved document context.

Document Intelligence supports configured OpenAI and Gemini models through LangChain provider integrations.

The chat workflow is:

  1. Receive a question.
  2. Retrieve relevant document chunks.
  3. Use the retrieved chunks as context.
  4. Generate an answer using the configured language model.
  5. Return the answer and source chunks.

The application prompt instructs the model to use the supplied context and avoid presenting financial advice as fact.

The non-streaming chat endpoint returns the answer along with the retrieved sources.

This allows users to inspect the document content used to generate the response.

The project also supports streaming responses through server-sent events.

API Endpoints

The API currently exposes the following endpoints:

Endpoint Purpose
GET /health Check API and MongoDB availability
POST /files Upload documents and queue ingestion
GET /files List documents
GET /files/{document_id} Inspect document status, content, and metadata
DELETE /files/{document_id} Delete document records, chunks, vectors, and original files
GET /search Retrieve relevant document chunks
POST /chat Ask questions using retrieved document context
POST /chat/stream Stream generated answers

Document processing tracks the following states:

QUEUED → PROCESSING → COMPLETED

If processing fails, the document is marked:

FAILED

The worker records the error and retries processing up to three times with backoff.

Running the Project Locally

Prerequisites

You will need:

  • Python 3.14 or later
  • uv
  • Docker Engine and Docker Compose
  • Tesseract
  • OCRmyPDF and its system dependencies
  • An API key for OpenAI or Gemini if you want to use chat

Clone the repository 

git clone https://github.com/yagnesh97/document-intelligence.git
cd document-intelligence
Enter fullscreen mode Exit fullscreen mode

Install dependencies 

cp .env.example .env
uv sync --all-groups
Enter fullscreen mode Exit fullscreen mode

Configure the required environment variables in .env.

At minimum, the settings class requires:

  • MONGODB_URI
  • S3_ACCESS_KEY
  • S3_SECRET_KEY

Configure the object-storage endpoint and create the bucket specified by S3_BUCKET.

Start infrastructure 

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

The Compose configuration starts the local infrastructure, including Redis, MongoDB, MinIO AIStor, Qdrant, and Flower.

Start the API 

uv run uvicorn app.api.main:app --reload
Enter fullscreen mode Exit fullscreen mode

Start the Celery worker

In another terminal:

uv run celery -A app.worker.celery:celery worker --loglevel=info --pool=solo
Enter fullscreen mode Exit fullscreen mode

Verify the API 

curl http://127.0.0.1:8000/health
Enter fullscreen mode Exit fullscreen mode

Interactive API documentation is available at:

http://127.0.0.1:8000/docs

Example: Uploading a Document 

curl -X POST http://127.0.0.1:8000/files \
  -F 'files=@./statement.pdf;type=application/pdf'
Enter fullscreen mode Exit fullscreen mode

The response contains a document ID.

Use that ID to inspect processing status:

curl http://127.0.0.1:8000/files/<document-id>
Enter fullscreen mode Exit fullscreen mode

Wait until the document reaches COMPLETED before searching its content.

Example: Searching Document Content 

curl --get 'http://127.0.0.1:8000/search' \
  --data-urlencode 'q=What was the operating income?' \
  --data-urlencode 'limit=5'
Enter fullscreen mode Exit fullscreen mode

The search response contains retrieved chunks, their scores, document identifiers, and metadata.

Search can also be scoped to a specific document using document_id.

Example: Asking Questions 

curl -X POST http://127.0.0.1:8000/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "question": "Summarize the reported revenue.",
    "model_id": "<llm-model-object-id>",
    "document_id": "<document-id>",
    "limit": 5
  }'
Enter fullscreen mode Exit fullscreen mode

The response contains:

  • Generated answer
  • Retrieved source chunks
  • Document identifiers
  • Chunk identifiers
  • Retrieval scores
  • Source metadata

The model_id must be the MongoDB ObjectId of an enabled LLM model configuration.

Testing

The project uses pytest.

Run the test suite:

./cmd.sh test
Enter fullscreen mode Exit fullscreen mode

Or:

uv run pytest
Enter fullscreen mode Exit fullscreen mode

Run specific tests:

uv run pytest tests/api/test_files.py
Enter fullscreen mode Exit fullscreen mode 
uv run pytest tests/services/parsers/pdf/test_analyzer.py
Enter fullscreen mode Exit fullscreen mode

Run code checks:

./cmd.sh check
Enter fullscreen mode Exit fullscreen mode

Format the code:

./cmd.sh format
Enter fullscreen mode Exit fullscreen mode

Integration tests require the relevant external services and must be selected explicitly.

What I Learned

Building Document Intelligence brought several backend and AI engineering concepts together.

Some of the key areas explored include:

  • Designing asynchronous document ingestion with FastAPI and Celery
  • Handling multiple document formats through parser-specific workflows
  • Understanding the differences between native-text and scanned PDFs
  • Integrating OCR into document processing
  • Generating dense and sparse embeddings
  • Implementing hybrid retrieval with reciprocal-rank fusion
  • Connecting retrieved document context to language models
  • Managing document records, original files, chunks, and vectors across different storage systems

It also highlighted how important the document-processing stage is to the overall quality of a RAG application.

Retrieval quality depends on the content extracted from the original document, how that content is chunked, and how relevant chunks are selected.

Current Limitations

The project is still evolving.

Some current limitations include:

  • Paragraph-based chunking without configurable size or overlap
  • Hard-coded local Redis configuration for Celery
  • No API endpoint for listing or managing LLM model configurations
  • Manual creation of the configured object-storage bucket
  • Parser selection based on supported filename extensions
  • No application-level authentication or authorization

The service should be placed behind appropriate access controls before being exposed beyond a local development environment.

Financial documents may contain sensitive information, so storage and infrastructure access must also be configured carefully.

What's Next?

There are several areas where the project could be improved:

  • More flexible chunking strategies
  • Configurable chunk size and overlap
  • Improved document extraction and OCR accuracy
  • More configurable ingestion infrastructure
  • Better model configuration management
  • Additional document-processing workflows

These are potential improvements rather than features currently implemented.

Explore the Project

Document Intelligence is an open-source project for developers interested in document processing, OCR, hybrid search, and retrieval-augmented generation.

You can explore the architecture, run the application locally, experiment with different documents, and adapt the implementation to your own use cases.

🔗 GitHub Repository:

https://github.com/yagnesh97/document-intelligence

If you're building document-processing or RAG applications, feel free to explore the code, share feedback, or contribute improvements.

What would you add to a document-question-answering pipeline?

I'd be interested to hear your thoughts.

Python #FastAPI #RAG #DocumentProcessing #OCR #Qdrant #ArtificialIntelligence #OpenSource #BackendDevelopment

Top comments (0)