DEV Community: Adrien Laugueux

Beyond OCR: Building a Truly Multimodal Local RAG Pipeline

Adrien Laugueux — Wed, 11 Mar 2026 07:49:40 +0000

If you've ever tried to build a document chatbot over a collection of scanned reports, technical manuals, or mixed-content PDFs, you've probably run into the same wall: classic RAG pipelines are essentially blind.

They extract text, chunk it, embed it, and retrieve it — but the moment your document contains a scanned table, a wiring diagram, or an annotated chart, that information either gets mangled by OCR or vanishes entirely. The retrieved context is impoverished, and your chatbot's answers reflect that. Ask it about the diagram on page 12 and it will confidently summarise the paragraph next to it, which is arguably worse than saying nothing at all.

There's a better way. Instead of treating documents as bags of text, you can treat them the way a human would: read the page as a whole, visuals included. And for pages that do contain native, selectable text, you don't have to choose between precision and visual understanding — you can have both. Revolutionary, we know.

Why OCR-Based RAG Falls Short

The standard pipeline — OCR → chunking → embedding → vector search → LLM — was designed for text-native documents. When applied to rich, heterogeneous content, it breaks down in predictable ways:

A scanned table loses its structure and becomes an unreadable string of values
A technical diagram is reduced to a handful of disconnected labels
Spatial relationships — captions, callouts, annotations — are destroyed
Charts and graphs lose all their meaning once flattened to text

The root problem is that OCR reduces a two-dimensional, semantically rich object (a page) to a one-dimensional stream of characters. You can't recover what was never captured. It's a bit like describing a painting by reading the label on the frame — technically accurate, entirely useless.

The Core Idea: Combine Native Text Extraction and Vision Models

Native text extraction and Vision Language Models (VLMs) are not competing approaches — they are complementary. Each covers what the other misses:

Native text (via PyMuPDF) is exact, faithful to the character, and computationally free. It carries no risk of hallucination.
VLMs understand structure, visual semantics, and spatial relationships — things that text extraction is blind to.

The best pipeline uses both. On pages that contain native text alongside visual elements, native extraction handles the prose while the VLM focuses exclusively on what it does best: tables, diagrams, charts, and images. On fully scanned pages, the VLM takes over entirely.

The pipeline becomes:

Step 1 — Convert Pages to Images and Extract Native Text

import fitz  # PyMuPDF
from pdf2image import convert_from_path

doc = fitz.open("document.pdf")
page_images = convert_from_path("document.pdf", dpi=200)

for i, page in enumerate(doc):
    native_text = page.get_text()
    image = page_images[i]
    # pass both to the processing function

Step 2 — Detect Visual Elements

PyMuPDF lets you check whether a page actually contains visuals, so you can avoid unnecessary VLM calls on text-only pages:

def page_has_visuals(page):
    images = page.get_images()
    drawings = page.get_drawings()
    return len(images) > 0 or len(drawings) > 0

Step 3 — Build a Combined Page Description

import ollama

def describe_visuals(image_path: str) -> str:
    prompt = """Focus only on non-textual elements on this page:
    - Tables, diagrams, charts, images, graphs
    - Describe their content and what they convey
    - For tables, transcribe their content in Markdown or JSON
    - Ignore plain text paragraphs
    If there are no visual elements, say so briefly."""

    response = ollama.chat(
        model="llama3.2-vision",
        messages=[{"role": "user", "content": prompt, "images": [image_path]}]
    )
    return response["message"]["content"]

def describe_full_page(image_path: str) -> str:
    prompt = """Describe this document page exhaustively:
    - If you see a table: transcribe its full content in a structured way
    - If you see a diagram or chart: describe its elements and relationships
    - If you see text: transcribe it faithfully
    - If you see a graph: describe the data and visible trends
    Be precise and thorough."""

    response = ollama.chat(
        model="llama3.2-vision",
        messages=[{"role": "user", "content": prompt, "images": [image_path]}]
    )
    return response["message"]["content"]

def process_page(page, page_image_path):
    native_text = page.get_text().strip()
    has_text = len(native_text) > 100
    has_visuals = page_has_visuals(page)

    if has_text and has_visuals:
        # Best of both worlds: precise text + VLM for visuals
        visual_desc = describe_visuals(page_image_path)
        return f"## Extracted text\n{native_text}\n\n## Visual elements\n{visual_desc}"

    elif has_text:
        # Text-only page: no VLM needed
        return native_text

    else:
        # Fully scanned page: VLM takes over entirely
        return describe_full_page(page_image_path)

This approach gives you maximum precision on text — proper nouns, exact figures, technical references — with no risk of hallucination, while the VLM adds the semantic layer that text extraction can never provide.

The process_page function maps directly to this decision tree: check for native text, check for visuals, and route accordingly.

What This Unlocks

The combined approach improves retrieval in ways that neither method achieves alone:

Exact recall: a query for a specific article number or technical specification matches the native text verbatim
Semantic recall: a query about "the heat flow diagram" or "the comparison table" matches the VLM's description
Structural fidelity: tables are indexed as structured Markdown or JSON, not as a garbled sequence of cell values
Compute efficiency: the VLM only runs when there are actual visuals to describe, keeping ingestion time reasonable

Going Further: ColPali

For the most demanding use cases, ColPali takes a fundamentally different approach: it embeds document pages directly as images, without any intermediate text representation. Queries are embedded in the same visual space, and retrieval is based on visual similarity.

from colpali_engine.models import ColPali, ColPaliProcessor

model = ColPali.from_pretrained("vidore/colpali-v1.2")

# Both page images and text queries are embedded directly
# Retrieval happens in the visual embedding space

The benefit is zero information loss — the layout itself is part of the index. ColPali consistently ranks among the best-performing models on document retrieval benchmarks, particularly for visually complex pages. It can also be combined with the hybrid approach above: use ColPali for retrieval, then pass the retrieved page image plus its extracted text to the LLM for generation.

Recommended Stack (Fully Local)

from langchain_community.vectorstores import Chroma
from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma(embedding_function=embeddings, persist_directory="./db")

# Index each combined page description
vectorstore.add_texts(
    texts=[page_description],
    metadatas=[{"source": filename, "page": page_num}]
)

Practical Advice

Chunk at the page level. A page is a natural semantic unit for a VLM. Splitting mid-page breaks the visual context the model needs to produce a coherent description.

Keep the original images. Store the source page image alongside its description. When a page is retrieved, you can pass the image directly to the LLM as additional context — especially useful for complex visuals that are hard to describe fully in text.

Tailor your VLM prompts to document type. A technical schematic, a financial report, and a product datasheet each warrant different prompting strategies. Investing in prompt templates per document category pays off in description quality.

Request structured output for tables. When a page contains tabular data, explicitly ask the VLM to output Markdown or JSON. This preserves structure in a way that plain prose cannot, and makes the indexed content far easier for the LLM to reason over.

Conclusion

The classic OCR-based RAG pipeline was never designed for visually rich documents. The solution isn't to replace text extraction with a VLM — it's to use both in concert. Native text gives you precision and reliability; the VLM gives you visual understanding. Together, they produce page descriptions that are richer than either could achieve alone. Think of it as hiring both a speed-reader and an art critic, and making them share a desk.

Combined with a fully local stack, this approach gives you a document chatbot that can reason over tables, diagrams, charts, and mixed content, without any data leaving your infrastructure. The tooling is mature, the models are capable, and the entire pipeline runs on commodity hardware. There's no reason to settle for text-only anymore.

Bye bye Team Viewer

Adrien Laugueux — Thu, 27 Mar 2025 13:28:08 +0000

This article targets developers who'd like to replace Team Viewer by RustDesk, a FREE and open source solution hosted on their own server/VPN:

🔐 For data privacy and security
💰 For budget
🚀 To have zero limits
✨ Compliant with most of devices (windows, android, etc.)
😎 And super easy to setup

This tutorial explains step-by-step how to setup this solution, from scratch. If you already have your VPN, just skip this part 🚀.

My opinion is that Team Viewer is great for individuals using the free version or for big companies who need to handle a huge parc of machines.

But for smaller companies, I definitely suggest RustDesk.

Prerequisites of this tutorial

A device with a terminal and ssh and scp installed
A RustDesk client installed on every device (controller or to be controlled). Download from here
This tutorial assumes you get a server under Debian

First step: subscribing a VPS plan

There are many VPS providers. I personally chose the german Netcup for its competitive prices, and for the absence of commitment (I can stop when I want).

Here is the subscription page: https://www.netcup.eu/vserver/vps.php

VPS 200 G11s is enough for most of people. It has:

Processor: 2 vCore (x86)
Main memory: 2 GB
Hard disk: 64 GB SSD
Minimum availability: 96.6%

Second step: admin access

Once the order is maid (yes, there is no payment step), a confirmation email will be sent, followed by a few more a few minutes later.

The most important mail will contain your customer number and password, to access the administration part:

we are pleased to welcome you as a customer at netcup. Enclosed you will find your access data to the netcup CCP (customer control panel).
There you have the possibility to maintain your data and products, as well as to view past invoices.
Your access data to the CCP are as follows:
Customer number: 123456
Password: blablabla

So don't forget to note:

your customer number (only digits)
your password
the link to administrate Netcup VPS: https://www.customercontrolpanel.de/rechnungen.php
the link to technically administrate the VPS: https://www.servercontrolpanel.de/SCP/Home
the ip of your VPS
the initial root password (to be changed in the next part)

Third step: root password configuration

First, for security reason, the root password must be changed via the admin interface. Access it via: https://www.customercontrolpanel.de/rechnungen.php

We can see our VPS listed, in the Products section:

Then by clicking on the 🔎, the URL to admin panel is displayed:

It's always https://www.servercontrolpanel.de/SCP/Home but just in case you lose it, this way you know where you can find it.

Once there, we can start the configuration of the server (it's very fast):

General > give a nickname to your server (optional but i recommend it)
Control > click on Shutdown (ACPI). required to set a new root password
Access > change the root password and NOTE IT
Control > restart the server

⚠️ Do not even try to use the terminal available on the General tab, you'll probably never be able to simply enter your password, because of the poor keymap support. To access it we'll open a real SSH terminal.

Fourth step: ssh access

Open your favorite terminal with ssh installed and run this command (of course adapt it to your server).

ssh root@YOUR_IP
# To the question "Are you sure you want to continue connecting (yes/no/[fingerprint])?", press enter
Then enter the root password you created in the "Third step"

Here we are.

Fifth step: user creation

For security reasons (if you are interested you can check here), it is recommended to run your programs as non-root user.

Here is how to create a user named "admin" (feel free to set the name of your choice of course):

# user creation
useradd admin

# password creation
passwd admin
# here create a password for admin

# set user as sudoer
usermod -aG sudo admin

# Create his home directory
mkhomedir_helper admin

Sixth step: RustDesk server installation

# use your non-root user
su - admin

# install ufw
sudo apt install ufw

# open ports
ufw allow proto tcp from YOURIP to any port 22
ufw allow 21114:21119/tcp
ufw allow 8000/tcp
ufw allow 21116/udp
sudo ufw enable

# download and run the installer
wget https://raw.githubusercontent.com/techahold/rustdeskinstall/master/install.sh
chmod +x install.sh
./install.sh

Note: if you had the good taste to have a domain linked to your VPS IP, you can enter it when requested.

Important: Note well the public key, you'll need it.

Seventh step: Install RustDesk on the host (computer to be controlled)

Execute the exe, it should open this window:

Click on Install in the pink button on the left.

Open the settings:

Only enter the ID Server with the IP of your VPS, or the domain name if you have one and if configured during installation.

You should now see "Ready" written in green in the bottom.

My advice: for simplicity, set a permanent password in the security settings.

Last step: Install RustDesk on the client (computer to rule them all)

Same installation, excepted that in the Network settings you'll have to fill Key with the public key I told you to save in the "Sixth step".

Now add the ID and password (the one you set if you followed my advice, otherwise the rotating one).

And you are good, now you can access your device by double-clicking on it and configure as many devices as you want!

Useful links

RustDesk self-hosting doc

Netcup VPS and Contabo VPS which are my favorite ones

👋 Thanks for reading, and feel free to share your feelings!

How I created my personal VPN with static IP

Adrien Laugueux — Fri, 17 Mar 2023 19:36:00 +0000

This article targets developers who'd need their VPN:

For confidentiality (especially when working with public wifi)
If company resources are restricted to IP whitelists
For any other reason

Have a good reading =)

Why using a VPS to create a VPN

I identified 3 main ways to get a vpn with a static IP:

💰💰 Subscribe a VPN with static IP option. This option can be great if beyond the fixed IP and confidentiality aspects, you want to be able to be behind an IP of the country of your choice. It's the easiest and most powerful way, but... also the most expensive. Note that static IPs are only available in paying VPN (at least today, in March 2023), so normally they don't make money with your personal data, like some free VPN.
🧔 Install your own personal VPN server at home, like on your favorite Raspberry pie zero. Given the consumption of such a device, it's very probably the least expensive way. But you are limited to your up bandwidth, so if you're not fibered, you'll feel it.
❤️ What I chose and what I'll present here is installing OpenVPN on a VPS (Virtual Private Server). It's a bit more technical, but very fast if you have the right guidelines. I prefer this solution to the 2 other ones, because small VPS are enough and clearly less expensive than VPN, because I'm not fibered and because it's more reliable.

Prerequisites

A device with a terminal and ssh and scp installed
An OpenVPN client installed on the machine you want to use a vpn.

First step: subscribing a VPS plan

There are many VPS providers. I personally chose the german Netcup for its competitive prices, and for the absence of commitment (I can stop when I want).

Here is the subscription page: https://www.netcup.eu/vserver/vps.php

VPS 200 G10s is clearly enough to make our VPN. It has:

Processor: 2 vCore
Main memory: 2 GB
Hard disk: 40 GB SSD (RAID10)
Unthrottled traffic: 80 TB / month (80 000 GB per month!)

Second step: admin access

Once the order is maid (yes, there is no payment step), a confirmation email will be sent, followed by a few more a few minutes later.

The most important mail will contain your customer number and password, to access the administration part:

we are pleased to welcome you as a customer at netcup. Enclosed you will find your access data to the netcup CCP (customer control panel).
There you have the possibility to maintain your data and products, as well as to view past invoices.
Your access data to the CCP are as follows:
Customer number: 123456
Password: blablabla

So don't forget to note:

your customer number (only digits)
your password
the link to administrate Netcup VPS: https://www.customercontrolpanel.de/rechnungen.php
the link to technically administrate the VPS: https://www.servercontrolpanel.de/SCP/Home
the ip of your VPS
the initial root password (to be changed in the next part)

Third step: root password configuration

First, for security reason, the root password must be changed via the admin interface. Access it via: https://www.customercontrolpanel.de/rechnungen.php

We can see our VPS listed, in the Products section:

Then by clicking on the 🔎, the URL to admin panel is displayed:

It's always https://www.servercontrolpanel.de/SCP/Home but just in case you lose it, this way you know where you can find it.

Once there, we can start the configuration of the server (it's very fast):

General > give a nickname to your server (optional but i recommend it)
Control > click on Shutdown (ACPI). required to set a new root password
Access > change the root password and NOTE IT
Control > restart the server

Fourth step: ssh access

Open your favorite terminal with ssh installed and run this command (of course adapt it to your server).

ssh root@YOUR_IP
# To the question "Are you sure you want to continue connecting (yes/no/[fingerprint])?", press enter
Then enter the root password you created in the "Third step"

Here we are.

Fifth step: user creation

For security reasons (if you are interested you can check here), it is recommended to run your programs as non-root user.

Here is how to create a user named "admin" (feel free to set the name of your choice of course):

# user creation
useradd admin

# password creation
passwd admin
# here create a password for admin

# set user as sudoer
usermod -aG sudo admin

# Create his home directory
mkhomedir_helper admin

Sixth step: OpenVPN installation

For security reasons (if you are interested you can check here), it is recommended to run your programs as non-root user.

Here is how to create a user named "admin" (feel free to set the name of your choice of course):

# use your non-root user
su - admin

# download OpenVPN installer
wget https://raw.githubusercontent.com/Angristan/openvpn-install/master/openvpn-install.sh -O debian-11-vpn-server.sh

# make it executable
chmod -v +x debian-11-vpn-server.sh

# execute it
sudo ./debian-11-vpn-server.sh

# Then simply press enter multiple times until you get this message: 
# "The configuration file has been written to /root/mydesktopclient.ovpn.
# Download the .ovpn file and import it in your OpenVPN client."

Seventh and ultimate step: Connection of OpenVPN client

Exit the server (exit command multiple times), then let's use scp command to retrieve mydesktopclient.ovpn file on your PC/mac/whatever:

scp admin@YOUR_IP:/home/admin/mydesktopclient.ovpn .
# enter here the password you created for admin (not root, just in case)

Open your OpenVPN client (install it if you don't have it yet), select the File tab > Browse > select your mydesktopclient.ovpn file.

Confirm by clicking on Connect, and 🎉 we are done!! 🎉

To create a new client .ovpn file

If you want to share your vpn with someone, instead of sharing your .ovpn file you can create a new one easily:

Connect to the server with ssh:

ssh admin@YOUR_IP
# admin password

And run the installer:

sudo ./debian-11-vpn-server.sh
# admin password

# What do you want to do?
1

# Tell me a name for the client.
# => enter here a name matching this new client

# Do you want to protect the configuration file with a password?
1

Then proceed like in the "Seventh step" to retrieve the file, and share it.

👋 Thanks for reading