ColPali

The other day I intended to learn and implement retrieval augmented generation pipeline

then a thought flashed in my mind that what if my pdfs are complex presenting figures, tables and multi-column layout that are not easily extractable which got me the rabbit hole to reach ColPali.
ColPali uses vision language model (PaliGemma Based)
The concept is simple: it treats the whole pdf as a single image and breaks the image into patches further assigning the embeddings to each of them. The question is also broken into tokens and every token is compared with the patches of each page (MaxSim operation) which is better than traditional single-vector cosine similarity

The core advantage is the structure will not be lost while using VLMs based inference.