A multi-vector search index stores one vector for every word of every document, so it runs many times the size of an ordinary one. Merging the near-duplicate vectors cuts it by four, and a model trained to expect the merge loses less from it.
Engineer's view. This is a metrics rollup. You keep one average per group of similar samples instead of every sample. A multi-vector index stores a vector for every word, and many are near-copies. Merge each group into its average and the index shrinks fourfold. The cost is a rare word averaged away, like a hidden spike.
TL;DR. Merge each document's most similar word vectors into their average, and a multi-vector index gets about four times smaller for a small loss in search quality. Train the embedding model with that merge in its loss, and most of the remaining loss goes away.
Explain Like I'm 5
You keep a box of photos from a trip. Twelve of them are the same beach from almost the same spot. You could keep one beach photo and throw the rest away. Your box gets much smaller, and you still remember the beach.
But one photo shows the shell you found. If you lump it in with the beach photos, you lose the shell. The trick is to throw away the copies and keep the one-offs.
The Problem
You have shipped this bug before, and it had nothing to do with AI.
Your metrics store kept every raw sample. Disk use grew with traffic, and the bill grew with it. So you added a rollup. Every minute, a hundred samples became one average. Storage dropped by a hundred times, and the dashboards looked the same.
Then an incident hid inside one minute. A single request took nine seconds, and the average for that minute barely moved. The rollup had kept what was common and erased what was rare.
A multi-vector search index has that bug at a larger scale. An ordinary embedding model turns a whole document into one vector. A multi-vector model, such as ColBERT, keeps one vector per token, a word or a piece of a word. That makes search more precise, because a query can match a single phrase deep inside a long page.
It also makes the index huge. At 200 tokens, 128 dimensions and 4 bits per number, one document takes 12.8 KB. A million documents take 12.8 GB, where a single 1,024-dimension vector per document would take 2 GB. Most teams who try multi-vector search stop at that number.
Many of those 200 vectors are near-copies. A page about Mars mentions Mars, the planet and its orbit, and the model points those tokens the same way.
The fix: merge each document's near-identical token vectors into their average, and train the model so the merge costs it little.
Read the full post on The AI Commit
How to Shrink a Multi-Vector Search Index Without Losing Recall carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.
Top comments (0)