Multimodal search systems that store pages as a thousand‑plus patch vectors can be inverted to recreate the original images. The index alone supplies enough signal for an attacker to generate a faithful rendering of the source document, turning what was thought to be a harmless embedding store into a privacy liability.
Before this work engineers treated dense multi‑vector indexes as low‑risk artifacts because only raw pixel data were considered sensitive. Retrieval pipelines such as ViDoRe and other visual‑document retrievers typically dump per‑patch embeddings into third‑party vector databases, assuming the vectors cannot be turned back into readable content.
Inverting raw indices recovers nearly half of the page text and almost always identifies its source. On the ViDoRe v3 benchmark, reconstructed pages retrieve 47 % of the words and rank their originating document first in 98.4 % of queries [1].
Simple token‑pooling or shuffling defenses slash that leakage to a negligible level. Both cut word recall to about 8 % “Both cut word recall to about 8%.” [1]
Even when vectors are shuffled, an attacker can restore order and regain near‑perfect re‑identification. “A model that restores the order of a shuffled index raises the share of source pages ranked first from 3.8 % to 93.5 %, while inverting a pooled index remains open.” [1] The pooled variant still leaves the index vulnerable to inversion (though with low word recall), and shuffling alone offers only an illusion of safety.
The paper’s threat model assumes access to the exact vision‑language encoder and knowledge of page geometry, and its experiments focus on two retrievers; broader ecosystems may present additional challenges or mitigations not captured here. Moreover, defenses that merely obfuscate vector order can be undone by modest learning models, suggesting that privacy cannot rely on “cheap” tricks alone.
Consequently, any deployment that outsources multi‑vector indexes to external stores must treat those embeddings as confidential assets, encrypt them at rest, and incorporate inversion‑resistance tests into the evaluation suite. Ignoring this hidden vulnerability will leave commercial search services exposed to trivial data exfiltration.
Top comments (0)