[memo]AUTOHALLUSION: Automatic Generation of Hallucination Benchmarks for Vision-Language Models

Xiyang Wuが第一著者，メリーランド大学

Related works
Object hallucination: bad impact for robotics medical imageing, HCI
Recent efforts focus on improving dataset quality for fine-tuning

Definitions and objective
Find things that are correlated in the LLM but not present 
In the picture to induce hallucinations in LVLMs

Contextual distance between two descriptions or texts

Scene generation
By using DALL-E to create an image I_s rich in context.
Image manipulation
Abnormal object insertion
Insert q argmax of distance between its language prior and GT
Correlated object removal
Remove q argmin of distance between its language prior and GT

Question construction
Target object q is present in the image.
These questions are repeated multiple times

Hallucination detection
Using GPT-4V-turbo, hallucinations are detected.

感想
変な質問が生成されないのかなーみたいなのは気になりました．
変数をうまく設定すればいけるのか