[memo]Unified Hallucination Detection for Multimodal Large Language Models

An inadequate range of halucination categories addressed

Contribution

• Construct MHaluBench
  • fine-grained analytical features
• UniHD
  • detection of hallucinations

Preliminaries

• unified perspective
• Modality conflicting hallucination
• Fact conflicting hallucination
  • outputs from MLLMs may contradict established factual knowledges

Construction of MHaluBench

• curated to include a balanced distribution of instances across three pivotal tasks
• Hallucinatory exapmle collection
  • I2T: IC and VQA tasks, drawing samples from the MS-COCO
  • T2I: DrawBench
• Human annotation

UniHD

• まず、VLMの主張からextractする
• その後、Object detectionを使ってextractする
• それを用いてhallucinationをextractionする