My Lessons from a 7-Model Ablation Study on Multimodal Models

Comparing gated approaches for hate speech

Investigating whether a learned per-sample gating mechanism that dynamically weights visual and textual features can outperform static fusion strategies for multimodal classification.

Research Question
When fusing image and text for hate detection in memes, does a model that learns how to combine modalities per-sample outperform fixed fusion? And what does the model learn about when each modality matters?

Dataset
Facebook Hateful Memes (Kiela et al., NeurIPS 2020) -- 10,000 memes with binary labels. Designed with benign confounders: neither modality alone is sufficient.