Leaked test items can add as much as eleven macro‑F1 points to multimodal fact‑checking scores [1]. That inflation silently rewrites the performance landscape that practitioners rely on when comparing LLMs or purchasing services.
Before this work, MAFC evaluation largely depended on static benchmarks such as AVeriTeC that were assembled without regard to model exposure [1]. Researchers have recently turned to dynamic collections like ClaimReview2025Q4, assuming post‑cutoff claims are automatically uncontaminated, yet the assumption had never been empirically verified.
Contamination can induce statistically significant inflation in MAFC performance, increasing Macro‑F1 by up to 11.34 points and distorting system rankings [1]. The authors measured this effect by contrasting scores on the full test set with a manually filtered uncontaminated subset, finding that every model’s macro‑F1 dropped when leaked items were removed, with statistically significant decreases observed for several models.
Gemini‑3.0‑Flash achieves the highest Macro‑F1 score (61.22 %); however, on the uncontaminated subset the best Macro‑F1 shifts to GPT‑5.2 (52.81 %) [1]. This reversal shows that contamination not only inflates absolute numbers but also reshuffles the ranking of state‑of‑the‑art systems.
Dynamic benchmarks still leave 17.09 %–29.30 % of post‑cutoff claims potentially contaminated, meaning a non‑trivial fraction of evaluation remains biased [1]. The study focuses on multimodal fact checking alone, so the magnitude of similar effects on other tasks—retrieval, summarisation, or code generation—remains an open question.
Evaluation pipelines should publish contamination audits and report results on strictly uncontaminated subsets as the default metric for any LLM service. Without such safeguards, reported gains risk being illusory, and downstream decisions built on them will be fundamentally unsound.
Top comments (0)