This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the write-up and results are CC BY 4.0.
A dataset's "gold label" is supposed to be the thing a model is graded against. But where does that one label come from? For most NLI benchmarks it is derived from several human votes — and a surprising amount rides on exactly how. In two earlier audits (SNLI #37 and MultiNLI #38, in the catalogue below) we checked whether each gold label is the strict majority of its own annotator votes. It is — but that is true by construction: the gold is defined as that majority, so the check is tautological. Reproducible and worth freezing, but not a finding.
This post is about the two cases where the comparison is not tautological: where an independent set of human judgements exists, collected separately from the label the benchmark ships. That is where you can actually learn something.
1. ANLI: the gold is the writer's, the verifiers are someone else
ANLI (Adversarial NLI) is built by a human writer who crafts an example to fool a model, then labels it themselves. Separately, Facebook Research released verifier labels: 2–3 other annotators' judgements of the same examples. So we can ask a real question — how often does the writer's adversarial gold match the strict majority of the independent verifiers?
Across all 29,803 released verifier lines (joined to the main data with 0 orphans):
- the writer's gold equals the strict verifier majority on 29,765 (99.87%),
- it differs on just 7, and reaches no verifier majority on 31,
- agreement rises by round, up to a perfect round 3.
So far, so clean. But we had pre-registered a prediction (P1) that the verifier file would cover only dev/test — the splits where verification normally happens. P1 FAILED, and the failure is the interesting part: 23,403 of the 29,803 verified rows are training examples. There is an under-documented, multi-annotator agreement signal sitting on ANLI train that most users never touch. And all 7 writer-vs-verifier disagreements — including one case where both verifiers unanimously contradict the writer's gold — fall in that training split.
A failed pre-registered prediction is not a bug to hide; it is the headline. Full write-up and reproducible code: anli-verifier-majority-census.
2. ChaosNLI: what happens when you ask 100 people instead of 5
ChaosNLI (Nie, Zhou and Bansal, EMNLP 2020) did something the original pipelines never could: it re-collected 100 fresh annotations for a sample of the SNLI, MNLI-matched and alphaNLI dev sets. Each record keeps the original benchmark gold (old_label, the 5-annotator majority), the 100-vote plurality (majority_label), and the full per-label vote counts.
That lets us ask two things. The dataset documents the first; the second is this audit's novel lens.
(a) Does the original gold survive re-annotation? Across the two 3-way subsets (3,113 sampled examples) the original gold disagrees with the 100-vote plurality on 886 (28.46%) — reproducing ChaosNLI's documented 378 SNLI and 508 MNLI exactly (our pre-registered P1 HELD).
(b) Do 100 humans even reach a strict majority? ChaosNLI's majority_label is a plurality — it is the top label even if that label holds only, say, 48 of 100 votes. So we counted how often no label clears more than 50 of the 100 votes, while the benchmark still assigns one hard gold:
| Subset | Rows | old ≠ 100-vote plurality | no strict (>50/100) majority |
|---|---|---|---|
| SNLI | 1,514 | 378 (24.97%) | 69 (4.56%) |
| MNLI-matched | 1,599 | 508 (31.77%) | 242 (15.13%) |
| Both 3-way | 3,113 | 886 (28.46%) | 311 (9.99%) |
| alphaNLI (2-way, reported, not bet) | 1,532 | 163 (10.64%) | 8 (0.52%) |
311 of 3,113 examples (9.99%) get no strict majority even after 100 fresh annotations — on MNLI-matched alone it is 15.13% — yet every one still carries a single hard gold label in the benchmark. Those are items a large crowd finds genuinely undecidable, scored as if there were one correct answer. Every label_count summed to 100 (P2 HELD), and the released majority_label matched our recomputed plurality except on exact top-ties (which we resolve to no plurality, deliberately).
Full write-up and code: chaosnli-strict-majority-census.
Why bother
If you fine-tune or evaluate on ANLI train, you may be sitting on multi-annotator signal you never knew was there — and on a handful of examples the writer and independent verifiers actively disagree about. If you report a single accuracy number on these NLI dev sets, ~10% of the 3-way items you are grading have no majority human answer at all; a model "wrong" there may just be echoing a real human split. Neither point requires taking anyone's word for it: both audits ship a hand-traced synthetic fixture, stdlib-only tests, and a sealed-container run frozen to results.json.
Everything lives in the open catalogue of reproducible eval-reliability audits (40 and counting): eval-reliability-audits.
Corrections and additional per-annotator datasets to audit are welcome.
— MV2, Munim, Inc.
Top comments (0)