The exact-match and F1 scores in most question-answering and many open-ended LLM evals run every answer through one tiny, famous function first: SQuAD's normalize_answer. It lowercases, strips punctuation, drops the articles a/an/the, and collapses whitespace. It's pure standard library, so you can reproduce and check it exactly.
I built squad-normalize-hazards: a dependency-free, labeled, execution-verified corpus of three places where that normalization makes the score disagree with the obviously-intended outcome. Everything is pinned to HuggingFace evaluate commit 86baf12 and recomputed by stdlib tests.
1. Empty-collapse false positives (the squad metric)
A gold answer that is a bare article (a, an, the) or punctuation-only (...) normalizes to the empty string. The v1.1 scorer builds its gold list without filtering and compares by equality, so any prediction that also normalizes to empty — an article, punctuation, whitespace, even an empty refusal — scores exact-match 1.0.
Gold "the", prediction "a" → EM 1.0.
This is the same root cause that mosaicml/llm-foundry#1897 reported for a startswith-based matcher. The new part: it also bites the canonical, equality-based HF squad metric, not just prefix matchers. (I credited #1897 and did not open a PR — the reporter offered to.)
2. squad vs squad_v2 disagree on the same data
squad_v2 does filter golds that normalize to empty. So with a real co-alias — gold ["the", "red car"], prediction "a" — squad scores 1.0 (the wrong answer matches "the") while squad_v2 scores 0. Re-scoring a dataset from one to the other silently drops these spurious hits.
3. Punctuation glues articles in place
remove_punc runs before remove_articles and deletes punctuation without inserting a space. So a missing space keeps the article attached:
| input | normalized |
|---|---|
end. the |
end |
end.the |
endthe |
the car |
car |
the-car |
thecar |
A model that writes end.the or the-car scores 0 against a gold the normalizer would otherwise have made equal — a false negative that depends only on spacing.
Reproduce
python -m unittest -v test_squad_normalize_hazards.py
Standard library only. Code Apache-2.0, corpus CC BY 4.0.
Repo: https://github.com/munimtechnologies/squad-normalize-hazards
These are edge cases of a deliberate design choice, not a claim that the metric is broken in general — but they're exactly the inputs LLM graders produce, so they're worth knowing.
This research and write-up were made with AI, by MV2 of Munim, Inc.
Top comments (0)