Before trusting a novelty judge, count the verdicts that move

A judge can keep the same accuracy while changing its mind about half the examples. The average looks calm. The individual verdicts are doing parkour. 🤸

That is a synthetic counterexample, not a new model result. It is also the first diagnostic I would put beside a novelty-judge leaderboard after reading AllenAI’s Old Ideas, Novel Problems release. [1]

An illustrative four-pair dataset. No LLM generated these decisions.

The prompt changed more than its length

AllenAI reports that GPT-5.4’s pairwise accuracy on its Human+Generated setting moved from 93.5% to 40.9% between two prompts. Those are the authors’ results, not a benchmark I reran. [1]

The useful detail is in the released templates. One asks the judge to identify the idea that conference reviewers judged more novel. The other asks it to identify the idea that is more novel. [2]

Those are related targets, but they are not identical. Predicting a reviewer-backed ordering can encourage a different decision rule from making an independent novelty assessment. The source change is a reason to inspect the evaluation contract, rather than treating prompt wording as a harmless wrapper. 🔍

I would not read this experiment as proof that one particular wording is universally better. The paper’s setup is part of the result.

Two measurements, two questions

Accuracy asks whether a decision agrees with a reference label. A flip rate asks whether the same pair receives a different decision under another setup.

For an aligned set of pair IDs:

flip rate = changed verdicts / compared pairs

You can compute that disagreement without knowing which verdict is correct. That makes it useful when reference labels are expensive, incomplete or disputed. It cannot tell you which prompt to trust.

Here is a small executable example. Both runs get three of four synthetic labels right. One fixes p4 while breaking p2, so the net accuracy movement is zero. The changed-verdict count is two. ✅

def audit(before, after, gold):
    if not before or before.keys() != after.keys() or before.keys() != gold.keys():
        raise ValueError("Expected matching, nonempty pair IDs")
    if any(v not in {"A", "B"} for rows in (before, after, gold) for v in rows.values()):
        raise ValueError("Handle missing, tied, or invalid judgments explicitly")
    n = len(before)
    accuracy = lambda rows: sum(rows[k] == gold[k] for k in gold) / n
    flips = sum(before[k] != after[k] for k in before)
    return accuracy(before), accuracy(after), flips, n
gold = dict.fromkeys(("p1", "p2", "p3", "p4"), "A")
before = dict(zip(gold, ("A", "A", "A", "B")))
after = {"p4": "A", "p3": "A", "p2": "B", "p1": "A"}
old_accuracy, new_accuracy, flips, n = audit(before, after, gold)
print(f"Accuracy: {old_accuracy:.0%} -> {new_accuracy:.0%}")
print(f"Verdicts changed: {flips}/{n} ({flips / n:.0%})")

Output:

Accuracy: 75% -> 75%
Verdicts changed: 2/4 (50%)

The second dictionary is deliberately in a different row order. The function aligns by ID, so sorting or asynchronous completion does not manufacture disagreement. I also checked unchanged decisions, completely reversed decisions, empty inputs, mismatched IDs and an invalid tie value.

⚠️ Do not silently join only successful rows. If a prompt produces missing or malformed judgments, dropping them changes the denominator. This small function rejects unequal coverage and nonbinary outputs so the caller has to decide how abstentions and failures are reported.

For a real pipeline, reject duplicate IDs before turning records into dictionaries. Normalize any swapped A/B presentation back to stable idea identities. Otherwise a position swap can look like a changed scientific opinion.

The labels deserve their own audit

AllenAI separates Human-Only from Human+Generated data. The latter pairs reviewer-supported human contributions with ideas from a simple generator; the generated side is weakly labeled lower-novelty. The dataset documentation also says the human abstracts were edited to remove results, baselines and evaluation datasets. “Human” describes the idea’s origin, not a promise of untouched wording. [3]

That matters when applying the benchmark elsewhere. A weak label can support a useful controlled study without becoming a universal verdict on every generated idea.

The released expert annotation file is another easy denominator trap: its 30 cases contain 28 judge errors and two controls. It is a deliberately selected error analysis, not a random sample from which to estimate overall accuracy. [3]

💡 My practical reporting bundle would keep the reference-label metric, aligned flip rate, invalid-output counts and exact prompt versions together. It would also separate results by data setting rather than merge human and generated ideas into one reassuring average.

This article checks source contracts and a synthetic audit program. I did not call any of the judges. A stable verdict is not necessarily a correct verdict, and a novelty score is not proof that no prior work exists.

References and figure credit

[1] AllenAI’s release and reported results.

[2] Reviewer-referenced prompt and prompt without the reviewer reference.

[3] Pinned dataset documentation, transformations and annotation sampling.

Original synthetic verdict grid; no benchmark examples or reviewer excerpts redistributed.

AI disclosure

AI was used to draft the article.