Surface Form Competition: Why Log-Prob Answer Scoring Fails

Your eval harness scores four candidate answers by summing token log-probabilities and picks the highest. The model "knows" the answer is a nuclear power plant, but the harness returns coal — because the right answer is four tokens longer and every extra token multiplies in another probability less than one. Nothing is broken. The scoring function is doing exactly what you asked. It's just measuring the wrong thing.

This failure has a name: surface form competition. If you rank answer options by log P(option | prompt), you are ranking strings, not meanings, and strings compete with each other in ways that have nothing to do with whether the model knows the answer.

TL;DR

• Log-prob answer scoring conflates three things: whether the answer is right, how long its surface form is, and how a-priori likely that string was anyway.
• Length bias is the easy half. Normalize by bytes, not tokens — token counts are tokenizer-dependent, so per-token normalization is not comparable across models or even across options.
• Surface form competition is the hard half. Probability mass splits across paraphrases (computer / a computer / the computer), so no length normalization can rescue a correct answer whose mass is spread thin.
• PMI / domain-conditional normalization is the standard fix: score log P(a | question) − log P(a | domain premise) to subtract out the option's a-priori plausibility.
• Switching to A/B/C/D doesn't dodge the problem — it trades length bias for token-prior and position bias. Fix that with contextual calibration and cyclic permutation, not by hoping it averages out.

What is surface form competition in LLM evaluation?

Surface form competition is what happens when one meaning has many valid spellings and each spelling gets its own slice of the probability mass. An autoregressive LM must put mass on strings. If the correct concept can be written as a nuclear power plant, nuclear power, nuclear plants, or nuclear energy, each of those gets a fraction, and each competes against a single-form distractor like coal that hogs all of its own mass.

The model can be perfectly calibrated about the concept and still rank the wrong string first. This is a measurement artifact of your scoring function, not a knowledge failure — and it means an eval regression can appear when nothing about the model's understanding changed.

Length bias rides along on top of it. Under the chain rule:

log P(a | q) = Σ_i log P(a_i | q, a_<i)

Every additional token adds a negative term. If the model's average per-token log-probability in this context is roughly −H, the sequence score scales like −H · L. Longer answers lose by construction. This is why lm-evaluation-harness reports both acc and acc_norm, and why on datasets with long, uneven endings (HellaSwag is the canonical case) the two numbers are visibly different — sometimes by enough to reorder a leaderboard.

Does normalizing by length fix it?

It fixes the length half, and only if you normalize by the right unit.

Do not divide by token count. Tokenization is model-specific: the same answer string is 3 tokens under one tokenizer and 6 under another, and within a single option set, an answer full of rare proper nouns fragments far more than a common phrase. Per-token normalization silently rewards options that tokenize efficiently.

Divide by byte length (len(option.encode("utf-8"))). Bytes are tokenizer-invariant, so the same normalized score is comparable across models — this is exactly what acc_norm does.

What byte normalization does not fix is mass splitting. If a computer and the computer and computer each take a third of the concept's mass, dividing each by its own length leaves all three below a distractor that never had competition. You need a second correction.

What does PMI normalization actually do?

It divides out the option's prior. Instead of ranking by log P(a | q), rank by a pointwise-mutual-information-style score:

score(a) = log P(a | q) − log P(a | domain_premise)

The second term is the option's log-probability under a content-free premise from the same domain — "Answer:" for a QA task, "The sentence continues:" for a completion task. Subtracting it asks a sharper question: how much did seeing the question raise this string's likelihood? A string that was already probable (short, frequent, generic) gets docked; a string that only became probable because of the question gets rewarded.

Use a domain-conditional premise rather than the fully unconditional P(a). Unconditional normalization over-corrects toward rare, weird strings, because rarity alone maximizes the ratio. Keeping the premise inside the task's domain and format cancels the domain prior without handing the win to nonsense.

Here's all four scorers, computed from one forward pass per (prompt, option) pair:

import torch
import torch.nn.functional as F

@torch.no_grad()
def seq_logprob(model, tok, prompt: str, cont: str):
    """Sum of log P(cont | prompt) plus its token and byte lengths."""
    p_ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    full_ids = tok(prompt + cont, return_tensors="pt").input_ids.to(model.device)
    n_p = p_ids.shape[-1]

    # Boundary check: BPE can merge the last prompt char with the first
    # continuation char, which shifts every index below by one.
    assert torch.equal(full_ids[0, :n_p], p_ids[0]), "tokenizer merged across the boundary"

    logits = model(full_ids).logits                       # [1, T, V]
    logprobs = F.log_softmax(logits[:, :-1].float(), dim=-1)
    targets = full_ids[:, 1:]                             # next-token targets
    tok_lp = logprobs.gather(-1, targets.unsqueeze(-1)).squeeze(-1)[0]

    cont_lp = tok_lp[n_p - 1:]        # predictions for the continuation tokens
    return cont_lp.sum().item(), cont_lp.numel(), len(cont.encode("utf-8"))


def score_option(model, tok, question: str, option: str, premise: str = "Answer:"):
    lp, n_tok, n_bytes = seq_logprob(model, tok, question, option)
    lp_prior, _, _ = seq_logprob(model, tok, premise, option)
    return {
        "raw":       lp,               # length-biased; do not ship this
        "per_token": lp / n_tok,       # tokenizer-dependent; not comparable
        "per_byte":  lp / n_bytes,     # this is acc_norm
        "pmi_dc":    lp - lp_prior,    # domain-conditional PMI
    }

Two details that quietly corrupt results if you skip them:

1. Own the whitespace. Strip trailing spaces from the prompt and put the leading space on the continuation (" a nuclear power plant"). A prompt ending in a space makes the model predict a token that almost never starts a word, and it tanks the score of every option equally except the ones where the merge happens to work out.
2. Assert the prefix property. The assert above is not paranoia. Byte-pair merges across the prompt/continuation boundary shift n_p and make you score the wrong slice — silently, with plausible-looking numbers.

Why doesn't the A/B/C/D format avoid this?

Because it swaps one bias for two. Reformatting as "answer with a single letter" makes every option exactly one token long, which kills length bias and mass splitting outright. What you get instead:

• Token prior bias. The model has a baseline preference over " A", " B", " C", " D" before it reads anything. That prior is often far from uniform.
• Position bias. Move the correct answer to a different slot and the prediction changes. This is the same class of artifact as position bias in LLM-as-judge setups, and it does not average out on small eval sets.
• Symbol binding failure. Weaker or base models often "know" the content but can't reliably map it to the letter, which reads as a knowledge failure and isn't one.

The fix for the prior is contextual calibration: measure the model's letter distribution on a content-free input, then apply a diagonal affine correction that flattens it.

import numpy as np

LETTERS = ["A", "B", "C", "D"]

def letter_probs(logprob_fn, prompt: str) -> np.ndarray:
    """Softmax restricted to the four letter tokens."""
    lps = np.array([logprob_fn(prompt, letter) for letter in LETTERS])
    e = np.exp(lps - lps.max())
    return e / e.sum()

def calibrated_pick(logprob_fn, render, question, options,
                    content_free=("N/A", "", "[MASK]")):
    p = letter_probs(logprob_fn, render(question, options))
    # Same template, same options, no question content.
    p_cf = np.mean([letter_probs(logprob_fn, render(cf, options))
                    for cf in content_free], axis=0)
    W = 1.0 / p_cf                       # diag(p_cf)^-1, b = 0
    return LETTERS[int(np.argmax(W * p))]

The calibration set matters: use the same rendered template and the same option strings, with only the question replaced. You are estimating the template's bias, not the model's general letter preference.

If your provider doesn't return token log-probabilities, this method isn't available directly. OpenAI-style APIs expose logprobs / top_logprobs and you can pin the output to the letter set with logit_bias. Anthropic's Messages API does not return token log-probs, so for Claude Opus 4.x or Sonnet 4.x you get the equivalent debiasing behaviorally: run each item under cyclic permutations of the option order and take a majority vote. Four permutations for a four-way item costs 4× the calls, and it removes both position bias and letter-prior bias without needing scores at all. Structured output (a single-field schema constrained to the letter set) keeps parsing clean.

Which scorer should you actually ship?

Setup	Scorer	Why
Base model, cloze/completion format	per-byte + PMI-DC	Length and prior both bite; report both, they disagree informatively
Base model, uneven-length options	PMI-DC	Length norm alone leaves mass splitting untouched
Instruction-tuned model, letter format	contextual calibration + cyclic permutation	No length bias; prior and position bias dominate
No log-prob access (Claude, most hosted chat APIs)	cyclic permutation + majority vote	Behavioral equivalent of calibration

Two operating rules. First, report the scorer alongside the number — "72% on ARC-Challenge" is not a claim until you say whether that's acc, acc_norm, or PMI. Second, fix the scorer before you fix the model: a scoring change can move a benchmark several points, which is comfortably larger than most real gains you're chasing, and you don't want to spend a week attributing one to the other.

Failure-mode checklist

• Prompt ends in a space → whitespace collides with the continuation's leading token.
• Prompt is not a token-level prefix of prompt+continuation → your slice indices are off by one.
• Normalizing by token count → results not comparable across tokenizers.
• Unconditional PMI instead of domain-conditional → rare, malformed options win.
• Letter format with no calibration → you're partly measuring the model's fondness for " C".
• Fixed option order across the whole eval set → position bias baked into the headline number.

The short answer

Log-prob answer scoring fails because of surface form competition: log P(option | prompt) measures the likelihood of a string, which bundles together correctness, length, and the string's a-priori frequency. Longer answers accumulate more negative log terms, and correct answers with many valid paraphrases have their probability mass split across them, so a short single-form distractor can outrank an answer the model actually knows. Normalize by byte length to remove the length term, subtract a domain-conditional prior (PMI) to remove the frequency term, and if you switch to A/B/C/D letters, apply contextual calibration and permute the option order — because that format doesn't eliminate the bias, it just moves it somewhere less visible.