How we score an election forecast (and what the score does not measure)

TL;DR: We score the reasoning before the outcome. Each answer is rated 0 to 100 on specificity, grounding and calibration by a fixed judge. After certification we add winner accuracy, calibration curves and the Brier score. A model can be wrong and score well; a vague answer always scores badly.

The protocol is public at elections.lforla.org/methodology.html.

1. Pre-registration

Every answer was collected before the vote, between 26 August and 31 October 2026, and is published verbatim. The date attached to each model is the date its forecast entered the benchmark.

This matters more than any scoring detail. If a forecast can be edited after the first results, the score measures hindsight, not forecasting.

The snapshot is sealed with SHA-256:

26c1563ea12200b9436e9ab7fd03b1af7f59fd17aea5697a2bd96a25bd38b2d9

Git tag: elections-forecasts-2026-10-31.

2. The judge

Each answer is rated by a fixed judge: DeepSeek V4 Pro at temperature 0, with a fixed prompt. The judge does not see the election result. It sees the question, the answer, and a rubric.

Using one judge for every answer keeps the scale comparable. It also creates a conflict: several models under test are from the same family as the judge. We disclose it rather than hide it. A future version should use a panel of judges and publish the disagreement.

3. The three axes

Each axis is scored 0 to 100.

Specificity

Does the answer name a candidate or party and state a probability?

• A named winner with a probability and a seat range scores high.
• A named winner with no probability scores lower.
• "It depends" scores near zero.

Grounding

Does the answer use evidence that is relevant to the race?

Polls, incumbency, district structure, the electoral calendar, the rules of the second round. A confident answer with no grounding is a guess with better typography.

Calibration

Does the stated probability match the real uncertainty?

A 95% call on a coin-flip race is worse than a 60% call on the same race, even if both name the same winner. Calibration rewards honesty about what is unknown.

4. From axes to a score

The category score is the mean of its questions. The overall score is the mean of the two categories: US 2026 and France 2027. A refusal scores 0.

The current board is dominated by models that give a named call, a probability and a reason. That is the intended behaviour. The benchmark is not neutral about vagueness.

5. What the score does not measure

It does not measure whether the forecast is right. That is the next layer, and it can only be computed after certification.

It also does not measure:

• the quality of the underlying polls;
• whether the model's theory of the race is correct;
• writing style, beyond the judge's ability to control for length;
• the model's robustness to a different prompt language.

A model can be wrong and score well. A vague answer always scores badly. Those are different failure modes, and conflating them is how forecast benchmarks become entertainment.

6. After the vote: accuracy, calibration and Brier

Once results are certified, every probabilistic call becomes a number.

Winner accuracy

The share of races where the model named the winning party or candidate. Simple, intuitive, and easy to game with a safe consensus pick.

Calibration

Group calls by stated probability and compare with outcomes. Did 70% calls happen about 70% of the time? With 18 models and two races, the sample is too small for a real calibration curve. We will publish the curve with that caveat, not instead of it.

Brier score

The Brier score is the mean squared error of the probability assigned to the outcome.

For a binary race, convert every call to the implied probability of Democratic control:

• a Democratic call at 70% gives P(dem) = 0.70;
• a Republican call at 70% gives P(dem) = 0.30.

Then:

Brier = (P(dem) - y)^2

where y = 1 if Democrats control the chamber and y = 0 otherwise.

Examples:

Call	Outcome	Brier
70% Democratic	Democrats win	0.09
70% Democratic	Republicans win	0.49
55% Democratic	Republicans win	0.3025
55% Democratic	Democrats win	0.2025

Lower is better. The important property is that a confident wrong call costs more than a hedged one. That is what makes the score useful: it prices false confidence.

7. Limits

• Small sample. Two chambers and one French presidential election are not enough for strong claims about a model's forecasting ability. We report per-race results, not a universal ranking.
• Language. Prompts are in French. Models with weaker French are disadvantaged. We disclose the prompt language and publish the verbatim answers so readers can judge.
• Judge family. The judge and some contestants share a family. The rubric is fixed, but the conflict is real.
• Style. Some models answer in one sentence; others write four paragraphs. The judge controls for substance, not length, but style still matters at the margin.
• One snapshot. A forecast is a point in time. We do not average multiple runs or cherry-pick the best one.

8. Reproduce it

• Methodology
• Full board
• US 2026 forecasts
• France 2027 forecasts
• Results page

The forecast file and its hash are published in the repository. If you find a scoring bug, the answer is not to adjust the result; it is to publish the correction and re-run the score on the same sealed inputs.

Originally published at elections.lforla.org.