Election Forecasting Benchmarks: How LFORLA Scores Models Before the Vote

TL;DR: LFORLA's election forecasting benchmark forces models to commit to concrete scenarios for the 2027 French presidential and 2026 US midterms. A fixed judge scores each forecast on specificity, grounding and calibration. Because correctness is only verifiable after the vote, the benchmark rewards disciplined, falsifiable predictions rather than vague hedging. The current leaderboard shows a wide spread, from 33.3 to 100, with several models tied at the top.

How the evaluation works

The mechanism is forecast elicitation. Instead of asking a model for a general opinion, the benchmark requires it to produce a concrete, checkable scenario for two real elections: the 2027 French presidential election and the 2026 US midterms. Each submission must state expected outcomes, margins, and the conditions under which the forecast would change.

A fixed judge then scores every forecast on three axes:

1. Specificity — does the model commit to concrete numbers, dates, and scenarios, or does it hide behind qualifiers?
2. Grounding — are the claims tied to observable data, historical patterns, or stated assumptions?
3. Calibration — does the model express uncertainty in a way that matches the evidence, avoiding both overconfidence and empty hedging?

The judge is fixed, so every model faces the same rubric. This matters because correctness cannot be scored at submission time: the elections have not happened yet. The benchmark therefore measures the quality of the reasoning and the discipline of the forecast, not whether the prediction eventually comes true.

The real numbers

The leaderboard below is the actual LFORLA output for this benchmark. Scores are on a 0–100 scale.

Model	Score	Provider
GLM 5.2	100	opencode-zen
HY3 (free)	100	opencode-zen
DeepSeek V4 Pro	100	opencode-zen
X Preview F (free)	100	opencode-zen
DeepSeek V4 Flash	96.3	opencode-zen
Qwen2.5 Coder 7B Instruct	88.9	local
Qwen2.5 7B Instruct	55.6	local
Llama 3.1 8B Instruct	33.3	local

Why the ranking looks this way

The top four models all score 100. In practice, this means they produced forecasts that the fixed judge found fully specific, well grounded, and appropriately calibrated. They committed to concrete scenarios without collapsing into vague language.

DeepSeek V4 Flash at 96.3 is close behind, suggesting a forecast that was nearly as disciplined but lost a few points on one axis, perhaps calibration or grounding.

The local models show a clear spread. Qwen2.5 Coder 7B Instruct at 88.9 performs strongly for a local model, while Qwen2.5 7B Instruct at 55.6 and Llama 3.1 8B Instruct at 33.3 lag behind. The gap likely reflects differences in how well each model handles structured forecasting tasks: committing to numbers, citing assumptions, and expressing uncertainty without hedging.

It is important to read these scores as a measure of forecast quality under a fixed rubric, not as a prediction of which model will be right. A score of 100 means the judge found the forecast exemplary in form; it does not mean the forecast will be correct.

What this means for practitioners choosing a model

If you need a model for election forecasting or any scenario where you must commit to falsifiable predictions, this leaderboard is a useful signal. The top-scoring models demonstrate the ability to produce structured, grounded forecasts. The local models offer a range of options depending on your resource constraints: Qwen2.5 Coder 7B Instruct is a strong local choice, while Llama 3.1 8B Instruct may need more scaffolding to reach the same level.

For production use, pair the leaderboard with your own evaluation. A model that scores well on forecast elicitation may still struggle with other tasks. Use the benchmark as one input, not the final word.

How to read a leaderboard

• Check the rubric. Understand what the judge scores. Here it is specificity, grounding and calibration, not accuracy.
• Look at the spread. A wide spread tells you the task discriminates between models. A narrow spread may mean the task is too easy or too noisy.
• Separate form from outcome. High scores mean well-formed forecasts, not correct ones. Correctness is only verifiable after the vote.
• Consider the provider. Local models may behave differently from hosted ones. Check the provider column.
• Replicate if possible. Run the benchmark with your own prompts or a subset of tasks to see if the ranking holds for your use case.

Limitations

This benchmark has clear limits. First, correctness is unverifiable until the elections occur, so the scores measure forecast quality, not predictive accuracy. Second, the fixed judge introduces its own biases; a different judge might score differently. Third, the benchmark covers only two elections, so results may not generalize to other forecasting tasks. Finally, the leaderboard includes a small number of models, and the top scores are tied at 100, which limits discrimination at the high end.

Conclusion

Election forecasting is a demanding test of a model's ability to commit, ground, and calibrate. LFORLA's benchmark makes those demands explicit and scores them consistently. Use the leaderboard as a guide, but always validate against your own needs. Explore the full results and methodology at https://lforla.org.