How LLM Evaluation Produces Comparable Numbers: Inside the LFORLA Satellite Geo QCM Benchmark

TL;DR: LLM evaluation only becomes useful when every model faces the same prompts, the same scoring rules, and a public trail you can re-read. The LFORLA Satellite Geo QCM benchmark does exactly that: a deterministic multiple-choice task with no LLM judge, per-difficulty accuracy, and verbatim answers. The leaderboard shows a wide spread, from 66.67 to 100, and our own GLM 5.2 submission landed at 78.0.

How the evaluation works

The Satellite Geo QCM benchmark is a geolocation multiple-choice test. Each model sees an aerial or satellite image and must pick the correct location — a city, region, or landmark — among four options. That design removes a lot of the usual ambiguity in LLM evaluation.

Here is the mechanism, step by step:

1. Same prompts for every model. Every model receives the identical image and the identical four answer choices. No model gets a friendlier phrasing or a different crop.
2. No LLM judge. Scoring is fully deterministic: the answer is either the correct option or it is not. There is no second model grading the first, so there is no judge bias to argue about.
3. Per-axis rubrics. The benchmark reports accuracy overall and by difficulty. That means you can see whether a model struggles on easy items or only falls apart on hard ones.
4. Public verbatim trail. Anyone can re-read what a model actually answered. If a score looks surprising, you can inspect the exact response instead of trusting a summary.

This is the core of serious LLM evaluation: comparability comes from fixed inputs, fixed scoring, and transparency.

The real numbers

Here is the leaderboard as published by LFORLA for Satellite Geo QCM:

Model	Score	Provider
Nemotron 3 Ultra 550B (free, via OpenRouter)	100	opencode
Laguna S 2.1 (free, via OpenRouter)	100	opencode
Qwen 3.8 27B (free, via OpenRouter)	100	opencode
DeepSeek V4 Flash Vision (exp)	98.18	deepseek
Nemotron 3 Ultra (free, via opencode)	81.82	deepseek
Mimo v2.6 Flash (free, via opencode)	66.67	opencode
Space Bunny (free, via opencode)	66.67	opencode
Longcat 2.5 Preview (free, via opencode)	66.67	opencode

Our own submission, GLM 5.2, scored 78.0 overall.

Why the ranking looks this way

The scores are accuracy percentages on a multiple-choice task. A score of 100 means the model picked the correct location every time. A score of 66.67 means it got roughly two out of every three right. The gap between 98.18 and 81.82 is meaningful: DeepSeek V4 Flash Vision is nearly perfect, while Nemotron 3 Ultra via opencode is solid but clearly behind.

The three models tied at 66.67 sit at the same accuracy level, which suggests they share a similar failure pattern on this task. Because scoring is deterministic, those ties are real, not judge noise.

The 100 scores deserve a caveat: on a finite benchmark, a perfect score can mean the task is saturated for that model, or that the sample is small enough for a clean sweep. The per-difficulty breakdown and the verbatim trail are what let you tell the difference.

What this means for practitioners choosing a model

For LLM evaluation, a single leaderboard number is a starting point, not a verdict. If your application involves visual geolocation, the Satellite Geo QCM results are directly relevant: a model at 100 is a safer bet than one at 66.67. But if your task is text summarization, this benchmark tells you almost nothing.

Use the per-axis scores to match the benchmark to your workload. A model that is strong overall but weak on hard items may still be fine for easy, high-volume queries. And always check the verbatim trail: reading a few actual answers tells you more about failure modes than any aggregate score.

How to read a leaderboard: a short checklist

• Check the scoring method. Is it deterministic, or is an LLM judge involved? Deterministic scoring is easier to reproduce.
• Look at the spread, not just the top. A 100 at the top and a 66.67 at the bottom tells you the task discriminates between models.
• Read the per-axis breakdown. Overall accuracy can hide weak performance on hard items.
• Inspect the verbatim trail. Re-read actual model answers before trusting a score.
• Match the benchmark to your task. A geolocation benchmark is not a general intelligence test.

Honest limitations

This benchmark measures one narrow skill: picking a location from a satellite image among four options. It does not test reasoning, instruction following, or long-context behavior. The sample size is not stated in the data we have, so perfect scores should be treated cautiously. Provider labels in the leaderboard are inconsistent — some models listed under opencode are described as coming via OpenRouter — so read the model names carefully. Finally, our own GLM 5.2 result at 78.0 is a single submission, not a controlled comparison.

Conclusion

LLM evaluation becomes trustworthy when the mechanism is visible: same prompts, deterministic scoring, per-axis rubrics, and a public trail. The Satellite Geo QCM leaderboard is a clean example of that approach, and the numbers are worth reading closely. Explore the full leaderboard and methodology at lforla.org.