LLM Evaluation, Explained: How a Benchmark Produces Comparable Numbers (TL;DR)

TL;DR: LLM evaluation only becomes comparable when every model sees the same prompts, is scored by the same fixed judge, and leaves a public verbatim trail you can re-read. This post walks through how the LFORLA FreeCAD Fix benchmark does exactly that, then shows the real leaderboard numbers and what they mean in practice.

How the evaluation works

FreeCAD Fix is a CAD repair task: the model must diagnose and repair a broken parametric FreeCAD script. It is scored by a deterministic geometry oracle against real freecadcmd measurements — no LLM judge. That single design choice removes a huge source of variance in LLM evaluation.

Here is the mechanism, step by step:

1. Same prompts. Every model receives the identical broken script and the same repair instruction. No prompt tuning per model, no retries with different wording.
2. Same fixed judge. Scoring is done by the geometry oracle, not by a second model. The oracle runs freecadcmd, measures the resulting geometry, and compares it to the expected result. A deterministic judge cannot be talked into a better score.
3. Per-axis rubrics. The overall score is not a single vibe check. It is composed from named axes, each scored 0–100%: Safety, Runs Clean, Fix Locality, Script Validity, Feature Accuracy, Geometry Validity, and Dimension Accuracy. Each axis has a clear definition and a higher-is-better direction.
4. Public verbatim trail. Anyone can re-read what a model actually answered. The trail is the audit layer of LLM evaluation: if a number looks surprising, you can inspect the script the model produced and the measurements the oracle took.

Because the judge is deterministic and the prompts are fixed, two runs of the same model should produce the same score. That is what makes the numbers comparable across models and across time.

The real numbers

Model	Score	Provider
Qwen 3.8 27B (free, via OpenRouter)	99.33	opencode
DeepSeek V4 Pro	96.3	opencode-zen
Longcat 2.5 Preview (free, via opencode)	90	opencode
Nemotron 3 Ultra 550B (free, via OpenRouter)	83.33	opencode
Nemotron 3 Ultra (free, via opencode)	66.67	deepseek
Space Bunny (free, via opencode)	60.33	opencode
Ling 3.1 Flash (free, via opencode)	58.33	opencode
Laguna S 2.1 (free, via OpenRouter)	56.67	opencode

For transparency: we submitted our own model, GLM 5.2, to this benchmark. It scored 78.0 overall — below Qwen 3.8 27B and DeepSeek V4 Pro, above the Nemotron 3 Ultra variant at 66.67.

Why the ranking looks this way

A score of 99.33 means the model repaired the script in a way that satisfied nearly every axis: the script ran clean, the geometry was valid, the dimensions matched, and the fix was local rather than a rewrite. A score in the 50s or 60s means the model produced something that ran but drifted on geometry or dimensions — the kind of failure that looks fine until you measure it.

The spread is wide because CAD repair is unforgiving. A single wrong dimension fails Dimension Accuracy even if the script is syntactically perfect. That is why per-axis rubrics matter for LLM evaluation: the overall score hides which axis broke, and the verbatim trail lets you find out.

Note also that the same model family can land differently depending on the provider route. Nemotron 3 Ultra 550B scores 83.33 via OpenRouter, while Nemotron 3 Ultra via opencode scores 66.67. In a deterministic benchmark, that gap points to real differences in what was served, not judge noise.

What this means for practitioners

When you choose a model for engineering or code-repair work, a single headline number is not enough. Look at the axes. A model with high Script Validity but low Dimension Accuracy will produce code that runs and is still wrong. For CAD, dimensions are the product. For your domain, find the axis that maps to your failure mode and weight it accordingly.

How to read a leaderboard

• Check whether the judge is deterministic or another LLM. Deterministic judges are reproducible; LLM judges add variance.
• Confirm the prompts are fixed and published. If prompts vary per model, the comparison is not apples to apples.
• Read the per-axis breakdown, not just the overall score. Find the axis that matches your use case.
• Open the verbatim trail for at least one mid-ranked model. See what a 60 actually looks like.
• Check whether the benchmark authors submitted their own model, and where it landed. Ours did, and we published the 78.0.

Honest limitations

FreeCAD Fix measures CAD repair, not general reasoning. A high score here does not transfer automatically to other tasks. The benchmark is also narrow by design: one task family, one oracle. And provider routing can change what a model name actually serves, as the Nemotron split shows. Treat any single benchmark as one signal among several.

Conclusion

Comparable LLM evaluation comes from discipline: same prompts, a fixed judge, per-axis rubrics, and a public trail. FreeCAD Fix applies that discipline to CAD repair, and the numbers above are what come out. Explore the full leaderboard and the verbatim answers at https://lforla.org.