LLM Evaluation That Actually Compares: Inside the FreeCAD Fix Benchmark

TL;DR: Most LLM evaluation is a vibe check with a leaderboard. The LFORLA FreeCAD Fix benchmark is different: it uses the same prompts, a deterministic geometry oracle instead of an LLM judge, per-axis rubrics, and a public verbatim trail so you can re-read exactly what each model answered. Here is how it produces numbers you can actually compare.

How the evaluation works

The FreeCAD Fix benchmark asks a single, concrete task: diagnose and repair a broken parametric FreeCAD script. The model receives the same broken script and the same instructions every time. No prompt drift, no hidden context, no creative reinterpretation.

Scoring is where most benchmarks get fuzzy. This one does not use an LLM judge. Instead, a deterministic geometry oracle runs the repaired script through real freecadcmd measurements. That means the score is not an opinion about the answer; it is a measurement of what the script actually produces when executed. If the geometry is wrong, the oracle says so. If the script fails to run, that is recorded too.

The benchmark breaks performance into seven per-axis rubrics, each scored 0 to 100 percent: Safety, Runs Clean, Fix Locality, Script Validity, Feature Accuracy, Geometry Validity, and Dimension Accuracy. These axes separate failure modes that a single overall score would blur. A model that writes a safe but useless script scores differently from one that writes a risky script that happens to run.

Finally, every run leaves a public verbatim trail. Anyone can re-read what the model actually answered, not just the summary score. That is the difference between a leaderboard and an audit trail.

The real numbers

Here is the current leaderboard for the FreeCAD Fix benchmark. Scores are overall percentages.

Model	Score	Provider
Qwen 3.8 27B (free, via OpenRouter)	99.33	opencode
DeepSeek V4 Pro	96.3	opencode-zen
Longcat 2.5 Preview (free, via opencode)	90	opencode
Nemotron 3 Ultra 550B (free, via OpenRouter)	83.33	opencode
Nemotron 3 Ultra (free, via opencode)	66.67	deepseek
Space Bunny (free, via opencode)	60.33	opencode
Ling 3.1 Flash (free, via opencode)	58.33	opencode
Laguna S 2.1 (free, via OpenRouter)	56.67	opencode

We also submitted our own model, GLM 5.2, which scored 78.0 overall. That places it between the two Nemotron entries in this snapshot.

Why the ranking looks this way

The spread from 99.33 to 56.67 is large, and that is the point. A deterministic oracle punishes models that cannot produce valid geometry, regardless of how fluent their explanation sounds. Qwen 3.8 27B tops the list because it consistently produces scripts that run clean and match the required dimensions. DeepSeek V4 Pro is close behind, suggesting strong code reasoning under the same fixed prompt.

The drop to the 60s and 50s is where per-axis rubrics matter. A model can score well on Safety, meaning it avoids dangerous operations, while failing Geometry Validity or Dimension Accuracy. The overall score compresses those axes, but the verbatim trail lets you see which axis dragged the number down. That is actionable in a way a single rank is not.

Notice also that provider matters less than model behavior here. The same benchmark harness runs across opencode, opencode-zen, and OpenRouter, so differences reflect the model, not the plumbing.

What this means for practitioners choosing a model

If you are picking a model for CAD repair, code generation, or any task with a verifiable output, prefer benchmarks with deterministic scoring and public transcripts. A high score on a judge-based benchmark tells you a model writes answers that another model likes. A high score here tells you the script ran and the geometry was correct.

Use the per-axis breakdown to match your risk profile. If a wrong dimension is expensive, weight Dimension Accuracy and Geometry Validity heavily. If a script that runs but does nothing is acceptable, Runs Clean may matter more than Feature Accuracy.

How to read a leaderboard: a checklist

• Check whether the judge is deterministic or another LLM. Deterministic oracles are harder to game.
• Look for per-axis scores, not just an overall rank. The axes tell you where a model fails.
• Demand a public verbatim trail. If you cannot re-read the answer, you cannot verify the score.
• Confirm the prompts are fixed and published. Prompt drift makes comparisons meaningless.
• Note the snapshot date and provider. Leaderboards are snapshots, not permanent truths.

Honest limitations

This benchmark covers one task family: repairing broken parametric FreeCAD scripts. A model that excels here may not excel at open-ended design or natural-language CAD instruction. The overall score compresses seven axes into one number, which hides trade-offs unless you read the trail. And leaderboards age: model versions change, providers update routing, and scores move. Treat any single number as a snapshot, not a verdict.

Conclusion

LLM evaluation is only useful when it produces comparable, verifiable numbers. The FreeCAD Fix benchmark does that with fixed prompts, a deterministic geometry oracle, per-axis rubrics, and a public verbatim trail. You can inspect the full leaderboard and methodology at lforla.org. If you are choosing a model for verifiable tasks, start there and read the trail, not just the rank.