AI Agent Evaluation: How the LFORLA JS 3D Builder Leaderboard Actually Scores Models

TL;DR: AI agent evaluation here means one thing: a model reads a written building spec, plans a structure, sources the right Three.js parts, and assembles a build plan. Every step is scored against the same rubric for every model. No execution. No LLM judge. Just server-side static analysis of the code the agent produces.

How the evaluation works

The benchmark is called JS 3D Builder (Three.js). The task is procedural generation: from a written spec, the model must produce a 3D building in Three.js. The scoring is done by server-side static analysis. That means the evaluator parses the submitted code and checks it against the rubric. It does not run the code. It does not ask another model to grade it.

Here is the mechanism, step by step.

1. Prompt. The model receives a written spec for a building. The spec describes geometry, spatial relationships, and the parts needed.
2. Plan. The model must plan the structure before writing code. This is where reasoning and spatial categories come in.
3. Source parts. The model must select and reference the correct Three.js primitives and constructors. Wrong parts mean a lower score.
4. Assemble a build plan. The model outputs code that constructs the building. The code is the artifact under evaluation.
5. Score against the rubric. Server-side static analysis checks the code against the same rubric for every model. The rubric covers code, reasoning, 3d, and spatial categories.

The key design choice is that the rubric is fixed. Every model is measured the same way. That is what makes the leaderboard comparable.

The real numbers

Model	Score	Provider
Nemotron 3 Ultra (free, via opencode)	71.46666666666667	deepseek
GLM 5.2 (our own submission)	78.0	lforla

We submitted our own model to this benchmark. The real result is above. GLM 5.2 scored 78.0. Nemotron 3 Ultra scored 71.46666666666667.

Why the ranking looks this way

The gap is not huge, but it is real. A score in the low 70s means the model can produce a working build plan, but the static analysis finds gaps. Those gaps usually show up in spatial reasoning: the model places parts in the right general area but misses precise relationships. A score of 78.0 means fewer of those gaps. The code is more consistently aligned with the spec.

Because there is no execution and no LLM judge, the score reflects the code as written. A model that writes plausible-looking code with subtle spatial errors will score lower than a model that writes less elegant code with correct structure. That is the point of static analysis: it rewards correctness over presentation.

The categories matter too. Code, reasoning, 3d, and spatial are all scored. A model can be strong in code generation and weak in spatial reasoning, and the overall score will show it. This is why AI agent evaluation on this benchmark is not just a coding test. It is a planning and spatial test as well.

What this means for practitioners choosing a model

If you are choosing a model for a 3D or spatial task, a leaderboard like this is more useful than a general coding benchmark. It tells you how the model handles a spec, not just how it writes a function. The difference between 71.47 and 78.0 is the difference between a model that needs more correction and one that needs less.

But do not pick a model on one number. Look at the categories. If your task is heavy on spatial reasoning, a model that scores well there is worth more than a model with a slightly higher overall score.

How to read a leaderboard

• Check the rubric. If the scoring method is not stated, the score is not comparable.
• Check whether the judge is a human, an LLM, or static analysis. Each has different failure modes.
• Check the categories. An overall score hides where a model is weak.
• Check the submission date. Models change. A score from six months ago may not hold.
• Check whether the benchmark authors submitted their own model. We did, and we report it openly.

Honest limitations

This benchmark does not execute the code. A build plan that passes static analysis may still fail at runtime. The rubric is fixed, which makes comparison fair but also means it may not capture every dimension of agent capability. The leaderboard currently has a small number of entries. One model at 71.47 and one at 78.0 is a start, not a definitive ranking. And our own submission is our own model, which is a conflict of interest we are disclosing rather than hiding.

Conclusion

AI agent evaluation works best when the mechanism is transparent. Here, the mechanism is: plan, source parts, assemble a build plan, and score every step against the same rubric. The numbers are real. The limitations are real. If you want to see the full leaderboard and the benchmark details, go to https://lforla.org.