AI Agent Evaluation, End to End: How LFORLA Scores a Drone Build

TL;DR: AI agent evaluation only means something when every model faces the same task, the same rubric, and a deterministic judge. LFORLA's Drone Build benchmark does exactly that: the agent must plan a mission, source parts into a bill of materials under budget and physics constraints, and ship OpenSCAD frame source that a flight-physics oracle scores. Here is how the numbers are produced, what the real leaderboard looks like, and how to read it without fooling yourself.

How the evaluation works

AI agent evaluation is often reduced to a single chat score. LFORLA's Drone Build benchmark treats it as a pipeline. The agent is given a mission and must produce a complete build plan. Every step is scored against the same rubric for every model, so the comparison is apples to apples.

Step 1: Plan. The model reads the mission and decides what kind of drone is needed. There is no partial credit for a vague plan; the plan has to survive the constraints that follow.

Step 2: Source parts. The model assembles a bill of materials, picking components under a budget and under physics constraints. This is where budget compliance and BOM completeness are measured. A missing part is a missing part, not a rounding error.

Step 3: Assemble the build plan. The model declares all-up weight, thrust, endurance, and thrust-to-weight. These declarations are checked against the oracle truth within stated bands: mass within plus or minus two percent, thrust within plus or minus five percent, endurance within plus or minus ten percent. Internal consistency checks that declared masses sum to the declared all-up weight and that declared ratios are coherent.

Step 4: Ship CAD. The model writes OpenSCAD frame source. A deterministic flight-physics oracle evaluates it. CAD validity, printability, CAD-BOM consistency, and flight performance are scored from that source, not from prose.

The judge is not a second language model giving a vibe. The physics oracle is deterministic, which means the same CAD source produces the same score every time. That is the core of credible AI agent evaluation: the scoring surface is fixed, and the agent either meets it or does not.

The real numbers

Two models appear on the public leaderboard for this benchmark. We also submitted our own model, GLM 5.2, and report its overall score honestly.

Model	Provider	Overall score
Nemotron 3 Ultra (free, via opencode)	deepseek	34.166666666666664
DeepSeek V4 Pro	opencode-zen	34
GLM 5.2 (our submission)	LFORLA	78.0

Why the ranking looks this way

The gap between 34.17 and 34 is noise, not a hierarchy. Both models cleared the same rubric and landed at nearly the same overall score. What matters is what the score is made of. The rubric spans budget compliance, mass accuracy, thrust accuracy, endurance accuracy, BOM completeness, thrust-to-weight, flight performance, CAD validity, printability, compatibility, internal consistency, and CAD-BOM consistency. A model can be strong on planning and weak on CAD, or vice versa, and still land near the same total.

Our own submission, GLM 5.2, scored 78.0. We are reporting that because AI agent evaluation is only useful when the evaluator submits to the same rubric. A high score on a benchmark you designed is not proof of superiority; it is proof that the rubric is reachable and that the pipeline runs end to end.

In practice, a score in the mid-30s means the agent produced a build plan that partially satisfies the oracle. It may have stayed under budget but missed mass accuracy, or shipped CAD that failed printability. The overall number compresses those failures into one figure. That is useful for ranking and dangerous for diagnosis.

What this means for practitioners choosing a model

Do not pick a model because it tops a leaderboard by a fraction of a point. Pick it because the axes it scores well on match your workload. If your agent needs to produce valid CAD, look at CAD validity and printability, not the overall score. If it needs to reason about budgets, look at budget compliance and internal consistency. The overall score is a summary; the axes are the decision.

Also note the provider column. The same benchmark can be run through different providers, and the provider is part of the result. A free tier via opencode and a paid provider are not the same deployment, even when the model name looks similar.

How to read a leaderboard

• Check whether the judge is deterministic or another model. Deterministic oracles are reproducible; model judges drift.
• Read the axes, not just the total. A single number hides which capability failed.
• Look for the evaluator's own submission. If the benchmark authors compete on their own rubric, the scores are more credible.
• Treat sub-point differences as ties. 34.17 versus 34 is not a ranking.
• Confirm the task matches your use case. A drone build is not a customer support agent.

Honest limitations

The public leaderboard for this benchmark currently lists two models, which is too few to draw broad conclusions about the model landscape. The overall score compresses many axes into one number, and the rubric weights are not visible in the data we have. Our own GLM 5.2 result at 78.0 comes from the same pipeline, but a single submission is not a controlled study. The benchmark measures a specific physical design task; results may not transfer to other agent workloads.

Conclusion

AI agent evaluation is a mechanism, not a vibe. LFORLA's Drone Build benchmark shows what end-to-end measurement looks like: a fixed task, a fixed rubric, a deterministic oracle, and every model scored the same way. If you want to see the full rubric and run your own agent against it, start at lforla.org.