How an LLM Evaluation Benchmark Produces Comparable Numbers: Inside the LFORLA Mechanism

TL;DR: A leaderboard is only useful if you can trust the numbers. LFORLA's LLM evaluation makes scores comparable by fixing the prompts, fixing the judge, scoring on per-axis rubrics, and publishing a verbatim trail so anyone can re-read what a model actually answered. Below is how that works in practice, using the Team Recruitment (Oracle) benchmark as a concrete example.

How the evaluation works

The Team Recruitment (Oracle) benchmark is a recruitment-agent task. The model queries an oracle of résumés, then builds a team. It is scored on composition criteria, with binary gates for budget and seat constraints. The mechanism that turns those interactions into a number has four moving parts.

1. Same prompts for every model. Each model receives the identical task description, the identical oracle interface, and the identical constraints. No model gets a friendlier phrasing or extra context. This is the baseline for comparability: if the input differs, the output cannot be compared.

2. Same fixed judge. A single judge configuration evaluates every submission. The judge is not swapped per model, and it is not tuned after seeing results. Fixing the judge removes a common source of leaderboard drift, where a model appears to improve simply because the grading changed.

3. Per-axis rubrics. Instead of one opaque score, the evaluation breaks performance into axes. For this benchmark, composition criteria are scored separately, and the budget and seat constraints act as binary gates: pass or fail. Per-axis rubrics make it possible to see where a model is strong and where it collapses, rather than only seeing a single aggregate.

4. A public verbatim trail. Every answer is stored and published verbatim. Anyone can re-read what a model actually said, not just the score it received. This is the accountability layer: if a score looks surprising, you can inspect the transcript and judge for yourself.

The real numbers

Here is the current leaderboard for Team Recruitment (Oracle):

Model	Score	Provider
Nemotron 3 Ultra (free)	90.87	opencode-zen
HY3 (free)	83.1	opencode-zen
GLM 5.2 (our own submission)	78.0	—

Why the ranking looks this way

The gap between 90.87 and 83.1 is meaningful in this task because the benchmark rewards correct team composition under hard constraints. A model that satisfies the binary budget and seat gates while also meeting the composition criteria will pull ahead. A model that misses a gate or misallocates seats loses points on the axes tied to those criteria.

The 78.0 for GLM 5.2, our own submission, sits below both leaders. We are publishing it because a benchmark is only credible if the people running it are willing to show their own results, including the unflattering ones. The score tells you where our model stands on this specific task; it does not tell you it is worse at everything.

What this means for practitioners choosing a model

A leaderboard score is a signal about a specific task under specific constraints, not a universal ranking. The Team Recruitment benchmark measures recruitment-agent behavior: querying an oracle, composing a team, and respecting budget and seat limits. If your use case looks like that, the ranking is directly relevant. If your use case is summarization or code generation, it is not.

The per-axis rubrics are the more useful artifact. They tell you which dimensions were measured, so you can ask whether those dimensions matter for your workload. The verbatim trail lets you check whether a high score reflects genuine competence or a lucky pass on a binary gate.

How to read a leaderboard: a short checklist

• Check the task. Confirm the benchmark actually resembles your use case before you weight the score.
• Check the judge. Ask whether the judge is fixed and whether it is disclosed. A moving judge makes historical comparisons unreliable.
• Read the axes. Look at per-axis results, not just the aggregate. A single number hides where a model fails.
• Inspect the trail. Open a few verbatim answers. If the transcript does not support the score, treat the score with suspicion.
• Note the gates. Binary constraints like budget and seat limits can dominate a score. Understand which gates exist.

Honest limitations

This benchmark covers one task family. Scores on Team Recruitment do not transfer automatically to other domains. The binary gates mean a single constraint failure can have an outsized effect on the final number, which compresses nuance. The leaderboard shown here is a snapshot, and models change. Finally, we submitted our own model, so read our interpretation with that conflict of interest in mind; the verbatim trail exists precisely so you do not have to take our word for it.

Conclusion

Comparable LLM evaluation comes down to discipline: same prompts, same judge, per-axis rubrics, and a public trail. The Team Recruitment (Oracle) leaderboard shows Nemotron 3 Ultra at 90.87, HY3 at 83.1, and our GLM 5.2 at 78.0. Explore the full results and the verbatim answers at lforla.org.