How LLM Evaluation Actually Works: Inside the LFORLA Team Recruitment Benchmark

TL;DR: LLM evaluation only produces comparable numbers when every model faces the same prompts, the same fixed judge, and the same per-axis rubrics — and when the raw answers are public so anyone can re-read them. This post walks through how the LFORLA Team Recruitment benchmark does exactly that, using real scores from its leaderboard.

Why LLM evaluation needs a mechanism, not a vibe

Ask five people to rank language models and you will get five different answers. Ask them why and you will usually get a vibe: it felt smarter, it wrote nicer prose, it answered faster. That is fine for a demo and useless for a decision. A benchmark earns trust only when it can explain, step by step, how a number was produced. That is the whole point of a mechanism explainer.

The LFORLA Team Recruitment benchmark, called Team Recruitment (Oracle) with the slug recruit-equipe, is a good case study because its task is concrete: a recruitment agent must query an oracle of résumés and then build a team. The output is scored on composition criteria, with binary gates for budget and seat constraints. No prose quality, no charisma — just whether the team is valid and how well it is composed.

How the evaluation works

The mechanism has four moving parts, and each one exists to remove a source of noise.

1. Same prompts. Every model receives an identical task description and an identical oracle interface. The résumés do not change between runs. If model A sees a different candidate pool than model B, the comparison is meaningless. Fixed prompts are the baseline of any credible LLM evaluation.

2. Same fixed judge. Scoring is not delegated to a different model per submission. A single fixed judge applies the rubric consistently. This matters because judge drift is one of the quietest ways a leaderboard becomes fiction: if the judge changes, the scores change, and history becomes incomparable.

3. Per-axis rubrics. The benchmark does not collapse everything into one fuzzy score. It scores composition criteria on separate axes, and it applies binary gates for budget and seat constraints. A team that exceeds the budget or fills the wrong number of seats fails the gate regardless of how elegant the composition looks. Binary gates are harsh, but they are honest: they encode hard requirements as hard requirements.

4. A public verbatim trail. Every answer is preserved so anyone can re-read what a model actually answered. This is the part most benchmarks skip. Without a verbatim trail, you are trusting the judge. With it, you can audit the judge. You can disagree with a score and point to the exact response that produced it.

The real numbers

Here is the current leaderboard for Team Recruitment (Oracle):

Rank	Model	Score	Provider
1	Nemotron 3 Ultra (free)	90.87	opencode-zen
2	Qwen2.5 7B Instruct	88.46	local
3	HY3 (free)	83.1	opencode-zen

And for transparency: we submitted our own model, GLM 5.2, to this same benchmark. It scored 78.0 overall. We are publishing that number next to the leaders because a benchmark you only cite when you win is not a benchmark.

Why the ranking looks this way

The gap between first and third is about 7.8 points. On a task with binary gates, that gap usually does not come from one catastrophic failure. It comes from how often a model clears the gates and how well it composes the team once it does. A model that occasionally overshoots the budget or miscounts seats loses points in chunks, not slivers.

The interesting result is Qwen2.5 7B Instruct at 88.46 from a local provider, sitting between two hosted free models. That is a useful reminder that LLM evaluation measures task behavior, not brand size. A smaller model that follows the oracle protocol cleanly and respects the constraints can outperform a larger one that is sloppier about gates.

GLM 5.2 at 78.0 is a real result, not a rounding error. It tells us where our model stands on this specific task under this specific judge. It does not tell us GLM 5.2 is worse at everything — it tells us it is worse at this recruitment composition task as measured here.

What this means for practitioners choosing a model

If you are picking a model for an agent that must satisfy hard constraints — budgets, seat counts, schema validity — a benchmark with binary gates is more informative than a general chat score. Look for the task shape that resembles yours. A model that tops a prose leaderboard may still fail your budget gate.

Also look at the spread, not just the winner. A 90.87 versus an 88.46 is close enough that cost, latency, and provider reliability should decide. A 78.0 versus 90.87 is a different conversation.

How to read a leaderboard

• Check the prompts are fixed. If the task changes between runs, the scores are not comparable.
• Check the judge is fixed. A rotating judge makes history meaningless.
• Look for per-axis scores and gates. A single aggregate hides whether a model failed hard constraints.
• Demand the verbatim trail. If you cannot re-read the answer, you cannot audit the score.
• Read the provider column. Local and hosted runs are not the same deployment context.

Honest limitations

This benchmark measures one task: recruitment composition against an oracle with budget and seat gates. It does not measure reasoning breadth, multilingual ability, or long-context recall. The leaderboard is small — three models plus our own submission — so rank stability is limited. Scores are judge-dependent even with a fixed judge; a different rubric would produce different numbers. And a single overall score, like GLM 5.2 at 78.0, compresses per-axis detail that practitioners should inspect before drawing conclusions.

Conclusion

LLM evaluation is not a number. It is a mechanism: same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail. The Team Recruitment benchmark shows what that looks like in practice, including the uncomfortable parts — like publishing our own 78.0 next to a 90.87 leader.

Explore the full leaderboard and the verbatim answers at https://lforla.org.