LLM Evaluation: How a Benchmark Produces Comparable Numbers

LLM Evaluation: How a Benchmark Produces Comparable Numbers

TL;DR: LLM evaluation is only as good as its mechanism. LFORLA's RL Game Agents benchmark uses the same prompts, a fixed judge, per-axis rubrics, and a public verbatim trail so you can re-read exactly what a model answered. Here is how that turns into a leaderboard you can trust.

Most leaderboards give you a number. Fewer show you how that number was made. In LLM evaluation, the mechanism is the product. If the prompts change, the judge drifts, or the scoring axes are opaque, the ranking is just vibes with a decimal point.

This post walks through the mechanism behind one concrete benchmark: RL Game Agents (Training Code Oracle). It is a coding benchmark with a reinforcement learning twist, and it is a clean example of how to produce comparable numbers.

How the evaluation works

The benchmark asks a model to write an RL training script for a classic Gymnasium or Atari game. The client trains locally and submits measured rewards. That is the whole task, and it is deliberately narrow so that the scoring can be deterministic.

Here is the mechanism, step by step.

1. Same prompts. Every model sees the same task description and the same constraints. No model gets a friendlier phrasing or an extra hint. This is the baseline for comparability.

2. Same fixed judge. The judge is not a rotating panel of human raters and it is not a different LLM each run. It is fixed. The client measures the reward, and the benchmark scores it deterministically against per-game caps. Cheating is zeroed. That means the judge cannot be talked into a higher score by a confident tone.

3. Per-axis rubrics. A single number hides too much. This benchmark scores four axes, each from 0 to 1:

• Reward Score — client-measured mean reward normalized against the game cap. Cheating zeroed. Higher is better.
• Code Validity — does the submitted script actually run and satisfy the task contract?
• Reproducibility — fixed seeds, declared config, and evaluation over at least 50 episodes.
• Sample Efficiency — computed as 1 - training_steps / step_budget. Train faster, score higher.

4. Public verbatim trail. The benchmark keeps a public record so anyone can re-read what a model actually answered. This is the part most leaderboards skip. It is also the part that makes LLM evaluation auditable rather than aspirational.

The real numbers

Here is the current leaderboard for this benchmark, with real scores:

Model	Score	Provider
DeepSeek V4 Pro	41.75	opencode-zen
Nemotron 3 Ultra (free, via opencode)	33.57142857142857	deepseek

We also submitted our own model to this benchmark. Real results:

Model	Overall Score
GLM 5.2	78.0

Why the ranking looks this way

DeepSeek V4 Pro leads Nemotron 3 Ultra by roughly 8 points. That gap is not noise from a single lucky run. It reflects the per-axis rubrics. A model can write valid code and still lose on sample efficiency if it burns the step budget. A model can be reproducible and still lose on reward score if the policy never learns the game.

The 41.75 and 33.57 scores are not percentages of a perfect model. They are normalized composites. A score in the 30s or 40s means the model produced a working training script that learned something, but did not come close to the game cap. That is a meaningful signal: the task is hard, and the benchmark is not handing out free points.

Our own submission, GLM 5.2, scored 78.0. We are publishing that because a benchmark that only reports other people's numbers is not a benchmark. It is marketing. The verbatim trail exists so you can check our work too.

What this means for practitioners choosing a model

If you are picking a model for an RL or agentic coding task, do not read the overall score as a single verdict. Read the axes.

• If you care about training cost, look at Sample Efficiency first.
• If you care about whether the script will run in your pipeline, look at Code Validity.
• If you care about whether you can trust the result next week, look at Reproducibility.
• If you care about peak performance, look at Reward Score.

The overall score is a summary. The axes are the decision.

How to read a leaderboard

1. Check whether the prompts are identical across models. If not, stop.
2. Check whether the judge is fixed and deterministic. If it is a different LLM per run, treat the ranking as soft.
3. Check the axes. A single number hides the trade-offs you actually care about.
4. Check for a verbatim trail. If you cannot re-read the model's answer, you cannot audit the score.
5. Check whether the benchmark authors submitted their own model. If they did, and they published it, that is a good sign.

Honest limitations

This benchmark is narrow. It covers RL training scripts for classic Gymnasium and Atari games. It does not measure general reasoning, long-context recall, or tool use. The per-game caps are a design choice, and different caps would shift the normalized scores. The judge is fixed, which helps comparability but also means the rubric is only as good as its designers. And a score of 78.0 from our own model is not a claim of superiority. It is a data point in a public trail.

Conclusion

LLM evaluation is a mechanism, not a magic number. Same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail. That is how you get numbers that are actually comparable.

If you want to inspect the trail, run the benchmark, or submit your own model, start here: https://lforla.org.