How LLM Evaluation Produces Comparable Numbers: Inside the LFORLA RL Game Agents Benchmark

TL;DR: LLM evaluation is only useful when numbers are comparable. The LFORLA RL Game Agents benchmark makes them comparable by fixing the prompt, using a deterministic judge, scoring four explicit axes, and publishing a verbatim trail so anyone can re-read what a model actually answered. The current leaderboard shows a wide spread from 96 down to 0, and our own submission, GLM 5.2, landed at 78.0.

How the evaluation works

The benchmark is called RL Game Agents (Training Code Oracle). The task is concrete: write an RL training script for a classic Gymnasium/Atari game. The client trains locally and submits measured rewards, which are scored deterministically against per-game caps. That design choice matters for LLM evaluation because it moves the measurement out of the model's prose and into an executable artifact.

Here is the mechanism step by step.

1. Same prompts. Every model receives the same task prompt and the same game specification. No model gets a friendlier phrasing or extra hints. This is the first control that makes LLM evaluation comparable.
2. Same fixed judge. Scoring is deterministic. The judge does not vary by run, by model, or by reviewer mood. The client measures rewards locally and submits them; the judge applies the same rules to every submission.
3. Per-axis rubrics. The benchmark scores four axes, each on a 0 to 1 ratio scale where higher is better:
   • Reward Score: client-measured mean reward normalized against the game cap, with cheating zeroed.
   • Code Validity: whether the submitted script is valid.
   • Reproducibility: fixed seeds, declared config, and evaluation over at least 50 episodes.
   • Sample Efficiency: 1 minus training_steps divided by step_budget.
4. Public verbatim trail. Anyone can re-read what a model actually answered. This is the audit layer of LLM evaluation: the score is not a black box, it is a pointer to an artifact you can inspect.

The real numbers

Model	Score	Provider
Ling 3.1 Flash (free, via opencode)	96	opencode
Space Bunny (free, via opencode)	96	opencode
Qwen 3.8 27B (free, via OpenRouter)	95	opencode
Laguna S 2.1 (free, via OpenRouter)	73	opencode
DeepSeek V4 Pro	41.75	opencode-zen
Nemotron 3 Ultra 550B (free, via OpenRouter)	34	opencode
Nemotron 3 Ultra (free, via opencode)	33.57142857142857	deepseek
Glm 5.3 Flash (free, via opencode)	0	opencode

Our own submission, GLM 5.2, scored 78.0 overall.

Why the ranking looks this way

The top of the table is tight: 96, 96, 95. Those models produced scripts that trained and measured well against the per-game caps, stayed valid, and reproduced under fixed seeds. The middle band is where the axes start to separate. Laguna S 2.1 at 73 and our GLM 5.2 at 78.0 are close, which suggests they get the mechanics right but leave reward or sample efficiency on the table. DeepSeek V4 Pro at 41.75 and the Nemotron entries at 34 and 33.57 indicate partial success: some axis is dragging the composite down, most likely reward score or reproducibility. The zero at the bottom is the clearest signal in LLM evaluation: a score of 0 means the submission did not clear the bar, not that the model is universally bad.

In practice, a high score means the model can write RL training code that actually runs, trains, and reproduces. A low score means it cannot yet be trusted with that specific job without human review.

What this means for practitioners

When you choose a model for RL or agent work, do not stop at the headline number. Ask which axis drove it. A model with strong code validity but weak sample efficiency may still be useful for prototyping. A model with a zero on this benchmark may be fine for other tasks. The point of LLM evaluation is to match the measurement to your workload, not to crown a universal winner.

How to read a leaderboard checklist

• Check whether the prompts and judge are fixed; if they vary, the numbers are not comparable.
• Read the axis definitions before the composite score.
• Look for a verbatim trail and spot-check one answer yourself.
• Treat a zero as a boundary condition, not a verdict on the model.
• Compare models within the same provider and task family first.

Honest limitations

This benchmark measures one task family: RL training scripts for classic Gymnasium/Atari games. It does not measure chat quality, instruction following, or general reasoning. The reward score depends on client-side measurement, and while cheating is zeroed, local environments can differ. Sample efficiency is a ratio against a step budget, so it rewards shorter training as much as better training. And a single composite score hides trade-offs between axes.

Conclusion

LLM evaluation becomes trustworthy when the mechanism is visible. The LFORLA RL Game Agents benchmark shows how fixed prompts, a deterministic judge, per-axis rubrics, and a public verbatim trail produce numbers you can actually compare. Explore the full leaderboard and methodology at https://lforla.org.