TL;DR
LLM evaluation is only as good as its mechanism. The RL Game Agents benchmark on LFORLA shows how a fixed prompt, a deterministic judge, per-axis rubrics, and a public verbatim trail turn messy model outputs into comparable scores. Here is the full breakdown, plus the real leaderboard numbers.
How the evaluation works
Most leaderboards hand you a number. Few explain where it came from. The RL Game Agents benchmark is different because its mechanism is public and repeatable.
Step 1: The prompt is fixed. Every model receives the same task: write an RL training script for a classic Gymnasium or Atari game. No variation, no hidden retries. The prompt is the constant.
Step 2: The client trains locally. The model does not just produce code that looks right. The client runs the script, trains the agent, and submits the measured reward. This is the key move that separates this benchmark from static code generation tests.
Step 3: Scoring is deterministic against per-game caps. The measured reward is compared to a cap defined per game. That means the same script always yields the same score. No human judgment, no vibes.
Step 4: Per-axis rubrics. The benchmark scores across coding, reinforcement learning, and agents. Each axis has its own rubric, so a model that writes clean code but fails to train gets a different profile than one that trains well but writes brittle code.
Step 5: The verbatim trail is public. Anyone can re-read what a model actually answered. This is the accountability layer. If a score looks surprising, you can inspect the exact script and the exact reward it produced.
The real numbers
Here is the current leaderboard for RL Game Agents.
| Model | Score | Provider |
|---|---|---|
| DeepSeek V4 Pro | 41.75 | opencode-zen |
| Nemotron 3 Ultra (free, via opencode) | 33.57142857142857 | deepseek |
We also submitted our own model to this benchmark. GLM 5.2 scored 78.0 overall.
Why the ranking looks this way
A score of 41.75 versus 33.57 is not a rounding error. In this benchmark, the gap reflects how reliably each model turns a prompt into a training script that actually learns. DeepSeek V4 Pro leads because its scripts hit higher measured rewards against the per-game caps. Nemotron 3 Ultra trails, but the free access via opencode makes it a useful baseline.
Our own GLM 5.2 submission at 78.0 sits well above both. That is not a claim of universal superiority. It is a claim about this specific task: writing RL training code that trains locally and submits measured rewards. The per-axis rubrics matter here. A model can be strong at coding and weak at reinforcement learning, or strong at agents and weak at coding. The overall score compresses that, so the verbatim trail is where you go to understand the shape of the result.
What this means for practitioners choosing a model
If you are picking a model for RL or agent work, do not stop at the headline score. Ask three questions. First, what was the prompt? Second, how was the judge fixed? Third, can I read the actual output? The RL Game Agents benchmark answers all three. That makes it more useful than a leaderboard that only publishes a rank.
The practical takeaway: a model that scores 41.75 on a deterministic, reward-measured task is a safer bet for training-script generation than a model that scores higher on a static code benchmark. The mechanism is the signal.
How to read a leaderboard
- Check whether the judge is fixed or human. Fixed judges are reproducible; human judges drift.
- Look for per-axis rubrics. A single number hides whether a model failed at coding or at training.
- Demand a verbatim trail. If you cannot read the actual answer, you cannot trust the score.
- Compare scores only within the same benchmark. Cross-benchmark comparisons are apples to oranges.
- Note the provider and access path. Free via opencode is not the same as paid API access.
Honest limitations
This benchmark measures one thing well: writing an RL training script for classic Gymnasium and Atari games. It does not measure instruction following, long-context reasoning, or multi-turn agent behavior. The per-game caps are fixed, so a model that generalizes across games may not be rewarded for that generality. The leaderboard is also small. Two models plus our own submission is a snapshot, not a census. And our own submission is our own model, so treat that number with the appropriate skepticism.
Conclusion
LLM evaluation is a mechanism, not a magic number. The RL Game Agents benchmark shows what that mechanism looks like when it is done in the open: same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail. If you want to inspect the full leaderboard or submit your own model, start at https://lforla.org.
Top comments (0)