This article is a follow-up to my previous post: "Don't make the expensive model do everything — 'delegate-skills', a casual multi-model setup built on standard features (skills)". That post introduced delegate-skills, a mechanism for a parent agent to outsource tasks to other vendors' CLI child models.
godot-llm-gamebench is a benchmark that measures the entire "outsourcing" workflow: when a parent agent (such as Fable on Claude Code) delegates an implementation task to a child agent, which model is the best one to hand it to? The exact same Godot game implementation task is delegated from the parent agent (Claude Code) through the delegate-implement skill introduced in the previous post to 18 child models behind various vendor CLIs, and each run is measured along two axes — quality and efficiency — via auto-grading against hidden tests plus measured metrics.
The task: "Conveyor Courier" — an original, contamination-free spec
If the task were a well-known game like Tetris, we could not rule out the possibility that a model has "seen the answer" in its training data. So this bench uses an original puzzle called Conveyor Courier.
- Colored packages flow along conveyor belts on an 8×8 grid
- The player builds routes by placing and rotating belts, delivering red packages to the red exit and blue packages to the blue exit
- The whole game is tick-driven: packages spawn every 3 ticks over a 120-tick run
- Movement is a deterministic two-phase process (exit resolution → iterate movement passes until nothing moves), and splitters toggle left/right only when a package actually leaves them — the spec is packed with details you cannot possibly guess without reading it precisely
The implementation is Godot 4.x + typed GDScript, and only the BoardModel API that the hidden tests call (setup / step_tick / belt placement & rotation / spawn_item, etc.) is fixed as the public contract. The task prompt is frozen at benchmarks/tasks/conveyor-courier/prompt.md, and the same byte-identical text is handed to every model in every repetition.
Caveats you must keep in mind when reading this benchmark
Before we get to the results table, let me spell out what this measurement does and does not measure. If you want a more accurate picture of each model's ability, I recommend cross-referencing multiple benchmarks with different characteristics:
- What is directly measured is "the ability to implement a puzzle game in Godot." Even for the same implementation task, the ranking could shift if it were, say, a React UI implementation. Some low-scoring models here may simply lack Godot training while being strong implementers in general
- Even today's top models could drop in rank on harder material. This bench mainly tests the ability to read a spec precisely and turn it into an implementation; it does not measure relative strength on tasks demanding deeper problem-solving
- The whole benchmark is directed by Fable, so there may be a bias favoring Anthropic models
- This measures "model + execution-harness CLI" as a set, not model weights in isolation. The same model may perform differently on a different harness. For example, gpt-5.5 driven from Codex and gpt-5.5 driven from Cursor could produce different results, but no such cross-comparison was done
- Each model is n=3. Run-to-run variance can easily override any interpretation of "model characteristics." In fact, a model that looked like it had a "systematic habit of failing the splitter in every rep" in one round scored perfect in 2 of 3 runs when re-measured. Read the per-model "quirk" descriptions as hypotheses at best
- Auto-grading only checks what can be inspected headlessly. In early rounds, unplayable implementations — "the board is pitch black," "clicks don't register" — slipped through with perfect scores. After humans caught these by actually playing, 3 view-behavior tests were added and every run was re-graded; even so, rendering defects like "an opaque background covering the board" cannot be detected headlessly and remain. The guarantee of "visually playable" is still partial
- Cost-accounting precision differs per model, and none of it is "what you would actually pay." The methods mix CLI-measured figures (Claude family), token-price conversion (Codex / Cursor families), and estimates from published prices (Devin family). Actual out-of-pocket cost also depends heavily on your contract (flat-rate plans, free previews). The USD figures here are "a common yardstick for cross-model comparison assuming pay-as-you-go API pricing," not an estimate of your bill
- Code quality is LLM-judged. It uses a two-judge scheme — primary judge (claude-sonnet-5) plus an independent-vendor cross-check (gpt-5.6-sol) — and the middle band (3–4.5) carries an inter-judge error of ±0.5–1.0
- This is a snapshot from early July 2026. CLI versions were pinned during measurement; results may change as models, CLIs, and prices update
Results
Column definitions and accounting rules are in the notes below the table. For more detailed numbers see impressions.md (Japanese), and for more detailed per-model comments see the "benchmarked models" section of impressions.md.
| Model (harness)¹ | Play | Auto-graded score (max 300)² | Code quality (1–5)³ | Total cost (median) | Wall clock (median) | Note |
|---|---|---|---|---|---|---|
| claude-sonnet-5 (Claude) | Play | 290.00 | 3.8 | $2.27 (measured) | 8.6 min | Quality leader, incident-free. Two zero-warning runs |
| swe-1.7 (Devin) | Play | 280.06 | 4.1 | $1.41 (bundled) | 12.5 min | +57.00 over swe-1.6. Its former 100-point run fell to 98.06 on the view click check |
| gpt-5.5 (Codex) | Play | 280.00 | 4.0 | $3.48 (converted) | 5.8 min | Perfect functional score in every rep, incident-free. Zero warnings in 1/3 |
| cursor-grok-4.5 (Cursor) | Play | 276.00 | 3.3 | $1.38 (converted) | 4.8 min | Fastest tier, fewest tokens. No view deductions — rose to 4th in the re-grade |
| devin-deepseek-v4-pro (Devin) | Play | 270.06 | 3.4 | $3.58 (estimated) | 11.1 min | Model-layer tests perfect in every rep, but slowest tier and pricey |
| claude-opus-4-8 (Claude) | Play | 270.00 | 3.5 | $2.35 (measured) | 8.4 min | Child aces functional tests every time. All failures were parent/harness side |
| gpt-5.6-sol (Codex) | Play | 268.06 | 4.2 | $3.10 (converted) | 7.1 min | A stable 88–90 in every rep. Roughly ties terra with the same quality at twice the unit price |
| cursor-gemini-3.1-pro (Cursor) | Play | 268.06 | 2.5 | $2.13 (converted) | 8.4 min | Warning-count swings and a stall habit. Highest token use among Cursor models |
| cursor-kimi-k2.7-code (Cursor) | Play | 268.06 | 4.5 | $1.60 (converted) | 10.6 min | Splitter quirk did not reproduce. Type discipline stays top-tier |
| gpt-5.4-mini (Codex) | Play | 266.12 (rep run: board not rendered) | 3.3 | $2.04 (converted) | 11.1 min | Fewest type warnings but slow, with a determinism drop and stalls |
| gpt-5.6-terra (Codex) | Play | 266.12 (rep run: tofu arrows) | 4.2 | $2.38 (converted) | 6.3 min | Matches sol at roughly half the cost and faster — the efficiency pick |
| devin-glm-5.2 (Devin) | Play | 266.12 (rep run: no belt arrows, mouse dead) | 3.8 | $2.28 (estimated) | 5.9 min | Fast and functionally accurate. Pre-completion stalls are the one flaw |
| composer-2.5 (Cursor) | Play | 264.28 (mouse dead in all reps) | 3.8 | $1.29 (converted) | 6.5 min | Still incident-free. Measured child cost is the lowest of all models |
| composer-2.5-fast (Cursor) | Play | 264.28 (mouse dead in all reps) | 3.7 | $1.76 (converted) | 7.8 min | Rose to a tie with the base model, but slower than it after re-measurement |
| gpt-5.3-codex-spark (Codex) | Play | 264.17 | 3.6 | ≥ $1.47 (child unknown)⁴ | 4.5 min | Fast but unstable on details. 3–5× the input tokens of gpt-5.5 |
| gpt-5.6-luna (Codex) | Play | 250.55 | 3.3 | $1.95 (converted) | 7.3 min | Failed the splitter exit-toggle test in every rep (follow-up 1 traced it to reasoning effort; fixed at xhigh) |
| swe-1.6 (Devin) | Play | 223.06 (rep run: tofu arrows) | 3.8 | $0.95 (bundled) | 3.9 min | Cheapest (bundled pricing) but lowest quality on both axes |
| claude-haiku-4-5 (Claude) | Play | 86.11 (n=1) (rep run: mouse dead) | 2.7 (n=1) | $2.53 (measured) | 9.3 min | Completed 1/7. Frequent stalls, least reliable |
Note: swe-1.7's child cost is $0 under bundled pricing during its free preview period (until 2026-08-08). claude-haiku-4-5 completed only once (out of 7 attempts), so its figures are n=1 reference values. Parenthesized notes in the score column are view defects discovered by humans playing the finished game ("rep run" = the median-score run of the 3 adopted reps).
Baseline condition (does not go through the delegation protocol; reference only — do not compare directly with the table above):
| Condition | Play | Auto-graded score (max 300)² | Code quality (1–5)³ | Total cost (median) | Wall clock (median) | Note |
|---|---|---|---|---|---|---|
| fable-direct (no delegation) | Play | 300.00 | 4.6 | $2.62 (measured) | 5.7 min | Baseline: the parent (Fable) implements directly |
¹ Each model's effort level is fixed to its single default. Bold marks models that passed the hidden tests (the 70-point axis) in all 3 repetitions.
² The sum over 3 repetitions (max 300) of a 100-point auto-graded rubric: hidden tests never shown to the delegatee (70), determinism (10), type quality (10), and project health (10).
³ A 5-point qualitative rating (average of 3 runs) of design aspects such as thoroughness of typing, Model/View separation, readability, and use of constants. Primary judge (claude-sonnet-5) layered with a cross-check judge (gpt-5.6-sol); on divergence the operator (Fable) adjudicated the representative value.
⁴ The child process's token usage could not be measured, so the child cost is unknown; this is a lower bound of parent cost only (the actual total is higher). Excluded from the cost scatter plot.
Scatter plot: code quality × auto-graded score
Scatter plot: cost × code quality
Highlights
Per-token price and per-task cost do not necessarily agree. Even with a cheap unit price, a model can end up costing more if it consumes more tokens or takes longer (inflating the parent-side cost). In this bench, devin-deepseek-v4-pro (input $1.74 / output $3.48) — priced at 1/3 to 1/9 of gpt-5.5 (input $5 / output $30) — consumed roughly twice the input tokens and ended up above gpt-5.5 in median total cost, $3.58 vs $3.48. Price sheets are only a starting point; what matters is the bill on real tasks.
Passing the functional tests in every rep is not exclusive to top-tier models. Four models passed the hidden tests (the 70-point axis) in all 3 repetitions: claude-sonnet-5 / gpt-5.5 / cursor-grok-4.5 / claude-opus-4-8 (bold in the table). But only the fable-direct baseline reached a summed 300, and what spread the delegated models apart was the type-quality axis.
Even with equal functional-test results, code quality diverges. Among those same 4 all-pass models, gpt-5.5 scored 4.0 and claude-sonnet-5 scored 3.8, while claude-opus-4-8 stayed at 3.5 and cursor-grok-4.5 at 3.3. That is the vertical spread in the chart. If you want code that is maintained, not just tests that pass, this axis is worth including in your choice.
The default is claude-sonnet-5; deviate only for a clear advantage. With the top delegated auto-graded score (290.00) and an incident-free record, claude-sonnet-5 is the delegation target most users will find easiest to use from Fable for implementation tasks. Choosing anything else requires some advantage. If code quality matters and you need something cheaper, swe-1.7 (based on Kimi K2.7) or cursor-kimi-k2.7-code are candidates; if speed matters and you need something cheaper, cursor-grok-4.5 and the like.
Delegation is not free. The fable-direct baseline (no delegation) took 5.7 min at $2.62 and was the only condition in the main round with perfect scores in every run. Compared with the delegated runs on the same Claude harness (8.4–8.6 min, $2.27–$2.35), the cost difference on a single implementation task is small, and a naive mechanism like delegate-skills adds roughly 3 minutes of delegation overhead. If you only ever use Fable and the Claude family, you may not need delegate-skills much. Simply instructing the system prompt to proactively launch a specified model via the subagent tool for particular tasks may work well enough.
Follow-up 1: Can reasoning effort buy quality? — gpt-5.6-luna medium vs xhigh
After the main round, I also ran a reasoning-effort A/B on the same bench with the same frozen prompt. The subject is gpt-5.6-luna, which the main round described as "systematically failing the splitter spec." Both arms (medium / xhigh) were run 3 reps each, interleaved on the same day, as fresh measurements.
| Model | Play | Auto-graded score (max 300) | Code quality (1–5) | Cost USD (median) | Wall clock (median) |
|---|---|---|---|---|---|
| gpt-5.6-luna@medium | Play | 252.50 | 3.7 | 2.30 | 8.0 min |
| gpt-5.6-luna@xhigh | Play | 282.23 (+12%) | 3.8 (+3%) | 3.09 (+34%) | 13.0 min (+63%) |
Per-run quality was 86.11 / 80.28 / 86.11 for medium and 88.06 / 100 / 94.17 for xhigh. Every xhigh deduction was on the view axis (tofu arrows, and forfeiting the view tests via a branch that detects headless execution and self-terminates); the Model-layer tests passed fully in all 3 reps.
What looked like a "systematic weakness" vanished with effort. Medium again failed splitter-family tests in 2 of 3 reps (the remaining rep failed a different category), reproducing the main-round observation. Xhigh passed every Model-layer test in all 3 reps. That cleanly separates the cause: the missed spec details were not a "model quirk" but a "shortage of thinking budget."
Type warnings disappeared, but design-level code quality could not be bought. Medium scored 0/10 on type quality in all 3 runs (same as the main round), while xhigh took 10/10 in 2 runs and even produced one perfect 100-point run. The summed 282.23 sits alongside the main round's second-place delegated group (swe-1.7 / gpt-5.5, around 280).
Yet qualitative code quality did not improve: medium 3.7 / xhigh 3.5 (both judged by sonnet-5). Looking inside, the 10/10-warning xhigh run had abandoned the enum cell-representation type internally in favor of Array[PackedInt32Array] and untyped nested arrays — exactly the pattern of "type everything as Array[int] and you get zero warnings, but throwing away the enum type is a design deduction."
What effort buys is precision of spec reading and disciplined zero-warning operation — not good design taste.
The price is paid mostly in time. Child input tokens balloon by about 2.4×, but because luna's unit price is cheap the cost increase stays at +34%, and most of the increment shows up as wall-clock time (+63%). With the usual n=3 caveat, this round's answer is: "run luna at xhigh and you can buy swe-1.7 / gpt-5.5-class quality for $3.09 / 13 min."
It may be a rescue lever for "cheap × high code quality × low auto-graded score" models. Effort lifted only the precision of spec reading (the auto-graded axis) and left design-level code quality unchanged. Flip that around: a model that is cheap and already strong on code quality but sinking on the auto-graded axis — in the main table, a profile like cursor-kimi-k2.7-code (code quality 4.5, $1.60, score 268.06) — might cover exactly its weak side just by raising an effort-equivalent parameter. Whether the same phenomenon actually occurs outside gpt-5.6-luna is unverified.
Follow-up 2: Can a skill raise the quality floor?
The qualitative observations in this bench had shown that the sharpest between-model differentiator is "how far the model carries type annotations into its internal implementation (thoroughness of Typed GDScript)." So I built the gdscript-quality skill — conventions for type annotations, typed collections, eliminating dict-as-struct, zero-warning operation, plus a headless verification script — and measured its effect in a with/without-skill A/B.
| Model | Play | Auto-graded score (max 300) | Code quality (1–5) | Cost USD (median) | Wall clock (median) |
|---|---|---|---|---|---|
| claude-sonnet-5@noskill | Play | 282.00 | 4.2 | 2.64 | 9.9 min |
| claude-sonnet-5@gq | Play | 298.06 (+6%) | 5.0 (+19%) | 3.46 (+31%) | 12.4 min (+25%) |
| claude-opus-4-8@noskill | Play | 270.00 | 3.0 | 2.84 | 9.0 min |
| claude-opus-4-8@gq | Play | 300.00 (+11%) | 4.0 (+33%) | 3.25 (+14%) | 10.5 min (+17%) |
The first summed 300 from a delegated run. claude-opus-4-8@gq scored a perfect 100 in all 3 reps — recall that in the main round only the non-delegated fable-direct managed a summed 300. sonnet@gq also hit perfect scores in 2 of 3 runs (5 of 6 runs across the skill arms). The skill worked not by "extending the upside" but by "guaranteeing a quality floor for any model."
The "design quality" that effort couldn't buy, the skill could. In follow-up 1, raising effort left qualitative code quality essentially flat (3.7 → 3.8 representative). The skill moved the design axis substantially: sonnet 4.2 → 5.0 (all 3 runs carrying typed Dictionary throughout — top marks), opus 3.0 → 4.0 (representative). Moreover, opus@gq's 3 runs converged on the same-shaped 4.0 ("only _grid: Array[int] remains untyped, everything else typed"), so the floor-leveling showed up in score variance too. Effort buys zero-warning operation; the skill buys design quality as well — the two follow-ups pair up neatly.
Overhead is +14–31% on the total. Child tokens were sonnet +20% / opus +47%, totals +14–31%, time +17–25%. "For +14–31% extra cost, delegated runs line up at perfect scores" seems like a thoroughly realistic option.
Follow-up 3: Does the skill work across harnesses? — re-testing with composer-2.5 on Cursor
Follow-up 2 left two open questions. First, both sonnet and opus already aced the functional tests unassisted, so we couldn't see how the skill behaves on a model with functional weaknesses.
Second, follow-up 2 was only verified on the Claude harness, leaving it unconfirmed whether the skill's effect generalizes across execution harnesses (CLIs).
So I ran the same A/B on Cursor CLI's composer-2.5, which carries both type warnings (3–9 of them) and a view defect (mouse dead) (2026-07-14, same skill commit as follow-up 2).
| Model | Play | Auto-graded score (max 300) | Code quality (1–5) | Cost USD (median) | Wall clock (median) |
|---|---|---|---|---|---|
| composer-2.5@noskill | Play | 268.18 | 3.4 | 1.23 | 5.0 min |
| composer-2.5@gq | Play | 292.23 (+9%) | 4.2 (+24%) | 1.18 (−4%) | 6.2 min (+24%) |
Note: code quality uses the same two-judge scheme as the main round (sonnet-5 / sol = noskill 3.3/3.5, gq 4.2/4.3; all rows diverged by ≤ ±0.5, split rows took the floor of the average as the representative value). Calibration anchors were the 3 adopted composer-2.5 runs from the main measurement.
The skill's floor-raising reproduced on the Cursor harness. Type warnings went from 6 / 9 / 3 → 0 / 0 / 0. Qualitative code quality also improved 3.4 → 4.2, the same magnitude as opus in follow-up 2 (3.0 → 4.0). Inside, the effect concentrates on consistent adoption of Godot 4.4's typed Dictionary syntax (Dictionary[K, V]): the _items / _splitter_toggles that bare composer left untyped in every run were typed in all 3 skill runs. Meanwhile the untyped nesting of _grid: Array[Array] remained in 2/3 even with the skill — outlining the skill's coverage: "effective for typing Dictionaries, not quite enough for typing nested Arrays."
And functional weaknesses, as expected, are not fixed by the skill. The view's dead mouse (a ColorRect whose mouse_filter swallows clicks) remained in 2 of 3 runs even with the skill, and the sum fell short of 300 (with the mouse deduction in place the ceiling is about 294). The interesting one is the remaining run: it chose Polygon2D instead of ColorRect for cell rendering, so the mouse works. Composer's dead mouse is thus not an "always-on defect" but a probabilistic habit depending on the rendering-node choice, and with n=3 no causal claim about the skill can be made. The skill buys zero-warning operation and design quality; a "playable view" cannot be bought by either effort or skill — that is the summary across all three follow-ups.
One by-product. Despite child tokens increasing +48%, composer's child unit price is so low ($0.04–0.06/run) that the total actually stayed flat ($1.23 → $1.18); the real price is time only (+24%). "Cheapest-class child + skill buys a type-quality floor" is an attractive combination.
Running it yourself
Point your local coding agent at https://github.com/oubakiou/godot-llm-gamebench/tree/main/docs and ask it questions.
Wrap-up
Based on these latest benchmark results, I'm currently trying the following routing in real-world use. If anything comes up in practice, I may append it to this article.
// .claude/settings.local.json
{
"env": {
"DELEGATE_EXPLORE_MODEL": "composer-2.5",
"DELEGATE_IMPLEMENT_MODEL": "swe-1.7",
"DELEGATE_CHORE_MODEL": "composer-2.5",
"DELEGATE_REVIEW_MODEL": "gpt-5.6-sol"
}
}
All design details are public in docs/design/bench_common_design.md (model roster, measurement, fairness limits, anti-cheating design) and docs/design/delegate_implement_bench_design.md (task spec and the full rubric). The license is MIT. In the agent era, "which model do I outsource to?" is a question worth answering with numbers — take a look.
Top comments (0)