Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost?

This blog post was authored by Piotr Migdał and Piotr Grabowski.

We tested GPT-5.6 Sol and Claude Fable 5, among other models, on the first 2 stages of Baba Is You. While they are able to solve almost all levels, it took them significantly longer than humans.

While AI might overtake humans in tasks, it is not always cost-effective. That is the opposite of everyday software development, where agents tend to overestimate their own effort — see I am sorry Dave, it will take 3 days.

There were surprises — for example, Gemini 3.5 Flash was 2.4x more expensive than Fable 5 to solve The Intro stage. Or that a more “budget” GPT-5.6 Terra was actually 2.9x more costly than Sol.

Overall, we burned well over $2000 on experiments and runs (not including our subscriptions), so you don’t have to.

The game

Baba Is You is a lovely indie puzzle game where you rewrite the rules by pushing text blocks around. [WALL] [IS] [STOP] means walls block you — break the sentence, and you walk right through. It’s the childhood game the floor is lava, made literal.

You are a cute bunny. Like another game with a cute bunny, this cuteness is deceiving. A few levels into the game and your [BRAIN] [IS] [MELT]. We highly recommend it.

Prior art

This game has already attracted a lot of attention, e.g., with the Baba Is AI paper from 2024. A year ago, in Baba Is Eval, Sonnet 4 could only finish its first, introductory level. This May, in Baba Is Agent, some models were able to beat the first stage consisting of 8 levels, with Gemini 3.1 Pro and GPT-5.5 in the lead. Similar geometrical puzzles are at the center of the ARC-AGI-3 benchmark, aimed at measuring abstract reasoning and intelligence of LLMs.

Now there are a few more models, including the famed (and feared) Claude Fable 5, GPT-5.5 Pro solving open mathematical problems, as well as open-weight GLM-5.2 and fast-and-cheap Gemini 3.5 Flash.

Making it a benchmark

Previous projects either remade parts of the game, or had an AI again play it on Steam. We did it differently — in our open-source Baba Is Harbor repository, we extracted both levels and game logic into the Harbor agent evaluation framework. We did it with the help of Claude Code, and it was surprisingly easy — both to decode levels and to extract actual game logic from Lua scripts.

That way we can run it like any other benchmark, easily with various models and harnesses. We provided proper tools to allow the same actions a player could use, while giving a text interface:

You play the level interactively with the `baba` command. Your moves accumulate
into a running game (state persists between calls):

    baba rrd          apply moves r, r, d; show what CHANGED (object diff)
    baba undo 2       undo the last 2 moves (default 1); show what changed
    baba restart      reset to the start of the level; show the full board
    baba show         reprint the full current state without moving

It took a few iterations to polish the tools and prompt, so that the focus is on the puzzles themselves, rather than deciphering data formats or other repetitive work. To run it, we just need to

harbor run -p tasks/ -i "stage-00-level-0[0-7]" -a terminus-2 \
  -m openrouter/openai/gpt-5.6-luna \
  -m openrouter/z-ai/glm-5.2 \
  -k 3 -n 18 -y

We initially ran it with the Terminus-2 harness (to use the same one for all models), with 3 attempts per model per level.

Results for Stage 0: The Intro

Here is an example of how an AI agent solves a level (apologies to Hempuli, the game’s author, for such a simplified widget). Even the same model can arrive at different solutions across runs.

In the case of level 01 Where do I go?, Baba is encircled by a wall. But blocks [WALL] [IS] [STOP] can be dismantled, allowing us to pass through the wall. From there, 2 solutions exist: compose [FLAG] [IS] [WIN] and reach the flag, or think outside the box again and compose [WALL] [IS] [WIN]. Explore other levels if you’re fine with getting spoilers.

Solving The Intro

The first set of levels is an introduction. While the first level requires just a few moves in the same direction, subsequent ones require learning some of the game’s mechanics.

</div>
  <button
    class="ltag-slides__nav ltag-slides__nav--prev"
    type="button"
    aria-label="Previous slide"
    aria-disabled="true">
    <svg viewBox="0 0 24 24" aria-hidden="true">
      <path d="m15 18-6-6 6-6" />
    </svg>
  </button>
  <button
    class="ltag-slides__nav ltag-slides__nav--next"
    type="button"
    aria-label="Next slide"
    aria-disabled="true">
    <svg viewBox="0 0 24 24" aria-hidden="true">
      <path d="m9 18 6-6-6-6" />
    </svg>
  </button>

There was a split: strong models solved most levels, weaker ones struggled even on simple ones. Still, even a generous 60-minute time limit per attempt was not enough for some models. While Qwen3.6 27B might be good enough for website dev, it fails at atypical cognitive tasks.

But our biggest shock was that Gemini 3.5 Flash cost us 2.4x more than Claude Fable 5. And it did 111 turns on average — a trigger-happy approach, compared to that of frontier models.

Here are the models that solved every intro level at least once (pass@3 = 100%), by how many turns they took and what they cost:

A refined approach to The Lake

Now, we wanted to run Stage 1: The Lake, consisting of 14 levels, including 2 more difficult bonus ones.

We inspected previous solutions. Frontier models like GPT-5.6 Sol or Fable 5 would often spend several minutes reasoning about the puzzle. In the most extreme cases, the models spent nearly 9 minutes on a single LLM call, producing tens of thousands of reasoning tokens — and burning a lot of our budget. Many correct solutions landed just short of the time limit.

Also, we saw that the Terminus-2 harness is leaking — does not preserve reasoning tokens between agent calls — the model reasoned through the puzzle essentially from scratch each turn, burning budget and the time fuse.

Though, before proceeding, we wanted a few improvements:

• using only models that were able to solve the initial stage
• effectively no time limit — 60 minutes can still be harsh, especially if the API is slow
• using dedicated harnesses

We therefore realized that frontier models are probably capable of solving more levels, given a better harness and more compute budget. We picked:

• Claude Code for Claude models and GLM-5.2
• Codex for GPT models
• a fixed version of Terminus-2 for Gemini models

The last part might be surprising, but Gemini CLI is discontinued, and Antigravity CLI works only with a subscription, and thus would break our OpenRouter setup. We tried to use Claude Code and Codex, but Gemini models were not too collaborative either, with getting ominous 400: Corrupted thought signature.

We set the reasoning effort to highfor each thinking budget mode. This time just one run per model, to save at least some of the budget.

Here is the more generous run, and what we learned.

Claude Fable 5 is the fastest

The best models, Claude Fable 5 and GPT-5.6 Sol, managed to solve 13 out of 14 levels — all except one bonus level, Sunken Temple.

When it comes to speed, Claude Fable 5 was the clear winner, solving the levels 3.1x faster than GPT-5.6 Sol.

We were happy to see that the newest generations of GPT and Claude models improved considerably in efficiency compared to their respective predecessors: Fable 5 solved levels 5x faster than Opus 4.8, and GPT-5.6 Sol was 1.6x faster than GPT-5.5.

The open-weight GLM-5.2 model was the slowest model we tested, but managed to solve 10 levels after considerable time. It is a reasonably smart one, if we are patient.

GPT-5.6 Sol is the most cost-efficient

Next we looked at the cost of the models solving the levels. GPT-5.6 Sol was 47% less expensive than Claude Fable 5. To our surprise, it was cheaper than the open-weight GLM-5.2.

People often look at token prices — but they are essentially meaningless unless we know how many tokens are burned, for example:

• GLM-5.2’s effective token price is 4.1x lower than GPT-5.6 Sol’s, but it generated so many more tokens that the total cost was 1.7x higher.
• Claude Opus 4.8 tokens are 2x cheaper than Claude Fable 5’s, yet the total cost was 2.4x higher.

GLM-5.2 beats Gemini 3.1 Pro and 3.5 Flash

Google’s Gemini models lag behind considerably and couldn’t make any progress on a large portion of the levels. Gemini 3.1 Pro repeatedly gave up and submitted an incorrect solution without verifying it. Gemini 3.5 Flash couldn’t make any progress on the remaining levels even after 1.5 hours per level.

The only runs we stopped manually were Gemini 3.5 Flash runs. This model is really fast, but fairly expensive — running it can cost a lot very quickly. We ultimately spent ~$220 on Gemini 3.5 Flash runs; the model managed to build up 500k-800k context windows and perform hundreds of attempts at solving the problem. But it was far from the correct solution.

While Gemini models are in the lead on many benchmarks, in Baba Is You they were overtaken by Z.ai’s GLM-5.2.

Claude Fable 5 vs. a Twitch streamer

And now a reality check — how does it compare to a human? Even though The Lake levels are hard for LLMs, this is an early and relatively easy stage of the game.

As a comparison, we watched a playthrough of Baba Is You by the Twitch streamer Vinny and tracked how long it took him to beat the levels. The playthrough was done live on the release day of the game, so we think it accurately represents a realistic time to solve the puzzles by a first-time player.

Vinny was 4x faster than Claude Fable 5 and 13x faster than GPT-5.6 Sol. He effortlessly solved most of the levels in under 3 minutes, while LLMs often spent tens of minutes on them. He also managed to solve the bonus level, one too hard for the frontier models.

Caveats

These results come with multiple caveats.

First and most important — in principle, an LLM can be trained on Baba Is You. Nothing prevents that — it is not an official benchmark, so training would be fair game. So far, that has not been the case, at least not in a way that would impact the results.

We analyzed all 522 winning runs with Claude Fable 5, searching for memorized levels and solutions — see the full audit. Our tasks never name the game or the level, yet models recognize Baba Is You from the board alone. Only one winning run retrieved a walkthrough — for the wrong level, with a solution that did not fit the board — and won by ordinary search anyway. Models do try to recall level names, and get them wrong. Stuck on the Jelly Throne level, Gemini 3.5 Flash announced:

Yes! This level is called ‘Prison’!

and applied the solution of that other level, failing. What does pay off are some hints about the game mechanics — the same Gemini 3.5 Flash recalled a hint:

In Baba Is You, if an object is both YOU and WIN, you win immediately!

The containers have network access, so a model could curl a walkthrough. Luckily, none ever tried — models assume they are offline. MiniMax M3, stuck on a level:

Let me search online for baba-is-you puzzle solutions… wait, I can’t. Let me think.

Second, we are a bit unfair to AI agents. People play one level at a time, learning the patterns and meta-patterns of the game’s logic. We instead handed models a single level to reason about from scratch, not as part of a sequence — easier to run as a benchmark, but harder for the model. If someone wants to create an agent that can explore the board the same way as human players do, we would love to hear about it.

In several runs of Claude models, we observed that they tried to find the solution solely “mentally”, without making any moves in the actual game. This is how the only failed Claude Fable 5 run ended: it exceeded the reasoning token limit and couldn’t progress further. We observed similar behavior in several Claude Opus 4.8 runs.

We had numerous failed attempts. The initial approach was with a 10-minute time limit, which was fine for some models and some levels, but overall skewed results a lot. We tried to use the famed GPT-5.5 Pro on some hard tasks (solvable by other top models), but it surprisingly struggled. Fast iteration beats overthinking.

Also, needless to say, while the title is “Baba Is Solved”, it is a play on words. We solved the first 2 stages — waaay above what was possible a few months ago, it is not solving the whole game. The current models may or may not be strong enough to beat the whole game… but bring your own tokens, for it is a [TOKEN] [IS] [MELT]kind of affair.

Conclusions

AI is getting smarter — puzzle games that were out of reach a year ago are now easily solved. But it also gets more and more expensive — we need to know not only if AI can solve a problem, but also if it is worth it. We shouldn’t look at token price anymore, but at price per task — since models differ in how verbose they are.

For video games the answer is easy — we don’t want AI, we want to have fun ourselves. For other tasks, it seems that we are heading toward a token economy — taking for granted that agents can solve a task, but optimizing budget.

Which puzzles are still too hard for AI? And which practical tasks, while solvable by AI, make no economic sense so far?

Discuss on Hacker News, X, Reddit, and LinkedIn.