I Tested 6 Popular LLMs on the Same Chess Puzzle. Only One Solved It.

SEO-Zhuk — Tue, 14 Jul 2026 09:12:43 +0000

A few years ago, shortly after ChatGPT became publicly available, I asked it to solve a simple chess puzzle.

It failed.

Back then I wasn't particularly surprised. Large Language Models were still in their infancy, and everyone was discovering what they could—and couldn't—do.

Fast forward to today.

Modern AI models can write production-ready code, build websites, analyze datasets, generate SEO strategies, and even solve mathematical problems. As an SEO specialist, I use LLMs every day for technical audits, content creation, brainstorming, and automation.

That made me wonder:

Can modern LLMs actually solve a chess puzzle?

Instead of searching for benchmarks or research papers, I decided to run my own experiment.

Why chess?

Johann Wolfgang von Goethe once described chess as "the touchstone of the intellect."

I've loved chess since childhood.

Nobody in my family played chess, so I learned from an old self-study book I found in a small village library. Within a few months I was already beating classmates—and sometimes even elderly players in the local park.

Although I don't play very often anymore, chess taught me something I still use every day in SEO:

Think several moves ahead.

SEO isn't just about keywords or backlinks.

It's about planning.

Every decision affects the next one.

That's why I became curious whether modern language models could handle a problem that requires structured reasoning instead of simply generating convincing text.

The experiment

Every model received exactly the same prompt.

Black:
King — a8
Pawn — a7
Pawn — b7
Knight — a6

White:
King — c8
Knight — c4
Queen — e5

White to move.
Mate in 2.
Find the solution.

I deliberately described the position using board coordinates instead of an image.

The idea was simple:

If an LLM truly understands the position, it shouldn't need a chessboard.

The correct solution

The puzzle has only one correct first move.

1. Qa1!

After that there are only two meaningful replies.

If Black moves the knight:

2. Nb6#

If Black pushes the b-pawn:

2. Qh1#

Simple.

Or at least I thought so.

Results

Model	Result
Claude	❌ Failed
Microsoft Copilot	❌ Failed
DeepSeek	❌ Failed
Gemini	❌ Failed
ChatGPT	❌ Failed
Grok	✅ Solved

Only one model solved the puzzle correctly.

Claude

Claude spent several minutes reasoning before recommending Qc7.

The explanation sounded convincing.

The move immediately loses the queen.

Microsoft Copilot

Copilot chose Qb5.

Its analysis looked logical but completely ignored one of Black's defensive resources.

The forced mate simply wasn't there.

DeepSeek

DeepSeek produced something even stranger.

Instead of making strategic mistakes, it started generating illegal chess moves.

Pieces moved through other pieces.

Captures appeared from impossible squares.

It looked as though the model had completely lost track of the board.

Gemini

Gemini surprised me.

It actually found the correct first move:

Qa1

Unfortunately, it failed to calculate the second move.

It understood the tactical idea but couldn't finish the combination.

Among all unsuccessful models, Gemini probably came the closest.

ChatGPT

This wasn't my first chess test for ChatGPT.

I actually tried something similar back in 2022.

At that time it invented moves that didn't even exist.

Today's version is noticeably better.

Every move was legal.

The reasoning felt much more structured.

But it still overlooked one important defensive move available to Black.

Close.

But not correct.

Grok

Then came Grok.

Twenty-seven seconds later it returned the complete solution.

Correct first move.

Correct continuations.

No invented variations.

No illegal moves.

No unnecessary explanation.

Just the right answer.

It was the only model that solved the puzzle.

Why is chess still difficult for LLMs?

While writing this article I remembered Hikaru Nakamura playing blindfold chess.

He doesn't need to see the board.

Strong chess players maintain an internal representation of every piece and continuously update it after every move.

Language models appear to work differently.

They don't actually "visualize" a chessboard.

Instead, they predict the next most probable sequence of tokens.

For many tasks that's enough.

For chess...

Not always.

The experiment revealed several different failure modes:

some models generated illegal moves;
some ignored defensive resources;
some understood the strategy but failed in calculation;
some confidently explained completely incorrect lines.

That distinction is important.

Good language generation isn't always the same thing as good reasoning.

What surprised me the most

I expected some models to fail.

I didn't expect them to fail in completely different ways.

Some behaved almost like beginner chess players.

Others behaved like they had forgotten the rules entirely.

Only one model consistently tracked the position correctly.

Final thoughts

This wasn't intended to be a scientific benchmark.

It was simply an experiment driven by curiosity.

As someone working in SEO, I use AI every single day.

It's an incredible productivity tool.

But experiments like this remind me that AI should still be treated as an assistant—not an authority.

Confidence is easy.

Correctness is much harder.

If you're interested in the complete experiment—including screenshots of every model, detailed move-by-move analysis, and the original chess position—you can read the full Ukrainian article on my blog:

👉 https://seo-zhuk.com.ua/llm-i-shahovi-zadachi-doslidzhennya/

I'd also love to hear your thoughts.

Have you ever tested an LLM with chess or another structured reasoning task?

DEV Community: SEO-Zhuk