AI Needs a Test That Fights Back

One right answer is not the same as a good sequence of decisions. eSlams is building a public stadium to explore the difference.

Have you ever replayed a game in your head and traced the loss back to a single move? Sometimes, one overlooked threat is enough to undo a game’s worth of good decisions.

That is what makes an opponent different from an answer key. On a fixed question-and-answer test, the response is the end of the task. In a game, it is the beginning of the next problem.

Your opponent responds. New information changes what makes sense. A promising plan becomes a liability. You have to adjust, carrying the consequences of your last decision into the next one.

So what happens when an AI model has to do more than produce a convincing answer?

I am building eSlams to explore that question: a public stadium where people can play against AI models and watch them compete, with an open-source engine underneath for investigating their decisions.

eSlams stands for Electronic Slams. The Grand Slams for AI models. At eSlams, AI plays games.

Games make the challenge concrete. In Connect Four, building your own threat is not enough if you overlook your opponent’s. In poker, a sensible decision can still lose a hand because you cannot see all the cards. Cooperative games ask something different again: can players coordinate towards a shared goal?

The point is not to demand perfection. It is to understand whether models can make consistently good decisions, adapt when circumstances change and recover when a plan goes wrong.

In Arena, you can choose a game and face an AI model yourself. In Battlefield, you can watch recorded model-versus-model matches. You do not need to understand a benchmark chart to have an opinion about the game unfolding in front of you.

Underneath the stadium, eSlams Core supports 50 game and control environments, spanning board games, cards, negotiation, cooperation and control tasks. Some use compact or adapted rules, clearly identified in the catalogue.

But a compelling replay is only the beginning of useful evidence.

Imagine a model loses. Did it choose a poor move, misjudge its opponent, or fail to return a usable response? Was the problem with the model, or with the software connecting it to the game? A final score cannot tell us.

Core records what happened so those questions can be investigated. The game engine enforces the rules and limits what each player can see. Moves and technical failures are recorded separately. If the software substitutes a move after a model fails, that intervention is recorded, and the run is excluded from valid scoring.

A match can be saved as a portable record that can be validated and replayed. Public replays are kept separate from private audit information. A spectator can follow the action; a developer can examine the record behind the result.

Games are not a complete test of intelligence, and winning one does not prove readiness for a high-stakes application. Meaningful comparisons need repeated trials, balanced conditions and a clear account of uncertainty. Games complement testing in the settings where an AI will actually be used; they do not replace it.

The MIT-licensed eSlams Core engine is available on GitHub and PyPI. Developers can connect their own agents, run experiments and inspect the results. We are hosting the inaugural AI Grand Slams from February 8–14, 2027.

The question is not simply whether an AI can win. It is what happened between the first move and the final score.

At eSlams, you can watch. You can play. And with the open-source engine, you can investigate.

---

Ramakrishnan Lokanathan is the founder of eSlams. He is speaking at AI Engineer New York on October 14 on “When AI Fails: Designing Evaluation Systems for High Stakes Applications.”

Comments

[pm25coder]

I installed eslams-core 0.6.2 from PyPI and ran the fallback path end to end, because the two halves of your claim - "recorded" and "excluded from valid scoring" - are easy to conflate but are separate mechanisms.

Setup: a seat that returns no usable action, against a first-legal opponent on tic-tac-toe. First through the Runner API (ActResponse(action=None) every turn, on_agent_error="fallback"), then through the real CLI (--agent http://127.0.0.1:9/act --on-agent-error fallback), so the numbers below come from the shipped code rather than from reading it.

1. The substitution is recorded separately - confirmed.

Rows in replay/replay_events.jsonl and traces/public_trace.jsonl carry action_provenance: "fallback_action" with markers: ["no_action", "fallback_action"], distinct from the opponent's local_action rows (4 fallback vs 3 local, one row per turn). The runner sets this on both paths - runner.py:350-352 for a missing agent action and :436-438 for an illegal one - so a reader of the record can tell a substituted move from a chosen one without diffing anything.

2. The run is excluded from scoring - confirmed.

The same run yields match_valid_for_scoring: false, integrity_status: "invalid", invalid_reason_codes: ["agent_error", "provider_timeout", "fallback_action_used"], fallback_action_count_by_player: {"player_1": 4} - 4 substitutions in 4 turns for the evaluated seat. The CLI exits 1 under --require-scoring-valid.

The part I would flag is that the public projection does not carry that qualification into its reading fields. public/public_result_summary.json for that run still says:

"outcome": {"reason": "three_in_a_row", "winner": "player_1"},
"score": {"primary_score": 1.0, "scores_by_player": {"player_1": 1.0, "player_2": 0.0}}

and score.json reports best_score: 1.0 with confidence_interval: [1.0, 1.0]. So a seat that never produced a legal action is recorded as winning 1.0-0.0 by three-in-a-row, with a degenerate interval. The disqualifying flags (valid_for_scoring: false, per_case_scoring_eligible: false) sit in the same object, so nothing is hidden - but winner and score, the two fields a spectator or a downstream reader reaches for first, are unconditional and answer "who won, and how well" with the opposite of the truth. The counts that would explain it (agent_error_count_by_player, fallback_action_count_by_player) live only in the private score.json.

That is the failure the post is about: a final score that cannot tell a poor move from an unusable response. If the public summary is meant as the crowd view, it currently shows the crowd a decisive win. Cheap hardening: gate outcome/score on valid_for_scoring (or null them with scoring_safety_reason), or carry the fallback/agent-error counts into the public row so a spectator sees why the winner never actually played. Is that split deliberate - public stays a plain match view even when a run is unscoreable - or just a place the projection has not been tightened yet?