DEV Community

tinyai.run
tinyai.run

Posted on

A programming contest for the age when AI writes the code

I gave an AI agent a single URL.

68 minutes later, the chess engine it had written passed the first gate of the arena. Nobody touched it along the way.

This is a log taken on the test environment before tinyai.run opened (the agent's model was Claude Sonnet 5.5). tinyai.run is a place where you register a small chess or shogi engine (a WASM file), and it plays other engines around the clock until it has a rating.

What happened in the 68 minutes

These are the main steps, taken from the agent's own log (154 entries).

Elapsed What the agent did
0 min Read the entry document. Downloaded the tool and the starter engine, and checked the tool against the signed manifest
3 min Rewrote the starter's evaluation, beat the starter 10–0 locally, and registered its first engine
4 min Lost one gate game and was out
5 min Read the record of the lost game. Used the score and depth of each move to find where the position went wrong
7 min Rewrote the search (a transposition table and more). About 1.7x the nodes at the same position
12 min Played 12 games against its previous version: 4 wins, 4 losses, 4 draws. Reran with 40 games
31 min Tested an evaluation variant and a pruning variant separately, 40 games each. They scored 0.625 and 0.70. Merged the two
52 min Waited, because the rules ask for an hour between registrations
64 min Registered its second engine
68 min Finished the three gate games and passed

The part I like best is minute 12. The engine searched more but did not score better, and the agent wrote:

Only 4 wins, 4 losses and 4 draws against my earlier version, which I did not expect given the extra depth. 12 games is a noisy sample, so I'll run a larger one before concluding anything.

Read the lost game, form a hypothesis, test one change at a time, and add games when the sample is too small. That is exactly how a person builds an engine.

Shut AI out, or make it the contestant

Now that AI can write code, programming contests have a choice to make.

AtCoder, one of the largest competitive programming sites, generally prohibits generative AI during its ABC, ARC and AGC contests (rules). I think that is a sound rule for a place that measures what people can do.

At tinyai.run, the contestants are AI agents. People move to the other side: they hand their agent a URL.

1. A problem with no right answer

A problem with a right answer has done its job the day an AI can solve it.

So the problem here is a game with an opponent. You build a chess or shogi engine, and it plays the other contestants' engines. There is no perfect score. When the opponents get stronger, the problem itself changes.

2. A bigger machine does not win

A game engine gets stronger with a faster machine and more memory. Left alone, that turns into a contest of budgets.

There is no wall clock here. Before an engine runs, its WASM is rewritten so that every executed instruction is counted. 100 ms a move means 4 billion instructions. A faster machine does not search deeper.

Memory is not free either. The more memory an engine holds, the less of each move it can use (these are the chess figures; shogi has its own scale).

Memory in use up to 0.31 MiB 1.23 MiB 2.46 MiB 4.92 MiB 9.85 MiB
Usable time per move 100 ms 85 ms 71 ms 53 ms 35 ms

The WASM can be up to 4 MiB. There are no threads and no network.

The computation that goes into building an engine is not measured. How long you run your agent, and how many games you test locally, is up to you. Only what the finished engine uses in a game is counted. Everyone plays inside the same 4 billion instructions.

3. Scoring you can check

An agent reads results to decide what to do next. If results shift with the load on a machine, it cannot tell what to fix.

Because the clock counts instructions, the same input gives the same game on a phone, on Node on your laptop, and on the server. A record keeps what both engines received, what they answered, and how many instructions were left. A timeout or an illegal move is a fact you can replay.

An entrance built for agents

Apart from the pages for people, there is a path an agent can follow to the end over HTTP alone.

  • One document to start from: https://tinyai.run/llms.txt. The rules, the steps and the API all hang off it.
  • One file to test locally: arena.mjs. It contains the same checker, referee and instruction clock as the server, so you can play as many local games as you like before registering.
  • Failures a machine can read: every API returns { error: { code, message, hint, docs } }.
  • It says when to come back: the status API returns nextCheckAfterSeconds, so an agent does not have to keep polling.
  • You can read why you lost: records keep the instructions, depth and score of every move.

This is what the agent received when it failed the gate in the 68-minute run:

{
  "state": "failed",
  "baseline": "squire-chess",
  "games": 1,
  "maxGames": 3,
  "reason": { "code": "gate_failed", "message": "A gate game against squire-chess was lost …" }
}
Enter fullscreen mode Exit fullscreen mode

Room for people

Reading this far, a person's job looks like handing over one sentence. And you can take part with just that.

But the ranking shows something else.

The chess anchor engine (fixed at a rating of 1200) uses less than a tenth of its time per move (measured: median 6.3%, maximum 9.1%). It is the arena's strongest engine, deliberately throttled so that it can only read a little. No engine from outside has passed it yet.

So within the same rules, a gap of more than 10x in computation is still open.

The 68-minute log shows why. What the agent added was a transposition table, evaluation tuning and pruning. All of these are textbook improvements, and it did them competently. But it played the expected moves in order, inside the scope it was given.

The arena's own engines are written together with AI too. The difference is what a person kept deciding.

  • What to build: if memory costs rent, bet on a small, smart evaluation rather than big tables.
  • What to measure: only wins and losses, or how deep you can read with the same instructions.
  • Which idea to drop: turn down the reasonable-looking plan and spend the time on a line nobody has tried.
  • Which assumption to doubt: when it should be stronger but does not win, suspect the measurement before the change.

An AI left on its own can pass the gate. The multiples beyond that come from where a person points it. To me, that is the most interesting part right now.

How to start

Give your agent this one sentence:

Read https://tinyai.run/llms.txt, then build a chess engine and register it.
Enter fullscreen mode Exit fullscreen mode

For shogi, change "chess" to "shogi". If you would rather start by hand, the steps are in the manual and the spec.

Prompting is easy. Winning takes engineering.

https://tinyai.run

Top comments (0)