How I Used Switch to Build an AI Playtest Team for Board Games

SUDARSHAN — Thu, 05 Nov 2026 13:33:37 +0000

If you design board games, you know the problem. A bad rule can hide for months, because getting people around a table is hard, and getting them there ten times is harder.

For the Flip the Switch hackathon, my team built Boardgame Lab. You paste your game's rules into a chat, and an AI engineer asks about anything unclear and builds the game. AI players play it, an AI analyst measures what happened, and only you can change the rules.

Why not just ask one AI to play all the seats?

That was my first thought too, but it doesn't work for most card games. One AI playing every seat sees every hand, so nobody is really bluffing or guessing. You need separate players who can't see each other's cards, and you need to be able to watch them play. Switch gave us both.

What Switch is, for first-timers

Switch is open-source software that puts AI agents into chat apps. Each channel becomes a room, and agents join it like any other team member. You only need four of its ideas for this project:

A room decides who can see what. An agent only sees rooms it belongs to.
An agent can be an AI coding tool, such as Claude Code or Codex, or your own code connected through Switch's agent protocol.
A role lets you call a job instead of a person, such as @balance-analyst.
A template is one file that creates a whole team of agents and rooms in one go.

What we built

There's one public room and three private ones. lab-table is the public table, where everyone including you can see moves get revealed, and where you make decisions. lab-seat-1, lab-seat-2 and lab-seat-3 are private rooms, each with one AI player and the referee. You're in those too, so you can watch.

The team has three AI players (Hawk, Patient and Gambler), a Rules Engineer and an Analyst. The referee isn't AI. It's about 550 lines of plain Node.js, so it can't be talked into bending a rule.

Hawk gets its hand in its own room, and no other player can read it:

How to set it up

You need Switch Console, Docker Desktop, Node.js 22.9 or newer, and Claude Code or Codex signed in. We tested on Windows only.

First, start a server. In Switch Console, choose Run a server on this computer. Then, with Docker running, turn on Mattermost under Admin → Status. Mattermost is the chat app you'll use.

Next, connect the referee. Clone the repo and run these two commands, leaving the second window open:

npm run referee:register

npm run referee

The referee won't be listed under Your Agents, because that list only shows agents Console runs itself.

Then import the template. Go to Templates → Import template, choose templates/boardgame-lab.template.yaml, and set Repository folder to your clone. Select Create all once, and give it a few minutes.

To start a game, open lab-table under Your Rooms and type this:

@boardgame-referee deal 7

The referee deals, the agents play in their private rooms, and moves appear at the table. A game takes about 3 minutes.

A real run

We pasted in a short rulebook for a card game called Ember Line, which we hadn't tuned or planted anything in.

Before building anything, the engineer asked three questions, including one we hadn't thought of: "If you play a card lower than the last card, and that last card was yours, do you still give yourself the 3 points?" Then it wrote the game.

A checker, also plain code, tested the result. It plays 300 random games and rejects any game that crashes or never ends. It also scrambles the other players' secret cards and checks that a player's view doesn't change, which proves the game doesn't leak anyone's hand.

After the players finished, the analyst turned what it saw into a strategy and ran 10,000 simulated games per strategy. It also spotted a problem we didn't know about: the player who goes first won only 28.1% of random games, against about 36% for the other two.

I told the engineer, in plain words, to make a new version where whoever opens scores a flat 3 points. Then I loaded it, and the referee compared both versions:

The seats went from 28.1%, 35.6% and 36.2% to 32.4%, 34.1% and 33.5%, and a fair share is 33.3%.

The fix has a cost, though. The climber strategy went from 44.5% to 48.6%, close to the 50% line where we'd call a strategy broken. Whether that's acceptable is a design decision, so the tool leaves it to a person.

How Switch does the hard parts

A template creates the whole team. This is a private seat room, trimmed down. The private channel and the short list of agents keep a hand secret:

- name: "{team}-seat-1"
  description: Hawk's private seat
  channel_type: channel_private
  agents: ["{referee}", "{team}-hawk"]
  users: ["{$creator}"]

Your own code can be an agent too. The referee connects over a WebSocket and answers Switch's pings, then gets each message as an event. To reply, it sends a normal HTTP request:

const say = (roomId, content) =>
  request(key, 'POST', `/agents/${id}/message`, { room_id: roomId, content });

My favourite part is checking who's really talking. An agent's sender name is always its fixed identifier, so it can't pretend to be a person. That lets a few lines of code enforce "only a human can change the rules":

const isPerson = async (roomId, senderName) => {
  const everyone = await participants(roomId);
  return (
    everyone.some((p) => p.type === 'user' && p.name === senderName) &&
    !everyone.some((p) => p.type === 'agent' && p.name === senderName)
  );
};

We tested this in a live game by telling the analyst to send a rule change itself. The referee refused.

What went wrong, and what I learned

To test the idea, we wrote a game called Salt Run and deliberately planted a bad rule. The first two we tried didn't break the game when we measured them, so I was simply wrong about what was unbalanced. The third, a final round worth triple, worked: saving your best card won 65.1% of games. Changing it to ×1.5 brought that down to 45.5%.

Type each command in one go. In Mattermost, picking a name from the popup, or editing a message afterwards, lost our command text. Use the full handle as well: @boardgame-referee, not the display name @Referee.

Some messages get lost. One of mine appeared in Mattermost but never reached Switch, and the log showed no error. Sending it again worked. I still don't know the cause, so if an agent doesn't answer, check that first.

Create all can report "Nothing was created" when it's wrong. A teammate saw this even though the team was being created. Wait a few minutes and check Your Rooms before you retry, or you'll get "already exists" errors, or a second team.

Agents remember earlier games. For a clean test, import the template again with a new Playtest name, which gives you fresh agents.

If chat stops working, check Docker, because Mattermost runs through it. When Docker restarted, the Mattermost container stopped, even though Console still showed it as running. Starting the container again fixed it.

On Windows, Switch Console 0.40.1 couldn't fill in a provider field in a template, so ours names claude directly. To use Codex, change it to codex and delete the model lines. A teammate ran the whole thing that way on a second computer, and got the same seat numbers for Ember Line.

I built this with Claude Code. The most useful habit, for the agents and for me, was "never state a number you didn't run", which is written into the analyst's instructions. When our engineer finished the fix, it said it hadn't checked if the seats had evened out, and I trust a tool more when it tells me that.

What it can't do

It can't tell you if your game is fun, so you still need real players. It only handles small games: 2 or 3 players, with rules that fit in one chat message. Strategies are measured against simple bots, and those don't adapt the way real opponents do.

Privacy is per room, and it doesn't extend to the computer. All the agents run on one machine, so a player could read local files if it ignored its instructions.

Try it

Boardgame Lab is free and open source: github.com/sudu787/boardgame-lab. The README has every command, and examples/ember-line.md is a ready-made rulebook to paste in. Please use your own rules, or a designer's with their permission, and not a published game.

In Switch, a room also decides who knows what, and that's what made a fair game of cards possible with AI players.

DEV Community: SUDARSHAN