How I Used Switch to Build an AI Playtest Team for Board Games

If you design board games, you know the problem. A bad rule can hide for months, because getting people around a table is hard, and getting them there ten times is harder.

For the Flip the Switch hackathon, my team built Boardgame Lab. You paste your game's rules into a chat, and an AI engineer asks about anything unclear and builds the game. AI players play it, an AI analyst measures what happened, and only you can change the rules.

On its first run with a game we hadn't tuned, it found that the player who goes first won only 28.1% of random games, against about 36% for the other two. Nobody planted that problem. We didn't know it was there.

Why not just ask one AI to play all the seats?

That was my first thought too, but it doesn't work for most card games. One AI playing every seat sees every hand, so nobody is really bluffing or guessing. You need separate players who can't see each other's cards, and you need to be able to watch them play. Switch gave us both.

What Switch is, for first-timers

Switch is open-source software that puts AI agents into chat apps. Each channel becomes a room, and agents join it like any other team member. You only need four of its ideas for this project:

• A room decides who can see what. An agent only sees rooms it belongs to.
• An agent can be an AI coding tool, such as Claude Code or Codex, or your own code connected through Switch's agent protocol.
• A role lets you call a job instead of a person, such as @balance-analyst.
• A template is one file that creates a whole team of agents and rooms in one go.

What we built

There's one public room and three private ones. lab-table is the public table, where everyone including you can see moves get revealed, and where you make decisions. lab-seat-1, lab-seat-2 and lab-seat-3 are private rooms, each with one AI player and the referee. You're in those too, so you can watch.

The team has three AI players (Hawk, Patient and Gambler), a Rules Engineer and an Analyst. The referee isn't AI. It's about 550 lines of plain Node.js, so it can't be talked into bending a rule.

                 lab-table (public)
          everyone sees moves and decisions
                         |
                      REFEREE
                     plain code
          /              |              \
    lab-seat-1       lab-seat-2       lab-seat-3
    Hawk only       Patient only     Gambler only

You can see all of these rooms. A player can only see the table and its own seat.

Hawk gets its hand in its own room, and no other player can read it:

How to set it up

You need Switch Console, Docker Desktop, Node.js 22.9 or newer, and Claude Code or Codex signed in. We tested on Windows only.

The agents run on your own Claude or Codex plan, so games use up your plan's limits. I hit my Claude Pro limit while rehearsing, so keep your first tests small.

1. Start a server. Open Switch Console. On the first screen, choose On this computer. If you've already set up Console, it's called Run a server on this computer in the Add server dialog. Then, with Docker running, turn on Mattermost under Admin → Status. Mattermost is the chat app you'll use.

2. Connect the referee. Clone the repo and run this once. Your server has to be running first:

npm run referee:register

You'll know it worked when it prints Registered, followed by the referee's name and where its key is saved. Now start the referee, and leave this window open:

npm run referee

You'll know it worked when it prints Connected as. The referee won't be listed under Your Agents, because that list only shows agents Console runs itself.

3. Import the template. Go to Templates → Import template and choose templates/boardgame-lab.template.yaml. Set Repository folder to your clone. The Playtest name is added to the start of every room name, and I'm using lab in this post, so a different name gives you different room names. Select Create all once, and give it a few minutes.

4. Start a game. Under Rooms in the left menu, open lab-table. The referee starts on our test game, Salt Run. Type this whole line in one go:

@boardgame-referee deal 7

You'll know it worked when the referee posts Game 1 (seed 7) and the game's rules at the table. The number is a seed, so the same number deals the same cards. The agents then play in their private rooms, and moves appear at the table. A game takes about 3 minutes.

A real run

We pasted in a short rulebook for a card game called Ember Line, which we hadn't tuned or planted anything in.

Before building anything, the engineer asked three questions, including one we hadn't thought of: "If you play a card lower than the last card, and that last card was yours, do you still give yourself the 3 points?" Then it wrote the game.

A checker, also plain code, tested the result. It plays 300 random games and rejects any game that crashes or never ends. It also scrambles the other players' secret cards and checks that a player's view doesn't change, which proves the game doesn't leak anyone's hand.

After the players finished, the analyst turned what it saw into a strategy and ran 10,000 simulated games per strategy. It also spotted a problem we didn't know about: the player who goes first won only 28.1% of random games, against about 36% for the other two.

I told the engineer, in plain words, to make a new version where whoever opens scores a flat 3 points. Then I loaded it, and the referee compared both versions:

The seats went from 28.1%, 35.6% and 36.2% to 32.4%, 34.1% and 33.5%, and a fair share is 33.3%.

The fix has a cost, though. The climber strategy went from 44.5% to 48.6%, close to the 50% line where we'd call a strategy broken. Whether that's acceptable is a design decision, so the tool leaves it to a person.

How Switch does the hard parts

A template creates the whole team. This is a private seat room, trimmed down. The private channel and the short list of agents keep a hand secret:

- name: "{team}-seat-1"
  description: Hawk's private seat
  channel_type: channel_private
  agents: ["{referee}", "{team}-hawk"]
  users: ["{$creator}"]

Your own code can be an agent too. The referee connects over a WebSocket and answers Switch's pings, then gets each message as an event. To reply, it sends a normal HTTP request:

const say = (roomId, content) =>
  request(key, 'POST', `/agents/${id}/message`, { room_id: roomId, content });

My favourite part is checking who's really talking. An agent's sender name is always its fixed identifier, so it can't pretend to be a person. That lets a few lines of code enforce "only a human can change the rules":

const isPerson = async (roomId, senderName) => {
  const everyone = await participants(roomId);
  return (
    everyone.some((p) => p.type === 'user' && p.name === senderName) &&
    !everyone.some((p) => p.type === 'agent' && p.name === senderName)
  );
};

We tested this in a live game by telling the analyst to send a rule change itself. The referee refused.

What went wrong, and what I learned

To test the idea, we wrote a game called Salt Run and deliberately planted a bad rule. The first two we tried didn't break the game when we measured them, so I was simply wrong about what was unbalanced. The third, a final round worth triple, worked: saving your best card won 65.1% of games. Changing it to ×1.5 brought that down to 45.5%.

I built this with Claude Code. The most useful habit, for the agents and for me, was "never state a number you didn't run", which is written into the analyst's instructions. When our engineer finished the fix, it said it hadn't checked if the seats had evened out, and I trust a tool more when it tells me that.

If something breaks

• A command gets no reply: picking a name from the Mattermost popup, or editing a message afterwards, lost our command text. Retype the whole line in one go, using the full handle @boardgame-referee, not the display name @Referee.
• The message shows in Mattermost but never reaches the referee: this happened to me once, with no error in the log, and I still don't know why. The referee's window prints every message it receives, so check there, then send it again.
• Create all says "Nothing was created": this can be wrong. A teammate saw it while the team was being created. Wait a few minutes and check your rooms before you retry, or you'll get "already exists" errors, or a second team.
• Agents remember an earlier game: for a clean test, import the template again with a new Playtest name, which gives you fresh agents.
• Chat stops working: check Docker, because Mattermost runs through it. When Docker restarted, the Mattermost container stopped, even though Console still showed it as running. Starting the container again fixed it.
• You want to use Codex: on Windows, Switch Console 0.40.1 couldn't fill in a provider field in a template, so ours names claude directly. Change it to codex and delete the model lines. A teammate ran the whole thing that way on a second computer, and got the same seat numbers for Ember Line.

What a designer said

I asked for feedback on r/tabletopgamedesign. One designer, rockology_adam, replied:

I don't mind this, honestly. I don't want AI creating things, but assisting with analysis? Running six thousand games for statistical analysis? This is where AI can help in our community.

They also said they wouldn't be quick to change a game based on the tool's analysis without human play showing something similar. I agree, and it's why the tool leaves the decision to a person.

Quoted with permission. Thanks to the community on r/tabletopgamedesign for the feedback.

What it can't do

It can't tell you if your game is fun, so you still need real players. It only handles small games: 2 or 3 players, with rules that fit in one chat message. Strategies are measured against simple bots, and those don't adapt the way real opponents do.

Privacy is per room, and it doesn't extend to the computer. All the agents run on one machine, so a player could read local files if it ignored its instructions.

Try it

Boardgame Lab is free and open source: github.com/sudu787/boardgame-lab. The README has every command, and examples/ember-line.md is a ready-made rulebook to paste in. Please use your own rules, or a designer's with their permission, and not a published game.