If your game logic is a pure reducer, (state, action) => newState, then a bot is just a function from state to action. That means you can measure your rules by playing thousands of games, not just test that they do not crash.
I did this for Game Mode Sandbox, a front-end-only Next.js and TypeScript site that I build with two teammates. It runs one stock-drafting game under six rule sets, with fake companies and generated prices. I own the engine. This post walks through the "Mode Lab" I added this week: five bots, a seeded runner, a terminal command, and a browser page. Players see nothing new. It is a developer tool. (I built it with Cline.)
1. A bot is a function
The engine exposes createGameReducer(market), which returns (state, action) => newState. An action the rules do not allow returns the same state object, unchanged, and never throws. Given that, a bot only needs this:
// A bot sees what a player sees and returns one action. The runner calls it until the game ends.
export type Strategy = {
name: string;
description: string;
chooseAction(state: GameState, market: Market, rng: Rng): GameAction;
};
export const bigSpender: Strategy = {
name: "Big spender",
description: "Buys every round it can, always at the largest allowed amount. Never sells.",
chooseAction(state, _market, rng) {
return buyOrSkip(state, rng, largestAmount);
},
};
I wrote five, each one a probe for a specific rule:
- Random: sells now and then (25% chance when it can), skips 1 round in 5, and spends a random allowed amount.
- Big spender: always buys at the largest allowed amount. Never sells.
- Penny pincher: always buys at the smallest allowed amount. Never sells. Probes the minimum-spend rules.
- Loyalist: only buys from the industry of the oldest stock it owns. Probes the industry limits.
- Churner: sells everything the rules allow each round, then buys at the largest amount. Probes the selling rules.
2. The runner
playGame in lib/lab/run.ts starts a game with startGame(config, market, seed), then loops: ask the bot for an action, feed it to the reducer, count what happened. Here is a simplified version:
// Simplified from lib/lab/run.ts. Helper names are shortened.
const reduce = createGameReducer(market);
let state = startGame(config, market, seed);
while (!isOver(state)) {
if (!hasLegalAction(state)) { stats.stuck++; break; }
const action = bot.chooseAction(state, market, actionRng);
const next = reduce(state, action);
if (next === state) throw new Error(`${bot.name} made an illegal move`);
record(stats, state, next);
state = next;
}
Three choices are worth calling out.
Illegal moves fail loudly. The reducer returns the same object for an illegal move, so next === state means the bot broke the rules. Throwing there means a buggy bot cannot quietly skew the averages.
A stuck check. Before each move, the runner checks that the player has at least one legal action (buy, skip, or sell). An earlier version of the engine had a real stuck-game bug, so this is a regression guard.
Separate randomness. Actions use their own random stream derived from the game seed, so any single game can be replayed exactly.
Per game, the lab records buys, skips, rounds with nothing buyable, the share of cards blocked by rules, sales, auto-sells, idle cash, final value, and stuck games. The table shows per-game averages plus the 10th percentile, median, and 90th percentile of final value. From the terminal it runs with:
npm run lab -- --games=1000 --seed=2024
3. Fair comparisons: same seeds for every bot
Every bot plays the same seeds within a mode. Without that, one bot could simply draw better boards and you would be measuring luck. With it, differences between rows come from behavior.
A nice side effect is a built-in sanity check. In modes without selling, the Churner cannot sell, so its row should be identical to Big spender's. It is.
4. Watch out: count transitions, not steps
Before I opened the first PR, a review flagged that auto-sells were being double-counted.
Auto-sells happen as a new round starts, and the engine records them as events on that round's state. When the last round ends, the engine marks the game done but keeps that round's events on the final state. The lab counted auto-sell events after every move, so the last round's auto-sells got counted twice. Big spender in Active Trader showed 8.0 auto-sells per game instead of 7.0.
The check was arithmetic. Active Trader has 10 rounds and a 3-year hold, so stocks bought in rounds 1 to 7 reach their hold by round 10: 7 auto-sells.
The fix is to count only on transitions:
// Simplified: only count auto-sells when the round actually advances.
if (next.round !== state.round) {
stats.autoSells += countAutoSells(next);
}
I wrote the test first. It failed on the old code (got 4, expected 3 in Sector Focus), then passed with the fix. All 347 existing tests had passed with the bug present.
Rule of thumb: when state carries events forward, count them when the state changes phase, not on every step.
5. Running it in the browser without freezing
The /dev/lab page runs the same lab in a React client component. You pick games per bot and a seed, press "Run the lab", and the tables fill in one mode and bot pair at a time. Between pairs, it yields so the page can repaint.
My first version yielded with setTimeout(0). A full run took 13.0 seconds in Chrome, even though the terminal does the same work in about a second. The tell: 100 games per bot took as long as 1,000.
Chrome throttles chained timers in pages it treats as hidden. In my test, 20 chained setTimeout(0) calls took 11.5 seconds, and 20 MessageChannel messages took 0ms. So I yield the way React's scheduler does:
const nextTick = () => new Promise<void>((resolve) => {
const channel = new MessageChannel();
channel.port1.onmessage = () => {
channel.port1.close();
resolve();
};
channel.port2.postMessage(null);
});
The run dropped from 13.0s to 4.2s in a background window. A Web Worker would move the work off the main thread entirely. This was the smaller change, and it was enough.
6. Testing the page
The page and the terminal script share one list of column definitions, so they always print the same numbers. A test renders the page, runs it, and compares the cells to the runner's output for the same seed.
That test passed on my laptop and failed on CI after 1,267ms. Testing Library's waitFor gives up after 1 second by default, and the CI runner was just slower. I gave that waitFor a 10-second timeout and the test 20 seconds. A test that does real work has a time budget, so set it on purpose.
7. Was it useful?
Here is Sector Swap at 1,000 games per bot, seed 2024 (per-game averages):
| Bot | Buys | Sales | Idle cash | Median final value |
|---|---|---|---|---|
| Big spender | 8.0 | 0.0 | $0 | $11,571 |
| Penny pincher | 8.0 | 0.0 | $6,000 | $10,402 |
| Churner | 8.0 | 7.0 | $8,038 | $10,601 |
Two findings from the full run:
- In Sector Swap, selling every round caps you at one stock. A sale must be replaced from the same industry, and the replacement uses that round's pick. Two reasonable rules combine into a play style nobody designed.
- In Classic Draft, the Loyalist makes 1 buy in 8 rounds and leaves $8,750 idle, because the one-per-industry rule blocks the rest of its industry.
Neither is a bug. Both are things we would not have known from reading the rules. The suite is at 355 tests, and the whole lab is a few hundred lines on top of an engine that was already pure.
Top comments (0)