DEV Community

Cover image for I load-tested HQ Trivia 2.0 with 1,000,000 bots
Benyamin Radmard
Benyamin Radmard

Posted on Edited on

I load-tested HQ Trivia 2.0 with 1,000,000 bots

The code in this post is simplified to show the patterns. It isn't Savvy's production source.

TL;DR

  • 1,000,000 concurrent bots, every hub method under 200 ms. That was the goal, and the final run hit it.
  • A custom bot fleet: SignalR bots that play the real game, 200 per container instance, 5,000 Azure Container Instances for the full million.
  • A realistic join curve: 25% of the bots join in the same moment, the way players do when the "show is live" push notification lands.
  • A Redis remote control: runners wait for a start command, so a fleet that takes 30+ minutes to provision still starts in sync.
  • Word submission went from a 10 s average to under 200 ms: HybridCache for show config, player state in memory, and Redis writes moved to a background queue.
  • Smaller messages: trimmed payloads plus MessagePack kept broadcasts inside SignalR's 2 KB message unit and under the per-unit bandwidth ceiling.
  • A calm end of show: one post-game worker and Cosmos DB transactional batches per partition key, instead of every server writing at once.
  • Load tests in CI: a GitHub Action runs 10,000 bots against any branch, so the team sees what a change costs before it ships.

What Savvy is

Think HQ Trivia, but with Wordle, hosted by Scott Rogowsky, the original host of HQ. Savvy is a live game show where thousands of players race the host to solve the same word, at the same moment, for cash prizes.

Every round works like Wordle, and a time-based scoring system ranks every player. The idea was simple. Running it was not: everyone sits in one room, every player needs the same state at the same moment, and we wanted the backend to hold a million players if the show took off.

Our real peak in a live show was 6,000 concurrent players. This post is about how I made sure 1,000,000 would work too.

The game

A show has several rounds. In each round, players get six guesses to find the secret word. They can use powers to improve their odds, for example a letter bomb that removes letters from the keyboard, or a hint. While they play, they can see where the host is on the board and how many points the host has. Between rounds we show stats such as who was fastest.

Watch a full show: the season finale, April 2, 2026.

Requirements

  1. Every player receives broadcasts in real time, and all clients stay in sync.
  2. Every request completes in under 200 ms.
  3. The backend scales horizontally to millions of concurrent players.

The stack

  • Client: Unity, shipped as iOS and Android apps
  • Backend: .NET on Azure App Service
  • Real-time: ASP.NET Core SignalR, with Azure SignalR Service holding the client connections
  • State: Redis, so every backend instance stays stateless
  • Database: Cosmos DB

Architecture: Unity app, Azure SignalR Service, App Service instances, Redis and Cosmos DB

Clients only ever talk to SignalR Service. The App Service instances hold no state of their own, so we can add more of them.

Test 1: 1,000 bots on my laptop

The first goal was 1,000 concurrent players who join a show, submit guesses, chat, use powers and leave at the end.

I looked at off-the-shelf load-testing tools but couldn't find one that worked well with SignalR and also scaled. Most of them speak plain HTTP or WebSockets, not SignalR's hub protocol. And a bot here has to play a game with state, reacting to what the server broadcasts, not replay recorded requests. So I wrote a small bot coordinator. It manages a swarm of bots and has them join a given show the way real players do. I packaged it as a Docker image and ran it in Docker Desktop, with the plan to move it to Azure Container Instances once I needed to scale out.

A bot is just a SignalR client that plays like a person:

// One bot: join, then guess like a human every round
connection.On<RoundStarted>("RoundStarted", async _ =>
{
    for (var guess = 0; guess < 6; guess++)
    {
        await Task.Delay(Random.Shared.Next(2_000, 8_000)); // think
        var result = await stats.Time("SubmitAnswer",
            () => connection.InvokeAsync<GuessResult>("SubmitAnswer", Words.Random()));
        if (result.Solved) break;
    }
});

await connection.StartAsync();
await stats.Time("Join", () => connection.InvokeAsync("Join", showId));
Enter fullscreen mode Exit fullscreen mode

The join curve

How bots join matters as much as how many join. Players don't drift into a Savvy show one by one. When a show goes live, everyone gets a push notification, and a big group taps it in the same moment. In production, about a quarter of the audience joined at once, and the rest came in over about ten minutes.

So the coordinator doesn't ramp at a steady pace. It follows a join curve over ten minutes. The first minute is a normal trickle. In the second minute the notification lands and 25% of the bots join at the same moment. The rest arrive over the remaining eight minutes, slower and slower. That was close enough to what we saw in real shows, and it's the burst, not the average, that breaks things.

Join curve: 5% of bots join in minute one, 25% in minute two, then a tail over ten minutes

The 25% spike in minute two and the ten-minute window are the real figures. The exact shape of the tail is illustrative.

Before the first run I added reporting. Every run printed the error rate, the number of messages sent, the peak number of connected bots, and the p50, p75, p90, p95, p99 and max latency for each hub method. A report looked like this:

The format of every report. The numbers here are illustrative.

=== Load test report ===
Show:            loadtest-0412
Duration:        14m 32s
Bots requested:  1,000
Max connected:   1,000
Messages sent:   61,240
Messages recv:   1,284,903
Errors:          3 (0.005%)

Hub method       calls    p50   p75   p90   p95   p99   max  (ms)
Join             1,000     38    52    71    88   143   311
SubmitAnswer    23,914     41    55    78    96   162   408
UseLetterBomb    2,107     36    49    66    81   128   264
UseHint          1,862     34    47    63    79   121   237
SendChat        31,357     22    29    41    53    94   199
Leave            1,000     18    24    33    41    67   112
Enter fullscreen mode Exit fullscreen mode

That report became the scoreboard for everything that followed.

What I'd do again

  • Write bots that play the real game and react to broadcasts, not scripts that replay requests.
  • Report p95 and p99 per hub method from day one. Averages hide the players who are waiting.
  • Model the join curve on how real users arrive. A push notification means a burst, not a steady ramp.

Test 2: moving to Azure, and a remote control for 50,000 bots

My laptop topped out at about 3,000 bots, so the bots had to move to the cloud.

I tried Azure Container Apps first. For reasons we never fully pinned down, it couldn't open enough ports, and we never got past about 20,000 bots per run. After a couple of weeks of back and forth with Azure support, we switched to Azure Container Instances (ACI).

The setup was simple. The bot runner image lived in Azure Container Registry, and a Terraform script spun up as many runners as I needed. After some trial and error I settled on 1 vCPU and 2 GB of RAM per instance, which handled 200 bots comfortably. One million bots would mean 5,000 container instances. I started with a target of 50,000: 250 instances spread across several regions.

Bot fleet: a controller sends start, stop, status and exit over Redis pub/sub to 250 runners with 200 bots each

Then I waited. And waited. Provisioning took about 30 minutes, and that exposed a design problem. ACI starts the container as soon as the instance is ready, and there was no way to tell Azure to hold everything until the last one was up. Some runners started at minute 0 and others at minute 30, when the show was already over. I needed every bot to join and play at the same time.

Timeline: without a start signal, runners start whenever Azure provisions them

The fix was a remote control. I provisioned a separate Redis instance and used its pub/sub as a command channel. Every runner starts as soon as it's online, but it doesn't join the show until it receives a command. The controller was a small script on my machine with four commands: start, stop, status and exit.

Sequence: Terraform provisions runners, runners wait, the controller sends start, bots join the game

Inside every runner, the control loop is a few lines of StackExchange.Redis:

// Runners wait for orders instead of starting right away
await sub.SubscribeAsync(RedisChannel.Literal("loadtest:control"), (_, msg) =>
{
    var cmd = Command.Parse(msg!); // "start show-123 200"
    switch (cmd.Name)
    {
        case "start":  run = Coordinator.Start(cmd.ShowId, cmd.BotCount); break;
        case "stop":   run?.Stop(); break; // runner stays up for the next run
        case "status": sub.Publish(StatusChannel, Stats.Summary()); break;
        case "exit":   Environment.Exit(0); break;
    }
});
Enter fullscreen mode Exit fullscreen mode

And the controller can be as simple as redis-cli:

redis-cli -u "$CONTROL_REDIS" PUBLISH loadtest:control "start show-123 200"
Enter fullscreen mode Exit fullscreen mode

To my surprise, it worked. It also kept the join curve intact at scale. Each runner applies the curve to its own 200 bots, and because all 250 runners get start at the same moment, their bursts add up to one fleet-wide burst, just like a real notification.

The big bonus was that I could reuse the same instances run after run, with no 30-minute wait in between.

Now I could scale out and coordinate the bots. But the latency was bad. Really bad. On average, a bot waited 10 seconds for a word submission to come back.

What I'd do again

  • Separate "the machine is up" from "the test starts". Runners wait for a signal.
  • Reuse runners between runs. Provisioning is the slowest part of the whole loop.
  • Use a separate Redis for control, so the control channel never shows up in the results.

Test 3: in-memory caching and background writes

To fix the latency I went through the hub methods one at a time. The worst were SubmitAnswer, Join and UseLetterBomb.

First I built an Azure dashboard with CPU and memory for every resource side by side: App Service, SignalR Service, Redis and Cosmos DB. I also added the service-specific metrics that turned out to matter most: bandwidth and concurrent connections for SignalR Service, and cache hit rate, reads and writes for Redis.

The dashboard showed Redis CPU spiking during the join burst and in every round. That made sense. 50,000 players were joining, submitting words and using powers, and each of those hub methods read from and wrote to Redis. The question was whether I could remove those calls or make them cheaper.

I started with word submission. It looks simple, but it wasn't scaling. For every guess it had to:

  • check that the round hasn't ended
  • check that the player's last submission was at least a second ago (basic bot detection)
  • fetch the player's earlier submissions for this round from Redis
  • fetch the show's scoring configuration from Redis
  • calculate the points
  • append the new submission to the player's list in Redis
  • return the points and which letters are correct, in the wrong place, or not in the word

Two reads stood out: the configuration and the player's submissions.

The show configuration doesn't change during a live show, so fetching it from Redis for every single guess made no sense. I put it behind a hybrid cache, with an in-memory L1 on each server and Redis as L2. The first call loads it from Cosmos DB into Redis and into the server's memory, and every call after that reads it from memory.

I did the same for submissions. Each server keeps its players' submissions in memory and writes points and submissions to Redis through a background queue, so the response goes back right away. This works because a player stays connected to the same server instance for the whole show. If the connection drops and the player lands on another instance, that instance loads their state from Redis and they carry on where they left off.

Before and after: SubmitAnswer reading from Redis versus reading from memory

Here's the hub method before:

// Before: three Redis round trips on every guess
var config      = await _redis.GetAsync<ShowConfig>(configKey);
var submissions = await _redis.GetListAsync<Submission>(playerKey);
var result      = Scoring.Score(word, submissions, config);
await _redis.AppendAsync(playerKey, result.Submission);
return result;
Enter fullscreen mode Exit fullscreen mode

And after. .NET's HybridCache does the L1/L2 work: register Redis as the distributed cache, and it keeps a copy in memory on top.

builder.Services.AddStackExchangeRedisCache(o => o.Configuration = redis); // L2
builder.Services.AddHybridCache();                                         // L1 in memory
Enter fullscreen mode Exit fullscreen mode
// After: memory first, Redis in the background
var config = await _cache.GetOrCreateAsync(configKey, ct => LoadConfig(showId, ct)); // memory → Redis → Cosmos
var state  = _players.Get(playerId); // lives on this server for the whole show
var result = Scoring.Score(word, state.Submissions, config);

state.Submissions.Add(result.Submission);
_writes.Enqueue(new SaveSubmission(playerId, result)); // Redis write happens later
return result;
Enter fullscreen mode Exit fullscreen mode

The background queue is a Channel<T> drained by a hosted service, which sends writes to Redis in batches:

// BackgroundService: drain the channel, write to Redis in batches
while (await _channel.Reader.WaitToReadAsync(ct))
{
    var batch = redis.CreateBatch();
    while (_channel.Reader.TryRead(out var w))
        _ = batch.ListRightPushAsync(w.Key, w.Value);
    batch.Execute();
}
Enter fullscreen mode Exit fullscreen mode

A production version also needs to handle a full queue and drain what's left on shutdown.

From then on I asked three questions about every operation:

  1. Can a hybrid cache replace this Redis read?
  2. Can this run more efficiently?
  3. Can part of this run in the background so I can respond right away? Writing to Redis, for example.

That got us through the 50,000 barrier. Word submission went from a 10-second average to under 200 ms.

The other bottleneck: message size

Redis wasn't the only thing the dashboard flagged. SignalR Service's bandwidth chart was climbing far faster than the player count.

Azure SignalR Service counts messages in 2 KB units, and anything bigger counts as several messages. Each unit also has a bandwidth ceiling: Microsoft's performance guide defines it as the most traffic a unit can carry while 99% of messages still arrive within a second. Go past it and delivery slows down for everyone. Our broadcasts were plain JSON, and many of them were well over 2 KB. Every broadcast to 50,000 players became several messages per player, and we hit that bandwidth ceiling long before we ran out of connections.

The fix had two parts. First, I cut every message down to what the client actually needed. Second, I switched the hub protocol from JSON to MessagePack, a binary format that's much more compact. Most broadcasts now fit in a single 2 KB unit.

Message size: 7 KB JSON costs 4 billed messages per player, under 2 KB MessagePack costs 1

Example sizes, to show the arithmetic.

Switching protocols is one line on each side:

builder.Services.AddSignalR().AddAzureSignalR().AddMessagePackProtocol();  // Server
new HubConnectionBuilder().WithUrl(hubUrl).AddMessagePackProtocol().Build(); // Client
Enter fullscreen mode Exit fullscreen mode

That's the easy part. The step that's easy to miss is the models. MessagePack needs a key for every field it sends, so every type that goes over the hub has to be decorated: [MessagePackObject] on the class and a [Key] on every property.

[MessagePackObject] // shared by the server and the Unity client
public class RoundResultsDto
{
    [Key(0)] public int Round { get; set; }
    [Key(1)] public string SecretWord { get; set; }
    [Key(2)] public int HostPoints { get; set; }
}
Enter fullscreen mode Exit fullscreen mode

Treat the key numbers like a wire contract. Client and server must agree on them, and once a version of the app has shipped, you only add new keys. You never renumber or reuse old ones.

Trimming the payload mattered just as much:

// Before: the whole round object
await Clients.Group(showId).SendAsync("RoundResults", round);

// After: only what the screen shows
await Clients.Group(showId).SendAsync("RoundResults", RoundResultsDto.From(round));
Enter fullscreen mode Exit fullscreen mode

What I'd do again

  • Ask three questions about every hub method: can a hybrid cache replace this read, can it run more efficiently, can part of it run in the background?
  • Keep a player's state on the server they're connected to, and save it to Redis in the background.
  • Watch SignalR Service's outbound traffic metric. Message count rounds everything up to 2 KB and hides the problem.
  • Start with MessagePack. The protocol is one line on each side, but every model needs [MessagePackObject] and [Key] attributes, and it's much harder to switch once the client has shipped.
  • Treat [Key] numbers as a contract between app versions: add new ones, never renumber.

Test 4: 100,000 bots, and the problem after the game

The next milestone was 100,000 bots: 500 container instances and almost an hour of provisioning. I used to go for a walk and come back when they were online.

Latency during the game was fine this time, and we hit 100,000 easily. The problems came after the game.

When a show ends, we save everyone's score, rewards and power-up balance. Until then, every server instance did this for all its players at the same moment, one write after another. That meant adding servers made things worse: more servers meant more writers hitting the same Cosmos DB at once, and the database only had so much throughput.

I had two options. I could run a single worker that processes all players, so the load stays predictable, or I could scale the database to absorb the spike at the end of every show. I chose the worker.

I also found that batching writes cut the database load further. The catch is that a Cosmos DB transactional batch only works inside one container and one partition key. So I made sure every document that belongs to a user lives in the users container, partitioned by user, and all of a player's post-game writes go out as one batch.

Post-game writes: every server writing to Cosmos DB versus one worker writing batches per player

The worker goes through a finished show at a pace the database can take:

await Parallel.ForEachAsync(finishedPlayers,
    new ParallelOptions { MaxDegreeOfParallelism = 32 }, // fixed, predictable load
    SavePlayerAsync);
Enter fullscreen mode Exit fullscreen mode

And because all of a player's documents share one partition key, their writes go out in one transaction:

// One partition key per player, so all their writes are one transaction
await _users.CreateTransactionalBatch(new PartitionKey(p.PlayerId))
    .PatchItem(p.ProfileId, [PatchOperation.Increment("/totalScore", p.Points)])
    .UpsertItem(new RewardDoc(p.PlayerId, p.Reward))
    .PatchItem(p.PowersId, [PatchOperation.Set("/balance", p.PowerBalance)])
    .ExecuteAsync(ct);
Enter fullscreen mode Exit fullscreen mode

What I'd do again

  • Load-test what happens after the game, not just during it.
  • Check what grows with your instance count. More servers can mean more load downstream.
  • Choose partition keys based on how you write, so related writes can share one batch.

Test 5: 1,000,000 bots

Then it was time to see if the architecture held at a million. The final run was less eventful than the ones before it, because the big optimizations were already done. The annoying parts were waiting for 5,000 container instances to come up, and waiting for the Azure team to raise our App Service quota so we could get enough instances.

On the SignalR side, a million connections means 1,000 units, because each unit holds 1,000 concurrent connections. The Standard tier tops out at 100 units per instance, so we ran a single Premium instance scaled to 1,000 units.

With a million bots connected and playing, every hub method stayed under our 200 ms target.

What I'd do again

  • Ask Azure for quota increases weeks before the big run.
  • Check your SignalR tier early. Standard stops at 100 units (100,000 connections), so 1M needs Premium.

Making it automatic

The 1M run proved the backend could handle a million concurrent players. It also showed that one small change to a hub method could badly hurt performance at scale.

So I built a GitHub Action that load-tests any branch. It scales up the infrastructure, builds an image from the branch, spins up the container instances, runs the test three times and writes a report. The default is 10,000 bots. In my experience that's the sweet spot: big enough to catch the major performance problems, small enough to run on every change. The team could now extend the game server and see what their change cost before it shipped.

Takeaways

Scaling a real-time app is complex and takes a lot of time. The best early investment you can make is a fast load test that behaves like production. The second best is letting your whole team run it from CI.

After going to a million, scaling comes down to four things for me:

  • pick the right resources
  • move whatever you can into the background
  • cache in memory
  • optimize the hot paths in your code

Top comments (0)