Game Optimization Methodology, or How 3 Days of Building a Test Harness Gave Me More FPS Than a Month of Writing Code

About me

 I'm a mobile game developer. Most of my work is gameplay programming and building tools for development teams. I like Lua, C++, Godot, and making interesting game-dev-related hobby projects. This article is about one of them.

Glossary

• Godot – a popular C++ game engine with its own scripting language, GDScript
• Physics server – the engine component responsible for physics calculations
• Collision – in a physics simulation, an intersection between a pair of objects
• Broad phase – the first stage of collision detection, reducing the number of candidate pairs that later stages need to process
• BVH, spatial hashing (hash grid) – common techniques used for broad-phase collision detection in 2D
• Baking – moving calculations out of program runtime and into a preprocessing step
• Batching – in computer graphics, combining the geometry of multiple objects to reduce the number of draw calls sent to the GPU
• P95, P99 (95th and 99th percentile) – a value below which 95/99% of the other values in a sample fall

Introduction

 It all started with my love for the indie game Brotato. For those who haven't played it, here's the short version: "You're a potato with guns, you level up, and mow down endless hordes of enemies." From that description alone, you can already see the two main performance problems:

1) There are lots of enemies; enemies shoot a lot; you also have lots of projectiles trying to deal damage to those enemies

2) The game is endless, and the number of enemies keeps growing with every wave

Example of a high wave in Brotato

Hardcore players can reach waves 1000+, while for more casual players the practical limit is around wave 150–200 (~6,000 enemies in a single wave).

 On desktop PCs, this isn't much of a problem. The graphics are fairly lightweight, and modern CPUs and memory can handle the game logic just fine. But Brotato is also extremely popular on handheld devices thanks to its casual nature and simple controls. This story started when I began playing Brotato on an Anbernic RG34XXSP, a handheld ARM Linux console.

Stage 1. First attempt

 On the RG34XXSP's rather weak hardware, the game dropped to 5–10 FPS from around wave 20 onward. That pushed me to get the game files, extract the Godot project (fortunately, that's easy enough for Godot games), and try optimizing it. The goal was to do it without major changes to the game's scripts, so future updates wouldn't invalidate all the work.

RG34XXSP specs for reference

 I already had some experience optimizing mobile games, so I built a simple test harness: the potato spawned on wave 100 with a fixed seed, the game recorded frame time, draw calls, and collision pair counts for every rendered frame, then wrote average values to a file at the end. The harness and measurement approach had plenty of problems (more on that in Stage 2), but I didn't feel like doing the "boring preparation", so I decided it was good enough.

 After several runs on the console itself and a few debugger runs on PC, I got the following picture:

• The main bottleneck was collision processing inside Godot's physics server. This was visible from the huge number of collision pairs per tick, and I was able to confirm it by running the game at half the original physics TPS
• The secondary bottleneck was the number of draw calls and GPU state changes. Textures were stored in separate files, forcing resource switches between draws and breaking batching
• Scripts also created load, but profiler overhead made it impossible to reliably point at a specific hot path

I tried to solve the physics-server problem by replacing Godot's implementation with a lightweight one using a rebuilt-every-frame hash grid, SoA position/size storage, and a simplified solver. I went this route for two reasons: Brotato only uses circles as collision shapes, which makes the implementation much smaller, and a hash grid is easier to implement than a BVH (which Godot's physics server uses by default). As a proof of concept, I decided to implement the system in GDScript first, then replace it with a C++ plugin if the results looked promising.

 The draw-call problem had a simpler solution: merge all textures into a single atlas and bake fonts into bitmap format. That would let the GPU resources change only when switching layers (enemies → bullets → text) and allow Godot to batch draw calls.

 A couple of weeks later, when the result was almost ready, the console unexpectedly died (thanks, Anbernic, for the cheap power controllers). Without a way to test on the target hardware, my enthusiasm faded. I finished the custom physics server and batching scripts, ran the tests, saw the improvement I wanted to see, and moved on to other projects.

Results thrown together in a hurry after the console died

Stage 2. Realizing the mistake

 Some time later I remembered the Brotato project and decided to try my optimizations on the Android version. After running both the optimized and clean versions on a Samsung A51 (a weak phone even when it launched in 2019), I found that my optimizations gave no performance improvement beyond measurement noise compared to the original game.

So I decided to dig deeper into the theory and apply what I'd learned at work to figure out what I had done wrong. These were the main mistakes I found:

• No run metadata. Metadata lets you verify that two recordings were made under the same conditions and can actually be compared
• No dry runs. On mobile hardware this matters not only for warming caches and compiling shaders early, but also for getting more comparable thermal conditions
• Using elapsed time to control test duration. When FPS drops, Godot may execute fewer physics ticks per rendered frame, distorting the simulation while making the workload lighter. Stopping by physics-tick count guarantees that two runs execute the same amount of work
• Using only one test scenario. That makes you optimize the playthrough of that one scenario instead of optimizing the game
• Not saving raw run metrics. With that approach, any change to data processing invalidates all previous results
• Using the arithmetic mean as the main representation. The average is easily distorted by anomalous values. I'll talk about better representations in Stage 3

I decided to try again, but this time put most of the effort into a proper test harness rather than the optimizations themselves — something that would let me objectively observe the effect of every change.

Stage 3. Successful attempt

 As mentioned earlier, the second attempt produced significant results in just five days. I spent three of those days building the test harness. The biggest part was the in-game testing system, which included:

• A test-scenario system that could quickly reproduce any game scenario
• A scenario runner where I could configure the number of recorded runs and dry runs
• A metric collection system recording selected values every frame
• A test HUD showing all measured values in real time
• A build plugin that writes the Git commit into the game build
• A recording system that saves run results, scenario configs, and build metadata to disk after all tests finish

Test scenario configuration UI

 The second most important part was data processing. This time I moved it into an external script that runs on the data recorded by the game. For values inside a single run, I chose the median, P95, P99, and the worst value.

Why these metrics? The median shows a typical value and barely moves because of outliers. P95 and P99 show increasingly rare bad values. After calculating these values for every metric, the script estimated variance and built the final graph, with the goal of making anomalies visible instead of hiding them. The script also had a compare mode: it checked the metadata of two recordings and, if they were valid to compare, calculated how every metric changed relative to the other recording.

 The graph below already shows the main difference from the old measurement approach: FPS drops happen as spikes rather than as a lower average. These spikes were usually caused by long physics-server work, when it couldn't calculate the required number of ticks within a rendered frame. My main goal was to get rid of those spikes first, and only then improve median FPS.

Run analysis visualization

 The harness wasn't just software. I also had a Samsung A51 connected to my PC, with all tests automatically executed on the phone through ADB and the resulting data pulled back to the PC for analysis. The same setup would have been possible on the RG34XXSP too, but I learned about ADB mode in MuOS far too late.

This made real-hardware testing much easier and also made it possible to delegate the testing loop to an AI agent.

 I settled on the following test scenarios:

1) 6 slingshots with damage multiplied by 1,000,000 on wave 200. This tested performance on a boss wave (waves divisible by 20) with a realistic amount of player damage

2) The same 6 slingshots on wave 249. This mostly existed to catch regressions on non-boss waves

3) AFK test on wave 150. This simulated a survivability-focused character build with low damage

Test setup

 Then I tested the changes from Attempt 1 against these scenarios. The custom physics-server replacement not only failed to improve performance, it actually made scenarios 1 and 2 worse. The rendering optimizations showed no effect at all, which I blamed on the frame being completely dominated by physics, so I postponed them for later testing.

 This was the first successful result of the new harness. It didn't give me any +FPS, but it showed the real impact of my old solutions.

After that, I ran the harness a few more times with the debugger attached to look for abnormal code paths.

 One thing worth pointing out: looking at function execution time in the usual way was mostly pointless here. The game calls a huge number of very simple functions, so instrumentation overhead heavily distorted the execution times reported by the debugger.

Instead, I looked at the second available metric: call count. Once I found a suspicious candidate, I would usually disable it and check whether performance improved. Then I'd repeat that until I found the problematic path.

1) Collision-pair count was growing faster than enemy count. The problem turned out to be that enemy hitboxes and collision areas were not separated. Because of that, the broad phase generated a huge number of invalid enemy-enemy pairs when what I actually needed were player-enemy pairs. The fix was trivial: move enemy hitboxes to a separate layer

2) Small script inefficiencies visible in the debugger. A static map size was recalculated every frame, collision pairs for fire spreading were queried for characters that weren't burning, already-disabled collision areas were disabled again every frame. All of these were fixed with a couple of guard checks

3) BVH vs spatial hashing. As I mentioned earlier, Godot's physics server uses BVH for broad phase by default. What I didn't know during the first optimization attempt was that Brotato's developers had switched it to a hash grid. As an experiment, I switched it back to BVH. This improved performance in the AFK scenario — the one most dominated by physics — but made the slingshot scenarios slightly worse. I kept the change because the gain in scenario 3 was much larger than the regressions in scenarios 1 and 2

 With just a couple of lines of optimization code and a few hours of testing, I got the following results compared to the vanilla game:

Scenario	FPS* before	FPS after	FPS difference	Difference
Slingshots, wave 200	16.08	32.30	+16.22	+100.8%
Slingshots, wave 249	45.33	51.95	+6.62	+14.6%
AFK, wave 150	14.02	44.02	+30.00	+214.0%

*median of the median FPS values across all runs of the scenario

 After that I went back to physics-server optimization hypotheses, but none of them produced a positive result. So I switched to looking for bottlenecks in scenarios 1 and 2, since scenario 3 was about 80% physics-bound.

 My main hypothesis was still memory/CPU/GPU bandwidth and constant state switching. To verify it quickly, I did two runs: one at 25% resolution (to rule out GPU fill-rate pressure) and one with enemy rendering disabled.

The result was what I expected: reducing resolution did nothing, while hiding enemies gave me +50% FPS. I hadn't been able to see this in earlier runs because the frame was completely dominated by physics.

So I reapplied my batching optimizations and inspected the result in RenderDoc, where I found one group of objects I had missed: blood effects. I had missed them because in the first attempt I never even tested a scenario where enemies were actually being shot, and blood accounted for 30% of all draw calls.

 After reading the documentation, I decided to move the particles to MultiMeshInstance2D (in Godot 4+ this could also be done with GPUParticles2D, but we work with what we have). Instead of having lots of separate blood emitters, I made a separate layer for the effect that could draw all blood droplets in a single command.

Blood rendered in a single glDrawArraysInstanced

 Combined, the old batching optimizations plus the blood optimization produced the following results. I left the AFK test out because the improvement there was within the margin of error:

Scenario	FPS before	FPS after	FPS difference	Difference
Slingshots, wave 200	32.30	40.11	+7.81	+22.8%
Slingshots, wave 249	51.95	55.7	+3.75	+7.2%

The difference in gains comes from wave 249 having lots of small enemies that die quickly. So most of the 7.2% improvement there comes from optimizing blood and damage numbers, while the bosses on wave 200 benefit from all of the changes.

 At that point I decided to stop — not because there was nothing left to optimize, but because the next changes were going to be much more expensive in development time.

Final result

Average FPS is shown for the recorded segment, not the entire run

 I consider the experiment a success. I achieved the goal of "making Brotato playable on a device where it previously struggled, without major changes to the game code" (the final change count in the game's original source was just under 200 lines).

The wave 200 scenario is the result I'm most impressed with. The total percentage gain was smaller than in the AFK test, but this scenario is much more realistic to encounter in actual play, and I was able to find and fix two substantial bottlenecks there.

 My main takeaway is simple: in optimization, the quality of the experiment matters more than the speed of writing code. The cheaper it is to test a hypothesis, the faster you can discard bad ideas and focus on the ones that actually work.

And with AI getting better all the time, this approach becomes even more important. AI can iterate through a hundred hypotheses for you, but if the measurements are distorted or processed incorrectly, those hundred iterations will do more harm than good.

 P.S.

If you spot any mistakes or have questions, let me know in the comments. I'd be happy to hear any feedback. This write-up is translated from my original write-up in Russian, which you can read here

Extra images

Run metadata

Run monitoring HUD

CSV part of the run analysis