I've been in game development for almost my entire career, and most of that has entailed developing game backends for online session PC/console games. Recently, I changed my field of activity, and I would like to summarize the main points I noted during this time. I hope this will be useful for novice developers and teams.
Everything is based on my PC/console game backend experience and might not be relevant for other cases.
What is a game backend?
First, let's figure out what a game backend is. Here, I mean the backend of online session-based games because MMO RPGs are developed with different approaches. In some companies, the game backend is referred to as online services. The game backend is the server part, almost unrelated to the actual game process. We can say that it is all about the moment when the player is in the game but not in battle.
The main tasks of the game backend include
- authorizing the player
- loading their profile data (inventory, wallet, quests, battle passes, social connections)
- selling items to the player
- matching the player with opponents
- sending the player into battle
- handling battle results The actual battle is handled by other software not directly related to the game backend and is often made using other technologies.
In addition, there is a large area that is not directly related to the game but that is necessary for successful game operation. It includes
- server monitoring through metrics and logs
- analytics
- game server management
- interaction with external systems (billing/voice chat/authorization, etc.)
- game backend configuration
- special tools and APIs to operate the game
1. There are almost no generally accepted approaches and off-the-shelf solutions
If you take several similar game projects made by different companies, you will almost certainly get several unique stacks of technologies, architectures, and approaches to game backend implementation. For a number of reasons, there are no generally accepted practices and nearly no off-the-shelf solutions in gaming backends. This is especially surprising since most online session games are, in essence, very similar in terms of backend mechanics. I recommend that every studio tries to create its game backend in a way that can be reused in other games internally.
2. Replication is convenient but expensive
In general, information systems have two ways of communication: through shared memory and through messages. Replication is essentially the emulation of shared memory, and it is usually a one-way process. In the case of a game backend, this is when a change in some server state leads to an automatic change in the client state. And on the client, you can subscribe and react to these changes.
Replication is usually the primary way of transferring information during a game session. For example, it is built into Unreal Engine. Replication is sometimes also used for interaction between the game backend and the client, but there is no off-the-shelf solution, and each project implements this on its own. This is not an easy task and harbours many pitfalls, so I can't say for sure that it's worth it.
On the other hand, it is convenient to work with replication, and it eliminates the desynchronization of server and client states. This is especially helpful when server and client have been developed on different technologies, so the code can't be reused to apply server changes to the client. But I would not go there for a game with a not very complex state, especially if there is another way to unify server and client state updates.
3. C++ is incredibly fast
Everyone knows that C++ is used to develop performance-critical systems. Because of this, it is the primary language for developing game clients. But for other reasons, it is not so popular for server-side development.
I realized how fast it was when I worked on one game where the backend was an almost unscalable C++ monolith with most of the logic executed in one thread. This server was able to handle 70k CCU. I moved to this project from the Java world, and I was surprised to find that the millisecond precision of log timestamps configured in ELK was not enough for this server to sort events by time correctly.
The reason, of course, is not only the efficiency of the generated C++ machine code but also the ability to work with memory efficiently. So I recommend using C++ for your game backend if you have a team of professionals.
4. Monoliths are better than microservices
This is a provocative statement, but I'll try to clarify it. In the modern information technology world, a more or less generally accepted position is that only highly distributed systems can cope with a large load and achieve reliability, availability, and zero downtime system updates.
Looking at the top Steam games, you will see that only 10 have a peak CCU exceeding 100,000. According to my practice, on large numbers without ping messages, on average, one player makes less than 1 request to the server in 10 seconds. In this case, these 100,000 players generate not more than 10,000 RPS. Is it a lot or not? You can easily handle this load with a single server using binary network protocol and efficient database communication.
If you play games, sometimes you see announcements about scheduled maintenance when the game is unavailable. There are a few reasons for that. On the one hand, due to game specifics, it is difficult to do a zero-maintenance game backend. But on the other hand, most games are not critical services (some are, but most are not), and short game downtime does not lead to a noticeable user loss.
There is high entity connectivity in games and a large logic variability when developers try to create small elements which designers can combine into final mechanics on their own. This leads to the fact that developing a highly distributed system with independent parts adds significant overhead to development at all stages.
Considering the above, I recommend using a monolith for some parts of the game backend. It would be much cheaper to develop. And seamless backend updates can be organized in other ways, for example, through sharding, when a new game backend is launched for each new client version and players are routed to it. This approach requires additional development but is much less than spreading the logic over compact services. Also, you can support more than one monolith in shard to increase performance and availability.
5. Feature toggles would save you from sleepless nights
Using feature toggles is a common practice for mobile applications and websites, but sometimes, it is underestimated in game development. For those unfamiliar with feature toggles: it is the ability to dynamically enable and disable some particular features. For example, Roblox locks every new or modified line of code to feature flags. But outside such an extreme case, simply locking new features with feature flags allows you to test some features on production servers with a limited number of users or just on developers and switch off features with bugs. Also, integrating it with analytics would allow you to use it as an A/B testing solution.
6. A single authorization system would never be enough
Perhaps this is the most obvious observation. You should be ready to, at some point, change the authorization system completely (and usually billing along with it) or support several different systems simultaneously. For example, it can happen when you want to be on Steam and EGS simultaneously or when you decide to launch in China, where you would be required to support local game distribution systems. Also, don't forget crossplay between PC and consoles. At the beginning of the project, there is a temptation to use some implementation features of a specific system internally. You may later find out that Xbox authorization returns an alphabetic unique player identifier rather than digital.
7. The ability to collect thousands of metrics would help you a lot
Most likely, this observation is not specific to games. But the potential number of unique metrics in games is measured in hundreds. Thus, the complexity of the internal logic is such that the opportunity to understand what is happening in dynamics is appreciated.
By metrics, I mean numbers at a fixed rate in special storage that can be analyzed on different charts. An example is Prometheus/Graphite with Grafana. You usually improve your monitoring when the issues arise, but initially, it is worth configuring a system that allows you to do it.
I would recommend having a second-by-second accuracy for the last day, five seconds for the last month, a minute for the last year, and an hour for more than a year. And be ready to store 100k+ metrics because of the large number of metrics from game session servers.
8. Matchmaking is more challenging than it looks
Matchmaking is the backend service which matches players with teammates and enemies for a battle. Creating a good matchmaking service might be much more challenging than it first seems. Matchmaking problems usually do not appear with a large CCU. But with a standard life cycle, the game has maximum CCU shortly after the launch. A gradual decrease in the number of players begins after that. At the same time, the internal complexity and variety of mechanics and game modes begins to grow. In addition, depending on the specifics and the main game territories, the intraday CCU fluctuation is up to 10 to 1 (peak CCU is 10x of min CCU). Therefore, matchmaking problems will appear sometimes in some game modes.
I recommend starting with primitive matchmaking thanks to the large CCU at launch. But you should cover everything with metrics and analytics so when someone reports too long a queue or an unbalanced team, you can determine whether this situation is normal or an MM problem.
Thank you for reading. Of course, it is just a part of the game's backend-specific moments. But these seemed the most interesting personally. So please write in the comments what other moments you know that differ in game backend development. Also, stay tuned. I'm going to write another article with a detailed description of the architecture and approaches used in a real game backend.
Top comments (0)