DEV Community: Konstantin Usachev

DevOps in Game Development: Accelerate Production with Automated Workflows

Konstantin Usachev — Mon, 13 May 2024 13:19:39 +0000

Hi, my name is Konstantin Usachev, I'm a Software Engineer with over 15 years of experience behind my back, 12 of them – in game development. During these years, I've experienced firsthand the impact of DevOps on game production cycles.

In this article, I’ve assembled some practical insights on automated workflows in GameDev that helped me a lot in my career, and I hope will help you boost productivity and product quality on your projects. Below, I'll share strategies and tools that have proven their worth in automating and accelerating production. From managing complex assets with Git LFS to harnessing Jenkins for cross-platform builds, these are lessons learned from direct hands-on application. Keep in mind that the particular tools I use here in examples are not as important as the core ideas behind these approaches you can adopt.

So, if you’re a DevOps/Software Engineer working in GameDev or just preparing to get your hands dirty in this thrilling area – this article is for you!

DevOps Tools for Efficient Asset Management

Dealing with game assets in DevOps can test the limits of any system. If you’ve dealt with a modern game's assets, you know it: large files that make cloning a repository feel like a coffee break and push operations that are longer than some of your gaming sessions.

With files sometimes measuring in hundreds of megabytes for high-quality assets, the real issue arises when you try to fit these into a traditional CI pipeline. Git struggles with large binaries because it was optimized for text. Assets don’t compress well, and differencing tools are almost useless for binaries, ballooning your repositories and making efficient collaboration more of a wish than a reality. Binary asset management needs to handle the continuous updating and storing of files without the chokehold on your workflow.

To overcome the hefty challenge of asset management, we turn to specialized tools tailored for this purpose like Git Large File Storage that replaces large files with text pointers while storing the file contents on a remote server.

Below are a couple of simple-to-implement tricks that will help your workflows with Git LFS and a CI/CD pipeline.

Here is an example of using GitHub CI to checkout and build a Unity project:

name: Build
on: [push]
jobs:
  build:
    name: Build for ${{ matrix.targetPlatform }}
    runs-on: self-hosted
    strategy:
      fail-fast: false
      matrix:
        targetPlatform:
          - Android
    outputs:
      buildVersion: ${{ steps.build.outputs.buildVersion }}
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          # don't delete 'cache'
          clean: false
      - name: Git LFS Pull
        run: |
          git lfs pull
          git add .
          git reset --hard
      - name: Build Unity Project
        id: build
        uses: game-ci/unity-builder@v2
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
        with:
          projectPath: Game
          targetPlatform: ${{ matrix.targetPlatform }}
          androidAppBundle: false
          androidKeystoreName: publish.keystore
          androidKeystoreBase64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
          androidKeystorePass: ${{ secrets.ANDROID_KEYSTORE_PASS }}
          androidKeyaliasName: publish
          androidKeyaliasPass: ${{ secrets.ANDROID_KEYALIAS_PASS }}
          androidTargetSdkVersion: AndroidApiLevel31

Here, we automatically pull the repository and build an Android version of our game. We could store the result apk file and even send notifications via Slack.

For Docker, you might be interested in a multi-stage build process that can compile assets within a Docker environment and then use them in the final image, reducing the overall size and keeping build environments consistent. Here is an example Dockerfile where we use one Docker image to build some backend .NET application and another to form the final image:

# Container we use for final publish
FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
ARG PORT=8080
EXPOSE ${PORT}

# Build container
FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build

# Copy the code into the container
WORKDIR /../
COPY [".", "Application"]

# NuGet restore
WORKDIR "Application"
RUN dotnet restore Backend.sln
#COPY ["*", "Application"]

# Build the API
RUN dotnet build "Backend/Backend.csproj" -c Release -o /app/build

# Publish it
FROM build AS publish
RUN dotnet publish "Backend/Backend.csproj" -c Release -o /app/publish

# Make the final image for publishing
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Backend.dll"]

Automating Game Testing Workflows

With traditional software, you might be testing for functionality and performance, often in a predictable environment. In GameDev, however, you're also testing the unpredictable player interactions, some random generator-based mechanics, AI behaviors, and real-time rendering across various hardware. And you're not just looking to see if the game crashes; you're assessing whether it delivers the intended performance as well.

The key to automation in game testing lies in the ability not only to split complex game mechanics into separate pieces you can test but to replicate and evaluate complex player behaviors and scenarios. This means creating automated test suites that can not only press buttons but also make decisions as a player might.

At first, you start with unit tests, where you test small pieces of your game mechanics in an isolated way. Usually, games are too agile during the active development stage. So, it is not always feasible to unit test a big part without affecting development performance. But the most critical and stable mechanics might be covered.

Functional testing, which verifies complex game mechanics combinations, involves scripting AI players who can navigate the game world, engage with objects and NPCs, and execute sequences of actions that a human player would. Tools like Selenium or Unity's Test Framework can be programmed to perform complex in-game actions, simulating real gameplay.

In performance testing, when evaluating the game's stability and framerate in different scenarios, automation tools must go beyond the standard load tests. The popular practice is to have an automatic grid of cameras on each level in your game and a command to measure performance metrics from each of them. With this, it is trivial to start measuring performance in your CI pipeline to catch any performance degradation earlier.

You will likely also have to deal with compliance and localization testing – confirming the game adheres to platform-specific regulations and accurately translated. However, these issues fall beyond the scope of this article.

Mastering Cross-Platform Deployment in GameDev

Deploying games across multiple platforms is inherently complex. Each platform has its own requirements – different hardware, OSs, input methods, and even specific rules for store listings. What works effortlessly on a PC might be a hurdle on a console or a mobile device.

A key challenge is making sure that your game doesn't just run but also performs well and provides a consistent experience wherever it's played. Moreover, with games frequently updated or patched, integrating automated deployment into your DevOps process is essential.

Here’s how you can tackle these challenges effectively:

Unified Codebase: To manage code efficiently, use conditional compilation flags. This keeps your codebase unified, making updates straightforward. For instance, in Unity we can write:

#if UNITY_IOS
    // iOS-specific functionality
#elif UNITY_ANDROID
    // Android-specific functionality
#endif

and for an Unreal Engine project, we can use a very similar approach:

#if PLATFORM_IOS
    // iOS-specific functionality
#elif PLATFORM_ANDROID
    // Android-specific functionality
#endif

Advanced Automated Builds: Enhance your CI/CD to handle builds, tests, and deployments based on the outcomes of automated tests. Here’s a simple Jenkinsfile example that uses parallel stages and environment variables:

pipeline {
    agent any
    environment {
        PLATFORM = 'initialize based on trigger'
    }
    stages {
        stage('Build') {
            steps {
                script {
                    sh 'build.sh -platform $PLATFORM'
                }
            }
        }
        stage('Automated Tests') {
            parallel {
                stage('Unit Tests') {
                    steps { sh 'run_unit_tests.sh' }
                }
                stage('Integration Tests') {
                    steps { sh 'run_integration_tests.sh' }
                }
            }
        }
        stage('Deploy') {
            when {
                expression { return currentBuild.result == 'SUCCESS' }
            }
            steps {
                sh 'deploy.sh -platform $PLATFORM'
            }
        }
    }
    post {
        always {
            slackSend (message: "Build and Test Pipeline completed for $PLATFORM")
        }
    }
}

Docker for Consistent Environments: Docker can standardize your build environments, encapsulating SDKs and tools. Consider a multi-stage Docker build that ensures each phase of game production is reproducible:

FROM gcc:9.3 as builder
WORKDIR /src
COPY . .
RUN make all

FROM alpine:latest as tester
COPY --from=builder /src/game /game
RUN ./test_game

FROM alpine:latest
COPY --from=tester /game /usr/local/bin/game
CMD ["game"]

Emulators and Virtual Machines for Testing: Using Android Studio and Xcode, simulate gameplay on Android and iOS. For broader testing, VMWare or VirtualBox can mimic various operating systems, giving you a virtual test lab without physical hardware. This enables testing across potential player devices, ensuring that everyone has a great experience regardless of their platform. Conclusion

As we've explored, mastering DevOps in game development involves much more than streamlining processes – it's about enhancing the gaming experience across all platforms. By implementing the advanced strategies and tools discussed, from unified codebases and sophisticated CI/CD pipelines to Docker environments and emulator integrations, you ensure that your game not only meets but exceeds player expectations.

I hope that the strategies and practices covered in this article will give you some ideas on how to keep your development agile and your releases flawless. Automation allows you to face many challenges of efficient game development. Ready to transform your next game project? Let these strategies guide you to success and set new standards in the gaming world.

Optimizing Game Performance: Essential Techniques and Tools

Konstantin Usachev — Fri, 26 Apr 2024 10:49:10 +0000

Newzoo’s Global Games Market Report states that in 2023, the global games market shot up to levels never seen, reaching $184.0 billion in terms of revenues. With this pace of development, game developers have no choice but to create an immersive experience which will be enjoyed by gamers and at the same time ensure that there is no difference when shifting from one device to another. This is why it is such a big deal for game performance optimization.

Game performance optimization refers to improving various aspects of a game like frame rate, graphics quality, use of memory and how fast it responds to input commands. With these factors improved upon well enough, developers can optimize their gaming experience resulting in increased player satisfaction and involvement.

The importance of optimizing game performance cannot be overemphasized. Even minor performance issues can lead players to abhorrent situations which could force them not to continue playing. In addition, due to more complex features found in modern games and several devices on which they get played, achieving optimum performance presents numerous challenges for the developers involved in this industry.

Being a software developer with over 13 years in the game development industry, I conceived this article as an introduction to game optimization tools and techniques. We will discuss the main performance indicators, the primary optimization approaches, profiling and debugging software, and platform-specific ways of increasing speed. Both aspiring developers and gamers can acquire knowledge about game performance enhancement.

First and Foremost: Understanding Game Performance Metrics

Performance metrics are crucial indicators of a system's ability to deliver a seamless, immersive experience. Frame Per Second (FPS) is one of the most pivot metrics that directly affects both the smoothness and visual quality of a game. Higher FPS rates usually provide for smoother gameplay and reduce the unpleasant feeling of stuttering or screen tearing, which is especially disadvantageous during fast-paced genres like first-person shooters or racers.

Additionally, the employment of GPU (Graphics Processing Unit) and CPU (Central Processing Unit) gives an idea about the way the workload and efficiency are distributed in terms of gaming setup. In order to be well performing, this has to be perfectly balanced with neither GPU nor CPU becoming a bottleneck. For example, when CD Projekt Red released “Cyberpunk 2077”, lots of players experienced significant performance issues that stemmed from under-optimized CPU and GPU usage, thereby initiating discussions on hardware requirements as well as game optimization.

Another important metric is memory consumption. Here, how quickly a game can load up and switch between complex game environments without any delay relies upon it is extremely important. In some games like The Witcher 3, vast amounts of RAM are used for handling detailed open worlds and dynamic content—it clearly demonstrates how more memory translates to a better gameplay experience.

Input latency and network performance are equally significant; the former affects responsiveness, crucial in competitive gaming, and the latter impacts multiplayer gaming experiences. Bad network performance can lead to lagging: the real example is a notable issue during the early days of Destiny 2, where players experienced delayed actions in high-stakes battles— it underlines the importance of robust game server infrastructure.

These metrics lie the foundation for measuring a game's technical performance, but most importantly, they shape how players perceive and enjoy their experience, pushing developers to constantly improve their technologies.

Moving Forward and Diving Deeper

Let us proceed to more techniques. Note that it is important to approach these techniques with caution and do proper research to guarantee the stability of your game.

Level of Detail (LOD) management maintains performance simultaneously providing high visual fidelity. You can adjust the complexity of models based on their distance from the camera — it will help you significantly reduce the number of polygons rendered, thus boosting FPS. Employing a dynamic LOD system is a standard approach integrated out-of-the-box in most game engines. This system is crucial when we talk about balancing detailed environments and extensive draw distances without sacrificing performance. It's important to ensure that all high-polygon models have properly configured LODs, as this significantly impacts GPU performance.

Occlusion culling is another technique that allows for improved rendering efficiency—it does so by ensuring that only objects visible to the player are rendered. This method doesn’t let the GPU waste resources on drawing elements that are hidden behind other objects. Games with corridor-like level topology, such as DOOM Eternal, particularly benefit from this technique: the game's complex corridor-based environments allow for significant performance gains, as rooms separated by narrow passages can be selectively rendered without needing individual checks for each object within them. This betters the ability to maintain high frame rates during fast-paced action.

Dynamic batching and texture atlasing are common techniques used to address the problem of high numbers of draw calls in a game, which is often a bottleneck for low-end systems. Dynamic batching combines similar objects into one draw call which avoids switching textures between draw calls by combining several small textures into a single large one through texture atlasing. Unity Engine, for instance, provides built-in support for these techniques, which aids developers in creating more efficient games.

Shader optimization is vital in reducing the computational load on GPUs. Properly optimized shaders can be all that separates a visually stunning game that runs poorly from one that looks great and plays smoothly. For example, Epic Games optimized shaders for Fortnite across different platforms to make sure it performed well on both high-end PCs as well as mobile devices.

Furthermore, multithreading employs modern multi-core processors to distribute workloads among various threads, consequently enabling games to handle complex calculations for physics, AI, and more without slowing down the main gameplay loop. While multithreading is a feature used in all game engines to some degree, it is typically used in two primary threads: the main game logic thread and a separate graphics processing thread. The main thread processes all game logic and then passes necessary information to the draw thread, which prepares and sends this data to the GPU in the following frame, effectively operating a frame behind.

A newer approach in Unity's Data-Oriented Technology Stack (DOTS) allows for the application of multithreading to game logic as well; though widespread implementation in actual games is still emerging.

It is worth remembering that while profiling game performance, if considerable time is spent on complex calculations, developers might use the potential benefits of offloading these tasks to a separate thread to optimize performance.

Profiling and Debugging

No way we could stop at just discussing the optimization techniques–profiling and debugging tools are also indispensable in game development. They provide deep insights into performance bottlenecks and system inefficiencies. Unity and Unreal Engine, two of the leading game development platforms, offer built-in profilers that help developers enhance game performance. Unity's Profiler is used for optimizing game elements in real-time, and Unreal's Profiler offers potent options for pinpointing performance issues. A critical first step in this regard is to use these tools to determine whether the bottleneck is in the CPU or GPU, for which the built-in capabilities of these engines are typically sufficient.

On the hardware side, RenderDoc stands out as the most popular tool for GPU performance analysis. Additionally, NVIDIA Nsight and AMD GPU PerfStudio allow for detailed scrutiny, which is crucial for graphic-intensive games. In the same way, CPU profiling tools like Intel VTune and AMD CodeXL optimize computing operations and make gameplay smoother. In addition, balancing memory and network performance is crucial. Unity offers a Memory Profiler for efficient resource management, and tools like Wireshark can reduce latency.

Optimizing for Different Platforms

Finally, developers have a crucial job of optimizing video games for various platforms, which involves implementing specific strategies to guarantee top performance on different hardware setups.

When it comes to PCs, developers typically prioritize adaptable graphics settings to accommodate a variety of hardware capabilities. Games like the aforementioned Cyberpunk 2077 are a prime example of this approach, since it provides adjustable graphics features that allow both powerful rigs and more modest systems to smoothly run the game.

Gaming consoles have specific optimization challenges because of their unchangeable hardware. Developers have to carefully adjust their games to work within these limitations, as demonstrated by The Last of Us Part II on PlayStation, which pushed the boundaries of what can be done on older hardware through meticulous performance improvements.

On the other hand, optimizing games for mobile devices consists of minimizing resource usage without sacrificing gameplay quality. Titles like PUBG Mobile accomplish this by using lower-resolution graphics and simpler particle effects, guaranteeing a smooth experience on a wide range of devices.

When it comes to optimizing VR and AR technology, there are some tough challenges too since one has to make sure the frame rates are super high to avoid making people feel sick. Developers have to do a lot of testing and use special techniques to make games on platforms like Oculus Quest better and more enjoyable for players.

Outro

There is so much left to discuss regarding this topic still. The future of game optimization is closely connected to the advancement of technology and the challenges that come with it. With the rise of next-gen consoles and high-performance GPUs, game developers are now tasked with pushing the boundaries of graphics while also maintaining smooth gameplay, as we discussed earlier. Machine learning is playing a larger role in this field as well, with algorithms being developed to improve rendering methods and unify testing procedures. One standout technology is NVIDIA’s DLSS, which utilizes AI to enhance lower-resolution visuals in real-time, which results in clearer images without overburdening the hardware. These advancements in technology are set to revolutionize game development and increase player engagement, and as we proceed in this process, we will see the expansion of the techniques pool too.

One real game backend overview

Konstantin Usachev — Mon, 03 Oct 2022 12:41:52 +0000

A few weeks ago, I announced a post with an overview of some real game backend, and here it is. Unfortunately, I don't work for this developer anymore, so I can't name the game. In general, it is an online-session-based FPS game. And to recall, by "game backend", I assume it is a part of the server code that handles everything the player does when they are not in the battle.

Tech stack

C++ for the server and client
MySQL database
Node.js/Vue for the administrative interface
ELK for logs
ClickHouse/Graphite/Grafana for metrics
Hadoop/Hive/Impala/HUE for analytics

Architecture

There are several backend nodes:

Auth server

It is the first server client connects to
It handles client authorization through one of the external authorization services (like Steam)
It handles new player creation and EULA acceptance
It sends back to the client the current least-busy Gateway to connect to

Gateway server

It is the only server that supports horizontal scaling in this backend
All game clients keep a TCP connection to one of the Gateway servers
Only Auth and Gateway servers have publicly available addresses.
The Gateway server handles all client pings, and it is more than 90% of the clients' requests
It handles "global" chat, which is a chat per Gateway
It protects from spam messages
It keeps outcome message buffers for each client. It is something like 1 MB per connection, allowing for the distribution of some RAM consumption.

World server

It is the Monolith of this backend
It handles most parts of the game logic
It keeps in the RAM caches of all online players
It is the main performance bottleneck
It is the main crash bottleneck because, when it crashes, it takes some time to write more than 100 GB crash dump file with the current memory state

MatchMaking server

It handles matchmaking (i.e. requests to enter a battle and distributes players in teams)
It requests and manages new game server launches to send players to.
It handles match results

Master server

It is a custom-made server management system
It provides configuration to other nodes
It has information about each server load through special processes on each server called Commander (Brain Slugs in the picture)
It can send commands to Commanders to launch new processes
It handles MatchMaking server's requests to launch a new game server and has information about global game servers' current load and capacity
It provides some administrative API to do some cluster-related manipulations (stop/launch another Gateway server, for example)

AdminTool server

Administrative Web UI and Websocket API to execute administrative actions (like check player profile, ban player, give the player something and so on)
Allows through Websocket API to integrate with custom operator's tools to automate some actions (give everybody a loot box, or give something to a player because he purchased it on the game's webshop)

Metrics server

It handles all metrics and analytics messages for other nodes
It supports some primitive UI to access recent metrics (for development purposes mostly)
It sends metrics to Clickhouse through carbon-clickhouse. There are ~10k metrics for backend nodes only without game servers
It sends analytics events to MySQL where it gets replicated to an analytics MySQL instance and by daily job gets moved to Hadoop cluster

It is a bird's-eye view of the cluster's components. Each node is a process which can be launched on a separate physical or virtual server. However, most utilize very few threads and don't have any high load, so, in production, they fit on three physical servers.

Let's take a look inside a single server:

Every server contains a bunch of services. At the start, the main thread parses this node's XML config and creates services. The config is provided by the master server through the local commander. Each service launches one or more threads for its work. In this picture, you can see some examples of such services.

Services communicate with each other asynchronously through tasks. Essentially, every service is a conveyor that handles tasks specific to them. Every task is a data-only struct with the following information:

The service the task should be handled by
The task ID, so it can be cast properly
Information about the service and thread where this task was created
A pointer to a callback function for when the task is executed—this gets called from inside the thread it was created
Specifics of the task information required to execute it properly and to store execution results

Every service adds some boilerplate code to hide this internal complexity and for more convenient usage.

Cluster configuration

The whole cluster configuration lives in two XML files.

The primary XML file has sections for each node configuration and a long list of defined variables to reuse some values in different config parts and override them in the second XML config. This file is provided with a new server code version and is not supposed to be modified locally.
The secondary XML config is the list of overrides for variables specific to the current environment, like local dev cluster, test stand or production. It is a part of the current environment and can be modified locally or through the AdminTool.

When the master server starts, it combines these two files into a single XML document and sends it to each node. Also, the master server observes config modification and sends updated sections when changes in the XML document happen. Therefore, the modification of some server settings, such as switching a feature on or matchmaking configuration modification, is essentially just a modification of this XML file on a server with this cluster's master server process launched.

Network communication

Here, no special framework for network communication, and even serialization is used. As described before, each player has a TCP connection to one of the Gateway. All client-server communication is organized through serializable messages, and no PRC idiom is used. The Gateway server handles some message types, but most go farther to the World server.

Each serializable object is inherited from a special class and implements Write and Read methods to serialize its data to a stream and deserialize it.

Player state management

When the player logs in to the game, information related to him gets loaded on the World server and cached inside a single PlayerCache object. During this load process, some DB data might be irrelevant or incorrect. In this case, the load routine executes the required updates to fix it.

All player caches are stored inside the same process in a custom-made in-memory DB where any service can asynchronously request and lock some player's cache and release it to make it available for other services.

This approach reduces read queries to DB because you never need to read anything after the initial load. Also, it allows all the information in one place and synchronously do all the required checks before modification.

At the same time, this load consist of ~20 SQL select queries and a single cache might take up to 2 MB of RAM. So, if the cluster starts during a peak time, some system to limit players' login pace is required to handle this load.

The following is the standard workflow with a PlayerCache:

The World server handles a message from the player to execute an action.Because almost every message from the client is a modification command, a locked PlayerCache is provided to the message handle method.
All the checks to verify this action are performed
Send a request to the DB service to modify the DB state
When the DB state is modified, these changes are applied to the cache
Send modification information to the client and release the cache

When a player connects to the server, it receives the same PlayerCache. So to keep it in sync with the server, the following approach is used:

Every modification action we might want to do with the cache has two versions: the TryAction method and the DoAction method
The TryAction method checks whether this action can be performed, then returns an object with all the information about this modification
The DoAction method receives this modification object and executes the actual modification

So as described previously, on the server, we execute the Try method. If it is successful, then after the DB modification, we apply this modification to the server cache and send it to the client who also applies it. And because the server and the client are both made with C++ and can share some source code, there is no need to write these Try and Do methods twice on the server and client. Also, the client can use the Try method before sending a request to the server to check preconditions and notify the player in case of failure.

Administrative interface

There is a Node.JS-based tool called AdminTool to modify the server's config remotely and perform administrative actions. It connects to the World server and provides a WebSocket-based API. Also, it hosts a Vue-based web UI, which uses this WebSocket-based API to perform these actions. Some external tools use this API as well.

To do this, the World server provides some special API available only for the AdminTool and a special set of messages. These messages on the World server's side mostly use the same methods regular players use. And Node.JS part deserializes and serializes this message from a byte stream to a JSON. When performing some action with a player, AdminTool allows for the action and its cache's being locked on the World server, so if this player is currently online, it gets disconnected. AdminTool also provides all the required functionality to manipulate the player's state.

Analytics

Each analytics event can have up to 1 UUID field, up to 10 long fields and up to 10 double fields. In code, it is covered in a type-safe class with mapping info.

All analytics events come to MySQL in two tables:

A table with event schemes
A per-day partitioned table with the actual data

Of course, MySQL isn't an OLAP DB. So it only keeps the data for a few days until it moves it to the Hadoop cluster.

The scheme of both tables looks very similar:

Columns in the data table:

int EventId
int EventVersion
binary Guid
bigint Long1, Long2 … Long10
double Double1, Double2 … Double10

Columns in the schemes table:

varchar EventName
int EventId
int EventVersion
varchar GuidName
varchar Long1Name, Long2Name … Long10Name
varchar Double1Name, Double2Name … Double10Name

With information from the scheme table, ETL scripts can create the required Hive Views for convenient data representation and query data through HUE using friendly names. This process never involves the modification of the MySQL table schemes to add new events.

When there is more data to send than fits into a single event, you can always create another event with the rest of the data. When you change the event's scheme, you should increase its version so that a new row will be added to the scheme table and the Hive View be updated.

This approach looks a bit complicated, but it worked well with ~200 mln events a day.

This was just an overview of one existing game backend. Of course, it has some pros and cons, but it was able to handle 70k CCU peaks, and this game is still actively supported.

I believe that you can select any tech stack or architecture and use any approach to build your product. The only difference is in the cost and effort required to make it and support it later.

How to measure hardware controller lag

Konstantin Usachev — Sun, 04 Sep 2022 14:38:38 +0000

While I'm in the process of writing another post about game backend development announced recently, here is a short story from my current project. One part of this project is a custom hardware controller. At some point, we realised we needed a way to measure its lag because we started to feel it during tests.

The first thought would have been to do it programmatically. It has some benefits:

You can do it automatically, even put the hardware to CI infrastructure and measure it every firmware update
The results would be accurate and repeatable

But at the same time, there are some drawbacks:

First, it would require additional development, in some cases even adding two-way communication to the controller, which we don't need
It would require us to not only make the controller's firmware changes but also host device software changes
This additional lag measurement–related software itself and its updates might affect the result
We might have some mechanical design lag reasons, which we would also like to catch for new hardware iterations
We cannot directly compare it with other existing devices to understand where we relate to other controllers out there

Video-based measurement

So after a short discussion, we came up with a relatively simple approach that, in our case, covers all our needs. You can measure the lag using a high-FPS video camera with a high FPS display and counting the time it takes from the button press to reaction.

Let's try to calculate this approach error. If we have a display with DisplayFPS and a camera with CameraFPS, the overall error would be something like ±(1000/CameraFPS) + (1000/DisplayFPS) / 2. So the result would be higher on average than the actual value.

I used a 165 Hz display and iPhone slo-mo camera with 240 FPS. So it would give an error of ±4 ms + 3 ms for a single measured lag. Is it much? It depends. We jumped from 70 ms to 140 ms lag when we started noticing it with our controller, so it is far enough in our case. Also, several measurements increase accuracy.

The actual process

The main idea is to record a video with the button press and display the reaction simultaneously. The measurement accuracy is highly affected by the actual measurement protocol. I recommend using a holder or tripod for the camera and trying to hold the controller in a way that it is obvious when the button press started. In our case, we record ten presses and calculate a median value. Also, we notice some extreme values and try to determine whether it is a measurement or controller-related issue.

Some results for on-hand controllers

I've measured the lag of some controllers to demonstrate that it is accurate enough to detect Bluetooth overhead, for example.

Type	Name	Connection type	Lag
gamepad	Xbox One Controller	wired	38 ms
gamepad	Xbox One Controller	Bluetooth	46 ms
keyboard	Laptop keyboard	wired	29 ms
keyboard	Logitech MX keys	Bluetooth	42 ms
mouse	Logitech G502 Hero	wired	33 ms
mouse	Logitech MX Master 3	Bluetooth	38 ms

You can find several descriptions of this approach with different variations. Here, the idea is that even a trivial and straightforward variation is sufficient and can be used as a cheap way to monitor latency for a custom controller.

Did you encounter such a task in your practice and try a different approach?

What I've learned after 10 years of game backend development

Konstantin Usachev — Mon, 29 Aug 2022 15:35:00 +0000

I've been in game development for almost my entire career, and most of that has entailed developing game backends for online session PC/console games. Recently, I changed my field of activity, and I would like to summarize the main points I noted during this time. I hope this will be useful for novice developers and teams.

Everything is based on my PC/console game backend experience and might not be relevant for other cases.

What is a game backend?

First, let's figure out what a game backend is. Here, I mean the backend of online session-based games because MMO RPGs are developed with different approaches. In some companies, the game backend is referred to as online services. The game backend is the server part, almost unrelated to the actual game process. We can say that it is all about the moment when the player is in the game but not in battle.

The main tasks of the game backend include

authorizing the player
loading their profile data (inventory, wallet, quests, battle passes, social connections)
selling items to the player
matching the player with opponents
sending the player into battle
handling battle results The actual battle is handled by other software not directly related to the game backend and is often made using other technologies.

In addition, there is a large area that is not directly related to the game but that is necessary for successful game operation. It includes

server monitoring through metrics and logs
analytics
game server management
interaction with external systems (billing/voice chat/authorization, etc.)
game backend configuration
special tools and APIs to operate the game

1. There are almost no generally accepted approaches and off-the-shelf solutions

If you take several similar game projects made by different companies, you will almost certainly get several unique stacks of technologies, architectures, and approaches to game backend implementation. For a number of reasons, there are no generally accepted practices and nearly no off-the-shelf solutions in gaming backends. This is especially surprising since most online session games are, in essence, very similar in terms of backend mechanics. I recommend that every studio tries to create its game backend in a way that can be reused in other games internally.

2. Replication is convenient but expensive

In general, information systems have two ways of communication: through shared memory and through messages. Replication is essentially the emulation of shared memory, and it is usually a one-way process. In the case of a game backend, this is when a change in some server state leads to an automatic change in the client state. And on the client, you can subscribe and react to these changes.

Replication is usually the primary way of transferring information during a game session. For example, it is built into Unreal Engine. Replication is sometimes also used for interaction between the game backend and the client, but there is no off-the-shelf solution, and each project implements this on its own. This is not an easy task and harbours many pitfalls, so I can't say for sure that it's worth it.

On the other hand, it is convenient to work with replication, and it eliminates the desynchronization of server and client states. This is especially helpful when server and client have been developed on different technologies, so the code can't be reused to apply server changes to the client. But I would not go there for a game with a not very complex state, especially if there is another way to unify server and client state updates.

3. C++ is incredibly fast

Everyone knows that C++ is used to develop performance-critical systems. Because of this, it is the primary language for developing game clients. But for other reasons, it is not so popular for server-side development.

I realized how fast it was when I worked on one game where the backend was an almost unscalable C++ monolith with most of the logic executed in one thread. This server was able to handle 70k CCU. I moved to this project from the Java world, and I was surprised to find that the millisecond precision of log timestamps configured in ELK was not enough for this server to sort events by time correctly.

The reason, of course, is not only the efficiency of the generated C++ machine code but also the ability to work with memory efficiently. So I recommend using C++ for your game backend if you have a team of professionals.

4. Monoliths are better than microservices

This is a provocative statement, but I'll try to clarify it. In the modern information technology world, a more or less generally accepted position is that only highly distributed systems can cope with a large load and achieve reliability, availability, and zero downtime system updates.

Looking at the top Steam games, you will see that only 10 have a peak CCU exceeding 100,000. According to my practice, on large numbers without ping messages, on average, one player makes less than 1 request to the server in 10 seconds. In this case, these 100,000 players generate not more than 10,000 RPS. Is it a lot or not? You can easily handle this load with a single server using binary network protocol and efficient database communication.

If you play games, sometimes you see announcements about scheduled maintenance when the game is unavailable. There are a few reasons for that. On the one hand, due to game specifics, it is difficult to do a zero-maintenance game backend. But on the other hand, most games are not critical services (some are, but most are not), and short game downtime does not lead to a noticeable user loss.

There is high entity connectivity in games and a large logic variability when developers try to create small elements which designers can combine into final mechanics on their own. This leads to the fact that developing a highly distributed system with independent parts adds significant overhead to development at all stages.

Considering the above, I recommend using a monolith for some parts of the game backend. It would be much cheaper to develop. And seamless backend updates can be organized in other ways, for example, through sharding, when a new game backend is launched for each new client version and players are routed to it. This approach requires additional development but is much less than spreading the logic over compact services. Also, you can support more than one monolith in shard to increase performance and availability.

5. Feature toggles would save you from sleepless nights

Using feature toggles is a common practice for mobile applications and websites, but sometimes, it is underestimated in game development. For those unfamiliar with feature toggles: it is the ability to dynamically enable and disable some particular features. For example, Roblox locks every new or modified line of code to feature flags. But outside such an extreme case, simply locking new features with feature flags allows you to test some features on production servers with a limited number of users or just on developers and switch off features with bugs. Also, integrating it with analytics would allow you to use it as an A/B testing solution.

6. A single authorization system would never be enough

Perhaps this is the most obvious observation. You should be ready to, at some point, change the authorization system completely (and usually billing along with it) or support several different systems simultaneously. For example, it can happen when you want to be on Steam and EGS simultaneously or when you decide to launch in China, where you would be required to support local game distribution systems. Also, don't forget crossplay between PC and consoles. At the beginning of the project, there is a temptation to use some implementation features of a specific system internally. You may later find out that Xbox authorization returns an alphabetic unique player identifier rather than digital.

7. The ability to collect thousands of metrics would help you a lot

Most likely, this observation is not specific to games. But the potential number of unique metrics in games is measured in hundreds. Thus, the complexity of the internal logic is such that the opportunity to understand what is happening in dynamics is appreciated.

By metrics, I mean numbers at a fixed rate in special storage that can be analyzed on different charts. An example is Prometheus/Graphite with Grafana. You usually improve your monitoring when the issues arise, but initially, it is worth configuring a system that allows you to do it.

I would recommend having a second-by-second accuracy for the last day, five seconds for the last month, a minute for the last year, and an hour for more than a year. And be ready to store 100k+ metrics because of the large number of metrics from game session servers.

8. Matchmaking is more challenging than it looks

Matchmaking is the backend service which matches players with teammates and enemies for a battle. Creating a good matchmaking service might be much more challenging than it first seems. Matchmaking problems usually do not appear with a large CCU. But with a standard life cycle, the game has maximum CCU shortly after the launch. A gradual decrease in the number of players begins after that. At the same time, the internal complexity and variety of mechanics and game modes begins to grow. In addition, depending on the specifics and the main game territories, the intraday CCU fluctuation is up to 10 to 1 (peak CCU is 10x of min CCU). Therefore, matchmaking problems will appear sometimes in some game modes.

I recommend starting with primitive matchmaking thanks to the large CCU at launch. But you should cover everything with metrics and analytics so when someone reports too long a queue or an unbalanced team, you can determine whether this situation is normal or an MM problem.

Thank you for reading. Of course, it is just a part of the game's backend-specific moments. But these seemed the most interesting personally. So please write in the comments what other moments you know that differ in game backend development. Also, stay tuned. I'm going to write another article with a detailed description of the architecture and approaches used in a real game backend.