DevOps in Game Development: Accelerate Production with Automated Workflows

Hi, my name is Konstantin Usachev, I'm a Software Engineer with over 15 years of experience behind my back, 12 of them – in game development. During these years, I've experienced firsthand the impact of DevOps on game production cycles.

In this article, I’ve assembled some practical insights on automated workflows in GameDev that helped me a lot in my career, and I hope will help you boost productivity and product quality on your projects. Below, I'll share strategies and tools that have proven their worth in automating and accelerating production. From managing complex assets with Git LFS to harnessing Jenkins for cross-platform builds, these are lessons learned from direct hands-on application. Keep in mind that the particular tools I use here in examples are not as important as the core ideas behind these approaches you can adopt.

So, if you’re a DevOps/Software Engineer working in GameDev or just preparing to get your hands dirty in this thrilling area – this article is for you!

DevOps Tools for Efficient Asset Management

Dealing with game assets in DevOps can test the limits of any system. If you’ve dealt with a modern game's assets, you know it: large files that make cloning a repository feel like a coffee break and push operations that are longer than some of your gaming sessions.

With files sometimes measuring in hundreds of megabytes for high-quality assets, the real issue arises when you try to fit these into a traditional CI pipeline. Git struggles with large binaries because it was optimized for text. Assets don’t compress well, and differencing tools are almost useless for binaries, ballooning your repositories and making efficient collaboration more of a wish than a reality. Binary asset management needs to handle the continuous updating and storing of files without the chokehold on your workflow.

To overcome the hefty challenge of asset management, we turn to specialized tools tailored for this purpose like Git Large File Storage that replaces large files with text pointers while storing the file contents on a remote server.

Below are a couple of simple-to-implement tricks that will help your workflows with Git LFS and a CI/CD pipeline.

Here is an example of using GitHub CI to checkout and build a Unity project:

name: Build
on: [push]
jobs:
  build:
    name: Build for ${{ matrix.targetPlatform }}
    runs-on: self-hosted
    strategy:
      fail-fast: false
      matrix:
        targetPlatform:
          - Android
    outputs:
      buildVersion: ${{ steps.build.outputs.buildVersion }}
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          # don't delete 'cache'
          clean: false
      - name: Git LFS Pull
        run: |
          git lfs pull
          git add .
          git reset --hard
      - name: Build Unity Project
        id: build
        uses: game-ci/unity-builder@v2
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
        with:
          projectPath: Game
          targetPlatform: ${{ matrix.targetPlatform }}
          androidAppBundle: false
          androidKeystoreName: publish.keystore
          androidKeystoreBase64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
          androidKeystorePass: ${{ secrets.ANDROID_KEYSTORE_PASS }}
          androidKeyaliasName: publish
          androidKeyaliasPass: ${{ secrets.ANDROID_KEYALIAS_PASS }}
          androidTargetSdkVersion: AndroidApiLevel31

Here, we automatically pull the repository and build an Android version of our game. We could store the result apk file and even send notifications via Slack.

For Docker, you might be interested in a multi-stage build process that can compile assets within a Docker environment and then use them in the final image, reducing the overall size and keeping build environments consistent. Here is an example Dockerfile where we use one Docker image to build some backend .NET application and another to form the final image:

# Container we use for final publish
FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
ARG PORT=8080
EXPOSE ${PORT}

# Build container
FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build

# Copy the code into the container
WORKDIR /../
COPY [".", "Application"]

# NuGet restore
WORKDIR "Application"
RUN dotnet restore Backend.sln
#COPY ["*", "Application"]

# Build the API
RUN dotnet build "Backend/Backend.csproj" -c Release -o /app/build

# Publish it
FROM build AS publish
RUN dotnet publish "Backend/Backend.csproj" -c Release -o /app/publish

# Make the final image for publishing
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Backend.dll"]

Automating Game Testing Workflows

With traditional software, you might be testing for functionality and performance, often in a predictable environment. In GameDev, however, you're also testing the unpredictable player interactions, some random generator-based mechanics, AI behaviors, and real-time rendering across various hardware. And you're not just looking to see if the game crashes; you're assessing whether it delivers the intended performance as well.

The key to automation in game testing lies in the ability not only to split complex game mechanics into separate pieces you can test but to replicate and evaluate complex player behaviors and scenarios. This means creating automated test suites that can not only press buttons but also make decisions as a player might.

At first, you start with unit tests, where you test small pieces of your game mechanics in an isolated way. Usually, games are too agile during the active development stage. So, it is not always feasible to unit test a big part without affecting development performance. But the most critical and stable mechanics might be covered.

Functional testing, which verifies complex game mechanics combinations, involves scripting AI players who can navigate the game world, engage with objects and NPCs, and execute sequences of actions that a human player would. Tools like Selenium or Unity's Test Framework can be programmed to perform complex in-game actions, simulating real gameplay.

In performance testing, when evaluating the game's stability and framerate in different scenarios, automation tools must go beyond the standard load tests. The popular practice is to have an automatic grid of cameras on each level in your game and a command to measure performance metrics from each of them. With this, it is trivial to start measuring performance in your CI pipeline to catch any performance degradation earlier.

You will likely also have to deal with compliance and localization testing – confirming the game adheres to platform-specific regulations and accurately translated. However, these issues fall beyond the scope of this article.

Mastering Cross-Platform Deployment in GameDev

Deploying games across multiple platforms is inherently complex. Each platform has its own requirements – different hardware, OSs, input methods, and even specific rules for store listings. What works effortlessly on a PC might be a hurdle on a console or a mobile device.

A key challenge is making sure that your game doesn't just run but also performs well and provides a consistent experience wherever it's played. Moreover, with games frequently updated or patched, integrating automated deployment into your DevOps process is essential.

Here’s how you can tackle these challenges effectively:

1. Unified Codebase: To manage code efficiently, use conditional compilation flags. This keeps your codebase unified, making updates straightforward. For instance, in Unity we can write:

#if UNITY_IOS
    // iOS-specific functionality
#elif UNITY_ANDROID
    // Android-specific functionality
#endif

and for an Unreal Engine project, we can use a very similar approach:

#if PLATFORM_IOS
    // iOS-specific functionality
#elif PLATFORM_ANDROID
    // Android-specific functionality
#endif

1. Advanced Automated Builds: Enhance your CI/CD to handle builds, tests, and deployments based on the outcomes of automated tests. Here’s a simple Jenkinsfile example that uses parallel stages and environment variables:

pipeline {
    agent any
    environment {
        PLATFORM = 'initialize based on trigger'
    }
    stages {
        stage('Build') {
            steps {
                script {
                    sh 'build.sh -platform $PLATFORM'
                }
            }
        }
        stage('Automated Tests') {
            parallel {
                stage('Unit Tests') {
                    steps { sh 'run_unit_tests.sh' }
                }
                stage('Integration Tests') {
                    steps { sh 'run_integration_tests.sh' }
                }
            }
        }
        stage('Deploy') {
            when {
                expression { return currentBuild.result == 'SUCCESS' }
            }
            steps {
                sh 'deploy.sh -platform $PLATFORM'
            }
        }
    }
    post {
        always {
            slackSend (message: "Build and Test Pipeline completed for $PLATFORM")
        }
    }
}

1. Docker for Consistent Environments: Docker can standardize your build environments, encapsulating SDKs and tools. Consider a multi-stage Docker build that ensures each phase of game production is reproducible:

FROM gcc:9.3 as builder
WORKDIR /src
COPY . .
RUN make all

FROM alpine:latest as tester
COPY --from=builder /src/game /game
RUN ./test_game

FROM alpine:latest
COPY --from=tester /game /usr/local/bin/game
CMD ["game"]

1. Emulators and Virtual Machines for Testing: Using Android Studio and Xcode, simulate gameplay on Android and iOS. For broader testing, VMWare or VirtualBox can mimic various operating systems, giving you a virtual test lab without physical hardware. This enables testing across potential player devices, ensuring that everyone has a great experience regardless of their platform. Conclusion

As we've explored, mastering DevOps in game development involves much more than streamlining processes – it's about enhancing the gaming experience across all platforms. By implementing the advanced strategies and tools discussed, from unified codebases and sophisticated CI/CD pipelines to Docker environments and emulator integrations, you ensure that your game not only meets but exceeds player expectations.

I hope that the strategies and practices covered in this article will give you some ideas on how to keep your development agile and your releases flawless. Automation allows you to face many challenges of efficient game development. Ready to transform your next game project? Let these strategies guide you to success and set new standards in the gaming world.