External code in Unreal Engine game source

Introduction

As many game developers know, the Git 100MB files size limit renders vanilla Git useless. For us, LFS did not seem like a viable option (we may post our full report regarding selecting version tracker later) and we ended up gong with Plastic SCM. So what we have now is:

• A game built on Unreal Engine. We are using the source version of the engine as we need to touch engine code.
• Full game and engine source as well as game data stored in a Plastic SCM repository.
• All cloud services, backend code and shared code stored in our GitHub repository.

To be able to work with faster iteration times outside the engine, use the Git-workflows and tool sets and potentially share code with other Goals services, we have introduced the "External Packages Workflow".

In a nutshell, it is a GitHub repository inside the Goals game source and it allows for a couple of workflows:

• Iterate rapidly on a library and its tests outside the engine source.
• Iterate on a library, committing to the GitHub repository, but while testing it as part of the game source.
• Iterate on a library when using it as part of another goals service.
• Allow packages to be both Bazel packages and Unreal source at the same time without interfering with each other.

For the remainder of this article, I will use the Goals game-ai library as an example. It could also be used for instance for shared Protobuf definitions files or something similar. The key is that you can put anything there that does not interfere with Unreal build system.

Defining which packages to use

Since there was no natural place to add a list of external packages that we needed to use, we introduced the Packages.goals file that lives in the root of the Game source. This is a text file with a pretty self explanatory syntax:

# Defines which external Goals packages should be added to the source.
# If the Commit hash/releaese is omitted, latests will be fetched.
# Format:
# [Package name] [Remote git url] [Commit hash/Release tag]

# Game AI
game-ai https://github.com/goalsgame/game-ai.git 242a3d0ed0ba1cc3668962c90911ec0c8197a360

Wrapping Setup.sh/bat

To achieve this without adding an extra thing you have to do each time you sync we introduced GoalsSetup.sh/bat. What GoalsSetup does is:

• Wrap the Setup.sh/bat files shipped with Unreal and only run it if needed.
• Check if there are any missing external code repositories missing (as defined by the Packages.goals file) and if there are, fetch or update.
• Check if one of the external repositories have been touched locally (i.e. git status --porcelain) and if it has, leave it alone.

So, as a coder, the only thing you have to do after you sync the main game repository is run GoalsSetup and everything should be dandy. It is our goal to not have any other dependencies that are needed on your machine apart from the ones dictated by working with unreal (which is essentially a compiler).

Hooking into Unreal build system

The easiest way to hook into unreal build system is to look at all external packages as one Packages module. This way, unreal build system can parse the source for the needed files and add them as dependencies for the game. So, create a new module by adding a build file in your source. For example Packages.Build.cs:

using UnrealBuildTool;

public class Packages : ModuleRules
{
    public Packages(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PrivateDependencyModuleNames.AddRange(new string[] {  });
    }
}

Now, in your game, add this module as a dependency. For example in Game.Build.cs:

using UnrealBuildTool;

public class Game : ModuleRules
{
    public Game(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PublicDependencyModuleNames.AddRange(new string[] { 
            ...
            "Packages",
        });

        ...

    }
}

Now, each time you build your game project, the source of the Packages module will be parsed and needed files added to the project. Here is what it could look like in your source:

Note that the external package game-ai has some Git and Bazel related files in it but that is fine. And now, for each package we add to the Packages.goals text file a new directory is created on the side of the game-ai directory above and its files included in the game build.

Left to do and caveats

We still need to be able to handle including more complex packages. This could be for instance packages with source files that should not be included (test files) or files that are disruptive to the Unreal build system. This could be handled in Packages.Build.cs above by excluding certain directories. Maybe ones the include the word "test" for example. See how this is done in for instance RuntimeMeshLoader.

If we want to include packages that depend on other packages we would have to make sure the include paths are the same and the libraries available but ideally, all the external packages are atomic and do not rely on other things.