DEV Community: GOALS Engineering

Unit testing private functionality in Unreal Engine C++ classes

Göran Syberg Falguera — Mon, 27 Mar 2023 07:24:48 +0000

Unreal Engine has a pretty extensible suit of automation features. For instance, we run unit tests in our continuous integration system upon merge of new code.

For these unit tests we utilize UE's IMPLEMENT_SIMPLE_AUTOMATION_TEST or potentially IMPLEMENT_COMPLEX_AUTOMATION_TEST. Here is a minimal test from Unreal's documentation:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FPlaceholderTest, "TestGroup.TestSubgroup.Placeholder Test", EAutomationTestFlags::EditorContext | EAutomationTestFlags::EngineFilter)

bool FPlaceholderTest::RunTest(const FString& Parameters)
{
    // Make the test pass by returning true, or fail by returning false.
    return true;
}

This is all well and good but pretty soon after implementing your first test you realize that you will want to reach private functionality in the class that you are testing.

After pondering different more or less ugly solutions I talked to my colleague Tomas Hübner and he showed me this old trick that does not make this pretty but at least not terrible. And after pre-compiler is done the class we are testing is actually unaffected.

We do not want to make this private functionality public just for testing and we do not want to pollute the tested class with for instance a friend class that the original class should know nothing about. So, here is the trick:

We add an empty macro UNIT_TEST_FRIEND to the class we want to test:

#pragma once

#include "CoreMinimal.h"

#ifndef UNIT_TEST_FRIEND
#define UNIT_TEST_FRIEND
#endif

class FMyTestedClass : public FCoolUnrealClass
{
public:
    FMyTestedClass();

private:
    float PrivateFloat;
    void PrivateMethod();

    UNIT_TEST_FRIEND;
};

And then in the test class, we make sure to define the UNIT_TEST_FRIEND macro. The test class from above then becomes:

#include "Misc/AutomationTest.h"

#if WITH_AUTOMATION_TESTS

#ifdef UNIT_TEST_FRIEND
#undef UNIT_TEST_FRIEND
#endif
#define UNIT_TEST_FRIEND friend class FPlaceholderTest

#include "MyTestedClass.h"

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FPlaceholderTest, "TestGroup.TestSubgroup.Placeholder Test", EAutomationTestFlags::EditorContext | EAutomationTestFlags::EngineFilter)

bool FPlaceholderTest::RunTest(const FString& Parameters)
{
    FMyTestedClass TestedClass;

    // Using private member here
    float MyValue = TestedClass.PrivateFloat;

    // Calling private function here
    TestedClass.PrivateMethod();

    return true;
}

Note how we include MyTestedClass.h after defining the macro! Otherwise the UNIT_TEST_FRIEND will be empty when we parse the MyTestedClass.h header.

Tada!

Developing games on and for Mac and Linux

Göran Syberg Falguera — Thu, 16 Feb 2023 08:31:30 +0000

The case for multi platform game development.

Introduction

In my current organization we have come to a point where we will have to test our belief in our previous choices. Up until now we have said “We will support Windows, Mac and Linux as long as it does not take up too much of our time”. Now things are heating up and we need to prioritize where we allocate our resources.

In this article I will drill into the different aspects of multi platform development and multi platform development environments for personal computers. I will try to convince you that there is a case for multi platform development and if you can afford it, and have the skills and perseverance to follow through, there is much to gain.

The below arguments all assume planned support for the two biggest console manufacturers on the market, Sony Playstation and Microsoft XBOX.

The costs

When developing an application for multiple platforms there are different aspects that will incur costs. One way to slice it is development, quality verification (testing) and distribution. Depending on the platform and choices made, they may all have equal impact on total cost.

Development

This is where extra costs arise from taking into consideration hardware and certification requirements from different vendors or platform owners.

As for pure feature development, much of this should be solved for us by the chosen game engine, in our case Unreal Engine. We did indeed choose Unreal Engine much because of its wide platform support. This is a truth with some modifications as the platforms are far from on par feature wise in the engine. Due to this, we often stumble on issues on one platform even though it worked on another platform. Up until now most differences have been related to rendering and video codecs. This is not surprising as it is key things that platform owners try to differentiate themselves on. I.e. Windows, XBOX -> DirectX, Linux -> Vulkan, Mac -> Metal, etc. and video codecs that are proprietary and sometimes hardware accelerated with special purpose on board circuits.

On top of pure technical development costs, most platforms have certification requirements that the developer needs to pass to be allowed to release the game on that platform. This could be certain ways menus are used, things that are not allowed to take too long, etc. The effect of this is that many features need special treatment and development for each platform. And this leads into the next section where we need to verify all these features.

Quality verification

Simply put, all versions and combinations of features of the game need to be tested. The more versions, the more testing and hardware is needed. With cloud services available and making automated testing a priority early on, we can minimize the testing effort but there is no getting around that each platform and configuration needs to be tested.

Worth noting is that PC-platforms are disproportionately expensive to test due to the number of hardware configurations. As an example, the Playstation 4 only exists in a couple of different hardware configurations so testing them all becomes possible. A PC could be put together with any compatible hardware making the permutations that need to be tested almost infinite.

Distribution

This one is pretty straightforward. It’s the costs related to getting the game into the hands of the players. Every platform has its own quirks and associated cost (for instance, the cost of generating a code for a digital download, etc.) but generally speaking, the biggest cost is the licensing fee. I.e. How big of a cut does Steam take on PC or Sony on Playstation. This cost does not increase with the number of platforms but it may vary depending on which platforms are chosen.

The business case

Looking at mostly game development in this article, the interesting data point is what is commonly referred to as install base. It is a simple to understand metric that means essentially: “If you build a game for this platform, how many potential customers do you have on that platform”. As we can see in the diagram, Linux accounts for only 0.5% of the potential install base and Mac only 1%. If it goes really well and the game reaches a really wide audience, say 100 million players, then that means potentially 500 thousand and 1 million players respectively for Linux and Mac.

There is an argument to be made for different kinds of games and different audiences. A game that runs on cheaper hardware with less resources and is free to play could potentially have a larger install base on for instance Linux that is a free operating system. This is, however, only speculation.

So, is the case closed? There is no point building for Linux and Mac? Not quite, read on.

The technical reasons

Compiler

If we group the platforms in a slightly different way, namely on compiler, we see that MSVC C++ compiler used by the Microsoft platforms and Clang C++ compiler used by all the others are pretty equal. Taking into account the dedicated servers that all run on Linux regardless of client platform, we can consider them roughly equal in terms of numbers. This is not interesting because it gives us any relief in development costs but it is interesting because our game code always needs to compile on MSVC and Clang C++ compilers regardless of if we chose to support Mac or Linux. So additional platforms do not add much cost in this regard.

Rendering API

Although the compiler differs depending on which platform you are compiling for and on, much of the technical complexity should be abstracted by Unreal Engine. That is, before Unreal does a release, they make sure everything is working on the supported platforms. So in theory, when we take that release, everything should be working. It is however very hard for Epic (who builds and maintains Unreal Engine) to have test cases for every feature in all rendering API’s. Our experience from doing one major, 3 minor and 2 hot-fix upgrades on the game while in development and while being live, tells us that a lot of problems arise from differences in the rendering API’s and artifacts/errors due to that. What this means is that if the game is working nicely, and we do an engine upgrade, there is no guarantee that the game will run on all platforms after the upgrade if they did it before the upgrade. The more rendering API’s we support, the more special treatment we need for each platform. This is where it gets really bad for Mac and Linux. Since DirectX is a proprietary rendering API for Microsoft platforms and Apple stubbornly wants their own (Metal), Linux needs to use the open (and still fairly young) rendering API Vulkan. We end up with the following graph if we look at it from a Rendering API perspective:

Since we technically do not need rendering on the game servers (that run on Linux) and Mac is just a small percentage of the install base, it becomes difficult to motivate the increased development effort to maintain the game on Mac and Linux.

The practical implications

Currently, 15% of our studio uses Mac, 40% of the game engineers prefer Linux. Among game engineers it is more common with Linux and among cloud engineering, art and other departments Mac is more popular. What this means is that if we do not support Mac, a significant part of the studio will need other computers/consoles to play the game if we decide not to support Mac. A large percentage of the game engineers will have to develop and/or test the game on another platform than the one they feel more comfortable with.

I’ve worked at Windows only game development studios that only supported Windows PC clients but ran all servers on Linux. What ended up happening in a studio with several hundred developers was that one developer used Linux to natively debug and profile the server build effectively, many others used remote debugging from their Windows machine to track down specific issues. What happened in practice was that the Linux build, vis-a-vie the server build, was neglected due to the hassle to even run it. The low hanging fruit can usually be fixed and optimized on a Windows version of the build but if you want to get really fast or really stable (finding those intermittent bugs) then you need to be natively working on the platform.

Another aspect that is often overlooked is the importance of the engineering team learning early to consider multiple platforms. As mentioned above, automatic testing becomes important right from the start since you do not have time to test every platform manually. The consequence of this is that the organization makes the multi-platform investments needed eventually directly and can start leveraging benefits right away.

Technically, when forced to consider multiple platforms, engineers learn early that code needs to be modular and have minimal dependencies. The modularity is due to being able to make platform specific implementations and minimal dependencies is due to external dependencies seldom supporting all the platforms you need them to. Both of the above will require more senior developers to pull off.

Social responsibility and employer branding

A well established practice in game development, and software development in general, is exclusive content. What this means is that platform owners ensure, through any means possible, that some game, applications or content is available only on their platform.

After a while, consumers learn that their desired content is available on a specific platform and that platform grows. Next step is that some game studios are doing exactly the above calculation and realize the install base is too small on some platforms so they do not develop their game for it. The circle of platform lock-in is now complete when the platform owners do not have to pay for exclusive content anymore since the other platforms are too small to motivate developing games for them. Since we cannot see the future, this becomes a chicken and egg problem. Are the Mac and Linux install bases so small due to missing content or are they missing content because the install base is so small?

The socially responsible thing to do, all other things equal, is to develop the game to be accessible to as many people as possible. Making the game accessible to for instance color blind people (~4%) is something that we take for granted, but making the game accessible to Linux and Mac players (~1.5%) is seldom talked about in relation to accessibility.

Making a game at the level of ambition that GOALS is trying to achieve, takes a lot of good developers. Finding these developers is not easy but one good way of attracting them may be letting them work on their preferred operating system. According to Stack Overflow developer survey of 2022, the distribution between the operating systems for professionals is a bit different than the one for gamers:

State of the art

If you decide to go down this road there are some fantastic tools, applications and libraries who's makers share the vision of a multi platform gaming ecosystem:

Game engine

Unreal Engine: https://www.unrealengine.com/. Open Source and multi platform game engine.

Version tracking

Plastic SCM: https://www.plasticscm.com/. Not only multi platform but specifically built for the needs of a game studio.

Integrated Development Environment

Rider: https://www.jetbrains.com/rider/. An IDE with a large ecosystem of plugins and specific Unreal Engine support.

Engine binary distribution

Gamecure: https://github.com/goalsgame/gamecure. Our own cross platform game engine distribution application that we open sourced. Read more here: https://dev.to/goals/collaboration-on-large-teams-with-gamecure-1dfb.

Engine upgrades

UEIMPORTER: https://github.com/goalsgame/ueimporter. Open sourced and in house built tool set for upgrading Unreal Engine when under Plastic SCM version control.

Code generation and network serialization

Protocol Buffers: https://developers.google.com/protocol-buffers

Delta patching

golongtail: https://github.com/DanEngelbrecht/golongtail
Cross platform library for delta patching. Used by our end user game launcher and Gamecure (mentioned above)

Summary

On the surface, it looks like an easy decision. The numbers just don't add up. Drilling a bit deeper, we realize there is much to gain from sticking to multiple platforms right from the start and having come such a long way as GOALS, and having all infrastructure in place, my conclusion is that there is no question what is better and what is desirable, the question is what the organization can afford and what the people in the organization can manage.

References

Title image credit: Midjourney

Ball physics in GOALS

Göran Syberg Falguera — Mon, 06 Feb 2023 09:35:21 +0000

Title image credit: Midjourney

Introduction

When creating the ball for GOALS we realized fairly quickly that using Unreal’s off-the-shelf physics will not suffice for our ball. Footballs behave in peculiar ways when forces are applied and when they are deformed. We want a really configurable and realistic looking ball trajectory. But at the same time, we want to be able to leverage Unreal's built in physics for collisions and interactions with the rest of the world. The current solution uses different approaches depending on which state the ball is in and we will describe each of them in this article.

The different states of the ball

The ball will be in six distinct states and different calculations are applied in each of them. Starting with ball being possessed by one of the players:

Dribbling - One of the players (or the AI) is performing dribbling animations that effect the ball.
In air predicted - One of the players is about to shoot and all parameters are known.
In air simulated - The ball now leaves the player and it's parameters are calculated each game update in our custom ball physics simulation.
On ground simulated - As soon as the ball hits something, like the frame of the goals, the ground or another player, we let Unreal physics take over and manage the ball according to it's physical properties like weight, material, size, etc.
On ground predicted - For players and AI to be able to intercept the ball, without it looking strange, we need to predict the ball path.
Once ball is at rest, it is just considered a plain old static mesh with physics in Unreal.

Dribbling

This is when one of the players is possessing the ball. In this state, the ball movement is dictated by designer controlled parameters that gives the ball a physical impulse depending on the stats of the player. This way we can vary the control the player has with the impulse. As we progress to more advanced dribbling, we may have to add more sophisticated ball control mechanisms.

In air predicted

To be able to put other players in the right state, i.e. getting ready for a volley shot etc. but also to adjust aiming, we need to know where the ball will end up (if it does not get intercepted on the way).

We do this by emulating Unreal Chaos physics and create a path from where the ball is to where it would land unobstructed. This is all done in the frame (game loop update) where a player lets go of one of the shooting buttons. We now know which type of shot the players used as well as the power and the aim, we also know how much we need to adjust the shooting angle for the player to hit the intended target.

All this is now plugged into the actual shooting function.

In air simulated

Once ball is let go, at each game update we considers several forces that impact the trajectory; speed, air drag, spin force, gravity to create a natural feeling during game play. But we also make sure that the ball in this state is regular AStaticMeshActor so that we can have it collide with and bounce of things in the world without having to implement this ourselves. We have built a set of development tools for engineers and designers to see exactly how the ball behaves with different parameters set. In this image we see the spin of the ball illustrated on the ball bath as well as spin force, spin speed, drag and ball speed plotted on screen.

High quality ball simulation is accomplished by updating physics at a fixed 128 Hz rate on the server. To minimize client-divergences we use sub-stepping, a technique to decouple physics simulation rate from frame rate, so that even players with hardware that fail to render the game at 128 frames per second will update physics in roughly the same rate as the server.

On ground simulated

From our perspective, this is the simplest case as we let Unreal engine physics handle most of it for us. We will get physical properties while the ball interacts with the worlds for free and we can leverage Unreal material properties for friction etc. The thing we apply ourselves is drag force that is also applied when in air. We do this with a manually adjusted curve based on research on ball behavior.

In this short GIF we can see how the ball is sliding slightly before friction is enough for it to start rolling. Then after a while the ball comes to rest.

On ground predicted

To be able to steer AI and different lock-on scenarios, we need to be able to predict where the ball is going. Otherwise the players will just keep updating where they are going and create the strange round path shown in the image below.

The trick in this scenario, is figuring out what Unreal physics will be doing to the ball (remember that when we are in air we are controlling physics so we can more easily predict it). Up until now, we have used a simple constant deceleration to model the behavior but as we increase the quality this is not enough anymore. We are currently testing different models and will update this section once implemented.

Configuration and shot types

The system is built to be highly configurable and extendable by the game designers and adding new shot types is just a matter of configuration and adjusting some curves. Here is an example of a curved shot. The amount of curvature does not only depend on the input of the controller but also on the stats of the current player:

Networking the ball

The ball is simulated on the server as well as on the client. To ensure fair play, the server is authoritative and any client divergence will be smoothly corrected.

At the time of writing, we use off-the-shelf Unreal movement component to network the kinematics of the ball. One big task for GOALS is to find a way to make this component more streamlined and optimized for GOALS use case as networking the ball takes up most of our network resources.

Runtime Procedural Character Generation

Göran Syberg Falguera — Wed, 11 May 2022 13:22:15 +0000

Or storing 18 quintillion characters in a 64 bit integer.

Introduction

The purpose of this article is to give a high level survey of a field that I believe will become increasingly important in the future. I aim to not only describe the area but try to spark your imagination on things that can and will be done with this technology.

Increasing numbers of game developers are using procedural tools to create content for their games. A lot has been done for procedural generation of environments, levels, game modes, etc. and even though procedural character generation has been done, it has not been as popular a use case as for instance generation of thousands of game levels for replayability or for generating vibrant environments. The reason for the relative lack of progress in this area I attribute to game designs not needing it. For one thing, it turns out people get more attached to characters that they get to create themselves than having them generated. Even for NPC-creation, the created characters usually have a place in a story, and any arbitrary character wont be good enough. Maybe a specific gender or geographical inheritance is needed for the story.

There is however a group of games that will not only gain from this technology but requires it and I am working on one of them. After reading this, I think you will have the foundation to not only accept my proposed use cases but maybe think of a few new ones yourself. Maybe you want to come work on this at GOALS?

Asset Generation

Asset generation, as opposed to asset creation, alludes to some kind of automated process. The point, of course, being that you can create a lot more content faster. Now, you could go about this in a couple of different ways depending on what you need. For instance, you could generate characters based random inputs and get random outputs each time. This would be referred to as stochastic generation of content. Or you could have a set of parameters that you have decided on and generate say 100 different characters based on 100 different predefined values of characteristic X. Now, in computer science "randomness" is a tricky thing. For "true" randomness you have to look at some non-deterministic physical aspect outside of the computer, for instance the radioactive decay of an atom. This is a bit overkill for our current topic and most programming languages supply some kind of pseudo random number generator. The funny thing with these "random number generators" is that they are fully determined by their "seed" and hence not "random" at all. Or, you could generate the same random number every time you want to.

A seed is something you pass into the generator function to make it differ from the previous time you ran the function. The reason we need a seed is that a computer cant make stuff up. They are built specifically to not be random and to reproduce the exact same result given the same input.

So, if you pass the same seed into the same random number generator, you end up with the exact same "random" number every time you do it. A "pseudo random number generator" is actually a "procedural number generator"! How fun. More about this in the "Procedural Content" section. The point to take with you from this section is that the set of generated outcomes can be done in a deterministic, and hence reproducible way (same result every time), or in a stochastic unique (most often) way.

Now, if the seed is a 64 bit unsigned integer (unsigned means only positive numbers), which is the most common integer size in modern computers, we could then generate about 18 quintillion (18,446,744,073,709,551,615) different random numbers. And if a character, or other procedural content, was fully generated from this seed, it would mean we would have 18 quintillion different and fully deterministic (we could recreate them anytime we want) versions of that asset.

This is where my head exploded the first time I thought about this. The possibilities are just staggering. Keep this in mind for now and let's talk a bit about how these procedural algorithms could work.

Procedural creation of content

Imagine a game character in a world with screens only supporting 12x12 pixels:

To create this character in the traditional (manual) way, we would have to manually chose the color of each pixel. This gives us a lot of control but if we want to create another character, we would have to do it again. Hence, this process scales really badly.

What if we would instead of choosing the color of each pixel create a little program that did something like this for the head of the character:

Aligned to the top of the center of the body, draw a symmetrical elliptic shape with the parameters X and Y while X and Y vary between Z and I.

Obviously the "body" would have to be similarly defined ahead of this but by varying X and Y between Z and I we can create as much variation as the resolution permits.

If we specify the values we want to vary in a list, like for the colors in the image below, we can have quite a lot of control of the process. Or we just vary the RGB-values pseudo randomly and get some unexpected color. Varying randomly can be risky though and may require more rigorous testing.

Game Characters

Now its time to talk about some 3D game characters and what they are made up of so that we can figure out how to procedurally create them.

Game characters are generally made up of the skeleton, the model mesh and the skin and textures. Arguably cloth could be seen as a fourth part in modern game engines. Traditionally cloth and clothing was part of the model mesh.

Skeleton or rig

The skeleton in game characters is not used to keep the body upright (as in real life characters), these will not fall in a pile of skin without it. The reason to have a skeleton is to be able to reason about the movement of the limbs and the body in relation to each other.

The definition of a skeleton is commonly just a list or hierarchy of bones starting at the hip and going outwards to the extremities.

To manipulate a skeleton, given the same number of joints (joints being the points tying the whole thing together) one might just vary the bone length according to some rules. As long as we do it within certain limits, this fits perfect for our procedural tools! Now we can just set random variation on bone length similar to pixel characters head above.

It is conceivable to create more arbitrary skeleton configurations and it has been done in games to create wonderful variation of creatures. As you can imagine, animation and testing of these characters can be a daunting task. For our use case, a humanoid configuration will suffice. Maybe we have different limits on male and female skeletons etc.

The model mesh

Attached to the bone is what is arguably the most complex part of the character. The model or the mesh is what makes up the shape of the character. It is made up by a number of vertices, or points in space. Between these points we draw lines called edges and together they crate surfaces or faces. In the image below we have a mesh that is made up of quads, meaning four vertices, and you need at least 3 vertices to create a surface. (That is why you often hear talk about triangles when people talk about graphics in games).

To manipulate these meshes, you simply move one of the vertices. This is what applications like 3ds Max, Maya and Blender excel at. To do this in a procedural context and randomly move vertices around probably would not make sense as the outcome is very unpredictable. One could instead imagine we create two extreme characters, one light and one heavy. The same for big/small noses, etc. and then vary some input parameters to create variations between those extreme values. It is easy to imagine a classical character creator in a RPG with sliders being the randomized values.

Skin and textures

Now after creating our model mesh, we need some color on this gray clay model. And that is what skin and textures are for. We could make our lives easy and create something like the blue man group and then vary the colors just like on pixel character above. But we probably want something more sophisticated like this character made by an old colleague of mine Björn Arvidsson:

Looking at a character like this it can be a bit daunting to think about how we could possibly generate something looking so realistic but luckily for us, thanks to other businesses, procedural textures and specifically for skin, has gotten pretty far. Adding more variations for scars with some shader programming or tattoos with decals, we can create endless variation here as well.

Tools already available

There are other tools on the market to manipulate and create characters. Below is an example from Epic's MetaHuman. This is not a procedural tool per se but it would be trivial to provide MetaHuman with a bunch of random variables as input and create a lot of characters. The problem with tools like MetaHuman is that its a cloud service and we have no idea how much resources and assets go into creating those characters. Also, if you export these characters to disk they are pretty big and creating/storing millions of them is just not practical. But this takes us to the first word in the title of the article.

Runtime generation and the GOALS use case

At GOALS we need characters that can scale up and down since we are targeting both high and low end machines, they cannot be on disk as they don't exist (have not been born) when we ship and we cannot use any manual labor to create them, there are simply too many of them. Also, back to the MetaHuman example: Even if we were able to store them in the cloud, it is not a nice user experience to wait while we download them before you can play.

Enter runtime character generation. All we have done today needs to be done at runtime. Meaning while the game is starting or running. This is sometimes referred to as online procedural generation vs offline.

The idea is that when you boot the game, you get a list of seeds belonging to you. For each of them we pass it into our procedural box and create skeletons, meshes and textures that makes sense for Unreal engine in the GOALS use case. Then, to update with new cool features and looks etc. we just have to update the procedural algorithms. Tada!

Future use cases

Considering what has been mentioned above, imagine if someone would create engine adapters for multiple engines. Then you could in theory, provided you have the right to use a particular seed, use it as you pass between engines. This is the holy grail of the metaverse.

If this sounds exiting, I would recommend you come work on it at GOALS and share your work with the world!

Working with Unreal Engine source releases in Plastic SCM

Andreas Andersson — Thu, 31 Mar 2022 08:27:27 +0000

This guide is for Plastic SCM users, that want to build a game using Unreal Engine, and plan to compile and make changes to the engine code itself, while also upgrading engine releases as they are released by Epic.

Hi, I'm Andreas, a game developer here at GOALS. We are building a football game using Unreal Engine and use Plastic to manage our source code and assets. This article tells the story on how we bridged the gap between Git and Plastic to stay on top of new engine releases.

Building blocks

Full engine source code is served via Unreal Engine own GitHub repository, to which you get access by registering your GitHub user with Epic. See How do I access Unreal Engine 4 C++ source code via GitHub?

Each official engine release is labelled with a Git tag, for example 4.27.2-release or 5.0.0-preview-2. Here you also find a downloadable tarball or zip for the release.

Then we have a Plastic repository, where we want to develop our game.

The idea

The main idea is to see Unreal Engine as a third-party vendor library, albeit a really big one. One strategy for vendor libs is to keep a clean and unmodified copy of it in a separate branch, that is then merged into your development branch.

This idea isn't new, it's basically exactly what Karl Fogel describe in Tracking Third-Party Sources (Vendor Branches), a section of his 20+ years old book titled Open Source Development With CVS.

Sounds simple enough, but how do we apply it to Plastic and Unreal Engine?

Branch layout and upgrade flow

Let us start with deciding a branch layout. The game itself is developed on main, we keep unmodified Unreal Engine releases in vendor-unreal-engine that is merged down to main for each release.

Take this example, where we set up an empty repo with 4.27.0 that we upgrade to 4.27.1 and finally 4.27.2.

This image speaks a thousand words, in text it says:

Start with an empty Plastic repo
Add 4.27.0 to vendor-unreal-engine
Make a small tweak to the engine itself.
Merge it all down into main
Add your games main module
Upgrade vendor-unreal-engine with release 4.27.1
Merge the new release with main, and apply fixes to make your game module compile and work again.
Merge the new engine into main
Make local modification to the engine itself
Upgrade vendor-unreal-engine with release 4.27.2
Merge it with main, and apply even more fixes to your game so that it still works.
Publish the new release to main
And continue making local changes to the engine

Importing Unreal Engine releases

So, how do you import or upgrade the engine source code into our vendor-unreal-engine?

Import by delete and re-adding all files

One crude strategy is to delete all files on the vendor branch and simply copy all files from the new release, and then let Plastic detect which files have been added, removed, modified or moved.

This should work well. Alas, plastics move detection seem to miss most moves, maybe there are too many files involved in engine upgrades for it to be practically possible? It would be understandable, upgrading 4.27.2 to 5.0.0-early-access-1 modifies over 50k files.

As a result, moved files will be imported as a delete followed by an add.
If you have made changes to the file in the old location on your main-branch, Plastic will not help you merge these changes into the file in its new location. Instead, it will ask you how to resolve your changes to the old, deleted file. Forcing you to manually copy your changes into the new location.

Depending on how widespread your local engine changes are, this strategy might be good enough, for us at GOALS it was not.

Import by replicating changes from Git

Luckily, we can do better. Full revision history is available in the main Git repo, it knows which files has been modified, added or removed, and most importantly it also knows which files have been renamed or moved.

The command we use is git diff --name-status, here's the output of the diff between 4.27.1 and 4.27.2. Note that this is just an excerpt, the real diff contains roughly 460 changes.

$ git diff --name-status 4.27.1-release 4.27.2-release
M   Engine/Build/Build.version
A   Engine/Extras/Containers/Dockerfiles/linux/dev-slim/Dockerfile
D   Engine/Source/Programs/Enterprise/Datasmith/DatasmithSolidworksExporter/Private/Animations/AnimationExtractor.cs
R064    Samples/PixelStreaming/WebServers/SignallingWebServer/platform_scripts/cmd/run.bat  Samples/PixelStreaming/WebServers/SignallingWebServer/platform_scripts/cmd/run_local.bat

The leading column means:

M - File was modified in place
A - File was added
D - File was deleted
R* - File was renamed or moved. The number is a percentage of how certain Git is that the file was in fact moved, and not a delete followed by an add. There is some grey area when it comes to moves in Git, sometimes a file is moved, but then modified to fit in its new location. For example, a moved C++ file may need to have include paths tweaked to compile.

Git uses some fuzzy heuristics to discern moves from adds and deletes. Most of the time it seems to make good guesses. When it fails it is not a big deal; the old location will still be deleted, and the name, location and content of the added file will be correct.

Now, it's just a matter of replicating these changes in vendor-unreal-engine.

Dealing with modified files is simple, just check out in Plastic and copy the file from the new release.

Adding files is almost as easy, just copy their content. But, if the target folder does not exist in Plastic we need to first create and add it before coping the file.

For deletes, the opposite of adds, we start by deleting the file itself.
If the folder in which it existed became empty, we need to remove the folder, we also need to iterate up the hierarchy to delete any now empty parent directories.

For moves we start by creating and adding target directories to before we tell Plastic to move the file. Finally, we copy the contents of the file from the new release.

Announcing ueimporter

This git->plastic import process is very scriptable, and I wrote a command line tool called ueimporter that takes care of it all. I'm happy to announce that GOALS are now open-sourcing this tool, available in GitHub at goalsgame/ueimporter.

Ignoring the elephant in the room

If you have ever tried to store Unreal Engine source inside Plastic, you may have noticed a rather big elephant that I so far avoided; Plastics ignore file, and it's incompatibility with Gits equivalent.

Unreal Engines GitHub repo comes with a rather complex .gitignore file. Whenever you build or work with the engine various intermediate and temporary files is scattered all over your workspace, not to mention the thousands of files that is downloaded when you run Setup.bat|sh.
These files should not be committed into Git, and likewise we do not want them checked into Plastic.

It is not possible to directly translate Gits .gitignore file into Plastics ignore.conf, the two systems have rather different rules deciding in what order ignore patterns are applied.

Ignoring in Git

On one hand, we have Git, where each line specifies a file or directory pattern, any subsequent matching line will override preceding matches. A simple philosophy that is relatively easy to understand.

If we compress Unreals .gitignore into a nutshell, it can be described like this:

Start by ignoring all files
Add exceptions to not ignore certain extensions. For instance, .h and .cpp files.
Add exceptions to those exceptions so that temporary build folders are ignored. For example, everything under Engine/Intermediate should be ignored, or else the *.h and *.cpp files that UnrealHeaderTool generates during the build process would be tracked.

On and on the list goes, with more detailed exceptions and ignore patterns. There are close to 160 rules listed in the ignore file for the 5.0.0-preview-2 release.

Ignorance is not a bliss in Plastic

Then we have Plastic, it prioritizes patterns based on its type, rather than in what order it occur in its ignore.conf. Two patterns of the same type are applied in the order they appear in the file. Exception patterns take precedence over ignore patterns of the same type.

So what pattern types are we talking about? Quoting the Pattern evaluation hierarchy section of the Version Control, DevOps and Agile Development with Plastic SCM book.

Plastic SCM will try to match the path of an item using the patterns in the file in a predefined way.
This means that some pattern formats take precedence over others rather than processing the patterns
in the order they appear in the file.

Absolute path rules that match exactly

Catch-all rules

Name rules applied to the current item

Absolute path rules applied to the item directory structure

Name rules applied to the item directory structure

Extension rules

Wildcard and Regular expression rules

There are more devils in the details, but that's the gist of it.

Unreals .gitignore use most of these types, but relies on the order to accomplish desired behaviour. Just copy pasting this to ignore.conf does not work, because it wreaks havoc to this order.

An acceptable workaround

We at GOALS have wrestled quite a bit with our ignore.conf, trying to come up with equivalent behaviour as in Git. So far, we haven't nailed it, but have at least arrived at a config file that we can endure.

We simply ignore the entire Engine-folder, that gets rid of most of the intermediate and temporary files that is explicitly ignored in .gitignore.
The main drawback with this is that we must remember to manually add files to Plastic, whenever we add anything to Engine, or else it will not show up as a pending change that can be checked in. We can edit files that are already checked into Plastic just fine, they will be detected as changed.

For our games own modules and plugins it was relatively easy to write ignore rules, mainly because Unreal write most files to Engine during the setup process and build artefacts all end up in easily identified intermediate folders.

Ignore files and our vendor branch

Thankfully, the ignore file is irrelevant on our vendor-unreal-engine branch.
Here we always want Plastic to detect all files, so that we can check them in and later have them merged into main. This implies that you must clear out any private files before you start importing a new engine release to the vendor branch. You should not build or do anything to pollute your workspace here, do that on an upgrade branch after merging with main (where an ignore file is present).

Parting words

With this setup you have the power to change the engine at will and still stay up to date with new releases. How you wield this power is up to you.

Consider that any merge conflicts you get with new engine releases, after making local changes, will need to be resolved. This is a manual process that is hard to automate.

In the past, when working in another big game engine, I have seen many, many, many dev-months been sunk into resolving merge conflicts and follow up issues, due to local modifications when the engine was upgraded. Tears were shed, good night sleeps lost and dev-happiness fled down the drain. It was not pretty.

Keep your engine changes small and isolated, and tag changed lines with begin/end comments. If a change can be done in your game module or a plug-in that is the preferred way.

One benefit with this setup, is that you can cherry-pick fixes from Epics mainline and push directly into your own main. Later, when the fix gets included in an official release your divergence should just resolve itself in the upgrade process.

Finally, upgrade the engine often, the further you diverge from the mainline the harder it will be to catch up. Take one version at a time, even if you are more than one version behind. In their Fish Slapping Dance Monty Python teach us that it's better to be slapped with a small pilchard multiple times than it is to be slapped by a big fat halibut just once.

Take care, stay safe and happy game making.

Collaboration on large teams with GAMECURE

Göran Syberg Falguera — Thu, 31 Mar 2022 08:09:44 +0000

Introducing GAMECURE! An open source Unreal Editor build distribution system for Windows, Linux and Mac, utilizing Dan Engelbrechts eminent longtail delta patching and with a bunch of other nifty features to get going quickly with your source built Unreal Engine in Plastic SCM.

Yes, that introduction is a mouth full so let me break it down for you.

Most of my career I've spent developing games at EA DICE. This is what would be considered a large studio (not the largest by any means) but we had projects with hundreds and sometimes thousands of individuals contributing to game code and content. All these individuals can contribute up to thousands of changes to the repository every day. The trick to being this fast is putting a wall between the content creators (artists, game designers, 3d-modelers, animators, etc.) that produce content and coders that improve, fix and many times break the game.

The coders will have a source version of the game engine on their computers. To play the game or use the game editor the coders need to build (compile, link, etc.) the whole engine locally on their computers. The custom computers used by GOALS coders spend 12-13 minutes on this but a regular gaming computer will spend hours on it! This does not only take a lot of time but it distracts you, is error prone and complicated. As a content creator you just want to create amazing content, no hassle!

What we need is some kind of tool that can act as the central tool for all GOALS development if you are not a coder, managing the other tools you need and allows you to easily feedback to engineers what is working and not working as well as quickly stepping back and forth between Unreal Editor versions.

Another big studio/project is Epic Games who produce the game Fortnite. They also happen to be the owners and maintainers of Unreal Engine. There is a great article explaining their workflow and reasons behind it called: Workflow on Fortnite - Collaboration on large teams with UnrealGameSync. And this takes us to an application called "UnrealGameSync" which is essentially what we want! A separation of coder and content creator workflow and concerns. Perfect! But... UnrealGameSync only works with Perforce version tracker (which is not really intended to be replaced) and its downloading full binaries each time (no delta patching) and it only works on Windows sigh

For a studio, like ours, that is cloud native and allow their developers to work on Windows, Linux or Mac and uses Plastic SCM for version control, that is three deal breakers right away. It was deemed to much for us to work around so we decided to create our own tool - GAMECURE!

One of the core development philosophies at GOALS is simplicity. And for us that usually means as few dependencies and as few hoops to jump through as possible. The maximum amount of complexity for a content creator at GOALS setting up their environment, we decided should be: Get the simplified version of Plastic SCM (called Gluon) and get GAMECURE from a permanent link. The rest would be automagic. Hence the GAMECURE application could not have a bunch of external dependencies and a stack made up .NET 6 and AvaloniaUI that we compile to standalone binaries turned out to be a good combo.

Another important requirement for a cloud native company as GOALS is that we don't have to download a bunch of unnecessary stuff. Cloud native meaning that essentially all our infrastructure is in the cloud. All downloads we do internally and that other people do externally from our CDN's incurs costs. Minimizing the amount downloaded is paramount and if you ask me, is just plain better engineering. For this we turned to a library called longtail by Dan Engelbrechts. This allows us to only distribute what has actually changed to the content creators.

What GAMECURE will do for you

Set up your Plastic SCM workspace with a predefined configuration so it only downloads what is needed to work with the pre-built editor.
Setup and download Unreal dependencies.
Single sign on to our cloud so you don't have to authenticate each time you want to update.
Delta-downloads of the editor to decrease the time it takes to update your editor.
Notifications when there's a new version available on the main branch.
Single click to create a bug report for the editor in Jira.
Cross platform packages, .app bundle for Mac, MSI installer for windows and a zip for Linux with a bundled config that is put together at build time.

A typical workflow

A content creator may need new editor functionality or a new C++ blueprint class for instance.
The coder takes the specifications and writes the feature in code. Once tested and reviewed, she pushes the changes to Plastic SCM and kicks of the CI.
The coder can now ping the content creator and say "i pushed the changes you need in changeset xxx"
The content creator will get a notification in GAMECURE saying there is a new version available and can verify that the particular changeset is in there.
The content creator now hit download on the new editor and usually in just a few seconds (due to the delta patching) the new version is on disk.
The new editor can now be launched with a button press from GAMECURE.

Some technical details

.NET 6 and AvaloniaUI for cross platform GUI application
WIX installer for an easy to use MSI on Windows
Custom BuildTool built in .NET 6 to remove the need for multiple scripts for different platforms. This tool is used to generate a config and build the different packages.
GUI is separated from the core functionality so that a CLI(or any other UI) can easily be added.

Future work:

Auto-updating when there's a new version of GAMECURE
Better Jira integration, instead of opening a browser it should have a dialog inside GAMECURE and attach any log files or screenshots
Track the current changeset, so that the user doesn't download an editor that is very different from the changeset they are on (this is currently a bit hard and undefined in plastic)
PKCE implementation for authentication/authorization with GCloud to remove the need for a Client Secret.
Progress bars where it's possible, for example download the editor, copy the editor etc.
Add support for other version trackers than Plastic SCM.
Add support for other clouds than Google.

We welcome contributions and we are curious to know how you solved your workflow challenges!

Head over to our GitHub to get started with your GAMECURE - workflow!

Input lag, what's going on?

Jens Andersson — Wed, 30 Mar 2022 12:24:40 +0000

Input lag, input latency, bad servers, and low tick rate are some of the phrases used to describe unresponsive gameplay. Let’s break it down into smaller parts and talk about where and why this happens, and what we can do to improve the experience.

At GOALS we want to create the best player experience possible, and we know there are no shortcuts to achieving that. GOALS should feel good and responsive to play; there shouldn’t be any delay when you press a button. To make this happen we’re going to build tools that help us measure the delay early on in the development process.

The tests in this article were performed with an Unreal Engine 4 empty project with a dedicated server and one client on a single machine. These values might not be 100% accurate but they will give you some information about where the latency happens. The client was running at 120fps and the server was set to a tick rate of 60. The client FPS and server tick rate for these tests were selected to make it easier to measure and present for the reader.

Hardware

The hardware you use can be a factor for the perceived input lag. We have a controller, mouse, or keyboard connected through Bluetooth, USB, or Wi-Fi. All of these devices have a small delay and it varies between the specific device, polling rate, and connection type. Both Xbox and PlayStation controllers have a small delay but they have a different default polling rate which could cause a bigger variety in the delays. The polling rate of a device is the number of times the operating system/driver will ask for changes. For example, an Xbox 360 controller has a default polling rate of 120 times per second and a Playstation 4 controller of 250 times per second. This means that the Xbox controller has a max delay of 8.3 ms and the Playstation controller 4 ms.

The delay to render a frame on your Monitor or TV can vary between 1 ms and 200 ms. For a new gaming monitor, the delay is very low (usually between 1-2 ms) and for older monitors, it can be around ~2-6 ms. TVs are different, they have a bigger screen and are not optimized for fast rendering. The delay can vary between 14 ms and up to 200 ms. Some new TVs are GSync compatible and have a gaming mode setting where some of the filters are disabled to reduce the delay in rendering. The refresh rate on monitors and TVs differs as well. Monitors have a refresh rate between 60 and 360hz while TVs are between 50 and 120hz. The refresh rate is how often a frame will be drawn on the screen, higher values will make motions appear more smooth and responsive.

Client frame rate table

Frame rate	Time per frame	Comment
30	33 ms	Older TVs and consoles
60	16.7 ms	Most common refresh rate
120	8.3 ms
144	6.9 ms
240	4.2 ms

The server tick rate

Let’s take a closer look at how game servers, specifically how tick rate works, and how it affects the responsiveness.

Tick rate is the number of times a server will process input, update the game state, and send updates back to the clients per second. A server at 60 tick rate will process input every 16.7 ms and for a server set to 120 tick rate, it's every ~8.3 ms. A higher tick rate on the server will decrease the time it takes to process the input from the client and update the game logic. This comes at the cost of more processing power and network traffic since more data has to be synchronized and processed.

In a game where the game state is big and there are a lot of changes, the server might not be able to process the entire state more than 30 times per second (30 tick rate), while in a game where the state is really small it might be able to process the game state at 240 times per second without any problem.

The tick rate will affect the input latency between 0 and 1/[tick rate] seconds depending on when the input package arrives at the server.

Internet and networked communication

Multiplayer games over the internet will always have a varied latency and this is due to the connection type, the number of routes between the client and the server, as well as the amount of traffic on that physical line. The Internet is very unreliable and can cause packages to arrive in the wrong order or get lost on the way. When a package is lost the client/server has to decide if it should request a resend or just discard it. This is usually done upfront in the game engine, marking some packages as reliable or unreliable. Reliable packages require the server/client to respond with an ack (acknowledgment) when the package has arrived while the unreliable version is just fire-and-forget, and ignores lost packages.
Input should always be sent as a reliable package since that data should never be lost, while position updates can be sent as unreliable. Resending an old position of a player in the game might be a waste because by the time it reaches the destination the data is old (this varies between different types of games).

When packages are lost some kind of delay will happen, in some cases, the game might appear frozen and in other cases, you might experience that your character did not react to your input fast enough.

Rendering and synchronization

There are a handful of settings that can change how a frame is rendered on the screen to increase performance, reduce the latency, avoid screen tearing, etc. Here’s a brief overview of how they affect performance and the perceived input lag.

A frame can be rendered to the back buffer (the buffer that the display reads) either by writing directly to the buffer, writing to a temporary buffer and copying the buffer when it's ready or to several buffers and just swapping the source. The two latter ones are called double and triple buffering. Both of these will increase the performance of the game since it doesn't write directly to the back buffer but it will also introduce a small latency. (see references for in-depth information about this)

Screen tearing is something that happens when the monitor's refresh rate doesn’t match the frame rate in the game. VSYNC is a way to solve this by keeping the frame rate (writing to the back buffer) in sync with the monitor's refresh rate. This will give the player a very smooth experience with no screen tearing, but it will introduce a small latency on the input since each frame has to wait until the monitor is ready to render it.

GSync/Freesync is a new monitor technology that lets the GPU control how often the monitor should render instead of the other way around. This eliminates both the input lag and the screen tearing and will give the best possible experience. The downside is that it requires special hardware, both the GPU and the monitor must support either GSync(Nvidia) or Freesync(AMD).

The tests in Unreal Engine

We’ve built some simple tools to track the latency from a button press until the server responds with the updates.

The blue bar is the Client and that’s how long it takes for Unreal Engine to register a button click. In Unreal Engine the input is tied to the frame rate, a high frame rate will poll the input more often, lower frame rate will increase the latency. The red bar is the communication to the server where it will package the data, send the package, unpack the data and run the method on the server. The yellow bar is the time it took for the response to come back to the client.

If we break these down into smaller parts and compare them to what we’ve learned so far: The client is running at 120fps, which means that the input will be polled every ~8.3 ms which would give us an average input delay of ~4 ms. This can be seen in the graph above.
The server tick rate is set to 60 and each input will be read every 16.7 ms (average at ~8.3 ms), this is the red bar in the graph. The server sends the response back to the client and the client reads that response every 8.3 ms (120fps), this is the yellow part.
The worst-case in this scenario is 30 ms between a button click and the code gets run on the client, and the best is below 10 ms.

In the graph below we can see a comparison of how long it takes before a button release event happens (this is a normal click with the thumb on a PS4 controller). This means that if the game reacts on button-up instead of button-down you’ll have a 50-120 ms delay before the input has even been read and processed by the game loop. To reduce the input delay you should do the action on button-down events, this might not be possible in some cases though. For example, in games where you have a double tap or a press+hold, the game code has to wait for a certain amount of time before it can decide which action you’re doing. If you use the same button for a single tap you’ll still have to wait that amount of time before it knows which action you are performing.

Summary

We’ve talked about the hardware, the software, and the internet. There are a lot of steps between pressing a button until something happens on the screen. In the table below you can see all the steps that happen when you press a button in the test we did. When everything is perfect most of these steps will be very close to 0 and in the worst case it will add up to a high number based on the frame rate, tick rate, internet latency, input device, and rendering settings.

The rendering and display times were not measured by us, these numbers are from tests that can be found in the references section.

Source	Possible Latency	Comment
Input device (controller, keyboard, mouse)	0 - 10 ms	The delay in the physical device, the drivers, and the Operating System.
Game loop (Read input)	30 fps = 33 ms 60 fps = 16.7 ms 120 fps = 8.3 ms	The frame rate that the game is running at if the engine polls the input once each frame.
Send input to server	0 - 0.2 ms	Packaging/Serializing the data on the client.
Internet/Network	> 0 ms	This is the time it takes for a package to reach the server (This is not RTT or Ping). This can vary a lot depending on your location and your ISP.
Process input on the server	30 ticks = 0 - 33 ms 60 ticks = 0 - 16.7 ms 120 ticks = 0 - 8.3 ms	The package will be processed every tick on the server, so depending on when it arrives the delay will vary.
Send response to client	0 - 0.2 ms	Package/Serializing the data on the server.
Internet/Network	> 0 ms	Time for a package to reach the client.
Process message on client	30 fps = 0 - 33 ms 60 fps = 0 - 16.7 ms 120 fps = 0 - 8.3 ms	In most game engines the network packages will be processed at the start of the game loop and systems can look at the values in their tick/update function.
Render	0-10 ms	Depending on the rendering technique used this can vary between 0 and a couple of milliseconds. It also depends on when in the game loop graphics are processed.
Display on monitor/TV	1 - 200 ms	The time it takes for the monitor to show what the GPU has in the back buffer.

There are some things we didn’t mention in this article and that are issues that happen on the server, for example, a short “freeze” on the server that causes the processing time to take longer than it should, disconnects, or dropped packages due to overflow of input, etc.

What can you do?

There are a few things you can do to reduce the perceived input latency.

Do some testing with VSync and enable/disable double/triple buffering.
If your device supports a higher polling rate you can probably find something in the manual or some online forum on how to increase it.
If you’ve got an older Monitor or TV and are considering an upgrade, look for something that is made for gaming.
GSync/Freesync/GSync compatible monitors and TVs will make the game look and feel better, make sure your Graphics card supports it.

What will GOALS do?

Some of these things that we’ve talked about are not something that GOALS can do anything about, but there are some parts where we’ll do our best to improve the responsiveness of the game.

The performance on the client and the server is one part that affects the input latency and it's something we can work with. We'll add a lot of telemetry points early in the development of the game so we can track the impact of every change and feature we add. This will give us a better understanding of which parts can be further optimized to achieve a higher frame rate.

There are a lot of different techniques that can be used to improve the perceived latency by predicting the move that the player does, like starting an animation when the button is pressed instead of waiting for the server to approve it. The downside is when the client and server disagree on something and you’ll experience a small rubber-band effect.
We’re going to spend a lot of time testing out different solutions to this problem so that we can give the player a great experience even when the network condition isn’t optimal.

We’ll keep you updated with our progress and share details about how we’re measuring the performance and responsiveness.

Links and References

Monitor/TVs

Monitors Input lag for controllers

Rendering

External code in Unreal Engine game source

Göran Syberg Falguera — Thu, 24 Feb 2022 14:30:07 +0000

Introduction

As many game developers know, the Git 100MB files size limit renders vanilla Git useless. For us, LFS did not seem like a viable option (we may post our full report regarding selecting version tracker later) and we ended up gong with Plastic SCM. So what we have now is:

A game built on Unreal Engine. We are using the source version of the engine as we need to touch engine code.
Full game and engine source as well as game data stored in a Plastic SCM repository.
All cloud services, backend code and shared code stored in our GitHub repository.

To be able to work with faster iteration times outside the engine, use the Git-workflows and tool sets and potentially share code with other Goals services, we have introduced the "External Packages Workflow".

In a nutshell, it is a GitHub repository inside the Goals game source and it allows for a couple of workflows:

Iterate rapidly on a library and its tests outside the engine source.
Iterate on a library, committing to the GitHub repository, but while testing it as part of the game source.
Iterate on a library when using it as part of another goals service.
Allow packages to be both Bazel packages and Unreal source at the same time without interfering with each other.

For the remainder of this article, I will use the Goals game-ai library as an example. It could also be used for instance for shared Protobuf definitions files or something similar. The key is that you can put anything there that does not interfere with Unreal build system.

Defining which packages to use

Since there was no natural place to add a list of external packages that we needed to use, we introduced the Packages.goals file that lives in the root of the Game source. This is a text file with a pretty self explanatory syntax:

# Defines which external Goals packages should be added to the source.
# If the Commit hash/releaese is omitted, latests will be fetched.
# Format:
# [Package name] [Remote git url] [Commit hash/Release tag]

# Game AI
game-ai https://github.com/goalsgame/game-ai.git 242a3d0ed0ba1cc3668962c90911ec0c8197a360

Wrapping Setup.sh/bat

To achieve this without adding an extra thing you have to do each time you sync we introduced GoalsSetup.sh/bat. What GoalsSetup does is:

Wrap the Setup.sh/bat files shipped with Unreal and only run it if needed.
Check if there are any missing external code repositories missing (as defined by the Packages.goals file) and if there are, fetch or update.
Check if one of the external repositories have been touched locally (i.e. git status --porcelain) and if it has, leave it alone.

So, as a coder, the only thing you have to do after you sync the main game repository is run GoalsSetup and everything should be dandy. It is our goal to not have any other dependencies that are needed on your machine apart from the ones dictated by working with unreal (which is essentially a compiler).

Hooking into Unreal build system

The easiest way to hook into unreal build system is to look at all external packages as one Packages module. This way, unreal build system can parse the source for the needed files and add them as dependencies for the game. So, create a new module by adding a build file in your source. For example Packages.Build.cs:

using UnrealBuildTool;

public class Packages : ModuleRules
{
    public Packages(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PrivateDependencyModuleNames.AddRange(new string[] {  });
    }
}

Now, in your game, add this module as a dependency. For example in Game.Build.cs:

using UnrealBuildTool;

public class Game : ModuleRules
{
    public Game(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;

        PublicDependencyModuleNames.AddRange(new string[] { 
            ...
            "Packages",
        });

        ...

    }
}

Now, each time you build your game project, the source of the Packages module will be parsed and needed files added to the project. Here is what it could look like in your source:

Note that the external package game-ai has some Git and Bazel related files in it but that is fine. And now, for each package we add to the Packages.goals text file a new directory is created on the side of the game-ai directory above and its files included in the game build.

Left to do and caveats

We still need to be able to handle including more complex packages. This could be for instance packages with source files that should not be included (test files) or files that are disruptive to the Unreal build system. This could be handled in Packages.Build.cs above by excluding certain directories. Maybe ones the include the word "test" for example. See how this is done in for instance RuntimeMeshLoader.

If we want to include packages that depend on other packages we would have to make sure the include paths are the same and the libraries available but ideally, all the external packages are atomic and do not rely on other things.

Building docker images in docker, and dynamic sequential Jenkins stages

Jonas Bergström — Thu, 17 Feb 2022 09:51:29 +0000

A couple of things I thought would be super easy but turned out to require a few hours of research and trial-and-error...

At GOALS we run Jenkins in Kubernetes for various reasons.
Some build jobs generates Docker images as artefacts, for example when we build new versions of some backend service.
However, Kubernetes is deprecating Docker, and we are running all our nodes on containerd.
So, how to build Docker images then?

Turns out that Google has a project for it, named Kaniko. And since we're running GKE, and has Workload Identity properly configured, it should be super easy to build Docker images and push them to our GCP managed docker repo, right?

No. I get this error:

aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "europe-west1-docker.pkg.dev/XXX": creating push check transport for europe-west1-docker.pkg.dev failed: GET https://europe-west1-docker.pkg.dev/v2/token?YYY: UNAUTHORIZED: authentication failed

AWS credentials - wth?

After some testing I found this, which solved the issue :). For an example Jenkinsfile, see below.

Another issue I had was that I wanted to generate dynamic build stages in Jenkins, and execute them sequentially. There are a lot of examples of how to execute dynamically generated stages in parallel, but that's not what I wanted.
Turned out to be super simple in the end ofc, but it aint simple until you've learned.

Here's an example Jenkinsfile that demonstrates both Kaniko and dynamic sequential build steps:

def stageConfigs = [
  [name: "Sara", age: 20, color: "blue"],
  [name: "Mona", age: 10, color: "green"],
  [name: "Lotta", age: 8, color: "red"]
]

def generateBuildStage(stageConfig) {
  return {
    stage("Building ${stageConfig.name}") {
      container('build-container') {
        echo "Building ${stageConfig.name}"
        writeFile(
          file: "${stageConfig.name}.generated",
          text: "${stageConfig.name} is ${stageConfig.age} years old and loves ${stageConfig.color} things")
      }
    }
  }
}

def buildStages = stageConfigs.collectEntries {
    ["${it}" : generateBuildStage(it)]
}

podTemplate(
  inheritFrom: 'linux',
  containers: [
    containerTemplate(name: "build-container", image: "busybox", command: "sleep", args: "infinity"),
    containerTemplate(name: "kaniko", image: "gcr.io/kaniko-project/executor:343f78408c891ef7a85bab1ecbf2dd69367a58bc-debug", command: "sleep", args: "infinity", runAsUser: "0", ttyEnabled: true)])
  {
  node(POD_LABEL) {
    stage("Checkout") {
      checkout(scm)
    }

    stage("Build application") {
      // execute builds in parallel
      // parallel(buildStages)
      // execute builds sequentially
      for (stage in buildStages.values()) {
        stage.call()
      }
      container('build-container') {
        writeFile(
          file: "Dockerfile",
          text: '''
            FROM busybox
            COPY *.generated ./
          ''')
        sh "ls -al"
      }
    }

    stage('Build image') {
      container('kaniko') {
        sh "/kaniko/executor --context `pwd` --dockerfile `pwd`/Dockerfile --destination europe-west1-docker.pkg.dev/XXX"
      }
    }
  }
}

HOWTO; Connect to a private GKE cluster using a site-to-site VPN between a GCP VPC...

Jonas Bergström — Sat, 12 Feb 2022 16:50:05 +0000

... and a double NAT'ed Ubiquiti Dream Machine Pro

Wait, what? I just want to access a Kubernetes cluster in a secure way ... should be easy.

[2022-03-10] Update from Google! I just got this message: "We are writing to let you know that starting June 15, 2022, we will remove the restriction on the Internet Key Exchange (IKE) identity of peer Cloud VPN gateways."
This means in practice that NAT'ed setups will become easier to manage because GCP will not require remote IP to match remote id.

At GOALS we are cloud native, and we are serious about security. As a consequence our Kubernetes clusters are provisioned with private endpoints only, where all nodes have internal IP addresses. This is great, but the question immediately arrises - how do we operate such a cluster, when we cannot access it from outside the VPC?

Here is a high level overview of what we have:

At the top of the diagram we see the private Google managed Kubernetes (GKE) cluster. A Kubernetes cluster consists of a control plane and worker nodes. In the case of GKE, Google manages the control plane (api server, etcd nodes, etc), the underlying VM's the control plane is running on, and the underlying VM's that worker nodes are running on.
We have set up our own VPC and a subnet where the worker nodes are running, and Google creates a managed VPC where the control plane is running. Google automatically peers the control plane VPC with our VPC.

At the bottom of the diagram we see an overview of our office space network. Since we are a startup we rent office space and share the network with other tenants. We have connected a Ubiquity Dream Machine Pro to the office space network, and created our own GOALS network where we connect our work stations.

Obviously, running eg kubectl describe nodes from my workstation in our office network doesn't work since kubectl needs access to the cluster's api server. So, how can we connect our office network to our VPC in a secure way, and enable management of the GKE cluster using kubectl?

A note on our infrastructure

We use Terraform to provision all GCP resources. Google provides opinionated Terraform modules to manage GCP resources here.
Our infra leverages a shared VPC and we use the project factory module to create the host project and the service projects.
The VPN will be provisioned in the host project that owns the VPC.
Our GKE clusters are created with the private cluster terraform module.

Prerequisites and preparations

There are a few things we need to ensure and some information we must gather before we can start.
First, we need admin access to the Ubiquiti Dream Machine (UDM) and we need a Google user with Network Management Admin.
Now you need to gather the following:

The office space external IP address, eg 123.45.67.89
The Goals network subnet range, eg 192.168.1.0/24

Ensure that the Kubernetes cluster is prepared

Two configuration entries are necessary to get right when setting up GKE to enable VPN access to the cluster.

master_authorized_networks needs to include the office network subnet range, ie 192.168.1.0/24 in our case.
VPC peering must be configured to export custom routes - in this particular case it means that the custom network route that the VPN will create to enable communication from the VPC to the office network will also be available in the Google managed VPC that host the GKE control plane. This is enabled by adding the following terraform configuration to the GKE setup: ```

module "kubernetes_cluster" {
source = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
version = "18.0.0"
...
}

resource "google_compute_network_peering_routes_config" "peering_gke_routes" {
peering = module.kubernetes_cluster.peering_name
network = var.vpc_id
import_custom_routes = false
export_custom_routes = true
}


### Shared VPN secret
Generate a shared secret that will be used for authentication of the VPN peers and put it into a new secret in GCP Secret Manager, name it **office-vpn-shared-secret**.

### Create a Cloud VPN Gateway
For this part we will use the [VPN](https://github.com/terraform-google-modules/terraform-google-vpn) module, setting it up in the host project like this:

locals {
name = "office-vpn"
}

resource "google_compute_address" "vpn_external_ip_address" {
project = var.project_id
name = local.name
network_tier = "PREMIUM"
region = var.region
address_type = "EXTERNAL"
}

data "google_secret_manager_secret_version" "office_vpn_shared_secret" {
project = var.project_id
secret = "office-vpn-shared-secret"
}

module "office_site_to_site" {
source = "terraform-google-modules/vpn/google"
version = "2.2.0"

project_id = var.project_id
network = var.vpc_id
region = var.region
gateway_name = local.name
tunnel_name_prefix = local.name
shared_secret = data.google_secret_manager_secret_version.office_vpn_shared_secret.secret_data
ike_version = 2
peer_ips = [ var.office_public_ip ]
remote_subnet = var.office_subnet_ranges
vpn_gw_ip = resource.google_compute_address.vpn_external_ip_address.address
}

resource "google_compute_firewall" "allow_office_traffic" {
project = var.project_id
name = "${local.name}-allow-office-traffic"
network = var.vpc_id
description = "Allow traffic from the office network"
allow { protocol = "icmp" }
allow {
protocol = "udp"
ports = [ "0-65535" ]
}
allow {
protocol = "tcp"
ports = [ "0-65535" ]
}
source_ranges = var.office_subnet_ranges
}

`google_compute_address.vpn_external_ip_address` creates the external static IP address which becomes the VPN endpoint on the GCP end.
`google_secret_manager_secret_version.office_vpn_shared_secret` fetches the shared secret used to authenticate the VPN peers.
The `office_site_to_site` module creates a "classic" Cloud VPN. The UDM does not support BGP yet which means that we cannot create a "HA Cloud VPN" variant.
`google_compute_firewall.allow_office_traffic` allows traffic originating from the office subnet (`192.168.1.0/24`) to enter our VPC.

After applying the configuration the VPN tunnel will be in an error state because it cannot connect to it's peer. This is expected since we have not set up the UDM side yet.

### Create a VPN network on the UDM Pro
The UDM Pro does not support configuration by code as far as I know, so here we need to resort to manually use the management GUI.

Go to Settings->Networks->Add New Network, choose a name and select VPN->Advanced->Site-to-site->Manual IPSec.
**Pre-shared Secret Key** is the `office-vpn-shared-secret` from above.
**Public IP Address (WAN)** is the IP address the UDM has on the office space network, ie it is __not__ the public IP our office space provider has. For example `192.168.10.150`.
In the **Remote Gateway/Subnets** section, add the subnet ranges in your VPC that you want to access from the office, eg `10.0.0.0/8` and `172.16.0.0/16`.
The **Remote IP Address** is the public static IP that was created for the VPN endpoint in GCP, eg `123.45.67.99`.
Expand the Advanced section and choose `IKEv2`. Leave PFS and Dynamic routing enabled.
Save the new network.

Unfortunately we are not ready yet, because with the current configuration the UDM will identify itself using the WAN IP we have configured, which doesn't match the IP it connects to GCP with.
To fix this last part of configuration we need to `ssh` into the UDM Pro. Once you are on the machine, we can update the ipsec configuration.

$ cd /run/strongswan/ipsec.d/tunnels
$ vi .config

Add the following line just below the `left=192.168.10.150` line:

leftid=123.45.67.99

This makes the UDM identify itself using it's actual public IP when it connects to the VPN on the GCP end.
Finally, refresh the IPSec configuration:

$ ipsec update


## Thats it!
Verify that the connection on the UDM is up and running by invoking:

$ swanctl --list-sas

The output should list information about the tunnel, and the tunnel should be in {% raw %}`ESTABLISHED` state.
Now the VPN tunnel state in GCP should move into a green state as well.
And, finally, accessing the Kubernetes cluster from a workstation in the Goals office network is possible:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-XXXXX-controller-pool-28b7a87b-9ff2 Ready 17d v1.21.6-gke.1500

We now have a setup looking like below:

![Private GKE and office with VPN](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zlornjjrpq8ya2eldfkx.png)

## Troubleshooting
While being connected to the UDM, run

$ tcpdump -nnvi vti64

to see all traffic routed via the VPN tunnel.

Sometimes routes are cached on the workstations, eg on Mac you can run

$ sudo route -n flush

a couple of times and disable/enable Wifi to make sure that your routing configuration is up to date.