DEV Community: BrendanLoBuglio

Welcome to Rabbit Hell! Reliable AI Locomotion with TDD

BrendanLoBuglio — Thu, 23 May 2019 19:00:14 +0000

Test-driven development (TDD) is a software workflow where code is written alongside small "unit test" programs that quantify whether each component is working. While automated unit tests are hard to apply to game development due to randomness, 3D interaction, and unpredictable player input, we were able to utilize a TDD workflow to write stable and regression-proof creature locomotion for ElemenTerra.

The Basics of Test-Driven Development

Those experienced in TDD can skip this section; for everyone else, here is a basic primer.

Let’s imagine we’re coding a custom function that adds two numbers. In a traditional workflow we would just write it the way we think it should work and move on. But to use TDD, let’s instead start the process by making a placeholder function and some unit tests:

int add(int a, int b){
 return -1;
}
// Our unit tests that throw errors unless “add” produces correct results:
void runTests(){
 if (add(1, 1) is not equal to 2)
   throw error;
 if (add(2, 2) is not equal to 4)
   throw error;
}

Initially, our unit tests will fail because our placeholder function returns -1 for every input. But now we can go on to correctly implement add, and have it return a + b; all our tests will succeed! This may seem like a roundabout way to do things, but there are a few advantages:

If we didn’t get enough sleep and write add as a - b, our tests still fail and we’ll immediately know to fix the function. Without the tests we might not catch our mistake, and experience strange behavior that takes time to debug later on.
We can keep our tests around and run them every time we build our code. This means that if another coder accidentally changes add in the future, they’ll know immediately that they need to fix it because the tests will once again fail.

This is all unnecessary for this simple example, but with complex features like predictable state-machine behavior (after eating 100 food, is isFull true?) TDD saves time and improves a program’s stability.

TDD Application in Game Development

There are two problems with TDD in game development. First, many game features have subjective goals that defy measurement. And second, it’s difficult to write tests that cover the entire possibility space of worlds full of complex interacting objects. Developers who want their character movement to “feel good” or their physics simulations to “not look jittery” will have a hard time expressing these metrics as deterministic pass/fail conditions.

However, I believe that workflows more loosely based on TDD principles can still be applied to complex and subjective features like character movement, and in our game ElemenTerra we did just that.

Unit Tests vs Debug Levels

Before I get into my TDD practice, I want to make the distinction between an automated unit test and a traditional "debug level." It’s a common practice in gamedev to create hidden scenes with contrived circumstances that allow programmers and QA professionals to witness specific events.

A secret debug level full of different objects in The Legend of Zelda: The Wind Waker. Image Source<\a>

A secret debug level full of different objects in The Legend of Zelda: The Wind Waker. Image Source

We have many of these In ElemenTerra: a level full of problematic geometry for the player character, levels with special UIs that trigger certain game states, etc. Like unit tests, these debug levels can be used to reproduce and diagnose bugs, but a few aspects separate the two:

Unit tests divide systems into their atomic parts and evaluate each individually, whereas debug levels test features on a more holistic level. After observing a bug in a debug level, developers still may need to search for the point of failure manually.
Unit tests are automated and should produce deterministic results every time, whereas many debug levels are “piloted” by a player. This creates variance between sessions.

None of this is to suggest that unit tests are strictly superior to debug levels; in many cases the latter is a more practical tool. But I also believe that unit testing is underutilized in game development, and should be explored further with systems to which it is not traditionally applied.

Welcome to Rabbit Hell!

In ElemenTerra, players use mystical nature powers to rescue the creatures hurt by a cosmic storm. One of those powers is the ability to create pathways out of the ground which guide creatures to food and shelter. Because these pathways are dynamic player-created meshes, the creature locomotion needs to handle strange geometric edge cases and arbitrarily complex terrain.

Character movement is one of those nasty systems where "everything affects everything else"; if you’ve ever implemented such a system, you’ll know that it’s very easy to break existing functionality when writing new code. Need the rabbits to climb small ledges? Fine, but now they’re jittering up and down slopes! Trying to get your lizards to avoid each others’ paths? It looks like that works, but now their normal steering is all messed up.

As the person responsible for both the AI systems and most of the gameplay code I knew I didn’t have a lot of time to be surprised by bugs. I wanted to immediately catch regressions as they came up, and so Test-Driven Development seemed appealing. The next step was to set up a system where I could easily define each creature locomotion use case as a simulated pass/fail test:

This "Rabbit Hell" scene is composed of 18 isolated corridors, each with a creature body and a course designed to be traversable only if a specific locomotion feature is working. The tests are considered successful if the rabbit is able to continue indefinitely without getting stuck, and considered failures otherwise. Note that we’re only testing the creatures’ bodies ("Pawns" in Unreal terms), not their AI. In ElemenTerra creatures can eat, sleep, and react to the world, but in Rabbit Hell their only instructions are to run between two waypoints.

Here are a few examples of these tests:

1, 2, 3: Unobstructed Movement, Static Obstacles, and Dynamic Obstacles

8 & 9: Even slopes, and Uneven Terrain

10: "Navmesh Magnet" failsafe for floating creatures

13: Reproduction for a bug where creatures would infinitely circle around nearby targets

14 & 15: Step-Up ability on flat and complex ledges

Let’s talk about the similarities and differences between this implementation and "pure" TDD.

My system was TDD-like in that:

I started features by making failed tests, and then wrote the code needed to pass them.
I kept running old tests as I added new features, preventing me from pushing regressions to source control.
Each test measured only one feature in the system, allowing me to quickly pinpoint issues.
The tests were automated and did not require player input.

Unlike Strict TDD:

There was an element of subjectivity in evaluating the tests; while true pathing "failures" (did not get from A to B) could be detected programmatically, things like position popping, animation syncing issues, movement jitters, and whether the steering “looked smooth” required human evaluation.
The tests were mostly but not completely deterministic. Random factors like framerate variation caused small deviations, and a few courses had dynamic elements with randomized timing. Overall, creatures still usually followed the same paths and had the same successes/failures between sessions

Limitations

Using TDD to write the ElemenTerra creature locomotion was a huge boon for our schedule, but my approach did have a few limitations:

The unit tests evaluated each movement feature in isolation, and so bugs with combinations of multiple features were not encompassed; unfortunately I didn’t have time to make 18² courses! Thus, my unit tests sometimes had to be supplemented by traditional debug levels.
ElemenTerra has four creature species, but the tests only have rabbits. This is an artifact of our production schedule; the other three species were added much later in development. Fortunately all four share the same movement capabilities, but the larger body of our Mossmork caused a few problems. If I had to do this all again, I would make the tests dynamically spawn a chosen species instead of using pre-placed rabbit bodies.

This galloping Mossmork requires a little more room to turn than a rabbit

Evaluation: Is Test-Driven Development Right for You?

As developers it can be tempting to put a little too much work into testing scenes the player will never appreciate; I won’t deny that I had a lot of fun building Rabbit Hell. Internal features like this can be a big waste of work and jeopardize milestones, so we need to take a hard look at when & whether a given feature warrants a unit testing apparatus. Below I’ve identified a few criteria that, in my eyes, justified TDD for ElemenTerra’s creature locomotion.

1. Are your test cases time-consuming to produce manually?

Before spending time on automated testing, we need to check whether we can evaluate a feature with the regular game controls. If you want to make sure your keys unlock doors, spawn a key and unlock a door with it! Creating unit tests for this feature would be an irresponsible time sink because it takes only seconds to test manually.

2. Are your test cases difficult to produce manually?

Automated unit tests become justified when there are known, difficult-to-produce edge cases. Rabbit Hell course #7 tests creatures stepping off ledges, a circumstance their AI works very hard to avoid. Course #12 simulates the navmesh desyncing from the floor geometry, which only occurs during extreme lag. Such situations can be difficult or impossible to contrive with the game controls, while our tests produce them effortlessly.

3. Do you know the desired results won’t change?

Game design is all about iteration, and the goals of individual features may change as your game is redesigned. Even small changes in intent can invalidate the metrics by which you evaluate your features, and thus any unit tests along with them. While the creatures' eating, sleeping, and player interaction behaviors underwent several redesigns, we always needed them to get from point A to point B. Thus, the locomotion code and its unit tests remained valid throughout development.

4. Are regressions likely to go unnoticed?

Maybe you’ve been in this situation: you’re wrapping up one of your final tasks before shipping a game, and suddenly you find a game-breaking bug in a feature you finished ages ago. Games are giant interconnected systems, and so it’s only natural that adding New Feature B might break Old Feature A.

This isn’t so bad when the broken feature is ubiquitous, like the player’s ability to jump. If your core mechanic breaks, you’re bound to notice immediately. But the bug might slip under the radar if the broken feature is less frequently observed, like what happens when the player steps into a narrow crevice. Bugs detected in late development can jeopardize your schedule, and post-launch they hurt your player experience. Thus, unit tests can be great tools for maintaining edge case behavior, but are often redundant for functionality which already gets a lot of incidental testing.

5. What’s the the worst-case cost of having tests, and what’s the worst-case cost of not having tests?

Setting up a testing apparatus is a form of risk management. Let’s imagine that you’re deciding whether to buy insurance for a vehicle. The three questions you need to answer are:

How much do the monthly premiums cost?
How likely is it the vehicle will be damaged?
How expensive would the worst-case scenario be if you were uninsured?

For TDD we can imagine the monthly premiums as the production cost of maintaining our unit tests, the likelihood of vehicle damage as the likelihood of our feature breaking, and the cost of fully replacing the vehicle as the worst-case scenario for a regression bug.

If a feature’s tests take a lot of time to create, the feature is uncomplicated and not likely to be changed, or it would be manageable if it broke in late development, then unit tests may be more trouble than they’re worth. If the tests are simple to put together, the feature is volatile and interconnected, or its bugs would cost a lot of production time, then tests can help keep us on schedule.

The limits of automation

Unit tests can be a great supplement to catching and reducing bugs, but I want to emphasize that they don’t replace the need for professional quality assurance on large-scale games. Proper QA is an art that requires creativity, subjective judgment, and excellent technical communication, which means that you need skilled and well-taken-care-of humans!

Testing the Waters

While not the right choice for every circumstance, Test-Driven Development is a powerful tool that can and should be applied to more game development contexts. Let’s expand the horizons of what and how we test!

If you have opinions on the points made in this article or have stories of using TDD in your own games, please shoot us an email. If you liked what you read and are looking for a skilled group of developers to build, improve, or help finish your game, you can get in touch with Freeform Labs here. Until next time!

Freeform Labs, Inc. is a games and software development team with a commitment to providing the highest quality of digital craftsmanship, and a mission to inspire learning and creativity through cutting-edge experience design. The team is centrally located in Los Angeles, and maintains a network of trusted partners across the globe. With specialized experience in VR, AR, Game AI, and more, Freeform Labs provides an array of services including consultation, work-for-hire software development, and on-site "ship it" assistance. The company's portfolio includes work for Microsoft, Disney, and Starbreeze - as well as award-winning original content.

C#? In Unreal? Speeding up UE4 development with Mono-UE

BrendanLoBuglio — Fri, 19 Apr 2019 20:50:04 +0000

“Should I make my game in Unreal or Unity?” is a frequent conundrum for gamedevs, and one of the biggest sticking points is Unity’s use of C# versus Unreal’s C++. Freeform Labs' dilemma while developing ElemenTerra was that we preferred Unreal's renderer and editor but Unity’s C# scripting. Our solution was to integrate the open-source Mono-UE plugin, which lets us write C# code for our Unreal game!

C# versus C++ is a complex topic you can read about elsewhere. We’re fluent in both at Freeform Labs, but prefer C# for its faster development speeds:

C# is much quicker to compile; our C# solution builds in 3-5 seconds, while the C++ codebase takes an average of 100.
Because the language builds faster, C# IDEs are quicker and more responsive.
C# error messages are specific and coherent; C++ errors are verbose and often bury the lead.
C# nicely catches its exceptions, whereas most C++ errors cause Unreal to crash.

Overall, being able to work in C# was a huge productivity boost that justified jumping through a few hoops to integrate the plugin.

The Basics of Mono-UE

Let’s start with the core premise: Mono-UE works by hooking C# classes into the same bindings that connect Blueprints and C++.

Those who’ve used Unreal will be familiar with making “Blueprint” children of C++ classes. Blueprint classes are authored with a visual editor that can add new components, scripting nodes, and plenty more. In C++ programmers use macros like UCLASS, UPROPERTY, and UFUNCTION to choose which definitions to expose to Blueprints. The compiler then creates “bindings,” Blueprint-accessible wrappers around C++ code that allow the two ecosystems to communicate.

Mono-UE allows us to take these same bindings and create C# classes instead of Blueprints. The UCLASSes that are extensible by Blueprints can be inherited from in C#, and C# can call and override Blueprint-accessible UFUNCTIONs. This alone encompasses a huge swath of functionality, and lets us write complex gameplay logic in C# code. What’s more, C# classes can themselves be extended by Blueprint children!

Here’s a simple example of a Mono-UE C# Class, with comment blocks highlighting the connection points between C# and C++:




```

/// A new C# class that extends Unreal’s “AActor,” which is defined in C++
public class MyCSharpActor : Actor
{  
    /// Overridden class constructors:  
    protected MyCSharpActor(ObjectInitializer initializer) : base(initializer) {}  
    protected MyCSharpActor(IntPtr inNativeObject) : base(inNativeObject) {}   
   
    /// C# override of the ReceiveTick function, which is defined in C++  
    protected override void ReceiveTick(float deltaSeconds)  
    {     
        base.ReceiveTick(deltaSeconds)
        // C# can invoke the UFunctions SetActorLocation and GetActorLocation,
        //  just as we would from a Blueprint or C++:
        this.SetActorLocation(this.GetActorLocation() + new Vector3(1f, 0f, 0f));
    }
   
    /// Custom function that Blueprints can call because of the UFunction and
    ///  BlueprintCallable attributes  
    [UFunction, BlueprintCallable, Category("My C# Actor")]
    protected void MyBlueprintCallableFunction(int input)  
    {
        // ...  
    }
    
    /// Custom function with no UFunction attributes, callable only by other C# code
    public void MyCSharpOnlyFunction(int input)  
    {       
        // ...  
    }       
}

```

Mono-UE is open-source software maintained by Mikayla Hutchinson and a lovely group of other volunteers to whom we are deeply grateful. The setup process is detailed here. With the plugin still in development and some features missing or unfinished, be aware that you’ll need to pick up a handful of idiosyncratic workarounds as you learn to use it.

Our Mono-UE Workflow

Now that we’ve established the essentials, let’s talk about the day-to-day of working with this plugin! I spent a year of my life in Mono-UE and could go on forever, but for brevity here are just a few usage notes:

C# classes have access to almost all the functions available to Blueprints. This encompasses most of what we need, but sometimes we find C++ functions that don’t have bindings. This is a familiar problem for Blueprint scripters as well, and in both cases the solution is to write Blueprint-exposed C++ wrappers around the desired functions.
Most of our objects are “3-layer cakes” composed of a C++ base, a C# child class, and finally a concrete Blueprint. Generally, gameplay logic goes in C#, engine enhancements go in C++, and artists design audiovisuals in the Blueprint. Thus we spend the majority of our time in the language that’s fastest to write, work in C++ only when necessary, and keep our look-and-feel functionality in a format the artists can edit.

Every once in a while we had to modify the Mono-UE source. Packaging & shipping a standalone executable wasn’t available when we started ElemenTerra, but we fixed the bug and shipped the game! Our modification has now been accepted as a pull request.
“Hot reloading” wasn’t working during ElemenTerra development, so we needed to close and reopen the editor after changing our C# code. This was annoying, but we still found it much faster than recompiling our C++. At the time of writing, there is a not-yet-integrated pull request claiming a partial fix.

With these methods we enjoyed all the benefits of C# described above: faster build times, more responsive IDEs, better error messages, and exception handling that doesn’t crash the engine!

What about performance?

ElemenTerra is a VR game, which required us to hit 90+ fps. One of the main draws of C++ is that it’s a low-level language that can run extremely fast; does using C# then put us at a liability? The answer is not so black-and-white

To start with the facts, we’ve observed that Unreal C++ runs unconditionally faster than Mono-UE C#. Where things get gray is that sometimes C++ is significantly faster, but often the difference is trivial. For generic algorithms and math functions we saw very little difference, but C# was slower when we were doing a lot of thunks. “Thunk” is a silly software term which in this context refers to a call from C# into an Unreal C++ function. These jumps between runtime environments are inherently expensive, and rack up a performance cost when done en masse.

This is a factor that needs to be respected, but is by no means a dealbreaker on Mono-UE. Here are the best-practices we used:

Never optimize before profiling! C# objects that only had a few spawned instances or didn’t call thunks from loops usually didn’t chart on the profiler: no action necessary. This applied to the majority of our C# codebase.
Avoid redundant thunks. Instead of thunking GetActorLocation() 3 times, we can thunk it once and save its output in a variable. Rather than call DestroyActor() on every object in an array, we can write a new DestroyMultipleActors() C++ function and call it once.
Port expensive C# code to C++. When the profiler did show a piece of our C# code was expensive, we moved it down to C++. Often this was done in the simplest way imaginable: copy the whole script, paste it into .h and .cpp files, do a few find-replaces, and fix the errors until it worked. We were surprised by how few complications came from this approach; often the C++ worked on its first run! Our strict encapsulation policies at Freeform Labs made this approach quite painless, as each script had few external dependencies.

With these practices in place, we enjoyed the productivity boost of C# and still reached 90fps.

Is Mono-UE Right for Me?

This article is about why Mono-UE was the right choice for our team, and we have no regrets. If I've piqued your interest, I’ll close with advice for anyone interested in trying Mono-UE themselves.

If you’re a team with C# and C++ experience and want to be writing more of the former, Mono-UE might be right for you! Just be aware that there's an upfront cost to setting it up and that you may run into bugs you’ll need to fix in the plugin’s source.
If you’re a tech-inclined hobbyist looking for something new, give Mono-UE a look! It’s a great opportunity to learn more about how Mono and UE4 work under the hood, and as an open-source project it’s always looking for new contributors.
If you’re a Unity user who wants to try Unreal Engine and only knows C#, you’re in luck as long as you can tolerate the roadbumps that come with using still-in-development software. The first-time setup is tricky and you’ll encounter bugs which you need the community’s help to solve, but the payoff is definitely there.
If you’re new to programming and think C# would be easier to learn than C++, I’d recommend picking up Blueprint Scripting or Unity instead. Both of those tools already have an abundance of online resources for first-time learners, whereas the Mono-UE community is just getting started.
If you’re looking to ship to console or mobile, Mono-UE does not yet support those platforms but may in the future.
If you want to get your company set up with Mono-UE but don’t know where to start, perhaps we can help! You can get in touch here.

Programming Languages and Peace of Mind

As game developers, picking the right tool is important both for the quality of our craft and for the quality of our lives. Personally, I find Mono-UE extremely fun to use: C# is a language where I can get into a great flow state, and not being interrupted by compile times or obtuse errors helps me maintain focus. Whatever you do decide to use, I hope this article has inspired you to reflect on your own development workflow!

If you’re interested in trying Mono-UE yourself, this landing page will direct you to the source code and some community resources. Also feel free to contact us directly if you need a tip or want to talk more. Until next time!