Abstract
Imagine a world in the palm of your fingers where human society unfolds in a digital environment. We've built a simulation that captures the intricate dance of civilisation, from the rhythm of planting seasons to the pulse of economies. Every citizen, every passing day is meticulously modelled, allowing hundreds of years to play out in the blink of an eye. Tailor the starting conditions and witness the dramatic variations in humanity's journey.
For this paper, we’re going to primarily focus on the technical aspects of the project, detailing our milestones, and struggles as well as a handful of interesting insights derived after days of tireless reruns of the simulation in its current state.
Introduction
Simulating thousands of citizens across multiple worlds, with a multitude of interactions between them and their environment, is a complex and computationally demanding task. This intricate dance requires modelling citizen behaviour, their interactions with each other, and their responses to the world around them. Game engines excel at handling large numbers of objects and complex simulations. Given this high-throughput need, we opted for a game engine approach.
Techstack
After evaluating our options, we chose Rust and the Bevy game engine for its ability to efficiently manage thousands of entities with diverse properties and functions. Rust's versatility allows us to write performant low-level code while also providing high-level abstractions for easier development. This, combined with Bevy's minimalist design, fostered a smooth developer experience for adding new features. Here is a breakdown of the libraries and utilities:
- bevy_egui - allowed us to quickly create and extend a ui for the simulation using the egui library. Especially the egui_plot library allowed for the creation of easy plots
- bevy_async_task - allowed for easy async handling in Bevy without the issues that come with async.
- serde and serde_json - allowed for easy serialisation of the config files
- sqlx - easy connection to the Postgres database
- rand_distr - to generate random values according to normal distributions
Implementation and Methods
Rust
Rust is blazingly fast and memory-efficient: with no runtime or garbage collector. This focus on memory management empowers Rust's rich type system and ownership model to guarantee memory safety and thread safety. These features enable you to catch and eliminate many common bugs at compile time, saving you debugging time down the line.
Beyond performance, Rust offers a delightful development experience. Its documentation is top-notch, and the compiler provides friendly and informative error messages to guide you. Furthermore, Rust boasts a robust ecosystem of tooling, including an integrated package manager and build tool. For a smooth coding experience, it offers smart multi-editor support with features like auto-completion, type inspections, and even an automatic formatter. These tools combine to make Rust a pleasure to work with.
Community
Coming from the JavaScript and Java Ecosystems we were pleasantly surprised by the Rust Ecosystem and Community. If there was a need for a library to do something for us the ecosystem had our backs. Crates.io, a central repository, offered a vast library collection, but that's just the start. Rust's built-in documentation, strong typing, and open-source focus make using any library a breeze. Even small crates have excellent documentation and maintenance. The community thrives on Discord channels for libraries, tools, and frameworks, ensuring quick help when needed.
Errors and null values
Although Rust has a lot of things to offer, one of the things we found intriguing was error handling. Here’s why:
No Null Pointers: Unlike many languages, Rust eliminates the possibility of null pointer exceptions; a common source of error.
Explicit Error Handling: Rust forces you to explicitly address potential error points at compile time. This proactive approach allows you to prevent errors from propagating throughout the code, preemptively handling the error.
Non-null.
Anyone who has worked with languages like Java, Python, JavaScript, C++ and so many others knows the frustration of null and null pointers. These can cause crashes if not handled while checking for them becomes error-prone and tedious. Rust offers a refreshing alternative.
In “Safe” Rust, the concept of “null” simply doesn’t exist, meaning you can’t accidentally create null values or a pointer that points to nowhere. On the other hand, let's say you want a null value. Here's where Rust's Option<T> type comes in. It represents the possibility of a value being absent. There are two variants: Some(T): which contains an actual value of type T, and None: which indicates that there is no value present.
When you work with an Option<T>, you're forced to explicitly handle both cases: when the value exists and when it doesn't. This can seem like extra work at first, but it has several benefits:
Safer code: By explicitly dealing with missing values, you eliminate the risk of null pointer exceptions that can crash your program.
Improved clarity: Your code becomes more readable because it indicates how you expect to handle the absence of data.
While other languages might have similar features, Rust enforces the use of Option<T>, ensuring your code is always prepared for missing data. This focus on safety makes Rust a reliable and predictable language to work with.
Errors.
Similar to how Option<T> handles the absence of data, Rust uses an enum called Result<T, E> to manage errors. This approach promotes safer and more predictable code. This enum has two variants, Ok(T) which represents a successful outcome and contains the actual value of type T. Also Err(E) which indicates that an error occurred. The error type is specified by E. When working with a Result<T, E>, you need to explicitly handle both possibilities, If the operation is successful, you can access the value using pattern matching or methods like unwrap(). If an error occurs, you can extract the error details from the Err variant and decide how to handle it (e.g., retry, log the error, or return an error to the caller).
You might be thinking:
"Doesn't handling Result and Option for everything lead to a lot of extra code?"
Yes and no. While using Result and Option might initially seem like extra work for rare cases, they reveal more than just the success scenario. These types encourage you to consider potential edge cases throughout your code, by explicitly handling both the happy path (successful outcome) and the unhappy path (potential errors), you develop a habit of writing more reliable code. Over time, handling these edge cases becomes second nature in the development process.
While handling errors and optional values with Result and Option can add some boilerplate code, it prevents a significant amount of potential issues down the line. Consider the following code: with erroring_fn, we explicitly handle the possibility of an error. However, other_fn’s guaranteed success removes the need for error handling.
fn erroring_fn() -> Result<String, String>
fn other_fn() -> String
Bevy ECS (Entity Component System)
Using an ECS such as Bevy was something completely new to us. Compared to more traditional inheritance-based systems, such as OOP, both Rust and Bevy needed quite a paradigm shift. The Analogy that finally helped it click for us is, that our human body is our Entity, we are made up of many components such as our limbs and organs, and the systems are the functions that allow us to make use of our components. To be more specific and technical, two main things stand out as different compared to an inheritance-based system.
Data that is attached to entities is put in components instead of just being an attribute on a class for example as you would in OOP.
There is no hierarchy between entities. In Rust's terms, there is no way for an entity to "own" another entity.
There are a lot of implications of not having a hierarchy, but we think the root of all of it is a matter of separation of concerns. We could easily create a single entity that contains all data but that would be a mess in terms of separation of concerns. To not have this situation you create different entities for the different structures needed and then define the relationship through components.
This kind of relationship is purely logical, so if you need some data on the colony you need to query for both and match them up to find it.
Aside from the separation of concerns, there is another reason to create an entity out of a component. Bevy does not allow multiple components of the same type on the same entity. This means that if you want to add two of the same component to an entity that component needs to become its entity and the relationship will be defined in the above fashion.
Bevy's Parent and Child Component Relationship
The first thing we stumbled upon was the Parent and Children components provided by Bevy. These came with a bunch of utilities but were not necessarily what we wanted. Using these provided Components would have created a stronger tie than needed (at least in the beginning). Not only that, if we were to use the same Parent and Children components for all relationships then those relationships would not be descriptive enough.
In the following situation, what allows us to determine how two citizens are related is through what kind of component contains the relationship. So if a ChildOf component contains an Entity then that means that Entity is the (human) Parent of this Entity and if the Spouse component contains it then it's the spouse.
Components and Attributes
One of the major advantages of ECS is its speed and as far as we understand this is mostly achieved through Components. Components are the part of the Entity that contains the actual data. An entity can have as many components as needed (but only one of each type). Through an ECS we can query for individual components of an entity and not the entity as a whole which makes parallelization easier as it allows concurrent non-blocking mutable access to different components of a single entity by different systems.
The actual question though is how to split up the data into components. The below example is the way we decided to do it. birthday, height and weight data are grouped into components since it’s often accessed together but the two relationships (CitizenOf to the colony and Spouse to another citizen) have their separate components since they have completely different use cases.
The actual dilemma comes when we look at CitizenData specifically. The age of the citizen is needed way more often than the weight or height but when these two are needed age is usually also needed.
So to put these thoughts into practice one of the following options are possible. Either we stay with the previous setup or we make a separate component for the birthday.
This example is only to show what things need to be thought about when using an ECS system, but the general idea is that there are two extremes, putting all data in one component and every single data point in its component, but the best way is somewhere in between and completely depends on the specific use case.
A final thought on Bevy. We realised quite late that we could have utilised some of our domain-driven design knowledge with an ECS. All we had to do was consider the components as their own “domain” instead of the entity as a whole being the domain. It would have allowed for better abstraction in particular as our food system used 2 different units. In storage, it was kg and in consumption, we were considering calories. Secondly, it would have allowed us to have less duplicate code and simpler systems by implementing domain-specific functions and using them where needed.
Data and Visualisation
InfluxDB
InfluxDB is an open-source time series database developed by InfluxData. It excels at storing time-stamped data. We spent more time than we should've used Influxdb Because we assumed(wrongly) it would be fantastic at working with our data since it's all in essence time series data.
However, we found a major limitation for our use case, the timestamp field type, which has a data range of roughly 1969 to 2260. This constraint would restrict our simulations to 300 years in length. We also ran into multiple problems where the Influxdb user interface wouldn't display data that wasn't from the current timestamp as the current time. Additionally, a major concern was the amount of time every single query we made took. We soon learned that influx db is very fast at inputting data however querying for vast amounts of data frequently was just not what it was designed for. For these reasons, we switched to Postgres for our data storage.
Postgresql
PostgreSQL is an open-source, highly stable database system that provides support to different functions of SQL, like foreign keys, subqueries, triggers, and different user-defined types and functions. It further augments the SQL language proffering up several features that meticulously scale and reserve data workloads. It’s primarily used to store data for many mobile, web, geospatial, and analytics applications.
It was undeniably vital during the data collection phase of our project since it allowed us to overcome all the challenges we encountered using InfluxDB by allowing us to use register timestamps for more than the ~300-year limit imposed by InfluxDB’s systems. Additionally, it made creating the queries much smoother and faster since we already have extensive knowledge of SQL as well as the database systems just being more suited for reading vast amounts of data frequently.
Grafana
Grafana is an open-source interactive data-visualisation platform, developed by Grafana Labs, which allows users to see their data via charts and graphs that are unified into dashboards for easier interpretation and understanding. You’re then more easily able to analyse the data, identify trends and inconsistencies, and ultimately make your processes more efficient. A Major benefit to using Grafana is its dashboard setup: Grafana dashboards revolutionise data comprehension by offering a comprehensive platform to analyse and share insights derived from diverse data streams. With Grafana, we effortlessly construct customised dashboards tailored to the information we would like to convey, leveraging advanced querying and visualisation capabilities. The key features we used include versatile panel options and real-time data rendering.
Conclusion
Taking into consideration the little bit of experience working with other game engines like Unreal and Unity, implementing features is much easier with bevy. The speed at which changes can be made is impressive making it the perfect tool for our job. Although the new paradigm took a second to get used to we quickly got the hang of it and made it work for us. If we were to start this project over again we would probably use Rust’s features to a larger extent making our systems more generic and easier to extend. At the same time we would also reduce the depth of content and trade it for more breadth, making the whole simulation more complex and thus interesting.
If you are interested in our full implementation and/or our use of the technologies checkout our GitHub repo.
Top comments (0)