DEV Community: Samuel Kahn

Best Source Control for game development: The Verdict

Samuel Kahn — Fri, 08 May 2020 13:25:42 +0000

This is part of a series of posts on source control for game development. Read more in the Blog.

We have finally reached the end of this series trying to find the best source control for game development.

If you have followed through all of the articles, the answer to your questions may still be unclear. Each solution has its pros and cons, many different pricing and hosting options. If you're a small team or an indie developer, you do not know what the future holds. Perhaps you'll sign a big publishing deal and need to scale rapidly with a larger budget, in which case your choices are changed once again.

So, for the final article in this series, and perhaps the only one you should read, I'll try to provide an overview of our findings and provide simple guidelines to help you with your choice.

Available Options

So far we've been studying in-depth 4 of the most popular source control options for game development.

Git, very popular outside of Games, but its regular downsides become showstoppers for game development at scale.
Perforce, the game industry standard, very expensive but the best option for dealing with large and complex projects.
PlasticSCM, a very promising competitor to Perforce. Much better than SVN, but comes with a price tag.
Subversion (SVN), a very solid option for small to medium projects, free and open-source.

Why do I see those as being the most popular? Well, one indicator is that these are the 4 options included by default in Unreal Engine 4, for instance. Perforce seems to be the most commonly integrated source control among game engines, especially the proprietary ones that you're unlikely to get your hands on unless you're part of the company.

That being said, there are other options we'll try to cover briefly.

Mercurial

Mercurial is a Distributed Version Control System (DVCS for short) very similar to Git. It is written in Python and is an open-source project as well. Mercurial's data model, features, and functionalities are virtually identical to Git. This has caused sort of an internet feud over which system is superior, with each side trying to convince the other to no avail.

Proponents of Mercurial argue that it is much better designed, and I generally tend to agree. In particular, Mercurial's command-line seems to be elegant and straightforward, using similar terms as subversion which makes learning Mercurial much easier than Git. There are other aspects that differentiate the two, most notably, while Git allows rewriting history through rebasing and several other commands, Mercurial's history is more immutable. For a full breakdown, these two articles from Atlassian summarize very well why one would be better than the other.

As with most DVCS, it requires a local copy of the history and therefore the same problems than we would have with Git would also apply. Mercurial seems to provide a solution for handling large binary files, and is considering a Git-LFS-compatible protocol.

Mercurial is still very active and used today in open-source projects as well as large companies. The Wikipedia page lists Facebook as one of the major users of Mercurial, and it is the common example thrown around when discussing Mercurial's adoption in the real world.

However, it is undeniable that Git has won the popularity contest. With Bitbucket shutting down support for Mercurial repositories, and now GitHub private repositories being free, more and more projects are looking to migrate from Mercurial to Git, as indicated even by some projects still listed on Mercurial's page.

For games, it seems that this is a poor option due to the lack of large file support, and the typical downsides of DVCS for non-mergeable files. I have, however, heard of some game projects using Mercurial, and would be very interested to hear some testimonies.

Unity Collaborate

If you're using Unity for your game project, you may have heard of Unity Collaborate.

Unity Collaborate is an official Unity product and part of their Unity Teams offering. This seems to be a managed platform and service to host your project, run your continuous integration and automation. The pricing is also very reasonable, especially considering there are 25GB of storage included.

I haven't used Unity Teams myself, but it does seem quite similar to any of the cloud-hosting providers we already evaluated for Git. When compared to those, it also seems to lack in other features, such as project management tools, documentation tools, and so on.

Unity Teams is also a black-box, and it seems that you can't access your repository in any other way than through the Unity Editor itself. As far as I know, Unity Collaborate actually runs on a Git-LFS backend, which makes it quite similar to GitCentral, but for Unity. However, I'd argue that having your entire project in a black-box system is not something you should feel comfortable with. There will be many opportunities where dealing with your source control from command-line or dedicated tools will be necessary, and as the project scales, I would be surprised if the Unity editor could handle the most extreme cases, especially since almost no dedicated source control GUI manages to do so. GitCentral, on the other hand, allows you to access the underlying Git-LFS repository directly, and deal with your project in any way you want.

Unity Collaborate is still a good option for low-stakes projects, such as student projects and relatively simple projects. As soon as your project will scale or require specific features, you'll need more than the black-box can provide.

I would still love to hear some testimonies about Unity Collaborate since I've never tried it myself.

Anything else?

Well, this is where my knowledge ends. I haven't heard of any other viable solution, nor met game developers using any version control that was not on this list. As I mentioned in the introduction of this series, some people use systems that cannot be qualified as source control, such as Dropbox and other file repositories with some sort of versioning. But, since you should without any question be using source control, those options won't even be discussed here.

If, however, you know of any other option you'd like me to look at, please reach out. I'll be very interested.

Comparison Matrix

With all that said, let's put all of this research together and see if we can draw some conclusions.

Please bear in mind that the assessments represent my subjective opinion after working with these tools in the context of game development. If we were talking about more traditional code projects, this table and the results would be very different. There are plenty of articles comparing source control for code projects if you need to.

Verdict

Unsurprisingly, the two most attractive options are Perforce and PlasticSCM. Both of these were designed for enterprise scenarios, very large projects, and a high number of collaborators. They were optimized to handle large binary files efficiently and provide great graphical tools and user experience. However, they are also the most expensive options.

Git and SVN are both free and open-source. This makes them much more popular with the development community at large. While the basic package only contains the command-line tools, both Git and SVN are supported by a massive array of third-party tools, web applications, hosting providers, and integrations in almost anything you can think of.

In my experience, your budget will be the main factor in the choice you make. This is why I've settled on the following ranking:

Perforce: If you can afford it, you cannot go wrong by choosing Perforce. It has been the industry standard for more than a decade, and for good reason. It does exactly what you need and you can easily work around its caveats.
PlasticSCM: A bold choice, but it can be worth it. PlasticSCM is constantly improving and I expect it to seriously challenge Perforce over the coming years. The price is affordable for most serious teams.
SVN: A classic for indie studios. SVN is free and allows you to spend a bit on your own hosting infrastructure rather than using the cloud providers. It will work well enough for small to medium-sized projects, and if your project grows larger, you should have money to spend to upgrade. For most readers, this is the right choice.
Git: Certainly the most popular version control, its model simply doesn't fit the needs of game developers, where binary files and non-technical people concentrate the bulk of the action. I would not recommend Git unless you use GitCentral, or an equivalent. Note that if your project is using SVN, you can still use Git for your programming thanks to git-svn.

So there you have it. In my opinion, Perforce is still the best source control for game development. This is the one I've chosen for Darewise, and we're very happy with it so far. I will continue to evaluate PlasticSCM periodically, as I feel it has a lot of potential.

However, if your budget is tight, I'd strongly recommend to take a look at SVN. You can achieve similar results for a fraction of the price, and you'll be able to migrate to a better solution once your budget allows.

Final words

I put a lot of effort into this series. I researched the source control options I didn't use, learned about them and their ecosystem, browsed through mountains of documentation, and actually tried them out on a project to see how they behaved. It was a learning experience for me, and I hope it was interesting for you too.

If you're reading this, thank you. I hope that it will help new game development teams to make the right choice on source control, which is arguably the most important technical choice to make after choosing a game engine.

Feel free to contact me for comments, questions, new information, and ideas for improvements on this topic.

Subversion Source Control for Game Development

Samuel Kahn — Thu, 30 Apr 2020 09:57:21 +0000

Subversion Source Control for game development

This is part of a series of posts on source control for game development. Read more in the Blog.

We couldn't reach the end of this series without talking about a source control giant, I'm talking of course about Subversion or SVN for short.

Every programmer should have heard of Subversion. Its massive popularity from the 2000s until today has only been outshined by Git fairly recently. In the games industry, SVN remains a very popular choice due to being free and implementing the file-based source control model which maps so well to our needs, as outlined in the previous articles. While not a common choice in larger studios, smaller ones and indie teams have long been using SVN as their backbone and for good reason.

Personally, I haven't used SVN since my early programming days, and never in my professional career. This article was a good opportunity to get familiar with it again and see how it competes against the other solutions we have studied so far.

So, is SVN a good source control solution for game development? Let's dig in!

Background and History

Subversion was started in 2000 as a replacement and improvement over CVS, the most popular version control at the time. The main goal in designing Subversion wasn't to reinvent the wheel or a new source control model but to have a better CVS, without the bugs and the features deemed unnecessary or badly designed.

Subversion has since then become a pillar of the software development world, overtaking CVS and becoming the standard for many years, both in the open-source communities and in the enterprise world. Subversion 1.5, in particular, was a major release that introduced highly needed features, most notably automated merge-tracking which is seen today as essential in a source control solution, as well as mirroring capabilities.

In 2010, Subversion became part of the Apache Software Foundation, one if not the most well-known organization maintaining large open-source projects. Apache now oversees the ongoing development and distribution of the rebranded Apache Subversion and is still looking for contributors to the project, in true open-source fashion. Recently, Subversion celebrated its 20 years of existence and is arguably still the most widely used source control solution today.

While it has certainly lost some of its popularity with more recent open-source projects in the age of GitHub, Subversion is used by many large open-source projects and in many organizations. Subversion doesn't benefit from a large platform such as GitHub, but it remains a robust and proven solution. Existing projects and organizations may not want to pay the cost of migrating, especially when the development community at large becomes more aware of the limitations of Git and DVCS they may have to contend with.

Features

The official website lists the main features of SVN. Here we will go through what is relevant for us.

Centralized source control

SVN was designed as an improvement over CVS, which was centralized version control. Even though distributed version control existed at the time, a centralized model appears logical and simpler. Even in open-source projects, there is a central or "blessed" repository where the official releases emanate from. In enterprise projects, a centralized repository allows for better access control over the intellectual property stored in the version control system.

The centralized is generally the way game development teams collaborate on game projects. This implies that, in order to share changes to others, you need to be connected to the central server. This should not be an issue in our day and age, but it does mean that you cannot easily work offline and keep a local version history. As we will see later, there are in fact workarounds to this issue.

File-based model

SVN uses a traditional file-based model to track changes. Each object has a revision number, and each change is tracked by a globally unique incrementing number. This is very similar to other version control systems such as Perforce, and is very easy to follow and understand, especially in comparison to Git's SHA hash-based revision identifiers.

There are notable features of SVN's source control model that aren't commonly found in other. Directories are versioned as first-class objects, in the same way files are. Properties in the form of key-value pairs are versioned with each object (file or directory), which allows you to embed extra information, and potentially has creative uses.

Branching is done through the "copy" operation. This can sound strange at first, but a branch is, in fact, a copy of the entire repository in another directory. This is how most older source control systems operate with branches. Fortunately, there are series of commands that allow you to treat branches the same way you would in Git, and not have to actually maintain two directories with two versions of the project on your own machine. Internally, branches are managed using a copy-on-write mechanism, so branches are not as costly as they may seem, and the cost of branching does not increase with the size of the data in the repository.

Still, branching and merging is not as simple an operation as you may have gotten used to with Git. More importantly, branching is a server-side operation due to the centralized model of SVN. This means branches are visible to all users, and for that reason, managing branches is reserved for privileged users or administrators of larger repositories. Using feature branches for small development is usually not recommended, instead, you may use the shelving feature, which is similar to Git's stash.

SVN has an interesting feature in the form of properties. Properties are key-value pairs that can be attached to any SVN object, such as a file, directory, or revision. SVN itself uses properties to implement some of its advanced behaviors and features, for instance, file-locking. You may also creatively leverage properties for anything required in your workflow, such as tagging certain files or using it for automation.

User experience

Setup and administration

Setting up SVN can be challenging for non-technical users. The website and the documentation definitely have this old-school open-source look and feel, which could deter some people. The most complete documentation comes in the form of a free book, affectionately known as The Red Book, which is curated and kept up to date by the SVN community. You can consult the HTML version directly on its website, and we will refer to it throughout this article.

The first barrier of entry is that the Apache Foundation does not distribute binaries for subversion. You must download the sources and compile them yourself. They do, however, reference trusted sources of binary packages for you to download the version that matches your OS. For most of us, this means Windows, and I'll recommend using Tortoise SVN which also brings a very practical Windows Explorer contextual menu extension and a graphical user interface.

While installing Tortoise SVN, don't forget to install the command-line tools. Without those, you won't really have SVN installed.

When you first create a SVN repository using Tortoise SVN, you will be asked if you want to use the default folder structure. This creates directories for trunk, tags, and branches which are pretty self-explanatory. You should use the standard folder structure to start with, as most of the tooling around SVN is configured to handle these by default. Of course, it is up to you to decide how you want to structure your repository, and where you want to store your branches. SVN offers you all the flexibility you need through the copy command, and those folders are merely a convention for the users to follow.

Command-line

SVN's command line uses terms and verbs that have come to be the standard vocabulary when talking about source control, and are understood by most. Most of the semantics we use for source control nowadays can be traced back to historical software, notably CVS, from which SVN takes most of its roots. This means SVN's command line is intuitive and easily understood, unlike Git's which introduces many new terms and confusing mechanics.

All of the commands are well documented in the Red Book, and for the most part, I didn't have to consult the documentation much when operating on them. To get started quickly, you can learn the basics in a few minutes on the official SVN website.

All of the command-line output is designed both for human readability and for parsing by scripts and automation tools. On top of that, API bindings to several popular programming languages allow users to interoperate with SVN more efficiently and without having to start multiple processes and parse outputs.

Overall, the command line takes some time to learn but it is relatively straightforward. The commands are single purpose and with a wealth of options, and the mental model of the source control repository is very simple to follow. Some commands have several logical (or funny) aliases for convenience, so you should feel at home quickly no matter what other source control you may be used to.

Graphical User Interface

The Apache Subversion project does not provide, maintain or endorse any Graphical User Interface. As is true with everything SVN, you will have to find your own tools to attach to the version control core. You can find a list of SVN clients on Wikipedia.

In the context of writing this article, I have tried two popular SVN clients on Windows. They both provide graphical tools as well as a Windows Explorer contextual menu integration.

Tortoise SVN

I've already talked about Tortoise SVN before in the setup guide. Tortoise SVN has been popular for as long as I can remember and is fully free and open-source, released under GPL license.

The main way to interact with Tortoise SVN is through the contextual menu integration. Right-click on any file or folder and the Tortoise SVN menu offers pretty much every possible action.

Unfortunately, there is no Tortoise SVN integrated tool. Everything is done through the windows explorers and popup dialogs. Personally, I think this is a worse user experience, as I often want to see an overview of my repositories and rapidly get information over many things in my repository. On top of that, not all of the dialogs are created equal. They can differ in style and useability, and some are clearly more polished than others. For instance, the revision graph view could be made much more compact and user-friendly. Still, Tortoise SVN is very useable in its current form and will continue to be the workhorse of many developers.

SmartSVN

SmartSVN is another Graphical User Interface that caught my eye. It seems to be from the same developers as SmartGit, a popular Git client.

SmartSVN has a free version and a paid version. The free version contains only a subset of the features, and obviously the most advanced operations are left out.

While SmartSVN also provides a contextual menu integration, it presents itself first and foremost as a standalone tool. Using SmartSVN feels very familiar coming from Perforce. The main view is a repository view where you can browse for all files and see their status. You can use the toolbar to perform all the main operations, such as Adding and Removing files, Updating, and Committing.

You can browse the full list of features on the SmartSVN website. There are important features only present in the paid version of SmartSVN:

Changesets, allowing one to separate modified files in several commits.
Merge and conflict resolution. Alternative and probably better tools are readily available for this.
Revision Graph, very useful to visualize changes of a file through branching and merging.
JIRA Integration, often useful in professional projects.

The user experience of SmartSVN is, in my personal opinion, much better than that of Tortoise SVN, and the price could be well worth it. Alternatively, I'd consider using the free version alongside Tortoise SVN, command-line, as well as other tools. The paid version of SmartSVN seems to be the most advanced and well-integrated SVN client available at this time.

Code review tool

SVN does not come out of the box with a code review tool. There are several third-party options to choose from, some are free, and some others come at a premium.

Out of all the tools listed, I would recommend looking at:

Review Board, free and open-source. While lacking in features compared to paid competitors, it's done the job for many companies and I've used it myself professionally.
Collaborator, probably my favorite code review tool, but the price is steep.

Diff and Merge tools

SVN does not provide Diff or Merge tools out of the box either, outside of the command-line utilities.

SVN can, fortunately, be configured to use an external diff tool. You are free to choose yet again from a variety of options, free and paid. For those of you looking, here are some to consider:

Meld, free and open-source, written in python. Well featured but sometimes slow.
P4Merge is actually free, and this has been my workhorse for my entire career.
SemanticMerge from the company behind PlasticSCM. A paid solution, but with unique features. I went in more detail about SemanticMerge in my PlasticSCM breakdown.
Beyond Compare, a popular paid tool. I've also used this and enjoyed it, but I did not see enough benefit to warrant buying a license for myself.

Useability for game development

Handling large projects

SVN has proven time and time again to be dependable for game projects. Many independent studios have chosen SVN for their source control because of its price and file-based model. Therefore, we can safely say that by using SVN, you cannot go entirely wrong.

However, it is also known that SVN has performance issues on larger repositories, and generally, this is the main argument its competitors, namely Perforce and PlasticSCM, use as the first real reason to consider switching. It's going to be very hard for me to evaluate SVN with a very large project that would recreate the enterprise conditions, so I can only take the testimonies at face value by saying: SVN is generally less performant than Perforce and PlasticSCM.

Still, SVN handles binary files very well and its model should scale relatively well according to our use cases. Two lines on the features page stand out:

Binary files handled efficiently. Subversion is equally efficient on binary as on text files, because it uses a binary diffing algorithm to transmit and store successive revisions.
Costs are proportional to change size, not data size. In general, the time required for a Subversion operation is proportional to the size of the changes resulting from that operation, not to the absolute size of the project in which the changes are taking place.

This should put you at ease when it comes to managing the binary files your game project will be filled with.

SVN can be brought up to an enterprise level by leveraging several features and extensions, some built-in and some using third-party software and services. SVN has been successfully used in enterprise scenarios for many years, in fact, the debate is still ongoing as to whether Git is really an improvement over SVN in many cases, and I tend to agree with some of the points made.

VisualSVN Server seems to be the most common server on offer, and it comes with multiple features useful for enterprises:

ActiveDirectory / LDAP integration, which enables Access Control features
For globally distributed teams, offers the ability to replicate your repository on remote sites in write mode
Backup and restore policy
Logging and auditing features

VisualSVN Server is unfortunately not free, though it seems to be the best enterprise SVN server available. Some of these features can be solved through other means, such as using svnsync to create read-only copies of your repository, but then you'll have to manage them yourself.

SVN also contains all the more advanced source control features you might need, for instance, you'll find an equivalent to Git's submodules in the Externals feature of SVN. This really shines when you are managing multiple projects at once.

There are also some more exotic tools that I found when doing my research. For example, SVK is an SVN client that adds more operations and allows for offline and decentralized workflow. I can't vouch for it as I haven't used it, but this goes to show that there is probably a solution out there for most of your SVN problems.

File-locking

File-locking is fully supported by SVN. On top of the basic locking and unlocking features, SVN allows for a user to leave a message to explain why they locked a particular file, which is a surprisingly simple and useful addition that I haven't seen elsewhere.

There are also ways to override locks when necessary. Of course, there are traditional administrator commands, but on top of that, SVN allows users to break and steal locks from other users. Of course, this is configurable and you may allow only privileged users to break locks instead of everyone. This can become useful when someone goes away for some reason while having the lock on a file, and you don't always have an administrator available to help.

Finally, files can be tagged to require a lock, and SVN will automatically set the file mode to read-only when you don't have the lock. This works beautifully and avoids making mistakes and wasting time.

Ecosystem

SVN has a long history and was very popular before the age of Git. Therefore, we should expect a great ecosystem that includes everything we need.

Integrations

SVN is well integrated with most typical development tools, such as IDEs and Continuous Integration tools. Here are some of the most useful for us:

Visual Studio, the IDE you're most likely to use, has an up to date SVN integration.
Jenkins and I'd expect most Continuous Integration tools.
Unreal Engine has first-class support for SVN, and the plugin is available out of the box when downloading the engine. Sadly, Unity does not.
Project management and bug tracking tools such as Jira also often support an SVN integration.
Communication apps such as Slack or MS Teams.

SVN is almost as well supported if not better than Git. Of course, most modern software will aim to interoperate with Git before SVN these days, but there is more than enough to be happy with and to satisfy all of our workflow needs.

Interoperability

SVN used to be the ubiquitous source control, which makes it a prime target for competitors trying to convert users. Here are a few bridges between SVN and other source control software:

git-svn, an officially supported bidirectional integration between Git and SVN. You're most likely going to need this even if you use SVN as your primary repository since it's likely that you will need to collaborate with third-parties using GitHub.
HgSubversion, a bidirectional bridge with Mercurial.
Perforce provides a migration guide and toolkit
PlasticSCM recommends using git-svn and their own Git interoperability tools

Hosting

SVN has been around for a long time, and it is not surprising to see many providers of cloud hosting. Unfortunately, there isn't a hugely popular website setting the standard, so we'll have to dig ourselves to find the best options. Fortunately, Wikipedia contains a good overview of source control hosting we can start from.

Here are some of the notable providers:

SourceForge, one of the most well-known hosting providers for open-source projects. SourceForge offers both the project management backend and provides a landing and download page for the software for its users.
Assembla, which is also the sole provider of cloud Perforce hosting, and can be considered a reliable hosting provider. The lower tier starts at $9.16/user/month for up to 5 users and 5GB of storage. Above this threshold, the price goes to $14.67/user/month with "500gb+" storage included.
Helix TeamHub by Perforce. Free up to 5 users and up to 1Gb storage. The paid tier comes at a very low $19/user/year, but still with a relatively limited storage capacity.
Cloudforge, for $2/user/month, but the storage capacity is tied to the number of users. Higher tiers allow for more storage.
Phabricator, an open-source hosting provider platform. While it is $20/user/month for the cloud option, Phabricator is a very good choice to host internally to manage your projects and repositories.
RhodeCode provides enterprise solutions starting at $8/user/month, with increasing costs depending on the hardware and storage capacity chosen.

Most of these providers have been active for a long time, and it also shows on their website and their offers. On this front, SVN does not enjoy the modern luxury that Git hosting has become. The user experience and features of the above are most likely not up to par. More importantly, neither is the pricing model, especially in light of GitHub's recent pricing updates.

That being said, this is not such a big issue if you're considering SVN for your game development project. Any of these solutions will give you a server with high availability, and that is all you really need. You can use other tools for project management if your hosting provider's platform does not meet your needs. For smaller teams, you should be fine using those.

Pricing

SVN is free and open-source. However, source control for game development can never be completely free.

You will have to host your project in a shared location, either on the cloud or on-premises. We touched on the cost of cloud hosting before. Of course, you may host SVN yourself on your own hardware, or using public cloud instances. The costs and maintenance involved can vary widely to be as expensive as you make it: hardware purchase and maintenance costs, backup and restore strategy, network infrastructure, and so on... It is up to you to find the best solution for your team size, configuration, and most importantly, your budget.

In comparison to its competitors in our study, SVN is the second cheapest source control for game development. Its cloud hosting options are more expensive than Git, but there are no license costs of any kind. For the budget game developer, and considering the technical limitations of Git for game development, it appears to be the cheapest, and therefore the best choice.

Conclusion

Subversion is a robust and proven software. It was once the dominant source control solution and remains very competitive to this day. It is free and open-source, still being maintained, and has an active ecosystem, even in the age of Git. I would love for SVN to have a more visible roadmap, as the current one doesn't provide much information. Some versions of SVN benefit from Long Term Support which is good news and a strong guarantee for you if you're starting today.

While the bulk of the software industry has moved on to the next shiny thing, SVN remains a very relevant source control for game development. Why? Its file-based model is exactly what we need, and its handling of binary files is only surpassed by expensive proprietary software such as Perforce or Plastic SCM.

SVN's flaws come mostly from its open-source legacy and its age. Much like Git, SVN is only the command-line core of the source control and does not provide the full package of tools and integrations a user will need on a day-to-day basis. Unlike Git, the tools and web services that are part of the SVN ecosystem aren't up to the latest standards of user experience and design. You will, therefore, have to research and choose for yourself which tools to use on top of SVN, and you may allow each of your team members to go with what fits them best.

But this is mostly cosmetic, and while cosmetics matter very much to some, it should not be the most important aspect for you if you're in business for a source control solution for your game development project. Price and features are the primary axes of comparison, and on that front, SVN is by far a better solution than Git. If your budget allows, then probably PlasticSCM or Perforce would offer either more features or more performance. For independent developers, it is often not worth the cost, and it explains why SVN has found a niche there.

We're reaching the end of the in-depth studies in this series. In the final article, I will try to reach a conclusion from all the learnings, and a summary for those of you who are still (but rightfully) confused.

Plastic SCM Source Control for Game Development

Samuel Kahn — Mon, 24 Feb 2020 20:43:52 +0000

Plastic SCM Source Control for game development

This is part of a series of posts on source control for game development. Read more in the Blog.

Plastic SCM is a lesser-known source control solution for game development. In fact, I expect that a lot of the readers will be hearing about it for the first time. I personally discovered it only a few years ago. So why is it worth an article in this series?

Well, Plastic SCM promises a lot: Plastic SCM brands itself as "the version control for games", and promises to deliver all the benefits of Git, as well as all the speed and power of Perforce, enterprise-grade features and support, unparalleled user experience, and unique features on top of that!

In fact, I was so excited by the product that I decided to evaluate it when I started Darewise, and back then I was very impressed. So let's jump right in and see how it looks like today!

Background and History

According to wikipedia, Codice Software, the company behind Plastic SCM, started in 2005 to create a commercial version control system that would be better than SVN, a benchmark at the time. The first version of Plastic SCM was released in October 2006, and several major versions have been released since.

One of the notable points to note from the history of Plastic SCM is that Plastic SCM 4.0 changed the source control model from a per-file model to a per-changeset model. This is interesting for us because, as we have noted previously, the per-file model is usually preferred for games, at least when it comes to not enforcing the entire workspace to be up to date before being allowed to commit.

Plastic SCM 4.0 is also the version that allowed Plastic SCM to better interoperate with Git, as well as a notable focus of the product on game development.

Another big focus of Plastic SCM and its side products is the focus on better diff and merge tools. One of the major axis of improvement is the ability to understand the code better and track methods as they are moved within a file or from a file to another. This should raise a few eyebrows for programmers as better diff and merge tools are always a great addition to our toolset.

Finally, the current version of Plastic is version 8.0.x which receives several updates every month. Plastic SCM is clearly being actively maintained and improved, and that can only be a good sign.

As you can see if you dig through the website and the release notes, Codice employs a cheeky tone in their communication which is refreshing, and this makes the product all the more appealing to me. It feels like a tool by devs, for devs, and a dream come true when looking at the list of features.

Features

Before you dig deeper, or if you want the short story, here is a SCM comparison matrix from Plastic SCM showing its strengths, but also drawing a fairly good picture. Perhaps I'll make a matrix of my own at the end of this series.

Distributed and centralized

Plastic SCM is the only source control that is truly both distributed and centralized. Plastic SCM is architectured around a client-server model, and each installation of the Plastic SCM client is paired with a local server. The beauty of this model is that there is virtually no difference in behavior or workflow when working against the local server or a remote server.

Here are the typical use cases you may perform:

Create repositories on your local server and work on them as easily as you would on any distributed SCM such as Git.
Work on a repository located on a remote server.
"Pull" from a remote repository, which transfers public branches of that repository to your local server, allowing you to work locally on your own copy of the repository.
Maintain private branches and history of your own changes on your local copy of the remote repository.
"Push" your changes or a branch from your local server to the remote server.

As discussed in previous articles, game development is a centralized effort, and it is most important that your version control is optimized for a centralized workflow. In this instance, Plastic SCM's primary model is always client-server, and therefore the centralized workflow is seamless and very efficient. You are also not required to use your local server at all, and you may avoid storing any redundant information locally, avoiding the typical Git issues and saving hard-drive space and download time.

You can easily combine distributed and centralized workflow in creative ways depending on your needs. Typically, programmers will enjoy the benefits of a distributed workflow to allow them to have private branches and history they do not wish to share. Less technical people will work directly against the central server, which is used as the single source of truth.

In fact, all of the common game development scenarios are covered: common repository for the engine but separate game projects, fully centralized or fully distributed, The combination of distributed and centralized setups elegantly solves most of the common collaboration issues, but everyone must be using Plastic SCM. If you're working with third-party companies, this is still unlikely.

Ideally, you will need to automate the pushes and pulls from the replicas to the master to achieve seamless multi-site setups. Plastic SCM also provides proxy servers for multi-site teams, but recommend using the distributed workflow instead, which doesn't suffer if the connection between the sites is interrupted. Where Perforce provides specific products to solve multi-site issues, Plastic SCM's distributed model appears like a more elegant solution and allows you to truly set up a multi-site infrastructure easily by using familiar elements of your regular workflow.

Changeset model

Plastic SCM changed its history model to a per-changeset model. This is closer to how Git presents the history. Each changeset is a collection of file changes, and changesets are presented in a tree-like hierarchy. Unlike Git, changesets are identified by an incremental number, and this makes it much more understandable.

While this is more modern than the older-generation per-file model, for us it could be an issue as we pointed out in our evaluation of Git. We do not want to have to download all the latest changes to submit a single file. Fortunately, Plastic SCM allows you to filter which files you are downloading using the cloaked files list. The other option is their Gluon tool, and we'll see how well it integrates into a typical game development workflow later in the article.

One of the main advantages of this approach is extremely easy branching, but also faster merging. And talking about the merge engine, Plastic SCM's is very good. While they claim it's better than Git at merging complex cases, the reality is that this is hard to test and you may find that it works better or worse depending on your merge cases. I certainly didn't have many issues and most complex merges seemed to resolve beautifully. This means time gained both in raw performance but also manual work for merges. In this aspect, Plastic SCM wins over Perforce in both speed and reliability of the merge engine.

User experience

Setup and administration

Plastic SCM's first-time user experience is very pleasant. Download and install like a regular program, and it just works! The documentation is very complete and will guide even the most non-technical users through all the steps.

Administrating the server is done through a mix of a web interface, configuration files, and executing commands from the GUI client or the command line. Again, it is surprisingly straightforward and doesn't involve a steep learning curve.

Even advanced administration procedures that can be scary in other systems seem to be simpler here. For backup, simply stop the server and backup a specific set of files, copy them back for restore. Although the initial setup is as simple as it gets, there is a wealth of configuration available to fine-tune the behavior, performance, access, and automation that you would expect from an enterprise version control system.

When evaluating Plastic SCM for the first time, I was blown away by the experience. After you register for evaluation, you are immediately offered a very thorough evaluation guide complete with test repositories to play with. On top of that, you are invited to a free webinar with one of the developers who will personally show you how to use the software and answer your questions. You even get a free t-shirt if you complete the guide, although I didn't apply for it in the end, so I can't confirm it actually works!

What I can remember though, is the impressive amount of support I received during and after the evaluation, as a non-paying customer. All of my emails and questions were answered quickly by the development team, and when I challenged certain design decisions or explained the reasoning behind us choosing Perforce, they became very eager to understand and hear my thoughts. Several very productive exchanges ensued via e-mail, conference call and discussions at game conferences. All I can say is: the devs behind Plastic SCM are truly interested in our user stories and solving our real-life problems, and it shows throughout the entire product.

Command-line

Version control always starts at the command-line, and Plastic SCM is no exception. The command-line tools are well-designed and well documented. There are in total less than 30 commands that provide enough options, alongside a concise syntax. The result is a very intuitive and fairly powerful command-line utility. The find command is a good example of an elegant and powerful command, which beats Git log in every way.

During my evaluation time, I didn't find any operation that would be difficult or impossible to perform using the command-line. Perhaps more surprisingly, I didn't find myself using the command-line as much as I expected, and that is thanks to all the graphical tools provided, where Plastic SCM really shines.

Graphical User Interface

Plastic SCM clearly aims to be user-friendly. It provides not one, but two graphical applications in order to solve the learning curve problem of version control. The main tool Plastic GUI, is fully featured, while the other, Gluon, is specifically tailored for game development artists and their specific workflow. Both tools are supported on Windows, Mac and Linux for maximum compatibility.

The main Plastic GUI is comparable to P4V and is similarly featured. You can perform all the operations from there without using the command-line. The main panels are a file browser, a view of the pending changes (detected automatically much like Git), and the ability to browse the history, labels, and shelves at the click of a button. You can also perform all of the administrative commands on your repositories from the client.

On top of this, Plastic provides a full suite of powerful tools you may have come to expect. There is a very graphical branch explorer to visualize the branching and merge history of the repository. This is extra powerful as you can use this view on a single file or directory, allowing you to filter and visualize the history in the most visual and intuitive way.

The annotate view, which is similar to git blame, allows you to see who last changed a particular line. A time-lapse view would have been excellent, but this will do.

Overall the Plastic GUI is great. There are still reports of some minor bugs and the occasional crash, but it hasn't been an issue for me.

The most annoying thing, in my opinion, is its tab and window management. There are things that only open as windows that you would rather have as tabs and vice versa. It also creates a new tab for almost every view, rather than reusing them. Again this has both pros and cons, but the result is that working on a repository will quickly end up with way more tabs than you can manage. I'd love for them to improve on that front by adding, for instance, the ability to pin tabs, to force certain tools to always populate the same tab, and to detach and dock tabs and windows freely.

Finally, as a power user, I'd love to have a log window much like P4V, where I can track all the commands issued by the GUI and the resulting console output. This helps you by keeping a history of your recent actions and comes very handy when debugging source control related issues.

Code review tool

Plastic SCM comes bundled with a full code review suite! Without having to install and configure an extra tool, you can review changes and even establish complex workflow rules out of the box.

The review tool is fairly well featured, but you may find that you require more control and configuration as you scale. It allows you to review changesets that have been shared on the central server. In other words, reviews happen post-commit. This is fairly typical in distributed workflow with per-task branches, but an option to be able to review shelves would have been nice. Reviews can also easily be tied to automation, for instance, a completed review may be automatically merged into the common branch.

While not perfect, it is very impressive to have it included at no extra cost.

Semantic Diff and Merge tools

Naturally, Plastic SCM provides a diff tool and a three-way merge tool. However, there is more than meets the eye here.

In parallel, Codice is working on a standalone product called Semantic Merge, which doesn't simply analyze text differences. It performs additional static analysis which allows you to track changes not just as lines added and removed, but as functions moved, symbols renamed, and follow changes even across multiple files.

In my eyes, this is a true breakthrough in programming tools. Using this, it is much easier to visualize what happened during a refactor, rather than deduce it from textual difference. The semantic engine is already very powerful, and hopefully will get better over time, as more and more diff tools take this direction.

Lately, Plastic SCM has been bundled with the semantic diff and merge capabilities, so that every diff view within the main Plastic GUI benefits from this engine. You don't need to buy a license for Semantic Merge separately anymore. This is simply amazing!

Semantic diff and merge are available for selected programming languages. There is official support for C++, C, C#, Java, and PHP, with the ability to create custom parsers to support more languages.

Gluon

Gluon is a tool designed for users working on a single branch on binary files that can't be merged, using exclusive locking. It also allows you to choose and download only the files you require, as well as share your changes on one file at a time. Thanks to exclusive locking, you will never have to deal with merging or conflicts. This is exactly what's needed for the bulk of game development. It seems Gluon was developed in close collaboration with Telltale and the result is impressive.

Gluon has a very simplified user interface. After creating your workspace, you can perform the following operations:

Configure which files and folders you want to work on
Update to the latest changes
Checkout and exclusively lock one or multiple files
Checkin changes
View file history and diff between revisions

And that's it! Gluon is exactly what the artists and less technical people need. It has all the features you would find in a basic file explorer integration, yet it also adds the necessary features of day to day workflow, without adding complexity. It is very easy to learn as there are only a few buttons and options. The main selling point is that it lifts the restrictions on having to download all the changes to check in a single file. This is great when working on very large art repositories where you are only likely to work on a few files at a time.

As a matter of fact, I'd love to see a Gluon equivalent for Perforce, and while GitCentral is essentially the same for Git, it's only accessible from within Unreal Engine and not as a standalone application.

There are some edge cases in which Gluon gets stuck. You generally have to discard your changes, or perform a manual operation with Plastic GUI to resolve the issue. It shouldn't happen often but it's worth noting.

Useability for game development

Handling large projects

Plastic SCM was famously used by Telltale Games, which should reassure you that shipping world-class game projects with Plastic SCM is possible. Codice Software has been using Telltale as its main showcase for game studios before Telltale tragically closed doors. Besides that, there are only a few game studios using Plastic SCM that I know of. At the very least, even if there is not widespread adoption, Plastic SCM has proven to work for a number of studios, which warrants our interest and pushes us to dig deeper. Therefore, let's examine the features of Plastic SCM that support large-scale projects.

Performance is clearly a focus of the development teams. There are numerous blog articles and benchmarks available which focus on the optimization of a particular feature, sometimes with comparison to competing products. Some recent client optimization around multithreading and process management allowed them to push the performance even further, as they shared some numbers privately with me.

Adding to that, the database backing Plastic SCM can be configured to fit your needs. Interestingly, the Jet storage is a special-purpose storage built for Plastic SCM, and now the default. It supposedly performs 3 to 5 times better than any of the other SQL backends offered, but you may still confirm that by yourself. Jet also has a "high scalability" mode that further increases performance for large repositories, but this is only available in the Enterprise version.

Finally, for enterprise networks and intercontinental data transfer, Plastic SCM supports a specific network protocol based on UDP which performs much better than TCP. I've seen studios build custom data transfer tools for this reason, so this should be very interesting for large publishers.

There are also some advanced features that really shine in larger organizations. Component-based development using Xlinks is almost essential and very well integrated. There more configuration and automation of the workspace than you would expect. You may achieve remarkably advanced tricks such as moving, renaming mounting directories locally while keeping the repository structure intact.

Access control and Security

Access control and permissions are very easy to configure through the web interface and the Plastic GUI client. The permissions are granular enough to allow almost everything you could want. You can create group permissions to easily manage a large number of users. You can also integrate with LDAP or Active Directory as you'd expect from an enterprise product.

Security is enabled through SSL connections and file encryption. As a bonus, you can easily audit your server where errors and destructive actions leave a paper trail.

File-locking

Plastic SCM provides file-locking out of the box. You can configure which files must be locked, as well as who can perform the operation.

It's simple and straightforward until you start working with branches. A file is locked globally on the repository, and not on a per-branch level. This makes it impractical to use in typical scenarios. For example, when you stabilize a build, you usually create a separate branch in which more fixes can be committed. However, if someone works on the same file in the development branch, you won't be able to lock the file in the stabilization branch. In this case, since you probably did not intend to merge back, you would have never had a conflict, and this workflow should be allowed. The only workaround available is to create a separate repository for stabilization, which is of course, not ideal.

Full and partial workspaces

Now, there is one caveat of Plastic SCM that I always have some trouble with. There are two GUIs, there are two sets of commands, and there are in fact two types of workspaces. A regular Plastic SCM workspace functions similarly to a Git workspace. It is synchronized with a specific changeset and may contain some local changes. A partial workspace is a Gluon workspace, in which you choose the files that are downloaded, and each file may be synchronized at a different changeset and may be committed individually.

This might seem logical, but becomes a problem in practice. Artists and designers will almost exclusively use Gluon and will be fine. Programmers and technical users, however, will require Plastic GUI and its advanced features. This doesn't mean they necessarily want to forego the per-file workflow when dealing with large binary assets such as the content of a game project.

It's not as bad as it sounds. Plastic GUI and Gluon will happily open a workspace and make the best of its current state. Gluon can mostly deal with Plastic workspaces, as they contain more information than the partial workspace would. In the worst case, you can resolve any blockers by discarding your changes. However, the Plastic GUI will require you to update your workspace to a changeset before being able to commit again. This may involve conflict resolution.

This forces programmers to stick to the Plastic GUI (per-changeset) workflow and avoid using Gluon altogether. For other team members, such as technical artists, it may not be so simple, as their usage of Plastic SCM is usually occasional.

It is an annoyance, but not a dealbreaker. Ideally, both workspace types would be fully compatible with one another. As a second choice, you could configure files and folders in a repository to be restricted to the per-file workflow. This, in fact, would solve the issue entirely. I tried several tricks using two repositories and mounting one workspace inside another to achieve this result, but I could not work around all the issues. It seems clear from the recent updates of Plastic SCM that "mixed-mode" workspace support is high on the priority list. Let's hope to hear some good news in the near future.

Ecosystem

Naturally, we expect Plastic SCM to have a smaller ecosystem than the major solutions such as Git of Perforce. However, when taking a closer look, Plastic SCM seems to have everything you need.

Integrations

You can find a list of official integrations on the website. The most notable integrations are:

Project management tools, in particular, the popular tools of Atlassian as well as Mantis.
Developer tools, especially Visual Studio which everyone is likely to be using for games.
Continuous Integrations tools, such as Jenkins and TeamCity.
Communication tools are not mentioned on the page, but an example of Slack integration is shared on GitHub.

On top of that, there are deep integrations in major commercial game engines. The Unity plugin is available with paid editions of Plastic SCM. The Unreal Plugin was developed by a coworker and friend of mine, in collaboration with Epic and Codice. It is now bundled with Unreal Engine by default. During my time evaluating Plastic SCM I tried using the Unreal integration and for the most part, it was functional and well integrated. There are definitely some edge cases especially as a programmer who plays with both Gluon and Plastic GUI. The Unreal integration seems to work with Plastic workspaces in most simple scenarios, but it works best using Gluon workspaces, as Unreal's workflow is almost identical to Gluon's.

For continuous integration, Plastic SCM features the mergebots which enable deeper integration and automation with external systems. For instance, you can use mergebots to automate merging a feature branch when the code review has been completed, or create merge queues to avoid the competing merge issues that can famously arise on graph-based source control. Think of mergebots as a powerful server-side trigger system to allow you to create and customize your workflow to your needs, and remove manual steps, especially manual merge actions which can be complex and time-consuming.

Interoperability

Plastic SCM interoperates natively with Git, Perforce and TFS. However, there is a definite focus on the Git interoperability through various key features:

Git Server: Every Plastic SCM server is also a Git server to which git clients can connect. When enabled, Plastic SCM repositories can be accessed as if they were Git repositories using a Git client.
Git Sync: A protocol allowing to perform distributed workflow operations (push/pull) against a Git repository not hosted on the Plastic server. For instance, you could use Git Sync to work seamlessly on a Git repository hosted on GitHub. This is great if you find Git's user experience complex, as Plastic is much more artist-friendly.
Plastic SCM now supports the Git fast-import format which is the de-facto standard for source control import and export. This allows for easy migration from another version control system to Plastic SCM.

This remarkable Git interoperability is not without its caveats. Plastic SCM does not have an equivalent to the Git rebase which allows you to rewrite your history in a more coherent form before sharing it. Rebase and fast-forward merges are therefore prohibited. It also relies on specific naming conventions and other little gimmicks which should be fine as long as you use it mostly as a read-only replica of the central Plastic SCM server.

The bigger issue remains, Git's properties still make it unwieldy for game projects filled with large binary files. To my knowledge, GitSync does not implement Git-LFS. The GitSync replicated repositories will need to contain the entire history of the binary files, which pretty much prevents using it on a large project. The same goes for cloaked files in Plastic SCM, which do not have equivalent in Git. If you use Plastic SCM primarily, you will probably fare better by avoiding the Git interoperability features unless they are required. That being said, it can have one very practical application: you can use Plastic SCM as your central repository even when contributing to external GitHub projects, which are commonplace nowadays even when collaborating with proprietary software.

Hosting

Codice provides an option for cloud hosting of Plastic SCM repositories with all of the different versions of the software. There is a specific cloud edition priced at $6.95/user/month which includes 5 GB storage space but is limited to teams of 15 members.

Here is the full pricing breakdown when increasing storage space. A 100 GB server will cost you $19.95/user/month on top of the basic license fee if you are using the Team or Enterprise edition. As you scale, it may be worth considering moving to your own hosting solutions.

Plastic SCM is, of course, made to be self-hosted on beefy hardware for enterprise needs. In this case, you will have to factor in the cost of purchasing and maintaining the hardware over time, which can be significant. Another option is to use public cloud instances to host your Plastic SCM server. This gives you more control over the hardware and the storage space, but also requires technical knowledge to set up, administer, and has an operational cost that may not be easy to forecast for novices.

While Plastic SCM doesn't have as many cloud hosting providers as Git does, the cloud edition is fairly priced and serves as a good starting point. It is assumed that, as your needs and budgets grow, you will move into more enterprise-like hosting solutions, and Plastic SCM comes with all the necessary features to support it.

Pricing

Plastic SCM is commercial software and therefore must be licensed. There is a tiered pricing model that targets different types of teams with different offerings. We already discussed the cost of data storage on cloud-hosted servers, so let's focus here on the licensing costs.

Plastic SCM is free for open-source software and non-profit. It is also free for individuals and hobbyists. Those versions come with a limited feature set focusing on the core source control. In particular, you may not use the Gluon GUI or the Unity plugin.
For small teams that don't fit in the above category, there is a cutoff at 15 users. You can choose between the cloud-hosted version ($6.95/user/month + storage costs) or the self-hosted version ($9.95/user/month).
Finally, the enterprise version allows you to scale above 15 users and adds the enterprise features such as LDAP/Active Directory integration and the high-performance storage and network features you are likely to need on very large projects. The full version is priced at $23.25/user/month, which makes it quite affordable overall, and certainly more affordable than its main enterprise competitor: Perforce.

Thanks to this tiered pricing, Plastic SCM can be affordable for many types of teams. When comparing the feature set with its competitors and taking the price into account, it is a very compelling offer. In particular, smaller teams with lower budgets may find that Plastic SCM has all of the features they need for a fraction of the budget and favor Plastic SCM over Perforce. Price was, in fact, the main reason why I evaluated Plastic SCM against Perforce when starting Darewise.

Conclusion

Plastic SCM is an amazing suite of tools, backed by an impressive and dedicated team at Codice Software. I would like to personally congratulate them for a great product.

There is a lot to love here: a huge and complete list of features, great UX and performance, one of the best Git interoperability, enterprise features ... but perhaps the thing that stands out the most is the amazing team behind Plastic SCM and the support that they are eager to provide. I would like to thank them again for their availability and openness during all of my interactions with them.

I've evaluated Plastic SCM twice now: once when starting Darewise, once again today. I was expecting to write the same notes as previously, but instead, it's amazing how much more there is to say, two years later. Most of my original points have been addressed, or are on the right track. There is a long list of new features and improvements at every level. Plastic SCM seems more viable than ever, supporting both Programmer and Artist workflow elegantly and providing solutions to all the typical situations you will find yourself in as a game developer, no matter your profile. It tries to combine the best features of Git and Perforce in one coherent solution, and it's not far from hitting the mark!

For enterprise users, Plastic SCM is more affordable than its main competitor, Perforce, but the choice is still not clear-cut and it will be up to every team to decide. I encourage every game studio to evaluate Plastic SCM hands-on, as they may be pleasantly surprised. While it is expensive to migrate ongoing projects, it could be a very reasonable option for upcoming ones. Old habits die hard and it may be a long road before game developers widely adopt a new tool, especially of such importance as version control. I've heard of a few studios trying Plastic SCM only to switch back to Perforce recently, and it's hard to know what were their reasons. Others, who came from Git and SVN are adopting Plastic SCM to overcome their hurdles. As for me, I am impressed but there are a few remaining obstacles. I would like to see resolved before seriously considering it for my professional projects.

For smaller and newer teams, it's a very different story. Plastic definitely beats Git and other free solutions for game development and should be within their financial reach. The time gain alone will be well worth the price, as non-technical team members will be operational in minutes and rarely get stuck.

I'd love to hear more testimonials of developers with Plastic SCM, whether good or bad, as I expect this article will need revision in the coming years.

In the next article, we will take a look back at an open-source source control that used to be the gold standard and still stands the test of time, I'm talking of course about SVN.

Perforce Source Control for game development

Samuel Kahn — Mon, 24 Feb 2020 20:40:58 +0000

Perforce Source Control for game development

This is part of a series of posts on source control for game development. Read more in the Blog.

Perforce is the most widely used source control solution for game development studios.

At Darewise, we evaluated many options and finally opted for Perforce, much like the rest of the game industry, or at least those who can afford it. Pretty much all major AAA studios, and medium to large teams are using Perforce as their main source control, so there must be some valid reasons behind this choice. I have personally used Perforce in every single one of my professional projects. I have worked primarily with workflow issues and source control, so this is a product I know very well and I might be slightly biased towards it.

In the previous article, I explained why Git is not the most optimal source control solution for game development. This time, let's examine why is Perforce so popular.

Background and History

According to wikipedia, Perforce (the company) was started in 1995 with its first product being the eponymous Perforce (the source control solution). The goal was to provide a system allowing companies to keep track of changes in large projects containing both source code and binary files. This already sounds interesting to us.

Perforce went through a few branding (mis)adventures as the years went by. I still remember its slogan being "the fast source control". Later, the company was bought and more products were added over the years to compliment the source control. These days the official branding for the source control product is "Helix Core", but in my heart and for this article, we will still refer to it as Perforce.

As stated on the official website, Perforce focuses on security, managing binary artifacts, performance at scale and teams distributed globally. They also claim that 19 out of 20 game development studios are using Perforce. While this claim is hard to verify, in my experience this is true when it comes to medium to large teams. In fact, I do not know of any sizeable studio not using Perforce.

On their officially disclosed list of customers, you can find Perforce is used by major game studios. EA, Ubisoft, Epic, Capcom, Crytek, CD Projekt Red just to name a few.

Microsoft famously maintains (or maintained?) a fork of Perforce called Source Depot. This was used notably for the Windows repository, which has now been migrated to Git, but they could only achieve this incredible feat after creating VFS for Git (previously called GVFS). This shows how much effort was necessary to adapt Git to handle as much as Perforce could.

Perforce has a history of being successfully used on large projects and by reputable companies. Some are migrating away after massive engineering efforts, others like the game industry are sticking with it.

Features

Perforce is packed full of features and I'm still discovering some of them. Let's go over the main points.

Centralized source control

Game development is at its core a centralized effort. Everyone needs to collaborate on the same project and be allowed to change it and receive updates as fast as possible.

Where open source leverages the distributed model to make up for a lack of trust, game development projects are most likely comprised of people trusting each other to work in a central repository. The only really interesting part of the distributed workflow is evident for programmers: the ability to create commits and branches which are only on your local machine and are not shared with everyone. This is great for programmers working on several features at the same time, amongst other things.

Perforce is often criticized for a lack of decentralized workflow, even though it provides several tools for it. The most basic tool is shelving which is the equivalent of Git stashes. I will agree that it is mostly a workaround, and feels lackluster after using Git for a while. However, Perforce does, in fact, have a decentralized option called DVCS. This works in command-line in a similar way to Git and has recently been added to the P4V client. Unfortunately, not many people know or use DVCS, so let's hope this article gets the word out.

Changelists and per-file versioning model

Each change in Perforce is grouped together into a Changelist. A changelist is a list of files, their revision numbers and the operations performed on each file.

Each changelist is given a unique incremental number. This makes it very easy to understand which change precedes or follows another. The number is unique on the entire Perforce server, even if you have multiple projects. In fact, one changelist can theoretically contain files from multiple projects, although I would not encourage you to do this often.

The change model is also per-file. While each changelist is a collection of changes, each file has a history with incremental revision numbers. Again, this is clear and easy to follow, as opposed to git's object model and commit hashes.

What's more, your workspace does not need to be up to date with every single file when submitting a changelist. You only need to be up to date with the files you are planning to change. This means you can iterate much faster than Git, especially when dealing with many large files in a game project. However, this also means your workspace may not be in a fully working state, as some of your files are up to date while others may not be. In practice, this is not as big of an issue as it sounds, you can easily synchronize your workspace to a specific changelist, or more frequently the latest change, when a problem arises with your workspace. As you don't have to sync so often, your iteration time is increased.

The branching model of Perforce is not as simple as Git's. Since each file is versioned independently, branches are performed at a file-level and are harder to visualize when looking at the big picture.

In the past, working on several branches was cumbersome and required the creation of a new folder on your hard drive. Since the introduction of streams, you can easily switch branches within the same folder, as you do with Git. On top of that, streams formalize the branching workflow by enforcing which files belong to which branch and controlling how you can integrate or copy from one branch to another, at a granular level. This is great because your guidelines are present in the tool rather than simply by convention, which helps avoid human error.

Large project support

Perforce puts the accent on performance when handling large files. This is evident in all of their communication and can be verified in practice. Storing and retrieving files is fast, especially if you are hosting your server on-premises.

On top of that, Perforce provides multiple solutions to allow geographically distributed teams to work together more efficiently. You can use basic caching proxies and read-only replicas, all the way to a network of "edge" servers that act as your local server and ensure eventual consistency with the master. This, combined with enterprise-grade networks and bandwidth, allows large organizations to work around the world on a centralized repository. Of course, setting up such an infrastructure involves massive costs as well as constant maintenance.

Another key enterprise feature is the integration with LDAP or Active Directory for user management, as well as the myriad of access control options to restrict access in all kinds of way. Large organizations can, therefore, use a single Perforce infrastructure for all of their projects while maintaining the security and confidentiality of their data. Additionally, Perforce can be configured to use the SSL protocol for greater security.

User experience

Command-line

For programmers and advanced users out there, Perforce provides a command-line interface. Having used it for years, I can say for certain that it is very well designed, consistent, easily understood, very powerful and very well documented.

When compared to Git and the criticism I emitted in the previous article, it's night and day. Perforce's command line is almost intuitive in its design and has remained very stable over the years. This has made my job easier when integrating Perforce in our automation systems or in various game engines in my career, where updates to the Perforce client or server rarely if ever require code changes. There are a few subtle issues that experienced users will eventually run into, but they are very minor inconveniences in practice, and Perforce's command line is a joy to use.

Graphical User Interface

This is where things get interesting. Perforce ships with P4V, the official a graphical client for Perforce and the only one worth mentioning. This is one of the biggest differentiating factors, as Perforce has always put emphasis on user experience especially for less technical users. One of the major arguments for the adoption of Perforce against other solutions is P4V.

Now, let's be honest, this is source control and it can never really be trivial to use, so there is a bit of a learning curve, but it is my personal opinion that P4V is the best source control GUI bar none.

For beginner users, it intuitively presents the file tree of the local workspace and the remote repository, provides basic functionality such as showing the history of a file or a directory, organizing your changelists and of course submitting your changes. This means non-technical users will never have to see or use the command-line. The same can't be said for most competitors.

For power users, it is a fully-featured Perforce client that can perform almost all the tasks you will ever need in practice. In fact, as a very experienced Perforce user and administrator, I rarely have to use the command line. I can perform complex queries over the files and the history and visualize all of the results in very well designed tools. With the administrator GUI, I can perform the most regular administrator tasks such as creating and managing my users and groups. The only time where I really need the command line is to perform complex and unfrequent server configuration or maintenance.

Here are some of the most advanced tools which don't have equivalents elsewhere.

Revision Graph: Easily visualize and follow the changes and integrations of a file. This is similar to visualizations of a Git branching tree, except it only shows you the evolution of a file which is often more useful in practice and lacking in Git interfaces.

Time-Lapse View: The most useful of them all. This is git blame on steroids. It allows you to browse a file and see for each line who was the last person to edit it, just like git blame. However, you can also scroll through time! Using this tool, you can follow a line and see the surrounding context evolve with revisions. You can quickly and easily understand how some code came to be, and when appropriate, blame the right person for their mistakes. The attempts to replicate this tool for Git still have a long way to go before they reach the same efficiency.

P4V is certainly not perfect, and Perforce development teams are releasing improvements and updates on a regular basis. What I would like to address here is the common belief that Perforce's user interface is difficult to handle. Those complaints often come from non-technical people who don't have a lot of experience with source control and find the core concepts confusing. However, I have not encountered a better graphical tool to handle source control than P4V so far. The concepts are daunting at first, but once understood, P4V allows you to efficiently manipulate them. I would be interested, of course, in knowing your opinion on this, but ultimately this is a tool that has been proven to work in real-life situations. Even the less technical minded artists and designers end up learning and understanding enough to work with it and they do get over the initial pain. In fact, the reputation of having a bad UX probably comes from not having been presented with the alternatives. That's not to say that UX of source control software cannot be improved further, but this is almost as good as it gets in 2019.

Merge tool

If all of this wasn't enough, Perforce also maintains and distributes a free diff and merge tool: P4Merge.

This is, of course, included and well-integrated with P4V, but it is available for anyone to use. It is one of the better diff and merge tools out there, and includes most of the features you'd expect. It is easy to use and consistent with the other tools from Perforce. As an architect and today a CTO, you can trust me on the fact that I have performed a lot of very complex integrations using many different source control solutions and setups. For all of those, I have been using P4Merge and still do to this day.

I have seen some people prefer Beyond Compare or Araxis Merge over it, but those aren't free and I never really understood what makes them so much better. I'd love to hear some compelling arguments. There is, in fact, a very promising and much more interesting merge tool I'm looking at these days, but we'll get to this in another article.

Useability for game development

Handling large projects

Perforce can handle source control for game development projects of any real-life scale. It performs reasonably well even with huge projects containing terabytes of data, hundreds of thousands of files and hundreds of users.

I have some personal experience with Ubisoft's Perforce infrastructure which is probably one of the most complex in the industry, and it undeniably does the job. Teams are building on top of Perforce to improve and optimize where needs be: transferring large amounts of data across the globe can be slow, but this becomes a rich man's problem when reaching such scales. For reasonably large teams, using Perforce out of the box should cover all of your collaborative needs.

Setup and administration

Perforce is complex software to deal with as an administrator. Where you can start using Git simply by typing "git init", for Perforce you'll have to take your time and learn.

The initial investment is significant: you will need hardware to host your server and learn to set it all up. Perforce can be hosted on Windows or Linux but for many reasons Linux is preferable, and this is once again not beginner-friendly. You also need to figure out a backup and restore strategy, to plan for the worst. You do not want to lose your data, as this is your most valuable asset.

Perforce requires configuration to fine-tune to your needs. You should configure the authentication, typemap, depots and streams before you start working. Sounds like a lot, but the upside is that you can get exactly what you want out of it. For more customization, you can create triggers, the equivalent of Git hooks, and server-side extensions in Lua. On the client-side, you can create plug-ins for P4v, or use the client APIs with C++, Perl, PHP, Python, Ruby, Java, Groovy and perhaps more.

While the learning curve is steep, Perforce allows you to truly implement your desired workflow down to the specifics. This can be extremely valuable over time and especially in larger organizations.

Tricky operations

Branching and merging operations are harder than on Git. This is generally reserved for power users, even with experienced teams. Branching can be slow and error-prone, and the merge engine is not as efficient as Git's. Some specific cases are very delicate to handle: those of you who have integrated moved and modified files will know what I'm talking about.

You will also encounter limitations of P4V one day or another. It performs extremely well on regular operations, but when dealing with several thousand files, you are better off using the command-line. Again, those operations should be unfrequent in most game projects and generally reserved for power users.

Perforce will be a great tool to leverage, provided you have technical people to support it. For small teams with artists and designers only, this could be daunting. For most serious game development projects, one of the programmers will be capable of taking this responsibility.

Ecosystem

Integrations

Perforce (the company) provides an array of products on top of the source control. Each one of them has its own licensing model. While some are free, some others require a separate paid license.

One of the most relevant for us is Helix Swarm, which is a free code review tool for Perforce. This is a great addition to the toolbox, and an indispensible one if you are serious about programming. Swarm is well integrated with Perforce and provides enough features to do the job. Just be aware that it is not the best tool on the market. Here is a list of alternatives. I hear that Perforce is planning for an update of Swarm, and I can't wait to try it out.

Because it is used everywhere in the game industry, Perforce is also well integrated with game engines. Unreal, Unity, Cryengine, and every proprietary engine I've worked on had a Perforce integration of some sort.

Perforce is also integrated with IDEs, most notably Visual Studio with its official plugin P4VS.

Of course, Perforce also has integrations with all of the major Continuous Integration solutions. For instance, they maintain and distribute the official Jenkins integration. This is what we use here at Darewise.

There is a Slack integration, sadly it is part of a separate Perforce product. A little research will tell you if your favorite tool has an Integration, but don't count on it.

Interoperability

In recent years, Perforce has fully embraced Git as an attempt to increase interoperability with Perforce source control as well as their other products. They provide several tools to that effect:

git-fusion: a legacy python script that allows two-way copies between a Git repository and Perforce server. While it is no longer listed on the website, it is still in use and I don't think there is a replacement for this particular use case. git-fusion is, for instance, how Epic Games and Cryengine provide a public Git endpoint, even though they work on Perforce internally.
Helix4Git: a Git server available on-premises or in the cloud. The unique selling point being, of course, its performance as it is backed by a Perforce server, presumably at least the same database engine (oracle). This allows for hosting both Perforce and Git repositories on the same server.
Helix TeamHub: a full project management solution including hosting the repositories, access control, kanban boards. It even supports SVN and Mercurial. Basically this is the equivalent of GitLab by Perforce, although it lacks the DevOps/Continuous Integration which makes GitLab so popular.

Hosting

Simple hosting solutions for Perforce are rare. Most teams choose to host on-premises using their own hardware or public cloud options. For managed solutions, it seems that Assembla is the exclusive provider of cloud hosting.

Overall, the ecosystem of Perforce is smaller than that of Git, as expected. That being said, all of the major points are covered: everything necessary for you to perform your daily tasks is available and pretty high quality. If that's not enough, there are several P4 to Git options to choose from.

Pricing

Perforce sells itself as an "enterprise" software and comes with an "enterprise" level price. In other words, Perforce is expensive.

You can use Perforce for free if you host it yourself. It is however limited to 5 users and 20 workspaces. In practice, most people will exceed this limitation. Each user needs to create several workspaces, at least one per workstation. It is recommended to create "system" users such as an administrator user, and probably one for your swarm and for your CI system. Of course, you can ignore these recommendations but the free tier remains very limiting.

If you require more, you must negotiate a licensing agreement with Perforce, with an undisclosed price and licensing model. While I am using Perforce myself, I am not sure I am at liberty to disclose our agreement. We'll just stick to public information to be safe.

For cloud-hosted servers with Assembla, the pricing comes in two tiers. The basic price starts at $44.50/user/month with a 12-month commitment. Moreover, extra features, as well as single-tenant hosting, can be negotiated with a custom pricing agreement.

We can safely assume all the undisclosed pricing is at least the same if not more expensive, which makes Perforce at least $534/user/year. This is before you factor in hardware and maintenance costs if you choose to self-host, which many studios do for greater control and data security. This is a large investment for most teams, and it is the most expensive source control solutions of all we are going to be looking at in this series.

I would love for Perforce to reconsider their licensing model to allow for more teams to access the software. This is the backbone of the game industry and it would surely benefit from a larger ecosystem. I'm sure many small teams would be inclined to switch to Perforce if they were offered a reasonable monthly model per user. The free tier limitations could also be relaxed to allow for small teams to use it properly, but it would require some development on Perforce's side due to how the license system is implemented. Let's keep out hopes up and see what the future holds.

Conclusion

Perforce is a great source control option for game development if you have the means to afford it, both financially and technically. For newer or smaller teams, it can be scary to get into, not to mention expensive.

It is still the undisputed champion of the game industry and the workhorse of all the major studios. It was adopted for good reason, as its ability to handle large projects was unrivaled, and this is the biggest issue facing game development projects, far before user experience or interconnectivity with the open-source ecosystem. However, the game industry has also accumulated enough legacy on top of Perforce that even if there was a better solution available today, the adoption would be slow due to the ever-increasing costs of migration.

Perforce is constantly improving in the meantime. There are several yearly releases of the server and the client, as well as the satellite products. A lot of these updates are going in the right direction, improving the user experience, adding features to support our workflows and interoperability with more systems. The company is clearly investing and consolidating its ecosystem to cover all grounds of software development, from source control to application lifecycle and project planning with more recent acquisitions. I believe it has a great opportunity to increase adoption by proposing new business models to attract new customers, and I will certainly keep an eye out for major updates.

In the meantime, newer teams should aim to start with the best tools available presently and within their budget. Which is the best fit is certainly not obvious, this is why I'm writing this after all. As an industry, we should never stop looking for better solutions, challenge our assumptions and established processes in order to improve our workflow. Making games is one of the most challenging tasks in software and art development of our time, and every improvement to our quality of life is worth it.

In the next article, we will look at a challenger that could rival Perforce on its own territory: Plastic SCM.

Git Source Control for Game Development

Samuel Kahn — Wed, 06 Nov 2019 13:58:56 +0000

This is part of a series of posts on source control for game development. Read more in the Blog.

Git is probably the first result people stumble onto when researching source control for their game development project.

As evidenced by my activity on this very website, I am very interested in Git. I have used it extensively both professionally as well as for personal projects including game projects. I have come to love its beautiful design, and hate its limitations. Because of those, I created GitCentral to overcome Git's shortcomings and create a better-suited source control workflow for game development.

This article is a good opportunity to examine the motivations behind creating GitCentral, and shed some light on why Git is both great, and difficult to use for game development.

Background and History

Git no longer needs to be introduced. Its steady rise to popularity alongside GitHub makes it nearly impossible to avoid.

If you're a programmer or aspire to become one, you need to learn Git now! Whether you want to participate in open-source projects, fork someone else's cool program, or just for your local hobby projects, Git is the tool you need. Git will be useful for you even in enterprise scenarios where you may use a different solution for the primary repository, and work on closed-source code. Many companies have started to use GitHub as a practical means to collaborate on proprietary code.

Features

A complete list of features of Git can be found in the official documentation. Here we will focus on the major features that separate git from other source control solutions. We assume that all of them provide the most common source control features of versioning, collaboration, and data integrity.

Distributed source control

Git was born out of the needs of the open-source community. These design intentions can be seen and felt in all the features and implementation of Git.

It uses a distributed model, which means there is no central server holding the source of truth. Instead, each developer holds an entire copy of the code and all the history. At their discretion, they may redistribute the code or its artifacts, or contribute their modifications to someone who does. Teams of maintainers curate popular repositories, and sometimes, a difference of needs or opinions leads to a fork which takes a life of its own.

In the most common form, the open-source distributed workflow can look like this:

Graph model

Git uses a graph model to represent the history. Each change is stored as an object and a link to its parent. The beauty of this representation means that every change is potentially a branch. Git also provides a very powerful merge engine.

Those features encourage a liberal usage of branching and merging. This workflow is very practical for programmers and is a big driver behind Git's popularity. By comparison, branches and merges in older version control systems such as SVN were time-consuming and error-prone. They tended to be avoided or reserved for elite programmers. Not only the ability to branch and merge liberates programmers and allows them to experiment freely, it also encourages merging more often, which leads to smaller patches that are easier to review and maintain. The result is increased overall productivity, stability and maintainability.

Git is very fast at performing all the regular source control operations with a local repository. It's when working with a remote repository that its limits start to show. The implementation of the graph model requires the repository to be fully consistent with the revision from which the current changes are being built upon, in order to allow committing. In practice, this means before every commit, one must get all the updates from the remote repository. By contrast, other version control systems only require you to download the updates to the files you have changed. With Git you must download everything. Git is not particularly fast at file transfer either, but more on this later.

User experience

Git is primarily a command-line tool. This makes it harder to use for beginners and less technical users. Its model can be confusing and it can be hard to understand what is going on when something went wrong.

The steep learning curve has become somewhat of a meme, spawning many websites and tongue-in-cheek literature.

Git is a very powerful tool, but we should also recognize its flaws. I share the opinion of many that the commands and the user experience could be improved. Some commands such as checkout or reset perform far too many different actions which would have been better separated into unique verbs. Gitless is a notable attempt to provide a better user experience on top of Git, and this might be just enough for beginners to get into it.

Still, even with a lot of experience using Git and diving into the plumbing while working on GitCentral, I find myself often looking at the documentation, begin confused by the inconsistent semantics used, or facing undocumented behavior.

Graphical User Interface

Git does not provide any Graphical User Interface. It is only a command-line tool. Here is an official list of third-party GUIs from the official website. However, they all seem to be lacking something when put to the test.

Personally, I have tried a couple of the major GUI tools and currently settled for SourceTree. Even then, I perform most of the actions using the command-line, which means the graphical tool has not provided me with a better alternative. You also have to add your own diff and merge tool. As I am used to Perforce in my professional life, I really miss some of the most advanced features which make me more efficient such as Time-Lapse View, though there is, in fact, a similar tool for Git. I'm sure some people are faster using only the command-line, but I am certainly less efficient and I would bet that the majority of developers are in the same category, especially the less experienced ones.

Having a graphical tool at your disposal to perform basic actions doesn't solve the learning curve entirely though. Git's underlying model and semantics are complex and hard to grasp for non-technical users. The best way to work with Git is through simple integrations, such as TortoiseGit which adds actions in the contextual menu of Windows Explorer. GitCentral does essentially the same from within Unreal Engine.

Overall, Git attracts for its features and its great model for programmers, but it is hindered by its complexity and user experience.

Useability for game development

Downsides of the distributed graph model

Game projects are rarely distributed efforts. More often than not, you want to collaborate with your entire team on a central repository. This goes against Git's model and creates very specific issues.

No single file versioning operations

As we already mentioned, in order to share any change, you must first download all the latest updates from the central server. This is very sensible for code, as it forces you to test your changes with the latest version before committing. On game projects where large binary files may have been changed, this could mean waiting a very long time for the updates to download. If anyone pushed their work in the meantime, you have to do it all over again until you're fast enough to commit your changes. When the number of collaborators increases, it can become almost impossible to commit a change manually. This sort of "traffic jam" situation can be solved using a lot of extra tooling and automation, but it is an extra hassle that many other source control solutions don't have. Interestingly, open-source workflow generally does not suffer from this because relatively few maintainers work on the central repositories.

On top of this, you may not want to get all the latest changes yourself. The very latest may be unstable, which is a reality of most game projects. More importantly, it may require you to stop your work and start a long process of compiling and "baking" the new code and assets. With some game engines and enough changes to download, this could take hours, only to find yourself with a potentially unstable version of the game. The ability to push or pull one file at a time is therefore very important in game development scenarios. Of course, Git's approach is safer. Pushing only a few files without testing with the latest changes could create unexpected issues. However, depending on your iteration time, the tradeoff may be well worth it.

Binary files need a centralized workflow

A more subtle issue is that the default function of git is to work against your local version of the repository. You will not see the changes made to a specific file on the central repository until you fetch the information and run specific commands. Since binary files cannot be merged, it is more important to know the status of a file when compared with its latest version on the remote repository, rather than the local one. When using Git traditionally, you will only see the conflict when you try to merge your changes with the remote changes, and it may already be too late at this point. File locking, as explained below, helps avoiding conflicts, but not entirely.

In essence, the workflow of Git does not match the collaboration model of the majority of game development teams. Game projects will find that they often have a better iteration time when using a centralized, file-based model. This is how many other version control solutions are implemented. Unfortunately achieving this workflow with Git is very complex and requires using many plumbing commands, as I did with GitCentral. You may be able to adapt to Git's distributed graph model, but the issues will start appearing when adding more people to the project, and the iteration time will be dramatically impacted.

Handling large projects

Git's distributed model means that every developer has a copy of the entire repository and all of the history on their hard drive. As the project gets larger, this causes the repository to grow in size to the point where it is too large for the hard drives and too slow to download. Git was created to handle text files primarily, but in game projects, binary files will quickly grow a Git repository beyond reasonable limits.

To allow Git to handle binary files more gracefully, several extensions have been created. Git-LFS, backed by GitHub has emerged as the dominant solution for a couple of years now. Git-LFS is transparent and stores binary files remotely to avoid bloating the main Git repository and its history. This solves the size issue and doesn't create an additional layer of complexity for the user. However, Git-LFS is very slow when compared to other source control solutions. I could not find a good benchmark online, but it is very noticeable. It seems clear that the current design of Git-LFS and how it integrates with Git could be optimized for performance.

Git's scaling issues are well known and there are still many attempts to improve on it. In the future, those problems may be completely resolved, but as of this day, Git scales poorly with game projects especially when compared to some other solutions. Meanwhile, there are some workarounds.

File locking

File locking is the action of "locking" a specific file, which means no other user can modify it until the lock is released. This avoids conflicts, and is critical for game development where binary files cannot be merged.

Git-LFS provides a basic file locking implementation since the release of Git-LFS 2.0.0. Most of the hosting providers support Git-LFS and locking by now.

There are a few caveats to note:

Only the files tracked by Git-LFS can be locked.
Locking a file locks it for all the branches of the repository. This can prevent two people from working on the same file on different branches.
The implementation of locking depends on your Git-LFS server, so your mileage may vary.

It is very important to note that two people may not work on separate branches at the same time. When working on a stabilization branch, say for a demo or a release, you may have to coordinate with others working on the next features in the master branch, and avoid using the lock at the same time. This can work with small teams, but the more people you add, the more this will slow you down and create issues. You may not anticipate it yet, but your source control workflow will get more complex over time, adding more branches for features, demos, releases and patches, and all those small problems at first will become worse.

Access control

In a game project, you often want to control who can access your files. You may even want some files to be read or written only by privileged users, for security or confidentiality reasons.

Because of its distributed nature, Git does not provide any access control out of the box. You must add it to your repository server. Luckily, virtually all the cloud hosting providers and self-hosted solutions offer private repositories and user management. However, I haven't found more fine-grained control such as per-directory or per-file permissions. You can often enforce additional rules, such as locking a branch, requiring a successful code review and automated tests before the change can be committed to the central repository. Generally, git offers less access control options than its competitors, but this is likely to improve in the future.

Of course, you can implement your dream workflow using git hooks, but this comes at a cost.

Ecosystem

Git has access to the most comprehensive ecosystem of tools and integrations available. I did not attempt to make an exhaustive list as it would be too long and quickly outdated. Virtually every tool or technology you are likely to require integration with your source control will provide some level of support. Git is available for all desktop platforms used in game development (Windows, Mac, Linux)

The most notable integrations include:

Game engines: Unreal has a Git integration by default, but GitCentral allows for a better workflow adapted to gamedev. If you decide to use Git and Unreal, then this will greatly improve your quality of life.
Developer tools from IDEs like Visual Studio to DCC tools.
Continuous integration tools such as Jenkins or the aptly named GitLab
Productivity tools like Jira or Trello
Communication apps (Slack, MS Teams...)
Cloud hosting providers, these often provide extra tools on top of hosting the repository, including many of the above

Note that some of these integrations only work with GitHub, which can limit you if you're not hosting your project there.

There are also scripts to allow migrating to and from most version control solutions to Git. This can be useful to migrate, and if you ever need to replicate your history in another version control for some reason. Git-SVN is available out of the box. Perforce has various tools as well, but I'll get deeper into those when talking about Perforce in a future post.

Pricing

Git is open-source and entirely free. However, there are costs to running a game development project with Git, as you still need to host your repository and its data in a shared location.

In a previous blog article, I compared the major Git hosting providers and their features. This should give you a good overview of the features and costs involved, and pointers to choose the best solution. Of course, you may host on your own hardware, which comes with its own time and financial costs.

Nonetheless, there are no license fees, which makes Git one of the cheapest and by far the most accessible source control solution for game development.

Conclusion

Git is here to stay and you will have to learn to work with it, whether you want it or not.

Thanks to its popularity, you have access to the biggest ecosystem of integrations and interoperability. You also have plenty of great cloud hosting options for your repository and your productivity tools.

Despite these great features, Git is not very well suited for game development. Its distributed model does not match the typical centralized workflow and does not scale very well: the larger your project gets or the more collaborators you add, the more you will run into issues. On top of this, it is one of the hardest source control solutions to learn. If your project has less technical people such as artists or designers, you will likely spend a lot of time helping them understand Git, and fix many mistakes made along the way. Git can be good enough for small-sized teams comprised of fairly technical people, and where the pace of change is slow enough to not create too many conflicts with binary files. As your project scales, you will be looking for a better solution.

On my personal game projects, I quickly encountered issues with Git's model and user experience. I decided to create GitCentral to solve them. It has since allowed me to collaborate in a convenient way with Git and Unreal Engine. Even so, GitCentral is best used on smaller projects, since the Git-LFS backend does not scale so well when your game project gets inevitably larger. In retrospect, I should have extracted the logic of GitCentral into a library from which to create several tools, such as a Windows Explorer integration, or allowing to integrate it in Unity and other common game development tools. Perhaps this will be a future project if there is demand.

In my professional life, when starting Darewise, we evaluated many options and finally opted for Perforce, which is the game industry standard, and what I have used in all of my previous positions. In the next article, I will show you what motivated this decision.

What is the best Source Control for game development ?

Samuel Kahn — Wed, 06 Nov 2019 13:54:57 +0000

Choosing the best Source Control for a game development project is a hot topic in the indie community. Many teams struggle with making the right choice, and the advice available online is often misleading. The answer is surprisingly not obvious, and there is no one size fits all solution.

Earlier this year I talked at the Nordic Game Conference about Source Control, and how to structure your team's workflow, automation and continuous integration and deployment pipeline around it. You can download the slides here. As this is one of my areas of expertise, I wanted to go deeper into the subject and break down the options available in order for game development teams to make an educated choice about source control.

In this series, we will study the major options available to you, their various pros and cons, and highlight the reasons why you should choose one over another depending on your team size, budget, and makeup.

Should I use Source Control?

Let's get one thing out of the way first, Yes you should always use source control. Yes, even if you're a solo developer working on a hobby project. Yes, even more so if you are a beginner.

Don't be afraid of source control, this is your most useful tool in the toolbox as a programmer, and the earlier you learn it, the more you will appreciate its many features and benefits.

There are many articles and threads outlining why you should use source control, but let me list a few points that I feel are relevant:

Source control (with a remote repository) acts as a backup for your code. If your hard drive or computer burns, you are safe. Even if you do not value your code much, having a backup can't hurt.
History and branching allow you to quickly explore several solutions and go back in time
Comments allow you to understand what you previously did and why. You may not believe me, but you will forget your own code sooner than you think. Here are some tips on how to write good commit messages for your future self and others.
You may have several computers on which you want to work on the project. Source control will make this easy.
One day your small project may grow. Perhaps more team members, perhaps you will sign a publishing deal and go big! You'll be very happy to have source control then.

Please note that Dropbox or any cloud drive equivalent is not a source control tool. Yes, it provides remote backup and some history, but it lacks all the tools and specific functionality that make source control software so useful.

Source control for game development

Thanks to Git and GitHub, source control has now become ubiquitous and collaboration has never been easier.

So, why is source control for game development so problematic? What makes game projects special?

Source control was mostly designed for code, which generally translates to text files. Game projects consist of code as well as game data, usually in the form of large binary files. Binary files generally cannot be diffed, nor merged. This hinders the possibility of using branches in the same fashion as you would with text files. It also requires exclusive locking mechanisms or conflict resolution.
Project size. Even the largest code-only projects remain manageable. Game projects can very quickly take more than a hundred of gigabytes, especially in the age of asset marketplaces. Some source control solutions struggle to handle large projects as they were not designed for this use case.
Non-technical team members. Code projects mostly involve programmers who can find their way around complex command-line tools such as Git. Game projects have many Artists and Designers who require a good user experience to properly use Source Control.

The available source control solutions for game development all have major drawbacks, whether it is price, complexity, user experience, or inability to handle large game projects. This makes the choice of source control very difficult.

This is why I created GitCentral. GitCentral allows using Git as a source control backend but solves the user experience, workflow, and technical issues by providing full integration in Unreal Engine.

Candidate choices and comparison axes

In this series, we will compare the major source control options, provide the information, and draw conclusions in order for you to make the best choice for your team.

In order to compare them properly, we will evaluate each option using the following criteria:

Features and source control model
Ability to support game projects, handling large projects and binary files
Iteration time
User Experience out of the box
Ecosystem of tools and integrations, notably with continuous integration
Cloud hosting providers
Price

Here are the source control solutions that we will evaluate:

Git, the ubiquitous source control software popularized by the open-source community and GitHub.
Subversion (SVN), another open-source staple. SVN is losing popularity to Git but is still one of the major source control systems out there.
Perforce, the AAA game industry standard choice.
Plastic SCM, a source control specifically designed for game development and large projects

There are other players to note in the source control field, such as Mercurial, Team Foundation, Alienbrain or CVS. We will touch briefly on them whenever relevant, but their reported low rates of adoption, lack of specific functionality relevant for game projects, and them being arguably inferior products to the ones mentioned above, makes them secondary choices. We will also explore engine-specific solutions for commercial engines. Of course, if there is an interesting solution I do not know about or you want me to dig deeper into, please contact me.

Stay tuned for the first article where we will break down Git's pros and cons for game development projects.

C++ Tricks: Fast RTTI and Dynamic Cast

Samuel Kahn — Tue, 08 Oct 2019 07:50:22 +0000

C++ Tricks is a series of posts on core libraries for game engines and language experiments shared as the Kahncode Core Libraries.

As introduced in the first post of these series, I will share the first piece of KCL: an implementation of RTTI and Dynamic Cast. The code can be found on GitHub.

If you don't know what dynamic casting is, then I suggest you read some online resources before diving into this article.

I will avoid debating whether or not you need RTTI. Some say that it correlates with bad architectural choices and violates the SOLID principles. However in practice most game engines have an extensive RTTI and introspection system and make wild use of it. Unreal Engine solves this issue using code generation in their object system.

Following the philosophy of KCL, our RTTI system will fulfill the following requirements:

Must support at least: dynamic cast, multiple inheritance, and retrieving the type name as a string.
As elegant syntax as possible. Only two macros are necessary. A one macro version is possible as well.
No code generation, use only the preprocessor and compiler.
Minimalist: there are less than 350 lines of code including all comments.
Efficient: one of the main complaints about STL RTTI and dynamic_cast is that is it slow. We must outperform the STL at least in runtime performance.
Avoid polluting the global namespace except for our version of dynamic_cast and the necessary macros.

The Solution

Before diving into the code, let's explain how we are going to solve this problem.

To better understand the following part, I will clarify some terminology. We will build the following:

TypeID: a unique identifier for a single type.
TypeData: the necessary information to perform a dynamic cast
TypeInfo: a structure which contains the TypeID, the type name and the TypeData

Plan of action:

We must add at least one virtual method to a polymorphic type, in order to retrieve the type information at runtime. This is unavoidable.
The type info itself should contain enough information to identify the type, its name, and all the necessary information to perform dynamic cast. The entirety of the information must be contiguous in memory for best performance, we will therefore avoid linked lists and pointers of any kind.
The cost of a dynamic_cast will therefore be that of calling a virtual method, loading the type info, and iterating over a contiguous region of memory.

TypeInfo breakdown

First let's create the TypeId by assigning one unique integer identifier for each type. For most programs 16 bits should be enough, so the typedef allows you to adjust the memory footprint of the RTTI system.

static typeId_t GenerateId()
{
    static typeId_t theTypeIdCounter = 0;
    return ++theTypeIdCounter;
}

We could build the TypeInfo as a simple array of all the Type IDs of the type and its base types. However there is one more information necessary to perform dynamic casts.

In a multiple inheritance scenario, the dynamic cast of a pointer may actually change the value of the pointer due to pointer adjustment. In order to understand this mechanic, let's take an example:

class A
{
    int i;
};
class B
{
    int j;
};
class C : A, B
{
    int k;
};

C* pC = new C; 
A* pA = (A*)pC; // No offset adjustment, pA == pB 
B* pB = (B*)pC; // Offset adjustment, pB != pA

The pointer difference can be explained when examining the memory layouts of these objects. In order to return a valid pointer to an instance of B, the compiler must return the adress of B in the memory layout of C.

The compiler may also introduce padding between the types, so the offset of B* is not necessarily A* + sizeof(A).

We will use the compiler itself to determine the offset accurately:

template<typename Derived, typename Base>
static ptrdiff_t ComputePointerOffset()
{
    Derived* derivedPtr = (Derived*)1;
    Base* basePtr = static_cast<Base*>(derivedPtr);
    return (intptr_t)basePtr - (intptr_t)derivedPtr;
}

Here is the final TypeInfo info design we come up with, containing all the necessary information. The data array is later implemented as TypeData<T>.

The data starts as a TypeID string with a size and an array of TypeIDs. The first type is the current type's TypeID. In the cases of multiple inheritance, it follows with more TypeID strings as well as their pointer offset adjustment information.

We can observe that the first list does not have an offset. This is because all first bases can be casted without pointer offset adjustment. We can therefore perform a slight optimization by omitting this offset value.

As none of the following lists should have 0 as pointer adjustment value, this allows us to treat a null offset value as the end marker to the chain.

The recursive nature of the TypeInfo will allow algorithms to walk through the data structure and extract the necessary information during dynamic cast.

Building the type info

The TypeInfo struct must not be templated, and contains all the necessary API to access the RTTI. A pointer to the static instance of this TypeInfo will be returned by the virtual function on the object.

struct TypeInfo
{
    const char* myName;

    const char* GetName() const 
    const char* GetTypeData() const 
    typeId_t GetTypeId() const 
    intptr_t CastTo(intptr_t aPtr, typeId_t aTypeId) const
    {
        const char* data = GetTypeData();
        size_t byteIndex = 0;
        ptrdiff_t offset = 0;

        while (true)
        {
            typeId_t size = *(typeId_t*)(data + byteIndex);
            byteIndex += sizeof(typeId_t);

            for (typeId_t i = 0; i < size; i++, byteIndex += sizeof(typeId_t))
            {
                if (*(typeId_t*)(data + byteIndex) == aTypeId)
                    return aPtr + offset;
            }

            offset = *(ptrdiff_t*)(data + byteIndex);
            if (offset == 0)
                return 0;

            byteIndex += sizeof(ptrdiff_t);
        }
    }
};

The TypeInfo contains the name, and assumes that is it followed in memory by the Type Data as described above. The type data starts with the current type's TypeID, which is returned by GetTypeId. The CastTo method iterates through the type data and tries to find the TypeID we are trying to dynamic cast to. If it is found, it performs the offset adjusment and returns the new pointer value.

The most complex part is still ahead of us: we must use the compiler to compute this Type Data. Variadic templates will allow us to fill the Type Data recursively:

template<typename T>
TypeData<T> : public TypeDataImpl<T, BaseTypes...>

Simple Type

Let's start with the most simple type, which does not inherit from anything:

template<typename Type>
struct TypeDataImpl<Type>
{
    TypeDataImpl() : mySize(1), myTypeId(GenerateId()), myEndMarker(0) 
    const char* GetData() const 
    typeId_t mySize;
    typeId_t myTypeId;
    ptrdiff_t myEndMarker;
};

We simply generate a new TypeID for this type and set the size to 1 and the end marker.

Single Inheritance

Let's examine now the case of a type with single inheritance:

template<typename Type, typename Base>
struct TypeDataImpl
{
    TypeDataImpl()
    {
        myTypeId = GenerateId();
        myBaseTypes.FillBaseTypeData<Type>(0 /* No offset with first base */, mySize);
        mySize++; // Size is the base's size + 1 to account for current type id
        myEndMarker = 0;
    }

    const char* GetData() const 

    typeId_t mySize;
    typeId_t myTypeId;
    BaseTypeData<Base> myBaseTypeData;
    ptrdiff_t myEndMarker;
};

Again we build the type data starting with a size and a fresh TypeID. However we will follow it by the type data of the base type. Therefore the size of the first TypeID list in the array is one more than the base type as we include the new type's Type ID.

In order to generate BaseTypeData we retrive the TypeDataImpl of the base type and iterate through it. Don't worry, this is the most complex part of the whole thing.

template<typename Base>
struct BaseTypeData<Base>
{
    void FillBaseTypeData(ptrdiff_t aOffset, typeId_t& outHeadSize)
    {
        // Retrieves the TypeData<Base> instance, explained below in the article
        const TypeData<Base>* baseTypeId = (TypeData<Base>*)(GetTypeInfo<Base>::Get()->GetTypeData());

        // return size of head list
        outHeadSize = baseTypeId->mySize;

        const char* data = baseTypeId->GetData();
        size_t byteSize = baseTypeId->mySize * sizeof(typeId_t);

        // copy type list
        memcpy(myData, data, byteSize);

        size_t byteIndex = byteSize;
        ptrdiff_t offset = *(ptrdiff_t*)(data + byteIndex);
        while (offset != 0)
        {
            // fill next offset and add pointer offset
            *(ptrdiff_t*)(myData + byteIndex) = offset + aOffset;
            byteIndex += sizeof(ptrdiff_t);

            // fill next size
            const typeId_t size = *(typeId_t*)(data + byteIndex);
            *(typeId_t*)(myData + byteIndex) = size;
            byteSize = size * sizeof(typeId_t);
            byteIndex += sizeof(typeId_t);

            // copy types
            memcpy(myData + byteIndex, data + byteIndex, byteSize);
            byteIndex += byteSize;

            offset = *(ptrdiff_t*)(data + byteIndex);
        }
    }

    // We only need the previous type data array, but not its size or end marker as we will insert them ourselves
    char myData[sizeof(TypeData<Base>) - sizeof(ptrdiff_t) - sizeof(typeId_t)];
};

This code iterates through the base type's data and fills the derived type's data. It finally stops when the offset found is 0, which denotes the end.

Interesting things to note:

It doesn't simply copy the data but it will add an extra potential offset to the base's data to account for pointer offset adjustment. This is why in the single inheritance example, we passed an offset of 0.
The size of the buffer constructed is based on the base types TypeData structure, of which we only need the internal array. We have added our own size and end markers in the TypeDataImpl above.

Multiple Inheritance

In a multiple inheritance scenario, we use variadic templates to build the BaseTypeData recursively:

template<typename Type, typename... BaseTypes>
struct TypeDataImpl
{
    TypeDataImpl()
    {
        myTypeId = GenerateId();
        myBaseTypeData.FillBaseTypeData<Type>(0 /* No offset with first base */, mySize);
        mySize++; // Size is the base's size + 1 to account for current type id
        myEndMarker = 0;
    }

    const char* GetData() const 

    typeId_t mySize;
    typeId_t myTypeId;
    BaseTypeData<BaseTypes...> myBaseTypeData;
    ptrdiff_t myEndMarker;
};

The base type data is built taking into account pointer offset. We can calculate it as the base types are unpacked in the variadic expansion, and the final derived type is passed as a template parameter to FillBaseTypeData. This is why we needed to use a static method and not a constructor. The sizes of the following lists are not incremented as no Type ID is added.

template<typename FirstBase, typename SecondBase, typename... Next>
struct BaseTypeData<FirstBase, SecondBase, Next...>
{
    template<typename Derived>
    void FillBaseTypeData(ptrdiff_t aOffset, typeId_t& outHeadSize)
    {
        myFirst.FillBaseTypeData<Derived>(ComputePointerOffset<Derived, FirstBase>(), outHeadSize);

        myOffset = ComputePointerOffset<Derived, SecondBase>();
        myNext.FillBaseTypeData<Derived>(myOffset, mySize);
    }

    BaseTypeData<FirstBase> myFirst;
    ptrdiff_t myOffset;
    typeId_t mySize;
    BaseTypeData<SecondBase, Next...> myNext;
};

One word of warning: this technique may be compromised by the compilers. Padding may be added which will not keep the type data contiguous in memory and the algorithms will fail. We can however prevent that using #pragma pack which is supported by most compilers, or other compiler-specific packing options.

Here is the mechanic visualized:

Registering a type

With the TypeDataImpl and TypeInfo taken care of, it is now time to implement the mechanic to instanciate these and retrieve them.

We need to assemble a TypeInfo with the name of the type followed by its TypeData<T>:

template<typename T>
struct TypeInfoImpl
{
    const TypeInfo myInfo;
    const TypeData<T> myData;
};

We can now use this in a type registration macro. The macro is needed to trigger template specialization and therefore trigger the instanciation of this TypeInfoImpl. Conveniently, we also use the macro to retrieve the type name.

First we declare tye TypeData<T> type, using macro expansion tricks to extract the first argument and inherit from TypeDataImpl<Type, BaseTypes...>.

#define KCL_RTTI_REGISTER(...)                                                     \
template<>                                                                         \
struct TypeData<KCL_FIRST_ARG(__VA_ARGS__)> : public TypeDataImpl<__VA_ARGS__> ; \
// continued...

Finally the static instance of the TypeInfo (contained in the TypeInfoImpl) is accessed through a static method on the specialization of the GetTypeInfo stuct.

// continuation... TYPE is the type being registered
template<>                                                                     \
struct GetTypeInfo<TYPE>                                                       \
{                                                                              \
    static const TypeInfo* Get()                                               \
    {                                                                          \
        static TypeInfoImpl<TYPE> ourInstance = ;   \
        return &ourInstance.myInfo;                                            \
    }                                                                          \
};

We can now retrieve the TypeInfo at compile time, but we also need it at runtime. For this we must use a virtual method.

This is declared in a second macro that must be used within the derived type declaration scope:

#define KCL_RTTI_IMPL()                                               \
                                                                      \
    virtual const TypeInfo* KCL_RTTI_GetTypeInfo() const              \
    {                                                                 \
        typedef std::remove_pointer<decltype(this)>::type ObjectType; \
        return GetTypeInfo<ObjectType>::Get();                        \
    }                                                                 \

Dynamic Cast

The goal is finally within our reach. We have built all the type information necessary to perform a dynamic cast and we can access it at runtime on an instance of a registered type.

However, there are still a few subtleties in the dynamic cast implementation:

We must not forget to handle the case of a null pointer. Interestingly, the pointer offset is not performed when the parameter of static_cast or dynamic_cast is called.
We can avoid the expensive dynamic_cast logic if we can safely call static_cast instead. We will use if constexpr here but SFINAE can be used as polyfill.
The offset present in the TypeData<T> is relative to the canonical pointer of the object, that is the pointer to an instance of its most derived type. If we hold a pointer to this object at an offset, we must perform the dynamic cast from the canonical this pointer. Fortunately the final overrider rule in the standard allows us to do this using a virtual method.

Here is the final implementation of dynamic_cast:

//continuation of KCL_RTTI_IMPL, this method is declared in the object and allows to always begin from the canonical "this" pointer
    virtual intptr_t KCL_RTTI_DynamicCast(typeId_t aOtherTypeId) const                \
    {                                                                                 \
        typedef std::remove_pointer<decltype(this)>::type ObjectType;                 \
        return GetTypeInfo<ObjectType>::Get()->CastTo((intptr_t)this, aOtherTypeId);  \
    }

//Implementation of dynamic_cast in the global namespace
template<typename Derived, typename Base>
Derived kcl_dynamic_cast(Base* aBasePtr)
{
    static_assert(std::is_pointer<Derived>::value, "Return type must be a pointer");
    typedef std::remove_pointer<Derived>::type DerivedObjectType;
    if constexpr (std::is_base_of<DerivedObjectType, Base>::value)
        return static_cast<Derived>(aBasePtr);
    else if (aBasePtr)
        return reinterpret_cast<Derived>(aBasePtr->KCL_RTTI_DynamicCast(GetTypeId<DerivedObjectType>()));
    else
        return nullptr;
}

And there we have it. The real implementation is slightly more complex to ensure more type safety and avoid polluting the global scope, but the principle is essentially the same.

Benchmark

We have done our best to design a RTTI system by packing all the information in a contiguous region of memory accessible at the cost of one virtual method call.

Let's see how it performs against the standard implementation and confirm that ours is indeed faster.

Each benchmark runs 3000000 dynamic_casts, and the measure is taken 10 times. Tests have shown that increasing those numbers significantly yields similar results so we will consider this enough to take conclusions.

Single Inheritance

Here we tested downcasting to the most derived type of a single inheritance chain of varying levels.

We can see the cost of dynamic_casting increasing with the number of base types. KCL performs better on all compilers.

Multiple Inheritance

Similar results can be obtained when examining multiple inheritance. The nested multiple inheritance case is interesting as some of its second bases themselves have second bases. This validates the pointer offset adjustment logic and is assumed to be one of the most complex cases for linked-list implementations to follow.

Once again KCL is dramatically faster in all scenarios.

Wrong casts

Here let's examine the cost of dynamic_cast to a type that is not part of the type's hierarchy. This is the very worst case as the algorithm has to iterate through the entire type information before failing to cast.

We will also test casts with a nullptr input value.

The results are similar again, KCL performs better and the speed gain is higher the more complex hierarchies are traversed. Nullptr cast is virtually the same except for MSVC where it is up to 5 times slower at 10ms!

Findings

Averaged over all the results, KCL performs more than 3.5 times faster with MSVC, more than 4.5 times faster with GCC, and more than 6 times faster with clang!
The cost of the standard implementations dramatically increases with the size of the hierarchy. For KCL it only increases linearly.
The initial cost of a dynamic_cast is always lower with KCL. We can assume retrieving the TypeInfo is more complex in the standard implementation.
clang performs consistently better than GCC and MSVC.
Interestingly, the compilers with the fastest implementation of dynamic cast also seem to perform better optimizations of KCL which results in an even bigger spead increase over their own implementation.
I did not report the metrics of upcasts that can be optimized with static_cast. This is because all compilers perform this successfully as well as KCL, compiling down to the exact same code.

Limitations and possible improvements

Our approach has some caveats and limitations as opposed to the standard library's dynamic cast. Let's try to list them all:

The TypeInfo is created on first access in the static method call. This means the startup performance should be less than the standard, which can compute embed all type info in the executable.
This is not thread safe. There could be race conditions in which the TypeID of a type is generated twice at the same time. This can be alleviated in various ways.
The static mechanic used to build the TypeID make this not safe to pass across module boundaries. This could be improved by generating a TypeID using a hash of the symbol name.
The TypeData architecture duplicates some information by design, and therefore uses more executable space than is stricly necessary. Using linked lists we could theoretically use less space but the runtime performance would be impacted significantly.
The design relies on compiler tricks and assumptions that may not hold true, especially with regards to padding. More static asserts, unit testing and instrumentation could be added to validate the integrity of the memory layout.
Behaviors related to std::bad_cast exception is not supported as opposed to the standard. If you really need it, it can be added.
In addition, we do not support dynamic casting references. This is because references should never be null, therefore a failed cast could break otherwise correct assumptions. The standard dynamic cast uses exceptions which we cannot use. This is however not a big issue as pointer logic can always be substituted.
You may have noticed I am using the std type traits. This is to allow you to easily port the code. If you do not want to depend on std at all you can reimplement this easily and better manage your compilation times.

As far as I'm concerned, these tradeoffs are worth the performance boost, and most of the issues can be alleviated by building upon this base.

If you find more issues with my code, please reach out to me, or better send a pull request!

I hope you enjoyed this breakdown of a possible RTTI implementation as much as I enjoyed making it. More is coming for KCL, so stay tuned for the next article.