DevLog 20250916: [Issue] Binary Design Project Version Control

#design #devjournal #tooling

Overview

Not having a way to back up and "go back to any time" has the issue of:

Not having a sense of security of "completely restore" something
Not having point-in-time reference, and thus needing to make manual "progress records"
Generally hurdle iteration since we need to worry about "keeping old stuff" for latter use

Existing "Strategy" (Not Working)

We could pack everything in VHDX and archive the whole thing on cloud store (like our backup strategy) or HHD, but it's not efficient, especially since design projects can easily become very big - that creates the issue of having to carefully plan "revision points".

The Verdict

We don't want to waste time expanding cv and develop our own - it seemed Perforce is a must have. On the other hand, it's time to try git + LFS again, if we use it carefully, it's certainly better than having no version control at all for design projects.

On the other hand, Fossil is a great single-user alternative that neatly stores everything in a single SQLite file. Fossil doesn’t attempt to diff or merge binaries; it just versions them, with history, commit notes and tags. Its main limitations in:

SQLite BLOB Size Limit: SQLite, and therefore Fossil, has a practical limit on the size of individual BLOBs, which is around 2GB. This means individual binary files larger than this limit cannot be directly stored within Fossil.
Performance with Numerous Large Binaries: While Fossil can handle binary files, a large number of very large binary files can impact performance, especially during operations like cloning, syncing, and committing, due to the single-file repository structure and the need to manage large BLOBs.

Compromised Strategy

Strategies for Managing Large Binary Files:

Unversioned Files: If historical versions of certain large binary assets are not critical, they can be managed outside of Fossil as unversioned files.
Repository Segmentation: For projects with a substantial amount of large binary data alongside source code, consider separating the project into multiple Fossil repositories. One repository could manage the source code and smaller assets, while another could handle the large binaries, potentially with scripting to maintain alignment.

Repository segmentation is slightly easier to do with Fossil, but still doable with Git (just ignore and don't commit things as submodules).

The Experimental Proposal

Since most of my game projects already use Git, and it's a bit confusing to add LFS, I decide to give fossil another try, specifically for overall (non-code) binary version control.

So the final toolset looks like this: