Open Source Adventures: Episode 47: How BATTLETECH DLCs Store Data

Just after I finished the weapons stats script I noticed it lacked some of the weapons like UACs. Oh right, they're not in the base game, they're in the DLCs.

Most games store all data, "base game" data, and "DLC" data, the same way. It's either in the same place, with DLC parts just disabled with some flags; or it's in some dlc/whatever/ with the same structure.

BATTLETECH is the first game I've seen that decided to store DLC data in a completely different way from base game data.

How BATTLETECH base game data is stored

It's simply in BattleTech_Data\StreamingAssets\data, as loose JSON files.

DLC data is not stored like that. All files for each DLC are packaged together into a single UnityFS archive, such as BattleTech_Data/StreamingAssets/data/assetbundles/heavymetal.

For weapons we only need to access contents of heavymetal DLC. If we want to run analysis of mechs, flashpoint and urbanwarfare also seem to contain some.

Why game store data in archives?

It's very common for games to store files as archives instead of loose files. There is one very simple reason for it - Windows operating system has ridiculously poor performance at handling large numbers of small files.

On Linux there's essentially no difference between huge number of small files, or one big archive with same data, so such archives would be fairly pointless.

On Windows, where most games are ran, loose small files are a performance disaster, so games do what operating system should be doing and have those read only virtual filesystems.

How to unpack UnityFS archives with UnityPack

I tried UnityPack, but its dependency decrunch crashes if I try to pip3 install decrunch.

I used workaround suggested by the issue:

$ sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/malloc/malloc.h /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/malloc.h

Then I ran unityextract --all heavymetal... and it crashed anyway:

Written 369790 bytes to 'barrel.obj'
Traceback (most recent call last):
  File "/usr/local/bin/unityextract", line 159, in <module>
    main()
  File "/usr/local/bin/unityextract", line 155, in main
    exit(app.run())
  File "/usr/local/bin/unityextract", line 57, in run
    self.handle_asset(asset)
  File "/usr/local/bin/unityextract", line 106, in handle_asset
    self.write_to_file(d.name + ".cg", d.script)
  File "/usr/local/lib/python3.9/site-packages/unitypack/engine/object.py", line 6, in _inner
    ret = self._obj[f]
KeyError: 'm_Script'

OK, that crashed unpacking some object file, but I only want the JSONs.

So I tried unityextract --text heavymetal instead, and got a different crash:

Traceback (most recent call last):
  File "/usr/local/bin/unityextract", line 159, in <module>
    main()
  File "/usr/local/bin/unityextract", line 155, in main
    exit(app.run())
  File "/usr/local/bin/unityextract", line 57, in run
    self.handle_asset(asset)
  File "/usr/local/bin/unityextract", line 82, in handle_asset
    for id, obj in asset.objects.items():
  File "/usr/local/lib/python3.9/site-packages/unitypack/asset.py", line 84, in objects
    self.load()
  File "/usr/local/lib/python3.9/site-packages/unitypack/asset.py", line 110, in load
    self.tree.load(buf)
  File "/usr/local/lib/python3.9/site-packages/unitypack/type.py", line 117, in load
    self.target_platform = RuntimePlatform(buf.read_uint())
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/enum.py", line 384, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/enum.py", line 702, in __new__
    raise ve_exc
ValueError: 520093696 is not a valid RuntimePlatform

OK, it's clear that's not going to work.

So what's next?

As the files are not directly available and I couldn't get UnityPack working, I have a few options:

• look for other tools
• implement UnityFS unpacker myself - the format looks simple enough, but all the UnityPack crashes suggest that maybe it's not so simple after all
• try fixing UnityPack - tbh fixing other people's code is often harder than just writing it all from scratch
• see if anyone extracted JSONs and provides them for download
• just drop the project