That's a lot of data!
While creating a roguelike game in golang I got to the point of needing to save the objects. Naturally my first thought was to use a database, and of course I didn't want to use something heavy and external, but rather light and embeddable, so I chose storm, a toolkit for bbolt. bbolt is a key/value datastore that writes to disk. But when one Screen's worth of objects is a few hundred MB, and you have 240 Screens in the world, well... the query results tend to get mangled after saving just 2-3 Screens. I had very weird results wherein the last 12% to 40% of the results from one Screen query weren't being processed correctly. Combine this with the query taking a few seconds due to the DB easily growing by 300MB for each Screen added, and it becomes apparent that a different method is needed.
The world is flat!
Enter the flat-file database. Instead of one giant datastore, I decided that each Screen would be a named folder, and inside there would be a separate file for each object. This yielded much better speeds for the program. For this method I used scribble. This project uses JSON as the file structure, but I don't need the files to be human-readable so I modified it to use encoding/binary. There is a library that was created to be able to use encoding/binary as a drop-in replacement of encoding/json by implementing the Marshaler and Unmarshaler interfaces: github.com/kelindar/binary
By using encoding/binary instead of encoding/json the file sizes are reduced and there's also a bit of a performance increase.
Obviously I need to tune the program to not use so much space, but I like to test things under a high load to see how it performs, and in this case it paid off.
My version of Scribble can be found here:
https://github.com/nanohard/scribble
Top comments (1)
Pretty cool, have you thought about using image file formats for each screen even if you use greyscale? And snappy compression algorithm would help out with the space.