FastDB, a speedy persistent KV store

Introduction

I've been using several KV (key-value) store solutions, but was never satisfied with the speed. We all know that the database-part of any application is most of the times the weakest link.

So what if I could make this faster as the fastest one I was using? Would the application that I am working on, benefit a lot? The answer is yes, it does. So now my application runs faster, smoother than before.

Semi-technical stuff

The fastest key-value store is just a simple map, I think. But that was not enough, because I wanted to be able to use buckets. (Buckets are a kind of a "box" in which you store key-values of the same kind.)

So I made the map like this:
map[string]map[int][]byte

I tried the basics of this against the usual solution and it was faster. But I still had to make a solution where the data would be stored on disk. I saw some AOF (append only file) solution somewhere, but wasn't satisfied with the speed, so I've tweaked that a bit more.

So now, when you open the file, you tell it to synchronise the data to disk every X milliseconds. (I would suggest 100.)

Since this is only a KV-store, there is no backed-in mechanism for sorting the output. Perhaps I will make one later, but for now there is a solution which is quite fast.

Example (pseudo code)

Here you can see how to open a database and store data in a bucket "texts".

store, _ := fastdb.Open("your.db", 100)

record := &someRecord{ID: 1, UUID: "UUIDtext", Text: "a text"}
recordData, _ := json.Marshal(record)

_ = store.Set("texts", record.ID, recordData)

Here you can see how to get data.

memData, ok := store.Get("texts", 1)
memRecord := &someRecord{}
_ = json.Unmarshal(memData, &memRecord)

Github link

https://github.com/marcelloh/fastdb