A lightweight kv storage engine based on Golang.

Leon Ding — Sat, 19 Mar 2022 09:43:12 +0000

What is Bottle🤔?

👨🏻‍💻: Bottle is a lightweight kv storage engine based on LSM-TREE.

Project Home pages: https://bottle.ibyte.me/

Project Introduction

The first thing to note is that Bottle is a KV embedded storage engine, not a KV database.

Other

I am a novice, self-taught go language programming, and then realized this project through the bitcask paper, I hope you can give me a star, thank you, my English is not very good.

The function implementation of this project is completely based on the bitcask paper.

Interested friends can go and see this course. If you think it is good, you can give me a small star ⭐ Thank you.

Install Module🚀

You just need to install the Bottle module in your project to use:

go get -u github.com/auula/bottle

Bottle Source code🌲

.
├── LICENSE
├── README.md
├── bottle-logo.svg
├── bottle.go           # Main File
├── bottle_test.go      
├── encoding.go         # Data encoding module
├── encrypted.go        # Data encrypted module
├── encrypted_test.go
├── example             # Exmaple code
│   ├── config.yaml
│   ├── demo_exchange.go
│   └── demo_put_get.go
├── go.mod
├── go.sum
├── gopher-bottle.svg
├── hashed.go          # Hashed function
├── item.go            # Data log Entry
└── option.go          # Bottle Configure 

1 directory, 17 files

Data Directory 📑

Since the bottle design is based on a single-process program, each storage instance corresponds to a data directory, data is the log merge structure data directory, and index is the index data version.

Log consolving structural data The current version is merged every time the data is started. The default is that all data files under the DATA data folder occupies the total and more than 1GB of more than 1GB will trigger a merge, and the data that is not used after the merger is discarded.

Of course, if the dirty data merge requirements are not reached, the data file is archived in the size of the configured, each data has a version number, and is set to read-only mount, the process work directory structure is as follows:

./testdata
├── data
│   └── 1.data
└── index
    ├── 1646378326.index
    └── 1646378328.index

2 directories, 3 files

When the storage engine starts working, the folder and files can only be operated by this process to ensure data security.

Basic API ⚙

Below is a usage example of a basic data manipulation I wrote:

package main

import (
    "fmt"
    "github.com/auula/bottle"
)

func init() {
    // Open a storage instance by default configuration
    err := bottle.Open(bottle.DefaultOption)
    // And handle possible errors
    if err != nil {
        panic(err)
    }
}

// Userinfo test data structure
type Userinfo struct {
    Name  string
    Age   uint8
    Skill []string
}

func main() {

    // PUT Data
    bottle.Put([]byte("foo"), []byte("66.6"))

    // If it is converted to string, it is a string
    fmt.Println(bottle.Get([]byte("foo")).String())

    // If there is no default, it is 0
    fmt.Println(bottle.Get([]byte("foo")).Int())

    // If it is not success, it is false.
    fmt.Println(bottle.Get([]byte("foo")).Bool())

    // If it is not success, it is 0.0
    fmt.Println(bottle.Get([]byte("foo")).Float())

    user := Userinfo{
        Name:  "Leon Ding",
        Age:   22,
        Skill: []string{"Java", "Go", "Rust"},
    }

    var u Userinfo

    // Save the data object by BSON, and set the timeout for 5 seconds
    // TTL timeout can not set demand
    bottle.Put([]byte("user"), bottle.Bson(&user), bottle.TTL(5))

    // Analysis of structures by unwrap
    bottle.Get([]byte("user")).Unwrap(&u)

    // Print value
    fmt.Println(u)

    // Remove key 'foo'
    bottle.Remove([]byte("foo"))

    // Close handleable error that may happen
    if err := bottle.Close(); err != nil {
        fmt.Println(err)
    }
}

The following is a separate variable assignment and catch handling errors:

    data := bottle.Get([]byte("user"))

    if data.IsError() {
        fmt.Println(data.Err)
    } else {
        fmt.Println(data.Value)
    }

Encryptor🔐

The data encryptor records the value of the data, that is, the block encryption at the field level, instead of encrypting the entire file once, which will cause performance consumption, so the block data segment encryption method is adopted.

The following example uses the bottle.SetEncryptor(Encryptor,[]byte) function to set the data encryptor and configure the 16-bit data encryption key.

func init() {
    bottle.SetEncryptor(bottle.AES(), []byte("1234567890123456"))
}

You can also customize the interface to implement the data encryptor:

// SourceData for encryption and decryption
type SourceData struct {
    Data   []byte
    Secret []byte
}

// Encryptor used for data encryption and decryption operation
type Encryptor interface {
    Encode(sd *SourceData) error
    Decode(sd *SourceData) error
}

The following code is the implementation code of the built-in AES encryptor, just implement the bottle.Encryptor interface, and the data source is the bottle.SourceData structure field:

// AESEncryptor Implement the Encryptor interface
type AESEncryptor struct{}

// Encode source data encode
func (AESEncryptor) Encode(sd *SourceData) error {
    sd.Data = aesEncrypt(sd.Data, sd.Secret)
    return nil
}

// Decode source data decode
func (AESEncryptor) Decode(sd *SourceData) error {
    sd.Data = aesDecrypt(sd.Data, sd.Secret)
    return nil
}

The specific encryptor implementation code can be viewed in encrypted.go

Hashed 🧰

If you need to customize the salad function, you can implement the bottle.Hashed interface:

type Hashed interface {
    Sum64([]byte) uint64
}

Then complete your hash function configuration by built-in bottle.SetHashFunc(hash Hashed) settings.

Index Size 🔎

The index pre-set size will greatly affect your program acception and read data. If you can expect the index size required to run during initialization, and in initialization, you can decreaseThe performance issues that run data migration and expansion brought by the program during operation.

func init() {
    // setting indexes size
    bottle.SetIndexSize(1000)
}

Configuration Information

You can also use the default configuration, you can initialize your storage engine with the structure of the built-in bottle.Option, and the configuration instance is as follows:

func init() {
    // Custom configuration information
    option := bottle.Option{
        // Working directory
        Directory:       "./data",
        // Algorithm opens encryption
        Enable:          true,
        // Customize the secret, you can use a built-in secret
        Secret:          bottle.Secret,
        // Custom data size, storage unit is KB
        DataFileMaxSize: 1048576,
    }
    // Custom configuration information
    bottle.Open(option)
}

Of course, you can also use the built-in bottle.Load(path string) function to load the configuration file to start Bottle, the configuration file format is YAML, the configurable items are as follows:

# Bottle config options
Enable: TRUE
Secret: "1234567890123456"
Directory: "./testdata"
DataFileMaxSize: 536870912

It is important to note that the key implemented by the built-in encryptor must be 16 bits, if you are a custom implementation encrypler to set your custom encryprators through bottle.SetEncryptor(Encryptor,[]byte), then this secretThe number of digits will not be restricted.

Vision 🎉

If you have any questions about this project, you can contact me from the GitHub home page, you can also initiate a pull request, after your code is merged, you will appear in the following list~

DEV Community: Leon Ding