DEV Community: WangXiangUSTC

tidb-lite: used for unit testing of Golang database-related code

WangXiangUSTC — Sun, 14 Feb 2021 16:37:29 +0000

Use tidb-lite to run TiDB in mocktikv mode in Golang code.

tidb-lite can be used for unit testing of database-related code. If your application uses MySQL/TiDB, there will be a lot of database-related code to be unit tested.

In addition, some applications hope to persist some data locally and store it in a database to facilitate data management. Tidb-lite can also be used in this scenario. Developers use SQL compatible with MySQL protocol to process data locally.

This article mainly introduces how to use tidb-lite for unit testing of database-related codes. So how to write unit tests for database-related code?

The simple way is to start a database instance during the unit test, but this is not very elegant, because the unit test has certain requirements on the environment.

Another common method is to mock SQL services in the test. At present, the more popular solution is to use go-sqlmock.

The problem with go-sqlmock

First, let's take a look at how to use go-sqlmock for unit testing of database-related code.

For example, we have the following codes:

package main

import (
    "database/sql"

    _ "github.com/go-sql-driver/mysql"
)

func recordStats(db *sql.DB, userID, productID int64) ( err error) {
    tx, err := db.Begin()
    if err != nil {
        return
    }

    defer func() {
        switch err {
        case nil:
            err = tx.Commit()
        default:
            tx.Rollback()
        }
    }( )

    if _, err = tx.Exec("UPDATE products SET views = views + 1"); err != nil {
        return
    }
    if _, err = tx.Exec("INSERT INTO product_viewers (user_id, product_id) VALUES (? , ?)", userID, productID); err != nil {
        return
    }
    return
}

func main() {
    // @NOTE: the real connection is not required for tests
    db, err := sql.Open("mysql", "root@/blog")
    if err != nil {
        panic(err)
    }
    defer db.Close()

    if err = recordStats(db, 1 /*some user id*/, 5 /*some product id*/); err != nil {
        panic(err)
    }
}

The recordStats function will increase the number of product views by one and add the user to the list of viewers of the product. The code for unit testing this function using go-sqlmock is as follows:

package main

import (
    "fmt"
    "testing"

    "github.com/DATA-DOG/go-sqlmock"
)

// a successful case
func TestShouldUpdateStats( t *testing.T) {
    db, mock, err := sqlmock.New()
    if err != nil {
        t.Fatalf("an error'%s' was not expected when opening a stub database connection", err)
    }
    defer db.Close()

    mock.ExpectBegin()
    mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
    mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3). WillReturnResult(sqlmock.NewResult(1, 1))
    mock.ExpectCommit()

    // now we execute our method
    if err = recordStats(db, 2, 3); err != nil {
        t.Errorf("error was not expected while updating stats: %s", err)
    }

    // we make sure that all expectations were met
    if err := mock.ExpectationsWereMet(); err != nil {
        t.Errorf("there were unfulfilled expectations: %s", err )
    }
}

go-sqlmock needs to define each step of the database operation and sequence in advance (including the begin and commit of executing the transaction). If the actual operation or steps are inconsistent, an error will be reported. And it is necessary to define the expected return data. If the table is more complicated, it is very troublesome.

In fact, our unit test may only need to pay attention to whether the return result of this function is correct, not the order of execution of each operation of the function.

The advantages of tidb-lite

You can use tidb-lite instead of go-sqlmock.

Simplicity

One of the most important advantages of tidb-lite is simplicity. Run a TiDB directly in the code instead of running a MySQL/TiDB instance before running the unit test. This ensures that the unit test does not depend on the external environment; in addition, we don’t need to write a lot of redundancy and boring test code like go-sqlmock but focus on the correctness of the function.

Compatible with MySQL protocol

TiDB is highly compatible with MySQL protocol. Using tidb-lite can almost completely simulate the MySQL environment.

The usage of tidb-lite

For example, we have the following codes:

package example

import (
    "context"
    "database/sql"
    "fmt"

    "github.com/pingcap/errors"
    "github.com/pingcap/log "
    "go.uber.org/zap"
)

// GetRowCount returns row count of the table.
// if not specify where condition, return total row count of the table.
func GetRowCount(ctx context.Context, db *sql.DB , schemaName string, tableName string, where string) (int64, error) {
    /*
        select count example result:
        mysql> SELECT count(1) cnt from `test`.`itest` where id> 0;
        +----- -+
        | cnt |
        +------+
        | 100 |
        +------+
    */

    query := fmt.Sprintf("SELECT COUNT(1) cnt FROM `%s`.`%s` ", schemaName, tableName)
    if len(where)> 0 {
        query += fmt.Sprintf(" WHERE %s", where)
    }
    log.Debug("get row count", zap.String("sql", query) )

    var cnt sql.NullInt64
    err := db.QueryRowContext(ctx, query).Scan(&cnt)
    if err != nil {
        return 0, errors.Trace(err)
    }
    if !cnt.Valid {
        return 0, errors.NotFoundf ("table `%s`.`%s`", schemaName, tableName)
    }

    return cnt.Int64, nil
}

GetRowCount is used to get the number of eligible rows in the table, the unit test code of this function using tidb-lite is as follows:

package example

import (
    "context"
    "testing"
    "time"

    tidblite "github.com/WangXiangUSTC/tidb-lite"
    . "Github.com/pingcap /check"
)

func TestClient(t *testing.T) {
    TestingT(t)
}

var _ = Suite(&testExampleSuite{})

type testExampleSuite struct{}

func (t *testExampleSuite) TestGetRowCount(c *C) {
    tidbServer, err: = tidblite.NewTiDBServer(tidblite.NewOptions(c.MkDir()))
    c.Assert(err, IsNil)

    dbConn, err := tidbServer.CreateConn()
    c.Assert(err, IsNil)

    ctx, cancel := context.WithTimeout (context.Background(), 10*time.Second)
    defer cancel()

    _, err = dbConn.ExecContext(ctx, "create database example_test")
    c.Assert(err, IsNil)
    _, err = dbConn.ExecContext(ctx , "create table example_test.t(id int primary key, name varchar(24))")
    c.Assert(err, IsNil)
    _, err = dbConn.ExecContext(ctx, "insert into example_test.t values(1, ' a'),(2,'b'),(3,'c')")
    c.Assert(err, IsNil)

    count, err := GetRowCount(ctx, dbConn, "example_test", "t", "id > 2")
    c.Assert(err, IsNil)
    c.Assert(count , Equals, int64(1))

    count, err = GetRowCount(ctx, dbConn, "example_test", "t", "")
    c.Assert(err, IsNil)
    c.Assert(count, Equals, int64(3))
    tidbServer.Close()

    tidbServer2, err := tidblite.NewTiDBServer(tidblite.NewOptions(c.MkDir()))
    c.Assert(err, IsNil)
    defer tidbServer2.Close()

    dbConn2, err := tidbServer2.CreateConn()
    c .Assert(err, IsNil)
    _, err = dbConn2.ExecContext(ctx, "create database example_test")
    c.Assert(err, IsNil)
}

First, we use NewTiDBServer to create a TiDB instance and use CreateConn to get a link to this database. And then you can use this link to access the database, generate test data, and verify the correctness of the function. You can see README for detailed usage.

go-mysql-mongodb: replicate data from MySQL to MongoDB

WangXiangUSTC — Tue, 09 Feb 2021 15:28:11 +0000

This year’s Spring Festival holiday is relatively idle, so I organized my open source project go-mysql-mongodb.

This tool is used to synchronize MySQL data to MongoDB. It has been developed for a long time, but there has been no maintenance. A few days ago, I suddenly received an email from a user asking about the problems encountered during use. I realized that this tool is still used by some people. I took advantage of the holiday to maintain it, and hope that I can help more people in the future.

Origins

This project goes back to 2017. At that time, my job was mainly to investigate various big data platforms, and I needed to synchronize MySQL data to databases such as Elasticsearch and MongoDB.

tungsten-replicator

I searched for some solutions on Google. At first, I used tungsten-replicator to synchronize MySQL data to MongoDB. This tool is powerful and used for data synchronization between multiple heterogeneous databases. For example, the following is the topology diagram of tungsten-replicator synchronizing MySQL/Oracle data to heterogeneous databases:

Workflow:

Deploy a Replicator service on the Master and Slave servers respectively.
Master Replicator pulls binlog/CDC data from Master DB and converts it into a common THL format data.
Master Replicator transfers THL data to Slave Replicator.
Slave Replicator converts THL data into SQL and synchronizes it to Slave DB according to the type of Slave DB.

This tool is relatively mature, you can see their Usage Document, a total of more than 300 pages, various scenarios have instructions. But there are mainly these problems in the process of use:

The structure is too heavy. As you can see from the above example, you need to deploy one on the Master and Slave servers, and you need to save THL data, which will take up a lot of disks.
The deployment and configuration are more complicated. It supports very rich functions, but its powerful functions also require complex configurations to support. But in fact, we only need to use a small part of the functions.
Development language (not counting the problem of this tool). This tool was developed in Java, and my group at the time mainly used Python and Golang to develop. If there are new requirements or bugs that need to be fixed, maintenance will be more difficult.

go-mysql-elasticsearch

later I needed to synchronize MySQL data to Elasticsearch, I found another tool go-mysql-elasticsearch, try it After a while, I found that this tool is relatively lightweight and simple to configure and deploy. The workflow of this tool is as follows:

Use mysqldump to export the full data of MySQL.
Import the full amount of data into Elasticsearch.
Pull MySQL binlog data from the binlog postion position of the full data.
Convert binlog to Elasticsearch format data, and synchronize to Elasticsearch in the form of RESTful API.

It can be seen that this tool is relatively lightweight, and only needs to deploy one service for the synchronization of a MySQL instance; in addition, it is developed in the Golang language, which I am familiar with. I can also implement it myself when I encounter new requirements during use.

go-mysql-mongodb

After successfully applied go-mysql-elasticsearch to the quasi-production environment, I got the idea of replacing tungsten-replicator.

MongoDB and Elasticsearch are similar, both belong to NoSQL, and the stored data is all document type. So I reused most of the logic in go-mysql-elasticsearch, just modify the code of the Elasticsearch client in the code to MongoDB and it can basically run. In this way, the project go-mysql-mongodb was formed.

go-mysql-mongodb function

As go-mysql-mongodb mainly refers to go-mysql-elasticsearch, the function is basically the same.

To configure the data source,

you must set which tables to synchronize MySQL to MongoDB. Example configuration:

[[source]]
schema = "test"
tables = ["t1", t2]

[[source]]
schema = "test_1"
tables = ["t3", t4]

It also supports some simple expressions, such as:

[[source]]
schema = "test"
tables = ["test_river_[0-9 ]{4}"]

In this way, tables like test_river_0001 and test_river_0002 in the test library are selected.

Conversion rules

go-mysql-mongodb supports synchronizing tables in MySQL to collections specified in MongoDB, and also support conversion of field names in tables, for example:

[[rule]]
schema = "test"
table = " t1"
database = "t"
collection = "t"

    [rule.field]
    mysql = "title"
    mongodb = "my_title"

This configuration will synchronize the table test.t1 in MySQL to collection t.t in MongoDB, the title field in the table will also be renamed to my_title.

Filter fields

go-mysql-mongodb support to synchronize only the fields specified in the table, for example:

[[rule]]
schema = "test"
table = "tfilter"
database = "test"
collection = "tfilter"

# Only sync following columns
filter = ["id", "name"]

This configuration will only synchronize the data in the two columns of id and name in the table test.tfilter.

For more functions, please refer to the README of the project.

go-mysql-mongodb Status Quo

This project is still under development. The basic functions should be fine, but more tests are needed to ensure it.

In addition, we need to pay attention to the changes of go-mysql-elasticsearch and bring some fixes to go-mysql-mongodb in time.

Hope this gadget can help you. If you encounter problems during use, you can raise an issue, or you can directly email me wx347249478 at gmail.com.

chaos-mesh-action: Integrate Chaos Engineering into Your CI

WangXiangUSTC — Mon, 21 Sep 2020 03:10:32 +0000

This article was originally published at www.chaos-mesh.org on September 18, 2020.

Chaos Mesh is a cloud-native chaos testing platform that orchestrates chaos in Kubernetes environments. While it’s well received in the community with its rich fault injection types and easy-to-use dashboard, it was difficult to use Chaos Mesh with end-to-end testing or the continuous integration (CI) process. As a result, problems introduced during system development could not be discovered before the release.

In this article, I will share how we use chaos-mesh-action, a GitHub action to integrate Chaos Mesh into the CI process.

chaos-mesh-action is available on GitHub market, and the source code is on GitHub.

Design of chaos-mesh-action

GitHub Action is a CI/CD feature natively supported by GitHub, through which we can easily build automated and customized software development workflows in the GitHub repository.

Combined with GitHub actions, Chaos Mesh can be more easily integrated into the daily development and testing of the system, thus guaranteeing that each code submission on GitHub is bug-free and won’t damage existing code. The following figure shows chaos-mesh-action integrated into the CI workflow:

Using chaos-mesh-action in GitHub workflow

chaos-mesh-action works in Github workflows. A GitHub workflow is a configurable automated process that you can set up in your repository to build, test, package, release, or deploy any GitHub project. To integrate Chaos Mesh in your CI, do the following:

Design a workflow.
Create a workflow.
Run the workflow.

Design a workflow

Before you design a workflow, you must consider the following issues:

What functions are we going to test in this workflow?
What types of faults will we inject?
How do we verify the correctness of the system?

As an example, let’s design a simple test workflow that includes the following steps:

Create two Pods in a Kubernetes cluster.
Ping one pod from the other.
Use Chaos Mesh to inject network delay chaos and test whether the ping command is affected.

Create the workflow

After you design the workflow, the next step is to create it.

Navigate to the GitHub repository that contains the software you want to test.
To start creating a workflow, click Actions, and then click the "New workflow" button:

A workflow is essentially the configuration of jobs that take place sequentially and automatically. Note that the jobs are configured in a single file. For better illustration, we split the script into different job groups as shown below:

Set the workflow name and trigger rules.

This job names the workflow "Chaos.” When the code is pushed to the master branch or a pull request is submitted to the master branch, this workflow is triggered.

    name: Chaos

    on:
     push:
       branches:
         - master
     pull_request:
       branches:
         - master

Install the CI-related environment.

This configuration specifies the operating system (Ubuntu), and that it uses helm/kind-action to create a Kind cluster. Then, it outputs related information about the cluster. Finally, it checks out the GitHub repository for the workflow to access.

    jobs:
     build:
       runs-on: ubuntu-latest
       steps:

       - name: Creating kind cluster
         uses: helm/kind-action@v1.0.0-rc.1

       - name: Print cluster information
         run: |
           kubectl config view
           kubectl cluster-info
           kubectl get nodes
           kubectl get pods -n kube-system
           helm version
           kubectl version

       - uses: actions/checkout@v2

Deploy the application.

In our example, this job deploys an application that creates two Kubernetes Pods.

    - name: Deploy an application
         run: |
           kubectl apply -f https://raw.githubusercontent.com/chaos-mesh/apps/master/ping/busybox-statefulset.yaml

Inject chaos with chaos-mesh-action.

    - name: Run chaos mesh action
        uses: chaos-mesh/chaos-mesh-action@xiang/refine_script
        env:
          CFG_BASE64: YXBpVmVyc2lvbjogY2hhb3MtbWVzaC5vcmcvdjFhbHBoYTEKa2luZDogTmV0d29ya0NoYW9zCm1ldGFkYXRhOgogIG5hbWU6IG5ldHdvcmstZGVsYXkKICBuYW1lc3BhY2U6IGJ1c3lib3gKc3BlYzoKICBhY3Rpb246IGRlbGF5ICMgdGhlIHNwZWNpZmljIGNoYW9zIGFjdGlvbiB0byBpbmplY3QKICBtb2RlOiBhbGwKICBzZWxlY3RvcjoKICAgIHBvZHM6CiAgICAgIGJ1c3lib3g6CiAgICAgICAgLSBidXN5Ym94LTAKICBkZWxheToKICAgIGxhdGVuY3k6ICIxMG1zIgogIGR1cmF0aW9uOiAiNXMiCiAgc2NoZWR1bGVyOgogICAgY3JvbjogIkBldmVyeSAxMHMiCiAgZGlyZWN0aW9uOiB0bwogIHRhcmdldDoKICAgIHNlbGVjdG9yOgogICAgICBwb2RzOgogICAgICAgIGJ1c3lib3g6CiAgICAgICAgICAtIGJ1c3lib3gtMQogICAgbW9kZTogYWxsCg==

With chaos-mesh-action, the installation of Chaos Mesh and the injection of chaos complete automatically. You simply need to prepare the chaos configuration that you intend to use to get its Base64 representation. Here, we want to inject network delay chaos into the Pods, so we use the original chaos configuration as follows:

    apiVersion: chaos-mesh.org/v1alpha1
    kind: NetworkChaos
    metadata:
     name: network-delay
     namespace: busybox
    spec:
     action: delay # the specific chaos action to inject
     mode: all
     selector:
       pods:
         busybox:
           - busybox-0
     delay:
       latency: "10ms"
     duration: "5s"
     scheduler:
       cron: "@every 10s"
     direction: to
     target:
       selector:
         pods:
           busybox:
             - busybox-1
       mode: all

You can obtain the Base64 value of the above chaos configuration file using the following command:

    $ base64 chaos.yaml

Verify the system correctness.

In this job, the workflow pings one Pod from the other and observes the changes in network delay.

    - name: Verify
         run: |
           echo "do some verification"
           kubectl exec busybox-0 -it -n busybox -- ping -c 30 busybox-1.busybox.busybox.svc

Run the workflow

Now that the workflow is configured, we can trigger it by submitting a pull request to the master branch. When the workflow completes, the verification job outputs of the results that look similar to the following:

do some verification
Unable to use a TTY - input is not a terminal or the right kind of file
PING busybox-1.busybox.busybox.svc (10.244.0.6): 56 data bytes
64 bytes from 10.244.0.6: seq=0 ttl=63 time=0.069 ms
64 bytes from 10.244.0.6: seq=1 ttl=63 time=10.136 ms
64 bytes from 10.244.0.6: seq=2 ttl=63 time=10.192 ms
64 bytes from 10.244.0.6: seq=3 ttl=63 time=10.129 ms
64 bytes from 10.244.0.6: seq=4 ttl=63 time=10.120 ms
64 bytes from 10.244.0.6: seq=5 ttl=63 time=0.070 ms
64 bytes from 10.244.0.6: seq=6 ttl=63 time=0.073 ms
64 bytes from 10.244.0.6: seq=7 ttl=63 time=0.111 ms
64 bytes from 10.244.0.6: seq=8 ttl=63 time=0.070 ms
64 bytes from 10.244.0.6: seq=9 ttl=63 time=0.077 ms
……

The output indicates a regular series of 10-millisecond delays that last about 5 seconds each. This is consistent with the chaos configuration we injected into chaos-mesh-action.

Current status and next steps

At present, we have applied chaos-mesh-action to the TiDB Operator project. The workflow is injected with the Pod chaos to verify the restart function of the specified instances of the operator. The purpose is to ensure that tidb-operator can work normally when the pods of the operator are randomly deleted by the injected faults. You can view the TiDB Operator page for more details.

In the future, we plan to apply chaos-mesh-action to more tests to ensure the stability of TiDB and related components. You are welcome to create your own workflow using chaos-mesh-action.

If you find a bug or think something is missing, feel free to file an issue, open a pull request (PR), or join us on the #project-chaos-mesh channel in the CNCF slack workspace.