Integration tests with dockertest and Go

I recently had the task of writing some integration tests for my Go services and stumbled upon dockertest.

https://github.com/ory/dockertest

Dockertest allows you to run your Go integration tests against third-party services. Instead of mocking database interactions, you can run your tests against a real database that is destroyed after testing. The dockertest library makes all this really straight forward.

I keep my Go services within a mono repo and wanted to be able to share as much setup code as possible within each service. I currently keep all library code in a directory named support in the root of the repository, within this support package I have a database package, this is where my dockertest setup resides.

I store a *sql.DB exported var within the database package, this can then be used in the individual service tests and they can use the connection to interact with the database docker container.

type dockerDBConn struct {
    Conn *sql.DB
}

var (
    // DockerDBConn holds the connection to our DB in the container we spin up for testing.
    DockerDBConn *dockerDBConn
)

I make use of the TestMain function within Go, for those of you who haven't used TestMain it's a function that allows you to perform whatever setup and teardown that is needed for your tests, here we can do some setup such as initialize the database container we want to run our tests against, run the tests and return the exit code to see whether they passed or failed.

func TestMain(m *testing.M) {
    pool, resource := initDB()
    code := m.Run()
    closeDB(pool, resource)
    os.Exit(code)
}

The next stage is rather long and has been split into four functions. First, it sets up the Postgres config, then creates the database container and attempts to ping the DB to ensure we have a connection. Once we've established a connection we run a few migrations that are needed for our service, set the *sql.DB var to the new database URL and return the pool and resource so we can close it within our TestMain method when we've finished testing.

In dockertest the pool represents a connection to the Docker API and is used to create and remove docker images and the resource represents the docker container.

// refs: https://github.com/ory/dockertest
func initDB() (*dockertest.Pool, *dockertest.Resource) {
    pgURL := initPostgres()
    pgPass, _ := pgURL.User.Password()

    runOpts := dockertest.RunOptions{
        Repository: "postgres",
        Tag:        "latest",
        Env: []string{
            "POSTGRES_USER=" + pgURL.User.Username(),
            "POSTGRES_PASSWORD=" + pgPass,
            "POSTGRES_DB=" + pgURL.Path,
        },
    }

    pool, err := dockertest.NewPool("")
    if err != nil {
        log.WithError(err).Fatal("Could not connect to docker")
    }

    resource, err := pool.RunWithOptions(&runOpts)
    if err != nil {
        log.WithError(err).Fatal("Could start postgres container")
    }

    pgURL.Host = resource.Container.NetworkSettings.IPAddress

    // Docker layer network is different on Mac
    if runtime.GOOS == "darwin" {
        pgURL.Host = net.JoinHostPort(resource.GetBoundIP("5432/tcp"), resource.GetPort("5432/tcp"))
    }

    DockerDBConn = &dockerDBConn{}
    // exponential backoff-retry, because the application in the container might not be ready to accept connections yet
    if err := pool.Retry(func() error {
        DockerDBConn.Conn, err = sql.Open("postgres", pgURL.String())
        if err != nil {
            return err
        }
        return DockerDBConn.Conn.Ping()
    }); err != nil {
        phrase := fmt.Sprintf("Could not connect to docker: %s", err)
        log.Error(phrase)
    }

    DockerDBConn.initMigrations()

    return pool, resource
}

func closeDB(pool *dockertest.Pool, resource *dockertest.Resource) {
    if err := pool.Purge(resource); err != nil {
        phrase := fmt.Sprintf("Could not purge resource: %s", err)
        log.Error(phrase)
    }
}

func (db dockerDBConn) initMigrations() {
    driver, err := postgres.WithInstance(db.Conn, &postgres.Config{})
    if err != nil {
        log.Fatal(err)
    }

    migrate, err := migrate.NewWithDatabaseInstance(
        "file://testdata/migrations/",
        "mydatabase", driver)
    if err != nil {
        log.Fatal(err)
    }

    err = migrate.Steps(2)
    if err != nil {
        log.Fatal(err)
    }
}

func initPostgres() *url.URL {
    pgURL := &url.URL{
        Scheme: "postgres",
        User:   url.UserPassword("myuser", "mypass"),
        Path:   "mydatabase",
    }
    q := pgURL.Query()
    q.Add("sslmode", "disable")
    pgURL.RawQuery = q.Encode()

    return pgURL
}

The migrations are stored in the service where we run the integration tests within a folder called testdata and Golang-Migrate is used to run them. Now I have the base set up and teardown for the tests, within each service I create a test file and add the below code to the start of my tests.

import "github.com/mono-repo/support/database"

func TestMain(m *testing.M) {
    database.TestMain(m)
}

This will call the TestMain from my database package and will set up my DB container to run my tests against. And finally, within my individual test, I can set my DB connection to be the var we exported from the test database package.

func TestCompany(t *testing.T) {
    s := service.Service{
        Data: &data.Data{DB: database.DockerDBConn.Conn},
    }

        req := &pb.SelectCompanyReq{
        UserID: userID,
    }
    res, err := s.SelectCompany(context.Background(), req)
    assert.NoError(t, err)
}

I hope someone finds this useful, feel free to ping me if you have any further questions regarding this post.