DEV Community: Morten Vistisen

A practical approach to structuring Golang applications

Morten Vistisen — Wed, 02 Feb 2022 17:02:14 +0000

2022 update

I wrote this article a year ago and there is much I would change about it today. I still post it because I think there are something to be learned from this, at least to get a end 2 end description of one way to structure Go applications. There are many other options, and today I would suggest that you check out this repository.

Go is an amazing language. It’s simple, easy to reason about and gives you much tooling right out of the box. However, when I started with Go I struggled to figure out how to structure my applications in a way that didn’t use “enterprisey” approaches. This is my take on structuring Golang applications that start out simple, but with the flexibility to grow, and what I would have wanted when I started with Go.

Disclaimer

This article is based on the following people (link to their Twitter):

All of the above-mentioned people have articles and tutorials that are amazing on their own, which you should definitely checkout.

If you just want to see the code for this article’s project, check out the Github repository.

Project & tools

It is always frustrating to learn a new language and having to create the same to-do app over and over again. So we are going to make an application that allows people to create an account, where they can track their weight and calculate their nutritional requirements (inspired by weight increase during lockdown). Feel free to add to this application with features you see fit.

Now, I’m assuming some familiarity with Golang and programming in general, as this is a practical example of how to structure a Go application. We will be using the following:

Gin Web Framework
Go version 1.15+
Postgresql
Golang-migrate
Your favorite IDE (could be Goland or VS code — I highly suggest Goland, it’s amazing)

Elements of a robust and scalable structure

Spend some time working in software development and you quickly learn how programs sometimes can be brittle and other times rigid. Therefore, I look for these three things in my application structure:

Testability
Readability
Adaptability

Writing programs in a way that also makes it easy to write tests dramatically affect developer productivity. Along with readability, it not only makes it easier for you to maintain and improve old code. You also make it easier for new people to add to the project as they can feel somewhat safe when changing or adding code as you already have test in place. Lastly, having adaptability means your project can adapt to changing needs and complexity.

Starting out as simple as possible have the advantage of you being able to quickly iterate on new ideas getting to the actual problem solving right from the get-go. Most developers don’t need to think about domain-driven development when starting new projects, and it would probably be a waste of time in most cases (unless, of course, you are working in Big Co. but then why would you be reading an article for beginner Go developers).

The tutorial is split into three parts:

The application structure
User Service implementation
TDD implementation of Weight Service

This should hopefully give you an overview of the overall structure, a step-by-step guide on how to add a service and finally how easy it is to write test and develop with TDD.

The application structure

Alright, enough talk, let’s go going. Create a new folder wherever you like to store your projects called weight-tracker and run the following command inside the folder:

go mod init weight-tracker

The application is going to follow a structure that looks something like this:

weight-tracker
- cmd
  - server
    main.go
- pkg
  - api
    user.go
    weight.go
  - app
    server.go
    handlers.go
    routes.go
  - repository
    storage.go

Go ahead and create the above structure in your designated folder.

Most go projects seem to be following a convention of having a cmd and a pkg directory. The cmd will be the entry point into the program and gives you the flexibility to interact with the program in several ways. The pkg directory will contain everything else:

routes
database interactions
services etc.

We will be working with four packages:

main
api
app
repository

The main package should be self-explanatory if you’ve ever done any golang programming before. All of our services, i.e. user and weight service, are going into the api package along with a definitions file that will contain all of our structs (we will create this later). In the app package, we will have our server, handlers and routes. Lastly, we have repository that will contain all of our code related to database operations.

Open up main.go and add the following:

package main

import (
    "database/sql"
    "fmt"
    "github.com/gin-contrib/cors"
    "github.com/gin-gonic/gin"
    "os"
    "weight-tracker/pkg/api"
    "weight-tracker/pkg/app"
    "weight-tracker/pkg/repository"
)

func main() {
    if err := run(); err != nil {
        fmt.Fprintf(os.Stderr, "this is the startup error: %s\\n", err)
        os.Exit(1)
    }
}

// func run will be responsible for setting up db connections, routers etc
func run() error {
    // I'm used to working with postgres, but feel free to use any db you like. You just have to change the driver
    // I'm not going to cover how to create a database here but create a database
    // and call it something along the lines of "weight tracker"
    connectionString := "postgres://postgres:postgres@localhost/**NAME-OF-YOUR-DATABASE-HERE**?sslmode=disable"

    // setup database connection
    db, err := setupDatabase(connectionString)

    if err != nil {
        return err
    }

    // create storage dependency
    storage := repository.NewStorage(db)

    // create router dependecy
    router := gin.Default()
    router.Use(cors.Default())

    // create user service
    userService := api.NewUserService(storage)

    // create weight service
    weightService := api.NewWeightService(storage)

    server := app.NewServer(router, userService, weightService)

    // start the server
    err = server.Run()

    if err != nil {
        return err
    }

    return nil
}

func setupDatabase(connString string) (*sql.DB, error) {
    // change "postgres" for whatever supported database you want to use
    db, err := sql.Open("postgres", connString)

    if err != nil {
        return nil, err
    }

    // ping the DB to ensure that it is connected
    err = db.Ping()

    if err != nil {
        return nil, err
    }

    return db, nil
}

You should have quite a few errors showing in your IDE now, we are fixing that in a moment.
But first off, let me explain what is going on here. In our func main() we call a method named run, that returns an error (or nil, if there is no error). If run should return an error our programs exits and gives us an error message. Setting up our main function in this way allows us to test it and thereby following the element of a robust service structure, testability. This way of setting up the main function comes from Mat Ryer, who talks more about it in his blog post.

The next two topics we need to discuss are Clean Architecture and Dependency Injection. You might have heard about Clean Architecture under a different name, as a few people wrote about it around the same time. The majority of my inspiration comes from Robert C. Martin so this is who I will be referencing.

We are not going to be religiously following Clean Architecture but mainly adopt the idea of The Dependency Rule. Take a look at the below picture:

Source: Robert C. Martin’s blog — Clean Coder Blog

A quote from Robert Martin’s article helps define this rule:

This rule says that source code dependencies can only point inwards. Nothing in an inner circle can know anything at all about something in an outer circle.

In rough terms, the inner layers should not know about the outer layers. This allows us to practically change the database we use. Say we are changing from PostgreSQL to MySQL or exposing our API through gRPC instead of HTTP. You would most likely never do this but it does speak for the adaptability of this concept.

To achieve this, we are going to use dependency injection (DI for short). Basically, DI is the idea that services should receive their dependencies when they are created. It allows us to decouple the creation of a service’s dependencies from the creation of the service itself. This will be helpful when we get to testing our code. If you want to read more about DI, I suggest this article.

I learn best by looking at actual code, so let’s start by adding some of the missing code that will make the code in main.go make more sense. Open up storage.go and add the following:

package repository

import (
    "database/sql"
    _ "github.com/lib/pq"
)

type Storage interface{}

type storage struct {
    db *sql.DB
}

func NewStorage(db *sql.DB) Storage {
    return &storage{
        db: db,
    }
}

This allows us to create a new storage component where and whenever we want, as long as it receives a valid db argument of type *sql.DB.

As you may have noticed, we have a lowercase and an uppercase version of storage, one is a struct and one is an interface. We will define any methods (like CRUD operations) on the lowercase version of storage and define the methods in the uppercase version. By doing so, we can now easily mock out storage for unit testing. Also, we now get some nice suggestions from our IDE when we add methods to the storage interface when we haven’t implemented the methods.

Now, lets set up the base structure of one of our services, user.go. This should give you a feel for how the API package is going to be structuring services. You will have to repeat this for the weight.go service as well. Just copy-paste the content of user.goand change the naming. Open up user.go and add the following:

package api

// UserService contains the methods of the user service
type UserService interface{}

// UserRepository is what lets our service do db operations without knowing anything about the implementation
type UserRepository interface{}

type userService struct {
    storage UserRepository
}

func NewUserService(userRepo UserRepository) UserService {
    return &userService{
        storage: userRepo,
    }
}

Notice that our user service have a repository dependency. We will define repository methods (i.e. database operations) here later on, but the important part is that we only define the methods we need. Since we are not interested in the implementation of these methods, only the behaviour, we can write mock functions that suit a given test case.

This might seem a bit fluffy right now but bear with me for now, it will make sense later on. Let’s get a running application that we can add actual logic to the services.

Open up server.go and add the following:

package app

import (
    "github.com/gin-gonic/gin"
    "log"
    "weight-tracker/pkg/api"
)

type Server struct {
    router        *gin.Engine
    userService   api.UserService
    weightService api.WeightService
}

func NewServer(router *gin.Engine, userService api.UserService, weightService api.WeightService) *Server {
    return &Server{
        router:        router,
        userService:   userService,
        weightService: weightService,
    }
}

func (s *Server) Run() error {
    // run function that initializes the routes
    r := s.Routes()

    // run the server through the router
    err := r.Run()

    if err != nil {
        log.Printf("Server - there was an error calling Run on router: %v", err)
        return err
    }

    return nil
}

Next, open routes.go and add the following:

package app

import "github.com/gin-gonic/gin"

func (s *Server) Routes() *gin.Engine {
    router := s.router

    // group all routes under /v1/api
    v1 := router.Group("/v1/api")
    {
        v1.GET("/status", s.ApiStatus())
    }

    return router
}

We are going to take advantage of gin’s group functionality so we can easily group our endpoints after the resources they intend to serve. Next up, let’s add a handler so we can actually make calls to the status endpoint and verify that our application is running. Open up handlers.go:

package app

import (
    "github.com/gin-gonic/gin"
    "net/http"
)

func (s *Server) ApiStatus() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Header("Content-Type", "application/json")

        response := map[string]string{
            "status": "success",
            "data":   "weight tracker API running smoothly",
        }

        c.JSON(http.StatusOK, response)
    }
}

At this point, we only need to sync a few dependencies: Gin Web Framework and a driver for PostgreSQL. Go ahead and type the following into your terminal: go get github.com/gin-contrib/cors , go get github.com/gin-gonic/ginand go get github.com/lib/pg.

Everything should be ready now, so go into your terminal and write: go run cmd/server/main.go, visit http://localhost:8080/v1/api/status, and you should receive a message along the lines of:

{
    "data": "weight tracker API running smoothly",
    "status": "success"
}

If you don’t get the above message, please go back to the previous steps and see if you missed something or checkout the accompanying GitHub repository.

User Service implementation

At this point, we are ready to start building the meat of the application. To do so, we probably need to set up a database with tables and relations. To do this, we are going to use the library [golang-migrate](https://github.com/golang-migrate/migrate/). I personally like this library as it has a CLI tool to add migrations which enables me to do things like adding Makefile commands to create migrations. I encourage you to give the library’s documentation a look, it is an amazing project.

I won’t cover how to set this up as the article would be too long. For now, go into your terminal, make sure you are in the repository folder and run:

git clone https://gist.github.com/ed090d782dc6ebb35e344ff82aafdddf.git

This clones the migrations needed for the project. You should also now have a folder named ed090d782dc6ebb35e344ff82aafdddf, ****lets change that to migrations by running:

mv ed090d782dc6ebb35e344ff82aafdddf migrations

The last thing we need now is to add the RunMigrations method to storage.go:

// update your imports to look like this:
import (
    "database/sql"
    "errors"
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
    "path/filepath"
    "runtime"
)

// add the RunMigrations method to our interface
type Storage interface {
    RunMigrations(connectionString string) error
}

// add this below NewStorage
func (s *storage) RunMigrations(connectionString string) error {
    if connectionString == "" {
        return errors.New("repository: the connString was empty")
    }
    // get base path
    _, b, _, _ := runtime.Caller(0)
    basePath := filepath.Join(filepath.Dir(b), "../..")

    migrationsPath := filepath.Join("file://", basePath, "/pkg/repository/migrations/")

    m, err := migrate.New(migrationsPath, connectionString)

    if err != nil {
        return err
    }

    err = m.Up()

    switch err {
    case errors.New("no change"):
        return nil
    }

    return nil
}

To run the migrations, open up main.go and add the following:

// everything stays the same, so add this below
// storage := repository.NewStorage(db)
// run migrations

// note that we are passing the connectionString again here. This is so
// we can easily run migrations against another database, say a test version,
// for our integration- and end-to-end tests
err = storage.RunMigrations(connectionString)

if err != nil {
    return err
}

You should now have two tables in your database: user and weight. Let’s get started on writing some actual business logic.

We want to let people create an account through our API. So let’s started by defining what a user request is, create a file called under the API folder definitions.go and add the following:

package api

type NewUserRequest struct {
    Name          string `json:"name"`
    Age           int    `json:"age"`
    Height        int    `json:"height"`
    Sex           string `json:"sex"`
    ActivityLevel int    `json:"activity_level"`
    WeightGoal    string `json:"weight_goal"`
    Email         string `json:"email"`
}

We are defining what a new user request should look like. Note that this might be different from what a user struct would look like, we define our struct to only include the data we need. Next up, open user.go and the following to UserService and UserRepository:

// user.go
type UserService interface {
    New(user NewUserRequest) error
}

// User repository is what lets our service do db operations without knowing anything about the implementation
type UserRepository interface {
    CreateUser(NewUserRequest) error
}

Here we define a method on our UserService, called New and a method on the UserRepository called CreateUser. Remember we talked about The Dependency Rule earlier? This is what’s going on with the CreateUser method on UserRepository, our service does not know about the actual implementation of the method, what type of database it is etc. Just that there is a method called CreateUser and takes in a NewUserRequest and returns an error. The benefit of this is twofold: we get some indication from our IDE that a method is missing (open up main.go and check api.NewUserService) and what it needs, and it allows us to easily write unit tests. You should also see an error from NewUserServicein user.go, telling us that we are missing a method. Let’s fixed that, add the following:

// add these imports after the package declaration
import (
    "errors"
    "strings"
)

// add this after NewUserService
func(u * userService) New(user NewUserRequest) error {
    // do some basic validations
    if user.Email == "" {
        return errors.New("user service - email required")
    }

    if user.Name == "" {
        return errors.New("user service - name required")
    }

    if user.WeightGoal == "" {
        return errors.New("user service - weight goal required")
    }

    // do some basic normalisation
    user.Name = strings.ToLower(user.Name)
    user.Email = strings.TrimSpace(user.Email)

    err := u.storage.CreateUser(user)

    if err != nil {
        return err
    }

    return nil
}

We do some basic validations and normalisation, but this method could definitely be improved upon. We still need to add the CreateUser method, so open up storage.go and add the following to the CreateUser method to the Storage interface:

// storage.go
type Storage interface {
    RunMigrations(connectionString string) error
    CreateUser(request api.NewUserRequest) error
}

Notice that this makes the error in main.go go away, but results in a new error on the NewStorage function. We need to implement the method, just like we did with the UserService. Add this below RunMigrations:

// add "log" to imports to your imports look like this:
import (
    "database/sql"
    "errors"
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
    "log"
    "path/filepath"
    "runtime"
    "weight-tracker/pkg/api"
)

// add this below RunMigrations
func (s *storage) CreateUser(request api.NewUserRequest) error {
    newUserStatement := `
        INSERT INTO "user" (name, age, height, sex, activity_level, email, weight_goal)
        VALUES ($1, $2, $3, $4, $5, $6, $7);
        `

    err := s.db.QueryRow(newUserStatement, request.Name, request.Age, request.Height, request.Sex, request.ActivityLevel, request.Email, request.WeightGoal).Err()

    if err != nil {
        log.Printf("this was the error: %v", err.Error())
        return err
    }

    return nil
}

Again, this code could probably do with some improvements, but I’m going to keep it short.

Now, all that is left to do is to expose this through HTTP so that people can actually start using our API and create their accounts. Open handlers.go and add the following:

// update your imports to look like this
import (
    "github.com/gin-gonic/gin"
    "log"
    "net/http"
    "weight-tracker/pkg/api"
)

// add this below APIStatus method
func (s *Server) CreateUser() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Header("Content-Type", "application/json")

        var newUser api.NewUserRequest

        err := c.ShouldBindJSON(&newUser)

        if err != nil {
            log.Printf("handler error: %v", err)
            c.JSON(http.StatusBadRequest, nil)
            return
        }

        err = s.userService.New(newUser)

        if err != nil {
            log.Printf("service error: %v", err)
            c.JSON(http.StatusInternalServerError, nil)
            return
        }

        response := map[string]string{
            "status": "success",
            "data":   "new user created",
        }

        c.JSON(http.StatusOK, response)
    }
}

We are going to accept a request with a JSON payload and using gin’s ShouldBindJSON method to extract the data. Go ahead and try it out!

It took some time to get here, so let me show you one of the benefits that I keep talking about: testability.

Create a file called user_test.go under the API folder and add the following:

package api_test

import (
    "errors"
    "reflect"
    "testing"
    "weight-tracker/pkg/api"
)

type mockUserRepo struct{}

func (m mockUserRepo) CreateUser(request api.NewUserRequest) error {
    if request.Name == "test user already created" {
        return errors.New("repository - user already exists in database")
    }

    return nil
}

func TestCreateNewUser(t *testing.T) {
    mockRepo := mockUserRepo{}
    mockUserService := api.NewUserService(&mockRepo)

    tests := []struct {
        name    string
        request api.NewUserRequest
        want    error
    }{
        {
            name: "should create a new user successfully",
            request: api.NewUserRequest{
                Name:          "test user",
                WeightGoal:    "maintain",
                Age:           20,
                Height:        180,
                Sex:           "female",
                ActivityLevel: 5,
                Email:         "test_user@gmail.com",
            },
            want: nil,
        }, {
            name: "should return an error because of missing email",
            request: api.NewUserRequest{
                Name:          "test user",
                Age:           20,
                WeightGoal:    "maintain",
                Height:        180,
                Sex:           "female",
                ActivityLevel: 5,
                Email:         "",
            },
            want: errors.New("user service - email required"),
        }, {
            name: "should return an error because of missing name",
            request: api.NewUserRequest{
                Name:          "",
                Age:           20,
                WeightGoal:    "maintain",
                Height:        180,
                Sex:           "female",
                ActivityLevel: 5,
                Email:         "test_user@gmail.com",
            },
            want: errors.New("user service - name required"),
        }, {
            name: "should return error from database because user already exists",
            request: api.NewUserRequest{
                Name:          "test user already created",
                Age:           20,
                Height:        180,
                WeightGoal:    "maintain",
                Sex:           "female",
                ActivityLevel: 5,
                Email:         "test_user@gmail.com",
            },
            want: errors.New("repository - user already exists in database"),
        },
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            err := mockUserService.New(test.request)

            if !reflect.DeepEqual(err, test.want) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, err, test.want)
            }
        })
    }
}

A lot of things are happening here so let us go through them step-by-step.

We start by creating a struct called mockUserRepo. Our UserServiceknows that it needs a UserRepository with the following method: CreateUser. However, it does care about the actual implementation of said method, which allows us to mock the behaviour any way we want. In this case, we say that when the Name of the request is equal to “test user already created” (a bad name I know, but hopefully you get the point), return an error, and if not, just return nil. This allows us to mock the behaviour of our database to make it fit different situations, and test if our logic handles it in a way we expect it too.

Next, we create a new variable called mockRepo of type mockUserRepo We then create a mockUserService and pass the mockRepo to it, and we are ready to go! The actual test is what is known as a table-driven test. I won’t go into details about it as it is beyond the scope of this article, but if you want to know more, check out Dave Chaney’s article about it here.

Now, whenever we add a method to the UserRepository we would also have to add it here, in our mockUserRepo. We would of course also like to have some integration test, but what I really wanted to show with this article was how to do unit tests as these are cheap and easy to write.

Run go test ./... from the root directory and all tests should pass.

TDD implementation of Weight Service

Our application is not worth much right now. We can only create a user but not track our weight or calculate the amount of calories we need. Let’s change that!

Our application is not worth much right now. We can only create a user but not track our weight or calculate the number of calories we need. Let’s change that!I’m a big fan of test-driven development (TDD) and the way this application is structured makes it really easy to use. We are going to need three methods on our weight service: New, CalculateBMR and DailyIntake and two on our repository: CreateWeightEntry and GetUser. Open weight.go and add the following to WeightService and WeightRepository:

// weight.go

// WeightService contains the methods of the user service
type WeightService interface {
    New(request NewWeightRequest) error
    CalculateBMR(height, age, weight int, sex string) (int, error)
    DailyIntake(BMR, activityLevel int, weightGoal string) (int, error)
}

type WeightRepository interface {
    CreateWeightEntry(w Weight) error
    GetUser(userID int) (User, error)
}

Next, we need to add three structs to our definitions file: ´NewWeightRequest, Weight and User. Open up weight.go and add the following:

// add this after the package declaration
import "time"

// add this below NewUserRequest
type User struct {
    ID            int       `json:"id"`
    CreatedAt     time.Time `json:"created_at"`
    UpdatedAt     time.Time `json:"updated_at"`
    Name          string    `json:"name"`
    Age           int       `json:"age"`
    Height        int       `json:"height"`
    Sex           string    `json:"sex"`
    ActivityLevel int       `json:"activity_level"`
    WeightGoal    string    `json:"weight_goal"`
    Email         string    `json:"email"`
}

type Weight struct {
    Weight             int `json:"weight"`
    UserID             int `json:"user_id"`
    BMR                int `json:"bmr"`
    DailyCaloricIntake int `json:"daily_caloric_intake"`
}

type NewWeightRequest struct {
    Weight int `json:"weight"`
    UserID int `json:"user_id"`
}

Now, you will see an error on NewWeightService about missing methods. We don’t want to write the actual implementation yet, as we are doing TDD, so for now, just add this below NewWeightService:

// add these below NewWeightService
func (w *weightService) CalculateBMR(height, age, weight int, sex string) (int, error) {
    panic("implement me")
}

func (w *weightService) DailyIntake(BMR, activityLevel int, weightGoal string) (int, error) {
    panic("implement me")
}

func (w *weightService) New(request NewWeightRequest) error {
    panic("implement me")
}

Open up main.go and you will see that we also have some missing methods on storage passed to api.NewWeightService. Open up storage.go and add these:

// add this to the Storage interface
type Storage interface {
    CreateWeightEntry(request api.Weight) error
    GetUser(userID int) (api.User, error)
}

// add this below the CreateUser method
func (s *storage) CreateWeightEntry(request api.Weight) error {
    panic("implement me!")
}

func (s *storage) GetUser(userID int) (api.User, error) {
    panic("implement me!")
}

Lets now add the tests that we will need, create a file calledweight_test.go in the API folder and add the following:

package api_test

import (
    "errors"
    "reflect"
    "testing"
    "weight-tracker/pkg/api"
)

type mockWeightRepo struct{}

func (m mockWeightRepo) CreateWeightEntry(w api.Weight) error {
    return nil
}

func (m mockWeightRepo) GetUser(userID int) (api.User, error) {
    if userID != 1 {
        return api.User{}, errors.New("storage - user doesn't exists")
    }

    return api.User{
        ID:            userID,
        Name:          "Test user",
        Age:           20,
        Height:        185,
        WeightGoal:    "maintain",
        Sex:           "female",
        ActivityLevel: 5,
        Email:         "test@mail.com",
    }, nil
}

func TestCreateWeightEntry(t *testing.T) {
    mockRepo := mockWeightRepo{}
    mockUserService := api.NewWeightService(&mockRepo)

    tests := []struct {
        name    string
        request api.NewWeightRequest
        want    error
    }{
        {
            name: "should create a new user successfully",
            request: api.NewWeightRequest{
                Weight: 70,
                UserID: 1,
            },
            want: nil,
        }, {
            name: "should return a error because user already exists",
            request: api.NewWeightRequest{
                Weight: 70,
                UserID: 2,
            },
            want: errors.New("storage - user doesn't exists"),
        },
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            err := mockUserService.New(test.request)
            if !reflect.DeepEqual(err, test.want) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, err, test.want)
            }
        })
    }
}

func TestCalculateBMR(t *testing.T) {
    mockRepo := mockWeightRepo{}
    mockUserService := api.NewWeightService(&mockRepo)

    tests := []struct {
        name   string
        Height int
        Age    int
        Weight int
        Sex    string
        want   int
        err    error
    }{
        {
            name:   "should calculate BMR for a female",
            Height: 170,
            Age:    22,
            Weight: 65,
            Sex:    "female",
            want:   1441,
            err:    nil,
        }, {
            name:   "should calculate BMR for a male",
            Height: 170,
            Age:    22,
            Weight: 65,
            Sex:    "male",
            want:   1607,
            err:    nil,
        }, {
            name:   "should return error because sex wasn't properly specified",
            Height: 170,
            Age:    22,
            Weight: 65,
            Sex:    "",
            want:   0,
            err:    errors.New("invalid variable sex provided to CalculateBMR. needs to be either male or female"),
        },
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            BMR, err := mockUserService.CalculateBMR(test.Height, test.Age, test.Weight, test.Sex)
            if !reflect.DeepEqual(err, test.err) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, err, test.want)
            }

            if !reflect.DeepEqual(BMR, test.want) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, BMR, test.want)
            }
        })
    }
}

func TestDailyIntake(t *testing.T) {
    mockRepo := mockWeightRepo{}
    mockUserService := api.NewWeightService(&mockRepo)

    tests := []struct {
        name          string
        BMR           int
        ActivityLevel int
        weightGoal    string
        want          int
        err           error
    }{
        {
            name:          "should calculate daily intake for activity level 1 with weight loss as goal",
            weightGoal:    "loose",
            BMR:           1441,
            ActivityLevel: 1,
            want:          1229,
            err:           nil,
        }, {
            name:          "should calculate daily intake for activity level 2 with weight loss as goal",
            weightGoal:    "loose",
            BMR:           1441,
            ActivityLevel: 2,
            want:          1481,
            err:           nil,
        }, {
            name:          "should calculate daily intake for activity level 3 with weight loss as goal",
            weightGoal:    "loose",
            BMR:           1441,
            ActivityLevel: 3,
            want:          1733,
            err:           nil,
        }, {
            name:          "should calculate daily intake for activity level 4 with weight increase as goal",
            weightGoal:    "gain",
            BMR:           1441,
            ActivityLevel: 4,
            want:          2985,
            err:           nil,
        }, {
            name:          "should calculate daily intake for activity level 5 with weight maintenance as goal",
            weightGoal:    "maintain",
            BMR:           1441,
            ActivityLevel: 5,
            want:          2737,
            err:           nil,
        },
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            BMR, err := mockUserService.DailyIntake(test.BMR, test.ActivityLevel, test.weightGoal)
            if !reflect.DeepEqual(err, test.err) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, err, test.want)
            }

            if !reflect.DeepEqual(BMR, test.want) {
                t.Errorf("test: %v failed. got: %v, wanted: %v", test.name, BMR, test.want)
            }
        })
    }
}

The actual content of the tests is not so important as the fact that we can quickly write tests and mock external dependencies such as the methods to interact with the database.

Run go test ./... from the root directory and you should receive a lot of failed tests. We are going to fix this by implementing the logic of these methods, starting with the three in our WeightService. Add this below NewWeightService:

// add this after the package declaration
import "errors"

// we define some activity multipliers here - these are just part of the
// formula I choose to calculate your daily caloric intake.
const (
    // Very Low Intensity activity multiplier - 1
    veryLowActivity = 1.2
    // Light exercise activity multiplier - 3-4x per week - 2
    lightActivity = 1.375
    // Moderate exercise activity multiplier - 3-5x per week 30-60 mins/session - 3
    moderateActivity = 1.55
    // High exercise activity multiplier - (6-7x per week for 45-60 mins) - 4
    highActivity = 1.725
    // Very high exercise activity multiplier - for people who is an Athlete - 5
    veryHighActivity = 1.9
)

func (w *weightService) New(request NewWeightRequest) error {
    if request.UserID == 0 {
        return errors.New("weight service - user ID cannot be 0")
    }

    user, err := w.storage.GetUser(request.UserID)

    if err != nil {
        return err
    }

    bmr, err := w.CalculateBMR(user.Height, user.Age, request.Weight, user.Sex)

    if err != nil {
        return err
    }

    dailyIntake, err := w.DailyIntake(bmr, user.ActivityLevel, user.WeightGoal)

    if err != nil {
        return err
    }

    newWeight := Weight{
        Weight:             request.Weight,
        UserID:             user.ID,
        BMR:                bmr,
        DailyCaloricIntake: dailyIntake,
    }

    err = w.storage.CreateWeightEntry(newWeight)

    if err != nil {
        return err
    }

    return nil
}

func (w *weightService) CalculateBMR(height, age, weight int, sex string) (int, error) {
    var sexModifier int

    switch sex {
    case "male":
        sexModifier = -5
    case "female":
        sexModifier = 161
    default:
        return 0, errors.New("invalid variable sex provided to CalculateBMR. needs to be either male or female")
    }

    return (10 * weight) + int(float64(height)*6.25) - (5 * age) - sexModifier, nil
}

func (w *weightService) DailyIntake(BMR, activityLevel int, weightGoal string) (int, error) {
    var maintenanceCalories int

    switch activityLevel {
    case 1:
        maintenanceCalories = int(float64(BMR) * veryLowActivity)
    case 2:
        maintenanceCalories = int(float64(BMR) * lightActivity)
    case 3:
        maintenanceCalories = int(float64(BMR) * moderateActivity)
    case 4:
        maintenanceCalories = int(float64(BMR) * highActivity)
    case 5:
        maintenanceCalories = int(float64(BMR) * veryHighActivity)
    default:
        return 0, errors.New("invalid variable activityLevel - needs to be 1, 2, 3, 4 or 5")
    }

    var dailyCaloricIntake int

    switch weightGoal {
    case "gain":
        dailyCaloricIntake = maintenanceCalories + 500
    case "loose":
        dailyCaloricIntake = maintenanceCalories - 500
    case "maintain":
        dailyCaloricIntake = maintenanceCalories
    default:
        return 0, errors.New("invalid weight goal provided - must be gain, loose or maintain")
    }

    return dailyCaloricIntake, nil
}

With these methods added all of the tests should pass.

The last thing we need is to add the methods needed to interact with the database. Open storage.go and add the following:

// replace the methods: CreateWeightEntry and GetUser with the code below
func (s *storage) CreateWeightEntry(request api.Weight) error {
    newWeightStatement := `
        INSERT INTO weight (weight, user_id, bmr, daily_caloric_intake)
        VALUES ($1, $2, $3, $4)
        RETURNING id;
        `

    var ID int
    err := s.db.QueryRow(newWeightStatement, request.Weight, request.UserID, request.BMR, request.DailyCaloricIntake).Scan(&ID)

    if err != nil {
        log.Printf("this was the error: %v", err.Error())
        return err
    }

    return nil
}

func (s *storage) GetUser(userID int) (api.User, error) {
    getUserStatement := `
        SELECT id, name, age, height, sex, activity_level, email, weight_goal FROM "user"
        where id=$1;
        `

    var user api.User
    err := s.db.QueryRow(getUserStatement, userID).Scan(&user.ID, &user.Name, &user.Age, &user.Height, &user.Sex, &user.ActivityLevel, &user.Email, &user.WeightGoal)

    if err != nil {
        log.Printf("this was the error: %v", err.Error())
        return api.User{}, err
    }

    return user, nil
}

We didn’t do this in true TDD style, however, it should again paint a picture of how we can structure Go applications and develop them using TDD. Feel free to re-implement this section and do it in true TDD style: create one test, make it pass, create the next one, and so on.

We have almost everything we need now. The last thing we need is to add routes and a handler to create a weight entry for a given user. I’m going to leave this up to the reader to implement, as the building blocks should be in place. If you are feeling lazy you can just check out the accompanying Github repository.

Conclusion

Hopefully, this tutorial gave you some insights on how to structure your Golang applications. It was a long one I know, but hopefully, you didn’t waste your time. In a part two, I will show how to add Makefile commands, integration tests and easily deploy the application. If you have any questions or criticisms, please do not hesitate to reach out.

My twitter and github.

Resources

Integration tests, docker and how it all Go(es) together

Morten Vistisen — Wed, 02 Feb 2022 16:56:01 +0000

Picture this: you've just left Node for the promised land of Go. You've learned about composition, interfaces, simplicity, domain-driven design (and understood nothing) and unit tests. You feel all grown-up and ready to take on concurrency and ride off into the promised land. You've built a sweet CRUD application that will revolutionise how we do X. Since this is a sure-fire success; you've of course already contacted a real estate agent to view that penthouse apartment you've always wanted and tweeted at multiple VC funds and investors. You're basically ready to go live but before you release your code into the wild, you whip out your favourite terminal and type in go run main.go. It runs, glory days!

You set up multiple demos and investor meetings. You add "CEO and Founder" to your Linkedin profile (and tinder, of course) and tell your mom that you're going to vacate the basement any day now. You show up to your first meeting, do a couple of power poses and start demoing that really-crucial-endpoint-that-is-the-backbone-of-the-app. With your right thumb on the ENTER key (and your left thumb mentally ready to pop the champagne), you go ahead. Aaaaaand, it completely crashes. What the fuck. Not only do you get laughed at your super important meeting, but you also have to tell the real estate agent that you probably can't afford that penthouse apartment, you have also to stay in your mom's basement (and continue trying to convince your tinder dates that it's definitely a temporary solution).

Hopefully, you haven't experienced the above scenario and are just here because you typed some words into google. I'm going to show you how I set up and run integration tests in Go, using docker and some nifty Makefile cmds.

Aim of the article

A lot has been written on this subject (check out the resources list at the end of the article). This is not an attempt to improve on what is currently available. It's an attempt to show how you can set up and configure integrations tests, both locally and externally (CI env), that is extendable and portable. I'm sure there are ways to improve upon what I'm going to show you here, however, the main goal is to provide you with the tools to build out a great test suite. Once you've understood the concepts and configurations it's much easier to customise them to your need. Furthermore, you get a better idea of what to google (unknown unknowns are a tricky subject), so you can solve your own issues that might come up later on.

Requirements

Alright, first thing first. You'll need some tools, so please install and setup the
following:

Go
Docker

There are more than one way to skin a cat

As with everything in software, there are a lot of ways of doing the same thing and even more opinions about how to do that thing. With integration testing, we might discuss how to set up/configure the test suite, what library (if any) to use and even when something classifies as an integration vs. unit test, E2E test, and so forth. This can make concepts more complex than they need to be especially when you're just starting out. I think one of the best ways to approach such situations is to agree on some sort of guiding principles/first principles, that becomes the basis for what we do. I've found one of the best principles for this is:

test behaviour, not implementation details

To expand upon this, we want to have our tests cover cases/scenarios of how a user might actually use our software. I use the terms user and software very broadly here as it's very context-specific. A user might be a co-worker who calls some method in the codebase that we have created or it might also be someone consuming our API. Our tests shouldn't really worry about how we actually implemented the code, only that given X input we see Y response. We are in a sense testing that the "contract" we have made as developers by exposing our code/functions/methods to the world, is actually kept.

If we want a more concrete definition of an integration test we can dust off the big ol' google book and see how they define it:

...medium-scoped tests (commonly called integration tests) are designed to verify interactions between a small number of components; for example, between a server and a database

Source: Software Engineering at Google, chap. 11

If we borrow some terminology from clean architecture we can think of it as when we include code from the infrastructure layer in our tests, we're in the integration testing territory.

What are we doing

When I started out writing integration tests in Go; what I struggled the most with was how to configure and set it up for real-world usage. This, again, will also differ based upon the developer/use-case/reality as this approach might not work for well a big multinational company. My criteria for my integration tests are that they should run across different operating systems, be easy to run and play nicely with CI/CD workflows (basically, play nice within a dockerized environment).

I'm a strong believer in YAGNI (you're not going to need it), so I'm going to show you two ways of setting up integration tests:

the vanilla only-using-the-standard-library approach
using a library

This should hopefully illustrate how you can start out relatively simple (we could skip the Docker part, but honestly, that would make it a little trickier to set up our CI/CD flow) and then add on as needed.

What we are testing

I'm going to re-use some of the code from my article on how to structure Go apps (which can be found here. If you haven't read it, it basically builds a small app that lets your track your weight gain during the lockdown. The article is due for an update (I would suggest checking this [repo(https://github.com/bnkamalesh/goapp) for a great example on how to structure Go apps using DDD and clean architecture), so we're just going to focus on adding integration tests based on the code (a good example of testing behaviour vs. implementation details). We want to make sure that calls to our services behave as expected.

You can find the repo here.

"Infrastructure" Setup

Much of modern web development uses Docker and this tutorial will be no exception. This is not a tutorial on Docker so I won't be going into much detail about the setup, but provide some foundations on how to get started. There are ways to improve upon this setup by extending the docker-compose file, using tags inside of our Dockerfile etc. But I often find that to give me more headaches than gains. We're violating DRY, to some extent, but it does give us the ability to have completely separate dev and test environment. And after reading this, you could yourself try to shorten the docker setup.

Docker test environment setup

FROM golang:1-alpine

RUN apk add --no-cache git gcc musl-dev

RUN go get -u github.com/rakyll/gotest

RUN go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

WORKDIR /app

version: "3.8"
services:
    database:
        image: postgres:13
        container_name: test_weight_tracker_db_psql
        environment:
            - POSTGRES_PASSWORD=password
            - POSTGRES_USER=admin
            - POSTGRES_DB=test_weight_tracker_database
        ports:
            - "5436:5432" # we are mapping the port 5436 on the local machine
              # to the image running inside the container
        volumes:
            - test-pgdata:/var/lib/postgresql/data

    app:
        container_name: test_weight_tracker_app
        build:
            context: .
            dockerfile: Dockerfile.test
        ports:
            - "8080:8080"
        working_dir: /app
        volumes:
            - ./:/app
            - test-go-modules:/go
        environment:
            - DB_NAME=test_weight_tracker_database
            - DB_PASSWORD=${DB_PASSWORD}
            - DB_HOST=test_weight_tracker_db_psql
            - DB_PORT=${DB_PORT}
            - DB_USERNAME=${DB_USERNAME}
            - ENVIRONMENT=test
        depends_on:
            - database

volumes:
    test-pgdata:
    test-go-modules:

With that in place we can now easily run our integration tests, which can be done with:

make run-integration-tests

Approach 1: Vanilla setup

Side note: if you want to see the code separated from the rest, check out the branch vanilla-approach/running-integration-tests-using-std-library

There is a tendency in the Go community to lean more towards the standard library than to pull in external libraries. And for a good reason, you can do a lot with just the standard library. I'm by no means a purist in this regard and we are also using external libraries for the routing, migration etc. here, but I think it gives a great understanding of what is happening by starting with the standard library and then using other libraries as you go.

In the earlier segment, we got our infrastructure up and running so we have an active database, a running application, or in this case, the possibility to trigger a test run against our app.

To maximize our confidence in our code we want our integration tests to mimic our production environment as much as possible. This means that we need to have some setup and teardown functions that can run migrations, populate the database with seed data and tear everything down after the test so we have a clean environment each time. The last part is important as we want to have a reliable test environment so we don't want previously run tests to affect the current one running.

It's worth noting here that this also means that our integrations tests have to run sequentially, and not in parallel, as we most likely would have our unit test do which means it would take longer running the test suite. I wouldn't
worry too much about this at the beginning and simply focus on having some integration tests with decent coverage. Once you start to reach unbearable long integration test runs then it's time to start looking into other setups/improving the current one. For example, we could create a new database for each test and use that, but it would add a more complex setup. Or we could use an SQLite database for our integration tests, but again, cross that bridge once you get to it.

It would be a great exercise to try and change this code to try out different strategies to speed up the test runs.

Migrations

I'm a big fan of the golang-migrate library so that is what we are going to use to write our migrations. In short, it generates up/down migration pairs and treats each migration as a new version, so you can roll back to the last working
version if you need to.

I'm not going to touch upon migration strategies here, so our tests will be written with the assumption that we have a database with all the latest migrations. To achieve this in our tests we will run the up version before each test is run. For that we need the following function:

func RunUpMigrations(cfg config.Config) error {
    _, b, _, _ := runtime.Caller(0)
    basePath := filepath.Join(filepath.Dir(b), "../migrations")
    migrationDir := filepath.Join("file://" + basePath)
    db, err := sql.Open("postgres", cfg.GetDatabaseConnString())
    if err != nil {
        return errors.WithStack(err)
    }
    defer db.Close()
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return errors.WithStack(err)
    }
    defer driver.Close()

    m, err := migrate.NewWithDatabaseInstance(migrationDir, "postgres", driver)
    if err != nil {
        return errors.WithStack(err)
    }

    if err := m.Up(); err != nil {
        if errors.Is(err, ErrNoNewMigrations) {
            return errors.WithStack(err)
        }
    }
    m.Close()
    return nil
}

After the test is down, we would like our environment to be clean so we also need this function:

func RunDownMigrations(cfg config.Config) error {
    _, b, _, _ := runtime.Caller(0)
    basePath := filepath.Join(filepath.Dir(b), "../migrations")
    migrationDir := filepath.Join("file://" + basePath)
    db, err := sql.Open("postgres", cfg.GetDatabaseConnString())
    if err != nil {
        return errors.WithStack(err)
    }
    defer db.Close()
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return errors.WithStack(err)
    }
    defer driver.Close()

    m, err := migrate.NewWithDatabaseInstance(migrationDir, "postgres", driver)
    if err != nil {
        return errors.WithStack(err)
    }

    if err := m.Down(); err != nil {
        return errors.WithStack(err)
    }

    return nil
}

Basically, we get a new connection to the database, create a new migrate instance where we pass a path to the migrations folder, the database we use and the driver. We then run the migrations and close the connection again, pretty straightforward.

Next up, we need some data in our database to run our tests against. I'm a pretty big fan of just keeping it in a SQL file and then having a helper function run the SQL script against the database. To do that we just need a function similar to the two above:

func LoadFixtures(cfg config.Config) error {
    pathToFile := "/app/fixtures.sql"
    q, err := os.ReadFile(pathToFile)
    if err != nil {
        return errors.WithStack(err)
    }

    db, err := sql.Open("postgres", cfg.GetDatabaseConnString())
    if err != nil {
        return errors.WithStack(err)
    }

    _, err = db.Exec(string(q))
    if err != nil {
        return errors.WithStack(err)
    }
    err = db.Close()
    if err != nil {
        return errors.WithStack(err)
    }

    return nil
}

With that setup we are ready to write our first tests.

Our first test

Technically, we could test the code interacting with the database by providing a mocked version of the methods in the database/SQL package. But that doesn't really give us much as it would be tricky to mock a situation where you, for example, miss a variable in your .Scan method or have some syntax issue. Therefore, I tend to write integration tests for all my database functionality. Let's add a test for the CreateUser function. We need the following:


// testing the happy path only - to improve upon these tests, we could consider
// using a table test
func TestIntegration_CreateUser(t *testing.T) {
    // create a NewStorage instance and run migrations
    cfg := config.NewConfig()
    storage := psql.NewStorage()

    err := psql.RunUpMigrations(*cfg)
    if err != nil {
        t.Errorf("test setup failed for: CreateUser, with err: %v", err)
        return
    }

    // run the test
    t.Run("should create a new user", func(t *testing.T) {
        newUser, err := entity.NewUser(
            "Jon Snow", "male", "90", "thewhitewolf@stark.com", 16, 182, 1)
        if err != nil {
            t.Errorf("failed to run CreateUser with error: %v", err)
            return
        }

        // to ensure consistency we could consider adding in a static date
        // i.e. time.Date(insert-fixed-date-here)
        // creationTime := time.Now()
        err = storage.CreateUser(*newUser)
        // assert there is no err
        if err != nil {
            t.Errorf("failed to create new user with err: %v", err)
            return
        }

        // now lets verify that the user is actually created using a
        // separate connection to the DB and pure sql
        db, err := sql.Open("postgres", cfg.GetDatabaseConnString())
        if err != nil {
            t.Errorf("failed to connect to database with err: %v", err)
            return
        }
        queryResult := entity.User{}
        err = db.QueryRow("SELECT id, name, email FROM users WHERE email=$1",
            "thewhitewolf@stark.com").Scan(
            &queryResult.ID, &queryResult.Name, &queryResult.Email,
        )
        if err != nil {
            t.Errorf("this was query err: %v", err)
            return
        }

        if queryResult.Name != newUser.Name {
            t.Error(`failed 'should create a new user' wanted name did not match 
                returned value`)
            return
        }
        if queryResult.Email != newUser.Email {
            t.Error(`failed 'should create a new user' wanted email did not match 
                returned value`)
            return
        }
        if int64(queryResult.ID) != int64(1) {
            t.Error(`failed 'should create a new user' wanted id did not match 
                returned value`)
            return
        }

    })

    // // run some clean up, i.e. clean the database so we have a clean env
    // // when we run the next test
    t.Cleanup(func() {
        err := psql.RunDownMigrations(*cfg)
        if err != nil {
            if errors.Is(err, migrate.ErrNoChange) {
                return
            }
            t.Errorf("test cleanup failed for: CreateUser, with err: %v", err)
        }
    })
}

We start by creating a new instance of config and storage (just like we would in main.go when running the entire application) and then run the up migrations function. If nothing goes wrong, we should have something similar to what we would have in production.

We then use the storage instance that we just set up to create a new user, open a new connection to query for the user we just created and verify that said user is created with the values we expect. After, use the Cleanup function provided by the testing package to call the down migrations. This basically clears the database.

One more thing you might notice is that we have a psql_test.go file. Open it, and you will find the following function:


// TestMain gets run before running any other _test.go files in each package
// here, we use it to make sure we start from a clean slate
func TestMain(m *testing.M) {
    cfg := config.NewConfig()
    // make sure we start from a clean slate
    err := psql.DropEverythingInDatabase(*cfg)
    if err != nil {
        panic(err)
    }

    os.Exit(m.Run())
}

TestMain is a special function that gets called before all other tests in the package it's located in. Here, we're being (justifiable, I would say) paranoid and call a function that drops everything in the database so we are sure we are starting from a clean slate. You can find the function in repository/psql.go if you want to take a closer look.

And that's basically it for running integration tests against our database functions. We could use table-driven tests here and probably should, but this will do for illustration purposes. See here for an explanation of table-driven tests if you don't know them. Next up, let's do a "proper" integration tests and ensure that our endpoints are working as expected!

Testing our endpoints

Now we get into the meat of the stuff. We've arrived at (or closer to, at least) the definition found in the ol' google book. We're testing that multiple parts of our codes work together, mostly in the infrastructure layer, works together
in the way we expect. Here, we want to ensure that whenever our API receives a request that is fulfilling the contract we as developers put out, it does what we want. That is, we want to test the happy path. Ideally, we would also want to test the sad path (not sure if that's the word for it, but this is my article, so now it is) but integration tests are more "expensive" so it's a delicate balance. You could choose to mock out database responses and test the sad path in a more unit-test kind of way, or you could add integration tests until the time it takes to run the test suite becomes unbearable. I would probably err on adding 1 integration test to many, and deal with the "cost" when it becomes too big.

Alright, enough rambling. Let's get started. A side note here; I'm using gofiber which is inspired by express, the Node web framework. The way I'm setting up the POST request sort of depends on how gofiber does things. I say sort of because the underlying thing when sending a post request from Go is using Marshaling. I will point it out when we get to it, but just be aware that if you like gorilla or gin you might have to google a bit.

A quick rundown of router setup

We're not going to be spending much time on this, as you can find the code in the repo. Basically, we have this:

type serverConfig interface {
    GetServerReadTimeOut() time.Duration
    GetServerWriteTimeOut() time.Duration
    GetServerPort() int64
}

type Http struct {
    router        *fiber.App
    serverPort    int64
    userHandler   userHandler
    weightHandler weightHandler
}

func NewHttp(
    cfg serverConfig, userHandler userHandler, weightHandler weightHandler) *Http {
    r := fiber.New(fiber.Config{
        ReadTimeout:  cfg.GetServerReadTimeOut(),
        WriteTimeout: cfg.GetServerWriteTimeOut(),
        AppName:      "Weight Tracking App",
    })
    return &Http{
        router:        r,
        serverPort:    cfg.GetServerPort(),
        userHandler:   userHandler,
        weightHandler: weightHandler,
    }
}

We set up an HTTP struct that has some dependencies to get our server up and running with a router. On that struct, we define some server-specific methods. It's pretty straightforward.

Testing our endpoint to create a new user

Our endpoint is pretty simple. There are no middleware and authentication, everybody can just spam our server with requests and create a ton of users. That's not ideal, but also not really what we care about right now. We just want to make sure our API does what it's supposed to do.


func TestIntegration_UserHandler_New(t *testing.T) {
    cfg := config.NewConfig()
    storage := psql.NewStorage()

    err := psql.RunUpMigrations(*cfg)
    if err != nil {
        t.Errorf("test setup failed for: CreateUser, with err: %v", err)
        return
    }

    userService := service.NewUser(storage)
    weightService := service.NewWeight(storage)

    userHandler := http.NewUserHandler(userService)
    weightHandler := http.NewWeightHandler(weightService)

    srv := http.NewHttp(cfg, *userHandler, *weightHandler)

    srv.SetupRoutes()
    r := srv.GetRouter()

    req := http.NewUserRequest{
        Name:          "Test user",
        Sex:           "male",
        WeightGoal:    "80",
        Email:         "test@gmail.com",
        Age:           99,
        Height:        185,
        ActivityLevel: 1,
    }
    var buf bytes.Buffer
    err = json.NewEncoder(&buf).Encode(req)
    if err != nil {
        log.Fatal(err)
    }
    rq, err := h.NewRequest(h.MethodPost, "/api/user", &buf)
    if err != nil {
        t.Error(err)
    }
    rq.Header.Add("Content-Type", "application/json")

    res, err := r.Test(rq, -1)
    if err != nil {
        t.Error(err)
    }

    if res.StatusCode != 200 {
        t.Error(errors.New("create user endpoint did not return 200"))
    }

    // query the database to verify that a user was created based on the request
    // we sent
    newUser, err := storage.GetUserFromEmail(req.Email)
    if err != nil {
        t.Error(err)
    }

    if newUser.Height != req.Height {
        t.Error(errors.New("create user endpoint did not create user with correct details"))
    }

    t.Cleanup(func() {
        err := psql.RunDownMigrations(*cfg)
        if err != nil {
            if errors.Is(err, migrate.ErrNoChange) {
                return
            }
            t.Errorf("test cleanup failed for: CreateUser endpoint, with err: %v", err)
        }
    })
}

Most of this looks similar to what we had in the repository tests. We set up the database, the services and lastly, the server. We create a request, encode it, send it to our endpoint and check the response. An important thing to note here is that we don't really know what is happening under the hood of this beast. We just know that we sent a request with some data and that returns OK and create a user in the database with the expected data. This is also known as black-box testing. We don't care how this is done, we care that the expected behaviour occurs.

One thing about the above code is that there is quite some repetitiveness in how we set up the tests and tear them down after each run. It would be nice if we didn't have to copy-paste all of this and take a long hot bath after each test run because we violated DRY. We could do this ourselves of course, or we could use Approach 2 - using test suites with Testify.

Approach 2 - using Testify to run our integration tests

For this, we are going to use the testify package which I have used for quite some time now. The main thing this does for us is save some lines on configuration and ensuring consistency in our tests suites. It's easy enough to have the entire codebase in your head when it's only this size, but as it grows, having the setup and configuration done in one place makes things so much easier. Let's see how the setup is done for our handler integration tests:

type HttpTestSuite struct {
    suite.Suite
    TestStorage *psql.Storage
    TestDb      *sql.DB
    TestRouter  *fiber.App
    Cfg         *config.Config
}

func (s *HttpTestSuite) SetupSuite() {
    log.SetFlags(log.LstdFlags | log.Lshortfile)
    cfg := config.NewConfig()

    db, err := sql.Open("postgres", cfg.GetDatabaseConnString())
    if err != nil {
        panic(errors.WithStack(err))
    }

    err = db.Ping()
    if err != nil {
        panic(errors.WithStack(err))
    }
    storage := psql.NewStorage()

    userService := service.NewUser(storage)
    weightService := service.NewWeight(storage)

    userHandler := http.NewUserHandler(userService)
    weightHandler := http.NewWeightHandler(weightService)

    srv := http.NewHttp(cfg, *userHandler, *weightHandler)

    srv.SetupRoutes()
    r := srv.GetRouter()

    s.Cfg = cfg
    s.TestDb = db
    s.TestStorage = storage
    s.TestRouter = r
}

We basically take the entire setup step and automate it for each test suite. If we check the documentation for the SetupSuite method we see that it's basically a method that runs before the test in a suite is run. So the whole setup we did with the standard library like here:


func TestIntegration_UserHandler_CreateUser(t *testing.T) {
    cfg := config.NewConfig()
    storage := psql.NewStorage()

    ..... irrelevant code removed

    userService := service.NewUser(storage)
    weightService := service.NewWeight(storage)

    userHandler := http.NewUserHandler(userService)
    weightHandler := http.NewWeightHandler(weightService)

    srv := http.NewHttp(cfg, *userHandler, *weightHandler)

    srv.SetupRoutes()
    r := srv.GetRouter()

    ..... irrelevant code removed

}

is automated for us, nice! Now, we also did have some other requirements, namely that we had a "fresh" environment for each test run. This means that we need to run up/down migrations to ensure our database is clean. This was done in the setup and teardown portion before in each test, but with testify, we can just define beforeTest and afterTest where we can run the same methods as we did before, without having to copy-paste them for each test.

One thing you will notice if you check out the repo is that we have almost the entirely same code in the repository as we do here. Only except for the TestRouter in the struct. I don't really mind the duplication here as my needs for the endpoints test could change in the future, and keeping my dependencies as few as possible is desirable. So you could, if you wanted, make one large integration test suite. I just prefer to split things up, each to their own.

In conclusion

Will the above steps prevent you from the disaster we went through at the beginning of the article? Maybe, it depends (every senior developer's favourite reply). But, it will definitely increase the amount of confidence you can have
in your code. How many integration tests to have is always a balance since they do take some time longer to run, but there are ways at fixing that, so testing the happy path until things becomes unbearably slow is a good rule of thumb. Cross that bridge when you get to it.

As mentioned in the introduction, this is not an attempt to add something new and revolutionise the way we do integration tests in the Go community. But to give you another perspective and some boilerplate code to get going with your integration testing adventures. Should you want to continue and learn from
people way smarter than me (you definitely should), check out the resource section.

Resources

Learn go with tests - basically read this entire thing. Chris does an amazing job showing you how to get started with Go and test-driven development. Definitely worth the read.

HTML forms, databases, integration tests - though it's not in Go, but in Rust, Luca does a great job explaining integration testing. Always try to look for what concepts transcend programming languages and what doesn't. It's always beneficial to have a nuanced view.