DEV Community: TECH SCHOOL

Backend Master Class [Go + Postgres + Kubernetes + AWS]

TECH SCHOOL — Sun, 30 Jan 2022 10:41:14 +0000

Finally, Tech School's Backend Master Class course is completed and published on Udemy. Here's a discount coupon link if you want to learn it on Udemy and earn a certificate of completion.

In this course, you will learn step-by-step how to design, develop and deploy a backend web service from scratch. I believe the best way to learn programming is to build a real application. Therefore, throughout the course, you will learn how to build a backend web service for a simple bank. It will provide APIs for the frontend to do the following things:

Create and manage bank accounts.
Record all balance changes to each of the accounts.
Perform a money transfer between 2 accounts.

The programming language we will use to develop the service is Golang, but the course is not just about coding in Go. The course is divided into 4 main parts:

In the first part, you will learn deeply about how to design the database, generate codes to talk to the DB in a consistent and reliable way using transactions, understand the DB isolation levels, and how to use it correctly in production. Besides the database, you will also learn how to use Docker for local development, how to use Git to manage your codes, and how to use Github Action to run unit tests automatically.
In the second part, you will learn how to build a set of RESTful HTTP APIs using Gin - one of the most famous Golang frameworks for building web services. This includes everything from loading app configs, mocking DB for more robust unit tests, handling errors, authenticating users, and securing the APIs with JWT and PASETO access tokens.
In the third part, you will learn how to build your app with Docker and deploy it to a production Kubernetes cluster on AWS. The lectures are very detailed with a step-by-step guide, from how to build a minimal docker image, set up a free-tier AWS account, create a production database, store and retrieve production secrets, create a Kubernetes cluster with EKS, use Github Action to automatically build and deploy the image to the EKS cluster, buy a domain name and route the traffics to the service, secure the connection with HTTPs and auto-renew SSL/TLS certificate from Let's Encrypt.
The last part is a work-in-progress, where we discuss more advanced backend topics such as managing user sessions, building gRPC APIs, using gRPC gateway to serve both gRPC and HTTP with 1 single implementation of the handler, and embedding Swagger documentation as part of the backend service, etc. We will keep making and uploading new videos over time, so please come back here to check them out from time to time.

This course is designed with a lot of details, so that everyone, even with very little programming experience can understand and do it by themselves. I strongly believe that after the course, you would be able to work much more confidently and effectively in your projects.

If you like the course, please give me some meaningful feedback on Udemy, and feel free to share it with your friends and colleagues. And don't forget to check out my channel for more videos & courses: http://techschool.guru/

Implement login user API that returns PASETO or JWT access token in Go

TECH SCHOOL — Sat, 24 Apr 2021 08:13:04 +0000

Hello everyone! Welcome back to the backend master class!

In the previous lecture, we’ve implemented the token maker interface using JWT and PASETO. It provides 2 methods to create and verify tokens.

So today we’re gonna learn how to use it to implement the login API, where the username and password are provided by the client, and the server will return an access token if those credentials are correct.

Here's:

Link to the full series playlist on Youtube
And its Github repository

OK let’s start!

Add token Maker to the Server

The first step is to add the token maker to our API server. So let’s open api/server.go file!

In the Server struct, I’m gonna add a tokenMaker field of type token.Maker interface.

type Server struct {
    store      db.Store
    tokenMaker token.Maker
    router     *gin.Engine
}

Then let’s initialize this field inside the NewServer() function! First we have to create a new token maker object. We can choose to use either JWT or PASETO, they both implement the same token.Maker interface.

I think PASETO is better, so let’s call token.NewPasetoMaker(). It requires a symmetric key string, so we will need to load this from environment variable. For now, let’s just put an empty string here as a placeholder.

If the returned error is not nil, we return a nil server, and an error saying "cannot create token maker". The %w is used to wrap the original error.

func NewServer(store db.Store) (*Server, error) {
    tokenMaker, err := token.NewPasetoMaker("")
    if err != nil {
        return nil, fmt.Errorf("cannot create token maker: %w", err)
    }

    server := &Server{
        store:      store,
        tokenMaker: tokenMaker,
    }

    ...

    return server, nil
}

OK, so now we have to change the return type of the NewServer() function to include an error as well. Then in the statement to create a Server object, we add the tokenMaker object that we’ve just created.

Alright, now let’s come back to the symmetric key parameter. I’m gonna add a new environment variable to the app.env file. Let’s call it TOKEN_SYMMETRIC_KEY.

And as we’re using PASETO version 2, which uses ChachaPoly algorithm, the size of this symmetric key should be exactly 32 bytes.

TOKEN_SYMMETRIC_KEY=12345678901234567890123456789012
ACCESS_TOKEN_DURATION=15m

We should also add 1 more variable to store the valid duration of the access token. It’s a best practice to set this to a very short duration, let’s say, just 15 minutes for example.

OK, now we have to update our config struct to include the 2 new variables that we’ve just added.

First, the TokenSymmetricKey of type string. We have to specify the mapstructure tag for it because viper uses mapstructure package to parse the config data. Please refer to the lecture 12 of the course if you don’t know how to use viper.

type Config struct {
    ...
    TokenSymmetricKey   string        `mapstructure:"TOKEN_SYMMETRIC_KEY"`
    AccessTokenDuration time.Duration `mapstructure:"ACCESS_TOKEN_DURATION"`
}

The next field is AccessTokenDuration of type time.Duration. And its mapstructure tag should be this environment variable’s name: ACCESS_TOKEN_DURATION.

As you can see, when the type of a config field is time.Duration, we can specify the value in a human readable format like this: 15m.

OK so now we’ve loaded the secret key and token duration into the config, let’s go back to the server and use them. We have to add a config parameter to the NewServer() function. Then in the token.NewPasetoMaker() call, we pass in config.TokenSymmetricKey.

We should also add a config field to the Server struct, and store it here when initialize the Server object. We will use the TokenDuration in this config object later when creating the tokens.

type Server struct {
    config     util.Config
    store      db.Store
    tokenMaker token.Maker
    router     *gin.Engine
}

func NewServer(config util.Config, store db.Store) (*Server, error) {
    tokenMaker, err := token.NewPasetoMaker(config.TokenSymmetricKey)
    if err != nil {
        return nil, fmt.Errorf("cannot create token maker: %w", err)
    }

    server := &Server{
        config:     config,
        store:      store,
        tokenMaker: tokenMaker,
    }

    if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
        v.RegisterValidation("currency", validCurrency)
    }

    server.setupRouter()
    return server, nil
}

At the end of this function, we should return a nil error. And that will be it!

However, as we added a new config parameter to the NewServer() function, some unit tests that we wrote before are broken. So let’s fix them!

Fix broken unit tests

In the api/main_test.go file, I’m gonna define a function newTestServer() that will create a new server for test. It takes a testing.T object and a db.Store interface as input. And it will return a Server object as output.

In this function, let’s create a new config object, with TokenSymmetricKey is util.RandomString of 32 characters, and AccessTokenDuration is 1 minute.

func newTestServer(t *testing.T, store db.Store) *Server {
    config := util.Config{
        TokenSymmetricKey:   util.RandomString(32),
        AccessTokenDuration: time.Minute,
    }

    server, err := NewServer(config, store)
    require.NoError(t, err)

    return server
}

Then we create a new server with that config object and the input store interface. Require no errors, and finally return the created server.

Now get back to the api/transfer_test.go file. Here, instead of NewServer(), we will call newTestServer, and pass in the testing.T object and the mock store.

func TestTransferAPI(t *testing.T) {
    ...

    for i := range testCases {
        tc := testCases[i]

        t.Run(tc.name, func(t *testing.T) {
            ...

            server := newTestServer(t, store)
            recorder := httptest.NewRecorder()

            ...
        })
    }
}

We do the same for the server inside api/user_test.go file and api/account_test.go file as well. There are several calls of NewServer() in these files, so we have to change all of them to newTestServer().

Alright, now everything is updated. Let’s run the whole api package tests!

All passed! Excellent! So the tests are now working well with the new Server struct.

But there’s one more place we need to update, that’s the main entry point of our server: main.go

func main() {
    config, err := util.LoadConfig(".")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    conn, err := sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server, err := api.NewServer(config, store)
    if err != nil {
        log.Fatal("cannot create server:", err)
    }

    err = server.Start(config.ServerAddress)
    if err != nil {
        log.Fatal("cannot start server:", err)
    }
}

Here, in this main() function, we have to add config to the api.NewServer() call. And this call will return a server and an error.

If error is not nil, we just write a fatal log, saying "cannot create server". Just like that, and we’re done!

Now it’s time to build the login user API!

Implement login user handler

Let’s open the api/user.go file!

The login API’s request payload must contain the username and password, which is very similar to the createUserRequest:

type createUserRequest struct {
    Username string `json:"username" binding:"required,alphanum"`
    Password string `json:"password" binding:"required,min=6"`
    FullName string `json:"full_name" binding:"required"`
    Email    string `json:"email" binding:"required,email"`
}

So I’m gonna copy this struct, and paste it to the end of this file. Then let’s change the struct name to loginUserRequest and remove the FullName and Email fields, just keep the Username and Password fields.

type loginUserRequest struct {
    Username string `json:"username" binding:"required,alphanum"`
    Password string `json:"password" binding:"required,min=6"`
}

Next, let’s define the loginUserResponse struct. The most important field that should be returned to the client is AccessToken string. This is the token that we will create using the token maker interface.

type loginUserResponse struct {
    AccessToken string       `json:"access_token"`
}

Beside the access token, we might also want to return some information of the logged in user, just like the one we returned in the create user API:

type createUserResponse struct {
    Username          string    `json:"username"`
    FullName          string    `json:"full_name"`
    Email             string    `json:"email"`
    PasswordChangedAt time.Time `json:"password_changed_at"`
    CreatedAt         time.Time `json:"created_at"`
}

So to make this struct reusable, I’m gonna change its name to just userResponse. It will be the type of the User field in this loginUserResponse struct:

type userResponse struct {
    Username          string    `json:"username"`
    FullName          string    `json:"full_name"`
    Email             string    `json:"email"`
    PasswordChangedAt time.Time `json:"password_changed_at"`
    CreatedAt         time.Time `json:"created_at"`
}

type loginUserResponse struct {
    AccessToken string       `json:"access_token"`
    User        userResponse `json:"user"`
}

Then let’s copy the userResponse object from the createUser() handler, and define a newUserResponse() function at the top.

func newUserResponse(user db.User) userResponse {
    return userResponse{
        Username:          user.Username,
        FullName:          user.FullName,
        Email:             user.Email,
        PasswordChangedAt: user.PasswordChangedAt,
        CreatedAt:         user.CreatedAt,
    }
}

The role of this function is to convert the input db.User object into userResponse. The reason we do that is because there’s a sensitive data inside the db.User struct, which is the hashed_password, that we don’t want to expose to the client.

OK, so now in the createUser() handler, we can just call the newUserResponse() function to create the response object.

func (server *Server) createUser(ctx *gin.Context) {
    ...

    user, err := server.store.CreateUser(ctx, arg)
    ...

    rsp := newUserResponse(user)
    ctx.JSON(http.StatusOK, rsp)
}

The newUserResponse() function will be useful for our new loginUser() handler as well.

Alright, now let’s add a new method to the server struct: loginUser(). Similar as in other API handlers, this function will take a gin.Context object as input.

Inside, we declare a request object of type loginUserRequest, and we call the ctx.ShouldBindJSON() function with a pointer to that request object. This will bind all the input parameters of the API into the request object.

func (server *Server) loginUser(ctx *gin.Context) {
    var req loginUserRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    ...
}

If error is not nil, we send a response with status 400 Bad Request to the client, together with the errorResponse() body to explain why it failed.

If there’s no error, we will find the user from the database by calling server.store.GetUser() with the context ctx and req.Username.

func (server *Server) loginUser(ctx *gin.Context) {
    ...

    user, err := server.store.GetUser(ctx, req.Username)
    if err != nil {
        if err == sql.ErrNoRows {
            ctx.JSON(http.StatusNotFound, errorResponse(err))
            return
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

If the error returned by this call is not nil, then there are 2 possible cases:

The first case is when the username doesn’t exist, which means error equals to sql.ErrNoRows. In this case, we send a response with status 404 Not Found to the client, and return immediately.
The second case is an unexpected error occurs when talking to the database. In this case, we send a 500 Internal Server Error status to the client, and also return right away.

If everything goes well, and no errors occur, we will have to check if the password provided by the client is correct or not. So we call util.CheckPassword() with the input req.Password and user.HashedPassword.

func (server *Server) loginUser(ctx *gin.Context) {
    ...

    err = util.CheckPassword(req.Password, user.HashedPassword)
    if err != nil {
        ctx.JSON(http.StatusUnauthorized, errorResponse(err))
        return
    }

    ...
}

If this function returns a not nil error, then it means the provided password is incorrect. We will send a response with status 401 Unauthorized to the client, and return.

Only when the password is correct, then we will create a new access token for this user.

Let’s call server.tokenMaker.CreateToken(), pass in user.Username, and server.config.AccessTokenDuration as input arguments.

func (server *Server) loginUser(ctx *gin.Context) {
    ...

    accessToken, err := server.tokenMaker.CreateToken(
        user.Username,
        server.config.AccessTokenDuration,
    )
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    rsp := loginUserResponse{
        AccessToken: accessToken,
        User:        newUserResponse(user),
    }
    ctx.JSON(http.StatusOK, rsp)
}

If an unexpected error occurs, we just return 500 Internal Server Error status code.

Otherwise, we will build the loginUserResponse object, where AccessToken is the created access token, and User is newUserResponse(user). We then send this response to the client with a 200 OK status code.

And that’s basically it! The loginUser() handler function is completed:

func (server *Server) loginUser(ctx *gin.Context) {
    var req loginUserRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    user, err := server.store.GetUser(ctx, req.Username)
    if err != nil {
        if err == sql.ErrNoRows {
            ctx.JSON(http.StatusNotFound, errorResponse(err))
            return
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    err = util.CheckPassword(req.Password, user.HashedPassword)
    if err != nil {
        ctx.JSON(http.StatusUnauthorized, errorResponse(err))
        return
    }

    accessToken, err := server.tokenMaker.CreateToken(
        user.Username,
        server.config.AccessTokenDuration,
    )
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    rsp := loginUserResponse{
        AccessToken: accessToken,
        User:        newUserResponse(user),
    }
    ctx.JSON(http.StatusOK, rsp)
}

Add login API route to the server

The next step is to add a new API endpoint to the server that will route the login request to the loginUser() handler.

I’m gonna put it next to the create user route. So router.POST(), the path should be /users/login, and the handler function is server.loginUser().

func NewServer(config util.Config, store db.Store) (*Server, error) {
    ...

    router := gin.Default()

    router.POST("/users", server.createUser)
    router.POST("/users/login", server.loginUser)

    ...
}

And we’re done!

However, this NewServer() function is getting quite long now, which makes it harder to read.

So I’m gonna split the routing part into a separate method of the server struct. Let’s call it setupRouter(). Then paste in all the routing codes.

func (server *Server) setupRouter() {
    router := gin.Default()

    router.POST("/users", server.createUser)
    router.POST("/users/login", server.loginUser)

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccounts)

    router.POST("/transfers", server.createTransfer)

    server.router = router
}

We should move the gin router variable here as well. And at the end, we should save this router to the server.router field.

Then all we have to do in the NewServer() function is to call server.setupRouter().

func NewServer(config util.Config, store db.Store) (*Server, error) {
    tokenMaker, err := token.NewPasetoMaker(config.TokenSymmetricKey)
    if err != nil {
        return nil, fmt.Errorf("cannot create token maker: %w", err)
    }

    server := &Server{
        config:     config,
        store:      store,
        tokenMaker: tokenMaker,
    }

    if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
        v.RegisterValidation("currency", validCurrency)
    }

    server.setupRouter()
    return server, nil
}

And now we’ve really completed the login user API’s implementation. It’s pretty easy and straightforward, isn’t it?

Run the server and send login user request

Let’s run the server and send some requests to see how it goes!

❯ make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /users                    --> github.com/techschool/simplebank/api.(*Server).createUser-fm (3 handlers)
[GIN-debug] POST   /users/login              --> github.com/techschool/simplebank/api.(*Server).loginUser-fm (3 handlers)
[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (4 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (4 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccounts-fm (4 handlers)
[GIN-debug] POST   /transfers                --> github.com/techschool/simplebank/api.(*Server).createTransfer-fm (4 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8080

As you can see here, the login user API is up and running.

Now I’m gonna open Postman, create a new request and set it method to POST. The URL should be http://localhost:8080/users/login, then select body, raw, and JSON format.

The JSON body will have 2 fields: username and password. In the database, there are 4 users that we already created in previous lectures. So I’m gonna use the first user with username quang1 and the password is secret.

OK let’s send this request:

Voilà! It's successful!

We’ve got the PASETO v2 local access token here. And all the information of the logged in user in this object. So it worked!

Let’s try login with an invalid password: xyz. Send the request.

Now we’ve got 400 Bad Request because the password we sent was too short. That's because we have a validation rule for the password field to have at least 6 characters:

type loginUserRequest struct {
    ...
    Password string `json:"password" binding:"required,min=6"`
}

So let’s change this value to xyz123. And send the request again.

This time we’ve got 401 Unauthorized status code, and the error is: "hashed password is not the hash of the given password", or in other words, the provided password is incorrect.

Now let’s try the case when username doesn’t exist. I’m gonna change the username to quang10, and send the request again.

This time, we’ve got 404 Not Found status code. That’s exactly what we expected! So the login user API is working very well.

Before we finish, I’m gonna show you how easy it is to change the token types.

Change the token type

Right now, we’re using PASETO, but since it implements the same token maker interface with JWT, it will be super easy if we want to switch to JWT.

All we have to do is just change the token.NewPasetoMaker() call to token.NewJWTMaker() in the api/server.go file:

func NewServer(config util.Config, store db.Store) (*Server, error) {
    tokenMaker, err := token.NewJWTMaker(config.TokenSymmetricKey)
    if err != nil {
        return nil, fmt.Errorf("cannot create token maker: %w", err)
    }

    ...
}

That’s it! Let’s restart the server, then go back to Postman and send the login request one more time.

As you can see, the request is successful. And now the access token looks different because it’s a JWT token, not a PASETO token as before.

OK, now as we’ve confirmed that it worked, I’m gonna revert the token type to PASETO because it’s better than JWT in my opinion.

func NewServer(config util.Config, store db.Store) (*Server, error) {
    tokenMaker, err := token.NewPasetoMaker(config.TokenSymmetricKey)
    if err != nil {
        return nil, fmt.Errorf("cannot create token maker: %w", err)
    }

    ...
}

And that wraps up this lecture about implementing login user API in Go.

I hope you find it useful. Thanks a lot for reading, and see you soon in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

How to create and verify JWT & PASETO token in Golang

TECH SCHOOL — Mon, 05 Apr 2021 13:02:05 +0000

Hello everyone!

In the previous lecture, we have learned about token based authentication and why PASETO is better than JWT in term of security practice.

Today we will learn how to implement both of them in Golang to see why PASETO is also much easier and simpler to implement compared to JWT.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Declare token Maker interface

Alright, let’s start!

First, I’m gonna create a new package called token. Then create a new file maker.go inside this package.

The idea is to declare a general token.Maker interface to manage the creation and verification of the tokens. Then later, we will write a JWTMaker and a PasetoMaker struct that implements this interface. By doing so, we can easily switch between different types of token makers whenever we want.

So this interface will have 2 methods:



type Maker interface {
    CreateToken(username string, duration time.Duration) (string, error)
    VerifyToken(token string) (*Payload, error)
}

The CreateToken() method takes a username string and a valid duration as input. It returns a signed token string or an error. Basically, this method will create and sign a new token for a specific username and valid duration.

The second method is VerifyToken(), which takes a token string as input, and returns a Payload object or an error. We will declare this Playload struct in a moment. The role of this VerifyToken() method is to checks if the input token is valid or not. If it is valid, the method will return the payload data stored inside the body of the token.

Declare token Payload struct

OK, now let’s create a new payload.go file, and define the Payload struct inside it. This struct will contain the payload data of the token.

The most important field is Username, which is used to identify the token owner.

Then an IssuedAt field to know when the token is created.

When using token based authentication, it’s crucial to make sure that each access token only has a short valid duration. So we need an ExpiredAt field to store the time at which the token will be expired.



type Payload struct {
    ID        uuid.UUID `json:"id"`
    Username  string    `json:"username"`
    IssuedAt  time.Time `json:"issued_at"`
    ExpiredAt time.Time `json:"expired_at"`
}

Normally these 3 fields should be enough. However, if we want to have a mechanism to invalidate some specific tokens in case they are leaked, we need to add an ID field to uniquely identify each token.

Here I use the UUID type for this field. The type is defined in the google/uuid package, wo we have to run go get command to download and add it to the project.



go get github.com/google/uuid

Next, I’m gonna define a function NewPayload() that takes a username and a duration as input arguments, and returns a Payload object or an error. This function will create a new token payload with a specific username and duration.



func NewPayload(username string, duration time.Duration) (*Payload, error) {
    tokenID, err := uuid.NewRandom()
    if err != nil {
        return nil, err
    }

    payload := &Payload{
        ID:        tokenID,
        Username:  username,
        IssuedAt:  time.Now(),
        ExpiredAt: time.Now().Add(duration),
    }
    return payload, nil
}

First, we have to call uuid.NewRandom() to generate a unique token ID. If an error occurs, we simply return a nil payload and the error itself.

Else, we create the payload, where ID is the generated random token UUID, Username is the input username, IssuedAt is time.Now(), and ExpiredAt is time.Now().Add(duration).

Then we just return this payload object and a nil error. And we’re done!

Implement JWT Maker

Now we’re gonna implement a JWTMaker. We will need a JWT package for Golang, so let’s open the browser and search for jwt golang.

There might be many different packages, but I think this one is the most popular: https://github.com/dgrijalva/jwt-go. So let’s copy its URL, and run go get in the terminal to install the package:



go get github.com/dgrijalva/jwt-go

OK, the package is installed. Now let’s go back to visual studio code.

I’m gonna create a new file jwt_maker.go inside the token package. Then declare a new type JWTMaker struct. This struct is a JSON web token maker, which implements the token.Maker interface.

In this tutorial, I will use symmetric key algorithm to sign the tokens, so this struct will have a field to store the secret key.



type JWTMaker struct {
    secretKey string
}

Next, let’s add a function NewJWTMaker() that takes a secretKey string as input, and returns a token.Maker interface, or an error as output.

By returning the interface, we will make sure that our JWTMaker must implement the token.Maker interface. We will see how the go compiler checks this for us in a moment.

Now, although the algorithm we’re gonna use doesn’t require how long the secret key should be, it’s still a good idea to ensure that the key should not be too short, for better security. So I will declare a constant minSecretKeySize = 32 characters.



const minSecretKeySize = 32

func NewJWTMaker(secretKey string) (Maker, error) {
    if len(secretKey) < minSecretKeySize {
        return nil, fmt.Errorf("invalid key size: must be at least %d characters", minSecretKeySize)
    }
    return &JWTMaker{secretKey}, nil
}

Then inside this function, we check if the length of the secret key is less than minSecretKeySize or not. If it is, we just return a nil object and an error saying that the key must have at least 32 characters.

Otherwise, we return a new JWTMaker object with the input secretKey, and a nil error.

Now you can see a red line here, because the JWTMaker object that we’ve created doesn’t implement the required methods of the token.Maker interface, which is the return type of this function.

So in order to fix this, we have to add the CreateToken() and VerifyToken() methods to this struct.

Let’s copy them from the token.Maker interface, and paste them here. Then let’s add the JWTMaker receiver in front of each method.



func (maker *JWTMaker) CreateToken(username string, duration time.Duration) (string, error) {}

func (maker *JWTMaker) VerifyToken(token string) (*Payload, error) {}

OK, now the red line is gone! Let’s implement the CreateToken() method!

Implement the JWT CreateToken method

First we create a new token payload by calling NewPayload(), and pass in the input username and valid duration.

If error is not nil, we return an empty token string and the error. Else, we create a new jwtToken by calling the jwt.NewWithClaims() function of the jwt-go package.

This function expects 2 input arguments:

First, the signing method (or algorithm). I’m gonna use HS256 in this case.
Then the claims, which actually is our created payload.



func (maker *JWTMaker) CreateToken(username string, duration time.Duration) (string, error) {
    payload, err := NewPayload(username, duration)
    if err != nil {
        return "", err
    }

    jwtToken := jwt.NewWithClaims(jwt.SigningMethodHS256, payload)
    return jwtToken.SignedString([]byte(maker.secretKey))
}

Finally, to generate a token string, we call jwtToken.SignedString(), and pass in the secretKey after converting it to []byte slice.

Here we have an error because our Payload struct doesn’t implement the jwt.Claims interface. It’s missing one method called Valid().

The jwt-go package needs this method to check if the token payload is valid or not. So let’s open the payload.go to add this method.

The signature of this method is very simple. It doesn’t take any input argument, and only return an error in case the token is invalid. You can easily find this method in the implementation of the jwt-go package.



var ErrExpiredToken = errors.New("token has expired")

func (payload *Payload) Valid() error {
    if time.Now().After(payload.ExpiredAt) {
        return ErrExpiredToken
    }
    return nil
}

The simplest but also the most important thing we must check is the expiration time of the token.

If time.Now() is after the payload.ExpiredAt, then it means that the token has expired. So we just return a new error saying: token has expired.

We should declare this error as a public constant: ErrExpiredToken, so that we can check the error type from outside.

If the token is not expired, then we simply return nil. And that’s it! The Valid function is done.

Now back to our jwt_maker.go file, we can see that the red line on the payload object is gone.

As we’ve imported the jwt-go package, we should run go mod tidy in the terminal to add it to the go.mod file.



module github.com/techschool/simplebank

go 1.15

require (
    github.com/dgrijalva/jwt-go v3.2.0+incompatible
    github.com/gin-gonic/gin v1.6.3
    github.com/go-playground/validator/v10 v10.4.1
    github.com/golang/mock v1.4.4
    github.com/google/uuid v1.1.4
    github.com/lib/pq v1.9.0
    github.com/o1egl/paseto v1.0.0
    github.com/spf13/viper v1.7.1
    github.com/stretchr/testify v1.6.1
    golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad
)

The jwt-go version I’m using is 3.2.0. In the future, it might be possible that you will use a newer version, such as 4.0, then the function and interface signatures might be different. However, the idea should be similar.

OK, now the CreateToken() method is done. Let’s move on to the VerifyToken() method.

Implement the JWT VerifyToken method

This will be a bit more complicated. First, we have to parse the token by calling jwt.ParseWithClaims and pass in the input token string, an empty Payload object, and a key function.

What is a key function? Well, basically, it’s a function that receives the parsed but unverified token. You should verify its header to make sure that the signing algorithm matches with what you normally use to sign the tokens.

Then if it matches, you return the key so that jwt-go can use it to verify the token. This step is very important to prevent the trivial attack mechanism as I explained in the previous lecture.

Alright, I’m gonna copy this function signature, and paste it here. Let’s set this input argument’s name to token, and its type should be jwt.Token. Then we just pass the keyFunc to the ParseWithClaims() call.



func (maker *JWTMaker) VerifyToken(token string) (*Payload, error) {
    keyFunc := func(token *jwt.Token) (interface{}, error) {
        _, ok := token.Method.(*jwt.SigningMethodHMAC)
        if !ok {
            return nil, ErrInvalidToken
        }
        return []byte(maker.secretKey), nil
    }

    jwtToken, err := jwt.ParseWithClaims(token, &Payload{}, keyFunc)
    ...
}

In the key function, we can get its signing algorithm via the token.Method field. Note that its type is a SigningMethod, which is just an interface. So we have to try converting it to a specific implementation.

In our case, we convert it to SigningMethodHMAC because we’re using HS256, which is an instance of the SigningMethodHMAC struct.

This conversion can be successful or not. If it is not ok, then it means that the algorithm of the token doesn’t match with our signing algorithm, so it’s clearly an invalid token.

We have to return a nil key with an ErrInvalidToken. I’m gonna declare this new error inside the payload.go file, the same place as the ErrExpiredToken. They are different types of error that will be returned by our VerifyToken() function.



var (
    ErrInvalidToken = errors.New("token is invalid")
    ErrExpiredToken = errors.New("token has expired")
)

OK, back to the JWTMaker. If the conversion is successful, then it means the algorithm matches. We can just return the secret key that we’re using to sign the token after converting it to []byte slice, and a nil error.

OK now the key function is ready. Let’s continue with the ParseWithClaims function call. If it returns a not nil error, then there might be 2 different scenarios: either the token is invalid or it is expired.

But now things get more complicated when we want to differentiate these 2 cases. If we follow the implementation of the jwt-go package, we can see that it automatically calls token.Claims.Valid() function under the hood.

And in our implementation of this function, we’re returning ErrExpiredToken error. However, jwt-go has secretly hiden this original error inside its own ValidationError object.

So in order to figure out the real error type, we have to convert the returned error of the ParseWithClaims() function to jwt.ValidationError



func (maker *JWTMaker) VerifyToken(token string) (*Payload, error) {
    ...

    jwtToken, err := jwt.ParseWithClaims(token, &Payload{}, keyFunc)
    if err != nil {
        verr, ok := err.(*jwt.ValidationError)
        if ok && errors.Is(verr.Inner, ErrExpiredToken) {
            return nil, ErrExpiredToken
        }
        return nil, ErrInvalidToken
    }

    ...
}

Here I assign the converted error to the verr variable. If the conversion is OK, we use the errors.Is() function to check if the verr.Inner is actually the ErrExpiredToken or not.

If it is, we just return a nil payload and the ErrExpiredToken. Otherwise, we just return nil and ErrInvalidToken.

In case everything is good, and the token is successfully parsed and verified, we will try to get its payload data by converting jwtToken.Claims into a Payload object.



func (maker *JWTMaker) VerifyToken(token string) (*Payload, error) {
    keyFunc := func(token *jwt.Token) (interface{}, error) {
        _, ok := token.Method.(*jwt.SigningMethodHMAC)
        if !ok {
            return nil, ErrInvalidToken
        }
        return []byte(maker.secretKey), nil
    }

    jwtToken, err := jwt.ParseWithClaims(token, &Payload{}, keyFunc)
    if err != nil {
        verr, ok := err.(*jwt.ValidationError)
        if ok && errors.Is(verr.Inner, ErrExpiredToken) {
            return nil, ErrExpiredToken
        }
        return nil, ErrInvalidToken
    }

    payload, ok := jwtToken.Claims.(*Payload)
    if !ok {
        return nil, ErrInvalidToken
    }

    return payload, nil
    }

If it’s not OK, then we just return nil and ErrInvalidToken. Else, we return the payload object and a nil error.

And that’s it! The JWTMaker is completed. Now let’s write some unit test for it!

Test JWT Maker

I’m gonna create a new file jwt_maker_test.go inside the token package. Then let’s add a new function TestJWTMaker() that takes a testing.T object as input.

First, we have to create a new maker by calling the NewJWTMaker() function and pass in a random secret key of 32 characters. We require no errors to be returned here.

Next, we generate a username with the util.RandomOwner() function, and let’s say the token valid duration is gonna be 1 minute.

Let’s also declared 2 variables to compare the result later:

The issuedAt time should be time.Now()
And we add the duration to this issuedAt time to get the expiredAt time of the token.



func TestJWTMaker(t *testing.T) {
    maker, err := NewJWTMaker(util.RandomString(32))
    require.NoError(t, err)

    username := util.RandomOwner()
    duration := time.Minute

    issuedAt := time.Now()
    expiredAt := issuedAt.Add(duration)

    token, err := maker.CreateToken(username, duration)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    payload, err := maker.VerifyToken(token)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    require.NotZero(t, payload.ID)
    require.Equal(t, username, payload.Username)
    require.WithinDuration(t, issuedAt, payload.IssuedAt, time.Second)
    require.WithinDuration(t, expiredAt, payload.ExpiredAt, time.Second)
}

OK, now we generate the token by calling maker.CreatToken function, and pass in the username and duration. Require no errors, and require the output token to not be empty.

Next, we call maker.VerifyToken to make sure that the token is valid and also get back its payload data. We require no errors, and require the payload object to be not empty.

Then we need to check all fields of the payload object.

First the payload.ID should be not zero.
Then the payload.Username should equal to the input username.
We use require.WithinDuration to compare the payload.IssuedAt field with the expected issuedAt time we saved above. They should not be different by more than 1 second.
Likewise, we compare the payload.ExpiredAt field with the expected expiredAt time in the same manner.

And we’re done! Let’s run this unit test!

It passed. Cool! So that’s how we test the happy case.

Now let’s add another test to check the expired JWT token case.

Similar as before, we first have to create a new JWTMaker. Then we will create an expired token by calling maker.CreateToken(), pass in a random username and a negative duration.



func TestExpiredJWTToken(t *testing.T) {
    maker, err := NewJWTMaker(util.RandomString(32))
    require.NoError(t, err)

    token, err := maker.CreateToken(util.RandomOwner(), -time.Minute)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    payload, err := maker.VerifyToken(token)
    require.Error(t, err)
    require.EqualError(t, err, ErrExpiredToken.Error())
    require.Nil(t, payload)
}

We require no errors to be returned, and the token should not be empty. Now we will verify this output token.

This time, we expect an error to be returned. And more specifically, it should be ErrExpiredToken. Finally, the output payload should be nil.

OK, let’s run the test!

It passed. Excellent!

The last test we’re gonna write is to check the invalid token case, where a None algorithm header is used. This is a well-known attack technique that I have told you in the previous lecture.

First I’m gonna create a new payload with a random username and a duration of 1 minute. Require no errors. Then let’s make a new token by calling jwt.NewWithClaims() with the jwt.SigningMethodNone and the created payload.

Now we have to sign this token using the SignedString() method. But we cannot just use any random secret key here, because the jwt-go library has completely forbidden from using the None algorithm to sign the token.

We can only use it for testing when we pass in this special constant: jwt.UnsafeAllowNoneSignatureType as the secret key.

If you follow the implementation of this value, you can see that normally the None sign method is disallowed, unless the input key is this special constant. It basically means that you’re aware of what you’re doing. Make sure you only use it for testing, and not for production.



func TestInvalidJWTTokenAlgNone(t *testing.T) {
    payload, err := NewPayload(util.RandomOwner(), time.Minute)
    require.NoError(t, err)

    jwtToken := jwt.NewWithClaims(jwt.SigningMethodNone, payload)
    token, err := jwtToken.SignedString(jwt.UnsafeAllowNoneSignatureType)
    require.NoError(t, err)

    maker, err := NewJWTMaker(util.RandomString(32))
    require.NoError(t, err)

    payload, err = maker.VerifyToken(token)
    require.Error(t, err)
    require.EqualError(t, err, ErrInvalidToken.Error())
    require.Nil(t, payload)
}

OK let’s get back to our code. We have to create a new JWTMaker as in the other tests. And now we call maker.VerifyToken() to verify the token we’ve signed above.

This time, the function should also return an error, and the error should be equal to ErrInvalidToken. The output payload should also be nil.

Alright, now let’s run the test!

It passed! Awesome!

So now you know how to implement and test JWT in go.

Although I think the jwt-go package was quite well implemented in terms of preventing security mistakes, it’s still a bit complicated and difficult to use than necessary, especially for the token verification part.

Implement PASETO Maker

Now I’m gonna show you how to implement the same token maker interface but using PASETO instead. It would be much easier and cleaner than JWT.

OK, let’s open the browser and search for paseto golang. Open its Github page and copy the URL: https://github.com/o1egl/paseto. Then run go get with this URL to download the package:



go get github.com/o1egl/paseto

Now get back to our project, I’m gonna create a new file: paseto_maker.go inside the token folder.

Similar to what we’ve done with JWT, Let’s declare a type PasetoMaker struct, which will implement the same token.Maker interface, but use PASETO instead of JWT.

We’re gonna use the latest version of PASETO at the moment, which is version 2. So the PasetoMaker struct will have a paseto field of type paseto.V2.



type PasetoMaker struct {
    paseto       *paseto.V2
    symmetricKey []byte
}

And as I just want to use the token locally for our banking API, we will use symmetric encryption to encrypt the token payload. Therefore, we need a field to store the symmetricKey here.

OK now let’s add a function NewPasetoMaker(), which takes a symmetricKey string as input, and returns a token.Maker interface or an error. This function will create a new PasetoMaker instance.

Paseto version 2 uses Chacha20 Poly1305 algorithm to encrypt the payload. So here we have to check the length of the symmetric key to make sure that it has the correct size that’s required by the algorithm.



import (
    "github.com/aead/chacha20poly1305"
    "github.com/o1egl/paseto"
)

func NewPasetoMaker(symmetricKey string) (Maker, error) {
    if len(symmetricKey) != chacha20poly1305.KeySize {
        return nil, fmt.Errorf("invalid key size: must be exactly %d characters", chacha20poly1305.KeySize)
    }

    maker := &PasetoMaker{
        paseto:       paseto.NewV2(),
        symmetricKey: []byte(symmetricKey),
    }

    return maker, nil
}

If the key length is not correct then we just return a nil object and an error saying invalid key size. It must have exactly this number of characters.

Else, we just create a new PasetoMaker object that contains paseto.NewV2() and the input symmetricKey converted to []byte slice.

Then we return this maker object and a nil error.

Again, here we see a red line under maker object because it’s not implementing the token.Maker interface yet. So let’s do the same as what we’ve done for JWTMaker.

I’m gonna copy these 2 required methods of the token maker interface, and add the PasetoMaker receiver in front of them.



func (maker *PasetoMaker) CreateToken(username string, duration time.Duration) (string, error) {}

func (maker *PasetoMaker) VerifyToken(token string) (*Payload, error) {}

OK, now the red line is gone. Let’s implement the CreateToken() method.

Implement PASETO CreateToken method

Similar as before, we first have to create a new payload with the input username and duration. If error is not nil, we return an empty string and the error to the caller.

Otherwise, we return maker.paseto.Encrypt(), and pass in the maker.symmetricKey, and the payload object. The last argument is an optional footer, which we don’t need, so I put nil here.



func (maker *PasetoMaker) CreateToken(username string, duration time.Duration) (string, error) {
    payload, err := NewPayload(username, duration)
    if err != nil {
        return "", err
    }

    return maker.paseto.Encrypt(maker.symmetricKey, payload, nil)
}

And that’s it! Pretty short and simple, right?

If we follow the implementation of this Encrypt() function, we can see that, it is using Chacha Poly cipher algorithm.

And inside the newCipher() function, it also checks the input key size to make sure that it equals to 32 bytes.

Implement PASETO VerifyToken method

Alright, now let’s go back to our code and implement the VerifyToken() method. It’s very simple!

We just need to declare an empty payload object to store the decrypted data. Then call maker.paseto.Decrypt() with the input token, the symmetricKey, the payload and a nil footer.



func (maker *PasetoMaker) VerifyToken(token string) (*Payload, error) {
    payload := &Payload{}

    err := maker.paseto.Decrypt(token, maker.symmetricKey, payload, nil)
    if err != nil {
        return nil, ErrInvalidToken
    }

    err = payload.Valid()
    if err != nil {
        return nil, err
    }

    return payload, nil
}

If error is not nil, we return nil payload an ErrInvalidToken. Else, we will check if the token is valid or not by calling payload.Valid().

If there’s an error, we just return nil payload and the error itself. Otherwise, we return the payload and a nil error.

And that’s it! Very concise and much simpler than JWT, right?

Test PASETO Maker

OK, now let’s write some unit tests!

I’m gonna create a new file: paseto_maker_test.go inside the token package. Actually the test would be almost identical to the one we wrote for JWT, so I’m just gonna copy it here.

Change its name to TestPasetoMaker. Then here, instead of NewJWTMaker(), we call NewPasetoMaker().



func TestPasetoMaker(t *testing.T) {
    maker, err := NewPasetoMaker(util.RandomString(32))
    require.NoError(t, err)

    username := util.RandomOwner()
    duration := time.Minute

    issuedAt := time.Now()
    expiredAt := issuedAt.Add(duration)

    token, err := maker.CreateToken(username, duration)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    payload, err := maker.VerifyToken(token)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    require.NotZero(t, payload.ID)
    require.Equal(t, username, payload.Username)
    require.WithinDuration(t, issuedAt, payload.IssuedAt, time.Second)
    require.WithinDuration(t, expiredAt, payload.ExpiredAt, time.Second)
}

We don’t have to change anything else because PasetoMaker implements the same token.Maker interface as JWTMaker.

Let’s run the test!

It passed!

Now let’s copy the test for the expired token case! Change its name to TestExpiredPasetoToken, and update this call to NewPasetoMaker().



func TestExpiredPasetoToken(t *testing.T) {
    maker, err := NewPasetoMaker(util.RandomString(32))
    require.NoError(t, err)

    token, err := maker.CreateToken(util.RandomOwner(), -time.Minute)
    require.NoError(t, err)
    require.NotEmpty(t, token)

    payload, err := maker.VerifyToken(token)
    require.Error(t, err)
    require.EqualError(t, err, ErrExpiredToken.Error())
    require.Nil(t, payload)
}

Then run the test!

It also passed. Excellent!

We don’t need the last test because the None algorithm just doesn’t exist in PASETO. You can write another test to check the invalid token case if you want. I leave it as an exercise for you to practice.

And that brings us to the end of this lecture. We have learned how to implement both JWT and PASETO using Go to create and verify access tokens.

In the next article, I will show you how to use them in the login API, where users provide their username & password, and the server will return an access token if the provided credentials are correct.

Thanks a lot for reading, and see you soon in the next lecture!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Why PASETO is better than JWT for token-based authentication?

TECH SCHOOL — Sun, 21 Feb 2021 14:49:00 +0000

JWT vs PASETO

Nowadays, token-based authentication has become more and more popular in the development of web and mobile applications.

There are many different types of tokens, but among them, JSON web token (or JWT) is one of the most widely used.

However, in the past few years, we’ve also discovered several security issues regarding JSON web token, mainly because of its poorly designed standard.

So recently people have started migrating to other types of tokens such as PASETO, which promises to bring better security to the application.

In this lecture, we will learn everything about the security issues of JWT and how PASETO is designed to solve all of those problems.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Token-based authentication

First, let’s talk a bit about token-based authentication.

Basically, in this authentication mechanism, the client will make the first request to log in user, where it provides the username and password to the server.

The server will check if the username and password are correct or not. If they are, the server will create and sign a token with its secret or private key, then sends back a 200 OK response to the client together with the signed access token.

The reason it’s called access token is that later the client will use this token to get access to other resources on the server.

For example, let’s say the client wants to get the list of bank accounts that belong to the logged-in user. Then it will make a GET /accounts request to the server, where it embeds the user’s access token in the header of the request.

Upon receiving this request, the server will verify if the provided token is valid or not. If it is valid, the request will be authorized, and a 200 OK response will be sent back to the client with the list of user’s bank accounts.

Note that an access token normally has a lifetime duration before it gets expired. And during this time, the client can use the same token to send multiple requests to the server.

So that’s how the token-based authentication works.

JSON Web Token

Now let’s talk about JSON Web Token! Here’s an example of a JSON Web Token:

It is a base64 encoded string, composed of 3 main parts, separated by a dot.

The first part (with the color red) is the header of the token. When we decode this part, we will get a JSON object that contains the token type JWT, and the algorithm used to sign the token: HS256 in this case.

The second part (in purple) of the token is the payload data. This part is where we store information about the logged-in user, such as username, and also the timestamp at which this token will be expired.

You can customize this JSON payload to store any other information you want. In this case, we also have an ID field to uniquely identify the token. It will be useful in case we want to revoke access of the token in case it is leaked.

Keep in mind that all data stored in the JWT is only base64-encoded, not encrypted. So you don’t need the secret/private key of the server in order to decode its content.

It also means that we can easily encode the header and payload data without the key. So how can the server verify the authenticity of the access token?

Well, that’s the purpose of the third part of the token: the digital signature (in blue color). If you don’t know how the digital signature algorithm works, then I recommend you to read my post about SSL/TLS before continuing.

The idea is simple, only the server has the secret/private key to sign the token. So if a hacker attempts to create a fake token without the correct key, it will be easily detected by the server in the verification process.

The JWT standards provide many different types of digital signature algorithms, but they can be classified into 2 main categories.

Symmetric-key algorithm

The first one is symmetric-key algorithm, where the same secret key is used to both sign and verify the tokens.

And since there’s only 1 key, it should be kept secret. So this algorithm is suitable for local use only, or in other words, for internal services, where the secret key can be shared.

Some specific algorithms which belong to this symmetric-key category are: HS256, HS384, and HS512.

Here HS256 is the combination of HMAC and SHA-256. HMAC stands for Hash-based Message Authentication Code, and SHA is the Secure Hashing Algorithm. While 256/384/512 is the number of output bits.

Symmetric-key algorithm is very efficient and suitable for most applications.

However, we cannot use it in case there’s an external third-party service that wants to verify the token, because it would mean we must give them our secret key.

In that case, we must use the second category: asymmetric-key algorithm.

Asymmetric-key algorithm

In this type of algorithm, there’s a pair of keys instead of just 1 single secret key.

The private key is used to sign the token, while the public key is used only to verify it.

Therefore, we can easily share our public key with any external third-party services without worrying about leaking our private key.

Within this asymmetric-key category, there are several groups of algorithms, such as RS group, PS group, or ES group.

Here, RS256 is basically RSA algorithm with PKCSv1.5 and SHA256.

PS256 is also RSA algorithm but with Probabilistic Signature Scheme and SHA256. It was designed to be more secured than PKCSv1.5

And the last one ES256 is simply Elliptic Curve Digital Signature Algorithm with SHA256.

Problems of JWT

OK, so far it sounds like JWT is a good standard, and it gives us a lot of flexibility to choose whatever signing algorithms we want. So what exactly are its problems?

Weak algorithms

Well, the first problem is weak signing algorithms. JWT gives developers too many algorithms to choose from, including the algorithms that are already known to be vulnerable, such as:

RSA with PKCSv1.5 is susceptible to a padding oracle attack.
Or ECDSA can face an invalid-curve attack.

For developers without deep experience in security, it would be hard for them to know which algorithm is the best to use.

So the fact that JWT gives developers too much flexibility to choose the algorithm is like giving them a gun to shoot themselves in the foot.

Trivial Forgery

But it’s not the worst. JSON web token makes token forgery so trivial, that if you are not careful in your implementation or if you choose a poorly implemented library for your project, your system will easily become a vulnerable target.

One bad thing about JWT is that it includes the signing algorithm in the token header.

Because of this, we have seen in the past, an attacker can just set the alg header to none to bypass the signature verification process.

Of course, this issue has been identified and fixed in many libraries, but it’s something you should carefully check when choosing the community-developed library for your project.

Another more dangerous potential attack is to purposely set the algorithm header to a symmetric-key one, such as HS256 while knowing that the server actually uses an asymmetric-key algorithm, such as RSA to sign and verify the token.

Let me explain how!

Basically, the server’s RSA public key is clearly known to the public because it’s a public key.

So the hacker can just create a fake token of the admin user, where he purposely set the algorithm header to HS256, which is a symmetric-key algorithm.

Then, he just signs this token with the server’s public key and uses it to access resources on the server.

Now, keep in mind that, the server normally uses an RSA algorithm, such as RS256 to sign & verify the token, so it will use the RSA public key as the key to verify the token signature.

However, since the token’s algorithm header is saying HS256, the server will verify the signature with this symmetric algorithm HS256 instead of RSA.

And because the same key is used by the hacker to sign the token payload, this signature verification process will be successful, and the request of the hacker will be authorized.

This kind of attack is very simple, but still so powerful and dangerous, and it has actually happened in the past because the developers didn’t check the algorithm header before verify the token signature.

So, in order to prevent this attack, it’s crucial that in your server code, you must check the token’s algorithm header to make sure that it matches with the one your server uses to sign and verify tokens.

OK, so now you know why JSON web token is not a very well-designed standard. It opens the door to many potential threats.

Therefore, many people are trying to stay away from it, and migrate to something more robust.

PASETO - Platform Agnostic Security Token

PASETO, or Platform Agnostic Security Token is one of the most successful designs that is being widely accepted by the community as the best-secured alternative to JWT.

Strong algorithms

It solves all issues of JSON web token by first, provide strong signing algorithms out of the box.

Developers don’t have to choose the algorithm anymore. Instead, they only need to select the version of PASETO they want to use.

Each PASETO version has already been implemented with 1 strong cipher suite. And at any time, there will be only at most 2 latest versions of PASETO are active.

Right now, 2 active PASETO versions are version 1 and version 2.

PASETO version 1

Version 1 is older, and should only be used for legacy systems that cannot use modern cryptography.

Similar to JWT, PASETO also has 2 algorithm categories for 2 main use cases.

For local or internal services, we use a symmetric-key algorithm.

But unlike JWT, which only does base64-encode the payload, and sign the token, PASETO actually encrypts and authenticates all data in the token with a secret key, using a strong Authenticated Encryption with Associated Data (or AEAD) algorithm. If you don’t know what AEAD is, you can watch my video about SSL/TLS.

The AEAD algorithm used in PASETO version 1 is AES256 CTR with HMAC SHA384.

For public cases, where there are external services that need to verify the token, we have to use an asymmetric-key algorithm.

In that case, PASETO uses the same approach as JWT, which means, it doesn’t encrypt the token data, but only base64-encode it, and uses the private key to sign the content with a digital signature.

The chosen asymmetric-key algorithm in PASETO version 1 is RSA PSS with SHA384.

PASETO version 2

In the latest version of PASETO (version 2), 2 more secured and modern algorithms are being used.

For local symmetric-key scenario, it uses XChacha20 with Poly1305 algorithm.

And for a public asymmetric-key scenario, Edward-curve digital signature algorithm with curve 25519 is used.

This choice reminds me of how TLS 1.3 was designed to improve the security of its older version TLS 1.2, and also simplify & reduce the number of TLS cipher suites at the same time.

Non-trivial forgery

Now with the design of PASETO, token forgery is no longer trivial.

Because the algorithm header doesn’t exist anymore, so the attacker cannot set it to none, or force the server to use the algorithm it chose in this header.

Everything in the token is also authenticated with AEAD, so it’s not possible to tamper with it.

Moreover, if you use a local symmetric-key algorithm, the payload is now encrypted, not just encoded, so it’s impossible for hackers to read or change the data stored in the token without knowing the server’s secret key.

Sounds amazing, right?

In the next lecture, I’m gonna show you how to implement both JWT and PASETO using Golang.

You will see PASETO not only makes it safer but also easier and so much simpler to implement, compared to JWT.

For now, let’s take a look at the structure of a PASETO token.

This is a PASETO version 2 token for local usage purposes. There are 4 main parts of the token, separated by a dot.

The first part is PASETO version (with red color), which is version 2.

The second part is the purpose of the token, is it used for local or public scenarios? In this case, it is local, which means using a symmetric-key authenticated encryption algorithm.

The third part (with green color) is the main content or the payload data of the token. Note that it is encrypted, so if we decrypt it using the secret key, we will get 3 smaller parts:

First, the payload body. In this case, we just store a simple message and the expiration time of the token.
Second, the nonce value that is used in both encryption and message authentication process.
And finally the message authentication tag to authenticate the encrypted message and its associated unencrypted data.

In this case, the unencrypted data is the version, the purpose, and the footer of the token (with purple color).

You can store any public information in the footer because it won’t be encrypted like the payload body, but only base64 encoded. So anyone who has the token can decode it to read the footer data.

In this case, it is Paragon Initiative Enterprises, the one who invented PASETO.

Note that the footer is optional, so you can have a PASETO token without a footer. For example, this is another PASETO token for the public usage scenario:

It only has 3 parts, with no footer. The first part is PASETO version, which is version 2.

The second part is its purpose: public in this case, which means an asymmetric-key digital signature algorithm is used to sign the token, and its payload data will not be encrypted, but only base64 encoded.

As you can see here, the green part of the payload is actually the encoded body, which we can easily decode to get this JSON object.

While the blue part of the payload is the signature of the token, created by the digital signature algorithm using the private key.

The server will use its paired public key to verify the authenticity of this signature.

And that’s it for today’s lecture about token-based authentication.

We have learned about the design flaws of JSON Web Token that causes many security issues in the past, and how PASETO was invented to fix all of those problems, and thus, make our developers’ life much easier.

I hope you like this article, and see you in the next one, where we will write codes to create and verify JWT and PASETO tokens in Golang.

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

How to write stronger unit tests with a custom go-mock matcher

TECH SCHOOL — Sun, 21 Feb 2021 12:58:49 +0000

Hello everyone!

In this lecture, we will learn how to write a custom gomock matcher to make our Golang unit tests stronger.

Here's:

Link to the full series playlist on Youtube
And its Github repository

The weak unit test

In the last lecture, we have learned how to securely store users’ password using bcrypt. We have also implemented the API to create a new user for our simple bank.



func (server *Server) createUser(ctx *gin.Context) {
    var req createUserRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    hashedPassword, err := util.HashPassword(req.Password)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    arg := db.CreateUserParams{
        Username:       req.Username,
        HashedPassword: hashedPassword,
        FullName:       req.FullName,
        Email:          req.Email,
    }

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    rsp := createUserResponse{
        Username:          user.Username,
        FullName:          user.FullName,
        Email:             user.Email,
        PasswordChangedAt: user.PasswordChangedAt,
        CreatedAt:         user.CreatedAt,
    }
    ctx.JSON(http.StatusOK, rsp)
}

Although I didn’t show you how to write unit tests for this API, since it would be very similar to what we have done in lecture 13 of the course. With the help of go-mock, any API unit tests can be written with ease.

However, if you have tried to write unit tests for the create user API yourself, you might find it a little bit tricky, due to the fact that the input password param is hashed before storing in the database.

To understand why, let’s look at the simple version of the tests that I’ve already written here.



func TestCreateUserAPI(t *testing.T) {
    user, password := randomUser(t)

    testCases := []struct {
        name          string
        body          gin.H
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(recoder *httptest.ResponseRecorder)
    }{
        {
            name: "OK",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(1).
                    Return(user, nil)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchUser(t, recorder.Body, user)
            },
        },
        {
            name: "InternalError",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(1).
                    Return(db.User{}, sql.ErrConnDone)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusInternalServerError, recorder.Code)
            },
        },
        {
            name: "DuplicateUsername",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(1).
                    Return(db.User{}, &pq.Error{Code: "23505"})
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusForbidden, recorder.Code)
            },
        },
        {
            name: "InvalidUsername",
            body: gin.H{
                "username":  "invalid-user#1",
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(0)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusBadRequest, recorder.Code)
            },
        },
        {
            name: "InvalidEmail",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     "invalid-email",
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(0)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusBadRequest, recorder.Code)
            },
        },
        {
            name: "TooShortPassword",
            body: gin.H{
                "username":  user.Username,
                "password":  "123",
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(0)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusBadRequest, recorder.Code)
            },
        },
    }

    ...
}

Basically, we first create a random user to be created.

Then we declare a table of test cases, where we can define the input request body, and 2 functions to build store stubs and check the response of the API.

There are several different cases we can test, such as:

The successful case
Internal server error case
Duplicate username case
Invalid username, email, or password case

We iterate through all of these cases, and run a separate sub-test for each of them.



func TestCreateUserAPI(t *testing.T) {
    ...

    for i := range testCases {
        tc := testCases[i]

        t.Run(tc.name, func(t *testing.T) {
            ctrl := gomock.NewController(t)
            defer ctrl.Finish()

            store := mockdb.NewMockStore(ctrl)
            tc.buildStubs(store)

            server := NewServer(store)
            recorder := httptest.NewRecorder()

            // Marshal body data to JSON
            data, err := json.Marshal(tc.body)
            require.NoError(t, err)

            url := "/users"
            request, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(data))
            require.NoError(t, err)

            server.router.ServeHTTP(recorder, request)
            tc.checkResponse(recorder)
        })
    }
}

In each sub-test, we create a new gomock controller, and use it to build a new mock DB store.

Then we call the buildStubs() function of the test case to set up the stubs for that store.

After that, we create a new server using the mock store, and create a new HTTP response recorder to record the result of the API call.

Next we marshal the input request body to JSON, and make a new POST request to the create-user API endpoint with that JSON data.

We call server.router.ServeHTTP() function with the recorder and request object. And finally just call tc.checkResponse() function to check the result.

It’s pretty simple, just like what we’ve learned in Lecture 13.

I highly recommend you to read it first to make sure you fully understand the code before continue.

Now, for today’s lecture, we only need to focus on 1 case: the successful one.



func TestCreateUserAPI(t *testing.T) {
    user, password := randomUser(t)

    testCases := []struct {
        name          string
        body          gin.H
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(recoder *httptest.ResponseRecorder)
    }{
        {
            name: "OK",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Any()).
                    Times(1).
                    Return(user, nil)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchUser(t, recorder.Body, user)
            },
        },
        ...
    }

    ...
}

As you can see here, the request body’s parameters are all valid.

In the build stubs function, we expect the CreateUser() function of the store to be called with 2 parameters. In this simple version, we’re using the gomock.Any() matcher for both of them.

Note that the first argument of the store.CreateUser() function is a context, which we don’t care about its value, so it makes sense to use any matcher.

However, using that same matcher for the second argument will weaken the test, because it won’t be able to detect if the createUserParams object passed in the CreateUser() function is correct or not. I’m gonna show you how in a moment. For now, let’s just keep this gomock.Any() matcher.

And this CreateUser function is expected to be called exactly once, and it will return the user object with no errors, since this is the happy case.

For the check response function, we just check the HTTP status code to be 200 OK, and make sure that the response body matches the created user object.

That’s it! Let’s run the test.

All passed. So we’re good, right?

Well, not really!

As I said before, this gomock.Any() matcher will make the test weaker.

How?

Let’s see what will happen if in this createUser() handler, I set the argument variable to an empty CreateUserParam{} object.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    arg := db.CreateUserParams{}

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

This will discard all the input parameters of the request, and will create a completely empty user in the database. So it should not be allowed, and we expect the test to fail, right?

However, if we run the test again:

It still passed!

This is very bad, because the implementation of the handler is completely wrong, but the test could not detect it!

Another case that this test would not be able to detect is like this:

Let’s remove this set empty argument statement, but I will ignore the user’s input password, and just hash a constant value, such as "xyz" here.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    hashedPassword, err := util.HashPassword("xyz")
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    arg := db.CreateUserParams{
        Username:       req.Username,
        HashedPassword: hashedPassword,
        FullName:       req.FullName,
        Email:          req.Email,
    }

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

Then go back to the test, and run it again.

As you can see, it still passed!

This is unacceptable! The test we wrote is too weak! We need to fix it!

Try using gomock.Eq

One thing that might come to your mind is: what if instead of using gomock.Any() matcher, we use something else, such as the gomock.Eq() matcher? Let’s try it!

First I will declare a new arg variable of type db.CreateUserParams, where username is user.Username.

For the HashedPassword field, we need to hash the input naked password, so let’s go up to the top of the test. Here, after generating the random user object, we call util.HashPassword, and pass in the generated password value.

This function will return a hashedPassword value and an error, so we have to make sure error is nil using require.NoError().



func TestCreateUserAPI(t *testing.T) {
    user, password := randomUser(t)

    hashedPassword, err := util.HashPassword(password)
    require.NoError(t, err)

    testCases := []struct {
        name          string
        body          gin.H
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(recoder *httptest.ResponseRecorder)
    }{
        {
            name: "OK",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                arg := db.CreateUserParams{
                    Username: user.Username,
                    HashedPassword: hashedPassword,
                    FullName: user.FullName,
                    Email: user.Email,
                }
                store.EXPECT().
                    CreateUser(gomock.Any(), gomock.Eq(arg)).
                    Times(1).
                    Return(user, nil)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchUser(t, recorder.Body, user)
            },
        },
        ...
    }

    ...
}

Next, the FullName should be user.FullName, and finally Email should be user.Email.

With this object, we can now replace gomock.Any() matcher with gomock.Eq(arg)

OK now, let’s try to test the case where all input parameters are discard.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    arg := db.CreateUserParams{}

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

Since we have used a stronger Eq() matcher here, the test should fail, right? Let’s run it to confirm!

Yes, that’s right! The test failed as expected.

The logs tell us it failed because of the missing calls, which is true, because although the CreateUser function was called, it was not called with the correct input argument as we wanted.

So is that it? Did we fixed the issue?

Unfortunately no! Let’s see what will happen if I remove this line arg := db.CreateUserParams{}



func (server *Server) createUser(ctx *gin.Context) {
    ...

    arg := db.CreateUserParams{
        Username:       req.Username,
        HashedPassword: hashedPassword,
        FullName:       req.FullName,
        Email:          req.Email,
    }

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

Now the create user handler function is correctly taken into account all input parameters, so we expect the test to pass, right? Let’s run it to confirm!

Sadly, the test didn’t pass. It still failed due to missing call.

And the real reason is that the CreateUser() function is called with an input argument that doesn’t match with the one we expect.

In the log, we can see clearly what value the mock store got, compared to what it wants to receive.

It looks like the Username, FullName, and Email are all matched. Only the HashedPassword values are different. Do you know why?

Well, if you still remember what we learned in the last lecture about bcrypt, it uses a random salt when hashing the password to prevent rainbow table attack.

So even if we pass the same password value into the hash function, it will always produce a new hashed value every time.

Because of this, the hashed value that we created in the test and the one in the create user handler will always be different. So we cannot simply use the built-in gomock.Eq() matcher to compare the argument.

The only way to fix this properly is to implement a new custom matcher on our own in this case. Although it sounds a bit annoying, it’s actually very easy to implement. And I think it would be useful for you if you ever encounter some special cases like this in your real project.

OK, let’s learn how to do it!

Implement a custom gomock matcher

First I will remove the hashedPassword because it is not needed in the custom matcher that we’re going to implement.

We will have to replace the gomock.Eq matcher with our own matcher. So let’s open its implementation.



func Eq(x interface{}) Matcher { return eqMatcher{x} }

type Matcher interface {
    // Matches returns whether x is a match.
    Matches(x interface{}) bool

    // String describes what the matcher matches.
    String() string
}

It’s simply a function that takes the expected argument x as input and returns a Matcher interface. In this specific case, it returns an implementation of the matcher that matches on equality: eqMatcher.

For our custom matcher, we will have to write a similar implementation of the Matcher interface, which has only 2 methods:

The first one is Matches(), which should return whether the input x is a match or not.
And the second one is String(), which just describes what the matcher matches for logging purpose.

There are several built-in implementations of the Matcher interface. For example, this one is anyMatcher, which will always return true regardless of the input argument.



type anyMatcher struct{}

func (anyMatcher) Matches(interface{}) bool {
    return true
}

func (anyMatcher) String() string {
    return "is anything"
}

Then this is the equal matcher that we’re using:



type eqMatcher struct {
    x interface{}
}

func (e eqMatcher) Matches(x interface{}) bool {
    return reflect.DeepEqual(e.x, x)
}

func (e eqMatcher) String() string {
    return fmt.Sprintf("is equal to %v", e.x)
}

It uses reflect.DeepEqual to compare the actual input argument with the expected one.

The custom matcher that we’re going to implement will be very similar to this one, so I’m gonna copy all of these functions, and paste them to the top of the user_test.go file.

Then let’s change the name of this struct to eqCreateUserParamsMatcher. In order to compare the input arguments correctly, we will need to store 2 fields this struct:

First the arg field of type db.CreateUserParams
And second, the password field to store the naked password value.



type eqCreateUserParamsMatcher struct {
    arg      db.CreateUserParams
    password string
}

OK, now let’s implement the Matches() function. Since the input parameter x is an interface, we should convert it to db.CreateUserParams object.

If the conversion is not OK, then we just return false. Otherwise, we will check if the hashed password matches with the expected password or not by calling util.CheckPassword() function with e.password and arg.HashedPassword.



func (e eqCreateUserParamsMatcher) Matches(x interface{}) bool {
    arg, ok := x.(db.CreateUserParams)
    if !ok {
        return false
    }

    err := util.CheckPassword(e.password, arg.HashedPassword)
    if err != nil {
        return false
    }

    e.arg.HashedPassword = arg.HashedPassword
    return reflect.DeepEqual(e.arg, arg)
}

If this function returns an error, then we return false. Else, we will set the hashedPassword field of the expected argument e.arg to the same value with the input arg.HashPassword.

And we use reflect.DeepEqual to compare the expected argument e.arg with the input argument arg.

That’s all! Pretty simple, right?

OK, now let’s update this message of the String function to include the expected argument and naked password values.



func (e eqCreateUserParamsMatcher) String() string {
    return fmt.Sprintf("matches arg %v and password %v", e.arg, e.password)
}

And our custom matcher is done.

Next we will add a function to return an instance of this matcher, just like the Eq() function that returns an EqMatcher instance.

I’m gonna change this function’s name to EqCreateUserParams(), and it will have 2 input arguments: a db.CreateUserParams object, and a naked password string.

This function will return a Matcher interface, which in our case is the eqCreateUserParamsMatcher object with the input argument and password.



func EqCreateUserParams(arg db.CreateUserParams, password string) gomock.Matcher {
    return eqCreateUserParamsMatcher{arg, password}
}

Alright, so now we have everything we need for the new custom matcher. Let’s use it in the unit test to see how it goes.

I’m gonna change the gomock.Eq() to EqCreateUserParams(), then pass in the CreateUserParams argument arg and naked password.



func TestCreateUserAPI(t *testing.T) {
    user, password := randomUser(t)

    testCases := []struct {
        name          string
        body          gin.H
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(recoder *httptest.ResponseRecorder)
    }{
        {
            name: "OK",
            body: gin.H{
                "username":  user.Username,
                "password":  password,
                "full_name": user.FullName,
                "email":     user.Email,
            },
            buildStubs: func(store *mockdb.MockStore) {
                arg := db.CreateUserParams{
                    Username: user.Username,
                    FullName: user.FullName,
                    Email:    user.Email,
                }
                store.EXPECT().
                    CreateUser(gomock.Any(), EqCreateUserParams(arg, password)).
                    Times(1).
                    Return(user, nil)
            },
            checkResponse: func(recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchUser(t, recorder.Body, user)
            },
        },
        ...
    }

    ...
}

Just like that, and we’re done. Let’s rerun the test!

It passed. Excellent!

Now let’s try the case where we set this argument to an empty db.CreateUserParams{} object.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    arg := db.CreateUserParams{}

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

We expect the test to fail.

And it does fail, since the input argument doesn’t match the expected one.

OK, how about the case where we ignore the input password parameter, and just hash this constant password value: xyz



func (server *Server) createUser(ctx *gin.Context) {
    ...

    hashedPassword, err := util.HashPassword("xyz")
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ...
}

Let’s run the test!

It failed, just like what we expected.

So now our unit test has become much stronger than it was before. Thanks to the new custom matcher that we’ve just implemented.

And that’s it for today’s lecture. I hope you find it interesting and useful.

Thanks for reading, and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

How to securely store passwords?

TECH SCHOOL — Sat, 16 Jan 2021 22:11:59 +0000

Hello everyone, welcome back to the backend master class!

In this lecture, we’re gonna learn how to securely store users’ password in the database.

Here's:

Link to the full series playlist on Youtube
And its Github repository

How to store password

As you already know, we should never ever store naked passwords! So the idea is to hash it first, and only store that hash value.

Basically, the password will be hashed using brypt hashing function to produce a hash value.

Besides the input password, bcrypt requires a cost parameter, which will decide the number of key expansion rounds or iterations of the algorithm.

Bcrypt also generates a random salt to be used in those iterations, which will help protect against the rainbow table attack. Because of this random salt, the algorithm will give you a completely different output hash value even if the same input password is provided.

The cost and salt will also be added to the hash to produce the final hash string, which looks something like this:

In this hash string, there are 4 components:

The first part is the hash algorithm identifier. 2A is the identifier of the bcrypt algorithm.
The second part is the cost. In this case, the cost is 10, which means there will be 2^10 = 1024 rounds of key expansion.
The third part is the salt of length 16 bytes, or 128 bits. It is encoded using base64 format, which will generate a string of 22 characters.
Finally, the last part is the 24 bytes hash value, encoded as 31 characters.

All of these 4 parts are concatenated together into a single hash string, and it is the string that we will store in the database.

So that’s the process of hashing users’ password!

But when users login, how can we verify that the password that they entered is correct or not?

Well, first we have to find the hashed_password stored in the DB by username.

Then we use the cost and salt of that hashed_password as the arguments to hash the naked_password users just entered with bcrypt. The output of this will be another hash value.

Then all we have to do is to compare the 2 hash values. If they’re the same, then the password is correct.

Alright, now let’s see how to implement these logics in Golang.

Implement functions to hash and compare passwords

In the previous lecture, we have generated the code to create a new user in the database. And hashed_password is one of the input parameters of the CreateUser() function.



type CreateUserParams struct {
    Username       string `json:"username"`
    HashedPassword string `json:"hashed_password"`
    FullName       string `json:"full_name"`
    Email          string `json:"email"`
}

func (q *Queries) CreateUser(ctx context.Context, arg CreateUserParams) (User, error) {
    row := q.db.QueryRowContext(ctx, createUser,
        arg.Username,
        arg.HashedPassword,
        arg.FullName,
        arg.Email,
    )
    var i User
    err := row.Scan(
        &i.Username,
        &i.HashedPassword,
        &i.FullName,
        &i.Email,
        &i.PasswordChangedAt,
        &i.CreatedAt,
    )
    return i, err
}

Also, in this createRandomUser() function of the unit test in db/sqlc/user_test.go, we’re using a simple "secret" string for the hash_password field, which doesn’t reflect the real correct values this field should hold.



func createRandomUser(t *testing.T) User {
    arg := CreateUserParams{
        Username:       util.RandomOwner(),
        HashedPassword: "secret",
        FullName:       util.RandomOwner(),
        Email:          util.RandomEmail(),
    }

    ...
}

So today we’re gonna update it to use a real hash string.

Hash password function

First, let’s create a new file password.go inside the util package. In this file, I’m gonna define a new function: HashPassword().

It will take a password string as input, and will return a string or an error. This function will compute the bcrypt hash string of the input password.



// HashPassword returns the bcrypt hash of the password
func HashPassword(password string) (string, error) {
    hashedPassword, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
    if err != nil {
        return "", fmt.Errorf("failed to hash password: %w", err)
    }
    return string(hashedPassword), nil
}

In this function, we call bcrypt.GenerateFromPassword(). It requires 2 input parameters: the password of type []byte slice, and a cost of type int.

So we have to convert the input password from string to []byte slice.

For cost, I use the bcrypt.DefaultCost value, which is 10.

The output of this function will be the hashedPassword and an error. If the error is not nil, then we just return an empty hashed string, and wrap the error with a message saying: "failed to hash password".

Otherwise, we convert the hashedPassword from []byte slice to string, and return it with a nil error.

Compare passwords function

Next, we will write another function to check if a password is correct or not: CheckPassword().

This function will take 2 input arguments: a password to check, and the hashedPassword to compare. It will return an error as output.

Basically, this function will check if the input password is correct when comparing to the provided hashedPassword or not.

As the standard bcrypt package has already implemented this feature, all we have to do is to call bcrypt.CompareHashAndPassword() function, and pass in the hashedPassword and naked password, after converting them from string to []byte slices.



// CheckPassword checks if the provided password is correct or not
func CheckPassword(password string, hashedPassword string) error {
    return bcrypt.CompareHashAndPassword([]byte(hashedPassword), []byte(password))
}

And that’s it. We’re done!

Write unit test for HashPassword and CheckPassword functions

Now let’s write some unit tests to make sure these 2 functions work as expected.

I’m gonna create a new file password_test.go inside the util package. Then let’s define function TestPassword() with a testing.T object as input.

First I will generate a new password as a random string of 6 characters. Then we get the hashedPassword by calling HashPassword() function with the generated password.

We require no errors to be returned, and the hashedPassword string should be not empty.



func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword)

    err = CheckPassword(password, hashedPassword1)
    require.NoError(t, err)
}

Next we call CheckPassword() function with the password and hashedPassword parameters.

As this is the same password we used to create the hashedPassword, this function should return no errors, which means correct password.

Let’s also test the case where an incorrect password is provided!

I will generate a new random wrongPassword string, and call CheckPassword() again with this wrongPassword argument. This time, we expect an error to be returned, since the provided password is incorrect.



func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword)

    wrongPassword := RandomString(6)
    err = CheckPassword(wrongPassword, hashedPassword)
    require.EqualError(t, err, bcrypt.ErrMismatchedHashAndPassword.Error())
}

To be exact, we use require.EqualError() to compare the output error. It must be equal to the bcrypt.ErrMismatchedHashAndPassword error.

OK, the test is now completed. Let’s run it!

It passed! Awesome!

Update the existing code to use HashPassword function

So the HashPassword() function is working properly. Let’s go back to the user_test.go file and use it in the createRandomUser() function.

Here I’m gonna create a new hashedPassword value by calling util.HashPassword() function with a random string of 6 characters.

We require no errors, then change the "secret" constant to hashedPassword instead:



func createRandomUser(t *testing.T) User {
    hashedPassword, err := util.HashPassword(util.RandomString(6))
    require.NoError(t, err)

    arg := CreateUserParams{
        Username:       util.RandomOwner(),
        HashedPassword: hashedPassword,
        FullName:       util.RandomOwner(),
        Email:          util.RandomEmail(),
    }

    ...
}

Alright, let’s run the whole db package test!

All passed!

Now if we open the database in Table Plus and check the users table, we can see that the hashed_password column is now containing the correct bcrypt hashed string.

It looks just like the example that I shown you in the beginning of this video.

Make sure all hashed passwords are different

One thing we want to make sure of is: if the same password is hashed twice, 2 different hash values should be produced.

So let’s go back to the TestPassword() function. I’m gonna change the hashPassword variable’s name to hashedPassword1.

Then let’s duplicate the hash password code block, and change the variable’s name to hashedPassword2.



func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword1, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword1)

    err = CheckPassword(password, hashedPassword1)
    require.NoError(t, err)

    wrongPassword := RandomString(6)
    err = CheckPassword(wrongPassword, hashedPassword1)
    require.EqualError(t, err, bcrypt.ErrMismatchedHashAndPassword.Error())

    hashedPassword2, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword2)
    require.NotEqual(t, hashedPassword1, hashedPassword2)
}

What we expect to see is: the value of hashedPassword2 should be different from the value of hashedPassword1. So here I use require.NotEqual() to check this condition.

OK, let’s rerun the test.

It passed! Excellent!

To really understand why it passed, we have to open the implementation of the bcrypt.GenerateFromPassword() function.



func GenerateFromPassword(password []byte, cost int) ([]byte, error) {
    p, err := newFromPassword(password, cost)
    if err != nil {
        return nil, err
    }
    return p.Hash(), nil
}

func newFromPassword(password []byte, cost int) (*hashed, error) {
    if cost < MinCost {
        cost = DefaultCost
    }
    p := new(hashed)
    p.major = majorVersion
    p.minor = minorVersion

    err := checkCost(cost)
    if err != nil {
        return nil, err
    }
    p.cost = cost

    unencodedSalt := make([]byte, maxSaltSize)
    _, err = io.ReadFull(rand.Reader, unencodedSalt)
    if err != nil {
        return nil, err
    }

    p.salt = base64Encode(unencodedSalt)
    hash, err := bcrypt(password, p.cost, p.salt)
    if err != nil {
        return nil, err
    }
    p.hash = hash
    return p, err
}

As you can see here, in the newFromPassword() function, a random salt value is generated, and it is used in the bcrypt() function to generate the hash.

So now you know, because of this random salt, the generated hash value will be different everytime.

Implement the create user API

Next step, I’m gonna use the HashPassword() function that we’ve written to implement the create user API for our simple bank.

Let’s create a new file user.go inside the api package.

This API will be very much alike the create account API that we’ve implemented before, so I’m just gonna copy it from the api/account.go file.

Then let’s change this struct to createUserRequest.

The first parameter is username. It is a required field.

And let’s say we don’t allow it to contain any kind of special characters, so here I’m gonna use the alphanum tag, which is already provided by the validator package. It basically means that this field should contain ASCII alphanumeric characters only.

The second field is password. It is also required. And normally we don’t want the password to be too short because it would be very easy to hack. So here let’s use the min tag to say that the length of the password should be at least 6 characters.



type createUserRequest struct {
    Username string `json:"username" binding:"required,alphanum"`
    Password string `json:"password" binding:"required,min=6"`
    FullName string `json:"full_name" binding:"required"`
    Email    string `json:"email" binding:"required,email"`
}

The third field is full_name of the user. There’s no specific requirements for this field, except that it is required.

Then, the last field is email, which is very important because it would be the main communication channel between the users and our system. We can use the email tag provided by validator package to make sure that the value of this field is a correct email address.

There are many other useful built-in tags that were already implemented by the validator package, you can check them out in its documentation or github page.

Now let’s go back to the code to complete this createUser() function.



func (server *Server) createUser(ctx *gin.Context) {
    var req createUserRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    hashedPassword, err := util.HashPassword(req.Password)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    arg := db.CreateUserParams{
        Username:       req.Username,
        HashedPassword: hashedPassword,
        FullName:       req.FullName,
        Email:          req.Email,
    }

    ...   
}

Here we use the ctx.ShouldBindJSON() function to bind the input parameters from the context into the createUserRequest object.

If any of the parameters are invalid, we just return 400 Bad Request status to the client. Otherwise, we will use them build the db.CreateUserParams object.

There are 4 fields that need to be set: Username, HashedPassword, Fullname, and Email.

So first, we compute the hashedPassword by calling util.HashPassword() function and pass in the input request.Password value.

If this function returns a not nil error, then we just return a status 500 Internal Server Error to the client.

Else, we will build the CreateUserParams object, where Username is request.Username, HashedPassword is the computed hashedPassword, FullName is request.FullName, and Email is request.Email.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, user)
}

Then we call server.store.CreateUser() with this input argument. It will return the created user object or an error.

Just like in the create account API, if error is not nil, then there are some possible scenarios. Keep in mind that, in the users table, we have 2 unique constraints:

One is for the primary key username,
And the other is for the email column.

We don’t have a foreign key in this table, so here we only need to keep the unique_violation code name to return status 403 Forbidden in case an user with the same username or email already exists.

Finally, if no errors occur, we just return status 200 OK with the created user to the client.

OK, so now the createUser API handler is completed. The last step we must do is to register a route for it in the api/server.go file.

Here, in this NewServer() function, I’m gonna add a new route with method POST. Its path should be /users, and its handler function is server.createUser



// NewServer creates a new HTTP server and set up routing.
func NewServer(store db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
        v.RegisterValidation("currency", validCurrency)
    }

    router.POST("/users", server.createUser)

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccounts)

    router.POST("/transfers", server.createTransfer)

    server.router = router
    return server
}

And that’s it! We’re done!

Test the create user API

Let’s open the terminal and run make server to start the server.

I’m gonna use Postman to test the new API.

Let’s select method POST and fill in the URL: http://localhost:8080/users

For the request body, let’s choose raw, and select JSON format. I'm gonna use this JSON data:



{
    "username": "quang1",
    "full_name": "Quang Pham",
    "email": "quang@email.com",
    "password": "secret"
}

OK, let’s send this request!

It’s successful! We’ve got the created user object here with all correct field values.

Let’s open the database to find this user.

Here it is! So our API is working well in the happy case.

Now let’s see what happens if I send this same request the second time.

We’ve got a 403 Forbidden status code. And the reason is that the unique constraint for username is violated.

We’re trying to create another user with the same username, So clearly it should not be allowed!

Now let’s try changing the username to quang2, but keep the email value the same, and send the request again.

We still got 403 Forbidden. But this time, the error is because the email unique constraint is violated. Exactly what we expected!

If I change the email to quang2@email.com, then the request will be successful, since this email doesn’t belong to any other users.

OK, now let’s try an invalid username, such as quang#2:

This time, the status code is 400 Bad Request. And the reason is: the field validation for username failed on the alphanum tag. There’s a special character # in the username, which is not alphanumeric.

Next, let’s try an invalid email. I’m gonna change the username to quang3, and email to quang3email.com, without the @ character.

We’ve got 400 Bad Request status again. And the error is: field validation for email failed on the email tag, which is exactly what we want.

OK now let’s fix the email address, and change the password to a very short value, such as "123". Then send the request one more time.

This time, we’ve got an error because the password field validation failed on the min tag. It doesn’t satisfy the minimum length constraint of 6 characters.

API should not expose hashed password

Before we finish, there’s one more thing I want to tell you. Let’s fix the password value and send the request again.

Now it’s successful. But you can notice that the hashed_password value is also returned, which doesn’t seem right, because the client will never need to use this value for anything.

And it might raise some security concerns, as this piece of sensitive information is being transmitted in the public.

It would be better to remove this field from the response body.

To do that, I’m gonna declare a new createUserResponse struct in the api/user.go file. It will contain almost all fields of the db.User struct, except for the HashedPassword field that should be removed.



type createUserResponse struct {
    Username          string    `json:"username"`
    FullName          string    `json:"full_name"`
    Email             string    `json:"email"`
    PasswordChangedAt time.Time `json:"password_changed_at"`
    CreatedAt         time.Time `json:"created_at"`
}

Then here, at the end of the createUser() handler function, we build a new createUserResponse object, where Username is user.Username, FullName is user.FullName, Email is user.Email, PasswordChangedAt is user.PasswordChangedAt, and CreatedAt is user.CreatedAt.



func (server *Server) createUser(ctx *gin.Context) {
    ...

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    rsp := createUserResponse{
        Username:          user.Username,
        FullName:          user.FullName,
        Email:             user.Email,
        PasswordChangedAt: user.PasswordChangedAt,
        CreatedAt:         user.CreatedAt,
    }
    ctx.JSON(http.StatusOK, rsp)
}

Finally, we return the response object instead of user. And we’re done!

Let’s restart the server. Then go back to Postman, update the username and email to new values, and send the request.

It’s successful. And now there’s no hashed_password field in the response body anymore. Perfect!

So that brings us to the end of this lecture. I hope you have learned something useful.

Thank you for reading, and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

How to handle DB errors in Golang correctly

TECH SCHOOL — Fri, 01 Jan 2021 10:12:01 +0000

Hi guys, welcome back!

In the last lecture, we’ve added a new users table to the database schema. Today, let’s update our golang code to work with this table.

And while doing so, we’re also gonna learn how to correctly handle some specific errors returned by Postgres.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Alright, let’s start!

Generate code to create and get user

First I’m gonna create a new file user.sql inside the db/query folder. In this file, we will write 2 SQL queries to create and get users.

They should be similar to the ones we used to create and get accounts, so I’m gonna copy these 2 queries from the account.sql file and paste them to the user.sql file.



-- name: CreateAccount :one
INSERT INTO accounts (
  owner,
  balance,
  currency
) VALUES (
  $1, $2, $3
) RETURNING *;

-- name: GetAccount :one
SELECT * FROM accounts
WHERE id = $1 LIMIT 1;

Then let’s change the function name to CreateUser, the table name to users, and the field names are: username, hashed_password, full_name, and email.

We don’t have to specified the password_changed_at and created_at fields because they will be automatically filled with default value by Postgres.

There are 4 input fields, so we have to add 1 more parameter to the value list.



-- name: CreateUser :one
INSERT INTO users (
  username,
  hashed_password,
  full_name,
  email
) VALUES (
  $1, $2, $3, $4
) RETURNING *;

Next, the GetAccount function should be changed to GetUser, and the query is SELECT FROM users.

Note that we don’t have an ID column in the users table. Its primary key is username, so here we should get user by username instead.



-- name: GetUser :one
SELECT * FROM users
WHERE username = $1 LIMIT 1;

Alright, now the queries are completed, let’s open the terminal and run this command to generate golang codes for them.



❯ make sqlc
sqlc generate

Now back to visual studio code. In the db/sqlc/models.go file, a new User struct has been added:



type User struct {
    Username          string    `json:"username"`
    HashedPassword    string    `json:"hashed_password"`
    FullName          string    `json:"full_name"`
    Email             string    `json:"email"`
    PasswordChangedAt time.Time `json:"password_changed_at"`
    CreatedAt         time.Time `json:"created_at"`
}

And there’s a new file db/sqlc/user.sql.go that contains 2 functions to create and get user from the database:



// Code generated by sqlc. DO NOT EDIT.
// source: user.sql

package db

import (
    "context"
)

const createUser = `-- name: CreateUser :one
INSERT INTO users (
  username,
  hashed_password,
  full_name,
  email
) VALUES (
  $1, $2, $3, $4
) RETURNING username, hashed_password, full_name, email, password_changed_at, created_at
`

type CreateUserParams struct {
    Username       string `json:"username"`
    HashedPassword string `json:"hashed_password"`
    FullName       string `json:"full_name"`
    Email          string `json:"email"`
}

func (q *Queries) CreateUser(ctx context.Context, arg CreateUserParams) (User, error) {
    row := q.db.QueryRowContext(ctx, createUser,
        arg.Username,
        arg.HashedPassword,
        arg.FullName,
        arg.Email,
    )
    var i User
    err := row.Scan(
        &i.Username,
        &i.HashedPassword,
        &i.FullName,
        &i.Email,
        &i.PasswordChangedAt,
        &i.CreatedAt,
    )
    return i, err
}

const getUser = `-- name: GetUser :one
SELECT username, hashed_password, full_name, email, password_changed_at, created_at FROM users
WHERE username = $1 LIMIT 1
`

func (q *Queries) GetUser(ctx context.Context, username string) (User, error) {
    row := q.db.QueryRowContext(ctx, getUser, username)
    var i User
    err := row.Scan(
        &i.Username,
        &i.HashedPassword,
        &i.FullName,
        &i.Email,
        &i.PasswordChangedAt,
        &i.CreatedAt,
    )
    return i, err
}

Next, we will write tests for these 2 functions to make sure they’re working as expected.

Write tests for the generated functions

We’ve already learned how to do that in lecture 5 of the course.

So I’m gonna create a new file user_test.go file in this db/sqlc folder. Then I will copy the tests that we wrote for the create and get account function and paste them to this file.

Then let’s change the function name to createRandomUser. The argument variable will be of type CreateUserParams.

The first field is username, which we can leave as a random owner.

The second field is hashed_password. Normally we will have to generate a random password and hash it using bcrypt, but that would be done in another lecture. For now, I’m just gonna use a simple text value "secret" here.



func createRandomUser(t *testing.T) User {
    arg := CreateUserParams{
        Username:       util.RandomOwner(),
        HashedPassword: "secret",
        FullName:       util.RandomOwner(),
        Email:          util.RandomEmail(),
    }

    ...
}

The next field is full_name. We can use the same util.RandomOwner() function for it.

And the last field is email. We will need to add a new RandomEmail() function to the util package. So let’s open the util/random.go file and implement it.



// RandomEmail generates a random email
func RandomEmail() string {
    return fmt.Sprintf("%s@email.com", RandomString(6))
}

This RandomEmail function will return a string, which should be a randomly generated email. I’m gonna keep it simple here by using fmt.Sprintf to generate an email of the form: some random string at email.com.

Alright, now go back to the test. We have to change this function call to testQueries.CreateUser(), and the output result should be a user object.



func createRandomUser(t *testing.T) User {
    ...

    user, err := testQueries.CreateUser(context.Background(), arg)
    require.NoError(t, err)
    require.NotEmpty(t, user)

    require.Equal(t, arg.Username, user.Username)
    require.Equal(t, arg.HashedPassword, user.HashedPassword)
    require.Equal(t, arg.FullName, user.FullName)
    require.Equal(t, arg.Email, user.Email)
    require.NotZero(t, user.CreatedAt)
    require.True(t, user.PasswordChangedAt.IsZero())

    return user
}

We require this user to be not empty. Then we compare each field of the output user with those of the input argument:

arg.Username should be equal to user.Username
arg.HashedPassword should be equal to user.HashedPassword
arg.Fullname should be equal to user.Fullname
arg.Email should be equal to user.Email

Then the user.CreatedAt field should be not zero, since we expect the database to fill it with the current timestamp.

The last field we have to check is user.PasswordChangedAt. When the user is first created, we expect this field to be filled with a default value of a zero timestamp. The IsZero() function is used for checking this condition.

Then at the end, we should return the created user to the caller. OK, now let’s use this function in the tests!

First, for the TestCreateUser, we simply call createRandomUser with the input testing.T object.



func TestCreateUser(t *testing.T) {
    createRandomUser(t)
}

The next test is GetUser. We call the createRandomUser() function to create a random user1. Then we call testQueries.GetUser to fetch the user with user1.Username from the database.



func TestGetUser(t *testing.T) {
    user1 := createRandomUser(t)
    user2, err := testQueries.GetUser(context.Background(), user1.Username)
    require.NoError(t, err)
    require.NotEmpty(t, user2)

    require.Equal(t, user1.Username, user2.Username)
    require.Equal(t, user1.HashedPassword, user2.HashedPassword)
    require.Equal(t, user1.FullName, user2.FullName)
    require.Equal(t, user1.Email, user2.Email)
    require.WithinDuration(t, user1.PasswordChangedAt, user2.PasswordChangedAt, time.Second)
    require.WithinDuration(t, user1.CreatedAt, user2.CreatedAt, time.Second)
}

The output user2 of this query should match the input user1. So we compare each field of them to make sure they’re equal: username, hashed_password, full_name, and email.

For a timestamp field like created_at and password_changed_at, I often use require.WithinDuration to compare the values because sometimes there might be a very small difference.

Alright, the tests are completed. Let’s run them!

First the TestCreateUser.

It passed!

Then the TestGetUser.

Also passed!

Now if we open the database using Table Plus, we can see there are 2 records in the users table.

OK, so the 2 functions generated by sqlc worked correctly.

Let’s try to run the whole package test!

This time, there are many tests of the accounts CRUD function failed. And the reason is because of the foreign key constraint violation.

This is expected because at the time these tests were written, the foreign key constraint for the owner field didn’t exist yet.

Fix the failed tests

As you can see in the db/sqlc/account_test.go file, we’re just generating a random owner, and it doesn’t link to any existed users:



func createRandomAccount(t *testing.T) Account {
    arg := CreateAccountParams{
        Owner:    util.RandomOwner(),
        Balance:  util.RandomMoney(),
        Currency: util.RandomCurrency(),
    }

    account, err := testQueries.CreateAccount(context.Background(), arg)
    require.NoError(t, err)
    require.NotEmpty(t, account)

    require.Equal(t, arg.Owner, account.Owner)
    require.Equal(t, arg.Balance, account.Balance)
    require.Equal(t, arg.Currency, account.Currency)

    require.NotZero(t, account.ID)
    require.NotZero(t, account.CreatedAt)

    return account
}

func TestCreateAccount(t *testing.T) {
    createRandomAccount(t)
}

In order to fix this, we have to create a user in the database first. Then, instead of a random owner, we will use the created user’s username as the account owner:



func createRandomAccount(t *testing.T) Account {
    user := createRandomUser(t)

    arg := CreateAccountParams{
        Owner:    user.Username,
        Balance:  util.RandomMoney(),
        Currency: util.RandomCurrency(),
    }

    ...
}

OK, now it should work. Let’s rerun the package tests.

All passed this time. Excellent!

But note that this only runs all tests in the db package. We also have more tests in the api package.

So let’s open the terminal and run make test to run all of them.

We’ve got an error here because our MockStore doesn’t implement the db.Store interface. It’s missing some functions’ implementation.

That’s because when we run make sqlc before to generate codes, 2 new functions: CreateUser and GetUser has been added to the Querier interface. And the Querier interface is a part of the db.Store interface.

To fix this, we have to regenerate the code for the MockStore:



❯ make mock
mockgen -package mockdb -destination db/mock/store.go github.com/techschool/simplebank/db/sqlc Store

After this, we can see in the db/mock/store.go file, the implementation of the GetUser and CreateUser function has been added:



// CreateUser mocks base method
func (m *MockStore) CreateUser(arg0 context.Context, arg1 db.CreateUserParams) (db.User, error) {
    m.ctrl.T.Helper()
    ret := m.ctrl.Call(m, "CreateUser", arg0, arg1)
    ret0, _ := ret[0].(db.User)
    ret1, _ := ret[1].(error)
    return ret0, ret1
}

// CreateUser indicates an expected call of CreateUser
func (mr *MockStoreMockRecorder) CreateUser(arg0, arg1 interface{}) *gomock.Call {
    mr.mock.ctrl.T.Helper()
    return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateUser", reflect.TypeOf((*MockStore)(nil).CreateUser), arg0, arg1)
}

// GetUser mocks base method
func (m *MockStore) GetUser(arg0 context.Context, arg1 string) (db.User, error) {
    m.ctrl.T.Helper()
    ret := m.ctrl.Call(m, "GetUser", arg0, arg1)
    ret0, _ := ret[0].(db.User)
    ret1, _ := ret[1].(error)
    return ret0, ret1
}

// GetUser indicates an expected call of GetUser
func (mr *MockStoreMockRecorder) GetUser(arg0, arg1 interface{}) *gomock.Call {
    mr.mock.ctrl.T.Helper()
    return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetUser", reflect.TypeOf((*MockStore)(nil).GetUser), arg0, arg1)
}

So now the api unit tests should work. Let’s rerun make test in the terminal.

All passed this time. Perfect!

Handle different types of DB error

Alright, now let’s try to run the HTTP server.



❯ make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /users                    --> github.com/techschool/simplebank/api.(*Server).createUser-fm (3 handlers)
[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (3 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (3 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccounts-fm (3 handlers)
[GIN-debug] POST   /transfers                --> github.com/techschool/simplebank/api.(*Server).createTransfer-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8080

Then open Postman to test the existing API to create a new account.

First, I’m gonna try creating an account for an owner that doesn’t exist in the database.

As you can see, we’ve got an error because the foreign key constraint for the account owner is violated. This is expected, since there’s no account with this username in the database yet.

However, the HTTP response status code is 500 Internal Server Error. This status is not very suitable in this case since the fault is on the client’s side because it’s trying to create a new account for an inexisted user.

It’s better to return something like 403 Forbiden status instead. In order to do that, we have to handle the error returned by Postgres.

Here in the create account handler of api/account.go file, after calling store.CreateAccount, if an error is returned, we will try to convert it to pq.Error type, and assign the result to pqErr variable:



func (server *Server) createAccount(ctx *gin.Context) {
    ...

    account, err := server.store.CreateAccount(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            log.Println(pqErr.Code.Name())
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, account)
}

If the conversion is OK, let’s print out a log here to see this error’s code name.

Now I will restart the server. Then go back to Postman and resend the same request.

Now in the log, we can see the error’s code name is foreign_key_violation. We can use it to classify the error later.

But before that, I’m gonna try to create a new account for an existed user. Let’s copy this username from the users table, and paste it to this owner field’s value, then send the request.

This time the request is successful, and a new account is created. But what if we send this same request a second time?

Now we’ve got another error: duplicate key value violates unique constraints owner_currency_key. That’s because we’re trying to create more than 1 account with the same currency for the same owner.

In this case, we also want to return status 403 Forbidden instead of 500 Internal Server Error. So let’s look at the log to see its error code name.

It’s unique_violation. OK, now with this information, let’s go back to the code and update it.



func (server *Server) createAccount(ctx *gin.Context) {
    ...

    account, err := server.store.CreateAccount(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "foreign_key_violation", "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, account)
}

Here I will use a simple switch case statement to check the error code name. In case it is foreign_key_violation or unique_violation, we will send this error response with http.StatusForbidden status code.

Alright, let’s restart the server! Then resend the request.

As you can see, the returned status code is now 403 Forbidden as we expected.

Let’s try changing the owner field’s value to an inexisted username as before, and resend the request one more time.

We also get a 403 Forbidden status in this case. So it worked!

Last but not least, I’m gonna try to create a second account for this user, but with a different currency, such as EUR.

The request is successful. And in the database, we can see there are 2 new accounts linking to the same username, 1 account is EUR and the other is USD.

So that brings us to the end of this lecture. I hope you have learned something useful.

Thank you for reading and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Add users table with unique & foreign key constraints in PostgreSQL

TECH SCHOOL — Sun, 27 Dec 2020 16:40:45 +0000

Hi guys, welcome back to the backend master class!

So far we’ve implemented several features for our simple bank system such as create, update, retrieve, or transfer money between bank accounts.

However, there’s one very important feature that’s still missing: user authentication and authorization. Our banking system could not be completed until this feature is implemented.

So today, we’re gonna take the first step to implement it, which is adding a new users table to the database and link it with the existing accounts table via some db constraints.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Add table users

Alright, so this is the current schema of our simple bank database written using dbdiagram.io

Table accounts as A {
  id bigserial [pk]
  owner varchar [not null]
  balance bigint [not null]
  currency varchar [not null]
  created_at timestamptz [not null, default: `now()`]

  Indexes {
    owner
  }
}

Table entries {
  id bigserial [pk]
  account_id bigint [ref: > A.id, not null]
  amount bigint [not null, note: 'can be negative or positive']
  created_at timestamptz [not null, default: `now()`]

  Indexes {
    account_id
  }
}

Table transfers {
  id bigserial [pk]
  from_account_id bigint [ref: > A.id, not null]
  to_account_id bigint [ref: > A.id, not null]
  amount bigint [not null, note: 'must be positive']
  created_at timestamptz [not null, default: `now()`]

  Indexes {
    from_account_id
    to_account_id
    (from_account_id, to_account_id)
  }
}

At the moment, the accounts table has an owner field to tell us whom this account belong to. So we can use this field as a foreign key to link to the new users table that we’re going to create.

Let’s define Table users with an alias U.

The first field of this table is a username of type varchar. Every user should have a unique username, so this field can serve as the primary key of this table.

Next, we have the hashed_password field to store the hash value of the user’s password. Its type is also varchar, and it should be not null.

Why is it hashed_password and not just password? Well, basically we should never store a naked password in the database, because if we do then everyone who has access to the database will see the passwords of all users, which is a very critical security issue.

We will learn more about how to store and verify password in another lecture. For now, let’s just focus on adding this new users table.

Table users as U {
  username varchar [pk]
  hashed_password varchar [not null]
  full_name varchar [not null]
  email varchar [unique, not null]
  password_changed_at timestamptz [not null, default: '0001-01-01 00:00:00Z']
  created_at timestamptz [not null, default: `now()`]
}

OK, the next field will store the full_name of the user, so it is also of varchar type, and should not be null.

One more important field the users table should have is email. We will use it later to communicate with the users, for example, when they forget their password and want to reset it. So the type of this field should also be varchar. And note that it must be unique and not null, since we don’t want to have 2 users with the same email.

Then just like other tables, we will have a created_at field to store the time this user is created. This field is not null, and it has a default value of now(), so Postgres will automatically fill in the current timestamp when the new user record is inserted.

Moreover, for security reason, it’s often a good idea to ask users to change their password frequently, like once every month for instance. So we will need a field: password_changed_at to know when was the last time user changed their password. Its type should be timestamp with timezone, and should also be not null.

The reason I want every field to be not null is because it makes our developer’s life much easier since we don’t have to deal with null pointers.

If the password has never been changed, we will just use a default value which is a long time in the past. As we’re using golang, I’m gonna use a zero value timestamp of Go here. So it should be year 1, month 1, date 1, at 0 hour, 0 minute, 0 second, and the timezone is UTC: '0001-01-01 00:00:00Z'. The letter Z here means zero timezone.

OK so that’s the definition of the new users table.

Add foreign key constraint

Now keep in mind that we want to allow 1 user to have multiple accounts with different currencies, so what I’m gonna do is to link the owner field of the accounts table to the username field of users table. That would make the owner field become a foreign key.

Here in the definition of the owner, I’m gonna add a reference tag that points to U.username. To remind you, U is just an alias of the users table.

Table accounts as A {
  id bigserial [pk]
  owner varchar [ref: > U.username, not null]
  balance bigint [not null]
  currency varchar [not null]
  created_at timestamptz [not null, default: `now()`]

  Indexes {
    owner
  }
}

Now in the diagram, we can see that there’s a new link between the username field of users table and the owner field of the accounts table.

The number 1 and character * at the end of this link tell us that this is a 1-to-many relationship, which basically means, 1 user can have multiple accounts, but 1 account can only belong to exactly 1 single user.

Add unique constraint

There’s one more thing we should pay attention to here. We allow 1 user to have multiple accounts, but those accounts should have different currencies. For example, you can have 1 USD account and 1 EUR account, but clearly should not have 2 different USD accounts.

One way to set this constraint at the database level is to add a composite unique index to the accounts table. This index is composed of 2 fields: owner and currency. That’s why it’s called composite index - an index that involves more than 1 field.

Table accounts as A {
  id bigserial [pk]
  owner varchar [ref: > U.username, not null]
  balance bigint [not null]
  currency varchar [not null]
  created_at timestamptz [not null, default: `now()`]

  Indexes {
    owner
    (owner, currency) [unique]
  }
}

Export to PostgreSQL

OK so now our new schema is ready. Let’s export it to PostgreSQL.

Here we can see the code to create users table:

CREATE TABLE "users" (
  "username" varchar PRIMARY KEY,
  "hashed_password" varchar NOT NULL,
  "full_name" varchar NOT NULL,
  "email" varchar UNIQUE NOT NULL,
  "password_changed_at" timestamptz NOT NULL DEFAULT '0001-01-01 00:00:00Z',
  "created_at" timestamptz NOT NULL DEFAULT (now())
);

Then the alter table command to add foreign key constraint to the owner field:

ALTER TABLE "accounts" ADD FOREIGN KEY ("owner") REFERENCES "users" ("username");

And a composite unique index for the owner and currency.

CREATE UNIQUE INDEX ON "accounts" ("owner", "currency");

Add new schema change to our project

Next, we have to add these new changes to our simple bank project.

One way to do that is to replace the whole content of the init_schema migration file with the new one, then reset the database and rerun the migrate up command.

However, in a real-world project, it’s not the right way go. Why?

Because requirements change all the time, and it might come after the first version of our system is deployed to the production. And once we have data in the production DB, we cannot reset it to rerun the old migration.

So the right way to apply new schema change is to create a new migration version. Let’s open the terminal to generate a new migration.

We have learned how to do this in lecture 3 of this course. But if you don’t remember the command, just run:

migrate -help

To create a new migration, we run this command:

migrate create -ext sql -dir db/migration -seq add_users

We use some parameters to tell migrate to set the the output file extension to sql, the output directory to db/migration, use a sequential number as the file name prefix, and the migration name is add_users.

As you can see here, 2 migration files has been generated inside the db/migration folder:

Let’s implement them in visual studio code.

Implement the up migration

I’m gonna start with the up migration.

First, we need to create table users. So let’s copy the SQL query that dbdiagram has generated for us and paste it to this migration file 000002_add_users.up.sql

Next, I’m gonna copy the query that adds a new foreign key constraint to the owner field of accounts table. And finally, the query to create a unique composite index for the owner and currency as well.

CREATE TABLE "users" (
  "username" varchar PRIMARY KEY,
  "hashed_password" varchar NOT NULL,
  "full_name" varchar NOT NULL,
  "email" varchar UNIQUE NOT NULL,
  "password_changed_at" timestamptz NOT NULL DEFAULT('0001-01-01 00:00:00Z'),  
  "created_at" timestamptz NOT NULL DEFAULT (now())
);

ALTER TABLE "accounts" ADD FOREIGN KEY ("owner") REFERENCES "users" ("username");

CREATE UNIQUE INDEX ON "accounts" ("owner", "currency");

This would be good enough for our new migration up script. However, I’m gonna show you another way to ensure that each owner has at most 1 account for a specific currency.

Instead of using a direct unique index like this, we can add a unique constraint for the pair of owner and currency on the accounts table. It will be very similar to the command to add foreign key constraint above:

-- CREATE UNIQUE INDEX ON "accounts" ("owner", "currency");
ALTER TABLE "accounts" ADD CONSTRAINT "owner_currency_key" UNIQUE ("owner", "currency");

Basically, under the hood, adding this unique constraint will automatically create the same unique composite index for owner and currency as the command we wrote above.

Postgres will need that index to check and enforce the unique constraint faster. So you can choose either of these 2 commands, whatever you like.

Run the migration up

OK, now let’s open the terminal and run make migrateup to apply this new migration.

Oops, we’ve got an error. And the reason is that the foreign key constraint is violated. Why?

Well, that’s because we already have some existing rows in the accounts table, but their owner field is completely random and doesn’t link to any existed user. Of course, since the users table doesn’t exist until now.

So in this case, we have to clean up all the existing data before running migrate up. This is possible because our existing system is not ready to deploy to production yet.

But note that, as the previous migrate up run was failed, it will change the current schema migration to version 2 but in a dirty state.

So now if we run make migratedown with the purpose of cleaning up the data, we will get an error because of this dirty version.

To fix this, I’m gonna manually update the value of this dirty field to false, save it, and go back to the terminal to run make migratedown.

This time the migrations run successfully, and all tables in our database are gone. We can now run make migrateup again to get them back.

OK, the migrations are successful. And back to TablePlus, we can see the new users table. Let’s click on the Structure button to see its schema.

It has all the fields that we declared in the code: username, hashed_password, full_name, email.

The password_changed_at field has the correct default value, and similar for the created_at field.

Look at the bottom, there are 2 BTREE indexes:

One is for the primary key username, which should be unique.
And the other is for the email, which is a unique index as well.

Let’s check the accounts table.

Here in the owner field, we can see that it now has a foreign key constraint that links to the username field of the users table.

And at the bottom, there’s a new unique index for the owner and currency pair.

So our migration up script works perfectly! Now let’s go back to the code and complete the migration down.

Implement the migration down

When writing the migration down, we should reverse what was done in the migration up.

So first we have to drop the unique constraint for the owner and currency pair of the accounts table. The command is:

ALTER TABLE IF EXISTS "accounts" DROP CONSTRAINT IF EXISTS "owner_currency_key";

Next, we have to drop the foreign key constraint for the owner field in a similar way. But now how can we know the name of this foreign key constraint?

Well, it’s very easy! Let’s go back to Table Plus and click on the Info button at the bottom of the window.

Here in the table definition, we can see the foreign key constraint name: accounts_owner_fkey

Let’s copy it, and paste it to this command:

ALTER TABLE IF EXISTS "accounts" DROP CONSTRAINT IF EXISTS "accounts_owner_fkey";

The last step we should do is to drop the users table. So let’s add this command to the file:

DROP TABLE IF EXISTS "users";

And that’s it! The 000002_add_users.down.sql file is done. How can we test if it works or not?

Test the up and down migrations

At the moment, in the Makefile, we only have one make migratedown command to run all the migration down versions. But in this case, we only want to run 1 last migration down version.

So let’s add a new make command for this purpose. I’m gonna called it migratedown1. The migrate command would be the same as before, except that we need to add one more argument at the end.

migratedown1:
  migrate -path db/migration -database "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable" -verbose down 1

The number 1 here means that we only want to rollback 1 last migration, or more precisely, just run the last down migration version that was applied before.

Similarly, I will duplicate the migrateup command, and add a new migrateup1 command that will only applies 1 next migration version from the current one:

migrateup1:
  migrate -path db/migration -database "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable" -verbose up 1

Alright, now let’s add these 2 new commands to the PHONY list:

.PHONY: postgres createdb dropdb migrateup migratedown migrateup1 migratedown1 sqlc test server mock

And go back to the terminal and run:

make migratedown1

The migration is successful. Now in TablePlus, we can see the current version has been changed to 1.

The users table is gone, and in the accounts table, there’s no more foreign key constraint for the owner column, as well as the unique constraint for the owner and currency pair.

So the migration down script worked!

OK, now let’s run this command to update the schema to the latest version.

make migrateup1

As you can see, the new constraints in the accounts table are back.

The current migration version is 2, and the new users table is here as expected.

So today we have successfully added a new users table to the simple bank database.

And while doing so, we also learned how to add some foreign key and unique constraints to build up the relationship between tables and to ensure the consistency of the data.

In the next lectures, we’re gonna update our golang code to work with this new table, and then add more feature to authenticate and authorize users.

Thank you for reading, and I’ll see you guys in the next one very soon!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Implement transfer money API with a custom params validator in Go

TECH SCHOOL — Mon, 07 Dec 2020 21:46:24 +0000

In the previous lectures, we have implemented and tested the HTTP APIs to mange bank accounts for our simple bank project.

Today, we will do some more practice by implementing the most important API of our application: transfer money API.

And while doing so, I will show you how to write a custom validator to validate the input parameters of this API.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Implement the transfer money API handler

First I will create a new transfer.go file inside the api package. The implementation of the transfer money API will be very similar to that of the create account API.

The struct to store input parameters of this API should be transferRequest. It will have several fields:

type transferRequest struct {
    FromAccountID int64  `json:"from_account_id" binding:"required,min=1"`
    ToAccountID   int64  `json:"to_account_id" binding:"required,min=1"`
    Amount        int64  `json:"amount" binding:"required,gt=0"`
    Currency      string `json:"currency" binding:"required,oneof=USD EUR CAD"`
}

The FromAccountID of type int64 is the ID of the account where money is going out. This field is required, and its minimum value should be 1.
Similarly, the ToAccountID, also of type int64, is the ID of the account where the money is going in.
Next, we have the Amount field to store the amount of money to transfer between 2 accounts. For simplicity, here I just use integer type. But in reality, it could be a real number, depending on the currency. So you should keep that in mind and choose the appropriate type for your data. For the binding condition, this field is also required and it should be greater than 0.
The last field is the Currency of the money we want to transfer. For now, we only allow it to be either USD, EUR or CAD. And note that this currency should match the currency of both 2 accounts. We will verify that in the API handler function.

Alright, now the handler function’s name should be createTransfer. And the request variable’s type should be transferRequest.

We bind the input parameters to the request object, and return http.StatusBadRequest to the client if the any of the parameters is invalid.

func (server *Server) createTransfer(ctx *gin.Context) {
    var req transferRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    arg := db.TransferTxParams{
        FromAccountID: req.FromAccountID,
        ToAccountID:   req.ToAccountID,
        Amount:        req.Amount,
    }

    result, err := server.store.TransferTx(ctx, arg)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, result)
}

Next we have to create a db.TransferTxParam object, where FromAccountID is request.fromAccountID, ToAccountID is request.toAccountID, and Amount is request.Amount.

With this argument, we call server.store.TransferTx() to perform the money transfer transaction. This function will return a TransferTxResult object or an error. At the end, we just need to return that result to the client if no errors occur.

Alright, this create transfer handler is almost finished except that we haven’t taken into account the last input parameter: request.Currency.

What we need to do is to compare this currency with the currency of the from account and to account to make sure that they’re all the same. So I’m gonna define a new function called validAccount() for the Server struct.

This function will check if an account with a specific ID really exists, and its currency matches the input currency. Therefore, it will have 3 input arguments: A gin.Context, an accountID, and a currency string. And it will return a bool value.

func (server *Server) validAccount(ctx *gin.Context, accountID int64, currency string) bool {
    account, err := server.store.GetAccount(ctx, accountID)
    if err != nil {
        if err == sql.ErrNoRows {
            ctx.JSON(http.StatusNotFound, errorResponse(err))
            return false
        }

        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return false
    }

    if account.Currency != currency {
        err := fmt.Errorf("account [%d] currency mismatch: %s vs %s", account.ID, account.Currency, currency)
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return false
    }

    return true
}

First, we call server.store.GetAccount() to query the account from the database. This function will return an account object or an error. If error is not nil, then there are 2 possible scenarios:

The first scenario is when the account doesn’t exist, then we send http.StatusNotFound to the client and return false.
The second scenario is when some unexpected errors occur, so we just send http.StatusInternalServerError and return false.

Otherwise, if there’s no error, we will check if the account’s currency matches the input currency or not. If it doesn’t match, we declare a new error: account currency mismatch. Then we call ctx.JSON() with the http.StatusBadRequest to send this error response to the client, and return false.

Finally, if everything is good, and the account is valid, we return true at the end of this function.

Now let’s go back to the createTransfer handler.

func (server *Server) createTransfer(ctx *gin.Context) {
    var req transferRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    if !server.validAccount(ctx, req.FromAccountID, req.Currency) {
        return
    }

    if !server.validAccount(ctx, req.ToAccountID, req.Currency) {
        return
    }

    arg := db.TransferTxParams{
        FromAccountID: req.FromAccountID,
        ToAccountID:   req.ToAccountID,
        Amount:        req.Amount,
    }

    result, err := server.store.TransferTx(ctx, arg)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, result)
}

We call server.validAccount() to check the validity of the request.fromAccountID and currency. If it’s not valid, then we just return immediately. We do the same thing for the request.toAccountID.

And that’s it! The createTransfer handler is completed. Next we have to register a new API in the server to route requests to this handler.

Register the transfer money API route

Let’s open the api/server.go file.

I’m gonna duplicate the router.POST account API. Then change the path to /transfers, and the handler’s name to server.createTransfer.

func NewServer(store db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccounts)
    router.POST("/transfers", server.createTransfer)

    server.router = router
    return server
}

That’s all. Let’s open the terminal and run:

❯ make server

Test the transfer money API

Then I’m gonna use Postman to test the new transfer money API.

Let's create a new request with method POST, and the URL is http://localhost:8080/transfers.

For the request body, let’s select raw, and choose JSON format. I’m gonna use this sample JSON body:

{
    "from_account_id": 1,
    "to_account_id": 2,
    "amount": 10,
    "currency": "USD"
}

The from account id is 1, the to account id is 2, the amount is 10, and the currency is USD.

Let’s open TablePlus to see the current data of these 2 accounts.

Here we can see that their currencies are different. The currency of account 2 is USD, but that of account 1 is CAD.

So if we send this API request, we will get a currency mismatch error for the account 1.

To fix this, I will update the currency of account 1 to USD in TablePlus. Save it, and go back to Postman to send the request again.

This time the request is successful.

{
    "transfer": {
        "id": 110,
        "from_account_id": 1,
        "to_account_id": 2,
        "amount": 10,
        "created_at": "2020-12-07T19:57:09.61222Z"
    },
    "from_account": {
        "id": 1,
        "owner": "ojkelz",
        "balance": 734,
        "currency": "USD",
        "created_at": "2020-11-28T15:22:13.419691Z"
    },
    "to_account": {
        "id": 2,
        "owner": "ygmlfb",
        "balance": 824,
        "currency": "USD",
        "created_at": "2020-11-28T15:22:13.435304Z"
    },
    "from_entry": {
        "id": 171,
        "account_id": 1,
        "amount": -10,
        "created_at": "2020-12-07T19:57:09.61222Z"
    },
    "to_entry": {
        "id": 172,
        "account_id": 2,
        "amount": 10,
        "created_at": "2020-12-07T19:57:09.61222Z"
    }
}

And in the response, we have a transfer record with id 110 from account 1 to account 2 with amount of 10. The new balance of account 1 is 734 USD. While that of account 2 is 824 USD.

Let’s check the database records in TablePlus. Before the transaction, the original values of account 1 and account 2 were 744 and 814 USD.

When I press command R to refresh the data, we can see that the balance of account 1 has decreased by 10, and the balance of account 2 has increased by 10.

Moreover, in the response, we also see 2 account entry objects:

One entry is to record that 10 USD has been subtracted from account 1.
And the other entry is to record that 10 USD has been added to account 2.

We can find these 2 records at the end of the entries table in the database. Similar for the transfer record at the bottom of the transfers table, which matches with the transfer object returned in the JSON response.

Alright, so the transfer API works perfectly. But there’s one thing I want to show you.

Implement a custom currency validator

Here, in the binding condition of the currency field, we’re hard-coding 3 constants for USD, EUR and CAD.

type transferRequest struct {
    FromAccountID int64  `json:"from_account_id" binding:"required,min=1"`
    ToAccountID   int64  `json:"to_account_id" binding:"required,min=1"`
    Amount        int64  `json:"amount" binding:"required,gt=0"`
    Currency      string `json:"currency" binding:"required,oneof=USD EUR CAD"`
}

What if in the future we want to support 100 different types of currency? It would be very hard to read and easy to make mistakes if we put 100 currency values in this oneof tag.

Also, there will be duplications because the currency parameter can appear in many different APIs. For now, we already have 1 duplication in the create account request.

type createAccountRequest struct {
    Owner    string `json:"owner" binding:"required"`
    Currency string `json:"currency" binding:"required,oneof=USD EUR CAD"`
}

In order to avoid that, I’m gonna show you how to write a custom validator for the currency field.

Let’s create a new file validator.go inside the api folder. Then declare a new variable validCurrency of type validator.Func

package api

import (
    "github.com/go-playground/validator/v10"
    "github.com/techschool/simplebank/util"
)

var validCurrency validator.Func = func(fieldLevel validator.FieldLevel) bool {
    if currency, ok := fieldLevel.Field().Interface().(string); ok {
        return util.IsSupportedCurrency(currency)
    }
    return false
}

Visual studio code has automatically imported the validator package for us. However, we have to add /v10 at the end of this import path because we want to use the version 10 of this package.

Basically, validator.Func is a function that takes a validator.FieldLevel interface as input and return true when validation succeeds. This is an interface that contains all information and helper functions to validate a field.

What we need to do is calling fieldLevel.Field() to get the value of the field. Note that it’s a reflection value, so we have to call .Interface() to get its value as an interface{}. Then we try to convert this value to a string.

The conversion will return a currency string and a ok boolean value. If ok is true then the currency is a valid string. In this case, we will have to check if that currency is supported or not. Else, if ok is false, then the field is not a string. Therefore, we just return false.

Alright, now I will create a new file currency.go inside the util package. We will implement the function IsSupportedCurrency() to check if a currency is supported or not in this file.

First I will declare some constants for the currencies that we want to support in our bank. For now let’s say we only support USD, EUR, and CAD. We can always add more currencies in the future if we want.

Then let’s write a new function IsSupportedCurrency() that takes a currency string as input and return a bool value. It will return true if the input currency is supported and false otherwise.

package util

// Constants for all supported currencies
const (
    USD = "USD"
    EUR = "EUR"
CAD = "CAD"
)

// IsSupportedCurrency returns true if the currency is supported
func IsSupportedCurrency(currency string) bool {
    switch currency {
    case USD, EUR, CAD:
        return true
    }
    return false
}

In this function, we just use a simple switch case statement. In case the currency is USD, EUR, or CAD, we return true. Else, we return false.

And that’s it! Our custom validator validCurrency is done. Next we have to register this custom validator with Gin.

Register the custom currency validator

Let’s open the server.go file.

package api

import (
    "github.com/gin-gonic/gin"
    "github.com/gin-gonic/gin/binding"
    "github.com/go-playground/validator/v10"
)

func NewServer(store db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
        v.RegisterValidation("currency", validCurrency)
    }

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccounts)

    router.POST("/transfers", server.createTransfer)

    server.router = router
    return server
}

Here, after creating the Gin router, we call binding.Validator.Engine() to get the current validator engine that Gin is using (binding is a sub-package of Gin).

Note that this function will return a general interface type, which by default is a pointer to the validator object of the go-playground/validator/v10 package.

So here we have to convert the output to a validator.Validate object pointer. If it is ok then we can call v.RegisterValidation() to register our custom validate function.

The first argument is the name of the validation tag: currency. And the second argument should be the validCurrency function that we have implemented before.

Use the custom currency validator

Alright, now with this new validator registered, we can start using it.

Here in the createAccountRequest struct, we can replace the oneof=USD EUR CAD tag with just currency tag:

type createAccountRequest struct {
    Owner    string `json:"owner" binding:"required"`
    Currency string `json:"currency" binding:"required,currency"`
}

And similar for this Currency field of the transferRequest struct:

type transferRequest struct {
    FromAccountID int64  `json:"from_account_id" binding:"required,min=1"`
    ToAccountID   int64  `json:"to_account_id" binding:"required,min=1"`
    Amount        int64  `json:"amount" binding:"required,gt=0"`
    Currency      string `json:"currency" binding:"required,currency"`
}

OK, let’s restart the server and test it.

❯ make server

I’m gonna change the currency to EUR and send the Postman request.

This is a valid supported currency, but it doesn’t match the currency of the accounts, so we’ve got a 400 Bad Request status with the currency mismatch error.

If I change it back to USD, then the request is successful again:

Now let’s try an unsupported currency, such as AUD:

This time, we also get 400 Bad Request status, but the error is because the field validation for the currency failed on the currency tag, which is exactly what we expected.

So our custom currency validator is working very well!

So today we have learned how to implement the transfer money API with a custom parameter binding validator. I hope it is useful for you.

Although I didn’t show you how to write unit tests for this new API in the article because it would be very similar to what we’ve learned in the previous lecture, I actually still write a lot of unit tests for it and push them to Github.

I encourage you to check them out on the simple bank repository to see how they were implemented.

Thanks a lot for reading this article, and I will see you soon in the one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Mock DB for testing HTTP API in Go and achieve 100% coverage

TECH SCHOOL — Sat, 14 Nov 2020 13:39:41 +0000

If you have trouble isolating unit test data to avoid conflicts, think about mock DB! In this article, we will learn how to use Gomock to generate stubs for the DB interface, which helps us write API unit tests faster, cleaner, and easily achieve 100% coverage.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Why mocking DB

In the previous lectures, we have learned how to implement RESTful HTTP APIs in Go. When it comes to testing these APIs, some people might choose to connect to the real database, while some others might prefer to just mocking it. So which approach should we use?

Well, I would say it’s up to you. But for me, mocking is better because of the following reasons:

First, it helps us to write independent tests more easily because each test will use its own separate mock DB to store data, so there will be no conflicts between them. If you use a real DB, all tests will read and write data to the same place, so it would be harder to avoid conflicts, especially in a big project with a large code base.
Second, our tests will run much faster since they don’t have to spend time talking to the DB and waiting for the queries to run. All actions will be performed in memory and within the same process.
The third and very important reason for mocking DB is: It allows us to write tests that achieve 100% coverage. With a mock DB, we can easily set up and test some edge cases, such as an unexpected error, or a connection lost, which would be impossible to achieve if we use a real DB.

OK, that sounds great. But is it good enough to test our API with just a mock DB? Can we be confident that our codes will still perform well when a real DB is plugged in?

Yes, absolutely! Because our code that talks to the real DB is already tested carefully in the previous lecture.

So all we need to do is: make sure that the mock DB implements the same interface as the real DB. Then everything will be working just fine when being put together.

How to mock DB

There are 2 ways to mock DB.

The first one is to implement a fake DB, which stores data in memory. If you followed my gRPC course then you definitely have already known about it.

For example, here we have the Store interface that defines a list of actions we can do with the real DB.

Then we have a fake DB MemStore struct, which implements all actions of the Store interface, but only uses a map to read and write data.

This approach of using fake db is very simple and easy to implement. However, it requires us to write a lot more codes that only be used for testing, which is quite time-consuming for both development and maintenance later.

So today I’m gonna show you a better way to mock DB, which is using stubs instead of fake DB.

The idea is to use gomock package to generate and build stubs that return hard-coded values for each scenario we want to test.

In this example, gomock already generated a MockStore for us. So all we need to do is to call its EXPECT() function to build a stub, which tells gomock that: this GetAccount() function should be called exactly 1 time with this input accountID, and return this account object as output.

And that’s it! After setting up the stub, we can simply use this mock store to test the API.

Don’t worry if you don’t fully understand it right now. Let’s jump into coding to see how it really works!

Install gomock

First, we need to install gomock. Let’s open the browser and search for gomock. Then open its Github page.

Copy this go get command and run it in the terminal to install the package:



❯ go get github.com/golang/mock/mockgen@v1.4.4

After this, a mockgen binary file will be available in the go/bin folder.



❯ ls -l ~/go/bin
total 341344
...
-rwxr-xr-x  1 quangpham  staff  10440388 Oct 17 18:27 gotests
-rwxr-xr-x  1 quangpham  staff   8914560 Oct 17 18:27 guru
-rwxr-xr-x  1 quangpham  staff   5797544 Oct 17 18:27 impl
-rwxr-xr-x  1 quangpham  staff   7477056 Nov  2 09:21 mockgen

We will use this tool to generate the mock db, so it’s important to make sure that it is executable from anywhere. We check that by running:



❯ which mockgen
mockgen not found

Here it says mockgen not found. That’s because the go/bin folder is not in the PATH environment variable at the moment.

To add it to the PATH, I will edit the .zshrc file since I’m using zsh. If you’re using a bash shell then you should edit the .bash_profile or .bashrc file instead.



❯ vi ~/.zshrc

I'm using vim, so let's press i to enter the insert mode. Then add this export command to the top of the file:



export PATH=$PATH:~/go/bin

Press Esc to exit the insert mode, then :wq to save the file and quit vim.

Next we have to run this source command to reload the .zshrc file:



❯ source ~/.zshrc

Now if we run which mockgen again, we can see that it is now available in the go/bin folder.



❯ which mockgen
/Users/quangpham/go/bin/mockgen

Note that the .zshrc file will be automatically loaded when we start a new terminal window. So we don’t need to run the source command every time we open the terminal.

Define Store interface

Alright, now in order to use mockgen to generate a mock DB, we have to update our code a bit.

At the moment, in the api/server.go file, the NewServer() function is accepting a db.Store object:



type Server struct {
    store  *db.Store
    router *gin.Engine
}

func NewServer(store *db.Store) *Server {
    ...
}

This db.Store is defined in the db/sqlc/store.go file. It is a struct which will always connect to the real database:



type Store struct {
    db *sql.DB
    *Queries
}

So in order to use a mock DB in the API server tests, we have to replace that store object with an interface. I’m gonna duplicate this Store struct definition and change its type to interface.



type Store interface {
    // TODO: add functions to this interface
}

type SQLStore struct {
    db *sql.DB
    *Queries
}

Then the old Store struct will be renamed to SQLStore. It will be a real implementation of the Store interface that talks to a SQL database, which is PostgreSQL in this case.

Then this NewStore() function should not return a pointer, but just a Store interface. And inside, it should return the real DB implementation of the interface, which is SQLStore.



func NewStore(db *sql.DB) Store {
    return &SQLStore{
        db:      db,
        Queries: New(db),
    }
}

We also have to change the type of the store receiver of the execTx() function and the TransferTx() function to *SQLStore like this:



func (store *SQLStore) execTx(ctx context.Context, fn func(*Queries) error) error {
    ...
}

func (store *SQLStore) TransferTx(ctx context.Context, arg TransferTxParams) (TransferTxResult, error) {
}

Alright, now we have to define a list of actions that the Store interface can do.

Basically, it should have all functions of the Queries struct, and one more function to execute the transfer money transaction.

So first I’m gonna copy this TransferTx() function signature and paste it inside the Store interface:



type Store interface {
    TransferTx(ctx context.Context, arg TransferTxParams) (TransferTxResult, error)
}

For the functions of the Queries struct, of course, we can do the same, like going through all of them and copy-paste one by one. However, it will be too time-consuming because this struct can contain a lot of functions.

Lucky for us, the sqlc package that we used to generate CRUD codes also has an option to emit an interface that contains all of the function of the Queries struct.

All we have to do is to change this emit_interface setting in the sqlc.yaml file to true:



version: "1"
packages:
  - name: "db"
    path: "./db/sqlc"
    queries: "./db/query/"
    schema: "./db/migration/"
    engine: "postgresql"
    emit_json_tags: true
    emit_prepared_queries: false
    emit_interface: true
    emit_exact_table_names: false
    emit_empty_slices: true

Then run this command in the terminal to regenerate the codes:



❯ make sqlc

After this, in the db/sqlc folder, we can see a new file called querier.go. It contains the generated Querier interface with all functions to insert and query data from the database:



type Querier interface {
    AddAccountBalance(ctx context.Context, arg AddAccountBalanceParams) (Account, error)
    CreateAccount(ctx context.Context, arg CreateAccountParams) (Account, error)
    CreateEntry(ctx context.Context, arg CreateEntryParams) (Entry, error)
    CreateTransfer(ctx context.Context, arg CreateTransferParams) (Transfer, error)
    DeleteAccount(ctx context.Context, id int64) error
    GetAccount(ctx context.Context, id int64) (Account, error)
    GetAccountForUpdate(ctx context.Context, id int64) (Account, error)
    GetEntry(ctx context.Context, id int64) (Entry, error)
    GetTransfer(ctx context.Context, id int64) (Transfer, error)
    ListAccounts(ctx context.Context, arg ListAccountsParams) ([]Account, error)
    ListEntries(ctx context.Context, arg ListEntriesParams) ([]Entry, error)
    ListTransfers(ctx context.Context, arg ListTransfersParams) ([]Transfer, error)
    UpdateAccount(ctx context.Context, arg UpdateAccountParams) (Account, error)
}

var _ Querier = (*Queries)(nil)

And here you can see it declares a blank variable var _ Querier to make sure that the Queries struct must implement all functions of this Querier interface.

Now what we need to do is just embed this Querier inside the Store interface. That would make Store interface to have all of its functions in addition to the TransferTx() function that we’ve added before:



type Store interface {
    Querier
    TransferTx(ctx context.Context, arg TransferTxParams) (TransferTxResult, error)
}

Next, we have to go back to the api/server.go file and remove this * from *db.Store type because it is no longer a struct pointer, but an interface instead:



func NewServer(store db.Store) *Server {
    ...
}

Note that although we have changed the Store type from struct to interface, our code will still work well, and we don’t have to change anything in the main.go file because the db.NewStore() function is now also returning a Store interface with the actual implementation SQLStore that connects to the real SQL DB.



func main() {
    config, err := util.LoadConfig(".")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    conn, err := sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server := api.NewServer(store)

    err = server.Start(config.ServerAddress)
    if err != nil {
        log.Fatal("cannot start server:", err)
    }
}

Generate mock DB

Alright, so now as we have the db.Store interface, we can use gomock to generate a mock implementation of it.

First I will create a new mock folder inside the db package. Then let’s open the terminal and run:



❯ mockgen -help
mockgen has two modes of operation: source and reflect.

Source mode generates mock interfaces from a source file.
It is enabled by using the -source flag. Other flags that
maybe useful in this mode are -imports and -aux_files.
Example:
    mockgen -source=foo.go [other options]

Reflect mode generates mock interfaces by building a program
that uses reflection to understand interfaces. It is enabled
by passing two non-flag arguments: an import path, and a
comma-separated list of symbols.
Example:
    mockgen database/sql/driver Conn,Driver

    -aux_files string
        (source mode) Comma-separated pkg=path pairs of auxiliary Go source files.
    -build_flags string
        (reflect mode) Additional flags for go build.
    -copyright_file string
        Copyright file used to add copyright header
    -debug_parser
        Print out parser results only.
    -destination string
        Output file; defaults to stdout.
    -exec_only string
        (reflect mode) If set, execute this reflection program.
    -imports string
        (source mode) Comma-separated name=path pairs of explicit imports to use.
    -mock_names string
        Comma-separated interfaceName=mockName pairs of explicit mock names to use. Mock names default to 'Mock'+ interfaceName suffix.
    -package string
        Package of the generated code; defaults to the package of the input with a 'mock_' prefix.
    -prog_only
        (reflect mode) Only generate the reflection program; write it to stdout and exit.
    -self_package string
        The full package import path for the generated code. The purpose of this flag is to prevent import cycles in the generated code by trying to include its own package. This can happen if the mock's package is set to one of its inputs (usually the main one) and the output is stdio so mockgen cannot detect the final output package. Setting this flag will then tell mockgen which import to exclude.
  -source string
        (source mode) Input Go source file; enables source mode.
  -version
        Print version.
  -write_package_comment
        Writes package documentation comment (godoc) if true. (default true)

Mockgen gives us 2 ways to generate mocks. The source mode will generate mock interfaces from a single source file.

Things would be more complicated if this source file imports packages from other files, which is often the case when we work on a real project.

In this case, it’s better to use the reflect mode, where we only need to provide the name of the package and the interface, and let mockgen use reflection to automatically figure out what to do.

OK so I’m gonna run:



❯ mockgen github.com/techschool/simplebank/db/sqlc Store

The first argument is an import path to the Store interface. It's basically the simple bank module name github.com/techschool/simplebank followed by /db/sqlc because our Store interface is defined inside the db/sqlc folder.

The second argument we need to pass in this command is the name of the interface, which is Store in this case.

We should also specify the destination of the generated output file. Otherwise, mockgen will write the generated codes to stdout by default. So let’s use the -destination option to tell it to write the mock store codes to db/mock/store.go file:



❯ mockgen -destination db/mock/store.go github.com/techschool/simplebank/db/sqlc Store

Then press enter to run this command.

Now get back to visual studio code, we can see that a new file store.go is generated inside the db/mock folder:



// Code generated by MockGen. DO NOT EDIT.
// Source: github.com/techschool/simplebank/db/sqlc (interfaces: Store)

// Package mock_sqlc is a generated GoMock package.
package mock_sqlc

import (
    context "context"
    gomock "github.com/golang/mock/gomock"
    db "github.com/techschool/simplebank/db/sqlc"
    reflect "reflect"
)

// MockStore is a mock of Store interface
type MockStore struct {
    ctrl     *gomock.Controller
    recorder *MockStoreMockRecorder
}

// MockStoreMockRecorder is the mock recorder for MockStore
type MockStoreMockRecorder struct {
    mock *MockStore
}

// NewMockStore creates a new mock instance
func NewMockStore(ctrl *gomock.Controller) *MockStore {
    mock := &MockStore{ctrl: ctrl}
    mock.recorder = &MockStoreMockRecorder{mock}
    return mock
}

// EXPECT returns an object that allows the caller to indicate expected use
func (m *MockStore) EXPECT() *MockStoreMockRecorder {
    return m.recorder
}

...

In this file, there are 2 important struct: MockStore and MockStoreMockRecorder.

MockStore is the struct that implements all required functions of the Store interface. For example, here’s the AddAccountBalance() function of the MockStore, which takes a context and an AddAccountBalanceParams as input and returns an Account or an error:



// AddAccountBalance mocks base method
func (m *MockStore) AddAccountBalance(arg0 context.Context, arg1 db.AddAccountBalanceParams) (db.Account, error) {
    m.ctrl.T.Helper()
    ret := m.ctrl.Call(m, "AddAccountBalance", arg0, arg1)
    ret0, _ := ret[0].(db.Account)
    ret1, _ := ret[1].(error)
    return ret0, ret1
}

The MockStoreMockRecorder also has a function with the same name and the same number of arguments. However, the types of these arguments are different. They’re just general interface type:



// AddAccountBalance indicates an expected call of AddAccountBalance
func (mr *MockStoreMockRecorder) AddAccountBalance(arg0, arg1 interface{}) *gomock.Call {
    mr.mock.ctrl.T.Helper()
    return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AddAccountBalance", reflect.TypeOf((*MockStore)(nil).AddAccountBalance), arg0, arg1)
}

Later we will see how this function is used to build stubs. The idea is: we can specify how many times the AddAccountBalance() function should be called, and with what values of the arguments.

All other functions of the Store interface are generated in the same manner.

Note that the current package name that gomock generated for us is mock_sqlc, which doesn’t look very idiomatic, so I want to change it to something else, such as mockdb.

We can instruct mockgen to do that using the -package option. All we have to do is to add -package, followed by mockdb to this command:



❯ mockgen -package mockdb -destination db/mock/store.go github.com/techschool/simplebank/db/sqlc Store

Then now in the code, the package name has been changed to mockdb as we wanted:



// Code generated by MockGen. DO NOT EDIT.
// Source: github.com/techschool/simplebank/db/sqlc (interfaces: Store)

// Package mockdb is a generated GoMock package.
package mockdb

import (
    context "context"
    gomock "github.com/golang/mock/gomock"
    db "github.com/techschool/simplebank/db/sqlc"
    reflect "reflect"
)

// MockStore is a mock of Store interface
type MockStore struct {
    ctrl     *gomock.Controller
    recorder *MockStoreMockRecorder
}

// MockStoreMockRecorder is the mock recorder for MockStore
type MockStoreMockRecorder struct {
    mock *MockStore
}

...

Alright, before start writing API tests using the new generated MockStore, I’m gonna add a new mock command to the Makefile so that we can easily regenerate the code whenever we want.



...

mock:
    mockgen -package mockdb -destination db/mock/store.go github.com/techschool/simplebank/db/sqlc Store

.PHONY: postgres createdb dropdb migrateup migratedown sqlc test server mock

Now whenever we want to regenerate the mock store, we can simple run make mock in the terminal.

Write unit test for Get Account API

OK, now with the generated MockStore, we can start writing test for our APIs.

I’m gonna create a new file account_test.go inside the api package.

There are several API to mange bank accounts in our application. But for this lecture, we will only write tests for the most important one: Get Account API. You can easily based on that to write tests for other APIs if you want.

In the api/account_test.go file, I will define a new function TestGetAccountAPI() with the testing.T input parameter.



func TestGetAccountAPI(t *testing.T) {
}

In order to test this API, we need to have an account first. So let’s write a separate function to generate a random account.

It will return a db.Account object, where ID is a random integer between 1 and 1000, Owner is util.RandomOwner(), Balance is util.RandomMoney(), and Currency is util.RandomCurrency().



func randomAccount() db.Account {
    return db.Account{
        ID:       util.RandomInt(1, 1000),
        Owner:    util.RandomOwner(),
        Balance:  util.RandomMoney(),
        Currency: util.RandomCurrency(),
    }
}

Now go back to the test, we call randomAccount() function to create a new account.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()
}

Next we need to create a new mock store using this mockdb.NewMockStore() generated function. It expects a gomock.Controller object as input, so we have to create this controller by calling gomock.NewController and pass in the testing.T object.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()
}

We should defer calling Finish method of this controller. This is very important because it will check to see if all methods that were expected to be called were called.

We will see how it works in a moment. For now, let’s create a new store by calling mockdb.NewMockStore() with this input controller.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    store := mockdb.NewMockStore(ctrl)
}

The next step is to build the stubs for this mock store. In this case, we only care about the GetAccount() method, since it’s the only method that should be called by the Get Account API handler.

So let’s build stub for this method by calling store.EXPECT().GetAccount(). This function expects 2 input arguments of type general interface.

Why 2 input arguments? That’s because the GetAccount() method of our Store interface requires 2 input parameters: a context and an account ID.



type Querier interface {
    GetAccount(ctx context.Context, id int64) (Account, error)
    ...
}

Thus for this stub definition, we have to specify what values of these 2 parameters we expect this function to be called with.

The first context argument could be any value, so we use gomock.Any() matcher for it. The second argument should equal to the ID of the random account we created above. So we use this matcher: gomock.Eq() and pass the account.ID to it.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    store := mockdb.NewMockStore(ctrl)
    store.EXPECT().
        GetAccount(gomock.Any(), gomock.Eq(account.ID)).
        Times(1).
        Return(account, nil)
}

Now this stub definition can be translated as: I expect the GetAccount() function of the store to be called with any context and this specific account ID arguments.

We can also specify how many times this function should be called using the Times() function. Here Times(1) means we expect this function to be called exactly 1 time.

More than that, we can use the Return() function to tell gomock to return some specific values whenever the GetAccount() function is called. For example, in this case, we want it to return the account object and a nil error.

Note that the input arguments of this Return() function should match with the return values of the GetAccount function as defined in the Querier interface.

Alright, now the stub for our mock Store is built. We can use it to start the test HTTP server and send GetAccount request. Let’s create a server by calling NewServer() function with the mock store.



func TestGetAccountAPI(t *testing.T) {
    ...

    server := NewServer(store)
    recorder := httptest.NewRecorder()
}

For testing an HTTP API in Go, we don’t have to start a real HTTP server. Instead, we can just use the recording feature of the httptest package to record the response of the API request. So here we call httptest.NewRecorder() to create a new ResponseRecorder.

Next we will declare the url path of the API we want to call, which should be /accounts/{ID of the account we want to get}.

Then we create a new HTTP Request with method GET to that URL. And since it’s a GET request, we can use nil for the request body.



func TestGetAccountAPI(t *testing.T) {
    ...

    server := NewServer(store)
    recorder := httptest.NewRecorder()

    url := fmt.Sprintf("/accounts/%d", tc.accountID)
    request, err := http.NewRequest(http.MethodGet, url, nil)
    require.NoError(t, err)
}

This http.NewRequest function will return a request object or an error. We require no errors to be returned.

Then we call server.router.ServeHTTP() function with the created recorder and request objects.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    store := mockdb.NewMockStore(ctrl)
    store.EXPECT().
        GetAccount(gomock.Any(), gomock.Eq(account.ID)).
        Times(1).
        Return(account, nil)

    server := NewServer(store)
    recorder := httptest.NewRecorder()

    url := fmt.Sprintf("/accounts/%d", tc.accountID)
    request, err := http.NewRequest(http.MethodGet, url, nil)
    require.NoError(t, err)

    server.router.ServeHTTP(recorder, request)
    require.Equal(t, http.StatusOK, recorder.Code)
}

Basically, this will send our API request through the server router and record its response in the recorder. All we need to do is to check that response.

The simplest thing we can check is the HTTP status code. In the happy case, it should be http.StatusOK. This status code is recorded in the Code field of the recorder.

And that’s it! Let’s run the test.

It passed. Awesome!

Now I’m gonna show you what will happen if in the getAccount() handler function of api/account.go file we don’t call the store.GetAccount function. Let’s comment out this block of code and just set account to be an empty object.



func (server *Server) getAccount(ctx *gin.Context) {
    var req getAccountRequest
    if err := ctx.ShouldBindUri(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    // account, err := server.store.GetAccount(ctx, req.ID)
    // if err != nil {
    //     if err == sql.ErrNoRows {
    //         ctx.JSON(http.StatusNotFound, errorResponse(err))
    //         return
    //     }

    //     ctx.JSON(http.StatusInternalServerError, errorResponse(err))
    //     return
    // }
    account := db.Account{}

    ctx.JSON(http.StatusOK, account)
}

Save this file and rerun the unit test.

This time, the test failed. And the reason is due to a missing call to the store.GetAccount function. We expect that function to be called exactly once, but in the implementation, it’s not getting called.

So now you know one power of the gomock package. It makes writing unit tests so easy and saves us tons of time implementing the mock interface.

Now, what if we want to check more than just the HTTP status code? To make the test more robust, we should check the response body as well.

The response body is stored in the recorder.Body field, which is in fact just a bytes.Buffer pointer.

We expect it to match the account that we generated at the top of the test. So I’m gonna write a new function: requireBodyMatchAccount() for this purpose.

It will have 3 input arguments: the testing.T, the response body of type byte.Buffer pointer, and the account object to compare.



func requireBodyMatchAccount(t *testing.T, body *bytes.Buffer, account db.Account) {
    data, err := ioutil.ReadAll(body)
    require.NoError(t, err)

    var gotAccount db.Account
    err = json.Unmarshal(data, &gotAccount)
    require.NoError(t, err)
    require.Equal(t, account, gotAccount)
}

First we call ioutil.ReadAll() to read all data from the response body and store it in a data variable. We require no errors to be returned.

Then we declare a new gotAccount variable to store the account object we got from the response body data.

Then we call json.Unmarshal to unmarshal the data to the gotAccount object. Require no errors, then require the gotAccount to be equal to the input account.

And we’re done. Now let’s go back to the unit test and call requireBodyMatchAccount function with the testing.T, the recorder.Body, and the generated account as input arguments.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    store := mockdb.NewMockStore(ctrl)
    store.EXPECT().
        GetAccount(gomock.Any(), gomock.Eq(account.ID)).
        Times(1).
        Return(account, nil)

    server := NewServer(store)
    recorder := httptest.NewRecorder()

    url := fmt.Sprintf("/accounts/%d", tc.accountID)
    request, err := http.NewRequest(http.MethodGet, url, nil)
    require.NoError(t, err)

    server.router.ServeHTTP(recorder, request)
    require.Equal(t, http.StatusOK, recorder.Code)
    requireBodyMatchAccount(t, recorder.Body, account)
}

Then rerun the test.

It passed. Excellent!

OK, so the GetAccount API unit test is working very well. But it only covers the happy case for now.

Next, I’m gonna show you how to transform this test into a table-driven test set to cover all possible scenarios of the GetAccount API and to get 100% coverage.

Achieve 100% coverage

First, we need to declare a list of test cases. I’m gonna use an anonymous class to store the test data.

Each test case will have a unique name to separate it from others. Then there should be an account ID that we want to get.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    testCases := []struct {
        name          string
        accountID     int64
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(t *testing.T, recoder *httptest.ResponseRecorder)
    }{
        // TODO: add test data
    }
    ...
}

Moreover, the GetAccount stub for each scenario will be built differently, so here I have a buildStubs field, which is actually a function that takes a mock store as input. We can use this mock store to build the stub that suits the purpose of each test case.

Similarly, we have a checkResponse function to check the output of the API. It has 2 input arguments: a testing.T, and a httptest.ResponseRecorder object.

Now with this struct definition, let’s add the first scenario for the happy case.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    testCases := []struct {
        name          string
        accountID     int64
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(t *testing.T, recoder *httptest.ResponseRecorder)
    }{
        {
            name:      "OK",
            accountID: account.ID,
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    GetAccount(gomock.Any(), gomock.Eq(account.ID)).
                    Times(1).
                    Return(account, nil)
            },
            checkResponse: func(t *testing.T, recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchAccount(t, recorder.Body, account)
            },
        },
    }
    ...
}

Its name is "OK". The account ID should be account.ID. Next, for the buildStubs function, I’m gonna copy its signature here. Then move the store.EXPECT command into this function.

Similar for the checkResponse function. Let’s copy its signature. Then move the 2 require commands into it.

We will add more cases to this list later. For now, let’s refactor the code a bit to make it work for multiple scenarios.

We use a simple for loop to iterate through the list of test cases. Then inside the loop, we declare new tc variable to store the data of current test case.

We gonna run each case as a separate sub-test of this unit test, so let’s call t.Run() function, pass in the name of this test case, and a function that takes testing.T object as input. Then I’m gonna move all of these statements into that function.



func TestGetAccountAPI(t *testing.T) {
    ...

    for i := range testCases {
        tc := testCases[i]

        t.Run(tc.name, func(t *testing.T) {
            ctrl := gomock.NewController(t)
            defer ctrl.Finish()

            store := mockdb.NewMockStore(ctrl)
            tc.buildStubs(store)

            server := NewServer(store)
            recorder := httptest.NewRecorder()

            url := fmt.Sprintf("/accounts/%d", tc.accountID)
            request, err := http.NewRequest(http.MethodGet, url, nil)
            require.NoError(t, err)

            server.router.ServeHTTP(recorder, request)
            tc.checkResponse(t, recorder)
        })
    }
}

Note that the url should be created with the tc.accountID so that it will use the account ID defined for each test case.

We call tc.buildStubs() function with the mock store before sending the request, and finally call tc.checkResponse() function at the end to verify the result.

OK, let’s rerun the test to make sure our happy case still works.

Yee, it passed! So now it’s time to add more cases.

I’m gonna duplicate the happy case's test data. The second case we want to test is when the account is not found. So its name should be "NotFound".

We can use the same accountID here because mock store is separated for each test case. But we need to change our buildStubs function a bit.



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    testCases := []struct {
        name          string
        accountID     int64
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(t *testing.T, recoder *httptest.ResponseRecorder)
    }{
        {
            name:      "OK",
            accountID: account.ID,
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    GetAccount(gomock.Any(), gomock.Eq(account.ID)).
                    Times(1).
                    Return(account, nil)
            },
            checkResponse: func(t *testing.T, recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusOK, recorder.Code)
                requireBodyMatchAccount(t, recorder.Body, account)
            },
        },
        {
            name:      "NotFound",
            accountID: account.ID,
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    GetAccount(gomock.Any(), gomock.Eq(account.ID)).
                    Times(1).
                    Return(db.Account{}, sql.ErrNoRows)
            },
            checkResponse: func(t *testing.T, recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusNotFound, recorder.Code)
            },
        },
    }

    ...
}

Here instead of returning this specific account, we should return an empty Account{} object together with a sql.ErrNoRows error. That’s because in the real implementation of the Store that connects to Postgres, the db/sql package will return this error if no account is found when executing the SELECT query.

We also have to change the checkResponse function, because in this case, we expect the server to return http.StatusNotFound instead. And since the account is not found, we can remove the requireBodyMatchAccount call.

Alright, let’s run the test again.

Cool! Both tests passed.

Let’s run the whole package test to see the code coverge.

In the account.go file, we can see that this getAccount handler is not 100% covered.

Right now only the not found case and the successful case are covered. We still have to test 2 more cases: InternalServerError and BadRequest.

Once again, I’m gonna duplicate the test data, change its name to "InternalError".



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    testCases := []struct {
        name          string
        accountID     int64
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(t *testing.T, recoder *httptest.ResponseRecorder)
    }{
        ...
        {
            name:      "InternalError",
            accountID: account.ID,
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    GetAccount(gomock.Any(), gomock.Eq(account.ID)).
                    Times(1).
                    Return(db.Account{}, sql.ErrConnDone)
            },
            checkResponse: func(t *testing.T, recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusInternalServerError, recorder.Code)
            },
        },
    }

    ...
}

Now in the buildStubs function, instead of returning sql.ErrNoRows, I return sql.ErrConnDone, which basically is one possible error that the db/sql package can return when a query is run on a connection that has already been returned to the connection pool.

In this case, it should be considered an internal error, so in the checkResponse function, we must require the recorder.Code to be equal to http.StatusInternalServerError.

Let’s rerun the package test.

All passed. And we can now see that the InternalServerError branch in the code is now covered.

The last scenario we should test is BadRequest, which means the client has sent some invalid parameters to this API.

To reproduce this scenario, we will use an invalid account ID that doesn’t satisfy this binding condition.

So I’m gonna go back to the test file, duplicate this test data one more time, change its name to InvalidID, and update this accountID to 0, which is an invalid value because the minimum ID should be 1.

In this case, we should change the second parameter of the GetAccount function call to gomock.Any().



func TestGetAccountAPI(t *testing.T) {
    account := randomAccount()

    testCases := []struct {
        name          string
        accountID     int64
        buildStubs    func(store *mockdb.MockStore)
        checkResponse func(t *testing.T, recoder *httptest.ResponseRecorder)
    }{
        ...
        {
            name:      "InvalidID",
            accountID: 0,
            buildStubs: func(store *mockdb.MockStore) {
                store.EXPECT().
                    GetAccount(gomock.Any(), gomock.Any()).
                    Times(0)
            },
            checkResponse: func(t *testing.T, recorder *httptest.ResponseRecorder) {
                require.Equal(t, http.StatusBadRequest, recorder.Code)
            },
        },
    }

    ...
}

And since the ID is invalid, the GetAccount function should not be called by the handler. Therefore, we must update this to Times(0), and remove the Return function call.

For the checkResponse, we must change the status code to http.StatusBadRequest.

And that’s all. Let’s rerun the whole package tests!

All passed. Great! And looking at the code of the getAccount handler, we can see that it is 100% covered. So our goal is achieved!

However, the test log is currently containing too much information.

There are many duplicate debug logs written by Gin, which make it harder to read the test result.

The reason is that Gin is running in Debug mode by default. So let’s create a new main_test.go file inside the api package and config Gin to use Test mode instead.

The content of this file will be very similar to that of the main_test.go file in the db package, so I’m gonna copy this TestMain function, and paste it to our new file. Then let’s delete all statements of this function, except for the last one.

Now all we need to do is to call gin.SetMode to change it to gin.TestMode



func TestMain(m *testing.M) {
    gin.SetMode(gin.TestMode)
    os.Exit(m.Run())
}

And that’s it! We’re done. Now let’s go back to our test file and run the whole package tests.

All passed. And now the logs look much cleaner and easier to read than before.

Conclusion

OK, so today we have learned how to use gomock to generate mocks for our DB interface, and use it to write unit tests for the Get Account API to achieve 100% coverage. It really helps us write tests faster, easier, cleaner, safer, and much more robust.

You can apply this knowledge to write tests for other HTTP APIs in our simple bank project such as Create Account or Delete Account API.

I will push the code to Github so that you can have a reference in case you want to take a look.

Thanks a lot for reading and see you soon in the next lecture.

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Load config from file & environment variables in Golang with Viper

TECH SCHOOL — Mon, 02 Nov 2020 16:45:29 +0000

When developing and deploying a backend web application, we usually have to use different configurations for different environments such as development, testing, staging, and production.

Today we will learn how to use Viper to load configurations from file or environment variables.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Why file and environment variables

Reading values from file allows us to easily specify default configuration for local development and testing.

While reading values from environment variables will help us override the default settings when deploying our application to staging or production using docker containers.

Why Viper

Viper is a very popular Golang package for this purpose.

It can find, load, and unmarshal values from a config file.
It supports many types of files, such as JSON, TOML, YAML, ENV, or INI.
It can also read values from environment variables or command-line flags.
It gives us the ability to set or override default values.
Moreover, if you prefer to store your settings in a remote system such as Etcd or Consul, then you can use viper to read data from them directly.
It works for both unencrypted and encrypted values.
One more interesting thing about Viper is, it can watch for changes in the config file, and notify the application about it.
We can also use viper to save any config modification we made to the file.

A lot of useful features, right?

What will we do

In the current code of our simple bank project, we’re hard-coding some constants for the dbDriver, dbSource in the main_test.go file, and also one more constant for the serverAddress in the main.go file.



const (
    dbDriver      = "postgres"
    dbSource      = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
    serverAddress = "0.0.0.0:8080"
)

So in this tutorial, we will learn how to use Viper to read these configurations from file and environment variables.

Install Viper

OK, let’s start by installing Viper! Open your browser and search for golang viper.

Then open its Github page. Scroll down a bit, copy this go get command and run it in the terminal to install the package:



❯ go get github.com/spf13/viper

After this, in the go.mod file of our project, we can see Viper has been added as a dependency.

Create config file

Now I’m gonna create a new file app.env to store our config values for development. Then let’s copy these variables from main.go file and paste them to this config file. Since we’re using dot env format, we must change the way we declare these variables. It should be similar to how we declare environment variables:

Each variable should be declared on a separate line.
The variable's name is uppercase and its words are separated by an underscore.
The variable value is followed by the name after an equal symbol.



DB_DRIVER=postgres
DB_SOURCE=postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable
SERVER_ADDRESS=0.0.0.0:8080

And that’s it! This app.env file is now containing the default configurations for our local development environment. Next, we will use Viper to load this config file.

Load config file

Let’s create a new file config.go inside the util package.

Then declare a new type Config struct in this file. This Config struct will hold all configuration variables of the application that we read from file or environment variables. For now, we only have 3 variables:

First, the DBDriver of type string,
Second, the DBSource, also of type string.
And finally the ServerAddress of type string as well.



type Config struct {
    DBDriver      string `mapstructure:"DB_DRIVER"`
    DBSource      string `mapstructure:"DB_SOURCE"`
    ServerAddress string `mapstructure:"SERVER_ADDRESS"`
}

In order to get the value of the variables and store them in this struct, we need to use the unmarshaling feature of Viper. Viper uses the mapstructure package under the hood for unmarshaling values, so we use the mapstructure tags to specify the name of each config field.

In our case, we must use the exact name of each variable as being declared in the app.env. For example, the DBDriver’s tag name should be DB_DRIVER, the DBSource’s tag name should be DB_SOURCE, and similar for the ServerAddress, should be SERVER_ADDRESS.

OK, next I’m gonna define a new function LoadConfig(), which takes a path as input, and returns a config object or an error. This function will read configurations from a config file inside the path if it exists, or override their values with environment variables if they’re provided.



func LoadConfig(path string) (config Config, err error) {
    viper.AddConfigPath(path)
    viper.SetConfigName("app")
    viper.SetConfigType("env")

    ...
}

First, we call viper.AddConfigPath() to tell Viper the location of the config file. In this case, the location is given by the input path argument.

Next, we call viper.SetConfigName() to tell Viper to look for a config file with a specific name. Our config file is app.env, so its name is app.

We also tell Viper the type of the config file, which is env in this case, by calling viper.SetConfigFile() and pass in env. You can also use JSON, XML or any other format here if you want, just make sure your config file has the correct format and extension.

Now, besides reading configurations from file, we also want viper to read values from environment variables. So we call viper.AutomaticEnv() to tell viper to automatically override values that it has read from config file with the values of the corresponding environment variables if they exist.



// LoadConfig reads configuration from file or environment variables.
func LoadConfig(path string) (config Config, err error) {
    viper.AddConfigPath(path)
    viper.SetConfigName("app")
    viper.SetConfigType("env")

    viper.AutomaticEnv()

    err = viper.ReadInConfig()
    if err != nil {
        return
    }

    err = viper.Unmarshal(&config)
    return
}

After that, we call viper.ReadInConfig() to start reading config values. If error is not nil, then we simply return it.

Otherwise, we call viper.Unmarshal() to unmarshals the values into the target config object. And finally just return the config object and any error if it occurs.

So basically the load config function is completed. Now we can use it in the main.go file.

Use LoadConfig in the main function

Let’s remove all of the previous hardcoded values. Then in the main() function, let’s call util.LoadConfig() and pass in "." here, which means the current folder because our config file app.env is in the same location as this main.go file.

If there’s an error, then we just write a fatal log saying cannot load configuration. Else, we just change these variables to config.DBDriver, config.DBSource, and config.ServerAdress.



func main() {
    config, err := util.LoadConfig(".")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    conn, err := sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server := api.NewServer(store)

    err = server.Start(config.ServerAddress)
    if err != nil {
        log.Fatal("cannot start server:", err)
    }
}

And we’re done! Let’s try to run this server.



❯ make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (3 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (3 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccount-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8080

It’s working! The server is listening on localhost port 8080, just like what we specified in the app.env file.

Let’s open Postman and send an API request. I’m gonna call the list accounts API.

Oops, we’ve got a 500 Internal Server Error. It’s because our web server cannot connect to Postgres database on port 5432. Let’s open the terminal and check if Postgres is running or not:



❯ docker ps

There are no containers running. If we run docker ps -a, we can see that Postgres container has been exited. So we have to start it by running:



❯ docker start postgres12

OK, now the database is up and running. Let’s go back to Postman and send the request again.

This time, it’s successful. We’ve got a list of accounts here. So our code to load configurations from file is working well.

Let’s try overriding those configurations with environment variables. I will set the SERVER_ADDRESS variable to localhost port 8081 before calling make server.



❯ SERVER_ADDRESS=0.0.0.0:8081 make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (3 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (3 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccount-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8081

Here we can see that the server is now listening on port 8081 instead of 8080 as before. Now if we try to send the same API request in Postman to port 8080, we will get a connection refused error:

Only when we change this port to 8081, then the request will be successful:

So we can conclude that Viper has successfully overridden the values it read from the config file with environment variables. That’s very convenient when we want to deploy the application to different environments such as staging or production in the future.

Use LoadConfig in the test

Now before we finish, let’s update the main_test.go file to use the new LoadConfig() function.

First, remove all of the hard-coded constants. Then in the TestMain() function, we call util.LoadConfig().

But this time, the main_test.go file is inside the db/sqlc folder, while the app.env config file is at the root of the repository, so we must pass in this relative path: "../..". These 2 dots .. basically mean go to the parent folder.



func TestMain(m *testing.M) {
    config, err := util.LoadConfig("../..")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    testDB, err = sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    testQueries = New(testDB)

    os.Exit(m.Run())
}

If the error is not nil, then we just write a fatal log. Otherwise, we should change these 2 values to config.DBDriver and config.DBSource.

And that’s it! We’re done. Let’s run the whole package test.

All passed! And that wraps up this lecture about reading configuration from file and environment variables.

I highly recommend you to check out the documentation and try some other interesting features of viper such as live watching, or reading from remote system.

Happy coding and I will see you in the next lecture!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Implement RESTful HTTP API in Go using Gin

TECH SCHOOL — Mon, 26 Oct 2020 10:55:34 +0000

Hello and welcome back to the backend master class.

So far we have learned a lot about working with database in Go. Now it’s time to learn how to implement some RESTful HTTP APIs that will allow frontend clients to interact with our banking service backend.

Here's:

Link to the full series playlist on Youtube
And its Github repository

Go web frameworks and HTTP routers

Although we can use the standard net/http package to implement those APIs, It will be much easier to just take advantage of some existing web frameworks.

Here are some of the most popular golang web frameworks sorted by their number of Github stars:

They offer a wide range number of features such as routing, parameter binding, validation, middleware, and some of them even have a built-in ORM.

If you prefer a lightweight package with only routing feature, then here are some of the most popular HTTP routers for golang:

For this tutorial, I’m gonna use the most popular framework: Gin

Install Gin

Let’s open the browser and search for golang gin, then open its Github page. Scroll down a bit and select Installation.

Let’s copy this go get command, and run it in the terminal to install the package:



❯ go get -u github.com/gin-gonic/gin

After this, in the go.mod file of our simple bank project, we can see that gin is added as a new dependency together with some other packages that it uses.

Define server struct

Now I’m gonna create a new folder called api. Then create a new file server.go inside it. This is where we implement our HTTP API server.

First let’s define a new Server struct. This Server will serves all HTTP requests for our banking service. It will have 2 fields:

The first one is a db.Store that we have implemented in previous lectures. It will allow us to interact with the database when processing API requests from clients.
The second field is a router of type gin.Engine. This router will help us send each API request to the correct handler for processing.



type Server struct {
    store  *db.Store
    router *gin.Engine
}

Now let’s add a function NewServer, which takes a db.Store as input, and return a Server. This function will create a new Server instance, and setup all HTTP API routes for our service on that server.

First, we create a new Server object with the input store. Then we create a new router by calling gin.Default(). We will add routes to this router in a moment. After this step, we will assign the router object to server.router and return the server.



func NewServer(store *db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    // TODO: add routes to router

    server.router = router
    return server
}

Now let’s add the first API route to create a new account. It’s gonna use POST method, so we call router.POST.

We must pass in a path for the route, which is /accounts in this case, and then one or multiple handler functions. If you pass in multiple functions, then the last one should be the real handler, and all other functions should be middlewares.



func NewServer(store *db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    router.POST("/accounts", server.createAccount)

    server.router = router
    return server
}

For now, we don’t have any middlewares, so I just pass in 1 handler: server.createAccount. This is a method of the Server struct that we need to implement. The reason it needs to be a method of the Server struct is because we have to get access to the store object in order to save new account to the database.

Implement create account API

I’m gonna implement server.createAccount method in a new file account.go inside the api folder. Here we declare a function with a server pointer receiver. Its name is createAccount, and it should take a gin.Context object as input.



func (server *Server) createAccount(ctx *gin.Context) {
    ...
}

Why does it have this function signature? Let’s look at this router.POST function of Gin:

Here we can see that the HandlerFunc is declared as a function with a Context input. Basically, when using Gin, everything we do inside a handler will involve this context object. It provides a lot of convenient methods to read input parameters and write out responses.

Alright, now let’s declare a new struct to store the create account request. It will have several fields, similar to the createAccountParams from account.sql.go that we used in the database in previous lecture:



type CreateAccountParams struct {
    Owner    string `json:"owner"`
    Balance  int64  `json:"balance"`
    Currency string `json:"currency"`
}

So I’m gonna copy these fields and paste them to our createAccountRequest struct. When a new account is created, its initial balance should always be 0, so we can remove the balance field. We only allow clients to specify the owner’s name and the currency of the account. We’re gonna get these input parameters from the body of the HTTP request, Which is a JSON object, so I’m gonna keep the JSON tags.



type createAccountRequest struct {
    Owner    string `json:"owner"`
    Currency string `json:"currency"`
}

func (server *Server) createAccount(ctx *gin.Context) {
    ...
}

Now whenever we get input data from clients, it’s always a good idea to validate them, because who knows, clients might send some invalid data that we don’t want to store in our database.

Lucky for us, Gin uses a validator package internally to perform data validation automatically under the hood. For example, we can use a binding tag to tell Gin that the field is required. And later, we call the ShouldBindJSON function to parse the input data from HTTP request body, and Gin will validate the output object to make sure it satisfy the conditions we specified in the binding tag.

I’m gonna add a binding required tag to both the owner and the currency field. Moreover, let’s say our bank only supports 2 types of currency for now: USD and EUR. So how can we tell gin to check that for us? Well, we can use the oneof condition for this purpose:



type createAccountRequest struct {
    Owner    string `json:"owner" binding:"required"`
    Currency string `json:"currency" binding:"required,oneof=USD EUR"`
}

We use a comma to separate multiple conditions, and a space to separate the possible values for the oneof condition.

Alright, now in the createAccount function, we declare a new req variable of type createAccountRequest. Then we call ctx.ShouldBindJSON() function, and pass in this req object. This function will return an error.

If the error is not nil, then it means that the client has provided invalid data. So we should send a 400 Bad Request response to the client. To do that, we just call ctx.JSON() function to send a JSON response.

The first argument is an HTTP status code, which in this case should be http.StatusBadRequest. The second argument is the JSON object that we want to send to the client. Here we just want to send the error, so we will need a function to convert this error into a key-value object so that Gin can serialize it to JSON before returning to the client.



func (server *Server) createAccount(ctx *gin.Context) {
    var req createAccountRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    ...
}

We’re gonna use this errorResponse() function a lot in our code later, and it can be used for other handlers as well, not just for account handlers, so I will implement it in the server.go file.

This function will take an error as input, and it will return a gin.H object, which is in fact just a shortcut for map[string]interface{}. So we can store whatever key-value data that we want in it.

For now let’s just return gin.H with only 1 key: error, and its value is the error message. Later we might check the error type and convert it to a better format if we want.



func errorResponse(err error) gin.H {
    return gin.H{"error": err.Error()}
}

Now let’s go back to the createAccount handler. In case the input data is valid, there will be no errors. So we just go ahead to insert a new account into the database.

First we declare a CreateAccountParams object, where Owner is req.Owner, Currency is req.Currency, and Balance is 0. Then we call server.store.CreateAccount(), pass in the input context, and the argument. This function will return the created account and an error.



func (server *Server) createAccount(ctx *gin.Context) {
    var req createAccountRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    arg := db.CreateAccountParams{
        Owner:    req.Owner,
        Currency: req.Currency,
        Balance:  0,
    }

    account, err := server.store.CreateAccount(ctx, arg)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, account)
}

If the error is not nil, then there must be some internal issue when trying to insert to the database. Thus, we will return a 500 Internal Server Error status code to the client. We also reuse the errorResponse() function to send the error to the client, then return immediately.

If no errors occur, then the account is successfully created. We just send a 200 OK status code, and the created account object to the client. And that’s it! The createAccount handler is done.

Start HTTP server

Next, we have to add some more code to run the HTTP server. I’m gonna add a new Start() function to our Server struct. This function will take an address as input and return an error. Its role is to run the HTTP server on the input address to start listening for API requests.



func (server *Server) Start(address string) error {
    return server.router.Run(address)
}

Gin already provided a function in the router to perform this action, so all we need to do is calling server.router.Run(), and pass in the server address.

Note that the server.router field is private, so it cannot be accessed from outside of this api package. That’s one of the reasons we have this public Start() function. For now, it has just 1 single command, but later, we might want to add some graceful shutdown logics in this function as well.

OK, now let’s create an entry point for our server in the main.go file at the root of this repository. The package name should be main, and it should have a main() function.

In order to create a Server, we need to connect to the database and create a Store first. It’s gonna be very similar to the code that we’ve written before in the main_test.go file.

So I’m gonna copy these constants for the dbDriver and dbSource, paste them to the top of our main.go file. Then also copy the block of code that establishes connections to the database and paste it inside the main function.

With this connection, we can create a new store using db.NewStore() function. Then we create a new server by calling api.NewServer() and pass in the store.



const (
    dbDriver      = "postgres"
    dbSource      = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
)

func main() {
    conn, err := sql.Open(dbDriver, dbSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server := api.NewServer(store)

    ...
}

To start the server, we just need to call server.Start() and pass in the server address. For now, I’m just gonna declare it as a constant: localhost, port 8080. In the future, we will refactor the code to load all of these configurations from environment variables or a setting file. In case some error occurs when starting the server, we just write a fatal log, saying cannot start server.

One last but very important thing we must do is to add a blank import for lib/pq driver. Without this, our code would not be able to talk to the database.



package main

import (
    "database/sql"
    "log"

    _ "github.com/lib/pq"
    "github.com/techschool/simplebank/api"
    db "github.com/techschool/simplebank/db/sqlc"
)

const (
    dbDriver      = "postgres"
    dbSource      = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
    serverAddress = "0.0.0.0:8080"
)

func main() {
    conn, err := sql.Open(dbDriver, dbSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server := api.NewServer(store)

    err = server.Start(serverAddress)
    if err != nil {
        log.Fatal("cannot start server:", err)
    }
}

Alright, so now the main entry for our server is completed. Let’s add a new make command to the Makefile to run it.

I’m gonna call it make server. And it should execute this go run main.go command. Let’s add server to the phony list.



...

server:
    go run main.go

.PHONY: postgres createdb dropdb migrateup migratedown sqlc test server

Then open the terminal and run:



make server

Voila, the server is up and running. It’s listening and serving HTTP requests on port 8080.

Test create account API with Postman

Now I’m gonna use Postman to test the create account API.

Let’s add a new request, select the POST method, fill in the URL, which is http://localhost:8080/accounts.

The parameters should be sent via a JSON body, so let’s select the Body tab, choose Raw, and select JSON format. We have to add 2 input fields: the owner’s name, I will use my name here, and a currency, let’s say USD.



{
    "owner": "Quang Pham",
    "currency": "USD"
}

OK, then click Send.

Yee, it’s successful. We’ve got a 200 OK status code, and the created account object. It has ID = 1, balance = 0, with the correct owner’s name and currency.

Now let’s try to send some invalid data to see what will happen. I’m gonna set both fields to empty string, and click Send.



{
    "owner": "",
    "currency": ""
}

This time, we’ve got 400 Bad Request, and an error saying the fields are required. This error message looks quite hard to read because it combines 2 validation errors of the 2 fields together. This is something we might want to improve in the future.

Next I’m gonna try to use an invalid currency code, such as xyz.



{
    "owner": "Quang Pham",
    "currency": "xyz"
}

This time, we also get 400 Bad Request status code, but the error message is different. It say the validation failed on the oneof tag, which is exactly what we want, because in the code we only allow 2 possible values for the currency: USD and EUR.

It’s really great how Gin has handled all the input binding and validation for us with just a few lines of code. It also prints out a nice form of request logs, which is very easy to read for human eyes.

Implement get account API

Alright, next we’re gonna add an API to get a specific account by ID. It would be very similar to the create account API, so I will duplicate this routing statement:



func NewServer(store *db.Store) *Server {
    ...

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)

    ...
}

Here instead of POST, we will use GET method. And this path should include the id of the account we want to get /accounts/:id. Note that we have a colon before the id. It’s how we tell Gin that id is a URI parameter.

Then we have to implement a new getAccount handler on the Server struct. Let’s move to the account.go file to do so. Similar as before, we declare a new struct called getAccountRequest to store the input parameters. It will have an ID field of type int64.

Now, since ID is a URI parameter, we cannot get it from the request body as before. Instead, we use the uri tag to tell Gin the name of the URI parameter:



type getAccountRequest struct {
    ID int64 `uri:"id" binding:"required,min=1"`
}

We add a binding condition that this ID is a required field. Also, we don’t want client to send an invalid ID, such as a negative number. To tell Gin about this, we can use the min condition. In this case, let’s set min = 1, because it’s the smallest possible value of account ID.

OK, now in the server.getAccount handler, we will do similar as before. First we declare a new req variable of type getAccountRequest. Then here instead of ShouldBindJSON, we should call ShouldBindUri.

If there’s an error, we just return a 400 Bad Request status code. Otherwise, we call server.store.GetAccount() to get the account with ID equals to req.ID. This function will return an account and an error.



func (server *Server) getAccount(ctx *gin.Context) {
    var req getAccountRequest
    if err := ctx.ShouldBindUri(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    account, err := server.store.GetAccount(ctx, req.ID)
    if err != nil {
        if err == sql.ErrNoRows {
            ctx.JSON(http.StatusNotFound, errorResponse(err))
            return
        }

        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, account)
}

If error is not nil, then there are 2 possible scenarios.

The first scenario is some internal error when querying data from the database. In this case, we just return 500 Internal Server Error status code to the client.
The second scenario is when the account with that specific input ID doesn’t exist. In that case, the error we got should be a sql.ErrNoRows. So we just check it here, and if it’s really the case, we simply send a 404 Not Found status code to the client, and return.

If everything goes well and there’s no error, we just return a 200 OK status code and the account to the client. And that’s it! Our getAccount API is completed.

Test get account API with Postman

Let’s restart the server and open Postman to test it.

Let’s add a new request with method GET, and the URL is http://localhost:8080/accounts/1. We add a /1 at the end because we want to get the account with ID = 1. Now click send:

The request is successful, and we’ve got a 200 OK status code together with the found account. This is exactly the account that we’ve created before.

Now let’s try to get an account that doesn’t exist. I’m gonna change the ID to 100: http://localhost:8080/accounts/100, and click send again.

This time we’ve got a 404 Not Found status code, and an error: sql no rows in result set. Exactly what we expected.

Let’s try one more time with a negative ID: http://localhost:8080/accounts/-1

Now we got a 400 Bad Request status code with an error message about the failed validation.

Alright, so our getAccount API is working well.

Implement list account API

Next step, I’m gonna show you how to implement a list account API with pagination.

The number of accounts stored in our database can grow to a very big number over time. Therefore, we should not query and return all of them in a single API call. The idea of pagination is to divide the records into multiple pages of small size, so that the client can retrieve only 1 page per API request.

This API is a bit different because we will not get input parameters from request body or URI, but we will get them from query string instead. Here’s an example of the request:

We have a page_id param, which is the index number of the page we want to get, starting from page 1. And a page_size param, which is the maximum number of records can be returned in 1 page.

As you can see, the page_id and page_size are added to the request URL after a question mark: http://localhost:8080/accounts?page_id=1&page_size=5. That’s why they’re called query parameters, and not URI parameter like the account ID in the get account request.

OK, now let’s go back to our code. I’m gonna add a new route with the same GET method. But this time, the path should be /accounts only, since we’re gonna get the parameters from the query. The handler’s name should be listAccount.



func NewServer(store *db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccount)

    server.router = router
    return server
}

Alright, let’s open the account.go file to implement this server.listAccount function. It’s very similar to the server.getAccount handler, so I’m gonna duplicate it. Then change the struct name to listAccountRequest.

This struct should store 2 parameters, PageID and PageSize. Now note that we’re not getting these parameters from uri, but from query string instead, so we cannot use the uri tag. Instead, we should use form tag.



type listAccountRequest struct {
    PageID   int32 `form:"page_id" binding:"required,min=1"`
    PageSize int32 `form:"page_size" binding:"required,min=5,max=10"`
    }

Both parameters are required and the minimum PageID should be 1. For the PageSize, let’s say we don’t want it to be too big or too small, so I set its minimum constraint to be 5 records, and maximum constraint to be 10 records.

OK, now the server.listAccount handler function should be implemented like this:



func (server *Server) listAccount(ctx *gin.Context) {
    var req listAccountRequest
    if err := ctx.ShouldBindQuery(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    arg := db.ListAccountsParams{
        Limit:  req.PageSize,
        Offset: (req.PageID - 1) * req.PageSize,
    }

    accounts, err := server.store.ListAccounts(ctx, arg)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, accounts)
}

The req variable’s type should be listAccountRequest. Then we use another binding function: ShouldBindQuery to tell Gin to get data from query string.

If an error occurs, we just return a 400 Bad Request status. Else, we call server.store.ListAccounts() to query a page of account records from the database. This function requires a ListAccountsParams as input, where we have to provide values for 2 fields: Limit and Offset.

Limit is simply the req.PageSize. While Offset is the number of records that the database should skip, wo we have to calculate it from the page id and page size using this formula: (req.PageID - 1) * req.PageSize

The ListAccounts function returns a list of accounts and an error. If an error occurs, then we just need to return 500 Internal Server Error to the client. Otherwise, we send a 200 OK status code with the output accounts list.

And that’s it, the ListAccount API is done.

Test list account API with Postman

Let’s restart the server then open Postman to test this request.

It’s successful, but we’ve got only 1 account on the list. That’s because our database is quite empty at the moment. We just created only 1 single account. Let’s run the database tests that we’ve written in previous lectures to have more random data.



❯ make test

OK, now we should have a lot of accounts in our database. Let’s resend this API request.

Voila, now the returned list has exactly 5 accounts as expected. The account with ID 5 is not here because I think it’s got deleted in the test. We’ve got the account with ID 6 here instead.

Let’s try to get the second page.

Cool, now we get the next 5 accounts with ID from 7 to 11. So it’s working very well.

I’m gonna try one more time to get a page that doesn’t exist, let’s say page 100.

OK, so now we’ve got a null response body. Although it’s technically correct, I think it would be better if the server returns an empty list in this case. So let’s do that!

Return empty list instead of null

Here in the account.sql.go file that sqlc has generated for us:



func (q *Queries) ListAccounts(ctx context.Context, arg ListAccountsParams) ([]Account, error) {
    rows, err := q.db.QueryContext(ctx, listAccounts, arg.Limit, arg.Offset)
    if err != nil {
        return nil, err
    }
    defer rows.Close()
    var items []Account
    for rows.Next() {
        var i Account
        if err := rows.Scan(
            &i.ID,
            &i.Owner,
            &i.Balance,
            &i.Currency,
            &i.CreatedAt,
        ); err != nil {
            return nil, err
        }
        items = append(items, i)
    }
    if err := rows.Close(); err != nil {
        return nil, err
    }
    if err := rows.Err(); err != nil {
        return nil, err
    }
    return items, nil
}

We can see that the Account items variable is declared without being initialized: var items []Account. That’s why it will remain null if no records are added.

Lucky for us, in the latest released of sqlc, which is version 1.5.0, we have a new setting that will instruct sqlc to create an empty slice instead of null.

The setting is called emit_empty_slices, and its default value is false. If we set this value to true, then the result returned by a many query will be an empty slice.

OK, so now let’s add this new setting to our sqlc.yaml file:



version: "1"
packages:
  - name: "db"
    path: "./db/sqlc"
    queries: "./db/query/"
    schema: "./db/migration/"
    engine: "postgresql"
    emit_json_tags: true
    emit_prepared_queries: false
    emit_interface: false
    emit_exact_table_names: false
    emit_empty_slices: true

Save it, and open the terminal to upgrade sqlc to the latest version. If you’re on a Mac and using Homebrew, just run:



❯ brew upgrade sqlc

You can check your current version by running:



❯ sqlc version
v1.5.0

For me, it’s already the latest version: 1.5.0, so now I’m gonna regenerate the codes:



❯ make sqlc

And back to visual studio code. Now in the account.sql.go file, we can see that the items variable is now initialized as an empty slice:



func (q *Queries) ListAccounts(ctx context.Context, arg ListAccountsParams) ([]Account, error) {
    ...

    items := []Account{}

    ...
}

Cool! Let’s restart the server and test it on Postman. Now when I send this request, we’ve got an empty list as expected.

So it works!

Now I’m gonna try some invalid parameters. For example, let’s change page_size to 20, which is bigger than the maximum constraint of 10.

This time we’ve got 400 Bad Request status code, and an error saying the validation of page_size failed on the max tag.

Let’s try one more time with page_id = 0.

Now we still got 400 Bad Request status, but the error is because page_id validation failed on the required tag. What happens here is, in the validator package, any zero-value will be recognized as missing. It’s acceptable in this case because we don’t want to have the 0 page, anyway.

However, if your API has a zero value parameter, then you need to pay attention to this. I recommend you to read the documentation of validator package to learn more about it.

Alright, so today we have learned how easy it is to implement RESTful HTTP APIs in Go using Gin. You can based on this tutorial to try implementing some more routes to update or delete accounts on your own. I leave that as a practice exercise for you.

Thanks a lot for reading this article. Happy coding and I will see you soon in the next lecture!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.