DEV Community: Jacob Goodwin

Get Started with XState and Vue 3

Jacob Goodwin — Thu, 21 Jan 2021 05:26:17 +0000

Hey folks, I just wanted to let you all know that I have recently created a pair of tutorials on using XState with Vue 3.

In the first video, I go over creating a simple state machine with the XState Visualizer, and then show how to integrate this into a Vue 3 setup function by using the composable function provided by XState.

In the second video, I discuss how we can use Typestates along with Typescript to better type our state machines and give us great compile-time support inside of Vue templates. In this latter tutorial, I also go over scaffolding out a Vite application for Typescript. In this video, we work on asynchronous data fetching.

I hope you check out the videos and give'em a thumbs up!

17 - Updating TokenPair Model

Jacob Goodwin — Mon, 11 Jan 2021 03:23:25 +0000

Written Tutorial Ending 😥

Thanks to all of you have been following along with this tutorial series. After much consideration, I have decided to focus on the video series only.

I'm currently increasing my effort to land my first job in software development and have thankfully been getting a few interviews. I need a little more time to review topics and frameworks that I haven't used in some time as I tend to go all in with one tech stack and then forget what I've previously learned. I also should spend some more time on algorithm study as I do not have a CS background. Unfortunately, companies still haven't found a better way to choose candidates.

Furthermore, my YouTube channel has seen slow, but steady growth in minutes watched and subscribers, whereas these articles don't seem to be gaining as much traction. Also, as the app gets more complex, it's really tough to explain the details in writing.

I know some of you might find it harder to watch a video in your non-native language (and frankly, they're a little long), but I promise I'll try to speak slowly!

Thanks for those of you that have followed along!

As always, check out the Github repo with all of the code for this tutorial including a branch for each lesson and some unit tests not included in the tutorials!

Here's the video for today's tutorial where I'll be updating the TokenPair model!

¡Hasta pronto!

16 - Create Gin Middleware to Extract Authorized User

Jacob Goodwin — Mon, 04 Jan 2021 01:22:56 +0000

Today we'll create a middleware to extract a user from an ID token sent to our application on an Authorization header of an HTTP request. This ID token is the one we send to a user in JWT format when they sign up or sign in.

The middleware will do the following:

verify that we've received the appropriate Authorization header.
check that the token is valid and not expired.
"set" the user on the Gin context so that route handlers can use the verified user data to make authorized updates for this user (like updating their profile, for example).

In some role- or permission-based authorization schemes, the token might store a role, or the role might be fetched from a database based on an ID stored in the token. In our application, each user will only be authorized to make changes to their own account info, so we will not be looking up or assigning any permissions.

In order to create this middleware, we'll need to add a new method to our our TokenService called ValidateIDToken. As middleware is a sub-package of handler, we'll have to pass the TokenService as a parameter to the middleware.

After we've completed these tasks, we'll be able to run our application and send a Get Request with a user's valid ID token to the "/me" endpoint and receive the user's details.

As always, check out the Github repo with all of the code for this tutorial including a branch for each lesson and some unit tests not included in the tutorials!

If you prefer video

Review of Me Handler

If we look at the current code of our "me" handler, we see that we extract a user from the gin context with a Get method. This user is then cast to a *model.User. The middleware we're going to write will make sure that a user is available on the "user" key when this handler method is executed.



func (h *Handler) Me(c *gin.Context) {
    // A *model.User will eventually be added to context in middleware
    user, exists := c.Get("user")

  // This shouldn't happen, as our middleware ought to throw an error.
    // We can also use "MustGet" to get the key or panic
    if !exists {
        log.Printf("Unable to extract user from request context for unknown reason: %v\n", c)
        err := apperrors.NewInternal()
        c.JSON(err.Status(), gin.H{
            "error": err,
        })

        return
    }

  uid := user.(*model.User).UID

  // code omitted
}

Fix UserService.Signin

I previously failed to update the *model.User passed as a method parameter to UserService.Signin with the user returned by UserRepository.FindByEmail.

Let's fix this as follows:



  // prev
  u = uFetched

  // May be better to return new user from method
  *u = *uFetched

If I had to do things again, I would pass an email and a password (or else a struct containing these fields) to UserService.Signin, and return (*model.User, error). I believe this approach would be much easier to understand.

To avoid having to modify our tests and handler.Signin, we won't fix this now, but feel free to do this in your app.

Add a validateIDToken Function

We previously created a file with some utility functions for working with tokens in ~/service/tokens.go. Let's now update this file with with a function to validate an ID token.



// IDTokenCustomClaims holds structure of jwt claims of idToken
type IDTokenCustomClaims struct {
    User *model.User `json:"user"`
    jwt.StandardClaims
}

// CODE OMITTED ...

// validateIDToken returns the token's claims if the token is valid
func validateIDToken(tokenString string, key *rsa.PublicKey) (*IDTokenCustomClaims, error) {
    claims := &IDTokenCustomClaims{}

    token, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
        return key, nil
    })

    // For now we'll just return the error and handle logging in service level
    if err != nil {
        return nil, err
    }

    if !token.Valid {
        return nil, fmt.Errorf("ID token is invalid")
    }

    claims, ok := token.Claims.(*IDTokenCustomClaims)

    if !ok {
        return nil, fmt.Errorf("ID token valid but couldn't parse claims")
    }

    return claims, nil
}

This function takes in the stringified version of our JWT, tokenString, and the rsa.PublicKey which is a field of our tokenService. We first parse the JWT using the public RSA key. This will return a *jwt.Token. This token will contain a valid field indicating whether the token is valid (including if it is expired). It also contains the claims, or fields, of the token. We cast these claims to *IDTokenCustomClaims and then return them.

Use validateIDToken in TokenService.ValidateIDToken

Add Method to Interfaces

We need to update the TokenService interface by creating the method signature for ValidateIDToken. We do this in ~/model/interfaces.go.



// TokenService defines methods the handler layer expects to interact
// with in regards to producing JWTs as string
type TokenService interface {
    NewPairFromUser(ctx context.Context, u *User, prevTokenID string) (*TokenPair, error)
    ValidateIDToken(tokenString string) (*User, error)
}

We need not pass a context as this method will not be making calls to the repository layer and because our token functions do not require a context.

Create Mock ValidateIDToken

To get rid of warnings, I'll add a mock definition in ~/model/mocks/token_service.go.



// ValidateIDToken mocks concrete ValidateIDToken
func (m *MockTokenService) ValidateIDToken(tokenString string) (*model.User, error) {
    ret := m.Called(tokenString)

    // first value passed to "Return"
    var r0 *model.User
    if ret.Get(0) != nil {
        // we can just return this if we know we won't be passing function to "Return"
        r0 = ret.Get(0).(*model.User)
    }

    var r1 error

    if ret.Get(1) != nil {
        r1 = ret.Get(1).(error)
    }

    return r0, r1
}

ValidateIDToken Implementation

The implementation of TokenService.ValidateIDToken will be pretty simple. The main purpose is to extract and return the *model.User from off of the IDTokenCustomClaims. Let's update ~/service/token_service.go.

In the case of any error from validateIDToken, we'll return an authorization error.



// ValidateIDToken validates the id token jwt string
// It returns the user extract from the IDTokenCustomClaims
func (s *tokenService) ValidateIDToken(tokenString string) (*model.User, error) {
    claims, err := validateIDToken(tokenString, s.PubKey) // uses public RSA key

    // We'll just return unauthorized error in all instances of failing to verify user
    if err != nil {
        log.Printf("Unable to validate or parse idToken - Error: %v\n", err)
        return nil, apperrors.NewAuthorization("Unable to verify user from idToken")
    }

    return claims.User, nil
}

Create AuthUser Middleware

We can finally create the middleware to extract an authorized user! To do this, let's add a file, ~/handler/middleware/auth_user.go.



package middleware

// IMPORTS OMITTED - Make sure to import validator/v10
// My auto import always uses V9

type authHeader struct {
    IDToken string `header:"Authorization"`
}

// used to help extract validation errors
type invalidArgument struct {
    Field string `json:"field"`
    Value string `json:"value"`
    Tag   string `json:"tag"`
    Param string `json:"param"`
}

// AuthUser extracts a user from the Authorization header
// which is of the form "Bearer token"
// It sets the user to the context if the user exists
func AuthUser(s model.TokenService) gin.HandlerFunc {
    return func(c *gin.Context) {
        h := authHeader{}

        // bind Authorization Header to h and check for validation errors
        if err := c.ShouldBindHeader(&h); err != nil {
            if errs, ok := err.(validator.ValidationErrors); ok {
                // we used this type in bind_data to extract desired fields from errs
                // you might consider extracting it
                var invalidArgs []invalidArgument

                for _, err := range errs {
                    invalidArgs = append(invalidArgs, invalidArgument{
                        err.Field(),
                        err.Value().(string),
                        err.Tag(),
                        err.Param(),
                    })
                }

                err := apperrors.NewBadRequest("Invalid request parameters. See invalidArgs")

                c.JSON(err.Status(), gin.H{
                    "error":       err,
                    "invalidArgs": invalidArgs,
                })
                c.Abort()
                return
            }

            // otherwise error type is unknown
            err := apperrors.NewInternal()
            c.JSON(err.Status(), gin.H{
                "error": err,
            })
            c.Abort()
            return
        }

        idTokenHeader := strings.Split(h.IDToken, "Bearer ")

        if len(idTokenHeader) < 2 {
            err := apperrors.NewAuthorization("Must provide Authorization header with format `Bearer {token}`")

            c.JSON(err.Status(), gin.H{
                "error": err,
            })
            c.Abort()
            return
        }

        // validate ID token here
        user, err := s.ValidateIDToken(idTokenHeader[1])

        if err != nil {
            err := apperrors.NewAuthorization("Provided token is invalid")
            c.JSON(err.Status(), gin.H{
                "error": err,
            })
            c.Abort()
            return
        }

        c.Set("user", user)

        c.Next()
    }
}

In the middleware we:

Check for validation errors provided by Gin's ShouldBindHeader. We check if the error is a validation error. If it is, we extract a few fields off of each validator.ValidationErrors (individual errors are called FieldError) to send to the client as a BadRequest. We use invalidArgument to define the error fields we want to send to the client.
Check that the Authorization header is provided in the format Bearer {token}. We split the token off of the string, which is a stringified JWT, and check for any errors.
Finally, we reach out to the ValidateIDToken method we just created, which uses the JWT library to make sure the token can be verified with the public RSA key, and that the token is not expired.
If all of these cases pass, we can Set the user on our Gin context, and call the Next() handler.

Apply AuthUser to Me Handler

Recall that our Me handler will expect to extract a model.User from the gin context. To make this user available to our handler, we need to add the AuthUser middleware to this individual handler.

We can update this in ~/handler/handler.go.



  // Wish I had thought this through better!
    if gin.Mode() != gin.TestMode {
        g.Use(middleware.Timeout(c.TimeoutDuration, apperrors.NewServiceUnavailable()))
        g.GET("/me", middleware.AuthUser(h.TokenService), h.Me)
    } else {
        g.GET("/me", h.Me)
    }

Notice that we only add the middleware if we're not in test mode. That's because we previously created a handler test in isolation of the middleware. You can choose to do it this way, or to integrate middleware and handlers for your test.

I'd really love if any of you have any practical suggestions on how to handle this in production-grade applications!

Run and Test Application

From the application root, let's run our application!



docker-compose up

Sign In An Existing User

I'll first sign in an existing user.

If you want to see how to make this request in Postman, checkout the YouTube video.



➜ curl --location --request POST 'http://malcorp.test/api/account/signin' \
--header 'Authorization: Bearer {{idToken}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "email": "guy01@guy.com",
    "password": "avalidpassword123"
}'

The response is:



{
  "tokens": {
    "idToken":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6ImVlODZjYzJjLTg2ODEtNDU5YS1hOTc3LTNjYWY5NzUzZTE0YiIsImVtYWlsIjoiZ3V5MDFAZ3V5LmNvbSIsIm5hbWUiOiIiLCJpbWFnZVVybCI6IiIsIndlYnNpdGUiOiIifSwiZXhwIjoxNjA5MjY5NDI2LCJpYXQiOjE2MDkyNjg1MjZ9.RLC-RcH-YnSJfKqgbucUvvo2DV3sLcJwXSMlbvOqnEbPgjeWv_3ae61lZU909xUMq6Qrl-tpGLxgkrkk3FiXUuhu8J8bBdCYgSgBhPTkoVuALSuC9N-0mcVTLiQ2zZVwxpuDWHCxHtcjinCwt-XSq94CuSqfwDxjmc--Y0IiQMa5pRMa5Ol4qhs0ABkCI-cq0op8_HUOR7mctmiyR1xaKC8AmvLXbgYp7-g5DfKquYjdEDM640W4y99eBTvDRJwHqRTE5QBVYwzVylqFcy82yCriKPB0sgv60iACOjngkzTqatPzYI6C_QUtKOoaNY1NiIpRI99jiFrrW7z1IIt9NA","refreshToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJlZTg2Y2MyYy04NjgxLTQ1OWEtYTk3Ny0zY2FmOTc1M2UxNGIiLCJleHAiOjE2MDk1Mjc3MjYsImp0aSI6IjE0NDBlNTg4LWI2NjgtNGVjNy05ZmJiLTU5OTM0ODhjMTE4NCIsImlhdCI6MTYwOTI2ODUyNn0.Hd6j4jzD5IKswvWqnJKG7XFLIBw-IRMLeCD4ojAZedA"
  }
}

We can then add the idToken to our "Authorization" header while making a GET request to "/me".



➜ curl --location --request GET 'http://malcorp.test/api/account/me' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6ImVlODZjYzJjLTg2ODEtNDU5YS1hOTc3LTNjYWY5NzUzZTE0YiIsImVtYWlsIjoiZ3V5MDFAZ3V5LmNvbSIsIm5hbWUiOiIiLCJpbWFnZVVybCI6IiIsIndlYnNpdGUiOiIifSwiZXhwIjoxNjA5MjY5NDI2LCJpYXQiOjE2MDkyNjg1MjZ9.RLC-RcH-YnSJfKqgbucUvvo2DV3sLcJwXSMlbvOqnEbPgjeWv_3ae61lZU909xUMq6Qrl-tpGLxgkrkk3FiXUuhu8J8bBdCYgSgBhPTkoVuALSuC9N-0mcVTLiQ2zZVwxpuDWHCxHtcjinCwt-XSq94CuSqfwDxjmc--Y0IiQMa5pRMa5Ol4qhs0ABkCI-cq0op8_HUOR7mctmiyR1xaKC8AmvLXbgYp7-g5DfKquYjdEDM640W4y99eBTvDRJwHqRTE5QBVYwzVylqFcy82yCriKPB0sgv60iACOjngkzTqatPzYI6C_QUtKOoaNY1NiIpRI99jiFrrW7z1IIt9NA'

And we receive the user as a response.



{
  "user": {
    "uid":"ee86cc2c-8681-459a-a977-3caf9753e14b",
    "email":"guy01@guy.com",
    "name":"",
    "imageUrl":"",
    "website":""
  }
}

I recommend you try to send the request with a missing auth header, which should return an HTTP 401 error with a response such as:



{

    "error": {

        "type": "AUTHORIZATION",

        "message": "Must provide Authorization header with format Bearer {token}"

    }

}

Conclusion

That's all for today. Once again I'll remind you of the unit tests available in the Github repository!

Next time, we'll get working on a "tokens" handler which will be used by a client to get renewed id and refresh tokens. This helps clients remain logged in to applications in our company or domain!

Hasta pronto!

Creating a Re-usable Vue 3 Store Plugin with Typescript

Jacob Goodwin — Tue, 29 Dec 2020 16:47:30 +0000

In a previous article, I described how to use a library I created called vue-dfs-store for creating and accessing re-usable global data stores in Vue 3 applications.

To expound on the topic, I have created a video tutorial demo on how to create a store to maintain a list of favorite restaurants. The demo will work with an asynchronous API to fetch, create, update, and delete a list of favorite restaurants which will be maintained in (vue-dfs) store. You'll learn how to make your state read-only and how to update the state via "trackable" mutations.

A few tutorials already touch on the topic of providing a reactive object and functions to update this state down the application tree. However, the ones I have seen don't use Typescript, which is one of the benefits of Vue 3 and the composition API. They also don't show how you might create your own "API" for accessing and modifying the state.

What's more? I'll be showing you how to create Vue plugins and an example of a Vite development environment with Typescript and TailwindCSS setup!

I hope these additions will be helpful to intermediate and advanced Vue developers who are interested in using the latest and greatest!

Two Good Introductions to Creating Stores with Vue's "Reactive" Objects

Some Related Content

Here's some related, and maybe prerequisite, content that you might find useful while checking out my video!

Vue Composition API
Vue Provide and Inject
Vue Plugins
Purify-TS - A functional programming library for Typescript
Vite - Native ES Module development tool
TailwindCSS

15 - Add Signin to Service and Repository Layers

Jacob Goodwin — Sun, 27 Dec 2020 23:30:10 +0000

Last time, we created and tested a handler for signing in a user. This handler accepts and validates an email and password received in a JSON request body. Let's take a look at our progress diagram to see what we'll work on today!

We'll add functionality to sign-in a user in the service layer. To do this, we'll need to be able to find a user in our database by email, and then verify that the password entered by the user matches the encrypted stored password from the database. To make this possible, we'll add a FindByEmail method to our UserRepository interface, mock, and Postgres implementations. This method will retrieve the user attempting to sign-in, which includes their hashed password. We'll also review and use the function we previously created to compare the user's supplied password with the password stored in the database.

At the end of the tutorial, we'll spin up the application, create a new user with a known password, and then attempt to sign in as that user.

As always, checkout the repo on Github with all of the code for this tutorial including complete tests a branch for each lesson!

If you prefer video, check out the video version below!

I will continue to create unit tests for handler- and service-layer methods. However, I will not include these in written or video tutorials unless I feel they provide some new information. I feel a lengthy portion of the tutorials has been dedicated to testing. I did this deliberately because sometimes setting up tests is more difficult than actual coding logic (seriously), and only a few tutorials write more than rudimentary tests.

Add FindByEmail to UserRepository Interface

In ~/model/interfaces.go, let's add an expectation that a UserRepository can find a user by email address.

// UserRepository defines methods the service layer expects
// any repository it interacts with to implement
type UserRepository interface {
    Create(ctx context.Context, u *User) error
    FindByEmail(ctx context.Context, email string) (*User, error)
    FindByID(ctx context.Context, uid uuid.UUID) (*User, error)
}

You should now have errors because or pGUserRepository and mockUserRepository do not implement this newly added method. Let's add these implementations!

Add FindByEmail to mockUserRepository

Though I mentioned that I won't go over unit tests unless necessary, I'll still add the mock implementations so you don't have any errors.

Add the following to ~/model/mocks/user_repository.go.

// FindByEmail is mock of UserRepository.FindByEmail
func (m *MockUserRepository) FindByEmail(ctx context.Context, email string) (*model.User, error) {
    ret := m.Called(ctx, email)

    var r0 *model.User
    if ret.Get(0) != nil {
        r0 = ret.Get(0).(*model.User)
    }

    var r1 error

    if ret.Get(1) != nil {
        r1 = ret.Get(1).(error)
    }

    return r0, r1
}

Add FindByEmail to pGUserRepository

Let's now add the implementation to our the Postgres User Repository implementation, found in ~/repository/pg_user_repository.go.

// FindByEmail retrieves user row by email address
func (r *pGUserRepository) FindByEmail(ctx context.Context, email string) (*model.User, error) {
    user := &model.User{}

    query := "SELECT * FROM users WHERE email=$1"

    if err := r.DB.GetContext(ctx, user, query, email); err != nil {
        log.Printf("Unable to get user with email address: %v. Err: %v\n", email, err)
        return user, apperrors.NewNotFound("email", email)
    }

    return user, nil
}

In this code block, we simply create a select query to get a user with the given email, and return a NotFound error if the user cannot be found (and for all possible errors, which is maybe a bit lazy). If the user is found, it will be populated on the *model.User and returned.

User Service Signin Implementation

We can now reach out to FindByEmail within the UserService. Let's add this in ~/service/user_service.go!

// Signin reaches our to a UserRepository check if the user exists
// and then compares the supplied password with the provided password.
// If a valid email/password combo is provided, u will hold all
// available user fields
func (s *userService) Signin(ctx context.Context, u *model.User) error {
    uFetched, err := s.UserRepository.FindByEmail(ctx, u.Email)

    // Will return NotAuthorized to client to omit details of why
    if err != nil {
        return apperrors.NewAuthorization("Invalid email and password combination")
    }

    // verify password - we previously created this method
    match, err := comparePasswords(uFetched.Password, u.Password)

    if err != nil {
        return apperrors.NewInternal()
    }

    if !match {
        return apperrors.NewAuthorization("Invalid email and password combination")
    }

    u = uFetched
    return nil
}

After storing the user in uFetch, we make a call to comparePasswords which compares the supplied password (from the HTTP request) and the retrieved password (from the database). We previously created this function in ~/service/passwords.go. This function extracts the password salt, pwsalt[1], and then hashes the supplied password with this salt. If this matches the hashed password from the database, pwsalt[0], we know the user entered a valid password! We return whether or not the password is a match as a boolean. We can also return an error if decoding the stored password fails.

func comparePasswords(storedPassword string, suppliedPassword string) (bool, error) {
    pwsalt := strings.Split(storedPassword, ".")

    // check supplied password salted with hash
    salt, err := hex.DecodeString(pwsalt[1])

    if err != nil {
        return false, fmt.Errorf("Unable to verify user password")
    }

    shash, err := scrypt.Key([]byte(suppliedPassword), salt, 32768, 8, 1, 32)

    return hex.EncodeToString(shash) == pwsalt[0], nil
}

Run Application

From the root of the project, you should now be able to run docker-compose up! Let's create a new user by posting a request to our signup endpoint (I've deleted all previous users from Postgres prior to this tutorial). I'll use curl here to keep things simple, and Postman in the video if you prefer to check that out!

➜  curl --location --request POST 'http://malcorp.test/api/account/signup' \
--header 'Content-Type: application/json' \
--data-raw '{
    "email": "guy01@guy.com",
    "password": "avalidpassword123"
}'

We get a status 201, created, response with the following body.

HTTP/1.1 201 Created
Content-Length: 871
Content-Type: application/json; charset=utf-8
Date: Wed, 23 Dec 2020 01:39:09 GMT

{"tokens":{"idToken":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6ImVlODZjYzJjLTg2ODEtNDU5YS1hOTc3LTNjYWY5NzUzZTE0YiIsImVtYWlsIjoiZ3V5MDFAZ3V5LmNvbSIsIm5hbWUiOiIiLCJpbWFnZVVybCI6IiIsIndlYnNpdGUiOiIifSwiZXhwIjoxNjA4Njg4NDQ5LCJpYXQiOjE2MDg2ODc1NDl9.haU3a-15xfoUYrpllkkuUphKFDqNfZKckmPZP6LRN7BGhe15DAONdirhLnH1n5QHFaqQ31eOs1nAleqln5MTzeG_YYdw4VhbQ53wve_b156SeMEvfm664js8fSQYsfTG_PBzkmkRaL62jcSaNmSWkKhzzT5bBeYlBd4lUBqGV1nw12Jj9WgF6oWoDHN786bSMQz25TWmkVyE1-082DHUdAjqnnVy7J_G-CU1Ozdv_6KUurUeVfqBj0D4irghcMnfnk75vBzFhyOShl2-RkXprRKvqjNo0u28Fd5BZ6ZKLAv6k_iUxK8rb-F3atozlFhdWaNL77w18XI4ZkZ2YieLPw","refreshToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJlZTg2Y2MyYy04NjgxLTQ1OWEtYTk3Ny0zY2FmOTc1M2UxNGIiLCJleHAiOjE2MDg5NDY3NDksImp0aSI6IjIzODhkY2Y5LWVlODQtNGUzMy04ZGJlLTZmNTlkOGQ2NzFmNCIsImlhdCI6MTYwODY4NzU0OX0.Rsr2zFf62WYm2ExZXo5zVlq0Ot_jqnoUtL8xapjc75U"}}%

If we then send a request to the signin endpoint with the same email and password, we should hopefully get a token pair and a response of status 200 (ok).

➜ curl -i --location --request POST 'http://malcorp.test/api/account/signin' \
--header 'Content-Type: application/json' \
--data-raw '{
    "email": "guy01@guy.com",
    "password": "avalidpassword123"
}'
HTTP/1.1 200 OK
Content-Length: 871
Content-Type: application/json; charset=utf-8
Date: Wed, 23 Dec 2020 01:39:51 GMT

{"tokens":{"idToken":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6IjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsImVtYWlsIjoiZ3V5MDFAZ3V5LmNvbSIsIm5hbWUiOiIiLCJpbWFnZVVybCI6IiIsIndlYnNpdGUiOiIifSwiZXhwIjoxNjA4Njg4NDkxLCJpYXQiOjE2MDg2ODc1OTF9.YuGGc6m1ZaL7BMSGTwdS7hBt8QFIcxRn1MJ-PqjnOm9vtVUPrVsYbg0n_TcwypcqtAcuhsI3buIipFj9GJU657q3INZWcVzNzlWEzeaPPUKuoJtL2EUP6veGElKd8bAQWsg5eX1T48ff8x4CxW-s7PJ0ZLWMi2Al2TU4xbzz4wxGs6PfgD3T4UYuwnCvnC3GGRdL0htLmqc9EiGqs4M6fzu8HhrusKSRvDdbbKNBO6eELtOzRM8_YKcbBBKMGsS9gKxGnDY227_zqYsc1T1fpy7NYmz7SSxZjd4c6XEqcmItqG28L9tvELZk1HlMQvOI7_yTxW13ntCaLLKdkKnZ0A","refreshToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiIwMDAwMDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAiLCJleHAiOjE2MDg5NDY3OTEsImp0aSI6IjQ3MTFjMDE0LTVmMDgtNDZhNC04Njk3LWQwNGVjMDMwMTEyMCIsImlhdCI6MTYwODY4NzU5MX0.aG7Y01RldyOxxkmaFIT04iYhvb1joSkBw0bboIDSpmE"}}%

Conclusion

We now have the ability to sign up and sign in users. This was quite an effort, especially since we added unit tests and spent a great deal of time on architecture.

Next time, we'll create our second middleware, which will be used to extract the user from our idToken. You might recall that at the very beginning of the tutorial, we created a me handler and API endpoint, but didn't use this in our application. I did this deliberately to show that we could test the handler in isolation od middleware and service layers.

Next time, we'll create this middleware. This can be used to verify the user, and authorize them to do things like retrieve and update their user profile details.

Until next time, ¡chau!

14 - Add Signin Handler

Jacob Goodwin — Sun, 20 Dec 2020 21:47:02 +0000

Today's tutorial will be the first of 2 parts on implementing sign in functionality. We'll follow many of the same patterns we used to sign up users, implementing some new methods in the process!

As seen in the diagram, we'll start with the handler layer signin in method, and next time flush out the details of all the other layers, some of which functionality we've already implemented!

As always, checkout the repo on Github with all of the code for this tutorial, including a branch for each lesson!

If you prefer video, check out the video version below!

Add Signin to UserService Interface

When a user signs in, we want to accept a user's email and password, just like we would when a user signs up. While the implementation details of signing in and signing up differ, the method signatures are the same. Let's add Signin to the UserService interface in ~/model/interfaces.go. As with Signin, we'll pass a partially filled *model.User, and return an error if signing in fails. If signing in is successful, then the user passed to the method will be modified to contain all user details.

// UserService defines methods the handler layer expects
// any service it interacts with to implement
type UserService interface {
    Get(ctx context.Context, uid uuid.UUID) (*User, error)
    Signup(ctx context.Context, u *User) error
    Signin(ctx context.Context, u *User) error
}

This update will break our mockUserService and service-layer UserService as they do not implement this method. Let's add an incomplete implementation of Signup in ~/service/user_service.go, which we'll complete in the next tutorial.

// Signin reaches our to a UserRepository check if the user exists
// and then compares the supplied password with the provided password
// if a valid email/password combo is provided, u will hold all
// available user fields
func (s *userService) Signin(ctx context.Context, u *model.User) error {
    panic("Not implemented")
}

Add Mock Signin Method

Before adding our handler logic and unit test, let's update our ~/model/mocks/user_service.go so that with a Signin method. See previous tutorials on testing and testify to learn more!

func (m *MockUserService) Signin(ctx context.Context, u *model.User) error {
    ret := m.Called(ctx, u)

    var r0 error
    if ret.Get(0) != nil {
        r0 = ret.Get(0).(error)
    }

    return r0
}

Add Handler

Since we'll be adding a lot of code for this handler, let's copy the Signin handler from ~/handler/handler.go into its own file, ~/handler/signin.go. We'll also update the contents of this method as follows.

package handler

// IMPORTS OMITTED

// signinReq is not exported
type signinReq struct {
    Email    string `json:"email" binding:"required,email"`
    Password string `json:"password" binding:"required,gte=6,lte=30"`
}

// Signin used to authenticate extant user
func (h *Handler) Signin(c *gin.Context) {
    var req signinReq

    if ok := bindData(c, &req); !ok {
        return
    }

    u := &model.User{
        Email:    req.Email,
        Password: req.Password,
    }

    ctx := c.Request.Context()
    err := h.UserService.Signin(ctx, u)

    if err != nil {
        log.Printf("Failed to sign in user: %v\n", err.Error())
        c.JSON(apperrors.Status(err), gin.H{
            "error": err,
        })
        return
    }

    tokens, err := h.TokenService.NewPairFromUser(ctx, u, "")

    if err != nil {
        log.Printf("Failed to create tokens for user: %v\n", err.Error())

        c.JSON(apperrors.Status(err), gin.H{
            "error": err,
        })
        return
    }

    c.JSON(http.StatusOK, gin.H{
        "tokens": tokens,
    })
}

We do the following in this handler method.

Bind data to signinReq using the helper bind_data function we previously created. This function sends a "bad request" HTTP status 400 error if the data cannot be bound or fails validation. In reality, it would be wise to create a separate test for bind_data (turns out this tutorial is flawed 😂).
If we do have valid data, we create a partially filled *model.User from the request's email and password fields.
Extract the request context from the gin context using c.Request.Context().
Reach out to the UserService.Signin method and handle any errors. If there is any error, we use our apperrors.Status() function to determine is the error is among those defined in our custom apperrors.
If the user signs in successfully, we'll send a new token pair, just as we do when signing up. We also make sure to handle any errors.

Testing Signin

Let's create a new file for this test, ~/handler/signin_test.go. In this file we'll test the following.

Invalid request data - Although we won't test all validation error combinations, we will test one case to make sure that we receive the proper error JSON response for a bad request.
Error returned by UserService.Signin
Successful call to TokenService.NewPairFromUser (which means a successful handler result).
Error from TokenService.NewPairFromUser. Again, we just want to make sure the handler sends the proper HTTP response in this case.

Test Body and Setup

We'll instantiate our mock services, gin engine/router, and handler in the setup portion of our test (before the t.Run blocks).

However, I decided that it will be clearer to define our mock method responses, i.e. mock.On(...), inside of the individual t.Run blocks instead of in the setup. I did this because I ended up creating so many mock method call definitions and variables in the setup that it was hard to understand what variables corresponded to which test cases.

package handler

// IMPORTS OMITTED

func TestSignin(t *testing.T) {
    // Setup
    gin.SetMode(gin.TestMode)

    // setup mock services, gin engine/router, handler layer
    mockUserService := new(mocks.MockUserService)
    mockTokenService := new(mocks.MockTokenService)

    router := gin.Default()

    NewHandler(&Config{
        R:            router,
        UserService:  mockUserService,
        TokenService: mockTokenService,
  })

  // Tests will be added here below
  // ...
}

Bad Request Data Case

In this case we create a JSON request body with an invalid email address and send this to our handler which we instantiated in the setup portion of the test. We assert that an HTTP Status of 400, http.StatusBadRequest, is sent and that the mockUserService.Signin and mockTokenService.NewPairFromUser methods don't get called as the handler should return before reaching these methods.

    t.Run("Bad request data", func(t *testing.T) {
        // a response recorder for getting written http response
        rr := httptest.NewRecorder()

        // create a request body with invalid fields
        reqBody, err := json.Marshal(gin.H{
            "email":    "notanemail",
            "password": "short",
        })
        assert.NoError(t, err)

        request, err := http.NewRequest(http.MethodPost, "/signin", bytes.NewBuffer(reqBody))
        assert.NoError(t, err)

        request.Header.Set("Content-Type", "application/json")
        router.ServeHTTP(rr, request)

        assert.Equal(t, http.StatusBadRequest, rr.Code)
    mockUserService.AssertNotCalled(t, "Signin")
    mockTokenService.AssertNotCalled(t, "NewTokensFromUser")
    })

Error from UserService.Signin Case

In this case, we define a mock response for Signin which returns an error. We create an example of such an error and store it in mockError. We then assert that this error is relayed and sent as JSON to the user with HTTP status 401, http.StatusUnauthorized, and the assert that mockUserService.Signin method is called, but that mockTokenService.NewPairFromUser is not called.

    t.Run("Error Returned from UserService.Signin", func(t *testing.T) {
        email := "bob@bob.com"
        password := "pwdoesnotmatch123"

        mockUSArgs := mock.Arguments{
            mock.AnythingOfType("*context.emptyCtx"),
            &model.User{Email: email, Password: password},
        }

        // so we can check for a known status code
        mockError := apperrors.NewAuthorization("invalid email/password combo")

        mockUserService.On("Signin", mockUSArgs...).Return(mockError)

        // a response recorder for getting written http response
        rr := httptest.NewRecorder()

        // create a request body with valid fields
        reqBody, err := json.Marshal(gin.H{
            "email":    email,
            "password": password,
        })
        assert.NoError(t, err)

        request, err := http.NewRequest(http.MethodPost, "/signin", bytes.NewBuffer(reqBody))
        assert.NoError(t, err)

        request.Header.Set("Content-Type", "application/json")
        router.ServeHTTP(rr, request)

    mockUserService.AssertCalled(t, "Signin", mockUSArgs...)
    mockTokenService.AssertNotCalled(t, "NewTokensFromUser")
        assert.Equal(t, http.StatusUnauthorized, rr.Code)
    })

Success Case

In this case, we respond to mockUserService.Signin with no error (nil). We also mock a valid token response from mockTokenService.NewPairFromUser. We assert that both of the above methods are called and that a valid HTTP response with HTTP status 200, http.StatusOK, is returned.

  t.Run("Successful Token Creation", func(t *testing.T) {
        email := "bob@bob.com"
        password := "pwworksgreat123"

        mockUSArgs := mock.Arguments{
            mock.AnythingOfType("*context.emptyCtx"),
            &model.User{Email: email, Password: password},
        }

        mockUserService.On("Signin", mockUSArgs...).Return(nil)

        mockTSArgs := mock.Arguments{
            mock.AnythingOfType("*context.emptyCtx"),
            &model.User{Email: email, Password: password},
            "",
        }

        mockTokenPair := &model.TokenPair{
            IDToken:      "idToken",
            RefreshToken: "refreshToken",
        }

        mockTokenService.On("NewPairFromUser", mockTSArgs...).Return(mockTokenPair, nil)

        // a response recorder for getting written http response
        rr := httptest.NewRecorder()

        // create a request body with valid fields
        reqBody, err := json.Marshal(gin.H{
            "email":    email,
            "password": password,
        })
        assert.NoError(t, err)

        request, err := http.NewRequest(http.MethodPost, "/signin", bytes.NewBuffer(reqBody))
        assert.NoError(t, err)

        request.Header.Set("Content-Type", "application/json")
        router.ServeHTTP(rr, request)

        respBody, err := json.Marshal(gin.H{
            "tokens": mockTokenPair,
        })
        assert.NoError(t, err)

        assert.Equal(t, http.StatusOK, rr.Code)
        assert.Equal(t, respBody, rr.Body.Bytes())

        mockUserService.AssertCalled(t, "Signin", mockUSArgs...)
        mockTokenService.AssertCalled(t, "NewPairFromUser", mockTSArgs...)
    })

Error from TokenService.NewPairFromUser Case

In this case, we want the mockUserService.Signin method to be called successfully, but want mockTokenService.NewPairFromUser to produce an error. We then expect an HTTP 500, http.StatusInternalServerError, as a response.

  t.Run("Failed Token Creation", func(t *testing.T) {
        email := "cannotproducetoken@bob.com"
        password := "cannotproducetoken"

        mockUSArgs := mock.Arguments{
            mock.AnythingOfType("*context.emptyCtx"),
            &model.User{Email: email, Password: password},
        }

        mockUserService.On("Signin", mockUSArgs...).Return(nil)

        mockTSArgs := mock.Arguments{
            mock.AnythingOfType("*context.emptyCtx"),
            &model.User{Email: email, Password: password},
            "",
        }

        mockError := apperrors.NewInternal()
        mockTokenService.On("NewPairFromUser", mockTSArgs...).Return(nil, mockError)
        // a response recorder for getting written http response
        rr := httptest.NewRecorder()

        // create a request body with valid fields
        reqBody, err := json.Marshal(gin.H{
            "email":    email,
            "password": password,
        })
        assert.NoError(t, err)

        request, err := http.NewRequest(http.MethodPost, "/signin", bytes.NewBuffer(reqBody))
        assert.NoError(t, err)

        request.Header.Set("Content-Type", "application/json")
        router.ServeHTTP(rr, request)

        respBody, err := json.Marshal(gin.H{
            "error": mockError,
        })
        assert.NoError(t, err)

        assert.Equal(t, mockError.Status(), rr.Code)
        assert.Equal(t, respBody, rr.Body.Bytes())

        mockUserService.AssertCalled(t, "Signin", mockUSArgs...)
        mockTokenService.AssertCalled(t, "NewPairFromUser", mockTSArgs...)
    })

Conclusion

That's all for today, chicos! Next time, we'll write the concrete implementation of UserService.Signin as well as the needed UserRepository methods to get a user from the database and verify their entered password.

¡Chau, hasta luego!

13 - Gin Handler Timeout Middleware

Jacob Goodwin — Mon, 14 Dec 2020 20:05:12 +0000

We just added the ability to store refresh tokens in Redis. Today, let's take a break from the main application features to learn how to create our first Gin middleware!

If at any point you are confused about file structure or code, go to Github repository and check out the branch for the previous lesson to be in sync with me!

I also wanted to mentioned that I made two updates in the main branch of the repository for more easily getting the application running for newcomers to this project. The steps to run the project have been updated in the README, as well.

First, I decided to add the .env.dev file to the repository. For development, I don't plan to include any API secrets directly in this file. Later on, however, we'll reference a Google Cloud Configuration file in .env.dev, so you'll need to make sure to keep that configuration file secret as always.

Second, I've added a make rule, make init, which will start up our Postgres containers and run database migrations so that you'll be ready to start creating data (users for now) with the application!

If you prefer video, check out the video version below!

Handler Timeout Middleware

Here's the ol' diagram of what we'll be working on today.

If we look at the right-hand side of our middleware, we see our application layers. The handler layer methods all receive a *gin.Context. This context is provided to us by Gin and gives us access to numerous utility methods that make working with HTTP requests and responses easier.

The problem we're trying to solve is if a call to these handler methods (including all of the downstream calls to the service/repository layers and data sources) takes a long time, we want to be able to terminate the handler and send an error to the user.

We can do this with built-in functionality from the context package of Go, by wrapping the request context using the context.WithTimeout method. When we wrap our context with this method, we get access to the context.Done() method which "returns a channel that's closed when work done on behalf of this context should be canceled" (source). After using WithTimeout, the context will be cancelled for us automatically after the duration of time we pass to the method.

We'll save discussing the timeoutWriter for when we write the middleware code.

What About http.Server Timeout Fields?

If we look in ~/main.go, we initialize our server with Go's http.Server.



  srv := &http.Server{
        Addr:    ":8080",
        Handler: router,
  }

You might be asking why we don't just make use of the net/http package's Server fields like ReadTimeout or WriteTimeout. These timeouts are concerned with the time to read and write the body to and from the client (i.e. networking I/O) and don't deal with the case of long running-handlers. There's a fine answer on StackOverflow you might want to read.

Middleware In Gin

Custom Middleware Function

Looking back at the diagram, remember that our goal is to manipulate the request context, which we extract from the gin.Context using the Request() method. I'll reiterate that the Gin and request context are two different contexts, which can be confusing.

To create a custom middleware in Gin, we create a function that returns a gin.HandlerFunc, which is defined as:



type HandlerFunc func(*Context)

In middleware HandlerFunc, we can modify this gin.Context according to our needs. In our handler timeout middleware, we will wrap our the request context with a timeout and handle concurrency issues. Another simple example is a logger, as shown in the Gin docs. Yet another case you might see is extracting a user from a session or token, and setting this user as a key on gin.Context. In fact, we'll do just that in a handful of tutorials!

Usage in Gin

I recommend checking out the Using Middleware section of the Gin docs to see how you can call a middleware function for routes, groups, and individual routes! We'll end up applying middleware to our existing route group in ~/handler/handler.go with the Use method.



  // Create a group, or base url for all routes
    g := c.R.Group(c.BaseURL)

    if gin.Mode() != gin.TestMode {
        g.Use() // middleware will be called in Use()
    }

Why Make Our Own Middleware?

I found the following open source middlewares for setting a handler timeout.

The first is the Go http.TimeoutHandler. The downside to this handler is that it's not configured to directly work with Gin routes and groups. Another thing I didn't like about the handler is that it is configured to send an HTML response in the case of a timeout, and doesn't appear to have a way to send a different Content-Type for a response.

The second middleware I found was gin-contrib/timeout. Note that this repository of middlewares is not the official one by gin-gonic, gin-gonic/contrib, even though it has a similar name. This middleware had issues trying to rewrite headers to a response writer, which we'll discuss when we write our middleware.

The third middleware I found was vearne/gin-timeout. This middleware actually addressed the above issues, and so I have no problem recommending it. However, it did not provide a way to modify the error response sent to the user.

Since getting a timeout handler middleware to work as I would like would require modifying any of the available handlers, I decided it would just be easier to create our own, even if I'm borrowing heavily from http.TimeoutHandler and vearne/gin-timeout.

Middleware Prep

Add A Pretend Delay to Signin Handler

In ~/handler/handler.go, let's fake a handler that takes a long time to complete. We'll use the Signin handler method as it currently doesn't yet have a "real" implementation. We'll set the Sleep duration to 6 seconds, as we'll end up setting our timeout limit to 5 seconds.



// Signin handler
func (h *Handler) Signin(c *gin.Context) {
    time.Sleep(6 * time.Second) // to demonstrate a timeout
    c.JSON(http.StatusOK, gin.H{
        "hello": "it's signin",
    })
}

Update App Errors

I want to add a ServiceUnavailable (HTTP 503) to our ~/model/apperrors/apperrors.go to send when our handlers run longer than 5 seconds.



// update const
const (
    // ... other types excluded for brevity
    ServiceUnavailable   Type = "SERVICE_UNAVAILABLE"    // For long running handlers
)

// update status method switch case to handle ServiceUnavailable

func (e *Error) Status() int {
    switch e.Type {
    // other cases and default excluded for brevity
    case ServiceUnavailable:
        return http.StatusServiceUnavailable
    // cases omitted
    }
}

// add an error factory
// NewServiceUnavailable to create an error for 503
func NewServiceUnavailable() *Error {
    return &Error{
        Type:    ServiceUnavailable,
        Message: fmt.Sprintf("Service unavailable or timed out"),
    }
}

Timeout Duration Environment Variable

Let's make timeout duration configurable. We'll set a handler timeout of 5 seconds for now. In reality, we could set different timeout lengths for different handlers or groups of handlers. As an example, you would probably expect the time to upload a file to a cloud service to take longer than a handler that merely updates text fields in the database. I encourage you to do this in your applications.

In .env.go add the following.



HANDLER_TIMEOUT=5

We now need to update the instantiation of our handler via handler.Config to accept this field as a duration. In ~/handler/handler.go.



// Config will hold services that will eventually be injected into this
// handler layer on handler initialization
type Config struct {
    R               *gin.Engine
    UserService     model.UserService
    TokenService    model.TokenService
    BaseURL         string
    TimeoutDuration time.Duration
}

Let's also load the environment variable and pass it to our handler config in ~/injection.go of package main.



  // read in HANDLER_TIMEOUT
    handlerTimeout := os.Getenv("HANDLER_TIMEOUT")
    ht, err := strconv.ParseInt(handlerTimeout, 0, 64)
    if err != nil {
        return nil, fmt.Errorf("could not parse HANDLER_TIMEOUT as int: %w", err)
    }

    handler.NewHandler(&handler.Config{
        R:               router,
        UserService:     userService,
        TokenService:    tokenService,
        BaseURL:         baseURL,
        TimeoutDuration: time.Duration(time.Duration(ht) * time.Second),
    })

We now have a configurable handler timeout duration! Woot woot!

Timeout Middleware Code

After quite a preface, let's add the middleware code. Our middleware will be added to a sub-package of handler. Let's add a folder and file at the path ~/handler/middleware/timeout.go. We'll go step-by-step over the code we add to this middleware function.

Middleware Function Definition

Our middleware will accept a timeout duration, which will be received from handler.Config, and an error that we'll send to the user in case of a timeout. We'll end up passing the custom ServiceUnavailable error we recently created.



package middleware

// Imports omitted

func Timeout(timeout time.Duration, errTimeout *apperrors.Error) gin.HandlerFunc {
    return func(c *gin.Context) {
    // Code will go here
  }
}

Set Gin Writer to Custom Timeout Writer

We first set the writer on Gin's context, c, to a custom writer, tw. You'll also observe that this writer is passed as a field to our custom timeoutWriter. This is a little confusing, but will hopefully become clearer after we go over the timeoutWriter definition.



    // set Gin's writer as our custom writer
        tw := &timeoutWriter{ResponseWriter: c.Writer, h: make(http.Header)}
        c.Writer = tw

Wrap Context in Timeout

In order to apply a timeout duration, we can use context.WithTimeout as follows. We defer the cancel function's execution until the end of the function. When cancel is called, all resources associated with it context, ctx, will be freed up.



    // wrap the request context with a timeout
        ctx, cancel := context.WithTimeout(c.Request.Context(), timeout)
        defer cancel()

        // update gin request context
        c.Request = c.Request.WithContext(ctx)

Channels For Finished of Panic "Signals"

We create channels that we'll send values into if our handler successfully completes or if the application panics.



    finished := make(chan struct{})        // to indicate handler finished
        panicChan := make(chan interface{}, 1) // used to handle panics if we can't recover

Call Handler Inside of a GoRoutine

We then create and invoke a goroutine which calls c.Next(). Next is a helper method which calls the next middleware or handler function (you can "cascade" handler functions and middlewares). As of yet, we only have a single middleware, so c.Next() will call a handler method. If the method completes, we pass an empty struct into the finished channel.

We also defer a call to recover from a panic and send the captured value from the panic and send it to the panicChan (it turns out we don't do anything with this value, so send and empty struct, if you prefer).



    go func() {
            defer func() {
                if p := recover(); p != nil {
                    panicChan <- p
                }
            }()

            c.Next() // calls subsequent middleware(s) and handler
            finished <- struct{}{}
        }()

Handling Either Finished, Panic, or Timeout/context.Done()

After invoking the goroutine, we then create a select statement that listens for incoming values on one of three channels. In all of these cases, we then write a header and body to a response writer embedded inside of our custom timeoutWriter in variable tw.

This may be a little confusing, writing to a Writer inside of a Writer, but this is how we prevent overwriting response bodies and headers. We basically use the timeoutWriter as a temporary/proxy writer that also stores the state of whether or not we have either timed out or already sent the response from a handler.



    select {
        case <-panicChan:
            // if we cannot recover from panic,
            // send internal server error
            e := apperrors.NewInternal()
            tw.ResponseWriter.WriteHeader(e.Status())
            eResp, _ := json.Marshal(gin.H{
                "error": e,
            })
            tw.ResponseWriter.Write(eResp)
        case <-finished:
            // if finished, set headers and write resp
            tw.mu.Lock()
            defer tw.mu.Unlock()
            // map Headers from tw.Header() (written to by gin)
            // to tw.ResponseWriter for response
            dst := tw.ResponseWriter.Header()
            for k, vv := range tw.Header() {
                dst[k] = vv
            }
            tw.ResponseWriter.WriteHeader(tw.code)
            // tw.wbuf will have been written to already when gin writes to tw.Write()
            tw.ResponseWriter.Write(tw.wbuf.Bytes())
        case <-ctx.Done():
            // timeout has occurred, send errTimeout and write headers
            tw.mu.Lock()
            defer tw.mu.Unlock()
            // ResponseWriter from gin
            tw.ResponseWriter.Header().Set("Content-Type", "application/json")
            tw.ResponseWriter.WriteHeader(errTimeout.Status())
            eResp, _ := json.Marshal(gin.H{
                "error": errTimeout,
            })
            tw.ResponseWriter.Write(eResp)
            c.Abort()
            tw.SetTimedOut()
        }

In the case of a panic, we simply write an internal server error with the appropriate status code.

If we receive on the finished channel, this means that our custom timeoutWriter will have the headers written by Gin to its h field, which will store headers. We then copy these headers to the inner, gin.ResponseWriter, referenced by dst. We then send the status code from tw, and the actual response body from the handler which is stored in tw.wbuf. We make sure to lock tw using it's mutex field to prevent writes in the case we "simultaneously" receive a on another channel.

In the case when the context times out, we receive on the channel returned by ctx.Done(). We also create a response with the *apperrors.Error passed to the middleware as a parameter. We then call c.Abort() which prevents any subsequent handlers or middlewares from being called, and set tw's timedOut field to true so that if the handler response eventually comes through, it will not be written to the client.

Let's now discuss the timeoutWriter in detail.

Custom Timeout Writer Explanation

The code for timeoutWriter is included below.



// implements http.Writer, but tracks if Writer has timed out
// or has already written its header to prevent
// header and body overwrites
// also locks access to this writer to prevent race conditions
// holds the gin.ResponseWriter which we'll manually call Write()
// on in the middleware function to send response
type timeoutWriter struct {
    gin.ResponseWriter
    h    http.Header
    wbuf bytes.Buffer // The zero value for Buffer is an empty buffer ready to use.

    mu          sync.Mutex
    timedOut    bool
    wroteHeader bool
    code        int
}

// Writes the response, but first makes sure there
// hasn't already been a timeout
// In http.ResponseWriter interface
func (tw *timeoutWriter) Write(b []byte) (int, error) {
    tw.mu.Lock()
    defer tw.mu.Unlock()
    if tw.timedOut {
        return 0, nil
    }

    return tw.wbuf.Write(b)
}

// In http.ResponseWriter interface
func (tw *timeoutWriter) WriteHeader(code int) {
    checkWriteHeaderCode(code)
    tw.mu.Lock()
    defer tw.mu.Unlock()
    // We do not write the header if we've timed out or written the header
    if tw.timedOut || tw.wroteHeader {
        return
    }
    tw.writeHeader(code)
}

// set that the header has been written
func (tw *timeoutWriter) writeHeader(code int) {
    tw.wroteHeader = true
    tw.code = code
}

// Header "relays" the header, h, set in struct
// In http.ResponseWriter interface
func (tw *timeoutWriter) Header() http.Header {
    return tw.h
}

// SetTimeOut sets timedOut field to true
func (tw *timeoutWriter) SetTimedOut() {
    tw.timedOut = true
}

func checkWriteHeaderCode(code int) {
    if code < 100 || code > 999 {
        panic(fmt.Sprintf("invalid WriteHeader code %v", code))
    }
}

This writer implements an http.ResponseWriter with the required Write, WriteHeader, and Header methods. In the middleware, we replaced the writer on *gin.Context with the custom timeoutWriter. Therefore, when the Gin handlers write a response, they will end up calling timeoutWriter.Write, timeoutWriter.WriterHeader and timeoutWriter.Header.

Let's take a look at what would happen if we use this middleware and send a response from the Signin handler.



  c.JSON(http.StatusOK, gin.H{
        "hello": "it's signin",
  })

When Gin calls the JSON method it will set the response body by calling timeoutWriter.Write with the JSON body inside of the gin.H map. If we look at timeoutWriter.Write, you can see that it checks that we haven't already set the timedOut field. This is how we prevent overwriting our response body! If we haven't timed out, we write the body to our custom field, wbuf. This wbuf temporarily stores the valid response which gets "relayed" to the client in the finished channel branch of the switch case.

The c.JSON method will also call timeoutWriter.Header() to get access to our writer's headers, timeoutWriter.h. Gin then sets headers such as Content-Type and Content-Length.

Finally, c.JSON will also set the HTTP status via the timeoutWriter.WriteHeader() method. Our custom WriteHeader() method will check to see if the header has already been written or if we've already timed out via the fields timedOut and wroteHeader. If we've already timed out or written a header, we do nothing. Otherwise, we set wroteHeader and the HTTP status code to our struct.

We also have a mu sync.Mutex in our custom writer. You can see that anytime we write to our timeoutWriter, we call tw.mu.Lock(). When we add a sync.Mutex to a struct, this allows us to lock struct to updates by concurrent routines which may want to access our timeoutWriter data (sorry if my terminology is off). Once all of the updates to our timeoutWriter data have been made, we Unlock access to the fields/data.

Apply Middleware

In ~/handler/handler.go we'll use our middleware immediately after creating our route group.



// Create a group, or base url for all routes
    g := c.R.Group(c.BaseURL)

    if gin.Mode() != gin.TestMode {
        g.Use(middleware.Timeout(c.TimeoutDuration, apperrors.NewServiceUnavailable()))
    }

You should now be able to run your app with docker-compose up and send a request to the "signin" endpoint as follows (with Curl or other client):



➜ curl --location --request POST 'http://malcorp.test/api/account/signin' \
--header 'Authorization: Bearer {{idToken}}' \
--header 'Content-Type: application/json' \
--data-raw '{}'
{"error":{"type":"SERVICE_UNAVAILABLE","message":"Service unavailable or timed out process"}}%

If you then set the timeout in Signin to something less than 5 seconds (our application's handler timeout duration), you should receive a status 200 response with JSON.



// Signin handler
func (h *Handler) Signin(c *gin.Context) {
    time.Sleep(1 * time.Second) // to demonstrate normal use
    c.JSON(http.StatusOK, gin.H{
        "hello": "it's signin",
    })
}

Rerunning the same curl command we receive:



{"hello":"it's signin"}%

Conclusion

That was actually a relatively complex middleware. If you're like me when I was learning from other's work, you may have to stare at it for a while until it makes sense. Yet I hope I've laid it out in a manner that saves you from a few minutes of confusion! Heck, I'm not sure I did everything right, so let me know if you think there are any blaring mistakes!

Next time, we'll write and test the Signin handler. Following that, we'll add service and repository method implementations to complete the full signin functionality.

¡Hasta pronto!

12 - Store Refresh Tokens in Redis

Jacob Goodwin — Tue, 08 Dec 2020 00:54:51 +0000

Howdy ho, neighbor!

Last time we cleaned up our app by:

making some structs package private
using the proper request context, *gin.Context.Request(), to forward down the call chain
reading in all environment variables in package main
making sure to send the user a clear error message if they send a non-JSON body.

As there were a lot of changes, you may want to check our the Github repository's branch called lesson-11, to make sure you have properly updated all of your code!

We're in for another big one today, but hopefully the patterns we established while learning to signup users will become even clearer as we store tokens.

You might also find the video useful!

Why We'll Store Refresh Tokens

We recently implemented signing up a user (see parts 8-10 of this series). However, we omitted an important detail in our application - storing refresh tokens. You can see these details below in the calls from NewPairFromUser to the token repository. In the diagram, I've grayed out and checked off the completed portions required to sign up a user.

Why store these tokens at all? Because this gives us the ability to invalidate user's long-lived tokens.

Recall that we set the duration to expiration to 3 days for refresh tokens and 15 minutes for ID tokens. This allows a user with an expired idToken to reach out to this account application with their refreshToken to get a new idToken. The idToken will allow the user to access various applications in our domain.

At the end of the day, we do this to make accessing our applications more pleasant, since it would be a pain in the arse to need to login every 15 minutes.

However, there is a danger in creating a token that lives for 3 days. If the token is stolen, or if a user wants to sign out of all of their devices, they could still access the application for up to 3 days!

For this reason, we're going to create a Redis database that stores valid refresh tokens for users. We use Redis because it's in-memory database (fast), has an easy API for storing key-value pairs, and allows us to easily remove the tokens from the cache after 3 days. A diagram of our storage approach is shown below.

We call this store a white list of refresh tokens. When a user receives an idToken/refreshToken pair, the refresh token will always be stored in Redis. Later on, we'll add a token refresh route and logic to our application. In this case, the user already has a refresh token, which is required to get a new idToken. Nevertheless, we'll expire their old refreshToken in Redis and provide them a fresh one. This is of a type of "revolving" refresh token technique.

I'll admit there are probably more infrastructure-friendly ways to do this, but I think we'll still learn a lot implementing the white list approach, and you can adjust the approach to your needs.

I see that many people create a "black list" of invalidated tokens, instead of a "white list" of valid tokens. In the blacklist approach, if a user wants to reset their password, or log out of all devices, then all of their current refresh tokens would need to be moved to Redis from the main database.

But wait, isn't this the same as a white list? No, because these tokens might originally be stored in the main database, and only sent to the in-memory cache when tokens must be invalidated. This may be a better architectural decision in your case. You'll likely save money using this latter approach, as you would be storing fewer tokens inside of Redis (in-memory is pricier, to the best of my knowledge). You will probably add some complexity, though, by needing to push all of the user's tokens from your main DB to Redis, though that shouldn't be too difficult.

Topics We'll Cover Today

We'll need to perform the following steps to update our application for token storage.

Create a Redis Container in docker-compose.yml.
Instantiate a connection in package main. Recall that we have a data_sources.go file for initializing our data.
Add a TokenRepository interface to our ~/model/interfaces.go and specify the method signatures for SetRefreshToken and DeleteRefreshToken.
Update our TokenService struct and NewPairFromUser method so that we can store the user's current refresh token, and delete their previous refresh token (will be used in a refresh token endpoint will implement down the road).
Add a TokenRepository mock and update the unit test for NewPairFromUser.
Create a RedisTokenRepository and implement SetRefreshToken and DeleteRefreshToken.
Make sure to inject our RedisTokenRepository into TokenService and run the application.

Add Redis to docker-compose.yaml

Add the following service at the same indentation level as other services (e.g., postgres-account) in docker-compose.yml, located at the project root.



redis-account:
    image: "redis:alpine"
    ports:
      - "6379:6379"
    volumes:
      - "redisdata:/data"

We'll also need to make sure to add redis-account to the depends_on list of account



account:
  # ... content omitted here ...
  depends_on:
    - postgres-account
    - redis-account

Also include the volume redisdata under the volumes key at the bottom of the file.



volumes:
  pgdata_account:
  redisdata:

That's all we need to create our Redis container!

Establish Redis Connection

We'll establish the application's connection to Redis inside of ~/data_sources.go of package main. We'll be using the Redis client package github.com/go-redis/redis/v8. Make sure to add this package to go.mod by running go get github.com/go-redis/redis/v8 from inside of the account folder.

Then, update the dataSources struct as follows:



type dataSources struct {
    DB          *sqlx.DB
    RedisClient *redis.Client
}

Then, inside of initDs(), establish a the connection as follows, and make sure to return this connection in dataSources.



// Initialize redis connection
    redisHost := os.Getenv("REDIS_HOST")
    redisPort := os.Getenv("REDIS_PORT")

    log.Printf("Connecting to Redis\n")
    rdb := redis.NewClient(&redis.Options{
        Addr:     fmt.Sprintf("%s:%s", redisHost, redisPort),
        Password: "",
        DB:       0,
    })

    // verify redis connection

    _, err = rdb.Ping(context.Background()).Result()

    if err != nil {
        return nil, fmt.Errorf("error connecting to redis: %w", err)
    }

    return &dataSources{
        DB:          db,
        RedisClient: rdb,
    }, nil

Notice that we're making use of the REDIS_HOST and REDIS_PORT environment variables. Therefore, we need to add the host (inside of Docker networking, this will be the name of the service) and the default Redis port to our ~/account/env.dev file.



REDIS_HOST=redis-account
REDIS_PORT=6379

Finally, we want to make sure we close the connection in our close method at the bottom of the file.



// close to be used in graceful server shutdown
func (d *dataSources) close() error {
    // ...code omitted

    if err := d.RedisClient.Close(); err != nil {
        return fmt.Errorf("error closing Redis Client: %w", err)
    }

    return nil
}

You should now be able spin up the application by running docker-compose up inside of the project root folder! Hopefully you'll see a log showing your Redis container is up and running and that the application has established a connection!

Create Token Repository Interface

We have yet to define expectations for a token repository. We can do so inside ~/model/interfaces.go of the model layer. We'll also add the first two methods for this repository. We'll soon see how these methods are used.



// TokenRepository defines methods it expects a repository
// it interacts with to implement
type TokenRepository interface {
    SetRefreshToken(ctx context.Context, userID string, tokenID string, expiresIn time.Duration) error
    DeleteRefreshToken(ctx context.Context, userID string, prevTokenID string) error
}

Access TokenRepository from tokenService

We will access a TokenRepository interface from inside of our tokenService implementation. Currently this service has a single method, NewPairFromUser, which will access the repository methods we just created.

Let's make sure to add model.TokenRepository as a dependency to tokenService in ~/service/token_service.go as follows:



// tokenService used for injecting an implementation of TokenRepository
// for use in service methods along with keys and secrets for
// signing JWTs
type tokenService struct {
    TokenRepository       model.TokenRepository
    PrivKey               *rsa.PrivateKey
    PubKey                *rsa.PublicKey
    RefreshSecret         string
    IDExpirationSecs      int64
    RefreshExpirationSecs int64
}

// TSConfig will hold repositories that will eventually be injected into this
// this service layer
type TSConfig struct {
    TokenRepository       model.TokenRepository
    PrivKey               *rsa.PrivateKey
    PubKey                *rsa.PublicKey
    RefreshSecret         string
    IDExpirationSecs      int64
    RefreshExpirationSecs int64
}

// NewTokenService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewTokenService(c *TSConfig) model.TokenService {
    return &tokenService{
        TokenRepository:       c.TokenRepository,
        PrivKey:               c.PrivKey,
        PubKey:                c.PubKey,
        RefreshSecret:         c.RefreshSecret,
        IDExpirationSecs:      c.IDExpirationSecs,
        RefreshExpirationSecs: c.RefreshExpirationSecs,
    }
}

We'll now add the logic for storing and deleting the tokens in NewPairFromUser. We do this after the tokens have been generated. Check out part 9 if you want to learn more about generating tokens.



// set freshly minted refresh token to valid list
    if err := s.TokenRepository.SetRefreshToken(ctx, u.UID.String(), refreshToken.ID, refreshToken.ExpiresIn); err != nil {
        log.Printf("Error storing tokenID for uid: %v. Error: %v\n", u.UID, err.Error())
        return nil, apperrors.NewInternal()
    }

    // delete user's current refresh token (used when refreshing idToken)
    if prevTokenID != "" {
        if err := s.TokenRepository.DeleteRefreshToken(ctx, u.UID.String(), prevTokenID); err != nil {
            log.Printf("Could not delete previous refreshToken for uid: %v, tokenID: %v\n", u.UID.String(), prevTokenID)
        }
    }

First, we store the token, which is derived from a userID (as a string), refreshToken.ID and refreshToken.expiresIn. The user is passed to the function from the handler layer, and the refreshToken is generated in a utility function called generateRefreshToken. We return an internal server error should any error occur.

Next, we delete the user's current refresh token if one exists. This functionality is not used when signing up a user. In ~/handler/signup.go, we call NewPairFromUser with an empty string for the final parameter, which is prevTokenID. Note that if there is an error deleting an old refresh token, we do not return an error, but merely log this for internal use.

Here is the call in ~/handler/signup.go.



  // create token pair as strings
  tokens, err := h.TokenService.NewPairFromUser(ctx, u, "")

Even though we're not yet deleting refresh tokens, I wanted to add this logic now so that we can complete our NewPairFromUser method. We'll call this method again when we add token refreshing.

Testing Newly Added Functionality

We will now update our ~/service/token_service_test.go to account for the updated logic to NewPairFromUser. As always, we need to mock the behavior or our TokenRepository so that this service doesn't depend on any concrete repository implementation. Let's add a mock, leveraging the testify package as always!

Add Mock Token Repository

Create a ~/model/mocks/token_repository.go file with the following content for mocking a TokenRepository. These mocks will be fairly straightforward as both methods only return errors. As always, please check out the testify documentation or previous videos to better understand these mocks!



package mocks

// ... imports omitted

// MockTokenRepository is a mock type for model.TokenRepository
type MockTokenRepository struct {
    mock.Mock
}

// SetRefreshToken is a mock of model.TokenRepository SetRefreshToken
func (m *MockTokenRepository) SetRefreshToken(ctx context.Context, userID string, tokenID string, expiresIn time.Duration) error {
    ret := m.Called(ctx, userID, tokenID, expiresIn)

    var r0 error

    if ret.Get(0) != nil {
        r0 = ret.Get(0).(error)
    }

    return r0
}

// DeleteRefreshToken is a mock of model.TokenRepository DeleteRefreshToken
func (m *MockTokenRepository) DeleteRefreshToken(ctx context.Context, userID string, prevTokenID string) error {
    ret := m.Called(ctx, userID, prevTokenID)

    var r0 error

    if ret.Get(0) != nil {
        r0 = ret.Get(0).(error)
    }

    return r0
}

Update Unit Test

Let's make the following updates to ~/service/token_service_test.go:

assert that TokenRepository.SetRefreshToken gets called inside of the current test ("Returns a token pair with proper values"). Also make sure that TokenRepository.DeleteRefreshToken is called if we pass prevTokenID to NewPairFromUser
assert TokenService.NewPairFromUser returns an error if there is an error setting the refresh token.
assert that DeleteRefreshToken is not called if we supply an empty string, "", as the prevTokenID.

I want to setup the mock before the t.Run blocks. Previously, we set up the mocks inside of each t.Run block.

To do this, import and instantiate a MockTokenRepository, making sure to supply it to the NewTokenService factory.



  mockTokenRepository := new(mocks.MockTokenRepository)

    // instantiate a common token service to be used by all tests
    tokenService := NewTokenService(&TSConfig{
        TokenRepository:       mockTokenRepository,
        PrivKey:               privKey,
        PubKey:                pubKey,
        RefreshSecret:         secret,
        IDExpirationSecs:      idExp,
        RefreshExpirationSecs: refreshExp,
    })

And just before the t.Run, we add the following:



  // code after instantiating a tokenService

  // include password to make sure it is not serialized
    // since json tag is "-"
    uid, _ := uuid.NewRandom()
    u := &model.User{
        UID:      uid,
        Email:    "bob@bob.com",
        Password: "blarghedymcblarghface",
    }

    // Setup mock call responses in setup before t.Run statements
    uidErrorCase, _ := uuid.NewRandom()
    uErrorCase := &model.User{
        UID:      uidErrorCase,
        Email:    "failure@failure.com",
        Password: "blarghedymcblarghface",
    }
    prevID := "a_previous_tokenID"

    setSuccessArguments := mock.Arguments{
        mock.AnythingOfType("*context.emptyCtx"),
        u.UID.String(),
        mock.AnythingOfType("string"),
        mock.AnythingOfType("time.Duration"),
    }

    setErrorArguments := mock.Arguments{
        mock.AnythingOfType("*context.emptyCtx"),
        uErrorCase.UID.String(),
        mock.AnythingOfType("string"),
        mock.AnythingOfType("time.Duration"),
    }

    deleteWithPrevIDArguments := mock.Arguments{
        mock.AnythingOfType("*context.emptyCtx"),
        u.UID.String(),
        prevID,
    }

    // mock call argument/responses
    mockTokenRepository.On("SetRefreshToken", setSuccessArguments...).Return(nil)
    mockTokenRepository.On("SetRefreshToken", setErrorArguments...).Return(fmt.Errorf("Error setting refresh token"))
    mockTokenRepository.On("DeleteRefreshToken", deleteWithPrevIDArguments...).Return(nil)

Since we'll be setting up our mock responses before the t.Run blocks, I want to be explicit about the arguments that we pass to various method calls. That is why you see arguments stored in mock.Arguments structs. This will then make it easier to check that the methods were called with those specific arguments without having to rewrite all of the arguments in test-case assertions.

Let's now update our existing test to make sure that SetRefreshToken and DeleteRefreshToken are called with correct parameters. In order to test the call of DeleteRefreshToken, I make sure to pass prevID (defined in the test setup) to NewPairFromUser.



  ctx := context.Background()                                    // updated from context.TODO()
  tokenPair, err := tokenService.NewPairFromUser(ctx, u, prevID) // replaced "" with prevID from setup
  assert.NoError(t, err)

  // SetRefreshToken should be called with setSuccessArguments
  mockTokenRepository.AssertCalled(t, "SetRefreshToken", setSuccessArguments...)
  // DeleteRefreshToken should not be called since prevID is ""
  mockTokenRepository.AssertCalled(t, "DeleteRefreshToken", deleteWithPrevIDArguments...)

  // rest of test is unchanged

Take note that I updated context.TODO to context.Background to conform to our mock argument expectations (context.emptyCtx is the type of context.Background, and this is what we've been using in other tests).

Next, we'll add a test for when SetRefreshToken returns an error. In this test, we make sure an error is returned of the correct type, and that DeleteRefreshToken is not called, since and error will be returned from NewPairFromUser before DeleteRefreshToken can be called.



  t.Run("Error setting refresh token", func(t *testing.T) {
        ctx := context.Background()
        _, err := tokenService.NewPairFromUser(ctx, uErrorCase, "")
        assert.Error(t, err) // should return an error

        // SetRefreshToken should be called with setErrorArguments
        mockTokenRepository.AssertCalled(t, "SetRefreshToken", setErrorArguments...)
        // DeleteRefreshToken should not be since SetRefreshToken causes method to return
        mockTokenRepository.AssertNotCalled(t, "DeleteRefreshToken")
    })

Finally, let's add a test for the case when no prevTokenID is provided. We'll only test to make sure DeleteRefreshToken is not called. We've already tested for the response body in the first test.



  t.Run("Empty string provided for prevID", func(t *testing.T) {
        ctx := context.Background()
        _, err := tokenService.NewPairFromUser(ctx, u, "")
        assert.NoError(t, err)

        // SetRefreshToken should be called with setSuccessArguments
        mockTokenRepository.AssertCalled(t, "SetRefreshToken", setSuccessArguments...)
        // DeleteRefreshToken should not be called since prevID is ""
        mockTokenRepository.AssertNotCalled(t, "DeleteRefreshToken")
    })

You should now retest this method by running the following from the account directory:



go test -v ./service -run NewPairFromUser

RedisTokenRepository Implementation

We start by creating a ~/account/repository/redis_token_repository.go file.

As with other repositories and services, we start by defining a dependency struct and a factory function to initialize this struct.

Make sure that you import redis/v8 as auto-import may not import the correct version.



package repository

//... imports omitted

// redisTokenRepository is data/repository implementation
// of service layer TokenRepository
type redisTokenRepository struct {
    Redis *redis.Client
}

// NewTokenRepository is a factory for initializing User Repositories
func NewTokenRepository(redisClient *redis.Client) model.TokenRepository {
    return &redisTokenRepository{
        Redis: redisClient,
    }
}

Next, we'll implement our current two methods on the model.TokenRepository.



// SetRefreshToken stores a refresh token with an expiry time
func (r *redisTokenRepository) SetRefreshToken(ctx context.Context, userID string, tokenID string, expiresIn time.Duration) error {
    // We'll store userID with token id so we can scan (non-blocking)
    // over the user's tokens and delete them in case of token leakage
    key := fmt.Sprintf("%s:%s", userID, tokenID)
    if err := r.Redis.Set(ctx, key, 0, expiresIn).Err(); err != nil {
        log.Printf("Could not SET refresh token to redis for userID/tokenID: %s/%s: %v\n", userID, tokenID, err)
        return apperrors.NewInternal()
    }
    return nil
}

// DeleteRefreshToken used to delete old  refresh tokens
// Services my access this to revolve tokens
func (r *redisTokenRepository) DeleteRefreshToken(ctx context.Context, userID string, tokenID string) error {
    key := fmt.Sprintf("%s:%s", userID, tokenID)
    if err := r.Redis.Del(ctx, key).Err(); err != nil {
        log.Printf("Could not delete refresh token to redis for userID/tokenID: %s/%s: %v\n", userID, tokenID, err)
        return apperrors.NewInternal()
    }

    return nil
}

I've added some comments about the storage of these tokens, but let's make sure it's clear what we're doing!

We store the tokens with a key {userID}:{tokenID}. We append the token to the userID (as opposed to storing the tokenID alone) because this will allow us to use a Redis operation called scan if we need to invalidate all of a user's refresh tokens. This would be necessary if the user wanted to reset a password or sign out of all devices. We'll be able to scan for and delete all keys starting with the user's id. Scan does this in a non-blocking fashion so other operations can be performed on the database.

The DeleteRefreshToken is used to revolve refresh tokens. If a user wants a new idToken, we'll also give them a new refreshToken and invalidate their former token. But I think I'm repeating myself. 😉

Inject RedisTokenRepository and Run Application

Let's take our Redis data source, create a TokenRepository from its RedisClient field, and inject this into our TokenService implementation. The TokenService has already been injected into the handler. We make these updates in ~/injection.go of package main.



/*
     * repository layer
     */
    userRepository := repository.NewUserRepository(d.DB)
    tokenRepository := repository.NewTokenRepository(d.RedisClient)



  tokenService := service.NewTokenService(&service.TSConfig{
        TokenRepository:       tokenRepository,
        PrivKey:               privKey,
        PubKey:                pubKey,
        RefreshSecret:         refreshSecret,
        IDExpirationSecs:      idExp,
        RefreshExpirationSecs: refreshExp,
    })

Use Redis Client to View Cache

Let's re-run the application, sign up a new user, and then check to make sure that user has been created with a token. Note that I've deleted all users from Postgres before running these commands.



curl --location --request POST 'http://malcorp.test/api/account/signup' \
--header 'Content-Type: application/json' \
--data-raw '{
    "email": "jacob@jacob.com",
    "password": "avalidpassword123"
}'

You should get a response with the idToken and refreshToken stored as JWTs:



{
  "tokens":{
  "idToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6ImIyODRmZGQ5LTU2ZjItNDgyNC1hN2IwLTU1NWQ0NWQwYTFiNCIsImVtYWlsIjoiYm9iQGJvYi5jb20iLCJuYW1lIjoiIiwiaW1hZ2VVcmwiOiIiLCJ3ZWJzaXRlIjoiIn0sImV4cCI6MTYwNzM4OTQ5MywiaWF0IjoxNjA3Mzg4NTkzfQ.jeT0WyRN0rN9RuudXr7fG1rRmVtBw_t-efBRq8E3iaLyEgPMgG5SC6y16mheeMtXQ74ksVKAlDVV7siS9D94pNBh2PoknA7H2Pa_9k5RFSxJPw4g-qk2NIMnOQIJ6NBrpbc2g1HozndGYpX7wnoCMDxlJvuzGg3mpMXIXausQdUG7nr5VJH_izksybRdhmW_vaK4ZushH8oFJT8XpW8zmylI3tz7g7ICSeRd7wIFybW3XbMudmIw_NOhV-dxd_UGgELQfZG4WRvoBOgVFhd0NmWhwqtyVb1CL1NeJJ1zItfX5WriXGvLesGfbDBSgRThirWlWs2tRBFNVRi6kJYu0g",
  "refreshToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJiMjg0ZmRkOS01NmYyLTQ4MjQtYTdiMC01NTVkNDVkMGExYjQiLCJleHAiOjE2MDc2NDc3OTMsImp0aSI6ImMyMzYwMjJkLWE5MTAtNGE4NS1iMjdkLTljNGE1MTQ0NDhjNCIsImlhdCI6MTYwNzM4ODU5M30.DDuKOfQBPqXdSDAwfqVLy0vECcGJtUfScoWYRUFH4Gk"
  }
}

You can then copy and paste the refreshToken inside of the jwt.io debugger or in another way that you prefer.

I get the following body for the token:



{
  "uid": "b284fdd9-56f2-4824-a7b0-555d45d0a1b4",
  "exp": 1607647793,
  "jti": "c236022d-a910-4a85-b27d-9c4a514448c4",
  "iat": 1607388593
}

The jti key is where we store the refresh token ID. You can then check Redis for a token with the uid and jti separated by a colon: b284fdd9-56f2-4824-a7b0-555d45d0a1b4:c236022d-a910-4a85-b27d-9c4a514448c4.

There are various options to look at a Redis DB, including a GUI version.

For this tutorial, I'll show how to use the CLI version.

You can check for the existence of the key as follows (we're using the default Redis port, so you need not do anything special to connect from your local machine):



redis-cli get b284fdd9-56f2-4824-a7b0-555d45d0a1b4:c236022d-a910-4a85-b27d-9c4a514448c4

And you should receive a response of "0", the value we stored under the key (because I believe you have to store a value for each key).



redis-cli get b284fdd9-56f2-4824-a7b0-555d45d0a1b4:c236022d-a910-4a85-b27d-9c4a514448c4
"0"

You can also check the expiration on the key with the TTL (time to live) command.



➜ redis-cli TTL b284fdd9-56f2-4824-a7b0-555d45d0a1b4:c236022d-a910-4a85-b27d-9c4a514448c4
(integer) 258634

It looks like our time is just shy of 3 days, in seconds!

Conclusion

That was another big one! Thanks for hanging around! We're starting to reap the benefits of our application's architecture. Sure, it was a pain in the arse to set it up, but now I'm quite enjoying connecting everything!

Next time, we'll add middleware to set a handler timeout for each individual request. The tutorial won't involve modifying so many files, but it will be interesting in terms of working with Go and address some concurrency principles!

Hasta pronto!

11 - Account API Cleanup & Fixes

Jacob Goodwin — Sun, 29 Nov 2020 17:18:40 +0000

Today we're going to clean up some of the naming, variables, and parameters passed to a few methods. Nothing will be terribly intellectually challenging, but we will modify a lot of files.

If you find this boring or are uncertain about the changes made to the codebase, please check out the branch for this tutorial from the Github repository!

You might also find the video useful to make sure you don't miss any changes.

I also want to thank everybody who has subscribed to me feed and actually reads! Thanks for subjecting yourself to some seriously dense material. I hope you find a useful snippet here or there! 😊

Make Service and Repository Implementations Private

Our service and repository factories return interfaces. As an example, let's look at the NewUserService in ~/service/user_service.go

// UserService acts as a struct for injecting an implementation of UserRepository
// for use in service methods
type UserService struct {
    UserRepository model.UserRepository
}

// USConfig will hold repositories that will eventually be injected into this
// this service layer
type USConfig struct {
    UserRepository model.UserRepository
}

// NewUserService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewUserService(c *USConfig) model.UserService {
    return &UserService{
        UserRepository: c.UserRepository,
    }
}

We want to force external packages to access the service by means of the factory and its config struct (as opposed to instantiating UserService directly). Therefore, we will make the actual service and repository structs package private.

Update UserService

To get started with this, let's make the UserService implementation package private. We do this by making the service name lowercase, i.e. userService.

// userService acts as a struct for injecting an implementation of UserRepository
// for use in service methods
type userService struct {
    UserRepository model.UserRepository
}

// USConfig will hold repositories that will eventually be injected into this
// this service layer
type USConfig struct {
    UserRepository model.UserRepository
}

// NewUserService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewUserService(c *USConfig) model.UserService {
    return &userService{
        UserRepository: c.UserRepository,
    }
}

Furthermore, you will need to return this lower-case userService inside of NewUserService and to update all of the receiver methods to use userService.

As an example (you'll need to do this to all of the methods):

// Signup reaches our to a UserRepository to sign up the user.
// UserRepository Create should handle checking for user exists conflicts
func (s *userService) Signup(ctx context.Context, u *model.User) error {
    // omitting body...
}
// ... subsequent code omitted

Update TokenService

Let's do the same in TokenService, located in ~/service/token_service.go.

// tokenService used for injecting an implementation of TokenRepository
// for use in service methods along with keys and secrets for
// signing JWTs
type tokenService struct {
    // TokenRepository model.TokenRepository
    PrivKey       *rsa.PrivateKey
    PubKey        *rsa.PublicKey
    RefreshSecret string
}

// TSConfig will hold repositories that will eventually be injected into this
// this service layer
type TSConfig struct {
    // TokenRepository model.TokenRepository
    PrivKey       *rsa.PrivateKey
    PubKey        *rsa.PublicKey
    RefreshSecret string
}

// NewTokenService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewTokenService(c *TSConfig) model.TokenService {
    return &tokenService{
        PrivKey:       c.PrivKey,
        PubKey:        c.PubKey,
        RefreshSecret: c.RefreshSecret,
    }
}

Also remember to update the factory and receiver methods to use tokenService.

Update PGUserRepository

Let's do the same for our PGUserRepository.

// pgUserRepository is data/repository implementation
// of service layer UserRepository
type pgUserRepository struct {
    DB *sqlx.DB
}

// NewUserRepository is a factory for initializing User Repositories
func NewUserRepository(db *sqlx.DB) model.UserRepository {
    return &pgUserRepository{
        DB: db,
    }
}

Again, remember to update the factory and receiver methods to use pGUserRepository!

Context Fixes

I want to forward our Context down the call chain (handler -> service -> repository -> data sources), so that we can timeout any handler, along with calls all the way to the data sources, after a lengthy amount of time. To do this, though, we need to extract the Request().Context from off of the gin context.

We do this by passing the request context to service method calls inside of handler layer methods.

We need to update our current 2 handlers, Signup and Me.

Handler Updates

In ~/handler/me.go:


// ... code omitted

// use the Request Context
    ctx := c.Request.Context()
    u, err := h.UserService.Get(ctx, uid)

// ... code omitted

And in ~/handler/signup.go, update the calls to Signup and NewPairFromUser methods with the Context from the Request. We'll use the same context for both calls, though I can definitely see cases where this would not be prudent.

    ctx := c.Request.Context()
    err := h.UserService.Signup(ctx, u)

    // error checking here

    // create token pair as strings
    tokens, err := h.TokenService.NewPairFromUser(ctx, u, "")

Fix Handler Layer Tests

Now that we're using a plain Golang context to call service methods, we need to fix our signup_test.go and me_test.go.

You merely need to replace mock call arguments using:

mock.AnythingOfType("*gin.Context")

with the following in both test files.

mock.AnythingOfType("*context.emptyCtx")

We use *context.emptyCtx here, which is an alias for context.Background(), a non-nil, empty context (we don't have any context fields in the unit tests).

SQLX

I also want to use methods in SQLX that can receive the context. To do this, we'll update method calls on DB in the PGUserRepository to use "Context" versions of each method. As an example, let's look at the Create method and change r.DB.Get to r.DB.GetContext, remembering to pass in the context.

// Create reaches out to database SQLX api
func (r *pgUserRepository) Create(ctx context.Context, u *model.User) error {
    query := "INSERT INTO users (email, password) VALUES ($1, $2) RETURNING *"

  // update this method to "GetContext"
    if err := r.DB.GetContext(ctx, u, query, u.Email, u.Password); err != nil {
        // ... code omitted
    }
    // ... code omitted
}

Also make sure to update the Get call in FindByID to GetContext.

Update Environment Variables Usage

To make our app more "configurable," I want to make the following updates:

Pass the ACCOUNT_API_URL environment variable to the handler config.
Set the expiration duration for both JWTs as environment variables so a user can easily set them per their requirements or per environment.

Pass ACCOUNT_API_URL to Handler Config

Where possible, environment variables should be read in from the main package so the app can terminate early if there is any configuration error.

Let's add BaseURL as a field to our Config inside of ~/handler/handler.go. This way we can instantiate the handler with this BaseURL inside of the main package.

// Config will hold services that will eventually be injected into this
// handler layer on handler initialization
type Config struct {
    R            *gin.Engine
    UserService  model.UserService
    TokenService model.TokenService
    BaseURL      string
}

Then inside of the NewHandler function, we'll use this value to group our handlers.

// Create a group, or base url for all routes
    g := c.R.Group(c.BaseURL)

Let's read and pass this environment variable to NewHandler inside of injection.go.

// read in ACCOUNT_API_URL
    baseURL := os.Getenv("ACCOUNT_API_URL")

    handler.NewHandler(&handler.Config{
        R:            router,
        UserService:  userService,
        TokenService: tokenService,
        BaseURL:      baseURL,
    })

We have also instantiated our handler inside of unit tests, but we do not need to make any changes as the tests work with an empty string for the BaseURL. Omitted string fields in a struct will default to an empty string in Go.

JWT Expiration

We currently set the JWT expiration with hard-coded values inside of the token utility methods found in ~/service/tokens.go. I want to make these functions configurable. Let's add environment variables for expiration time in seconds in .env.dev.

ID_TOKEN_EXP=900 #15 mins in seconds
REFRESH_TOKEN_EXP=259200 #3 days in seconds

We'll read in and parse these environment variables the injection file in package main, as we did with the BaseURL. But first, we need to add these timeouts to our tokenService, TSConfig, and NewTokenService in ~/service/token_service.go. We'll add them as int64 type.

// tokenService used for injecting an implementation of TokenRepository
// for use in service methods along with keys and secrets for
// signing JWTs
type tokenService struct {
    // TokenRepository model.TokenRepository
    PrivKey               *rsa.PrivateKey
    PubKey                *rsa.PublicKey
    RefreshSecret         string
    IDExpirationSecs      int64
    RefreshExpirationSecs int64
}
// TSConfig will hold repositories that will eventually be injected into this
// this service layer
type TSConfig struct {
    // TokenRepository model.TokenRepository
    PrivKey               *rsa.PrivateKey
    PubKey                *rsa.PublicKey
    RefreshSecret         string
    IDExpirationSecs      int64
    RefreshExpirationSecs int64
}

// NewTokenService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewTokenService(c *TSConfig) model.TokenService {
    return &tokenService{
        PrivKey:               c.PrivKey,
        PubKey:                c.PubKey,
        RefreshSecret:         c.RefreshSecret,
        IDExpirationSecs:      c.IDExpirationSecs,
        RefreshExpirationSecs: c.RefreshExpirationSecs,
    }
}

Then inside of injection.go, we'll parse the environment variables and call NewTokenService with them:

// load expiration lengths from env variables and parse as int
    idTokenExp := os.Getenv("ID_TOKEN_EXP")
    refreshTokenExp := os.Getenv("REFRESH_TOKEN_EXP")

    idExp, err := strconv.ParseInt(idTokenExp, 0, 64)
    if err != nil {
        return nil, fmt.Errorf("could not parse ID_TOKEN_EXP as int: %w", err)
    }

    refreshExp, err := strconv.ParseInt(refreshTokenExp, 0, 64)
    if err != nil {
        return nil, fmt.Errorf("could not parse REFRESH_TOKEN_EXP as int: %w", err)
    }

    tokenService := service.NewTokenService(&service.TSConfig{
        PrivKey:               privKey,
        PubKey:                pubKey,
        RefreshSecret:         refreshSecret,
        IDExpirationSecs:      idExp,
        RefreshExpirationSecs: refreshExp,
    })

We use int64 as this is the type of int used golang's time package.

The final thing we need to do is accept these values in our utility functions, found in ~/service/token.go, for generating tokens.

Note that in generateRefreshToken we are working with time and time.Duration so we have to do some casting of our int64.

// generateIDToken generates an IDToken which is a jwt with myCustomClaims
// Could call this GenerateIDTokenString, but the signature makes this fairly clear
func generateIDToken(u *model.User, key *rsa.PrivateKey, exp int64) (string, error) {
    unixTime := time.Now().Unix()
  tokenExp := unixTime + exp
  // ... code omitted
}

// generateRefreshToken creates a refresh token
// The refresh token stores only the user's ID, a string
func generateRefreshToken(uid uuid.UUID, key string, exp int64) (*RefreshToken, error) {
    currentTime := time.Now()
    tokenExp := currentTime.Add(time.Duration(exp) * time.Second)
    tokenID, err := uuid.NewRandom() // v4 uuid in the google uuid lib

Let's now make sure to call these functions inside of token_service.go by passing in the expiration times!

// ... code omitted

  idToken, err := generateIDToken(u, s.PrivKey, s.IDExpirationSecs)

  // not showing code here

  refreshToken, err := generateRefreshToken(u.UID, s.RefreshSecret, s.RefreshExpirationSecs)

Update Token Service Test

Let's update our tests in ~/service/token_service_test.go to take in these new expiration parameters. We'll hard code the expiration durations in seconds ad the start of the test.

func TestNewPairFromUser(t *testing.T) {
    var idExp int64 = 15 * 60
    var refreshExp int64 = 3 * 24 * 2600

    priv, _ := ioutil.ReadFile("../rsa_private_test.pem")
    privKey, _ := jwt.ParseRSAPrivateKeyFromPEM(priv)
    pub, _ := ioutil.ReadFile("../rsa_public_test.pem")
    pubKey, _ := jwt.ParseRSAPublicKeyFromPEM(pub)
    secret := "anotsorandomtestsecret"

    // ... code omitted
}

We also need to update the expectations in this test to use the expiration values passed to NewTokenService. We do this by updating the expectedExpiresAt... statement for each token.

    // for idToken
    expiresAt := time.Unix(idTokenClaims.StandardClaims.ExpiresAt, 0)
        expectedExpiresAt := time.Now().Add(time.Duration(idExp) * time.Second)
    assert.WithinDuration(t, expectedExpiresAt, expiresAt, 5*time.Second)

    // for refreshToken
    expiresAt = time.Unix(refreshTokenClaims.StandardClaims.ExpiresAt, 0)
        expectedExpiresAt = time.Now().Add(time.Duration(refreshExp) * time.Second)
        assert.WithinDuration(t, expectedExpiresAt, expiresAt, 5*time.Second)

Accept Only JSON and multipart/form

Our struct tags for our model.User and model.TokenPair only support JSON. Currently, when we bind data with our bindData function in ~/handler/bind_data.go, we might receive form or XML data in the HTTP body. I want to make sure to send a clear error if the user sends anything other than application/json.

Let's add some code in ~/handler/bind_data at the top of the bindData function.

// send error if Content-Type != application/json
    if c.ContentType() != "application/json" {
        msg := fmt.Sprintf("%s only accepts Content-Type application/json", c.FullPath())

        err := apperrors.NewUnsupportedMediaType(msg)

        c.JSON(err.Status(), gin.H{
            "error": err,
        })
        return false
    }

Note that I added a new custom error type called UnsupportedMediaType. You can see this added in our ~/model/apperrors/apperrors.go.

// "Set" of valid errorTypes
const (
    Authorization        Type = "AUTHORIZATION"        // Authentication Failures -
    BadRequest           Type = "BADREQUEST"           // Validation errors / BadInput
    Conflict             Type = "CONFLICT"             // Already exists (eg, create account with existent email) - 409
    Internal             Type = "INTERNAL"             // Server (500) and fallback errors
    NotFound             Type = "NOTFOUND"             // For not finding resource
    PayloadTooLarge      Type = "PAYLOADTOOLARGE"      // for uploading tons of JSON, or an image over the limit - 413
    UnsupportedMediaType Type = "UNSUPPORTEDMEDIATYPE" // for http 415
)

// ... code omitted

// Status is a mapping errors to status codes
// Of course, this is somewhat redundant since
// our errors already map http status codes
func (e *Error) Status() int {
    switch e.Type {
    case Authorization:
        return http.StatusUnauthorized
    case BadRequest:
        return http.StatusBadRequest
    case Conflict:
        return http.StatusConflict
    case Internal:
        return http.StatusInternalServerError
    case NotFound:
        return http.StatusNotFound
    case PayloadTooLarge:
        return http.StatusRequestEntityTooLarge
    case UnsupportedMediaType:
        return http.StatusUnsupportedMediaType
    default:
        return http.StatusInternalServerError
    }
}

// ... code omitted

// NewUnsupportedMediaType to create an error for 415
func NewUnsupportedMediaType(reason string) *Error {
    return &Error{
        Type:    UnsupportedMediaType,
        Message: reason,
    }
}

Run Server

Having made the above fixes, you should now be able to run the application. If you would like to see examples of sending HTTP requests (including bodies in formats other than application/json), please check out the video!

docker-compose up

Run All Unit Tests

Let's change into the account folder, and make sure all of our tests pass.

cd account

go test -v ./...

Conclusion

I hope you're now in sync with me! Again, you can also go ahead and checkout the code for the lesson-11 branch on Github to have these updates.

Next time we'll get working on creating a TokenRepository for storing valid refresh tokens in Redis.

Hasta entonces, chau!

10 - Dependency Injection and App Demo

Jacob Goodwin — Mon, 23 Nov 2020 21:04:27 +0000

After about two months building and testing pieces of our application, it's time to put the pieces together and run the application again! I apologies for the tedium of the tutorials and commend you for hanging in there!

Here's the diagram we've been looking at over the last few tutorials, with check-marks next to things we've completed.

With these steps completed, we're now able to:

Instantiate a database connection to Postgres from within package main
Inject this connection into the UserRepository
Inject the UserRepository into the UserService
Inject the UserService and TokenService into the Handler layer.

Later we'll also inject a TokenRepository into the TokenService, but I wanted to run this dadgummed application before boring y'all to death!

If at any point you are confused about file structure or code, go to Github repository and check out the branch for the previous lesson to be in sync with me!

The final file structure should be:

If you prefer video, check out the video version below!

Migrating Users Table

Migrations, in my rough definition, are snapshots of each change you make to your database. For each migration, we create two files with SQL statements: one for updating the database, and another for reverting those updates. These files are usually prefixed/suffixed with a sequence or timestamp which indicate the order in which the migrations are applied.

Install golang-migrate CLI

I ended up using a tool called golang-migrate CLI to create and apply migrations. Check out the link for instructions on how to install the CLI on your OS. I'm just going to install the CLI directly on my machine as it's a bit simpler than setting it up inside of Docker.

After installing, make sure to check your installation by running:

➜ migrate --version
4.13.0

Migration Commands

Since remembering all of the command-line arguments for migrate can be difficult, let's update the Makefile with some commands for migrating a database.

.PHONY: keypair migrate-create migrate-up migrate-down migrate-force

PWD = $(shell pwd)
ACCTPATH = $(PWD)/account
MPATH = $(ACCTPATH)/migrations
PORT = 5432

# Default number of migrations to execute up or down
N = 1

# Create keypair should be in your file below

migrate-create:
    @echo "---Creating migration files---"
    migrate create -ext sql -dir $(MPATH) -seq -digits 5 $(NAME)

migrate-up:
    migrate -source file://$(MPATH) -database postgres://postgres:password@localhost:$(PORT)/postgres?sslmode=disable up $(N)

migrate-down:
    migrate -source file://$(MPATH) -database postgres://postgres:password@localhost:$(PORT)/postgres?sslmode=disable down $(N)

migrate-force:
    migrate -source file://$(MPATH) -database postgres://postgres:password@localhost:$(PORT)/postgres?sslmode=disable force $(VERSION)

What do these commands do?

migrate-create creates migration files in "sequential order" (-seq) inside of the MPATH folder. That folder will be a ~/migrations folder in the account application, which you should create.

The next commands are used for applying a discrete number of migrations, as defined by N. The default for N is 1. The migrate-up command applies database updates, and the migrate-down command reverts these updates.

migrate-force can be used to force a version number (in our case, the sequence number), which is often required if you have to fix an error in a previous migration. I'll admit I still struggle with this, and have to sort of "hack" or manually fix migration issues before forcing a version.

The good news is that the migrate CLI has reasonably readable warnings.

Create Migration Files

To create a migration, execute the following from the project root (where the Makefile is), where NAME describes the change you're making:

make migrate-create NAME=add_users_table

You'll now have 2 files in the migrations folder (which you have hopefully created). Notice the sequence number is added to the beginning of the file, and that the NAME is added to the end of the file.

Let's add the SQL statements for our first migration to 00001_add_users_table.up.sql.

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

CREATE TABLE IF NOT EXISTS users (
  uid uuid DEFAULT uuid_generate_v4() PRIMARY KEY,
  name VARCHAR NOT NULL DEFAULT '',
  email VARCHAR NOT NULL UNIQUE,
  password VARCHAR NOT NULL,
  image_url VARCHAR NOT NULL DEFAULT '',
  website VARCHAR NOT NULL DEFAULT ''
);

The first line adds an extension for creating UUIDs. When we create a user with email and password, UUIDs will be automatically generated for the uid column.

All other columns are non-nullable VARCHARs. We will use an empty value to represent no value (like no imageURL). We also require email addresses to be unique.

Let's add the "down" statements in 00001_add_users_table.down.sql.

DROP TABLE users;
DROP EXTENSION IF EXISTS "uuid-ossp";

The down file just drops the entire user table and then removes the UUID extension. While this looks simple, you should make sure to drop the table first, as the uid column depends on the UUID extension. Postgres will prevent us from deleting an extension if any remaining table column depends on it.

Update "Auth" to "Account" in Docker-Compose

We need to run our Postgres container to execute apply the migrations.

Before running the Postgres container, I want to update some of the naming in our docker-compose.yaml file at the root of our project.

postgres-account:
  ...
  volumes:
    - "pgdata_account:/var/lib/postgresql/data"
  ...
account:
  ...
  depends_on:
    - postgres-account
  ...
volumes:
  pgdata_account:

First, we'll change the auth suffix of postgres-auth to account.

Also make sure to change the depends_on value of the account service, and to change the volume name of postgres-account to pgdata_account.

With these updates, let's run this specific service.

docker-compose up postgres-account

You should now be able to log into your Postgres server with PSQL, PGAdmin, or your preferred SQL client. (If you check out the video, you can see this in PGAdmin.) If you're a boss and want to use PSQL, enter the following command, then entering "password" when prompted.

psql -h localhost -d postgres -U postgres

Apply First Migration

With the container running, open up another terminal in the project root.

Run make migrate-up.

This by default should run migration 00001..., if we were to have 2 migrations we could set N=2 to run multiple migrations.

Looking at the database, you should see a users table and a schema_migrations table, the latter tracking which migration we're on and whether the migration state is dirty. The dirty flag would be set, for example, if you created an SQL file with an error and tried to migrate it. You'd then need to resolve some issues manually and possibly run make migrate-force.

For kicks, let's run the down migration to make sure it works, and then reapply the up migration.

make migrate-down - make migrate-down

It might be worth going to the video to see me showing this inside of PGAdmin

Initialize DB and Inject

Queue the Hallelujah Chorus

Let's review the dependency injection flow (it may help to reference the big diagram at the top).

Initialize a connection to Postgres when our application starts up.
Provide or inject this connection into our UserRepository
Inject the UserRepository into the UserService
Initialize the TokenService
Inject the TokenService and UserService into the handler
Use the handler in the gin router/engine and start the application.

Then we'll be able to run our application and make HTTP requests to our signup endpoint.

Connect to Postgres

Lets exit out of our currently running docker-compose since we have already migrated our database table and because we need to run both our account application and Postgres containers.

docker-compose down or ctrl-c.

To keep our main file from getting unwieldy, I am going to create a ~/data_sources.go file. We'll also use this file later on for initializing our Redis container and Google Cloud Storage Client. We'll also add a close method on dataSources for shutting down all connections when the application closes.

package main

import (
    "fmt"
    "log"
    "os"

    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"
)

type dataSources struct {
    DB *sqlx.DB
}

// InitDS establishes connections to fields in dataSources
func initDS() (*dataSources, error) {
    log.Printf("Initializing data sources\n")
    // load env variables - we could pass these in,
    // but this is sort of just a top-level (main package)
    // helper function, so I'll just read them in here
    pgHost := os.Getenv("PG_HOST")
    pgPort := os.Getenv("PG_PORT")
    pgUser := os.Getenv("PG_USER")
    pgPassword := os.Getenv("PG_PASSWORD")
    pgDB := os.Getenv("PG_DB")
    pgSSL := os.Getenv("PG_SSL")

    pgConnString := fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=%s", pgHost, pgPort, pgUser, pgPassword, pgDB, pgSSL)

    log.Printf("Connecting to Postgresql\n")
    db, err := sqlx.Open("postgres", pgConnString)

    if err != nil {
        return nil, fmt.Errorf("error opening db: %w", err)
    }

    // Verify database connection is working
    if err := db.Ping(); err != nil {
        return nil, fmt.Errorf("error connecting to db: %w", err)
    }

    return &dataSources{
        DB: db,
    }, nil
}

// close to be used in graceful server shutdown
func (d *dataSources) close() error {
    if err := d.DB.Close(); err != nil {
        return fmt.Errorf("error closing Postgresql: %w", err)
    }

    return nil
}

An important note in this code is that we must import the Postgres driver, _ "github.com/lib/pq", in order to establish a connection to Postgres with the SQLX library.

We also read in environment variables with SQL connection information. I'll explain where these come from in a moment.

From the environment variables, we create a formatted connection string, and make sure we can ping the database server to verify our connection is working. This function returns a package private datasources struct which will be used as input for our dependency injection.

Environment variables

Let's go back to the environment variables.

Recall that in our docker-compose.yaml file, under our account service we have an env_file key which references our .env.dev file. Docker-compose will load environment variables from this file and make them available to our account application.

I actually want to move the .env.dev file into the account project folder. To continue following along, do so now!

Inside of docker-compose.yaml, let's update:

env_file: ./account/.env.dev

Then update the .env.dev file:

ACCOUNT_API_URL=/api/account
PG_HOST=postgres-account
PG_PORT=5432
PG_USER=postgres 
PG_PASSWORD=password 
PG_DB=postgres 
PG_SSL=disable
REFRESH_SECRET=areallynotsuperg00ds33cret
PRIV_KEY_FILE=./rsa_private_dev.pem
PUB_KEY_FILE=./rsa_public_dev.pem

In addition to Postgres connection string variables, I've also added paths to our RSA keys. We'll create the necessary keypair soon.

Back inside of main.go, let's initialize the datasources at the top of the function.

  log.Println("Starting server...")

    // initialize data sources
    ds, err := initDS()

    if err != nil {
        log.Fatalf("Unable to initialize data sources: %v\n", err)
    }

I also encourage you to add the close method at the end of main.go for shutting down the datasources.

  ...

  // The context is used to inform the server it has 5 seconds to finish
    // the request it is currently handling
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    // shutdown data sources
    if err := ds.close(); err != nil {
        log.Fatalf("A problem occurred gracefully shutting down data sources: %v\n", err)
    }

    // Shutdown server
    log.Println("Shutting down server...")
    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("Server forced to shutdown: %v\n", err)
  }

  ...

Now we can run docker-compose up, and our account application should successfully connect to the database (check account and postgres-account docker logs).

Dependency Injection

Let's create a file called injection.go at the root of the account application.

This file will be a little long, but aside from loading key files, it's mostly just making calls to the factories in our service and repository layers to make sure our handler gets access to the concrete implementations of our app's features.

package main

// your imports here

// will initialize a handler starting from data sources
// which inject into repository layer
// which inject into service layer
// which inject into handler layer
func inject(d *dataSources) (*gin.Engine, error) {
    log.Println("Injecting data sources")

    /*
     * repository layer
     */
    userRepository := repository.NewUserRepository(d.DB)

    /*
     * repository layer
     */
    userService := service.NewUserService(&service.USConfig{
        UserRepository: userRepository,
    })

    // load rsa keys
    privKeyFile := os.Getenv("PRIV_KEY_FILE")
    priv, err := ioutil.ReadFile(privKeyFile)

    if err != nil {
        return nil, fmt.Errorf("could not read private key pem file: %w", err)
    }

    privKey, err := jwt.ParseRSAPrivateKeyFromPEM(priv)

    if err != nil {
        return nil, fmt.Errorf("could not parse private key: %w", err)
    }

    pubKeyFile := os.Getenv("PUB_KEY_FILE")
    pub, err := ioutil.ReadFile(pubKeyFile)

    if err != nil {
        return nil, fmt.Errorf("could not read public key pem file: %w", err)
    }

    pubKey, err := jwt.ParseRSAPublicKeyFromPEM(pub)

    if err != nil {
        return nil, fmt.Errorf("could not parse public key: %w", err)
    }

    // load refresh token secret from env variable
    refreshSecret := os.Getenv("REFRESH_SECRET")

    tokenService := service.NewTokenService(&service.TSConfig{
        PrivKey:       privKey,
        PubKey:        pubKey,
        RefreshSecret: refreshSecret,
    })

    // initialize gin.Engine
    router := gin.Default()

    handler.NewHandler(&handler.Config{
        R:            router,
        UserService:  userService,
        TokenService: tokenService,
    })

    return router, nil
}

We'll initialize the router, or *gin.Engine, in this injection file instead of in the main file.

This means we can remove the initialization of gin in main.go, and instead call this inject function.

// router := gin.Default() <- remove this line

router, err := inject(ds)

if err != nil {
  log.Fatalf("Failure to inject data sources: %v\n", err)
}

srv := &http.Server{
  Addr:    ":8080",
  Handler: router,
}

In our environment file, .env.dev, we reference a "dev.pem" file. So let's create a second key-pair, though in reality we could just use test keys that we created in the last tutorial.

make create-keypair ENV=dev

Sending Requests

Let's send some requests to our application to make sure it works. I'll include the curl command here, but you can check out how to do this in Postman in the video.

curl --location --request POST 'http://malcorp.test/api/account/signup' \
--header 'Content-Type: application/json' \
--data-raw '{
    "email": "guy01@guy.com",
    "password":"validpassword123"
}'

You should get a response with an idToken and a refreshToken.

{
    "tokens": {
        "idToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVpZCI6ImY2ZTUxOTVhLWMzYjgtNGE4YS1hNTU0LWMzNzgxMGYxOTZmZiIsImVtYWlsIjoiZ3V5MDFAZ3V5LmNvbSIsIm5hbWUiOiIiLCJpbWFnZVVybCI6IiIsIndlYnNpdGUiOiIifSwiZXhwIjoxNjA1OTE0NDk2LCJpYXQiOjE2MDU5MTM1OTZ9.Q62fFJLNkTpG1uoB5ikMG_N2KgDXPNz12rSuXjOImVxW_JWditBeE3pYo6AC89cadKtMbDDW9M4D5sCT43LKLVpB7TUuWkGRxMTakXmF_aBg-bWaQMcQHPi9qzWooc_Hpd0zfFA06-mZNJTwFXQhY_p1rfj-L0BFEqFmm9xmBj3xHQaH14elKkzxA8f4RY9ihjpDio_uo_xGjDWfqbhX4rSt_C5OgX5YgfzgywACMILFZ--KWucTWbcBHTwvyJMzggqYjqkHoykWX1Py7aod96Oa-YGMh_mBE8pAZrnQ9-6I2O45DDUZa-4ZiK40u_0Vu9VGuF39fhnaV1SyIpTuiA",
        "refreshToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJmNmU1MTk1YS1jM2I4LTRhOGEtYTU1NC1jMzc4MTBmMTk2ZmYiLCJleHAiOjE2MDYxNzI3OTYsImp0aSI6ImEzYjQ5ODQ4LTZiZTItNDVkOC05YmYzLTJkYjFiOTE5NGE0YiIsImlhdCI6MTYwNTkxMzU5Nn0.7oZRdPmEWjQnYsa1u19IiAspO__Q3vJzArbE8V9oBLU"
    }
}

Sweetness! I recommend you copy these tokens into the jwt.io debugger to see if the payloads are what you expect. You can even copy the key files or refresh secret into the debugger to verify the token signatures!

If you try running the same request again, you should get a conflict error. since the user already exists.

{
    "error": {
        "type": "CONFLICT",
        "message": "resource: email with value: guy01@guy.com already exists"
    }
}

Conclusion

That's it for today. I'm not gonna lie, these last three tutorials were kinda of a b!@#$! For any of you that have read or watched even a segment here or there, thanks so much! That makes doing this worth while! I hope you've learned half as much as I have!

Next time, we'll do some cleanup. In two tutorials, we'll create a TokenRepository for storing refresh tokens in Redis. After that, we'll create a "handler timeout" middleware. While golang's http server has some nice settings for read and write timeouts between server and client, we want to create a timeout for the handler's themselves. While golang has this, it's not directly compatible with gin or with sending JSON responses.

¡Bueno! De nuevo, les agradezco. ¡Chau!

09 - Token Creation

Jacob Goodwin — Tue, 17 Nov 2020 16:58:59 +0000

We just implemented a Signup method in our UserService and a Create method in our UserRepository. We can see these completed features in the diagram below!

Referencing the diagram, we can see that we have two tasks to perform in order to send POST requests to our endpoint (/api/account/signup):

Create an implementation of the NewPairFromUser method of the TokenService.
Establish a connection to our development Postgres server from inside of our application, and perform dependency injection so that we can use concrete implementations of our services and repository. We should then be able to run docker-compose up to get our reload server environment running and make calls to our signup endpoint.

We'll complete number 1 today!

If at any point you are confused about file structure or code, go to Github repository and check out the branch for the previous lesson to be in sync with me!

If you prefer video, check out the video version below!

Create NewPairFromUser in TokenService

Let's go back to the Signup handler in ~/model/interfaces.go to remind ourselves of the call signature of NewPairFromUser.

type TokenService interface {
    NewPairFromUser(ctx context.Context, u *User, prevTokenID string) (*TokenPair, error)
}

The second parameter is a User. This user will be created in the Signup handler before we call NewPairFromUser. The final parameter, prevTokenID is for passing the ID of the user's current refresh token so that it can be invalidated before providing a new refresh token. We pass an empty string for this parameter when signing up a user as the user won't yet have a token.

We're not going handle this token ID for a couple of tutorials, as it requires setting up a Redis database and a TokenRepository.

Let's create a file, ~/service/token_service.go, for our TokenService implementation.

Inside, create a TokenService struct to define this service's dependencies.

// TokenService used for injecting an implementation of TokenRepository
// for use in service methods along with keys and secrets for
// signing JWTs
type TokenService struct {
    // TokenRepository model.TokenRepository
    PrivKey       *rsa.PrivateKey
    PubKey        *rsa.PublicKey
    RefreshSecret string
}

// TSConfig will hold repositories that will eventually be injected into this
// this service layer
type TSConfig struct {
    // TokenRepository model.TokenRepository
    PrivKey       *rsa.PrivateKey
    PubKey        *rsa.PublicKey
    RefreshSecret string
}

// NewTokenService is a factory function for
// initializing a UserService with its repository layer dependencies
func NewTokenService(c *TSConfig) model.TokenService {
    return &TokenService{
        PrivKey:       c.PrivKey,
        PubKey:        c.PubKey,
        RefreshSecret: c.RefreshSecret,
    }
}

// NewPairFromUser creates fresh id and refresh tokens for the current user
// If a previous token is included, the previous token is removed from
// the tokens repository
func (s *TokenService) NewPairFromUser(ctx context.Context, u *model.User, prevTokenID string) (*model.TokenPair, error) {
    panic("Not implemented")
}

Note: I've been making a mistake in creating the struct containers for the various services and repository. The TokenService should be package private, and therefore lowercase. The factory will return a model.TokenService interface. Making the TokenService struct package private will make it so the only way to instantiate the service is through the factory, and not by directly creating a struct. I'll clean this up in two tutorials!

What are the public and private keys, and refresh secret, in the TokenService? We'll be using JSON Web Tokens as our authorization mechanism. Let's do a little primer about what these are so we can then fill in the implementation details for the NewPairFromUser method.

JSON Web Token (JWT) Explanation

I highly recommend you check out the jwt.io Introduction as that's what I'm using for this explanation.

How They Are Used

For authorizing users to use resources in this application, or other applications/services within our organization. From jwt.io: "Once the user is logged in, each subsequent request will include the JWT, allowing the user to access routes, services, and resources that are permitted with that token."

Token Structure

It's just a 3-part string, xxxxx.yyyyy.zzzzz, where:

xxxxx represents a "header".
yyyyy represents a "payload".
zzzzz represents a "signature".

The header and payload are written in JSON, and then encoded as Base64URLs. When the JWT is encoded and signed, it looks something like:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c.

Header

The header tells what signing algorithm was used.

{
  "alg": "HS256",
  "typ": "JWT"
}

We're going to use RS256 (RSA) for signing the idToken and HS256 (HMAC) for signing the refreshToken.

Payload

The payload's JSON keys provide information about the user. There are some registered claims which most JWT libraries support by default (like "exp," the expiration time). You can add your own set of claims, some of which are standardized, or registered.

Signature

From jwt.io:

"The signature is used to verify the message wasn't changed along the way, and, in the case of tokens signed with a private key, it can also verify that the sender of the JWT is who it says it is."

As an example, let's say the payload has an isAdmin boolean key that is set to false. If someone were to steal this token and attempt to change isAdmin to true, the token would no longer be valid with the given signature and private signing key.

Note however, that the JWT is easily readable since Base64Urls can easily be read (for example, by copying it into a converter online or in libraries in most programming languages). Therefore, you do not want to include sensitive information in the token (like a National ID or SSN, date of birth, etc).

ID and RefreshTokens in Our App

Remember that we have defined a TokenPair to return upon successful signup. Let's take a look at another diagram of our authorization flow to understand how these are used.

One of the two purposes of this account application is to authorize users to access various services or applications in our domain.

When a user supplies a valid email and password combination (authenticated), the account application provides two tokens. Both of these tokens are JWTs, but they serve different purposes.

idToken

The idToken provides the information from the model.User we've been using in our application. Be careful not to return any sensitive user information as I previously mentioned.

The idToken is "short-lived." We'll make this token valid for 15 minutes to limit the time frame over which damage can be done if this token is stolen. You can make this longer or shorter in your application.

The idToken will be signed with an rsa private key, and verified with an rsa public key. We'll create this "key pair" soon. The public key will be provided to the various applications in our organization that need to verify that a user is authorized.

refreshToken

The refreshToken will be valid for a longer time frame. We'll set it to 3 days in our application. The purpose of this token is to improve the user-experience of the application so that they need not continually log in. If the user's idToken is expired, or if a client application loads for the first time, it may send its refreshToken to our Account Application to request a new idToken.

There are some major differences between these 2 tokens.

The refreshToken will not hold any user information beyond an id.
The refreshToken is only verified in the account application. We don't provide the secret to other applications or services. We'll also sign this token with a secret string (HMAC), instead of an RSA key. This is a symmetric signing algorithm, meaning the same key is used for signing and verifying JWTs.
We'll eventually store a list of valid refresh tokens in Redis. If a user feels their account is compromised, they can authenticate by logging in, and choose to log out of all devices. This will invalidate all of their refresh tokens. idTokens are not stored anywhere, but expire in 15 minutes.

An Important Note

There is a "friendly" debate about the failings of JWTs as well as how to store them client-side. I previously addressed this at the end of Tutorial 07.

Beyond using JWTs for authorization, you might consider requiring re-authentication for important actions like changing sensitive user details or performing financial transactions. There are also considerations such as using 2-factor authentication, which we won't implement.

Adding Token Utility Functions

We'll create some token utility functions in the same way we did for passwords.

Create ~/service/tokens.go.

We'll be working with a new package called github.com/dgrijalva/jwt-go. So make sure to include this in your imports and/or run go get github.com/dgrijalva/jwt-go.

Let's now add code for generating tokens. We'll add some functions for verifying tokens when we add signin and token refreshing features to our API.

idToken Generation

package service

import (
    "crypto/rsa"
    "log"
    "time"

    "github.com/dgrijalva/jwt-go"
    "github.com/google/uuid"
    "github.com/jacobsngoodwin/memrizr/account/model"
)

// IDTokenCustomClaims holds structure of jwt claims of idToken
type IDTokenCustomClaims struct {
    User *model.User `json:"user"`
    jwt.StandardClaims
}

// generateIDToken generates an IDToken which is a jwt with myCustomClaims
// Could call this GenerateIDTokenString, but the signature makes this fairly clear
func generateIDToken(u *model.User, key *rsa.PrivateKey) (string, error) {
    unixTime := time.Now().Unix()
    tokenExp := unixTime + 60*15 // 15 minutes from current time

    claims := IDTokenCustomClaims{
        User: u,
        StandardClaims: jwt.StandardClaims{
            IssuedAt:  unixTime,
            ExpiresAt: tokenExp,
        },
    }

    token := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)
    ss, err := token.SignedString(key)

    if err != nil {
        log.Println("Failed to sign id token string")
        return "", err
    }

    return ss, nil
}

Inside we do the following:

Define the structure of the payload portion the the JWT. If you recall, there are some predefined claims that JWTs can use, which our imported JWT library implements with jwt.StandardClaims. We also add properties of our User model as "public" or "custom" claims.
We set an expiry time of 15 minutes, using Unix time in seconds. (Don't let me forget to set this with an environment variable at some point 🤣).
We then create a Token, which is a type from the JWT library, then call the SignedString method to sign it with our RSA private key.
We return this signed string, ss.

refreshToken Generation

Refresh tokens will be generated in a similar manner with a few differences.

// RefreshToken holds the actual signed jwt string along with the ID
// We return the id so it can be used without re-parsing the JWT from signed string
type RefreshToken struct {
    SS        string
    ID        string
    ExpiresIn time.Duration
}

// RefreshTokenCustomClaims holds the payload of a refresh token
// This can be used to extract user id for subsequent
// application operations (IE, fetch user in Redis)
type RefreshTokenCustomClaims struct {
    UID uuid.UUID `json:"uid"`
    jwt.StandardClaims
}

// generateRefreshToken creates a refresh token
// The refresh token stores only the user's ID, a string
func generateRefreshToken(uid uuid.UUID, key string) (*RefreshToken, error) {
    currentTime := time.Now()
    tokenExp := currentTime.AddDate(0, 0, 3) // 3 days
    tokenID, err := uuid.NewRandom()         // v4 uuid in the google uuid lib

    if err != nil {
        log.Println("Failed to generate refresh token ID")
        return nil, err
    }

    claims := RefreshTokenCustomClaims{
        UID: uid,
        StandardClaims: jwt.StandardClaims{
            IssuedAt:  currentTime.Unix(),
            ExpiresAt: tokenExp.Unix(),
            Id:        tokenID.String(),
        },
    }

    token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
    ss, err := token.SignedString([]byte(key))

    if err != nil {
        log.Println("Failed to sign refresh token string")
        return nil, err
    }

    return &RefreshToken{
        SS:        ss,
        ID:        tokenID.String(),
        ExpiresIn: tokenExp.Sub(currentTime),
    }, nil
}

We will not return the signed string directly. Instead, we return a RefreshToken that we define in a struct. We do this to make reaching out to a TokenRepository (we'll implement this later) a bit easier. We also return an ExpiresIn field as we will use this for expiring old refresh tokens in Redis. We'll still only send the "signed string" in the HTTP response, as shown in our previous diagram.
We make use of the ID property (a stringified UUID) in standard claims. This id will be used to identify valid tokens in Redis.
We sign the token using the HS256 algorithm by using passing in our secret as a byte slice.

Now let's see how we can generate the RSA key pair!

Generating Public and Private Keys

I'm going to use the technique recommended by Google Cloud for generating 2048-bit RSA Key pairs. We'll then add these lines to a Makefile at our project root as follows:

You may need to download Make and OpenSSL to get this to work in Windows. I did it once, and it was a pain in the butt. Sorry I'm too lazy to write a tutorial now 😢. But it is possible, and probably easier than these tutorials!

Create a "Makefile" at project root (one directory above "account").

.PHONY: create-keypair

PWD = $(shell pwd)
ACCTPATH = $(PWD)/account

create-keypair:
    @echo "Creating an rsa 256 key pair"
    openssl genpkey -algorithm RSA -out $(ACCTPATH)/rsa_private_$(ENV).pem -pkeyopt rsa_keygen_bits:2048
    openssl rsa -in $(ACCTPATH)/rsa_private_$(ENV).pem -pubout -out $(ACCTPATH)/rsa_public_$(ENV).pem

Make sure to add *.pem to your .gitignore file at the project root to exclude all keys from being committed to your repository!

.env.dev
*.pem

We can then create keys with make create-keypair ENV=test, where you can replace the value of ENV with the environment for which you want to create the keypair.

Call Utility Methods in TokenService

With our utility functions completed, we can complete the implementation of NewPairFromUser! We'll generate both of our tokens and check for any errors. If everything works, we return the TokenPair.

func (s *TokenService) NewPairFromUser(ctx context.Context, u *model.User, prevTokenID string) (*model.TokenPair, error) {
    // No need to use a repository for idToken as it is unrelated to any data source
    idToken, err := generateIDToken(u, s.PrivKey)

    if err != nil {
        log.Printf("Error generating idToken for uid: %v. Error: %v\n", u.UID, err.Error())
        return nil, apperrors.NewInternal()
    }

    refreshToken, err := generateRefreshToken(u.UID, s.RefreshSecret)

    if err != nil {
        log.Printf("Error generating refreshToken for uid: %v. Error: %v\n", u.UID, err.Error())
        return nil, apperrors.NewInternal()
    }

    // TODO: store refresh tokens by calling TokenRepository methods

    return &model.TokenPair{
        IDToken:      idToken,
        RefreshToken: refreshToken.SS,
    }, nil
}

Add Test for NewTokenPairFromUser

Let's create the test file for this service, ~/service/token_service_test.go, and add a single test case to make sure we get a token pair when we call NewPairFromUser!

package service

import (
    "context"
    "io/ioutil"
    "testing"
    "time"

    "github.com/stretchr/testify/assert"

    "github.com/dgrijalva/jwt-go"
    "github.com/google/uuid"
    "github.com/jacobsngoodwin/memrizr-tutorial-script/account/model"
)

func TestNewPairFromUser(t *testing.T) {
    priv, _ := ioutil.ReadFile("../rsa_private_test.pem")
    privKey, _ := jwt.ParseRSAPrivateKeyFromPEM(priv)
    pub, _ := ioutil.ReadFile("../rsa_public_test.pem")
    pubKey, _ := jwt.ParseRSAPublicKeyFromPEM(pub)
    secret := "anotsorandomtestsecret"

    // instantiate a common token service to be used by all tests
    tokenService := NewTokenService(&TSConfig{
        PrivKey:       privKey,
        PubKey:        pubKey,
        RefreshSecret: secret,
    })

    // include password to make sure it is not serialized
    // since json tag is "-"
    uid, _ := uuid.NewRandom()
    u := &model.User{
        UID:      uid,
        Email:    "bob@bob.com",
        Password: "blarghedymcblarghface",
    }

    t.Run("Returns a token pair with values", func(t *testing.T) {
        ctx := context.TODO()
        tokenPair, err := tokenService.NewPairFromUser(ctx, u, "")
        assert.NoError(t, err)

        var s string
        assert.IsType(t, s, tokenPair.IDToken)

        // decode the Base64URL encoded string
        // simpler to use jwt library which is already imported
        idTokenClaims := &IDTokenCustomClaims{}

        _, err = jwt.ParseWithClaims(tokenPair.IDToken, idTokenClaims, func(token *jwt.Token) (interface{}, error) {
            return pubKey, nil
        })

        assert.NoError(t, err)

        // assert claims on idToken
        expectedClaims := []interface{}{
            u.UID,
            u.Email,
            u.Name,
            u.ImageURL,
            u.Website,
        }
        actualIDClaims := []interface{}{
            idTokenClaims.User.UID,
            idTokenClaims.User.Email,
            idTokenClaims.User.Name,
            idTokenClaims.User.ImageURL,
            idTokenClaims.User.Website,
        }

        assert.ElementsMatch(t, expectedClaims, actualIDClaims)
        assert.Empty(t, idTokenClaims.User.Password) // password should never be encoded to json

        expiresAt := time.Unix(idTokenClaims.StandardClaims.ExpiresAt, 0)
        expectedExpiresAt := time.Now().Add(15 * time.Minute)
        assert.WithinDuration(t, expectedExpiresAt, expiresAt, 5*time.Second)

        refreshTokenClaims := &RefreshTokenCustomClaims{}
        _, err = jwt.ParseWithClaims(tokenPair.RefreshToken, refreshTokenClaims, func(token *jwt.Token) (interface{}, error) {
            return []byte(secret), nil
        })

        assert.IsType(t, s, tokenPair.RefreshToken)

        // assert claims on refresh token
        assert.NoError(t, err)
        assert.Equal(t, u.UID, refreshTokenClaims.UID)

        expiresAt = time.Unix(refreshTokenClaims.StandardClaims.ExpiresAt, 0)
        expectedExpiresAt = time.Now().Add(3 * 24 * time.Hour)
        assert.WithinDuration(t, expectedExpiresAt, expiresAt, 5*time.Second)
    })
}

In the test we do the following:

Add setup code which reads in our test RSA keys and secret, initializes a TokenService, and creates an example User.
We assert that the returned idToken is a string.
We then parse the JWT (decode the Base64-encoded value), and verify that the token's "claims" contain the fields from the user.
However, we also assert that the password is not part of the claims as that would be very, very bad! Our User model's JSON tags are set so that the password should never be sent as JSON.
We check that the expiration time is within 5 seconds of 15 minutes.
We make the same steps for the refreshToken, except we don't have as many fields to verify, and assert an expiry time of 3 days.

Conclusion

Next time, we're going to create to run our Postgres database and create a users table with columns corresponding to the fields of our User model. After we successfully migrate create this table, we'll initialize a database connection and perform dependency injection.

With any luck, we'll then be able to fire up our application and sign up an actual user!

After that, I think we'll do a little bit of cleanup and maintenance before moving on to adding more functionality.

To be honest, it you get this part, the rest should begin to move a bit more quickly!

See you next time!

08 - Implement Signup in Service and Repository Layers

Jacob Goodwin — Thu, 12 Nov 2020 04:27:17 +0000

The last thing we did in our account application was to create and test a Signup handler. In this tutorial, we'll get work on the logic for signing up a user by adding service and repository layer method implementations.

Our goal over the next 3 tutorials will be to get the application working for signing up a user, including storing a newly created user in a database, and creating authorization tokens for a newly signed up, or a signed in user.

If at any point you are confused about file structure or code, go to Github repository and check out the branch for the previous lesson to be in sync with me!

If you prefer video, check out the video version below!

And for an overview of what the app we're building, see this.

Current Progress

The diagram below shows the methods in each application layer that we'll eventually need to implement, along with check marks next to those we've already completed. We started out by working on the me handler and UserService.Get method. We did this first just do get a feel for creating and unit testing handlers and services.

Now, though, we want to learn how to put these pieces together into a functioning web API application!

Looking at the code code inside of the Signup handler, we see that we reached out to UserService.Signup and TokenService.NewPairFromUser. We'll be working on creating concrete implementations of these methods in the next two tutorials so that we can send HTTP requests to our application and create real users inside of a PostgresQL database!

  // ...data binding and instantiation of User model...

    err := h.UserService.Signup(c, u)

    // ...error handling ...

    // ...create token pair as strings
    tokens, err := h.TokenService.NewPairFromUser(c, u, "")

In order to store users in an actual database, we'll need to implement a UserRepository, along with its Create method. This repository will accept a connection to a Postgres database. This Postgres database will be created inside of a docker container. I know it's been a while, but recall that we do have a docker-compose.yml file at the root of the project for defining the containers required to run our application.

In the next tutorial we'll work on creating tokens in the NewPairFromUser method. There's a little bit of context I'll need to provide on these token pairs, so it will require a second tutorial.

In the third tutorial, we're going to connect all of these pieces together by injecting a database connection into our UserRepository, injecting the UserRepository into the UserService, and then injecting the TokenService and UserService into the handler layer.

At that point, we'll finally be able to fire up our live reload environment and send requests to sign up users!

Are you intimidated yet? ... Or are you significantly more resilient than me?

After we have these pieces in place, we'll have implemented some of the most difficult parts of this project.

Add Create to UserRepository Interface

Let's add the expectation that our UserRepository should Create users inside of ~/model/interfaces.go.

// UserRepository defines methods the service layer expects
// any repository it interacts with to implement
type UserRepository interface {
    FindByID(ctx context.Context, uid uuid.UUID) (*User, error)
    Create(ctx context.Context, u *User) error
}

If a user is successfully created, it should update the underlying User passed to Create, otherwise an error should be returned.

Add Create to Mock Implementation

You're probably getting a warning now since our mockUserRepository does not yet implement the Get method. We add the implementation inside ~/mocks/user_repository.go. This will also allow us to test our UserService.Signup method. As always, check out the testify docs for more information about creating these mocks.

// Create is a mock for UserRepository Create
func (m *MockUserRepository) Create(ctx context.Context, u *model.User) error {
    ret := m.Called(ctx, u)

    var r0 error
    if ret.Get(0) != nil {
        r0 = ret.Get(0).(error)
    }

    return r0
}

UserService Signup

We've already scaffolded out this method inside of ~/service/user_service.go. Let's now add the implementation details.

If you look at the top of the file, you'll see that we've already provided access to a model.UserRepository in our UserService struct which will allow us to access the Create method.

// Signup reaches our to a UserRepository to sign up the user.
// UserRepository Create should handle checking for user exists conflicts
func (s *UserService) Signup(ctx context.Context, u *model.User) error {

    if err := s.UserRepository.Create(ctx, u); err != nil {
        return err
    }

    // If we get around to adding events, we'd Publish it here
    // err := s.EventsBroker.PublishUserUpdated(u, true)

    // if err != nil {
    //  return nil, apperrors.NewInternal()
    // }

    return nil
}

Wow! That was really easy!

Not so fast! We don't want to store passwords as plain text because that would make life really easy for hackers if our database were breached.

A few tutorials back, I said that we'd have to add more complexity and logic to some of our service methods, so here we go! We're going to add a functions for encrypting a password and another for comparing a supplied password with the encrypted password.

Before continuing, let me warn you that I'm not even a cryptography novice, but I'm going to add some code based on what I've found out there for generating a "salt" and then hashing the password with scrypt.

Password Utility Functions

Lets add a file called ~/service/passwords.go. You could also create a sub-package or a root-level package. There are many opinions on how to structure utility applications out there, but I'm just going to created a file inside of our service layer as we don't plan to use this code elsewhere (maybe this is a dumb decision).

package service

import (
    "crypto/rand"
    "encoding/hex"
    "fmt"
    "strings"

    "golang.org/x/crypto/scrypt"
)

func hashPassword(password string) (string, error) {
    // example for making salt - https://play.golang.org/p/_Aw6WeWC42I
    salt := make([]byte, 32)
    _, err := rand.Read(salt)
    if err != nil {
        return "", err
    }

    // using recommended cost parameters from - https://godoc.org/golang.org/x/crypto/scrypt
    shash, err := scrypt.Key([]byte(password), salt, 32768, 8, 1, 32)
    if err != nil {
        return "", err
    }

    // return hex-encoded string with salt appended to password
    hashedPW := fmt.Sprintf("%s.%s", hex.EncodeToString(shash), hex.EncodeToString(salt))

    return hashedPW, nil
}

func comparePasswords(storedPassword string, suppliedPassword string) (bool, error) {
    pwsalt := strings.Split(storedPassword, ".")

    // check supplied password salted with hash
    salt, err := hex.DecodeString(pwsalt[1])

    if err != nil {
        return false, fmt.Errorf("Unable to verify user password")
    }

    shash, err := scrypt.Key([]byte(suppliedPassword), salt, 32768, 8, 1, 32)

    return hex.EncodeToString(shash) == pwsalt[0], nil
}

The hashPassword function creates a hashed version of the provided password by:

Creating a 32-byte salt using golang's random package.
Creating a key, or hash, from a byte slice of the provided password, the salt we just created, and so-called cost parameters. The cost parameters provide a tradeoff between computational power required to encrypt the password and difficulty of recreating the encrypted password via brute-force attacks. We'll use the cost parameters recommended by the scrypt docs. The final parameter, 32, corresponds to the number of bytes for the key/hash.
In order to check a user's password when they try to login at a later date, we'll need the same salt we just generated. So what we'll do is store the salt with the hashed password. We do this by using fmt.Sprintf, appending the salt to the hashed password with a "." in between. We encode both the hash and salt as hexadecimal strings.

We also create a comparePassword function. This function:

Splits the string we created in the previous function to separate the salt and hashed password.
Hashes the supplied password, or what the user enters when logging in with the salt.
Compares the above value with the hashed password from the database. If the two match, we know the user has entered the correct password!

Applying hashPassword to Signup

Let's update our Signup method to use the hashed password!

func (s *UserService) Signup(ctx context.Context, u *model.User) error {
    pw, err := hashPassword(u.Password)

    if err != nil {
        log.Printf("Unable to signup user for email: %v\n", u.Email)
        return apperrors.NewInternal()
    }

    // now I realize why I originally used Signup(ctx, email, password)
    // then created a user. It's somewhat un-natural to mutate the user here
    u.Password = pw
    if err := s.UserRepository.Create(ctx, u); err != nil {
        return err
    }

    // ...

    return nil
}

That's it for this service method. We'll be able to test it by using the UserRepository mock we previously updated!

Unit Test UserService Signup

Unit tests for the Signup method have been added below and can be found in ~/service/user_service_test.go. I want to mention that I did learn something new in creating this test.

In the success case, we chain on a Run method to our testify mock call, which executes some code on the provided arguments to the Create method call. In this case, we add a UID to the user, which mocks actual repository behavior.

func TestSignup(t *testing.T) {
    t.Run("Success", func(t *testing.T) {
        uid, _ := uuid.NewRandom()

        mockUser := &model.User{
            Email:    "bob@bob.com",
            Password: "howdyhoneighbor!",
        }

        mockUserRepository := new(mocks.MockUserRepository)
        us := NewUserService(&USConfig{
            UserRepository: mockUserRepository,
        })

        // We can use Run method to modify the user when the Create method is called.
        //  We can then chain on a Return method to return no error
        mockUserRepository.
            On("Create", mock.AnythingOfType("*context.emptyCtx"), mockUser).
            Run(func(args mock.Arguments) {
                userArg := args.Get(1).(*model.User) // arg 0 is context, arg 1 is *User
                userArg.UID = uid
            }).Return(nil)

        ctx := context.TODO()
        err := us.Signup(ctx, mockUser)

        assert.NoError(t, err)

        // assert user now has a userID
        assert.Equal(t, uid, mockUser.UID)

        mockUserRepository.AssertExpectations(t)
    })

    t.Run("Error", func(t *testing.T) {
        mockUser := &model.User{
            Email:    "bob@bob.com",
            Password: "howdyhoneighbor!",
        }

        mockUserRepository := new(mocks.MockUserRepository)
        us := NewUserService(&USConfig{
            UserRepository: mockUserRepository,
        })

        mockErr := apperrors.NewConflict("email", mockUser.Email)

        // We can use Run method to modify the user when the Create method is called.
        //  We can then chain on a Return method to return no error
        mockUserRepository.
            On("Create", mock.AnythingOfType("*context.emptyCtx"), mockUser).
            Return(mockErr)

        ctx := context.TODO()
        err := us.Signup(ctx, mockUser)

        // assert error is error we response with in mock
        assert.EqualError(t, err, mockErr.Error())

        mockUserRepository.AssertExpectations(t)
    })
}

Make sure to run the tests, and also fail them, to make sure the tests work as expected.

go test -v ./service.

PGUserRepository Get and FindByID

We can now take a couple of different paths. We can either work on the TokenService or on the UserRepository. We'll work on the TokenService next time as it requires a decent amount of explanation.

Let's create a new folder called repository for our concrete repository implementations. Inside, add pg_user_repository.go, where we clearly name this implementation as one which uses Postgres.

Inside of this file, we're going to add an import to a new package called SQLX, which adds some utility methods on top of the built-in golang SQL library. This SQLX will be the only dependency of our UserRepository.

We'll also use a factory to initialize our UserRepository as in the service layer implementations.

package repository

import (
    "github.com/jmoiron/sqlx"
)

// PGUserRepository is data/repository implementation
// of service layer UserRepository
type PGUserRepository struct {
    DB *sqlx.DB
}

// NewUserRepository is a factory for initializing User Repositories
func NewUserRepository(db *sqlx.DB) model.UserRepository {
    return &PGUserRepository{
        DB: db,
    }
}

Also, I recently realized that my service and repository structs should probably be package private (starting with a lower-case letter). Our factories return an interface, and we don't want users to instantiate repository or service structs manually. That's the whole purpose of this factory pattern! We'll fix this in a few tutorials.

If your IDE or editor is any good, you should see an error because our implementation of a UserRepository does not have a Create method! We'll also need a FindByID method since we defined that method in our UserRepository interface while working on the "me" handler.

// Create reaches out to database SQLX api
func (r *PGUserRepository) Create(ctx context.Context, u *model.User) error {
    query := "INSERT INTO users (email, password) VALUES ($1, $2) RETURNING *"

    if err := r.DB.Get(u, query, u.Email, u.Password); err != nil {
        // check unique constraint
        if err, ok := err.(*pq.Error); ok && err.Code.Name() == "unique_violation" {
            log.Printf("Could not create a user with email: %v. Reason: %v\n", u.Email, err.Code.Name())
            return apperrors.NewConflict("email", u.Email)
        }

        log.Printf("Could not create a user with email: %v. Reason: %v\n", u.Email, err)
        return apperrors.NewInternal()
    }
    return nil
}

// FindByID fetches user by id
func (r *PGUserRepository) FindByID(ctx context.Context, uid uuid.UUID) (*model.User, error) {
    user := &model.User{}

    query := "SELECT * FROM users WHERE uid=$1"

    // we need to actually check errors as it could be something other than not found
    if err := r.DB.Get(user, query, uid); err != nil {
        return user, apperrors.NewNotFound("uid", uid.String())
    }

    return user, nil
}

In these methods, we create query strings, then execute them with application data by calling SQLX methods.

The Get method on the DB (sqlx connection) for inserting may seem a little strange, but this is a way we can use "Returning" in the SQL query and apply it to the user, u.

We then check for a very specific error returned by the SQL library when a unique violation occurs. If that is the case, let's return an apperrors.NewConflict error. Otherwise, we'll return an InternalServerError.

We'll configure the "email" column of our user database table to be unique in a couple of tutorials, so any attempt to add an existing email will produce a unique violation error.

Let's leave this for now. We'll inject a db connection, and in turn inject this repository into the UserService in a couple tutorials. I just wanted to give you an idea of what the repository layer will look like.

Setup a Docker Container for Postgres

Now I want to do something that doesn't necessarily fit in with this tutorial, but will save us some time later. Let's set up a Postgres container for our docker development environment.

We can setup a basic development Postgres container by adding the following to our docker-compose.yml file (at the project root).

Make sure to add the dependency key to the config of our account service. This makes sure to not spin up the container if there's an issue spinning up the postgres-auth container.

Also set up a named volume for the Postgres container's data. You can add this volume under the volumes key with no indentation (at the same indentation as the services key).


...

 postgres-auth:
    image: "postgres:alpine"
    environment:
      - POSTGRES_PASSWORD=password
    ports:
      - "5432:5432"
    #   Set a volume for data and initial sql script
    #   May configure initial db for future demo
    volumes:
      - "pgdata_auth:/var/lib/postgresql/data"
      # - ./init:/docker-entrypoint-initdb.d/
    command: ["postgres", "-c", "log_statement=all"]

...

  account:
    build:
      context: ./auth
      target: builder
    image: account # if we don't give image name, traefik won't create router 🤷‍♂️
    expose:
      - "8080" # seems necessary for Traefik to have internal expose of port
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.account.rule=Host(`wordmem.test`) && PathPrefix(`/api/account`)"
    environment:
      - ENV=dev
    volumes:
      - ./auth:/go/src/app
    depends_on:
      - postgres-auth

volumes:
  pgdata_auth:

Let's make sure we can run this.

cd .. to get to the root folder and run

docker-compose up.

Hopefully you get logs showing your account and postgres containers up and running!

)

Conclusion

Alright, I'm going to make an abrupt stop here. Thanks again for joining along!

Next time we're going to work on creating our idToken (which also serves as an access token), as well as a refreshToken. We'll describe the differences between these tokens and how to create them.

Until then, ¡cuídense!