DEV Community: Matija Krajnik

Go modules and private GIT repositories

Matija Krajnik — Sat, 26 Jun 2021 18:00:34 +0000

Some time ago i stumbled upon an issue that was giving me a headache. I had to download and import private Go module, and as it turns out, that can be little tricky. I'm using SSH keys for authenticating with my repositories, so this solution is applied only for that type of authentication.
Of course, i scouted internet looking for solutions, and there are some examples you can find, but none was working for me. Until i realized that there is small but crucial difference depending on which Git repository hosting service you are using. I will show you how to do it for GitHub and Bitbucket. Of course, all examples here are using URLs to my private repositories, and you will have to change them to match yours.

I prepared two simple private repositories for this little demo. Both are essentially the same. They have one package named hello with function World() returning string Hello World!. They are located on:

GitHub - github.com/matijakrajnik/goprivatemodule
Bitbucket - bitbucket.org/matijakrajnik/goprivatebucket

Now, if i tried simply running go get like below, that will not work.

go get github.com/matijakrajnik/goprivatemodule
go get bitbucket.org/matijakrajnik/goprivatebucket

First thing i had to do is set GOPRIVATE environment variable. This variable contains a list of module path prefixes that are considered to be private, so go will not use proxy nor trying to validate checksum from public database.

To set that variable, run:

go env -w GOPRIVATE=github.com/matijakrajnik/goprivatemodule

It's also possible to use glob pattern to match all of your repositories:

go env -w GOPRIVATE=github.com/matijakrajnik/*

And even make comma separated list to include multiple values:

go env -w GOPRIVATE=github.com/matijakrajnik/*,bitbucket.org/matijakrajnik/*

Setting GOPRIVATE variable is only first step. When you run go get command, Go uses git for downloading module source files. And by default, git uses HTTPS, which will not work because terminal prompts are disabled and git don't have any credentials for authentication. So you will end up will error like this:

$ go get -v github.com/matijakrajnik/goprivatemodule
go get: module github.com/matijakrajnik/goprivatemodule: git ls-remote -q origin in /home/matija/go/pkg/mod/cache/vcs/f53d9a18ffd3065378124236da32538eaf1adb1addf29c1ac217b2540997b252: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

Fortunately, if you are using SSH keys for authentication, solution is very simple. You just need to configure git to use git instead of https.

For GitHub repository use:

git config --global url."git@github.com:matijakrajnik/goprivatemodule.git".insteadOf "https://github.com/matijakrajnik/goprivatemodule"

For Bitbucket repository use:

git config --global url."git@bitbucket.org:matijakrajnik/goprivatebucket".insteadOf "https://bitbucket.org/matijakrajnik/goprivatebucket"

If you look carefully at these two commands above, you can see that subtle little difference i mentioned at the beginning. Notice that in GitHub URL you have to include .git suffix, while for Bitbucket you can't use it. This can also vary from one git service implementation to another, so if you have problem, try to add or remove that .git suffix.

You can check git configuration file located in ~/.gitconfig, to check that commands above took effect. If everything is ok, you will see something like this:

[url "git@github.com:matijakrajnik/goprivatemodule.git"]
  insteadOf = https://github.com/matijakrajnik/goprivatemodule
[url "git@bitbucket.org:matijakrajnik/goprivatebucket"]
  insteadOf = https://bitbucket.org/matijakrajnik/goprivatebucket

With that setup i was able to download my private modules with:

go get github.com/matijakrajnik/goprivatemodule
go get bitbucket.org/matijakrajnik/goprivatebucket

And use them in my project:

package main

import (
  "fmt"

  "github.com/matijakrajnik/goprivatemodule/hello"
)

func main() {
  fmt.Println(hello.World())
}

package main

import (
  "fmt"

  "bitbucket.org/matijakrajnik/goprivatebucket/hello"
)

func main() {
  fmt.Println(hello.World())
}

Docker deploy

Matija Krajnik — Thu, 24 Jun 2021 18:44:54 +0000

Our server is finished and almost ready for deploy, which will be done using Docker. Notice that i said it's almost ready, so let's see what is missing. This whole time we used React development server which is listening on port 3000 and redirecting all requests to our backend on port 8080. That makes sense for development since it makes it easier to develop frontend and backend at the same time, as well as debugging frontend React app. But in production we don't need that and it makes much more sense to have only our backend server running and to serve static frontend files to client. So instead of starting React development server using npm start command, we will build optimized frontend files for production, using command npm run build inside of assets/ directory. This will create new directory assets/build/ along with all files needed for production. Now we also have to instruct our backend where to find these files to be able to serve them. That is simply done using command router.Use(static.Serve("/", static.LocalFile("./assets/build", true))). Of course, we want to do that only if server is started in prod environment, so we need to slightly update few files.

First, we will update Parse() function in internal/cli/cli.go to return environment value as a string:

func Parse() string {
  flag.Usage = usage
  env := flag.String("env", "dev", `Sets run environment. Possible values are "dev" and "prod"`)
  flag.Parse()
  logging.ConfigureLogger(*env)
  if *env == "prod" {
    logging.SetGinLogToFile()
  }
  return *env
}

Then we will update Config struct NewConfig() function to be able to receive and set environment value:

type Config struct {
  Host       string
  Port       string
  DbHost     string
  DbPort     string
  DbName     string
  DbUser     string
  DbPassword string
  JwtSecret  string
  Env        string
}

func NewConfig(env string) Config {
  ...
  return Config{
    Host:       host,
    Port:       port,
    DbHost:     dbHost,
    DbPort:     dbPort,
    DbName:     dbName,
    DbUser:     dbUser,
    DbPassword: dbPassword,
    JwtSecret:  jwtSecret,
    Env:        env,
  }
}

Now we can update internal/cli/main.go to receive env value form CLI, and send it to new configuration creation which will be used for starting server:

func main() {
  env := cli.Parse()
  server.Start(conf.NewConfig(env))
}

Next thing we have to do is update router to be able to receive configuration argument, and set it to serve static files if started in production mode:

package server

import (
  "net/http"
  "rgb/internal/conf"
  "rgb/internal/store"

  "github.com/gin-contrib/static"
  "github.com/gin-gonic/gin"
)

func setRouter(cfg conf.Config) *gin.Engine {
  // Creates default gin router with Logger and Recovery middleware already attached
  router := gin.Default()

  // Enables automatic redirection if the current route can't be matched but a
  // handler for the path with (without) the trailing slash exists.
  router.RedirectTrailingSlash = true

  // Serve static files to frontend if server is started in production environment
  if cfg.Env == "prod" {
    router.Use(static.Serve("/", static.LocalFile("./assets/build", true)))
  }

  // Create API route group
  api := router.Group("/api")
  api.Use(customErrors)
  {
    api.POST("/signup", gin.Bind(store.User{}), signUp)
    api.POST("/signin", gin.Bind(store.User{}), signIn)
  }

  authorized := api.Group("/")
  authorized.Use(authorization)
  {
    authorized.GET("/posts", indexPosts)
    authorized.POST("/posts", gin.Bind(store.Post{}), createPost)
    authorized.PUT("/posts", gin.Bind(store.Post{}), updatePost)
    authorized.DELETE("/posts/:id", deletePost)
  }

  router.NoRoute(func(ctx *gin.Context) { ctx.JSON(http.StatusNotFound, gin.H{}) })

  return router
}

Last line needing update is in migrations/main.go file. Just change

store.SetDBConnection(database.NewDBOptions(conf.NewConfig()))

store.SetDBConnection(database.NewDBOptions(conf.NewConfig("dev")))

Actually, that's not last thing. You will also have to update all tests that use configuration and router setup, but that's entirely up to you and it's dependent on what tests you have implemented.

Now everything is ready for Docker deploy. Docker is not in the scope of this guide, so i will not go into details about Dockerfile, .dockerignore and docker-compose.yml contents.

First we will create .dockerignore file in project root directory:

# This file
.dockerignore

# Git files
.git/
.gitignore

# VS Code config dir
.vscode/

# Docker configuration files
docker/

# Assets dependencies and built files
assets/build/
assets/node_modules/

# Log files
logs/

# Built binary
cmd/rgb/rgb

# ENV file
.env

# Readme file
README.md

Now create new directory docker/ with two files, Dockerfile and docker-compose.yml. Content of Dockerfile will be:

FROM node:16 AS frontendBuilder

# set app work dir
WORKDIR /rgb

# copy assets files to the container
COPY assets/ .

# set assets/ as work dir to build frontend static files
WORKDIR /rgb/assets
RUN npm install
RUN npm run build

FROM golang:1.16.3 AS backendBuilder

# set app work dir
WORKDIR /go/src/rgb

# copy all files to the container
COPY . .

# build app executable
RUN CGO_ENABLED=0 GOOS=linux go build -o cmd/rgb/rgb cmd/rgb/main.go

# build migrations executable
RUN CGO_ENABLED=0 GOOS=linux go build -o migrations/migrations migrations/*.go

FROM alpine:3.14

# Create a group and user deploy
RUN addgroup -S deploy && adduser -S deploy -G deploy

ARG ROOT_DIR=/home/deploy/rgb

WORKDIR ${ROOT_DIR}

RUN chown deploy:deploy ${ROOT_DIR}

# copy static assets file from frontend build
COPY --from=frontendBuilder --chown=deploy:deploy /rgb/build ./assets/build

# copy app and migrations executables from backend builder
COPY --from=backendBuilder --chown=deploy:deploy /go/src/rgb/migrations/migrations ./migrations/
COPY --from=backendBuilder --chown=deploy:deploy /go/src/rgb/cmd/rgb/rgb .

# set user deploy as current user
USER deploy

# start app
CMD [ "./rgb", "-env", "prod" ]

And content of docker-compose.yml is:

version: "3"
services:
  rgb:
    image: kramat/rgb
    env_file:
      - ../.env
    environment:
      RGB_DB_HOST: db
    depends_on:
      - db
    ports:
      - ${RGB_PORT}:${RGB_PORT}
  db:
    image: postgres
    environment:
      POSTGRES_USER: ${RGB_DB_USER}
      POSTGRES_PASSWORD: ${RGB_DB_PASSWORD}
      POSTGRES_DB: ${RGB_DB_NAME}
    ports:
      - ${RGB_DB_PORT}:${RGB_DB_PORT}
    volumes:
      - postgresql:/var/lib/postgresql/rgb
      - postgresql_data:/var/lib/postgresql/rgb/data
volumes:
  postgresql: {}
  postgresql_data: {}

All files required for Docker deploy are now ready, so let's see how to build Docker image and deploy it. First we will pull postgres image from official Docker containers repository:

docker pull postgres

Next step is to build rgb image. Inside of project root directory run (change DOCKER_ID with your own docker ID):

docker build -t DOCKER_ID/rgb -f docker/Dockerfile .

To create rgb and db containers with resources, run:

cd docker/
docker-compose up -d

That will start both containers, and you can check their status by running docker ps. Finally, we need to run migrations. Open shell in rgb container by running:

docker-compose run --rm rgb sh

Inside of container we are able to run migrations same as before:

cd migrations/
./migrations init
./migrations up

And we are finished. You can open localhost:8080 in your browser to check that everything is working as it should, which means you should be able to create account and add new post:

It was quite long for a simple web app, but we have completed this guide. Hopefully it was useful to some of you. If you have any questions, remarks, or you have found any issues, feel free to contact me in comments. Good luck to everyone and happy coding :)

Tests

Matija Krajnik — Thu, 24 Jun 2021 18:28:56 +0000

Writing unit and integration tests is important part of software development and it's something ideally done during development, but in this guide i dedicated one chapter for that since there are some prerequisites required before we can start to write tests. For instance, main thing to do is to create test database. This will be done by using already created development database schema.

We will start by creating new test configuration. Add function below to internal/conf/conf.go:

func NewTestConfig() Config {
  testConfig := NewConfig()
  testConfig.DbName = testConfig.DbName + "_test"
  return testConfig
}

This creates new configuration which is same as usual configuration, but with _test suffix appended to database name. So in our case, test database will be named rgb_test. To create test database connect to postgres database in the same way as explained in chapter 7 and run:

DROP DATABASE IF EXISTS rgb_test;
CREATE DATABASE rgb_test WITH TEMPLATE rgb;

This will create new database rgb_test using schema from rgb. This needs to be executed every time you change development database schema.

Every test case must be independent from others, so we should work with empty database for every test case. To do that, function to reset our test database will be created and called at the beginning of every test case. In this function we will define slice of all table names we wish to reset, clear all those tables, and reset their counters so all ID sequencing will start from 1 again. We can add that function to internal/store/store.go:

func ResetTestDatabase() {
  // Connect to test database
  SetDBConnection(database.NewDBOptions(conf.NewTestConfig()))

  // Empty all tables and restart sequence counters
  tables := []string{"users", "posts"}
  for _, table := range tables {
    _, err := db.Exec(fmt.Sprintf("DELETE FROM %s;", table))
    if err != nil {
      log.Panic().Err(err).Str("table", table).Msg("Error clearing test database")
    }

    _, err = db.Exec(fmt.Sprintf("ALTER SEQUENCE %s_id_seq RESTART;", table))
  }
}

One thing we will need to do in most of the tests is to setup test env and create new user. We don't want to repeat that in every test case. Let's create file internal/store/main_test.go and add helper functions:

package store

import (
  "rgb/internal/conf"
  "rgb/internal/store"

  "github.com/gin-gonic/gin"
)

func testSetup() *gin.Engine {
  gin.SetMode(gin.TestMode)
  store.ResetTestDatabase()
  cfg := conf.NewConfig("dev")
  jwtSetup(cfg)
  return setRouter(cfg)
}

func addTestUser() (*User, error) {
  user := &User{
    Username: "batman",
    Password: "secret123",
  }
  err := AddUser(user)
  return user, err
}

Now we can start adding tests. Let's create new file internal/store/users_test.go and create first test:

package store

import (
  "testing"

  "github.com/stretchr/testify/assert"
)

func TestAddUser(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)
  assert.Equal(t, 1, user.ID)
  assert.NotEmpty(t, user.Salt)
  assert.NotEmpty(t, user.HashedPassword)
}

Another test that we can add for user account creation is when user tries to create account with already existing username:

func TestAddUserWithExistingUsername(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)
  assert.Equal(t, 1, user.ID)

  user, err = addTestUser()
  assert.Error(t, err)
  assert.Equal(t, "Username already exists.", err.Error())
}

To test Authenticate() function, we will create 3 tests: successful authentication, authenticating with invalid username, and authenticating with invalid password:

func TestAuthenticateUser(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)

  authUser, err := Authenticate(user.Username, user.Password)
  assert.NoError(t, err)
  assert.Equal(t, user.ID, authUser.ID)
  assert.Equal(t, user.Username, authUser.Username)
  assert.Equal(t, user.Salt, authUser.Salt)
  assert.Equal(t, user.HashedPassword, authUser.HashedPassword)
  assert.Empty(t, authUser.Password)
}

func TestAuthenticateUserInvalidUsername(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)

  authUser, err := Authenticate("invalid", user.Password)
  assert.Error(t, err)
  assert.Nil(t, authUser)
}

func TestAuthenticateUserInvalidPassword(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)

  authUser, err := Authenticate(user.Username, "invalid")
  assert.Error(t, err)
  assert.Nil(t, authUser)
}

Finally, we will test FetchUser() function with 2 tests: successfull fetch, and fetching not existing user:

func TestFetchUser(t *testing.T) {
  testSetup()
  user, err := addTestUser()
  assert.NoError(t, err)

  fetchedUser, err := FetchUser(user.ID)
  assert.NoError(t, err)
  assert.Equal(t, user.ID, fetchedUser.ID)
  assert.Equal(t, user.Username, fetchedUser.Username)
  assert.Empty(t, fetchedUser.Password)
  assert.Equal(t, user.Salt, fetchedUser.Salt)
  assert.Equal(t, user.HashedPassword, fetchedUser.HashedPassword)
}

func TestFetchNotExistingUser(t *testing.T) {
  testSetup()

  fetchedUser, err := FetchUser(1)
  assert.Error(t, err)
  assert.Nil(t, fetchedUser)
  assert.Equal(t, "Not found.", err.Error())
}

With that we finished writing tests for internal/store/user.go. You can try to add tests for rest of the files by yourself for practice, or check my solution in RGB GitHub.

Test functions above are testing only database communication, but our router and handlers are not tested here. For that we will need another set of tests. For start, we should create few more helper functions. We will create new file internal/server/main_test.go:

package server

import (
  "bytes"
  "encoding/json"
  "net/http"
  "net/http/httptest"
  "rgb/internal/store"
  "strings"

  "github.com/gin-gonic/gin"
  "github.com/rs/zerolog/log"
)

func testSetup() *gin.Engine {
  gin.SetMode(gin.TestMode)
  store.ResetTestDatabase()
  jwtSetup()
  return setRouter()
}

func userJSON(user store.User) string {
  body, err := json.Marshal(map[string]interface{}{
    "Username": user.Username,
    "Password": user.Password,
  })
  if err != nil {
    log.Panic().Err(err).Msg("Error marshalling JSON body.")
  }
  return string(body)
}

func jsonRes(body *bytes.Buffer) map[string]interface{} {
  jsonValue := &map[string]interface{}{}
  err := json.Unmarshal(body.Bytes(), jsonValue)
  if err != nil {
    log.Panic().Err(err).Msg("Error unmarshalling JSON body.")
  }
  return *jsonValue
}

func performRequest(router *gin.Engine, method, path, body string) *httptest.ResponseRecorder {
  req, err := http.NewRequest(method, path, strings.NewReader(body))
  if err != nil {
    log.Panic().Err(err).Msg("Error creating new request")
  }
  rec := httptest.NewRecorder()
  req.Header.Add("Content-Type", "application/json")
  router.ServeHTTP(rec, req)
  return rec
}

Most of those functions are nothing new, except last one, performRequest(). In that function we are creating new request using http package and new recorder using httptest package. We also need to add Content-Type header with value application/json to our test request. We are now ready to serve that test request using passed router and record response with recorder. Let's now see how to use these functions in action. Create new file internal/server/user_test.go:

package server

import (
  "net/http"
  "rgb/internal/store"
  "testing"

  "github.com/stretchr/testify/assert"
)

func TestSignUp(t *testing.T) {
  router := testSetup()

  body := userJSON(store.User{
    Username: "batman",
    Password: "secret123",
  })
  rec := performRequest(router, "POST", "/api/signup", body)

  assert.Equal(t, http.StatusOK, rec.Code)
  assert.Equal(t, "Signed up successfully.", jsonRes(rec.Body)["msg"])
  assert.NotEmpty(t, jsonRes(rec.Body)["jwt"])
}

Again, i will not write rest of the tests here, so you can try to add it by yourself. If stuck, you can check RGB GitHub repo.

One thing to keep in mind is to run all test sequentially, without parallelism. They can affect each other if run simultaneously since it's expected for database to be empty for every test case. Go by default uses multiple goroutines to run tests if your machine has multiple cores. To make sure, only 1 goroutine is used, add -p 1 option. That means you should run tests using command:

go test -p 1 ./internal/...

Graceful shutdown

Matija Krajnik — Thu, 24 Jun 2021 18:13:17 +0000

If our server process gets signal to shut it down, we would like to finish the request currently handling before that happens. For that we will use graceful shutdown provided by Gin. This example is explained in detail on Gin GitHub with description in comments, so i will not go into details myself. Our internal/server/server.go will now look like this:

package server

import (
  "context"
  "errors"
  "net/http"
  "os"
  "os/signal"
  "rgb/internal/conf"
  "rgb/internal/database"
  "rgb/internal/store"
  "syscall"
  "time"

  "github.com/rs/zerolog/log"
)

const InternalServerError = "Something went wrong!"

func Start(cfg conf.Config) {
  jwtSetup(cfg)

  store.SetDBConnection(database.NewDBOptions(cfg))

  router := setRouter()

  server := &http.Server{
    Addr:    cfg.Host + ":" + cfg.Port,
    Handler: router,
  }

  // Initializing the server in a goroutine so that
  // it won't block the graceful shutdown handling below
  go func() {
    if err := server.ListenAndServe(); err != nil && errors.Is(err, http.ErrServerClosed) {
      log.Error().Err(err).Msg("Server ListenAndServe error")
    }
  }()

  // Wait for interrupt signal to gracefully shutdown the server with
  // a timeout of 5 seconds.
  quit := make(chan os.Signal)
  // kill (no param) default send syscall.SIGTERM
  // kill -2 is syscall.SIGINT
  // kill -9 is syscall.SIGKILL but can't be catch, so don't need add it
  signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
  <-quit
  log.Info().Msg("Shutting down server...")

  // The context is used to inform the server it has 5 seconds to finish
  // the request it is currently handling
  ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
  defer cancel()

  if err := server.Shutdown(ctx); err != nil {
    log.Fatal().Err(err).Msg("Server forced to shutdown")
  }

  log.Info().Msg("Server exiting.")
}

Custom database errors

Matija Krajnik — Thu, 24 Jun 2021 18:11:10 +0000

Custom validation errors are added in previous chapter, but what about database errors? If you try to create account with already existing username, you will get error ERROR #23505 duplicate key value violates unique constraint "users_username_key". Unfortunately, there is no validator involved here and pg module returns most of the errors as map[byte]string so this can be little tricky.

One way to do it is manually check for every error case by doing database query. For example, to check if user with given username already exists in database we could do this before trying to create new user:

func AddUser(user *User) error {
  err = db.Model(user).Where("username = ?", user.Username).Select()
  if err != nil {
    return errors.New("Username already exists.")
  }
  ...
}

Problem is that this can become quite tedious. It needs to be done for every error case in every function which communicates with database. And on top of that, we are unnecessarily multiplying database queries. In this simple case there will be now 2 database queries instead of 1 for every successful user creation. There is another way, and that is to try to do query once, and then parse error if it happens. And that is tricky part, since we need to handle every error type using regex to extract relevant data needed to create more user friendly custom error messages. So let's start. As mentioned, pg errors are mostly of type map[byte]string, so for this particular error when you try to create user account with already existing username, you will get map on picture below:

To extract relevant data, we will use fields 82 and 110. Error type will be read from field 82 and we will extract column name from field 110. Let's add these functions to internal/store/store.go:

func dbError(_err interface{}) error {
  if _err == nil {
    return nil
  }
  switch _err.(type) {
  case pg.Error:
    err := _err.(pg.Error)
    switch err.Field(82) {
    case "_bt_check_unique":
      return errors.New(extractColumnName(err.Field(110)) + " already exists.")
    }
  case error:
    err := _err.(error)
    switch err.Error() {
    case "pg: no rows in result set":
      return errors.New("Not found.")
    }
    return err
  }
  return errors.New(fmt.Sprint(_err))
}

func extractColumnName(text string) string {
  reg := regexp.MustCompile(`.+_(.+)_.+`)
  if reg.MatchString(text) {
    return strings.Title(reg.FindStringSubmatch(text)[1])
  }
  return "Unknown"
}

With that in place we can call this dbError() function from internal/store/users.go:

func AddUser(user *User) error {
  ...

  _, err = db.Model(user).Returning("*").Insert()
  if err != nil {
    log.Error().Err(err).Msg("Error inserting new user")
    return dbError(err)
  }
  return nil
}

If we now try to create new account with already existing username, we will get nice error message:

Of course, this is only the beginning. You need to handle every type of error separately, but that handling is now in one place, and there is no need for additional queries. You can try handle rest of the error cases you wish to handle, or check RGB GitHub repo for my solution.

Custom validation errors

Matija Krajnik — Thu, 24 Jun 2021 18:07:13 +0000

If you try to create new account using too short password, you will get error Key: 'User.Password' Error:Field validation for 'Password' failed on the 'min' tag. This is not really user friendly, so it should be changed for better user experience. Let's see how we can transform that to our own custom error messages. For that purpose we will create new Gin handler functions in internal/server/middleware.go file:

func customErrors(ctx *gin.Context) {
  ctx.Next()
  if len(ctx.Errors) > 0 {
    for _, err := range ctx.Errors {
      // Check error type
      switch err.Type {
      case gin.ErrorTypePublic:
        // Show public errors only if nothing has been written yet
        if !ctx.Writer.Written() {
          ctx.AbortWithStatusJSON(ctx.Writer.Status(), gin.H{"error": err.Error()})
        }
      case gin.ErrorTypeBind:
        errMap := make(map[string]string)
        if errs, ok := err.Err.(validator.ValidationErrors); ok {
          for _, fieldErr := range []validator.FieldError(errs) {
            errMap[fieldErr.Field()] = customValidationError(fieldErr)
          }
        }

        status := http.StatusBadRequest
        // Preserve current status
        if ctx.Writer.Status() != http.StatusOK {
          status = ctx.Writer.Status()
        }
        ctx.AbortWithStatusJSON(status, gin.H{"error": errMap})
      default:
        // Log other errors
        log.Error().Err(err.Err).Msg("Other error")
      }
    }

    // If there was no public or bind error, display default 500 message
    if !ctx.Writer.Written() {
      ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": InternalServerError})
    }
  }
}

func customValidationError(err validator.FieldError) string {
  switch err.Tag() {
  case "required":
    return fmt.Sprintf("%s is required.", err.Field())
  case "min":
    return fmt.Sprintf("%s must be longer than or equal %s characters.", err.Field(), err.Param())
  case "max":
    return fmt.Sprintf("%s cannot be longer than %s characters.", err.Field(), err.Param())
  default:
    return err.Error()
  }
}

Constant InternalServerError is defined in internal/server/server.go:

const InternalServerError = "Something went wrong!"

Let's use new Gin middleware in internal/server/router.go:

func setRouter() *gin.Engine {
  // Creates default gin router with Logger and Recovery middleware already attached
  router := gin.Default()

  // Enables automatic redirection if the current route can't be matched but a
  // handler for the path with (without) the trailing slash exists.
  router.RedirectTrailingSlash = true

  // Create API route group
  api := router.Group("/api")
  api.Use(customErrors)
  {
    api.POST("/signup", gin.Bind(store.User{}), signUp)
    api.POST("/signin", gin.Bind(store.User{}), signIn)
  }

  authorized := api.Group("/")
  authorized.Use(authorization)
  {
    authorized.GET("/posts", indexPosts)
    authorized.POST("/posts", createPost)
    authorized.PUT("/posts", updatePost)
    authorized.DELETE("/posts/:id", deletePost)
  }

  router.NoRoute(func(ctx *gin.Context) { ctx.JSON(http.StatusNotFound, gin.H{}) })

  return router
}

As you can see, we are now using customErrors middleware in api group. But that's not the only change. Notice updated routes for signing in and signing up:

api.POST("/signup", gin.Bind(store.User{}), signUp)
api.POST("/signin", gin.Bind(store.User{}), signIn)

With these changes, we are trying to bind request data before even hitting signUp and signIn handlers, which means that handlers will only be reached if form validations are passed. With setup like this, handlers don't need to think about binding errors, because there was none if handler is reached. With that in mind, let's update these 2 handlers:

func signUp(ctx *gin.Context) {
  user := ctx.MustGet(gin.BindKey).(*store.User)
  if err := store.AddUser(user); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{
    "msg": "Signed up successfully.",
    "jwt": generateJWT(user),
  })
}

func signIn(ctx *gin.Context) {
  user := ctx.MustGet(gin.BindKey).(*store.User)
  user, err := store.Authenticate(user.Username, user.Password)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "Sign in failed."})
    return
  }

  ctx.JSON(http.StatusOK, gin.H{
    "msg": "Signed in successfully.",
    "jwt": generateJWT(user),
  })
}

Our handlers are much simpler now and they are only handling database errors. If you again try to create account with too short username and password, you will see more readable and descriptive errors:

We can now do same thing with Post handlers. I will not show you the solution here, so you can try it yourself for practice, but you can find it on RGB GitHub repo.

Add user posts

Matija Krajnik — Thu, 24 Jun 2021 17:51:15 +0000

With authentication in place, it is time to start using it. We will need authentication to create, read, update and delete user's blog posts. Let's start by adding new database migration which will create required data table with columns. Create new migration file migrations/2_addPostsTable.go:

package main

import (
  "fmt"

  "github.com/go-pg/migrations/v8"
)

func init() {
  migrations.MustRegisterTx(func(db migrations.DB) error {
    fmt.Println("creating table posts...")
    _, err := db.Exec(`CREATE TABLE posts(
      id SERIAL PRIMARY KEY,
      title TEXT NOT NULL,
      content TEXT NOT NULL,
      created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
      modified_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
      user_id INT REFERENCES users ON DELETE CASCADE
    )`)
    return err
  }, func(db migrations.DB) error {
    fmt.Println("dropping table posts...")
    _, err := db.Exec(`DROP TABLE posts`)
    return err
  })
}

And then run migrations with:

cd migrations/
go run *.go up

Now we create structure to hold post data. We will also add field constraints for Title and Content. Add new file internal/store/posts.go:

package store

import "time"

type Post struct {
  ID         int
  Title      string    `binding:"required,min=3,max=50"`
  Content    string    `binding:"required,min=5,max=5000"`
  CreatedAt  time.Time
  ModifiedAt time.Time
  UserID     int `json:"-"`
}

User can have multiple blog posts, so we have to add has-many relation to User struct. In internal/store/users.go, edit User struct:

type User struct {
  ID             int
  Username       string `binding:"required,min=5,max=30"`
  Password       string `pg:"-" binding:"required,min=7,max=32"`
  HashedPassword []byte `json:"-"`
  Salt           []byte `json:"-"`
  CreatedAt      time.Time
  ModifiedAt     time.Time
  Posts          []*Post `json:"-" pg:"fk:user_id,rel:has-many,on_delete:CASCADE"`
}

Function that can insert new post entry in database will be implemented in internal/store/posts.go:

func AddPost(user *User, post *Post) error {
  post.UserID = user.ID
  _, err := db.Model(post).Returning("*").Insert()
  if err != nil {
    log.Error().Err(err).Msg("Error inserting new post")
  }
  return err
}

To create post, we will add new handler which will call the function above. Create new file internal/server/post.go:

package server

import (
  "net/http"
  "rgb/internal/store"

  "github.com/gin-gonic/gin"
)

func createPost(ctx *gin.Context) {
  post := new(store.Post)
  if err := ctx.Bind(post); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  user, err := currentUser(ctx)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  if err := store.AddPost(user, post); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{
    "msg":  "Post created successfully.",
    "data": post,
  })
}

Post creation handler is ready, so let's add new protected route for crating posts. In internal/server/router.go, we will create new group which will use authorization middleware we implemented in previous chapter. We will add route /posts with HTTP method POST to that protected group:

func setRouter() *gin.Engine {
  // Creates default gin router with Logger and Recovery middleware already attached
  router := gin.Default()

  // Enables automatic redirection if the current route can't be matched but a
  // handler for the path with (without) the trailing slash exists.
  router.RedirectTrailingSlash = true

  // Create API route group
  api := router.Group("/api")
  {
    api.POST("/signup", signUp)
    api.POST("/signin", signIn)
  }

  authorized := api.Group("/")
  authorized.Use(authorization)
  {
    authorized.POST("/posts", createPost)
  }

  router.NoRoute(func(ctx *gin.Context) { ctx.JSON(http.StatusNotFound, gin.H{}) })

  return router
}

The recipe is same for all other CRUD (Create, Read, Update, Delete) methods:

implement function to communicate with database for required action
implement Gin handler which will use function from step 1
add route with handler to router

We have Create part covered, so let's move on to the next method, Read. We will implement function that will fetch all user's posts from database in internal/store/posts.go:

func FetchUserPosts(user *User) error {
  err := db.Model(user).
    Relation("Posts", func(q *orm.Query) (*orm.Query, error) {
      return q.Order("id ASC"), nil
    }).
    Select()
  if err != nil {
    log.Error().Err(err).Msg("Error fetching user's posts")
  }
  return err
}

And in internal/server/post.go:

func indexPosts(ctx *gin.Context) {
  user, err := currentUser(ctx)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  if err := store.FetchUserPosts(user); err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{
    "msg":  "Posts fetched successfully.",
    "data": user.Posts,
  })
}

To Update post, add these 2 functions to internal/store/posts.go:

func FetchPost(id int) (*Post, error) {
  post := new(Post)
  post.ID = id
  err := db.Model(post).WherePK().Select()
  if err != nil {
    log.Error().Err(err).Msg("Error fetching post")
    return nil, err
  }
  return post, nil
}

func UpdatePost(post *Post) error {
  _, err := db.Model(post).WherePK().UpdateNotZero()
  if err != nil {
    log.Error().Err(err).Msg("Error updating post")
  }
  return err
}

And in internal/server/post.go:

func updatePost(ctx *gin.Context) {
  jsonPost := new(store.Post)
  if err := ctx.Bind(jsonPost); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  user, err := currentUser(ctx)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  dbPost, err := store.FetchPost(jsonPost.ID)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  if user.ID != dbPost.UserID {
    ctx.AbortWithStatusJSON(http.StatusForbidden, gin.H{"error": "Not authorized."})
    return
  }
  jsonPost.ModifiedAt = time.Now()
  if err := store.UpdatePost(jsonPost); err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{
    "msg":  "Post updated successfully.",
    "data": jsonPost,
  })
}

Finally, to Delete post add to internal/store.posts.go:

func DeletePost(post *Post) error {
  _, err := db.Model(post).WherePK().Delete()
  if err != nil {
    log.Error().Err(err).Msg("Error deleting post")
  }
  return err
}

And in internal/server/post.go:

func deletePost(ctx *gin.Context) {
  paramID := ctx.Param("id")
  id, err := strconv.Atoi(paramID)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": "Not valid ID."})
    return
  }
  user, err := currentUser(ctx)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  post, err := store.FetchPost(id)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  if user.ID != post.UserID {
    ctx.AbortWithStatusJSON(http.StatusForbidden, gin.H{"error": "Not authorized."})
    return
  }
  if err := store.DeletePost(post); err != nil {
    ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{"msg": "Post deleted successfully."})
}

One new thing that you can notice here is paramID := ctx.Param("id"). We are using that to extract ID param from URL path.

Let's add all those handlers to router:

func setRouter() *gin.Engine {
  // Creates default gin router with Logger and Recovery middleware already attached
  router := gin.Default()

  // Enables automatic redirection if the current route can't be matched but a
  // handler for the path with (without) the trailing slash exists.
  router.RedirectTrailingSlash = true

  // Create API route group
  api := router.Group("/api")
  {
    api.POST("/signup", signUp)
    api.POST("/signin", signIn)
  }

  authorized := api.Group("/")
  authorized.Use(authorization)
  {
    authorized.GET("/posts", indexPosts)
    authorized.POST("/posts", createPost)
    authorized.PUT("/posts", updatePost)
    authorized.DELETE("/posts/:id", deletePost)
  }

  router.NoRoute(func(ctx *gin.Context) { ctx.JSON(http.StatusNotFound, gin.H{}) })

  return router
}

If user has no posts yet, User.Posts field will be nil by default. This complicates things for frontend since it must check for nil value, so it would be better to use empty slice. For that we will use AfterSelectHook which will be executed every time after Select() is executed for User. That hook will be added to internal/store/users.go:

var _ pg.AfterSelectHook = (*User)(nil)

func (user *User) AfterSelect(ctx context.Context) error {
  if user.Posts == nil {
    user.Posts = []*Post{}
  }
  return nil
}

JWT authentication

Matija Krajnik — Thu, 24 Jun 2021 17:40:09 +0000

Authentication is one of the most important part of almost every web application. We must ensure that every user can create, read, update and delete only data for which it's authorized. For that purpose we will use JWT (JSON Web Token). Fortunately, there are various Golang modules specialized for this. One that will be used in this guide can be found in this GitHub repo. Current latest version is v3 which can ne installed by running go get github.com/cristalhq/jwt/v3.

Since we will need secret key for generating and verifying tokens, let's add line export RGB_JWT_SECRET=jwtSecret123 to our .env file. Of course, in production you would want to use some long randomly generated string.
Next thing we should do is add new variable to internal/conf/conf.go. We will add constant jwtSecretKey = "RGB_JWT_SECRET" with the rest of our constants and then add new field JwtSecret of type string to Config struct. Now we can read new env variable and add it inside of NewConfig() function:

const (
  hostKey       = "RGB_HOST"
  portKey       = "RGB_PORT"
  dbHostKey     = "RGB_DB_HOST"
  dbPortKey     = "RGB_DB_PORT"
  dbNameKey     = "RGB_DB_NAME"
  dbUserKey     = "RGB_DB_USER"
  dbPasswordKey = "RGB_DB_PASSWORD"
  jwtSecretKey  = "RGB_JWT_SECRET"
)

type Config struct {
  Host       string
  Port       string
  DbHost     string
  DbPort     string
  DbName     string
  DbUser     string
  DbPassword string
  JwtSecret  string
}

func NewConfig() Config {
  host, ok := os.LookupEnv(hostKey)
  if !ok || host == "" {
    logAndPanic(hostKey)
  }

  port, ok := os.LookupEnv(portKey)
  if !ok || port == "" {
    if _, err := strconv.Atoi(port); err != nil {
      logAndPanic(portKey)
    }
  }

  dbHost, ok := os.LookupEnv(dbHostKey)
  if !ok || dbHost == "" {
    logAndPanic(dbHostKey)
  }

  dbPort, ok := os.LookupEnv(dbPortKey)
  if !ok || dbPort == "" {
    if _, err := strconv.Atoi(dbPort); err != nil {
      logAndPanic(dbPortKey)
    }
  }

  dbName, ok := os.LookupEnv(dbNameKey)
  if !ok || dbName == "" {
    logAndPanic(dbNameKey)
  }

  dbUser, ok := os.LookupEnv(dbUserKey)
  if !ok || dbUser == "" {
    logAndPanic(dbUserKey)
  }

  dbPassword, ok := os.LookupEnv(dbPasswordKey)
  if !ok || dbPassword == "" {
    logAndPanic(dbPasswordKey)
  }

  jwtSecret, ok := os.LookupEnv(jwtSecretKey)
  if !ok || jwtSecret == "" {
    logAndPanic(jwtSecretKey)
  }

  return Config{
    Host:       host,
    Port:       port,
    DbHost:     dbHost,
    DbPort:     dbPort,
    DbName:     dbName,
    DbUser:     dbUser,
    DbPassword: dbPassword,
    JwtSecret:  jwtSecret,
  }
}

We will create new file internal/server/jwt.go:

package server

import (
  "rgb/internal/conf"

  "github.com/cristalhq/jwt/v3"
  "github.com/rs/zerolog/log"
)

var (
  jwtSigner   jwt.Signer
  jwtVerifier jwt.Verifier
)

func jwtSetup(conf conf.Config) {
  var err error
  key := []byte(conf.JwtSecret)

  jwtSigner, err = jwt.NewSignerHS(jwt.HS256, key)
  if err != nil {
    log.Panic().Err(err).Msg("Error creating JWT signer")
  }

  jwtVerifier, err = jwt.NewVerifierHS(jwt.HS256, key)
  if err != nil {
    log.Panic().Err(err).Msg("Error creating JWT verifier")
  }
}

Function jwtSetup() will only create signer and verifier that will later be used in authentication. Now we can call this function from internal/server/server/go when starting server:

package server

import (
  "rgb/internal/conf"
  "rgb/internal/database"
  "rgb/internal/store"
)

func Start(cfg conf.Config) {
  jwtSetup(cfg)

  store.SetDBConnection(database.NewDBOptions(cfg))

  router := setRouter()

  // Start listening and serving requests
  router.Run(":8080")
}

To generate tokens, we will create function in internal/server/jwt.go:

func generateJWT(user *store.User) string {
  claims := &jwt.RegisteredClaims{
    ID:        fmt.Sprint(user.ID),
    ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour * 24 * 7)),
  }
  builder := jwt.NewBuilder(jwtSigner)
  token, err := builder.Build(claims)
  if err != nil {
    log.Panic().Err(err).Msg("Error building JWT")
  }
  return token.String()
}

And then we will call it from internal/server/user.go instead of hardcoded string we had so far for testing purposes:

package server

import (
  "net/http"
  "rgb/internal/store"

  "github.com/gin-gonic/gin"
)

func signUp(ctx *gin.Context) {
  user := new(store.User)
  if err := ctx.Bind(user); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  if err := store.AddUser(user); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  ctx.JSON(http.StatusOK, gin.H{
    "msg": "Signed up successfully.",
    "jwt": generateJWT(user),
  })
}

func signIn(ctx *gin.Context) {
  user := new(store.User)
  if err := ctx.Bind(user); err != nil {
    ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
  }
  user, err := store.Authenticate(user.Username, user.Password)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "Sign in failed."})
    return
  }

  ctx.JSON(http.StatusOK, gin.H{
    "msg": "Signed in successfully.",
    "jwt": generateJWT(user),
  })
}

Let's test this by signing up or signing in through our frontend. Open browser dev tools and check signIn or signUp response. You can see that our backend now generated random JWT:

Token is now created in signIn and signUp handlers, which means we can verify it for all secured routes. For that we will first implement verifyJWT() function in internal/server/jwt.go. This function will receive token in the form of string, verify its signature, extract ID from claims and if everything is ok, user's ID will be returned as int:

func verifyJWT(tokenStr string) (int, error) {
  token, err := jwt.Parse([]byte(tokenStr))
  if err != nil {
    log.Error().Err(err).Str("tokenStr", tokenStr).Msg("Error parsing JWT")
    return 0, err
  }

  if err := jwtVerifier.Verify(token.Payload(), token.Signature()); err != nil {
    log.Error().Err(err).Msg("Error verifying token")
    return 0, err
  }

  var claims jwt.StandardClaims
  if err := json.Unmarshal(token.RawClaims(), &claims); err != nil {
    log.Error().Err(err).Msg("Error unmarshalling JWT claims")
    return 0, err
  }

  if notExpired := claims.IsValidAt(time.Now()); !notExpired {
    return 0, errors.New("Token expired.")
  }

  id, err := strconv.Atoi(claims.ID)
  if err != nil {
    log.Error().Err(err).Str("claims.ID", claims.ID).Msg("Error converting claims ID to number")
    return 0, errors.New("ID in token is not valid")
  }
  return id, err
}

Functions for generating and verifying are done, and with that we are almost ready to write Gin middleware for authorization. Before that, we will add function that will fetch user from database based on its ID. In internal/store/users.go, add function:

func FetchUser(id int) (*User, error) {
  user := new(User)
  user.ID = id
  err := db.Model(user).Returning("*").WherePK().Select()
  if err != nil {
    log.Error().Err(err).Msg("Error fetching user")
    return nil, err
  }
  return user, nil
}

It's time to create new file internal/server/middleware.go:

package server

import (
  "net/http"
  "rgb/internal/store"
  "strings"

  "github.com/gin-gonic/gin"
)

func authorization(ctx *gin.Context) {
  authHeader := ctx.GetHeader("Authorization")
  if authHeader == "" {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "Authorization header missing."})
    return
  }
  headerParts := strings.Split(authHeader, " ")
  if len(headerParts) != 2 {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "Authorization header format is not valid."})
    return
  }
  if headerParts[0] != "Bearer" {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "Authorization header is missing bearer part."})
    return
  }
  userID, err := verifyJWT(headerParts[1])
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": err.Error()})
    return
  }
  user, err := store.FetchUser(userID)
  if err != nil {
    ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": err.Error()})
    return
  }
  ctx.Set("user", user)
  ctx.Next()
}

Authorization middleware extracts token from Authorization header. It first checks if header exists, if it's in valid format, and then calls verifyJWT() function. If JWT verification passes, user's ID is returned. User with that ID is fetched from database and set as current user for this context.

Getting current user from context is something that we will need fairly often, so let's extract this into helper function:

func currentUser(ctx *gin.Context) (*store.User, error) {
  var err error
  _user, exists := ctx.Get("user")
  if !exists {
    err = errors.New("Current context user not set")
    log.Error().Err(err).Msg("")
    return nil, err
  }
  user, ok := _user.(*store.User)
  if !ok {
    err = errors.New("Context user is not valid type")
    log.Error().Err(err).Msg("")
    return nil, err
  }
  return user, nil
}

First we check if user is set for this context. If not, error is returned. Since ctx.Get() returns interface, we must check if value is of type *store.User. If not, error is returned. When both checks are passed, current user is returned from context.

Authorization middleware is now ready to use for protected routes, as we will see in next chapter.

Logging

Matija Krajnik — Thu, 24 Jun 2021 17:33:55 +0000

Logging is also pretty important part of most web applications because we usually want to know what requests are coming in, and what's more important, whether there are any unexpected errors. So, as you may have guessed, this section will introduce logging and i will show you how to setup logging, and how to separate logging in dev and prod environments. Now we will use -env flag added in previous section.

For logging we will use zerolog module, which you can get by running go get github.com/rs/zerolog/log.

Now let's create another directory internal/logging/ and file logging.go inside of it:

package logging

import (
  "fmt"
  "io"
  "io/ioutil"
  "os"
  "path/filepath"
  "strings"
  "time"

  "github.com/gin-gonic/gin"
  "github.com/rs/zerolog"
  "github.com/rs/zerolog/log"
)

const (
  logsDir = "logs"
  logName = "gin_production.log"
)

var logFilePath = filepath.Join(logsDir, logName)

func SetGinLogToFile() {
  gin.SetMode(gin.ReleaseMode)
  logFile, err := os.Create(logFilePath)
  if err != nil {
    log.Panic().Err(err).Msg("Error opening Gin log file")
  }
  gin.DefaultWriter = io.MultiWriter(logFile)
}

func ConfigureLogger(env string) {
  zerolog.SetGlobalLevel(zerolog.DebugLevel)
  switch env {
  case "dev":
    stdOutWriter := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "15:04:05.000"}
    logger := zerolog.New(stdOutWriter).With().Timestamp().Logger()
    log.Logger = logger
  case "prod":
    createLogDir()
    backupLastLog()
    logFile := openLogFile()
    logFileWriter := zerolog.ConsoleWriter{Out: logFile, NoColor: true, TimeFormat: "15:04:05.000"}
    logger := zerolog.New(logFileWriter).With().Timestamp().Logger()
    log.Logger = logger
  default:
    fmt.Printf("Env not valid: %s\n", env)
    os.Exit(2)
  }
}

func createLogDir() {
  if err := os.Mkdir(logsDir, 0744); err != nil && !os.IsExist(err) {
    log.Fatal().Err(err).Msg("Unable to create logs directory.")
  }
}

func backupLastLog() {
  timeStamp := time.Now().Format("20060201_15_04_05")
  base := strings.TrimSuffix(logName, filepath.Ext(logName))
  bkpLogName := base + "_" + timeStamp + "." + filepath.Ext(logName)
  bkpLogPath := filepath.Join(logsDir, bkpLogName)

  logFile, err := ioutil.ReadFile(logFilePath)
  if err != nil {
    if os.IsNotExist(err) {
      return
    }
    log.Panic().Err(err).Msg("Error reading log file for backup")
  }

  if err = ioutil.WriteFile(bkpLogPath, logFile, 0644); err != nil {
    log.Panic().Err(err).Msg("Error writing backup log file")
  }
}

func openLogFile() *os.File {
  logFile, err := os.OpenFile(logFilePath, os.O_WRONLY|os.O_CREATE, 0644)
  if err != nil {
    log.Panic().Err(err).Msg("Error while opening log file")
  }
  return logFile
}

func curentDir() string {
  path, err := os.Executable()
  if err != nil {
    log.Panic().Err(err).Msg("Can't get current directory.")
  }
  return filepath.Dir(path)
}

And then we can update internal/cli/cli.go to configure logging based on environment instead of just printing it:

package cli

import (
  "flag"
  "fmt"
  "os"

  "rgb/internal/logging"
)

func usage() {
  fmt.Print(`This program runs RGB backend server.

Usage:

rgb [arguments]

Supported arguments:

`)
  flag.PrintDefaults()
  os.Exit(1)
}

func Parse() {
  flag.Usage = usage
  env := flag.String("env", "dev", `Sets run environment. Possible values are "dev" and "prod"`)
  flag.Parse()
  logging.ConfigureLogger(*env)
  if *env == "prod" {
    logging.SetGinLogToFile()
  }
}

This seems like a lot of code, but it's quite simple. First we configure our logger based on environment. If env is dev we will log everything to stdout, while for prod environment we will log into file. When logging into file we will first create logs directory if needed and backup previous log, so we have fresh log for every server start. Of course, you can create different logic for log rotation which better suits your needs. Another thing we need to do in this case is to tell Gin to run in release mode, which will reduce unnecessary and noisy output, and then also set default Gin writer to write into log file. You can also do that inside of a prod case block, but since we actually have 2 different loggers (Gin's attached logger and our zerolog logger) i like to separate those 2 parts of code. It's just a personal preference and you can do it your way. With this set we can now start logging some errors. For example, let's update logAndPanic() function inside internal/conf/conf.go:

func logAndPanic(envVar string) {
  log.Panic().Str("envVar", envVar).Msg("ENV variable not set or value not valid")
}

We can also log if error happens while generating password salt inside of internal/store/users.go:

func GenerateSalt() ([]byte, error) {
  salt := make([]byte, 16)
  if _, err := rand.Read(salt); err != nil {
    log.Error().Err(err).Msg("Unable to create salt")
    return nil, err
  }
  return salt, nil
}

It's entirely up to you to log whatever you want. So feel free to update code with logging of your choice or check out RGB GitHub repo to see how i did it. Let's now move on to the next section.

Deploy and stop scripts

Matija Krajnik — Thu, 24 Jun 2021 17:28:58 +0000

Now that we moved our configuration to ENV, we need to source .env every time we restart our machine or shell session, which can become a bit tedious. To make our lives easier, we will create bash scripts for starting and stopping server. While at it, we will also separate deploy for dev and prod environments. Let's start by creating new directory internal/cli/ and file cli.go inside of it with following content:

package cli

import (
  "flag"
  "fmt"
  "os"
)

func usage() {
  fmt.Print(`This program runs RGB backend server.

Usage:

rgb [arguments]

Supported arguments:

`)
  flag.PrintDefaults()
  os.Exit(1)
}

func Parse() {
  flag.Usage = usage
  env := flag.String("env", "dev", `Sets run environment. Possible values are "dev" and "prod"`)
  flag.Parse()
  fmt.Println(*env)
}

This code is pretty straightforward. We set usage text that will be printed if app is started with some invalid options or arguments, after that we set default value and description for -env option. Finally, parse all CLI flags with flag.Parse(). For the moment we don't do anything smart with that value, we are only printing it. But that will change latter when we add logging. For now let's use this by updating cmd/rgb/main.go file:

package main

import (
  "rgb/internal/cli"
  "rgb/internal/conf"
  "rgb/internal/server"
)

func main() {
  cli.Parse()
  server.Start(conf.NewConfig())
}

Everything is now ready to write bash scripts that will be used for deploying and stopping our app. I will not go into details about bash commands and syntax but i will explain logic behind it. Lets first create another directory inside of our project root and name it simply scripts/. Inside of it we will add our first script, deploy.sh:

#! /bin/bash

# default ENV is dev
env=dev

while test $# -gt 0; do
  case "$1" in
    -env)
      shift
      if test $# -gt 0; then
        env=$1
      fi
      # shift
      ;;
    *)
    break
    ;;
  esac
done

cd ../../rgb
source .env
go build -o cmd/rgb/rgb cmd/rgb/main.go
cmd/rgb/rgb -env $env &

In the script above we first set env=dev which means that dev environment will be selected by default. After that we are reading arguments passed to this script, and if we found -env argument, we will set env variable to passed value of that argument. This is done inside of while loop. When env variable is set, change directory to project root, source ENV variables and then run go build -o cmd/rgb/rgb cmd/rgb/main.go. This will create executable file named cmd/rgb/rgb which we can run to start our server. When app is built, we are starting server with cmd/rgb/rgb -env $env & which passes value of env variable as -env flag to our server and then starts it in background because of $ flag.

Let's now create very simple stop.sh script, also inside of scripts/ directory:

#! /bin/bash

kill $(pidof rgb)

This script will find process ID of our rgb app and then kills that process.

Before running both scripts, you will need to make them executable:

chmod +x deploy.sh
chmod +x stop.sh

Now we can start and stop our server by simply positioning inside of scripts/ directory and running:

./deploy.sh
./stop.sh

Server configuration

Matija Krajnik — Thu, 24 Jun 2021 17:25:11 +0000

Currently we have hardcoded values for server address and port. Same goes for database options required for connecting to our database. That is obviously not a great solution, so let's extract this configuration to ENV variables and add logic to read values from there. First we will create new directory internal/conf/ and there we will create conf.go file with content:

package conf

import (
  "log"
  "os"
  "strconv"
)

const (
  hostKey       = "RGB_HOST"
  portKey       = "RGB_PORT"
  dbHostKey     = "RGB_DB_HOST"
  dbPortKey     = "RGB_DB_PORT"
  dbNameKey     = "RGB_DB_NAME"
  dbUserKey     = "RGB_DB_USER"
  dbPasswordKey = "RGB_DB_PASSWORD"
)

type Config struct {
  Host       string
  Port       string
  DbHost     string
  DbPort     string
  DbName     string
  DbUser     string
  DbPassword string
}

func NewConfig() Config {
  host, ok := os.LookupEnv(hostKey)
  if !ok || host == "" {
    logAndPanic(hostKey)
  }

  port, ok := os.LookupEnv(portKey)
  if !ok || port == "" {
    if _, err := strconv.Atoi(port); err != nil {
      logAndPanic(portKey)
    }
  }

  dbHost, ok := os.LookupEnv(dbHostKey)
  if !ok || dbHost == "" {
    logAndPanic(dbHostKey)
  }

  dbPort, ok := os.LookupEnv(dbPortKey)
  if !ok || dbPort == "" {
    if _, err := strconv.Atoi(dbPort); err != nil {
      logAndPanic(dbPortKey)
    }
  }

  dbName, ok := os.LookupEnv(dbNameKey)
  if !ok || dbName == "" {
    logAndPanic(dbNameKey)
  }

  dbUser, ok := os.LookupEnv(dbUserKey)
  if !ok || dbUser == "" {
    logAndPanic(dbUserKey)
  }

  dbPassword, ok := os.LookupEnv(dbPasswordKey)
  if !ok || dbPassword == "" {
    logAndPanic(dbPasswordKey)
  }

  return Config{
    Host:       host,
    Port:       port,
    DbHost:     dbHost,
    DbPort:     dbPort,
    DbName:     dbName,
    DbUser:     dbUser,
    DbPassword: dbPassword,
  }
}

func logAndPanic(envVar string) {
  log.Println("ENV variable not set or value not valid: ", envVar)
  panic(envVar)
}

With configuration logic implemented, all that is left to do is use it. For that we will need to slightly refactor our existing code. Our files will then look like this:

internal/database/database.go:

package database

import (
  "rgb/internal/conf"

  "github.com/go-pg/pg/v10"
)

func NewDBOptions(cfg conf.Config) *pg.Options {
  return &pg.Options{
    Addr:     cfg.DbHost + ":" + cfg.DbPort,
    Database: cfg.DbName,
    User:     cfg.DbUser,
    Password: cfg.DbPassword,
  }
}

internal/server/server.go:

package server

import (
  "rgb/internal/conf"
  "rgb/internal/database"
  "rgb/internal/store"
)

func Start(cfg conf.Config) {
  store.SetDBConnection(database.NewDBOptions(cfg))

  router := setRouter()

  // Start listening and serving requests
  router.Run(":8080")
}

cmd/rgb/main.go:

package main

import (
  "rgb/internal/conf"
  "rgb/internal/server"
)

func main() {
  server.Start(conf.NewConfig())
}

One more change is required in migrations/main.go file. Just import rgb/internal/conf package and change line

store.SetDBConnection(database.NewDBOptions())

store.SetDBConnection(database.NewDBOptions(conf.NewConfig()))

We are now ready to read ENV variables required for our configuration. But one more thing is missing. We need to feed those values to ENV. To do that let's create new file in root project directory named .env:

export RGB_HOST=0.0.0.0
export RGB_PORT=8080
export RGB_DB_HOST=localhost
export RGB_DB_PORT=5432
export RGB_DB_NAME=rgb
export RGB_DB_USER=postgres
export RGB_DB_PASSWORD=postgres

We can feed these values to ENV using source command. So, now we can start our server like this:

source .env
go run cmd/rgb/main.go

Hashing password

Matija Krajnik — Thu, 24 Jun 2021 17:20:49 +0000

In last section we have created users table in our database, but we are currently storing user's password in plain text. This is something we should never do, and instead we need to store hashed password with random salt. For that we will use golang/crypto library. First we need to expand our User structure:

 type User struct {
  ID             int
  Username       string `binding:"required,min=5,max=30"`
  Password       string `pg:"-" binding:"required,min=7,max=32"`
  HashedPassword []byte `json:"-"`
  Salt           []byte `json:"-"`
  CreatedAt      time.Time
  ModifiedAt     time.Time
}

There are few new things here. First, we added pg:"-" for password validation. What that means? Well, this simply defines that there is no password column in users database table. This field is still required because user needs to provide it, but it will not be saved in database. Instead of that, we will generate random Salt and use Password field to generate HashedPassword. Then we will store Salt and HashedPassword in database. When authenticating user with Username and Password, we can use Salt from database to calculate HashedPassword based on provided Password and then compare it with HashedPassword stored in database. We also added json:"-" for HashedPassword, and Salt fields so they will not be sent to fronted in JSON response and will also be ignored if sent to backend as JSON. User struct is now ready to go, but we also need to update our migration file:

package main

import (
  "fmt"

  "github.com/go-pg/migrations/v8"
)

func init() {
  migrations.MustRegisterTx(func(db migrations.DB) error {
    fmt.Println("creating table users...")
    _, err := db.Exec(`CREATE TABLE users(
      id SERIAL PRIMARY KEY,
      username TEXT NOT NULL UNIQUE,
      hashed_password BYTEA NOT NULL,
      salt BYTEA NOT NULL,
      created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
      modified_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
    )`)
    return err
  }, func(db migrations.DB) error {
    fmt.Println("dropping table users...")
    _, err := db.Exec(`DROP TABLE users`)
    return err
  })
}

When this is done we are ready to update rest of our internal/store/users.go:

package store

import (
  "crypto/rand"
  "time"

  "golang.org/x/crypto/bcrypt"
)

type User struct {
  ID             int
  Username       string `binding:"required,min=5,max=30"`
  Password       string `pg:"-" binding:"required,min=7,max=32"`
  HashedPassword []byte `json:"-"`
  Salt           []byte `json:"-"`
  CreatedAt      time.Time
  ModifiedAt     time.Time
}

func AddUser(user *User) error {
  salt, err := GenerateSalt()
  if err != nil {
    return err
  }
  toHash := append([]byte(user.Password), salt...)
  hashedPassword, err := bcrypt.GenerateFromPassword(toHash, bcrypt.DefaultCost)
  if err != nil {
    return err
  }

  user.Salt = salt
  user.HashedPassword = hashedPassword

  _, err = db.Model(user).Returning("*").Insert()
  if err != nil {
    return err
  }
  return err
}

func Authenticate(username, password string) (*User, error) {
  user := new(User)
  if err := db.Model(user).Where(
    "username = ?", username).Select(); err != nil {
    return nil, err
  }
  salted := append([]byte(password), user.Salt...)
  if err := bcrypt.CompareHashAndPassword(user.HashedPassword, salted); err != nil {
    return nil, err
  }
  return user, nil
}

func GenerateSalt() ([]byte, error) {
  salt := make([]byte, 16)
  if _, err := rand.Read(salt); err != nil {
    return nil, err
  }
  return salt, nil
}

Since we changed our database schema, now we also need to reset our database:

cd migrations/
go run *.go reset
go run *.go up

Now let's go to our frontend and create new account again. If we check our database now, we can see that there is no more password column, but there are salt and hashed_password columns: