DEV Community: Marcos Filho

Validating input JSON

Marcos Filho — Thu, 17 Aug 2023 17:12:44 +0000

All the codes that you will see in this post you can check here

In the last post we did the POST method to create books in our application and in this post we will put a validation in our presenter layer (or data transfer object) that represent json input in our REST API.

In this case we will use a validator lib that will help us in our process.

So let's install the lib

go get github.com/go-playground/validator/v10

In our case we need to add some tags in our presenter layer to explain what we will validate.

//cmd/http/presenter/book.go

type BookPersist struct {
    Title       string `json:"title" validate:"required"`
    Author      string `json:"author" validate:"required"`
    NumberPages int    `json:"numberPages" validate:"required"`
}

In our BookPersist struct we added a validate tag, in our example we just add a required value to turn required values in ours fields but you can look into validator documentation to check every value that you can use.

But we need to change some places in our code to not impact our handler or usecase.

In this moment we have a place to decode the json to struct in our request.go file in the BindJson method. Let's update our BindJson to check this validations for us.

//cmd/http/helper/request.go

func BindJson(r *http.Request, destination interface{}) error {
    e := json.NewDecoder(r.Body).Decode(destination)
    if e != nil {
        return ErrDecodeJson
    }
    return validateJsonFields(destination)
}

func validateJsonFields(input interface{}) error {
    validator := validator.New()
    if err := validator.Struct(input); err != nil {
        return err
    }
    return nil
}

The validateJsonFields will help us to centralize the validation for every presenter/DTO that pass here.

So the validator.New will instantiate the lib and validator.Struct(...) will do the magic for us and if our json not respect the validations fields that we put in the validate tag the error will be generated.

BindJson will call this method for every decode and it's very important to tell you, if your struct don't use the validate tag BindJson will continue work without any problem, good right?

But our application don't know how to respond the validations errors to our user with friendly messages. To solve that we need to change one more place. Yes, our DealWith(...) method that handle our errors in our handlers. So let's improve our DealWith method.

//cmd/http/helper/error.go

func DealWith(err error) HttpError {
    if ok, httpError := isValidationError(err); ok {
        return *httpError
    } else if errCode, ok := errorHandlerMap[err]; ok {
        return HttpError{Status: errCode, Description: err.Error()}
    } else {
        return HttpError{
            Status:      http.StatusInternalServerError,
            Description: "Internal error, please report to admin",
        }
    }
}

func isValidationError(err error) (bool, *HttpError) {
    v := &validator.ValidationErrors{}
    if errors.As(err, v) {
        validationErrors := HttpError{Status: http.StatusBadRequest, Description: ErrJsonValidation.Error()}
        for _, err := range err.(validator.ValidationErrors) {
            message := generateErrorMessage(err)
            validationErrors.Messages = append(validationErrors.Messages, message.Error())
        }
        return true, &validationErrors
    }
    return false, nil
}

func generateErrorMessage(err validator.FieldError) error {
    return fmt.Errorf("error: field validation for '%s' failed on the '%s' tag", err.Field(), err.Tag())
}

First we will create the isValidationError(...) where we will check if the error that come from handler error is generated from our new validator lib.

We iterate the range err.(validator.ValidationErrors) because our struct could throw more than one error per field and in this range we append the errors to result in one message with all validations.

The generateErrorMessage(...) it's just a method to result in a clear message.

And in the DealWith method we have a new checkup the
if ok, httpError := isValidationError(err); ok {...}
It's very important this condition come first because the validation error will not be considered a domain error, for this reason we will not need to register the ErrJsonValidation in the errorHandlerMap.

Right now our application know how to deal with validation messages from our validator lib without change any handler or usecase.

And now we will dispatch a curl to check the response

curl --location 'http://localhost:8080/v1/books' \
--header 'Content-Type: application/json' \
--data '{
    "title": "title",
    "numberPages": 200
}'

{"description":"error validating json","messages":["error: field validation for 'Author' failed on the 'required' tag"]}

Right now our application get the error messages from presenter layer and delivery with a friendly message to our user.

In the next post we will start to configure our infrastructure to add docker and docker-compose to change our in-memory repository to a Postgresql.

Creating book data

Marcos Filho — Wed, 28 Dec 2022 19:15:37 +0000

All the codes that you will see in this post you can check here

In this post, we will create an Http POST method to include our books in the application and validate the fields.

Let's go to our handler and create the new endpoint to include our books

//cmd/http/handler/book.go

func (b *BookHandler) Route(r chi.Router) {
    r.Get("/hello", b.hello)
    r.Get("/", b.getAll)
    r.Post("/", b.create)
}

Before we show the create method we need to create a struct to represent the data that will represent the request body to our handler. It's called data transfer object pattern or DTO. The DTO it's a struct that will be used to represent a data for every input data in our handler.

So we will create a new 'presenter' folder in the cmd/http folder. The name could be DTO folder or anything else that represent the DTO layer. I like to use presenter name.

//cmd/http/presenter/book.go

type BookPersist struct {
    Title       string `json:"title"`
    Author      string `json:"author"`
    NumberPages int    `json:"numberPages"`
}

func (b *BookPersist) ToDomain() book.Book {
    return book.Book{
        Title:       b.Title,
        Author:      b.Author,
        NumberPages: b.NumberPages,
    }
}

This struct represent the JSON that will be received in the post body. You could be thinking why I not using the book domain struct and it's a simple reason. If your endpoint need to add some fields that not represent information about your domain then you need to add these fields in your domain model and this is not a good decision to keep with your application more cohesive.

And the ToDomain() method it's a way to propagate only the domain data through the application.

Let's go to our create method.

//cmd/http/handler/book.go

var ErrDecodeJson = errors.New("error decoding json")

func (h *BookHandler) create(w http.ResponseWriter, r *http.Request) {
    dto := presenter.BookPersist{}

    e := json.NewDecoder(r.Body).Decode(&dto)
    if e != nil {
        helper.HandleError(w, ErrDecodeJson)
        return
    }
    entity := dto.ToDomain()
    if _, err := h.bookService.Save(entity); err != nil { 
        helper.HandleError(w, err)
    } else {
        w.WriteHeader(http.StatusCreated)
    }
}

First we just instantiate the dto and decode the json body to our struct. After that we convert our DTO to domain struct using ToDomain() to pass only domain data to our core service.

I don't like to pass DTO to service layer just to our service layer know only the domain data. If one day your presenter layer change your service layer won't be affected.

But we have a little problem right now, our HandleError don't know how to deal with ErrDecodeJson.. let's see what happen when we dispatch a invalid json to our handler.

curl -X POST http://localhost:8080/v1/books \
   -H 'Content-Type: application/json' \
   -d '{"title":"title","author":"author","numberPages":100,}'

{"description":"Internal error, please report to admin"}

That's not the best message to say json problem, for this reason we will register our ErrDecodeJson to our map.

To a better maintainability we will move the ErrDecodeJson to our http helper package.

//cmd/http/helper/error.go

var (
    errorHandlerMap = make(map[error]int)

    ErrDecodeJson = errors.New("error decoding json")
)

// register how to deal with errors here to HTTP layer
func init() {
    errorHandlerMap[ErrDecodeJson] = http.StatusBadRequest

}

...

Now we initialize our errorHandlerMap to know how to deal with ErrDecodeJson.

Let's see what happens when we dispatch the same request again.

curl -v -X POST http://localhost:8080/v1/books \                                                                   15:28:03
   -H 'Content-Type: application/json' \
   -d '{"title":"title","author":"author","numberPages":100,}'

< HTTP/1.1 400 Bad Request
{"description":"error decoding json"}%

Wow, now we explain to the user what error happens and send a Bad Request HTTP Error.

To finish, we will improve our decode process and the response only http code to helper package.

//cmd/http/helper/request.go
package helper

import (
    "encoding/json"
    "net/http"
)

func BindJson(r *http.Request, destination interface{}) error {
    e := json.NewDecoder(r.Body).Decode(destination)
    if e != nil {
        return ErrDecodeJson
    }
    return nil
}

and

//cmd/http/helper/response.go

func Response(w http.ResponseWriter, status int) {
    addDefaultHeaders(w)
    w.WriteHeader(status)
}

Now let's see how our create method looks like.

//cmd/http/handler/book.go

func (h *BookHandler) create(w http.ResponseWriter, r *http.Request) {
    dto := presenter.BookPersist{}
    if err := helper.BindJson(r, &dto); err != nil {
        helper.HandleError(w, err)
        return
    }

    if _, err := h.bookService.Save(dto.ToDomain()); err != nil {
        helper.HandleError(w, err)
    } else {
        helper.Response(w, http.StatusCreated)
    }
}

In the next post we will include some validations to our BookPersist and improve our inputs data to not receive a lot of invalid datas.

Handling our API errors

Marcos Filho — Mon, 26 Dec 2022 21:12:15 +0000

All the codes that you will see in this post you can check here

In the last post, we build a DealWith() method to handle our API errors, but we do nothing at the moment. In this post we will build how to deliver a user-friendly way to show the user our problem. In case the problem is not a registered error (an error that we build and is a part of our application), we will centralize our error stack to share with the developer's team where the problem is (file, method name, and line) to the team solve the possible problem.

To represent our registered errors we will create a error map (errorHandlerMap) to store an error as a key and a http code as a value.

//cmd/http/helper/error.go
//...

var (
    errorHandlerMap = make(map[error]int)
)
//...

func DealWith(err error) HttpError {
    if errCode, ok := errorHandlerMap[err]; ok {
        return HttpError{Status: errCode, Description: err.Error()}
    } else {
        return HttpError{
            Status:      http.StatusInternalServerError,
            Description: "Internal error, please report to admin",
        }
    }
}

Now you can see our DealWith method checks if our errorHandlerMap has an error key registered in our map. If the error that we want to deal with is registered, we will represent a specific Http Code with a specific message. Anything else will be designated as an unregistered error or an error that we don't know how to deal with at the moment.

This map will increase while we creating this app. Not every application error needs to be in this map, but if your error should be delivered with a user friendly message, then this error needs to be registered in this map with a specific http code.

If you see our code probably thinking that our map don't have any error registered, and you right. We will start to register some validations errors in the next post.

Less verbose, better maintenance

Marcos Filho — Wed, 13 Apr 2022 04:05:19 +0000

All the codes that you will see in this post you can check here

In the last post we delivered the content to response in our controller but the code it's not too readable. In this post, we will refact the code to be more readable and easy to deliver our content.

Inside cmd/http folder we will create a new folder called helper. This package will group functions to help the http layer to deal with request and response that will be used by any other handler in our project.

First of all, lets refact our book handler in cmd/http/book.go and see how our getAll will be

func (h *BookHandler) getAll(w http.ResponseWriter, r *http.Request) {
    if results, err := h.bookService.FindAll(); err != nil {
        helper.HandleError(w, err)
    } else {
        if results != nil && len(*results) > 0 {
            helper.JsonResponse(w, results, http.StatusOK)
        } else {
            helper.NoContent(w)
        }
    }
}

Now the helper will help us to deal with error and with the response data.

In the helper package we will create a response.go and will move the addDefaultHeaders and writeData presents in book handler to our response.go file.

3 - We will start to the more easy method here, the NoContent function.

//cmd/http/helper/response.go
//...
func NoContent(w http.ResponseWriter) {
    addDefaultHeaders(w)
    w.WriteHeader(http.StatusNoContent)
}

We just will add the default headers and send a Header to No Content Status.

2 - We will write the json data to response

//cmd/http/helper/response.go
//...
func JsonResponse(w http.ResponseWriter, data interface{}, httpStatus int) {
    addDefaultHeaders(w)
    w.WriteHeader(httpStatus)
    writeData(w, data)
}

Again, we just move the boilerplate code to write data to a new function in helper package.

1 - Now, we will refact our error method to improve the process.

//cmd/http/helper/response.go
//...
func HandleError(w http.ResponseWriter, err error) {
    addDefaultHeaders(w)
    httpError := DealWith(err)
    w.WriteHeader(httpError.Status)

    if httpError.Status != http.StatusNoContent {
        writeData(w, httpError)
    }
}

look that we have a new method called DealWith this method will check if the error is a registered error or an unexpected error to our application.

At the moment we will create an error.go inside helper folder to create the DealWith method for our code works without any error.

//cmd/http/helper/error.go
//...
type HttpError struct {
    Status      int      `json:"-"`
    Description string   `json:"description,omitempty"`
    Messages    []string `json:"messages,omitempty"`
}

func DealWith(err error) HttpError {

    return HttpError{
        Status:      http.StatusInternalServerError,
        Description: "Internal error, please report to admin",
    }
}

In the next post we will describe the DealWith method to know a better way to produce errors in http layer.

Getting a response!

Marcos Filho — Wed, 19 Jan 2022 03:06:34 +0000

In the last step, we show our service layer consuming the repository layer but we don't see the data.

In this post, we will keep covering the code to show you how we consume the service layer in the handler and mount the response.

Before demonstrating the handler code, we need to make some changes in the container/container.go. Now we need to instantiate the repository layer that we talk in the last step and pass this new repository into the service layer in the constructor

// ...
func Inject() Container {

    //stores - this is new
    bookStore := repository.NewBookStoreInMemory()

    //init services
    bs := service.NewBookService(bookStore)
// ...
}

Now, as you can see, we instantiate the repository and delivery the implementation to the service layer, Now our Container with services will have a service that communicates with a persistence layer (in memory at the moment).

Looking to our handler/book.go you will see new methods and our constructor with a new required information, the service layer.

package handler

import (
    "encoding/json"
    "net/http"

    "github.com/go-chi/chi"
    "github.com/maaarkin/hero-api-golang/internal/service"
)

type BookHandler struct {
    bookService service.BookService
}

func NewBookHandler(bs service.BookService) *BookHandler {
    return &BookHandler{bs}
}

func (b *BookHandler) Route(r chi.Router) {
    r.Get("/hello", b.hello)
    r.Get("/", b.getAll)
}

func (*BookHandler) hello(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("Hello World"))
}

func (h *BookHandler) getAll(w http.ResponseWriter, r *http.Request) {
    if results, err := h.bookService.FindAll(); err != nil {
        addDefaultHeaders(w)
        writeData(
            w,
            map[string]interface{}{
                "Status":      http.StatusInternalServerError,
                "Description": "Internal error, please report to admin",
            },
        )
    } else {
        if results != nil && len(*results) > 0 {
            addDefaultHeaders(w)
            w.WriteHeader(http.StatusOK)
            writeData(w, results)
        } else {
            addDefaultHeaders(w)
            w.WriteHeader(http.StatusNoContent)
        }
    }
}

func addDefaultHeaders(w http.ResponseWriter) {
    w.Header().Set("Content-Type", "application/json; charset=utf-8")
    w.Header().Set("Access-Control-Allow-Origin", "*")
    w.Header().Set("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept")
}

func writeData(w http.ResponseWriter, data interface{}) {
    if bytes, e := json.Marshal(data); e != nil {
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
    } else {
        if _, e := w.Write(bytes); e != nil {
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        }
    }
}

Now you will see three new method (getAll, addDefaultHeaders and writeData). The getAll is our new method to show all books that our service will provide to us. The addDefaultHeaders is just a helper function to group some headers necessary to our response being formated in json and the writeData is a helper function too that will helper us to write our data into the response.

The getAll is too simple, we call our service layer and provide a specific HTTP error code and a message error if our service returns an error. If our service does not throw any error we just validate if the result has some content and delivery a 200 Http Code (OK) with the content or delivery a 204 Http Code (No Content).

Now if you start the application and get the endpoint /v1/books, you will receive a json with two objects.

λ curl http://localhost:8080/v1/books
[{"Id":1,"Title":"Title 1","Author":"MarkMark","NumberPages":101},{"Id":2,"Title":"Title 2","Author":"MarkMark 2","NumberPages":203}]

But... I think our handler is too verbose at the moment.. in the next step we will refactor the code to be more readable.

See ya!

Add some data in memory

Marcos Filho — Fri, 03 Dec 2021 21:37:40 +0000

All the code's that you will see in this post you can check here

After we structure our project to pass the service into the handler constructor, now we can access the service layer in our rest handler.

But our service layer does nothing at the moment, in this first time we will create our repository/persistence layer to persist the information in memory and we can access the information in the service layer, doing the same process that we do between service/handler now we will do between service/repository.

The repository layer will have the responsibility to communicate with the database (in-memory initially) and return only the domain code to the service layer. In our project, only the service can access the repository. Usually, I like to do this because it's easier to maintain the project. When you have multiple places accessing the repository being more difficult to track some problems in the code.

The structure from this project in this post will looks like this

📦hero-api-golang
 ┣ 📂cmd
 ┃ ┗ 📂http
 ┃ ┃ ┣ 📂handler
 ┃ ┃ ┃ ┣ 📜book.go
 ┃ ┃ ┃ ┗ 📜router.go
 ┃ ┃ ┗ 📜main.go
 ┣ 📂internal
 ┃ ┣ 📂container
 ┃ ┃ ┗ 📜container.go
 ┃ ┣ 📂domain
 ┃ ┃ ┗ 📂book
 ┃ ┃ ┃ ┗ 📜model.go
 ┃ ┣ 📂repository
 ┃ ┃ ┣ 📜book.go
 ┃ ┃ ┗ 📜book_inmemory.go
 ┃ ┗ 📂service
 ┃ ┃ ┗ 📜book.go
 ┣ 📜.gitignore
 ┣ 📜go.mod
 ┣ 📜go.sum
 ┗ 📜README.md

Now you can see a new repository folder, here we will group the files that will represent our persistence layer to connect with our database and return a domain code to our service.

//book.go
package repository

import "github.com/maaarkin/hero-api-golang/internal/domain/book"

type BookStore interface {
    GetAll() (*[]book.Book, error)
    Get(id uint64) (*book.Book, error)
    Create(item book.Book) (uint64, error)
    Update(id uint64, item book.Book) (*book.Book, error)
    Delete(id uint64) error
}

Here we will represent the default CRUD that we can see in any post in the internet using this interface.

In this first step we will store our data in memory for this reason we have a inmemory file in this folder.

//book_inmemory.go
package repository

import (
    "errors"
    "sync"

    "github.com/maaarkin/hero-api-golang/internal/domain/book"
)

var (
    keyInstance = uint64(3)

    books = map[uint64]book.Book{
        1: {Id: 1, Title: "Title 1", Author: "MarkMark", NumberPages: 101},
        2: {Id: 2, Title: "Title 2", Author: "MarkMark 2", NumberPages: 203},
    }
)

type bookStoreInMemory struct {
    mu sync.Mutex
}

func NewBookStoreInMemory() BookStore {
    return &bookStoreInMemory{}
}

func (store bookStoreInMemory) GetAll() (*[]book.Book, error) {
    m := make([]book.Book, 0)

    for _, value := range books {
        m = append(m, value)
    }

    return &m, nil
}

func (store bookStoreInMemory) Get(id uint64) (*book.Book, error) {
    book, has := books[id]
    if !has {
        return nil, errors.New("No book in database")
    }
    return &book, nil
}

func (store bookStoreInMemory) Create(item book.Book) (uint64, error) {
    store.mu.Lock()
    defer store.mu.Unlock()

    keyInstance = keyInstance + 1
    books[keyInstance] = item

    return keyInstance, nil
}

func (store bookStoreInMemory) Update(id uint64, item book.Book) (*book.Book, error) {
    store.mu.Lock()
    defer store.mu.Unlock()

    books[keyInstance] = item

    return &item, nil
}

func (store bookStoreInMemory) Delete(id uint64) error {
    delete(books, id)
    return nil
}

to represent an in-memory database we will use a map to store our book data. You will see the mu sync.Mutex, we use this dude to create a "safe zone" in the places that we need to increase the number that will represent our Id Generator. If we don't use this lock in these places, a scenario with multiple request (concurrency) trying to add in the map, using the keyInstance probably will be overridden in the map. To create a "safe zone" at the moment that we store the keyInstance, we will use the lock and unlock method to guarantee our process to simulate the id generation.

Now we will access this repository in the service layer, for this reason, we will inject the repository interface into our service constructor.

//internal\service\book.go
//...

type bookServiceImpl struct {
    bookStore repository.BookStore
}

func NewBookService(bookStore repository.BookStore) BookService {
    return bookServiceImpl{bookStore}
}

//...

When you put this interface in the NewBookService() constructor you will see that your 'container.go' will break, because we describe that we need the repository interface in the constructor method and there is the place that we inject/instantiate our components. Then we will change the 'container.go' too.

//internal\container\container.go
//...
func Inject() Container {

    //stores
    bookStore := repository.NewBookStoreInMemory()

    //init services
    bs := service.NewBookService(bookStore)

    //...
}
//...

Now we can access the repository in the service layer because the container will inject the repository correctly. And now we will change the service method to use the repository correctly.

Now our service/book.go will look like this:

package service

import (
    "github.com/maaarkin/hero-api-golang/internal/domain/book"
    "github.com/maaarkin/hero-api-golang/internal/repository"
)

type BookService interface {
    Save(book book.Book) (*book.Book, error)
    FindAll() (*[]book.Book, error)
    FindById(id uint64) (*book.Book, error)
    Delete(id uint64) error
    Update(book book.Book) error
}

type bookServiceImpl struct {
    bookStore repository.BookStore
}

func NewBookService(bookStore repository.BookStore) BookService {
    return bookServiceImpl{bookStore}
}

func (bs bookServiceImpl) Save(book book.Book) (*book.Book, error) {
    id, err := bs.bookStore.Create(book)

    if err != nil {
        return nil, err
    }

    book.Id = id
    return &book, nil
}

func (bs bookServiceImpl) FindAll() (*[]book.Book, error) {
    return bs.bookStore.GetAll()
}

func (bs bookServiceImpl) FindById(id uint64) (*book.Book, error) {
    return bs.bookStore.Get(id)
}

func (bs bookServiceImpl) Delete(id uint64) error {
    return bs.bookStore.Delete(id)
}

func (bs bookServiceImpl) Update(book book.Book) error {
    _, err := bs.bookStore.Update(book.Id, book)

    if err != nil {
        return err
    }

    return nil
}

At this point, we can access the repository layer through the service layer but we don't have any REST endpoint to get any information from this service. In the next step we will improve our handler/book.go to get information from the service layer and delivery the JSON data.

Utilizando uma estrutura concorrente para o disparo de emails

Marcos Filho — Wed, 05 May 2021 20:17:29 +0000

Recentemente aqui na OLX no time de accounts, recebemos uma solicitação para o disparo de emails para alguns clientes com o intuito de enviar um formulário para a obtenção de dados que possam ajudar a melhorar a experiencia do usuário.

Como essa solicitação era um envio de formulário para um grupo de usuários muito específico, não havíamos um serviço que atendesse essa demanda, o que tínhamos em mãos era um relatório contendo todos os emails desse grupo para os quais deveriam ser entregues os emails.

Aqui na OLX possuímos um serviço interno que já realiza envio de emails através de uma API, facilitando o disparo de emails, dessa forma o processo basicamente se limita a ler de um arquivo que contem todos emails que precisam ser enviados e o consumo dessa API. Simples não?

A resposta é sim, o envio é muito simples. O problema é que a base de envio desses emails é o relatório descrito anteriormente que possui aproximadamente 200 mil registros. Isso significa que para cada registro é necessário disparar um email. Acontece que o algoritmo de envio é simples, basta percorrer todos os emails e para cada email consumir a API e entregar o email solicitado e seguir para o próximo email, o problema é que seguir com esse algoritmo bloqueante irá demorar e muito o processamento dessa solicitação e tínhamos uma necessidade de entregar todos esses emails em um período curto.

Para isso foi concebido uma prova de conceito de um algoritmo para termos ideia do tempo que iria demorar através desse algoritmo simples, e para construir esse algoritmo utilizei o golang como ferramenta, pois sabia do potencial de concorrência da linguagem.

Todo os códigos apresentados nesse material encontra-se no github.

Iniciando o Projeto

Para executar e entendermos esse projeto, será necessário clonar o projeto do github, abra um terminal e execute:

git clone https://github.com/MAAARKIN/notification-async.git

Estrutura

A estrutura do projeto estará descrita dessa forma

notification-async/
┣ basic - Algoritmo bloqueante
┃ ┗ basic.go
┣ concurrent - Algoritmo concorrente
┃ ┗ concurrent.go
┣ model - Modelo que armazenará os dados de entrada
┃ ┗ option.go
┣ MOCK_DATA.csv - arquivo de mock com 1000 registros de email
┣ MOCK_DATA100.csv - arquivo de mock com 100 registros de email
┣ README.md
┣ base.csv
┣ go.mod
┗ main.go - simples CLI que designará para qual algoritmo será direcionado

Main.go

Nosso main.go será o entrypoint do projeto, tendo o papel de ser o responsável decidir se o código a ser executado será o bloqueante ou o concorrente a partir de dados que serão digitados pelo usuário.

Além disso, por ser o entrypoint do nosso projeto, será ele o responsável por criar um contexto que será responsável por finalizar nosso processo bloqueante ou concorrente caso venhamos a querer o fim do programa ANTES do termino de leitura do arquivo.

package main

import (
    "context"
    "flag"
    "log"
    "os"
    "os/signal"

    "github.com/MAAARKIN/notification-async/basic"
    "github.com/MAAARKIN/notification-async/concurrent"
    "github.com/MAAARKIN/notification-async/model"
)

func main() {
    // create a context
    ctx, cancel := context.WithCancel(context.Background()) //1
    defer cancel() //2
    // that cancels at ctrl+C
    go onSignal(os.Interrupt, cancel) //3

    // parse command line arguments
    op := new(model.Options)

    flag.StringVar(&op.Filename, "filename", "", "src file")
    flag.StringVar(&op.Event, "event", "", "event name")
    numberOfWorkers := flag.Int("workers", 2, "concurrent workers")
    flag.BoolVar(&op.Async, "async", false, "if the process will be async")
    flag.Parse() //4

    // check arguments
    if op.Filename == "" {
        log.Fatal("filename required")
    }

    if op.Event == "" {
        log.Fatal("event required")
    }

    if op.Async {
        // check arguments
        if numberOfWorkers == nil {
            log.Fatal("workers required")
        }
        concurrent.Start(ctx, *op, *numberOfWorkers) //5
    } else {
        basic.Start(ctx, *op)
    }

}

func onSignal(s os.Signal, cancel func()) {
    c := make(chan os.Signal, 1)
    signal.Notify(c, s)
    <-c
    cancel()
}

Criaremos um contexto que terá como responsabilidade a propagação de um possível cancelamento precoce, caso o usuário não queira aguardar a leitura de todo o arquivo. Mais pra frente detalhar sobre o contexto;
A criação do contexto a partir da função .WithCancel retorna dois valores, o primeiro será o contexto criado, com o qual iremos propagar nos nossos algoritmos e o segundo é uma função que ao ser chamada, emite um evento de cancelamento através de um canal, permitindo que possamos escutar esse sinal e possamos finalizar nosso programa;
A função onSignal terá o papel de ser executada de forma assíncrona onde iremos informar que caso o sinal de os.Interrupt (um Ctrl + C, irá emitir esse sinal de interrupção) seja executado, iremos chamar a função de cancelamento. E nos dois algoritmo iremos escutar esse evento de forma que possamos interromper os algoritmos e finalizar o programa. Vale salientar que iremos descrever com mais calma o papel da palavra reservada go mais a frente no processo concorrente;
O flag.Parse() tem o papel de levar os dados inseridos da linha de comando para os parâmetros dentro da variável op e dessa forma utilizar a estrutura de modelo Options;
O usuário que informar na chamada do comando de inicialização do sistema utilizar o valor do parametro de -async como verdadeiro, será chamado o algoritmo concorrente, caso contrário iremos consumir o algoritmo bloqueante.

Bloqueando o processo

Tanto nosso algoritmo bloqueante como o concorrente possuirá duas funcionalidades, o worker e o start.

O worker terá como papel ser a funcionalidade que possuirá o bloco que desejamos executar a chamada, ou seja, caso você queira realizar qualquer outro processo, seu bloco de código deve estar dentro do worker. No nosso caso, iremos construir um payload padrão com o dado do evento que será passado na chamada do terminal e iremos colocar um temporizador de 100 milli segundos para simular uma chamada a uma API, optei por não colocar a chamada via API para não ficar mais verboso nem expor dados de endpoint internos da OLX, mas um http client em go é fácil de encontrar na internet.

O start terá o papel de ser a funcionalidade que inicializará o bloco de código respectivo a cada algoritmo, que no nosso caso será bloqueante e concorrente. No caso do algoritmo bloqueante iremos contabilizar o tempo total de processamento, inicializar o arquivo que será passado via linha de comando e percorrer todo o arquivo de forma bloqueante e entregar ao worker cada linha lida para que a função possa realizar seu papel.

package basic

import (
    "bufio"
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/MAAARKIN/notification-async/model"
)

func worker(url string, event string) {
    payload := map[string]interface{}{
        "event":       event,
        "destination": map[string]string{"email": url},
        "source":      event,
    }

    data, _ := json.Marshal(payload)

    //my http client post here
    time.Sleep(100 * time.Millisecond) //to simulate the api.

    out := fmt.Sprintf("send event to %v, payload to send %v", url, string(data)) // do somethingg useful.
    fmt.Println(out)
}

func Start(ctx context.Context, op model.Options) {
    start := time.Now()

    csvfile, err := os.Open(op.Filename) //1
    if err != nil {
        log.Fatal(err) //2
    }

    defer csvfile.Close() //3

    scanner := bufio.NewScanner(csvfile) //4
    scanner.Split(bufio.ScanLines) //5

    for scanner.Scan() { //6
        select { //7
        case <-ctx.Done(): //8
            fmt.Println("worker finish")
            os.Exit(0) //9
        default:
            worker(scanner.Text(), op.Event) //10
        }
    }

    fmt.Printf("\n%2fs", time.Since(start).Seconds())
    fmt.Println()
}

Utilizamos a funcionalidade para inicializar e ter acesso ao arquivo desejado;
Caso a leitura do arquivo tenha algum problema, iremos finalizar o programa, não faz sentido continuar se não houver dados a processar ou se o arquivo apresentar problemas;
A palavra reservada defer no golang te garantirá que seu código seja executado ANTES que o bloco termine, garantindo assim que fechemos o arquivo ao final do processo;
O Scanner é será uma estrutura de armazenamento dos dados do arquivo num buffer para que possamos realizar a leitura;
Queremos que o scanner realize o split (a quebra dos dados) a partir de cada linha do arquivo;
a funcionalidade Scan() retorna um booleano caso haja linhas a serem lidas, parando somente quando não houver mais linhas, retornando assim um valor false e consequentemente finalizando o laço;
A palavra reservada select é uma uma palavra da linguagem utilizada para se trabalhar com channel, nesse caso ele realiza o bloqueio do código aguardando algum dos casos serem executados, similar ao switch case de outras linguagens mas voltado para canais. Vale salientar que se não existir o default o select mantem o código bloqueado até o canal receber informação, e caso o default exista e nenhum dos cases for atendido, então o default será executado, permitindo assim que o código continue. Enquanto o evento de cancelamento não for disparado, o default será executado;
Canal do contexto que escutará o evento de cancelamento ser executado para processar o bloco de código;
Funcionalidade do go para realizar a finalização do código;
Execução da função worker junto do scanner.Text() que é utilizado para obter a linha atual do arquivo.

Dentro do projeto notification-async teremos dois arquivos distintos de .csv com emails falsos que foram criados para simular um cenário de envio de emails, para nossos exemplos, utilizaremos o MOCK_DATA100.csv que contem 100 registros de emails falsos para testarmos nosso código.

Para executar nosso algoritmo executaremos o seguinte código no terminal

go run main.go -filename=MOCK_DATA100.csv -event=some_event -async=false

Esse projeto recebe como parametro alguns dados, como o arquivo que iremos ler, um nome de evento que irá estar presente no conteúdo que simulará o body do nosso serviço (que substituímos por um temporizador de 100ms) e o parametro -async serve pra designar para que nosso código main solicite para que o algoritmo bloqueante seja executado.

Resultado Bloqueante

Todos os testes realizados em minha maquina pessoal estão sempre em uma media de 10 a 11 segundos. Vale salientar que isso pode mudar de acordo com a maquina mas caso ocorra a repetição dos testes, será possível você identificar que o valor sempre irá diferenciar nas casas dos milli segundos, mantendo um valor muito próximo dos segundos.

...
send event to brackstraw2n@xinhuanet.com, payload to send {"destination":{"email":"brackstraw2n@xinhuanet.com"},"event":"teste","source":"teste"}
send event to nnosworthy2o@github.io, payload to send {"destination":{"email":"nnosworthy2o@github.io"},"event":"teste","source":"teste"}
send event to esmiley2p@opensource.org, payload to send {"destination":{"email":"esmiley2p@opensource.org"},"event":"teste","source":"teste"}
send event to gkording2q@desdev.cn, payload to send {"destination":{"email":"gkording2q@desdev.cn"},"event":"teste","source":"teste"}
send event to hgunthorpe2r@blogs.com, payload to send {"destination":{"email":"hgunthorpe2r@blogs.com"},"event":"teste","source":"teste"}

10.429105s

Context

Como pode ser observado dentro da função start o primeiro parâmetro, o context.Context será utilizado nos algoritmos para realizar o papel de "olheiro" do nosso sistema, ou seja, através dele teremos a possibilidade de emitir um sinal a partir da linha de comando para que possamos parar o programa caso a gente queira. Este evento de parada poderá ser executado através do 'Ctrl+C' no terminal para que por algum motivo precisemos parar o sistema. Não iremos explicar a fundo sobre o context pois este não é o objetivo, mas saiba que através do canal context.Done() é possível escutar o evento de cancelamento.

Processo Concorrente

Assim como no bloqueante, teremos duas funcionalidades, worker e start. Diferentemente do bloqueante, o worker aqui será executado através de goroutine que é uma funcionalidade de execução assíncrona, disponibilizada pela linguagem.

Por esse motivo, você verá que o worker terá alguns parâmetros diferentes, que são os canais e o contexto. Agora o processo de escutar o evento de cancelamento e parar a execução do código estará presente dentro da função worker e iremos explicar o motivo quando detalharmos o código.

Para evitar copiar códigos já mostrados anteriormente optarei por utilizar reticencias para descrever que é o mesmo código do bloqueante, sendo assim, basta verificar o código anterior. Também irei ignorar os imports pois eles podem ser visualizados no repositório do github. E irei modificar a ordem das funções para uma melhor didática.

package concurrent

func Start(ctx context.Context, op model.Options, numberOfWorkers int) {
    ...

    scanner := bufio.NewScanner(csvfile)
    scanner.Split(bufio.ScanLines)

    src := make(chan string) //1
    out := make(chan string) //2

    // use a waitgroup to manage synchronization
    var wg sync.WaitGroup //3

    // declare the workers
    for i := 0; i < numberOfWorkers; i++ { //4
        wg.Add(1) //5
        go func() { //6
            defer wg.Done() //7
            worker(ctx, out, src, op.Event) //8
        }()
    }

    go func() { //9
        for scanner.Scan() {
            src <- scanner.Text() //10
        }
        close(src) //11
    }()

    // drain the output
    go func() { //12
        for res := range out {
            fmt.Println(res)
        }
    }()

    // wait for worker group to finish and close out
    wg.Wait() //13
    close(out) //14

    fmt.Printf("\n%2fs", time.Since(start).Seconds())
    fmt.Println()
}

func worker(ctx context.Context, out chan<- string, src <-chan string, event string) {
    for { //15
        select { //16
        case url, ok := <-src: //17
            if !ok {
                return
            }

            payload := map[string]interface{}{
                "event":       event,
                "destination": map[string]string{"email": url},
                "source":      event,
            }
            ...
            out <- fmt.Sprintf("send event to %v, payload to send %v", url, string(data)) // do somethingg useful.
        case <-ctx.Done(): //18
            fmt.Println("worker finish")
            return
        }
    }
}

Para realizar a comunicação das linhas que serão lidas e entregues ao worker utilizaremos um canal chamado src. Vale salientar que a partir de agora o start estará sendo executado em uma rotina e o worker estará sendo executada em outra, por isso, necessitaremos dos canais.
Um canal para obter as respostas do worker de quais foram os registros enviados.
No golang para que as rotinas que foram criadas em paralelo sejam processadas, a rotina principal precisa estar ativa, ou seja, precisamos manter em algum momento um cenário onde falamos para que o nosso código aguarde que as rotinas executadas, sejam aguardadas até que finalizem. E para isso utilizaremos o sync.WaitGroup que tem o papel de conhecer quantas rotinas serão observadas para que possamos aguardar a finalização delas.
Iremos iterar num total de workers que serão definidos na entrada principal, ou seja, caso digamos que teremos um total de 5 workers trabalhando em paralelo, então esse laço irá iterar 5 vezes criando 5 workers.
Através da funcionalidade .Add(1) estamos informando para nosso WaitGroup que teremos mais uma goroutine a ser aguardada.
o go func é uma forma de criar uma goroutine de forma "embarcada" ou seja, podemos invocar uma função anônima de forma concorrente escrevendo a palavra reservada go + func() {...}.
Através do defer wg.Done() estamos informando para o WaitGroup que ao termino dessa nossa goroutine queremos que o WaitGroup seja notificado que uma das rotinas que estão sendo processadas já foi finalizada.
A chamada do worker com os novos parâmetros dentro de uma goroutine, ou seja esse bloco de código será executado de forma concorrente
Agora com os workers já criados e iniciados, iremos criar uma nova goroutine para percorrer o arquivo
Antes entregávamos a leitura da linha para dentro do worker, agora iremos entregar a leitura da linha para dentro de um canal, ou seja, iremos escrever essa linha atual que estamos iterando para dentro do canal através do símbolo ←, ou seja, se esse símbolo estiver a direita do nome do canal, significa que você estará escrevendo algo para dentro do canal, se estiver a esquerda do nome do canal, significa que você estará lendo os dados desse canal.
Após percorrer todo o arquivo e escrevermos tudo dentro do canal src, não iremos mais utilizar esse canal, sendo assim, devemos fechar esse canal através da funcionalidade close() que pertence a própria linguagem;
Iremos criar uma outra goroutine somente para ficar aguardando a leitura do canal de impressão dos dados processados dentro do worker, esse laço continuará repetindo enquanto houver dados a serem lidos do canal.
Através da funcionalidade .Wait() o WaitGroup bloqueia o código, aguardando que todos as funcionalidades wg.Done() sejam executadas, garantindo assim, que só iremos finalizar nossa rotina principal, após todas rotinas serem finalizadas;
Como chegamos ao final da leitura dos dados de saída do worker, iremos fechar o canal out. Vale salientar que esse bloco de código não será executado enquanto todas as goroutines declaradas no WaitGroup tenham sido processadas e o método .Wait() permita que esse código seja executado;
Dentro do worker iremos manter um laço "eterno" para ficar sempre escutando os canais.
Através do select iremos bloquear essa goroutine de forma que o bloco de código só seguirá se chegar uma mensagem no canal src ou se o contexto for cancelado. Vale salientar que o for não será iterado enquanto nenhum dos canais forem lidos, como falei anteriormente o select é bloqueante sem um default;
Através do símbolo ← a esquerda do canal src, estamos informando para o select que nesse case iremos aguardar a LEITURA do canal src, ou seja, toda vez que uma linha do arquivo for lida e escrita nesse canal, esse case será processado e executado, permitindo assim que o select seja desbloqueado e que possa ocorrer a próxima iteração do laço for;
A escuta do cancelamento do contexto foi transferida para dentro do worker pois caso o cancelamento ocorra, precisamos notificar TODOS as rotinas que foram criadas para que sejam canceladas através do return, que ao ser executado o return a função worker será finalizada, consequentemente finalizando também nosso laço "eterno". Permitindo assim que o defer wg.Done() ocorra e consequentemente desbloqueando o .Wait() e assim permitindo a finalização do nosso bloco de código do start;

Para executar nosso algoritmo executaremos o seguinte código no terminal

go run main.go -filename=MOCK_DATA100.csv -event=teste -async=true -workers=2

Resultado Concorrente

Testes com 2 workers

go run main.go -filename=MOCK_DATA100.csv -event=teste -async=true -workers=2
...
send event to massandri2i@cocolog-nifty.com, payload to send {"destination":{"email":"massandri2i@cocolog-nifty.com"},"event":"teste","source":"teste"}
send event to tberthon2k@photobucket.com, payload to send {"destination":{"email":"tberthon2k@photobucket.com"},"event":"teste","source":"teste"}
send event to estruthers2l@marriott.com, payload to send {"destination":{"email":"estruthers2l@marriott.com"},"event":"teste","source":"teste"}
send event to brackstraw2n@xinhuanet.com, payload to send {"destination":{"email":"brackstraw2n@xinhuanet.com"},"event":"teste","source":"teste"}
send event to adepinna2m@microsoft.com, payload to send {"destination":{"email":"adepinna2m@microsoft.com"},"event":"teste","source":"teste"}
send event to esmiley2p@opensource.org, payload to send {"destination":{"email":"esmiley2p@opensource.org"},"event":"teste","source":"teste"}
send event to nnosworthy2o@github.io, payload to send {"destination":{"email":"nnosworthy2o@github.io"},"event":"teste","source":"teste"}
send event to gkording2q@desdev.cn, payload to send {"destination":{"email":"gkording2q@desdev.cn"},"event":"teste","source":"teste"}
send event to hgunthorpe2r@blogs.com, payload to send {"destination":{"email":"hgunthorpe2r@blogs.com"},"event":"teste","source":"teste"}

5.133309s

Testes com 4 workers

go run main.go -filename=MOCK_DATA100.csv -event=teste -async=true -workers=4
...
send event to djuckes2h@dropbox.com, payload to send {"destination":{"email":"djuckes2h@dropbox.com"},"event":"teste","source":"teste"}
send event to dtippler2j@wsj.com, payload to send {"destination":{"email":"dtippler2j@wsj.com"},"event":"teste","source":"teste"}
send event to adepinna2m@microsoft.com, payload to send {"destination":{"email":"adepinna2m@microsoft.com"},"event":"teste","source":"teste"}
send event to brackstraw2n@xinhuanet.com, payload to send {"destination":{"email":"brackstraw2n@xinhuanet.com"},"event":"teste","source":"teste"}
send event to estruthers2l@marriott.com, payload to send {"destination":{"email":"estruthers2l@marriott.com"},"event":"teste","source":"teste"}
send event to tberthon2k@photobucket.com, payload to send {"destination":{"email":"tberthon2k@photobucket.com"},"event":"teste","source":"teste"}
send event to hgunthorpe2r@blogs.com, payload to send {"destination":{"email":"hgunthorpe2r@blogs.com"},"event":"teste","source":"teste"}
send event to gkording2q@desdev.cn, payload to send {"destination":{"email":"gkording2q@desdev.cn"},"event":"teste","source":"teste"}
send event to esmiley2p@opensource.org, payload to send {"destination":{"email":"esmiley2p@opensource.org"},"event":"teste","source":"teste"}
send event to nnosworthy2o@github.io, payload to send {"destination":{"email":"nnosworthy2o@github.io"},"event":"teste","source":"teste"}

2.562593s

Testes com 1 worker

go run main.go -filename=MOCK_DATA100.csv -event=teste -async=true -workers=1
...
send event to massandri2i@cocolog-nifty.com, payload to send {"destination":{"email":"massandri2i@cocolog-nifty.com"},"event":"teste","source":"teste"}
send event to dtippler2j@wsj.com, payload to send {"destination":{"email":"dtippler2j@wsj.com"},"event":"teste","source":"teste"}
send event to tberthon2k@photobucket.com, payload to send {"destination":{"email":"tberthon2k@photobucket.com"},"event":"teste","source":"teste"}
send event to estruthers2l@marriott.com, payload to send {"destination":{"email":"estruthers2l@marriott.com"},"event":"teste","source":"teste"}
send event to adepinna2m@microsoft.com, payload to send {"destination":{"email":"adepinna2m@microsoft.com"},"event":"teste","source":"teste"}
send event to brackstraw2n@xinhuanet.com, payload to send {"destination":{"email":"brackstraw2n@xinhuanet.com"},"event":"teste","source":"teste"}
send event to nnosworthy2o@github.io, payload to send {"destination":{"email":"nnosworthy2o@github.io"},"event":"teste","source":"teste"}
send event to esmiley2p@opensource.org, payload to send {"destination":{"email":"esmiley2p@opensource.org"},"event":"teste","source":"teste"}
send event to gkording2q@desdev.cn, payload to send {"destination":{"email":"gkording2q@desdev.cn"},"event":"teste","source":"teste"}
send event to hgunthorpe2r@blogs.com, payload to send {"destination":{"email":"hgunthorpe2r@blogs.com"},"event":"teste","source":"teste"}

10.241959s

Conclusão

É possível visualizar que para esse tipo de cenário a estrutura concorrente diminuiu drasticamente o tempo de processamento do algoritmo mas vale notificar com somente 1 worker não vale a pena utilizar o algoritmo concorrente pois estaremos só tornando nosso código mais verboso para uma estratégia onde somente uma rotina será executada.

Um outro motivo de se utilizar esse padrão para o consumo concorrente é que caso eu não limitasse, poderia gerar um aumento de recurso pois para cada linha estaríamos abrindo rotinas concorrentes, então em um cenário com um arquivo de 1000 linhas, teríamos 1000 rotinas em paralelo e talvez para processar essa quantidade de rotinas seria necessário mais recursos do computador, ou seja, não teríamos limites de processos ocorrendo de forma concorrente.

Sem falar que caso tivéssemos consumindo uma API, poderíamos derrubar ou até mesmo acabar tendo que aumentar os recursos dessa API pelo uso desenfreado de rotinas em paralelo. Essa estratégia de limitar a quantidade de rotinas processando em paralelo é conhecida como Worker Pool, muito utilizada para gerenciamento de conexões de banco de dados.

Se bem utilizado, a programação concorrente pode ajudar e muito nos processos das empresas. E obvio que a linguagem fornece uma facilidade enorme em utilizar esse tipo de programação, tendo em vista que para todo esse nosso estudo, não foi necessário nenhum tipo de serviço ou dependência de uma biblioteca terceira. Ou seja, a própria linguagem irá te fornecer o necessário para transformar teu algoritmo em um cenário concorrente de forma natural.

Domain, Service and IoC

Marcos Filho — Sun, 26 Jul 2020 02:03:18 +0000

All the code's that you will see in this post you can check here

To keep building our project, in this post we will build our service layer, domain model and our dependency injection container.

The structure from this project in this post will looks like this

📦hero-api-golang
 ┣ 📂cmd
 ┃ ┗ 📂http
 ┃ ┃ ┣ 📂handler
 ┃ ┃ ┃ ┣ 📜book.go
 ┃ ┃ ┃ ┗ 📜router.go
 ┃ ┃ ┗ 📜main.go
 ┣ 📂internal
 ┃ ┣ 📂container
 ┃ ┃ ┗ 📜container.go
 ┃ ┣ 📂domain
 ┃ ┃ ┗ 📂book
 ┃ ┃ ┃ ┗ 📜model.go
 ┃ ┗ 📂service
 ┃ ┃ ┗ 📜book.go
 ┣ 📜.gitignore
 ┣ 📜go.mod
 ┣ 📜go.sum
 ┗ 📜README.md

If you look to the structure will see a internal folder, in my projects i like to building my domain and services in the internal folder, when you name a package as internal, it can only be imported from its parent directory’s packages.

First we will define a domain folder and inside this folder we will create a folder to represent every domain that we will use in this project, in this case we will start a CRUD to a Book.

package book

type Book struct {
    Title       string
    Author      string
    NumberPages int
}

This struct will represent our book model with information that will store inside our database.

It's a simple struct because i want to show you how to build a entire project, after understand the structure you can improve this domain like you want.

After that we will create our service layer, inside internal folder you can check the service folder, in this folder we will group all services necessary to build this project, you will see a lot of post's represent this layer like UseCase layer. If we decide to put any business rules in this project, the service layer is the place.

package service

import (
    "github.com/maaarkin/hero-api-golang/internal/domain/book"
)

type BookService interface {
    Save(book book.Book) (*book.Book, error)
    FindAll() (*[]book.Book, error)
    FindById(id uint64) (*book.Book, error)
    Delete(id uint64) error
    Update(book book.Book) error
}

type bookServiceImpl struct {
}

func NewBookService() BookService {
    return bookServiceImpl{}
}

func (bookServiceImpl) Save(book book.Book) (*book.Book, error) {
    return nil, nil
}

func (bookServiceImpl) FindAll() (*[]book.Book, error) {
    return nil, nil
}

func (bookServiceImpl) FindById(id uint64) (*book.Book, error) {
    return nil, nil
}

func (bookServiceImpl) Delete(id uint64) error {
    return nil
}

func (bookServiceImpl) Update(book book.Book) error {
    return nil
}

In this moment we just create the service layer, you can see that we return nil in all services because we will conect this layer with the repository layer in the next post, but we will build right now to understand how we will inject this service layer in our handlers.

And we will do that through the DI Container, see what i like to do in my projects to represent this "container"...

In the internal folder you can check the container package and the code is very simple.

package container

import (
    "github.com/maaarkin/hero-api-golang/internal/service"
)

// TODO declare services/repositories/components, here
type Container struct {
    Services Services
}

type Services struct {
    BookService service.BookService
}

// Inject represent the starter of our IoC Container, here we will inject
// the necessary structs/functions that we need to build our project.
func Inject() Container {

    //init services
    bs := service.NewBookService()

    services := Services{
        BookService: bs,
    }

    return Container{
        Services: services,
    }
}

The container.go file is the place that we will make our "New()" functions to start any service, repository or component that we want.

And this container we will invoke in our main.go and pass this container through the layers and we will inject the dependencies that are necessary to our project start.

after that, our main.go will looks like this.

package main

import (
    "github.com/maaarkin/hero-api-golang/cmd/http/handler"
    "github.com/maaarkin/hero-api-golang/internal/container"
)

func main() {

    di := container.Inject()
    //delegate to start our Http Server
    handler.StartServer(di)
}

And in the router.go we change this functions

func initHandlers(di container.Container) handlers {
    return handlers{
                //here we will inject the service to the handler
        Book: NewBookHandler(di.Services.BookService),
    }
}

func StartServer(di container.Container) {
    //...
    handlers := initHandlers(di)

    //...
}

If you see the initHandlers(...) function you will see that we pass the container and in the moment that we start our BookHandler we pass the service to the NewBookHandler(di.Services.BookService).

That way, in the handler we can access the BookService interface that represent the service layer.

In this moment we know how to create our domain and service layer and understand how to start our components and pass the information in the project.

In the next post we will improve our handler to make the CRUD functions and access the service layer.

Init project

Marcos Filho — Mon, 06 Jul 2020 00:40:33 +0000

To start our project we will use this github to save our steps during the project.

In this first step we will define a folder structure, domain and controllers with a simple crud (GET, POST, PUT and DELETE).

First of all, we will see how our folder structure will be, after a lot of time reading articles and posts, creating projects and improving my code i decide to use a structure folder like golang standard layout with some changes.

Our Structure

In this first step our structure will look like this:

📦hero-api-golang
 ┣ 📂cmd
 ┃ ┗ 📂http
 ┃ ┃ ┣ 📂handler
 ┃ ┃ ┃ ┣ 📜book.go
 ┃ ┃ ┃ ┗ 📜router.go
 ┃ ┃ ┗ 📜main.go // Our entrypoint project
 ┣ 📜.gitignore
 ┣ 📜go.mod
 ┣ 📜go.sum
 ┗ 📜README.md

In this post we will see all this files and improve this structure in the next post while we building our API.

Initializing the project

To initialize the golang project we need to start the goland dependency mananger, we will use the go mod.

go mod init github.com/maaarkin/hero-api-golang

change /maaarkin/hero-api-golang to /your-user/your-project-name

Every application need a entrypoint, in our case we will represent by cmd/http/main.go file.

To minify the post, we will share only the code necessary to explain what we need, the complete code you can check in the github

func main() {

    //delegate to start our Http Server
    handler.StartServer()
}

the StartServer() function manage everything that envolve http entrypoint like handler register, middleware register and error handler.

package handler

import (
    "github.com/go-chi/chi"
    "github.com/go-chi/chi/middleware"
    "log"
    "net/http"
)

type handlers struct {
    Book *BookHandler
}

func initHandlers() handlers {
    return handlers{
        Book: NewBookHandler(),
    }
}

func StartServer() {
    log.Println("Initializing the Http Server at localhost:8080")
    handlers := initHandlers()

    r := chi.NewRouter()

    r.Use(middleware.RequestID)
    r.Use(middleware.RealIP)
    r.Use(middleware.Logger)
    r.Use(middleware.Recoverer)

    r.Route("/v1", func(r chi.Router) {
        r.Route("/books", handlers.Book.Route)
    })
    http.ListenAndServe(":8080", r)
}

To build our http server we will use chi, if you look the chi documentation you will see very similarities with our start code. The only difference here is that we separate the handler code to another file.

In this case you don't need the chi, the go provide a good http server but chi help us with a good idiomatic and composable router for building Go HTTP services, be free to choice anyone.

In initHandlers() we declare ours handlers, in other post i will tell you why we adopt this approach (IoC :D), but for now that function is responsible to group our API handlers.

And by now, check our BookHandler.

package handler

import (
    "github.com/go-chi/chi"
    "net/http"
)

type BookHandler struct{}

func NewBookHandler() *BookHandler {
    return &BookHandler{}
}

func (b *BookHandler) Route(r chi.Router) {
    r.Get("/hello", b.hello)
}

func (*BookHandler) hello(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("Hello World"))
}

The Route(...) function that we use in the StartServer(...) provide an extension for /books. If we expose any endpoint to the client and this endpoint envolve book domain, we use this handler to expose the service.

To start the handler we will begin with a Hello World, just to check if the http server it's ok.

λ curl http://localhost:8080/v1/books/hello
Hello World

If you check the server log, you will see something like this

2020/07/01 18:56:36 [LAPTOP-JNKKGDL0/hg3pWpHa7y-000002] "GET http://localhost:8080/v1/books/hello HTTP/1.1" from [::1]:50046 - 200 11B in 0s

At this moment we have our http server started and we are ready to go to the next step, write our domain and service layer. But this content belongs to step-2.

Creating a complete golang rest api from zero to Hero

Marcos Filho — Sun, 28 Jun 2020 15:22:11 +0000

After a lot of time spent building golang API's in my job and getting a lot of experience in this wonderful language, i think it's time to share some of this experience with the community that has helped me so much in this journey. This post will be the first of a from zero to hero series on how to build a complete REST API in golang.

I will show you how to build a simple CRUD. But you need to know others things that are necessary to take your project to another level, like to unify your error handling, provide a documentation with swagger, project logging and how to send those logs to a log center (ELK, Splunk, etc), produce a simple repository layer with GORM, use migrations to version your database scripts. I will show you all that and also how I normally structure my projects.

If you search for the tools that I used in this project individually you can find many tutorials online, including how to use they in golang of course, but I could not find any tutorial teaching how to use all of them to provide a complete project, of course you will see something like that in a paid course but I think that it's time to share this information completely free.

Please, feel free to share your opinion in the next posts, it may help me or other people. Or contact me directly.