DEV Community: mrthkc

GO Backend Service Skeleton w/Gin

mrthkc — Sat, 09 Jan 2021 12:36:27 +0000

Intro

Starting this post I assume that you have basic installation of GoLang on your local environment. If not you can follow steps from; Download and Install - The Go Programming Language

Go Directory

After download and install steps, your go(path) directory should be like below;

You need to create your awesome project's folder under src/ directory. If you are using GitHub to host your codes, it would be better to start project under path like;
~/go/src/github.com/{github-username}/{project-name}. That makes easier using go commands such as go get, go install etc.

Go Project Directory View

Now We are ready to open our preferred Go IDE to form project directory layout and code. Here you are my layout;

I will try to explain every file in this directory layout from top to bottom and you can start your backend project by making changes specific to you.

Cmd -/api

Under cmd we can keep starting the system scripts. In this example you can see api/main.go to start our local backend service. Also, you can store fetchers, data importers or fixing scripts under this directory. To run;
go run cmd/api/main.go, specifically for this sample.

Let's see main.go;

package main

import (
    "flag"

    "github.com/{github-username}/{project}/internal/api"
    "github.com/{github-username}/{project}/internal/pkg/conf"
    log "github.com/sirupsen/logrus"
)

func main() {
    env := flag.String("env", "local", "environment")
    flag.Parse()

    config, err := conf.NewConfig("config/default.yml", *env)
    if err != nil {
        log.Fatalf("Can not reead config: %v", err)
    }

    config.Env = *env
    config.DBCred = config.Mysql.Local
    config.Secret = config.JWT.Local.Secret
    if config.Env == "prod" {
        config.DBCred = config.Mysql.Prod
        config.Secret = config.JWT.Prod.Secret
    }

    log.Info("SDK API started w/Env: " + *env)
    api.Listen(config)
}

As you can see basically we are reading service's configuration and starting the API by using Listen function under api package.

Config -/default.yml

As you can guess, under this directory we are storing our configurations and credentials. For secrets we can think another location but basically it is OK to keep them here.

Also yml file is a personal decision, you can select your favorite format such as json, xml, etc.

mysql:
  local:
    host: localhost
    port: 3306
    db: project
    user: root
    password: 12345678
  prod:
    host: prod_db
    port: 3306
    db: project
    user: user
    password: strong_pass

jwt:
  local:
    secret: "test"
  prod:
    secret: "prod"

Internal

Under internal directory we store our service's base scripts. I divided the code base mainly two parts;

service (api) - it could be fetcher, importer, etc.
pkg - structs and functions that we will use for services.

Api (Service)

As I mentioned main functions for an API backend service locating here. Yes, main listen function, cors header function and router for this example;

base.go: by using configuration starting gin

// Listen : starts api by listening to incoming requests
func Listen(c *conf.Config) {
    service.Config = c

    if service.Config.Env == "prod" {
        gin.SetMode(gin.ReleaseMode)
    }
    r := gin.New()
    r.Use(CORS())

    // router.go
    route(r)

    server := &http.Server{
        Addr:         ":9090",
        Handler:      r,
        ReadTimeout:  60 * time.Second,
        WriteTimeout: 60 * time.Second,
    }

    err := server.ListenAndServe()
    if err != nil && err != http.ErrServerClosed {
        log.Fatalf("Server can not start: %v", err)
    }
}

CORS function that we use to set our response headers.
You can change them specifically by looking how you request and collect responses from backend service.

// CORS : basic cors settings
func CORS() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Writer.Header().Set("Access-Control-Allow-Origin", "*")
        c.Writer.Header().Set("Access-Control-Allow-Credentials", "true")
        c.Writer.Header().Set("Access-Control-Allow-Headers", "Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization, accept, origin, Cache-Control, X-Requested-With")
        c.Writer.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS, GET, PUT, DELETE")

        if c.Request.Method == "OPTIONS" {
            c.AbortWithStatus(204)
            return
        }
        c.Next()
    }
}

router.go: Classical router.
Important parts:

endpoint to function match
using middleware(JWTAuth()) and grouping endpoints.

func route(r *gin.Engine) {
    // Base
    r.GET("/api/", service.BaseHandler)

    // Register
    r.POST("/api/user", service.Register)
    r.POST("/api/login", service.Login)

    authorized := r.Group("/api/")
    authorized.Use(service.JWTAuth())
    {
        authorized.GET("token", service.Token)
        authorized.GET("user/:uid/profile", service.Profile)
    }
}

Pkg

We store our service's deep and detailed parts in here.

Adapters:

In my project, I am using MySQL DB only but by looking your project's requirements you can multiple adapters like a NoSQL which needs to establish a connection.

mysql.go:

func NewClient(host string, port string, dbName string, uName string, pass string) (db *sql.DB) {
    db, err := sql.Open("mysql", uName+":"+pass+"@tcp("+host+":"+port+")/"+dbName)
    if err != nil {
        panic(err.Error())
    }
    return db
}

Configuration Reader:

To read our configuration file into a struct:
conf/reader.go:

type Config struct {
    Env   string
    Mysql struct {
        Local struct {
            Host     string `yaml:"host"`
            Port     string `yaml:"port"`
            DB       string `yaml:"db"`
            User     string `yaml:"user"`
            Password string `yaml:"password"`
        } `yaml:"local"`
        Prod struct {
            Host     string `yaml:"host"`
            Port     string `yaml:"port"`
            DB       string `yaml:"db"`
            User     string `yaml:"user"`
            Password string `yaml:"password"`
        } `yaml:"prod"`
    } `yaml:"mysql"`
    JWT struct {
        Local struct {
            Secret     string `yaml:"secret"`
        } `yaml:"local"`
        Prod struct {
            Secret     string `yaml:"secret"`
        } `yaml:"prod"`
    } `yaml:"jwt"`

    DBCred struct {
        Host     string `yaml:"host"`
        Port     string `yaml:"port"`
        DB       string `yaml:"db"`
        User     string `yaml:"user"`
        Password string `yaml:"password"`
    }
    Secret string
}

// NewConfig returns a new decoded Config struct
func NewConfig(configPath string, env string) (*Config, error) {
    // Create config structure
    config := &Config{}

    // Open config file
    file, err := ioutil.ReadFile(configPath)
    if err != nil {
        return nil, err
    }

    // Init new YAML decode
    err = yaml.Unmarshal(file, config)
    if err != nil {
        return nil, err
    }

    return config, nil
}

Entity:

Under entity directory, first of all under entity/main.go I preferred to keep a DB variable for usage of all possible entities.

// DB : DB for all entities
var DB *sql.DB

And as you can guess, in entity/user.go we are using this DB variable for SELECT/INSERT/UPDATE querying functions. Also a map struct for DB user schema to make easier to read rows.

user.go:

// User : user entity
type User struct {
    ID          int        `json:"id"`
    Email       string     `json:"email"`
    Password    string     `json:"password"`
    Fullname    string     `json:"fullname"`
    TSLastLogin int32      `json:"ts_last_login"`
    TSCreate    int32      `json:"ts_create"`
    TSUpdate    int32      `json:"ts_update"`
    Permission  json.RawMessage `json:"permission"`
}

// GetUserByEmail : returns single user
func GetUserByEmail(email string) *User {
    user := new(User)
    row := DB.QueryRow("SELECT * from user WHERE email=?", email)
    err := row.Scan(&user.ID, &user.Email, &user.Password, &user.Fullname, &user.TSLastLogin, &user.TSCreate, &user.TSUpdate, &user.Permission)
    if err != nil {
        log.Errorln("User SELECT by Email Err: ", err)
        return nil
    }
    return user
}

Service:

Under service directory there are handlers / middleware functions. These functions are the base point to respond requests coming from any external service.

Under service/base.go, we can add a baseHandler to check health of the service.

// BaseHandler : home - health-test
func BaseHandler(c *gin.Context) {
    log.Info("Base")
}

Under service/jwt.go, you can find the middleware function to validate and auth token, that we called from router.go as it is mentioned.

// ValidateToken :
func ValidateToken(encodedToken string) (*jwt.Token, error) {
    return jwt.Parse(encodedToken, func(token *jwt.Token) (interface{}, error) {
        if _, isvalid := token.Method.(*jwt.SigningMethodHMAC); !isvalid {
            return nil, errors.New("invalid token")
        }
        return []byte(Config.Secret), nil
    })
}

// JWTAuth :
func JWTAuth() gin.HandlerFunc {
    return func(c *gin.Context) {
        var bearer = "Bearer"
        authHeader := c.GetHeader("Authorization")
        if authHeader == "" {
            c.AbortWithStatus(http.StatusUnauthorized)
            return
        }

        tokenString := authHeader[len(bearer):]
        token, err := ValidateToken(tokenString)
        if token.Valid {
            claims := token.Claims.(jwt.MapClaims)
            log.Debug(claims)
        } else {
            log.Error(err.Error())
            c.AbortWithStatus(http.StatusUnauthorized)
        }
    }
}

And lastly, service/user.go, there are functions related to user schema. Such as login, register, profile, etc. Here you are login function that we used in router.go to respond api/login endpoint.

// Login :
func Login(c *gin.Context) {
    entity.DB = adapter.NewClient(Config.DBCred.Host, Config.DBCred.Port, Config.DBCred.DB, Config.DBCred.User, Config.DBCred.Password)
    defer entity.DB.Close()

    user := new(entity.User)
    if err := c.BindJSON(user); err != nil {
        log.Error("Binding user error occured: ", err)
        c.JSON(http.StatusBadRequest, "Binding error")
        return
    }

    checkUser := entity.GetUserByEmail(user.Email)
    if checkUser == nil {
        c.JSON(http.StatusBadRequest, gin.H{
            "success": 0,
        })
        return
    }

    login := comparePasswords(checkUser.Password, []byte(user.Password))
    if login == false {
        c.JSON(http.StatusForbidden, gin.H{
            "success": 0,
        })
        return
    }

    token, err := GenerateToken(checkUser.ID)
    if err != nil {
        c.JSON(http.StatusForbidden, gin.H{
            "success": 0,
        })
        return
    }
    c.JSON(http.StatusOK, gin.H{
        "success": 1,
        "data":    gin.H{
            "uid": checkUser.ID,
            "uemail": checkUser.Email,
            "token": token,
        },
    })
    return
}

As a conclusion, I am using this layout while starting any side project's backend service. By adding new entities, service functions and endpoints, it is really sufficient to save your time that is spent in starting step.

Cheers.

Mapping with AWS S3 Select — Cost Efficient Solution

mrthkc — Sat, 07 Nov 2020 17:53:44 +0000

Welcome to my first 'Dev' post. I want you to know this is not my first post. I'm moving my post from medium because I know here is better platform for me to follow everything about my profession. Good reading.

From April 10th;

We, developers, love both creating and using high performing ( eg: fast responding) functions, softwares and applications. Generally, one of our finalization steps is performing speed and load tests on our products, especially if we are using a third party service with the related parts.

Nowadays fast and flexible NoSql technologies are very popular. There are many tools which serve the response instantly and consistently. Let’s look at AWS’ DynamoDB as an example. However, we have one main problem with DynamoDB. Write requests like INSERT / UPDATE and their reserved capacity are highly priced. If you want fast responses, you need to insert these data to DynamoDB first which could be really costly under high traffic load.

My ex work place was a platform that links advertisers who could be app owners and media companies with ad publishers. If you are working on mobile advertising data, you need to have some solutions to detect fraudulent traffic. While redirecting advertisement clicks to APP stores, we must make sure that possible install and in-app-events would come from real users.

As you can guess, we are operating on a lot of click, install and in-app-event data by matching them with each other and detecting untrustworthy ones.

Now let’s start with the logic of how we are matching all these data, under high traffic load despite the fraudulent traffic. First, we need to create a mechanism to generate smart identifiers for your objects which should be shared with external platforms and businesses. Smarts IDs should be encrypted, with a strong key, to not give any room to AD fakers but also contain some information about advertisement items which are progressing.

Let’s get back to AWS. It’s time to talk about how we were using some of their services. Remember smart identifiers, besides them, we need to keep some data like click date-time (to detect the passed time between an install or in-app-event and click). Relational databases could be very compulsive if you are not loaded and do not have many servers (Plus we need to think about the maintenance costs).

So, we needed a backup storage first, which we decided to go with S3(Simple Storage Service). You can use both JSON and CSV / TSV format to store my ex work place's traffic data. (eg: clicks, in-app-events etc.)

{id:<strong-generated-id>, ts: 987026400, …}
{id:<strong-generated-id>, ts: 1069974000, …}

We planned S3 stored data for only backup and side usages, we also had one design with NoSql to store and map our data, which is DynamoDB. We were inserting tons of click data to DynamoDB after buffering to match these clicks with installs and events as fast as we can.

We do not have to wait too long for this design to be a costly one. Let’s go over the February 6th DynamoDB writing cost as an example.

February 6th Write Capacity Unit(WCU): 138,798
February 6th Cost: $101.06

By looking at this you can see that one of our biggest expense is AWS DynamoDB write requests. (Think about some days WCU is raising to 175K)

At last, after all of the above analysis I am ready to explain our solution for the cost reduction.I told you about our smart IDs and storing them in S3. We noticed that with minor changes we could understand which click item was stored in which S3 file. Take a look at this;

Our smart IDs;

Encrypt (<execution_ts>+<related_ad_campaign>+<random_alpha_numeric>+<server_identifier>);
Encrypt (987026407+123321+ABcd1234EF56+55533);

So, S3 keys are like

S3_key: <item_type>+<buffered_ts>+<server_identifier>+<random_alpha_numeric>
S3_key: click+987026400+55533+ZZXX112233VVPP

Now, it’s easy to determine which traffic item is in which S3 file when an install or in-app-event comes with a converted click’s smart ID. It’s not done yet. In buffered S3 files there are many other click data and we need to find the correct one from the conversion ID. Because we stored the click details in JSON or CSV/TSV format we can use the S3-Select function https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-s3-select-is-now-generally-available/, like;

func S3Select(bucket, key, itemID){ …use your AWS library’s S3 select… }
item = S3Select(‘traffic’, ‘click+987026400+55533+ZZXX112233VVPP’, <encrypted-smart-identifier>)

Last of all, by organizing your item identifiers and S3 file names in a logic, you can use Simple Storage Service, ‘select’ function to attribute your relational items cost effectively. Before finishing my first medium post, I want to share DynamoDB cost graphs which includes the exact date that we updated our attribution logic;

Monthly perspective;

Line chart below is showing S3 API Requests’ costs for February;

As you can see there are no remarkable changes after mapping logic change.

If fast responses are not essential, if you can handle your pairing objects processes asynchronously and if you are not extremely wealthy, then you can prefer S3 instead of DynamoDB for both storing and retrieving your data.

Hello dev.to

mrthkc — Mon, 02 Nov 2020 10:47:19 +0000

Hello dev.to,

My name is Mert and I live in Ankara, 10 year experienced software developer. So far I have worked for five startup companies as a backend developer. I mostly used php, python3, GoLang and JS.

I just opened my account and my objective is stay in touch and follow related software development posts.

My interests;

python
Django w/python3
GoLang
React
Flutter
Cloud solutions (AWS, Oracle, Google Cloud etc.)

Also I'll try to share posts that include interesting development issues and solutions which I experience.

Happy to be here.