DEV Community: Synexis Microsystem

Database Migration: Take Care of Your Database Changes

Synexis Microsystem — Mon, 27 May 2024 03:47:38 +0000

Preamble

Database Migration is basically a process that should be initialized and started before starting up a service that need database. In this moment I would share my experience about Database Migration, that helps me for moving or duplicating a database which already good to go.

I will show you how to create an simple program for performing database migration using Golang Migrate. Before we proceed to next step make sure you already install its binary on your system, here is the release list with various operating system option, Golang Migrate Releases.

By creating this program, you can easily setup database design and its sample data. And I have repository you can clone for this here is it, while reading this post you will understanding how it works. And I recommend you to have docker installed in your pocket for this tutorial.

Database Migration Directory

I already have the repository for this one, inside the project there is package named database and inside it there is migration package where you put all generated migration files. Here is the structure of it.

Database Server Setup

For this moment you need to run mysql database server in your machine, if you are already have installed one, you can just go with it. If you want to use docker here is the command to spin up mysql server in simple way.

run -d -p 3306:3306 --name mysql-db -e MYSQL_ROOT_PASSWORD=root mysql:5.7.44

Install golang-migrate

To install golang-migrate you can choose which operating system did you use. Extract migrate binary file and register it your environment path. You can download migrate from its releases page golang migrate download.

Generate Migration Files

Once already download migrate binary file and register it to your environment path, you can check it by running this command.

migrate --version

It will show you which migrate version did you use.

For generate migration files you can use this command

migrate create -ext sql -dir path/to/migration/dir -seq migration_file_name

Explanation

-ext This arg is to tell migrate to use sql extension
-dir This arg is to tell migrate where migration files at
-seq This arg is to tell migrate to use sequential numbering for every migration file you generate this is the sample 000001_create_users_table.up.sql and 000001_create_users_table.down.sql

And we need to generate 2 migration files each of it will be have up and down version. So there will be 4 files. Let's generate migration file for users and profiles table. And make sure your current working directory is on the root of the project.

migrate create -ext sql -dir database/migration -seq create_users_table
migrate create -ext sql -dir database/migration -seq create_profiles_table

When you take a look in the project it will be looks like this

What is difference between up and down migration file?

up This migration file will does construction for your database
down This migration file will does deconstruction for your database

Or can be simplified up migration file will contains creation and modification action. In the other hand down will contains drop or removal action.

And here is the SQL Query that will performed when migration process running.

Users Table Migration Files

`000001_create_users_table.up.sql`

CREATE TABLE users
(
    id         SERIAL PRIMARY KEY,                              -- Auto-incremented primary key
    email      VARCHAR(255) NOT NULL UNIQUE,                    -- Email column with unique constraint
    password   VARCHAR(255) NOT NULL,                           -- Password column
    username   VARCHAR(50)  NOT NULL UNIQUE,                    -- Username column with unique constraint
    deleted_at TIMESTAMP NULL,                                  -- Timestamp for soft deletion
    updated_at TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Timestamp for last update
    created_at TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP  -- Timestamp for creation
);

-- Adding index for columns that might be frequently searched or filtered
CREATE INDEX idx_users_email ON users (email);
CREATE INDEX idx_users_username ON users (username);

-- Index for deleted_at to quickly filter non-deleted users
CREATE INDEX idx_users_deleted_at ON users (deleted_at);

`000001_create_users_table.down.sql`

DROP TABLE users;

Profiles Table Migration Files

`000001_create_profiles_table.up.sql`

CREATE TABLE profiles
(
    id              SERIAL PRIMARY KEY,                             -- Auto-incremented primary key
    user_id         INTEGER     NOT NULL,                           -- Assuming user_id will link to users table but without any constraint
    first_name      VARCHAR(50) NOT NULL,                           -- First name of the user
    last_name       VARCHAR(50) NOT NULL,                           -- Last name of the user
    bio             TEXT,                                           -- A short bio of the user
    profile_picture VARCHAR(255),                                   -- URL to the profile picture
    deleted_at      TIMESTAMP NULL,                                 -- Timestamp for soft deletion
    created_at      TIMESTAMP   NOT NULL DEFAULT CURRENT_TIMESTAMP, -- Timestamp for creation
    updated_at      TIMESTAMP   NOT NULL DEFAULT CURRENT_TIMESTAMP  -- Timestamp for last update
);

-- Adding index for columns that might be frequently searched or filtered
CREATE INDEX idx_profiles_user_id ON profiles (user_id);
CREATE INDEX idx_profiles_last_name ON profiles (last_name);

`000001_create_profiles_table.down.sql`

DROP TABLE profiles;

Implementation

In this section we jump to coding and how to use as it Standalone Program or Embedded routine inside your application. Inside repository project there are 3 golang file for database migration connector.go, mlog.go and migration.go.

`connector.go`

In this file you can add your own database connection function template for your desired database by copy paste existing sample and adjust based on your need. In this sample will be 2 MySQL and Postgres.

package database

import (
    "database/sql"
    "fmt"
    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/mysql"
    "github.com/golang-migrate/migrate/v4/database/postgres"
)

// connect database connector using builtin go sql library
func connect(dialect, connStr string) (*sql.DB, error) {
    db, err := sql.Open(dialect, connStr)
    if err != nil {
        return nil, err
    }
    err = db.Ping()
    if err != nil {
        return nil, err
    }
    return db, nil
}

// MySQLBuilder decorator function for constructing mysql connection string
func MySQLBuilder(cfg Config) (*migrate.Migrate, error) {
    // construct connection string and connect
    const formatString = "%s:%s@tcp(%s:%s)/%s?parseTime=true&multiStatements=true"
    credentials := []any{
        cfg.DatabaseUser, cfg.DatabasePasw, cfg.DatabaseHost,
        cfg.DatabasePort, cfg.DatabaseName,
    }
    finalCsf := fmt.Sprintf(formatString, credentials...)
    db, err := connect("mysql", finalCsf)
    if err != nil {
        return nil, err
    }

    // create migrate instance from connected database client
    driver, err := mysql.WithInstance(db, &mysql.Config{})
    if err != nil {
        return nil, err
    }
    filePath := fmt.Sprintf("file://%s", cfg.DatabaseMdir)
    mgrt, err := migrate.NewWithDatabaseInstance(filePath, cfg.DatabaseDrvr, driver)
    if err != nil {
        return nil, err
    }
    return mgrt, nil
}

// PostgresBuilder decorator function for constructing mysql connection string
func PostgresBuilder(cfg Config) (*migrate.Migrate, error) {
    // construct connection string and connect
    const formatString = "postgres://%s:%s@%s:%d/%s?sslmode=disable"
    credentials := []any{
        cfg.DatabaseUser, cfg.DatabasePasw, cfg.DatabaseHost,
        cfg.DatabasePort, cfg.DatabaseName,
    }
    finalCsf := fmt.Sprintf(formatString, credentials...)
    db, err := connect("postgres", finalCsf)
    if err != nil {
        return nil, err
    }

    // create migrate instance from connected database client
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return nil, err
    }
    filePath := fmt.Sprintf("file://%s", cfg.DatabaseMdir)
    mgrt, err := migrate.NewWithDatabaseInstance(filePath, cfg.DatabaseDrvr, driver)
    if err != nil {
        return nil, err
    }
    return mgrt, nil
}

`mlog.go`

This file contains logger implementation for migrate to log execution process. This log will be used inside migration.go file.

package database

import "log"

type (
    Log interface {
        Printf(format string, v ...interface{})
        Verbose() bool
    }

    // logImpl implements the golang-migrate Logger interface
    logImpl struct {
        logger *log.Logger
    }
)

func NewLog(logger *log.Logger) Log {
    return &logImpl{logger: logger}
}

func (l *logImpl) Printf(format string, v ...interface{}) {
    l.logger.Printf(format, v...)
}

func (l *logImpl) Verbose() bool {
    return true // or false, depending on whether you want verbose logging
}

`migration.go`

This file contains logger implementation for migrate to log execution process. This log will be used inside migration.go file.

package database

import (
    "errors"
    _ "github.com/go-sql-driver/mysql"
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/mysql"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
    "log"
)

type (
    // ConnectionFunc decorator function for constructing connection string
    ConnectionFunc func(cfg Config) (*migrate.Migrate, error)

    // Config database configuration detail for connecting to desired database server
    Config struct {
        DatabaseHost string // database host
        DatabasePort string // database port
        DatabaseName string // database name
        DatabaseUser string // database user
        DatabasePasw string // database password
        DatabaseDrvr string // database driver
        DatabaseMdir string // database migration dir
    }

    // Migration interface for migration
    Migration interface {
        Action(name string) error
        down() error
        up() error
    }

    // migrationImpl implementation struct
    migrationImpl struct {
        m *migrate.Migrate
    }
)

func NewMigration(cf ConnectionFunc, cfg Config) (Migration, error) {
    // initialize migration instance
    m, err := cf(cfg)
    if err != nil {
        return nil, err
    }
    m.Log = NewLog(
        log.New(log.Writer(),
            "migration: ",
            log.LstdFlags|log.Lshortfile,
        ),
    )
    mg := &migrationImpl{m: m}
    return mg, nil
}

// Action migrating desired migration version start constructing database
func (mgr *migrationImpl) Action(name string) error {
    mgr.m.Log.Printf("migaration.Action: %s", "starting to constructing database")
    switch name {
    case "UP":
        return mgr.up()
    case "DOWN":
        return mgr.down()
    default:
        mgr.m.Log.Printf("migaration.Action: %s (%s)", "unknown action name", name)
        return errors.New("unknown action")
    }
}

// down performing all SQL inside `down` migration files
func (mgr *migrationImpl) down() error {
    err := mgr.m.Down()
    if err != nil {
        if errors.Is(err, migrate.ErrNoChange) {
            mgr.m.Log.Printf("migaration.down: %s (%s)", "deconstructing database done", err.Error())
            return nil
        }
        mgr.m.Log.Printf("migaration.down: %s (%s)", "failed to deconstructing database", err.Error())
        return err
    }
    return nil
}

// up performing all SQL inside `up` migration files
func (mgr *migrationImpl) up() error {
    mgr.m.Log.Verbose()
    err := mgr.m.Up()
    if err != nil {
        if errors.Is(err, migrate.ErrNoChange) {
            mgr.m.Log.Printf("migaration.down: %s (%s)", "constructing database done", err.Error())
            return nil
        }
        mgr.m.Log.Printf("migaration.down: %s (%s)", "failed to constructing database", err.Error())
        return err
    }
    return nil
}

How to Use

We will use Migration implementation inside main.go file, and I will show you how to use it as Standalone Program or Embeded Routine for your application.

Standalone Program

We will compile this into executable binary. Here is the main.go content for that.

package main

import (
    "flag"
    "github.com/xoxoist/dm-tutor/database"
)

func main() {
    action := flag.String("action", "UP", "desired action for your database migration")
    flag.Parse()

    mg, err := database.NewMigration(database.MySQLBuilder, database.Config{
        DatabaseHost: "localhost",
        DatabasePort: "3306",
        DatabaseName: "md_tutor",
        DatabaseUser: "root",
        DatabasePasw: "root",
        DatabaseDrvr: "mysql",
        DatabaseMdir: "database/migration",
    })
    if err != nil {
        panic(err)
    }

    err = mg.Action(*action)
    if err != nil {
        panic(err)
    }
}

And let's compile it into executable binary named migrator

go build -o migrator main.go

Here is how to use that complied executable binary. And make sure you are on same current working directory with migrator

UP

./migrator -action UP

Response for UP action

migration: 2024/05/26 21:19:45 mlog.go:22: migaration.Action: starting database migration
migration: 2024/05/26 21:19:45 mlog.go:22: Start buffering 1/u create_users_table
migration: 2024/05/26 21:19:45 mlog.go:22: Start buffering 2/u create_profiles_table
migration: 2024/05/26 21:19:45 mlog.go:22: Read and execute 1/u create_users_table
migration: 2024/05/26 21:19:45 mlog.go:22: Finished 1/u create_users_table (read 932.759µs, ran 14.400196ms)
migration: 2024/05/26 21:19:45 mlog.go:22: Read and execute 2/u create_profiles_table
migration: 2024/05/26 21:19:45 mlog.go:22: Finished 2/u create_profiles_table (read 16.819281ms, ran 10.786267ms)

DOWN

./migrator -action DOWN

Response for DOWN action

migration: 2024/05/26 21:20:07 mlog.go:22: migaration.Action: starting database migration
migration: 2024/05/26 21:20:07 mlog.go:22: Start buffering 2/d create_profiles_table
migration: 2024/05/26 21:20:07 mlog.go:22: Start buffering 1/d create_users_table
migration: 2024/05/26 21:20:07 mlog.go:22: Read and execute 2/d create_profiles_table
migration: 2024/05/26 21:20:07 mlog.go:22: Finished 2/d create_profiles_table (read 28.291992ms, ran 11.132556ms)
migration: 2024/05/26 21:20:07 mlog.go:22: Read and execute 1/d create_users_table
migration: 2024/05/26 21:20:07 mlog.go:22: Finished 1/d create_users_table (read 41.51291ms, ran 8.649414ms)

Embedded to Application

How to use still remain the same with Standalone Program, but you just need this err = mg.Action(*action) into err = mg.Action("UP") or err = mg.Action("DOWN"). based on your need. and put this code on initialization phase of your application

mg, err := database.NewMigration(database.MySQLBuilder, database.Config{
        DatabaseHost: "localhost",
        DatabasePort: "3306",
        DatabaseName: "md_tutor",
        DatabaseUser: "root",
        DatabasePasw: "root",
        DatabaseDrvr: "mysql",
        DatabaseMdir: "database/migration",
    })
    if err != nil {
        panic(err)
    }
//err = mg.Action("DOWN")
err = mg.Action("UP")
    if err != nil {
        panic(err)
    }

Behavior

This migration has behavior to be known before you use it. When it started up your migration it will execute all your SQL query on your migration files, also automatically create table named schema_migrations to track where migration version sequence at, and is it dirty? if dirty it means there is something wrong in your SQL query syntax and need to be fixed before it moves to next version sequence of your migration file. Here is the schema_migrations table structure.

Schema Migration Table

Your defined tables inside migration files

For this case current version sequence of your migration files is 2, try to look column version.

When you try to down the migration process all table dropped and leave you schema_migrations table without any rows.

When there is SQL syntax error while up or down your migration. You need manually edit version to which version sequence before version sequence that has SQL syntax error, and reset dirty to 0 again on schema_migrations table.

If column dirty is 0 means there is no syntax error on your migration files
If column dirty is 1 means there is syntax error on your migration files.

Conclusion

Database Migration helps you construct your database design in a complete new environment, and you also could add new table or modify table by generate new migration file and write SQL query for that and re-run the program.

You can also put this program inside your application so when the application start it will runs the migration process and ignore the migration process when there is no changes in migration files.

You have 2 option make this program as standalone program or you can embed it inside your application code.

Its up to you and based on your needs, for my personal I would make it as standalone program, where I can keep track database changes and no distraction from others, so when ever someone need a new changes, I just ask the final query for that changes.

If I embed it inside application, people who contribute in that application development will freely add changes and I think it is not so good, every addition and modification of database structure should be discussed before it applied. So again back to your needs. Make it standalone or embed it inside application.

My Thanks

Thank you for visiting! I hope you found it useful and enjoyable. Don't hesitate to reach out if you have any questions or feedback. Happy reading!

Contextual Logging: Keep Tracks Execution Flow

Synexis Microsystem — Sat, 18 May 2024 09:07:09 +0000

Preamble

Have you ever been in a situation where you needed to trace the execution flow to find a bug, but had no idea what was happening, when a request started entering your business logic and flowed through various logic sets? At this moment, I want to share how to implement Contextual Logging to make it easier to trace the execution flow in business logic.

Demonstration Project

I have coded a project to demonstrate how contextual logging flows across applications, or services. In this project, we have three services: Alpha, Bravo, and Charlie. Alpha sends requests to Bravo and Charlie when certain conditions are met. Afterward, I merge the log files from these services to see the correlated log lines, tracking the request as it enters the Alpha application and proceeds to the sub-applications, Bravo and Charlie. This allows us to trace the execution flows between these applications. Please clone this repository before we proceed to next section. Contextual Logging Demonstration Git.

Business Logic Flow

The Business Logic Flow behind the project involves the execution of Alpha within the main function. Alpha requires two applications, Bravo and Charlie, which are used for string manipulation in a simple case. Once Alpha receives a request, it checks the content string for Bravo. If content is present, it sends a request to the Bravo app. Similarly, if there is content for Charlie, it sends a request to the Charlie app. The responses from Bravo and Charlie are then included in Alpha's response structure, which is returned using a print line. Throughout this process, logs are generated line by line to trace the execution flow. The sequence diagram provided offers a visual understanding of this flow.

Log Files

In this project, there are three log files, one for each application: Alpha, Bravo, and Charlie. These log files will be merged using Logstash, which I will discuss in the next episode. For now, we achieve this by merging them manually. Here is where the log file is located on the project.

Logger Library Setup

You are free to choose any library for logging in your application, but for this implementation, I will use Zap Logger and Lumberjack for log file writer. Zap Logger Git and Lumberjack Git. We need to initialize a Zap Logger instance and use it across applications. Here is what the logger/logger.go file content looks like, and here is where the logger.go file is located.

Logger `logger/logger.go`

package logger

import (
    "github.com/natefinch/lumberjack"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "os"
    "strings"
)

func NewZapLogger(logFileName, setMode, applicationVersion, applicationName string) *zap.SugaredLogger {
    // encode config mode checking and setup
    var encoderCfg zapcore.EncoderConfig
    if strings.Contains(setMode, "stg") ||
        strings.Contains(setMode, "prd") {
        encoderCfg = zap.NewProductionEncoderConfig()
    }
    encoderCfg.TimeKey = "timestamp"
    encoderCfg.EncodeTime = zapcore.ISO8601TimeEncoder

    // zap logging config setup using previous encoder config
    var config zap.Config
    config.Level = zap.NewAtomicLevelAt(zap.InfoLevel)
    config.Development = false
    config.DisableCaller = false
    config.DisableStacktrace = false
    config.Sampling = nil
    config.EncoderConfig = encoderCfg
    config.Encoding = "json"
    config.OutputPaths = []string{"stderr"}
    config.ErrorOutputPaths = []string{"stderr"}
    config.InitialFields = map[string]interface{}{
        "service_pid":     os.Getpid(),
        "service_name":    applicationName,
        "service_version": applicationVersion,
    }

    // Set up file output using Lumberjack
    fileSyncer := zapcore.AddSync(&lumberjack.Logger{
        Filename: logFileName,
    })

    // Create file core
    fileCore := zapcore.NewCore(zapcore.NewJSONEncoder(encoderCfg), fileSyncer, config.Level)
    core := zapcore.NewTee(
        zap.Must(config.Build()).Core(),
        fileCore.With([]zap.Field{
            zap.Int("service_pid", os.Getpid()),
            zap.String("service_name", applicationName),
            zap.String("service_version", applicationVersion),
        }),
    )

    // setup sugared log
    sugared := zap.New(core).Sugar()
    defer func(sugar *zap.SugaredLogger) {
        _ = sugar.Sync()
    }(sugared)
    return sugared
}

Applications Code

In this demonstration, there are three applications inside the applications package, each wrapped in a package named alpha, bravo, and charlie. Inside these packages, there are Go files containing the business logic code. Here is the content of those files. And here is where the applications package looks like.

Alpha `alpha.go`

package alpha

import (
    "context"
    "errors"
    "fmt"
    "github.com/xoxoist/cl-tutor/applications/bravo"
    "github.com/xoxoist/cl-tutor/applications/charlie"
    "go.uber.org/zap"
)

type (
    Request struct {
        AlphaContent   string
        BravoContent   string
        CharlieContent string
    }

    Response struct {
        BravoManipulatedContent   string
        CharlieManipulatedContent string
    }

    Alpha interface {
        AlphaLogicFlowEntry(ctx context.Context, req Request) (*Response, error)
        businessLogicFlow(ctx context.Context, req Request) (*Response, error)
        requestToBravoApp(ctx context.Context, req Request) (*bravo.Response, error)
        requestToCharlieApp(ctx context.Context, req Request) (*charlie.Response, error)
    }

    alpha struct {
        Name          string
        Version       string
        Package       string
        bravoClient   func(requestId string, request bravo.Request) (*bravo.Response, error)
        charlieClient func(requestId string, request charlie.Request) (*charlie.Response, error)
    }
)

func NewAlphaApplication(
    applicationName string,
    applicationVersion string,
    bravoClient func(requestId string, request bravo.Request) (*bravo.Response, error),
    charlieClient func(requestId string, request charlie.Request) (*charlie.Response, error),
) Alpha {
    return &alpha{
        Name:          applicationName,
        Version:       applicationVersion,
        Package:       "applications.alpha",
        bravoClient:   bravoClient,
        charlieClient: charlieClient,
    }
}

func (a *alpha) AlphaLogicFlowEntry(ctx context.Context, req Request) (*Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", a.Package, "AlphaLogicFlowEntry")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // executing business logic flow
    response, err := a.businessLogicFlow(ctx, req)
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while contacting charlie service")
    }

    // returning
    return response, nil
}

func (a *alpha) businessLogicFlow(ctx context.Context, req Request) (*Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", a.Package, "businessLogicFlow")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // var bravo and charlie manipulated string
    var bravoStr, charlieStr string

    // when string length inside req.BravoContent is 0
    // then calling bravo client
    if len(req.BravoContent) > 0 {
        br, err := a.requestToBravoApp(ctx, req)
        if err != nil {
            cl.Errorw(fName, "reason", err.Error())
            return nil, errors.New("error while contacting bravo service")
        }
        bravoStr = br.ManipulatedContent
    }

    // when string length inside req.CharlieContent is 0
    // then calling charlie client
    if len(req.CharlieContent) > 0 {
        cr, err := a.requestToCharlieApp(ctx, req)
        if err != nil {
            cl.Errorw(fName, "reason", err.Error())
            return nil, errors.New("error while contacting charlie service")
        }
        charlieStr = cr.ManipulatedContent
    }

    // returning
    return &Response{
        BravoManipulatedContent:   fmt.Sprintf("%s %s", req.AlphaContent, bravoStr),
        CharlieManipulatedContent: fmt.Sprintf("%s %s", req.AlphaContent, charlieStr),
    }, nil
}

func (a *alpha) requestToBravoApp(ctx context.Context, req Request) (*bravo.Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", a.Package, "requestToBravoApp")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // extract request id to be passed to bravo application
    requestId, ok := ctx.Value("Request-Id").(string)
    if !ok {
        return nil, errors.New("request id not pass through context")
    }

    // sending request to bravo application
    response, err := a.bravoClient(requestId, bravo.Request{
        Content: req.BravoContent,
    })
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while contacting bravo service")
    }

    // returning
    return response, nil
}

func (a *alpha) requestToCharlieApp(ctx context.Context, req Request) (*charlie.Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", a.Package, "requestToCharlieApp")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // extract request id to be passed to charlie application
    requestId, ok := ctx.Value("Request-Id").(string)
    if !ok {
        return nil, errors.New("request id not pass through context")
    }

    // sending request to charlie application
    response, err := a.charlieClient(requestId, charlie.Request{
        Content: req.CharlieContent,
    })
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while contacting charlie service")
    }

    // returning
    return response, nil
}

Bravo `bravo.go`

package bravo

import (
    "context"
    "errors"
    "fmt"
    "go.uber.org/zap"
    "strings"
)

type (
    Request struct {
        Content string
    }

    Response struct {
        ManipulatedContent string
    }

    Bravo interface {
        BravoLogicFlowEntry(ctx context.Context, req Request) (*Response, error)
        reverseString(ctx context.Context, content string) (string, error)
    }

    bravo struct {
        Name    string
        Version string
        Package string
    }
)

func NewBravoApplication(
    applicationName string,
    applicationVersion string,
) Bravo {
    return &bravo{
        Name:    applicationName,
        Version: applicationVersion,
        Package: "applications.bravo",
    }
}

func (b *bravo) BravoLogicFlowEntry(ctx context.Context, req Request) (*Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", b.Package, "BravoLogicFlowEntry")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // reverse the string
    manipulated, err := b.reverseString(ctx, req.Content)
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while manipulate string")
    }

    // returning
    return &Response{
        ManipulatedContent: manipulated,
    }, nil
}

func (b *bravo) reverseString(ctx context.Context, content string) (string, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", b.Package, "reverseString")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return "", errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // manipulator
    runes := []rune(content)
    for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
        runes[i], runes[j] = runes[j], runes[i]
    }
    manipulated := strings.ToUpper(string(runes))
    cl.Infow(fName, "manipulated_content", manipulated)

    // returning
    return manipulated, nil
}

Charlie `charlie.go`

package charlie

import (
    "context"
    "errors"
    "fmt"
    "go.uber.org/zap"
    "strings"
)

type (
    Request struct {
        Content string
    }

    Response struct {
        ManipulatedContent string
    }

    Charlie interface {
        CharlieLogicFlowEntry(ctx context.Context, req Request) (*Response, error)
        replaceVowelsWithAsterisk(ctx context.Context, content string) (string, error)
    }

    charlie struct {
        Name    string
        Version string
        Package string
    }
)

func NewCharlieApplication(
    applicationName string,
    applicationVersion string,
) Charlie {
    return &charlie{
        Name:    applicationName,
        Version: applicationVersion,
        Package: "applications.charlie",
    }
}

func (c *charlie) CharlieLogicFlowEntry(ctx context.Context, req Request) (*Response, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", c.Package, "BravoLogicFlowEntry")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // reverse the string
    manipulated, err := c.replaceVowelsWithAsterisk(ctx, req.Content)
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while manipulate string")
    }

    // returning
    return &Response{
        ManipulatedContent: manipulated,
    }, nil
}

func (c *charlie) replaceVowelsWithAsterisk(ctx context.Context, content string) (string, error) {
    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", c.Package, "replaceVowelsWithAsterisk")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return "", errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

    // manipulator
    vowels := "AEIOUaeiou"
    var result strings.Builder
    for _, char := range content {
        if strings.ContainsRune(vowels, char) {
            result.WriteRune('*')
        } else {
            result.WriteRune(char)
        }
    }
    manipulated := result.String()
    cl.Infow(fName, "manipulated_content", manipulated)

    // returning
    return manipulated, nil
}

Main `main.go`

package main

import (
    "context"
    "fmt"
    "github.com/xoxoist/cl-tutor/applications/alpha"
    "github.com/xoxoist/cl-tutor/applications/bravo"
    "github.com/xoxoist/cl-tutor/applications/charlie"
    "github.com/xoxoist/cl-tutor/logger"
    "go.uber.org/zap"
)

func ControllerAlphaApplication() (*alpha.Response, error) {
    // create logger
    const appName, logFile, logMode, version = "alpha", "logs/log.svc.a.log", "prd", "v1.0.0"
    lg := logger.NewZapLogger(logFile, logMode, version, appName)

    // prepare contextual logging
    const XRequestIDHeader = "1239893491327"
    requestData := alpha.Request{
        AlphaContent:   "John Doe",
        BravoContent:   "Hello World from Bravo to",
        CharlieContent: "Hello World from Charlie to",
    }
    lg = lg.With("request_id", XRequestIDHeader)
    lg = lg.With("request_data", requestData)

    // prepare context for contextual logging
    ctx := context.Background()
    ctx = context.WithValue(ctx, "Request-Id", XRequestIDHeader)
    ctx = context.WithValue(ctx, zap.SugaredLogger{}, lg)

    // calling business logic
    service := alpha.NewAlphaApplication(
        appName,
        version,
        ControllerBravoApplication,
        ControllerCharlieApplication,
    )
    response, err := service.AlphaLogicFlowEntry(ctx, requestData)
    if err != nil {
        return nil, err
    }
    return response, nil
}

func ControllerBravoApplication(requestId string, request bravo.Request) (*bravo.Response, error) {
    // create logger
    const appName, logFile, logMode, version = "bravo", "logs/log.svc.b.log", "prd", "v2.0.0"
    lg := logger.NewZapLogger(logFile, logMode, version, appName)

    // prepare contextual logging
    lg = lg.With("request_id", requestId)
    lg = lg.With("request_data", request)

    // prepare context for contextual logging
    ctx := context.Background()
    ctx = context.WithValue(ctx, "Request-Id", requestId)
    ctx = context.WithValue(ctx, zap.SugaredLogger{}, lg)

    // prepare context for contextual logging
    service := bravo.NewBravoApplication(appName, version)

    // calling business logic
    return service.BravoLogicFlowEntry(ctx, request)
}

func ControllerCharlieApplication(requestId string, request charlie.Request) (*charlie.Response, error) {
    // create logger
    const appName, logFile, logMode, version = "charlie", "logs/log.svc.c.log", "prd", "v3.0.0"
    lg := logger.NewZapLogger(logFile, logMode, version, appName)

    // prepare contextual logging
    lg = lg.With("request_id", requestId)
    lg = lg.With("request_data", request)

    // prepare context for contextual logging
    ctx := context.Background()
    ctx = context.WithValue(ctx, "Request-Id", requestId)
    ctx = context.WithValue(ctx, zap.SugaredLogger{}, lg)

    // prepare context for contextual logging
    service := charlie.NewCharlieApplication(appName, version)

    // calling business logic
    return service.CharlieLogicFlowEntry(ctx, request)
}

func main() {
    response, err := ControllerAlphaApplication()
    if err != nil {
        panic(err)
    }
    fmt.Println("BRAVO:", response.BravoManipulatedContent)
    fmt.Println("CHARLIE:", response.CharlieManipulatedContent)
}

Contextual Logging

Within each business logic function, there is a chunk of code responsible for combining the package name and function name. This combined information is stored inside the fName variable and is used for each line of logging. Additionally, the code extracts a *zap.SugaredLogger from the ctx where the log instance already contains request_id and request_data passed from the controller layer. Therefore, when the log instance is used, it already contains the necessary data for tracing. Here is the corresponding chunk of code.

    // extract zap logger inside context that passed from controller where
    // request received and logger from controller its already has unique
    // request id everytime request received.
    fName := fmt.Sprintf("%s.%s", a.Package, "businessLogicFlow")
    cl, ok := ctx.Value(zap.SugaredLogger{}).(*zap.SugaredLogger)
    if !ok {
        return nil, errors.New("contextual logging not pass through context")
    }
    cl.Infow(fName, "reason", "execution started")
    defer cl.Infow(fName, "reason", "execution ended")

In certain cases, such as requestToBravoApp and requestToCharlieApp functions, there is a need to extract the request_id value. This extracted value will be passed to the next sub-application. in order to keep the next application still retains the same request_id while executing its business logic. Here is the corresponding code snippet.

    // extract request id to be passed to charlie application
    requestId, ok := ctx.Value("Request-Id").(string)
    if !ok {
        return nil, errors.New("request id not pass through context")
    }

    // sending request to charlie application
    response, err := a.charlieClient(requestId, charlie.Request{
        Content: req.CharlieContent,
    })
    if err != nil {
        cl.Errorw(fName, "reason", err.Error())
        return nil, errors.New("error while contacting charlie service")
    }

So whenever you try to find logs which are contained certain request_id it will show you which are correlated line of logs. Each lines of log represents business logic execution flows, which are functions that executed while business logic function running. This is called Contextual Logging.

Merging Log Files and Test It

In this section we will try to retrieve certain lines of logs, by using this 1239893491327 request id. I have merge these log file log.svc.a.log, log.svc.b.log and log.svc.c.log into log.txt and now try to retrieve correlated lines by that request id using awk command. awk is tool for text processing similar with grep.

Command

awk '/"request_id":"1239893491327"/' log.txt

Response

{"level":"info","timestamp":"2024-05-17T18:41:21.312+0700","msg":"applications.alpha.AlphaLogicFlowEntry","service_pid":34716,"service_name":"alpha","service_version":"v1.0.0","request_id":"1239893491328","request_data":{"AlphaContent":"John Doe","BravoContent":"Hello World from Bravo to","CharlieContent":"Hello World from Charlie to"},"reason":"execution started"}
.......... other lines ..........
{"level":"info","timestamp":"2024-05-17T18:41:21.312+0700","msg":"applications.charlie.BravoLogicFlowEntry","service_pid":34716,"service_name":"charlie","service_version":"v3.0.0","request_id":"1239893491328","request_data":{"Content":"Hello World from Charlie to"},"reason":"execution ended"}

Even with simple command you can trace execution flow by the request id.

If you want use some formating that awk can does. Just perform your awk command.

Command

awk -F'[,:}]' '/"request_id":"1239893491327"/ {print "Function Name:", $8, "Timestamp:", $4, "Service Name:", $14, "Reason:", $22}' log.txt

Response

Function Name: "applications.alpha.AlphaLogicFlowEntry" Timestamp: "2024-05-17T18 Service Name: "v1.0.0" Reason: "CharlieContent"
.......... other lines ..........
Function Name: "applications.charlie.BravoLogicFlowEntry" Timestamp: "2024-05-17T18 Service Name: "v3.0.0" Reason: "execution ended"

To try different request_id you can change it in main.go file find this line code.

const XRequestIDHeader = "1239893491327"

And change the random number, in real case this number generated by client who sent the request, or you can generate it automatically, but you need to return the request id everytime you application respond to client, this is like the key if a request gone wrong. So you can trace its flow using its request id.

This will be more interesting when the log file loaded using Logstash and feed to Kibana, as your log finder.

Conclusion

Contextual Logging is crucial parts when you want to move into microservice architecture, where service might be communicate with other service. So when error happened in other service, you can still trace the execution flow across services. This might be helpful for improving maintainability, readability and help you to find error faster.

My Thanks

Thank you for visiting! I hope you found it useful and enjoyable. Don't hesitate to reach out if you have any questions or feedback. Happy reading!

Dependency Injection: A Straightforward Implementation in Golang

Synexis Microsystem — Fri, 10 May 2024 08:35:24 +0000

Preamble

Before we move into the implementation, we should know why we use this kind of thing? what is the purpose of implement dependency injection? So dependency injection helps to decouples components in a system by removing direct dependencies between them. making them more modular and easier to understand, maintainable, and testable. In simple explanation you do not have to do initialization of an instance everytime you need it, just take it from dependency injection, so your coding process will be straightforward. In this moment we will use Uber Dig to achive that.

Implementation

We will entering coding and instruction section. And I expecting you already knows basic commands of your lovely OS and Coding stuffs. Let's go to the detail.

Initialize new Golang Project.

mkdir di-tutor
cd di-tutor
go mod init

Make directory named di inside di-tutor directory.

mkdir di

Make file named init.go inside di directory

cd di
touch init.go

Content init.go file.

package di

import (
    "go.uber.org/dig"
)

// Container master containers for all dependencies injected
// this global variable will be accessed from main function
// and will provide needed instances across functionalities
var Container = dig.New()

// Injected this struct represents dependencies injections
// bank whole injected instance will be accessed from this
// structure.
type Injected struct {
    FooBar string
}

// NewInjected initialize dependencies injection entries
// for all dependencies based what this function params
// needed will be injected again using Injected struct.
func NewInjected() *Injected {
    return &Injected{
        FooBar: "Hello This is FooBar content",
    }
}

// init default initialization function from golang
func init() {
    var err error
    // Injecting needed dependencies across functionalities

    // Wrapping up all injected dependencies
    err = Container.Provide(NewInjected)
    if err != nil {
        panic(err)
    }
}

Inside this file is where all instances that the application needs, got injected. But for this moment it only inject *Injected struct with "Hello This is FooBar content" string inside FooBar attribute.

Load *Injected struct from main.go file.

package main

import (
    "fmt"
    "github.com/yourgituname/di-tutor/di"
)

func main() {
    err := di.Container.Invoke(func(inj *di.Injected) {
        fmt.Println(inj.FooBar)
    })
    if err != nil {
        panic(err)
    }
}

Inside the main.go file we try to load *Injected struct that defined inside init.go file on di directory. And let's try to print string content from FooBar attribute of *Injected struct.

Download needed libraries in project

go mod vendor

Run main.go

go run main.go

The output after you run main.go will look like this.

Use Case

In last section we talk about injecting a struct that responsible to holds all injected instance, but what is the real use case? I do not want to use Backend Engineering for the use case, its too specific and to advanced, let's use human anatomy as the use case. First we need to think what is human anatomy has? Let's breakdown.

Human has Head and Body
- Head Section has (Eyes, Ears) (Just 2 for simplification)
- Body Section has (Hands, Legs) (Just 2 for simplification)

Abstracting

In this sub-section let's make an abstraction for Human and their anatomy. Make new package that will look like this. Follow this image.

Head

ears.go

package head

import "fmt"

type Ears struct{}

func (e *Ears) Hearing() {
    fmt.Println("Hearing...")
}

func NewEars() *Ears {
    return &Ears{}
}

eyes.go

package head

import "fmt"

type Eyes struct{}

func (e *Eyes) Seeing() {
    fmt.Println("Seeing...")
}

func NewEyes() *Eyes {
    return &Eyes{}
}

di.go on head directory

package head

import (
    "go.uber.org/dig"
)

type DependenciesHolder struct {
    dig.In
    Ears *Ears
    Eyes *Eyes
}

func RegisterDependencies(container *dig.Container) error {
    var err error
    err = container.Provide(NewEars)
    err = container.Provide(NewEyes)
    if err != nil {
        return err
    }
    return nil
}

Body

hands.go

package body

import "fmt"

type Hands struct{}

func (h *Hands) TakeWithRightHand() {
    fmt.Println("Taking with right hand")
}

func (h *Hands) TakeWithLeftHand() {
    fmt.Println("Taking with left hand")
}

func (h *Hands) PunchWithRightHand() {
    fmt.Println("Punching with right hand")
}

func (h *Hands) PunchWithLeftHand() {
    fmt.Println("Punching with left hand")
}

func NewHands() *Hands {
    return &Hands{}
}

legs.go

package body

import "fmt"

type Legs struct{}

func (l *Legs) WalkWithRightLeg() {
    fmt.Println("Walking with right leg")
}

func (l *Legs) WalkWithLeftLeg() {
    fmt.Println("Walking with left leg")
}

func (l *Legs) KickWithRightLeg() {
    fmt.Println("Kicking with right leg")
}

func (l *Legs) KickWithLeftLeg() {
    fmt.Println("Kicking with left leg")
}

func NewLegs() *Legs {
    return &Legs{}
}

di.go on body directory

package body

import (
    "go.uber.org/dig"
)

type DependenciesHolder struct {
    dig.In
    Hands *Hands
    Legs  *Legs
}

func RegisterDependencies(container *dig.Container) error {
    var err error
    err = container.Provide(NewHands)
    err = container.Provide(NewLegs)
    if err != nil {
        return err
    }
    return nil
}

Human

human.go

package human

import (
    "github.com/yourgituname/di-tutor/human/body"
    "github.com/yourgituname/di-tutor/human/head"
)

type Human struct {
    HeadDependenciesHolder head.DependenciesHolder
    BodyDependenciesHolder body.DependenciesHolder
}

func (h *Human) RunningAllHumanFunction() {
    // head
    h.HeadDependenciesHolder.Ears.Hearing()
    h.HeadDependenciesHolder.Eyes.Seeing()

    // body
    h.BodyDependenciesHolder.Hands.TakeWithRightHand()
    h.BodyDependenciesHolder.Hands.TakeWithLeftHand()
    h.BodyDependenciesHolder.Hands.PunchWithRightHand()
    h.BodyDependenciesHolder.Hands.PunchWithLeftHand()
    h.BodyDependenciesHolder.Legs.WalkWithRightLeg()
    h.BodyDependenciesHolder.Legs.WalkWithLeftLeg()
    h.BodyDependenciesHolder.Legs.KickWithRightLeg()
    h.BodyDependenciesHolder.Legs.KickWithLeftLeg()
}

func NewHuman(
    headDependenciesHolder head.DependenciesHolder,
    bodyDependenciesHolder body.DependenciesHolder,
) *Human {
    return &Human{
        HeadDependenciesHolder: headDependenciesHolder,
        BodyDependenciesHolder: bodyDependenciesHolder,
    }
}

Inject Head and Body `DependenciesHolder` and `*Human` struct on `init.go` file

init.go

package di

import (
    "github.com/yourgituname/di-tutor/human"
    "github.com/yourgituname/di-tutor/human/body"
    "github.com/yourgituname/di-tutor/human/head"
    "go.uber.org/dig"
)

// Container master containers for all dependencies injected
// this global variable will be accessed from main function
// and will provide needed instances across functionalities
var Container = dig.New()

// Injected this struct represents dependencies injections
// bank whole injected instance will be accessed from this
// structure.
type Injected struct {
    FooBar string
    Head   head.DependenciesHolder
    Body   body.DependenciesHolder
    Human  *human.Human
}

// NewInjected initialize dependencies injection entries
// for all dependencies based what this function params
// needed will be injected again using Injected struct.
func NewInjected(
    hd head.DependenciesHolder,
    bd body.DependenciesHolder,
    hm *human.Human,
) *Injected {
    return &Injected{
        FooBar: "Hello This is FooBar content",
        Head:   hd,
        Body:   bd,
        Human:  hm,
    }
}

// init default initialization function from golang
func init() {
    var err error
    // Injecting needed dependencies across functionalities
    err = head.RegisterDependencies(Container)
    err = body.RegisterDependencies(Container)
    err = Container.Provide(human.NewHuman)

    // Wrapping up all injected dependencies
    err = Container.Provide(NewInjected)
    if err != nil {
        panic(err)
    }
}

Load it from `main.go`

main.go

package main

import (
    "github.com/yourgituname/di-tutor/di"
)

func main() {
    err := di.Container.Invoke(func(inj *di.Injected) {
        inj.Human.RunningAllHumanFunction()
    })
    if err != nil {
        panic(err)
    }
}

Now try to run updated main.go file

go run main.go

The output of updated main.go will look like this.

If you want to read directly from github, I have push the repository so hope you got better understanding

Dependency Injection Tutorial Github

Conclusion

Hope you understand the whole objective of this tutorial, the conclusion is. With uber dig dependency injection you don't need to directly initialize an instance, just by put initialize function of an instance uber dig automatically finds injected instance by referencing parameters type, if the type is same will it automatically use injected instance and otherwise it will error and telling you the instance with certain type is not there. And by implement dependency injection will help you structure your dependency accros your code base, minimalize coupling by initialize every dependency at the first run.

My Thanks

Thank you for visiting! I hope you found it useful and enjoyable. Don't hesitate to reach out if you have any questions or feedback. Happy reading!

Microservice: Consideration Before Migrating

Synexis Microsystem — Thu, 09 May 2024 14:54:43 +0000

Preamble

Planning to shift to Microservices? Before you make the leap, let's discuss a few key things to keep in mind. Considering these factors upfront can make your transition smoother and ensure you maximize the benefits of Microservices.

Maintenance Cost

When your team decide to go Microservice, you will need multiple compute engine where your service put on. In my opinion for this problem you can go with Domain-Driven-Design concept, it teaches you not to overslice your big service into "Too Much Services" and it show you ways how to categorizing service based on its busines domain, and decrease amout of compute engine should be prepared for your services. Ofc it affects your cost to spend on compute engines. Here's an example for Domain-Driven-Design.

Scaling Factors

Before moving to Microservice you should check your service, is it Stateless or Stateful service? because it help you to decide using Horizontal or Vertical scaling?

What is Stateful and Stateless Service?

Here is a illustration of Stateful and Stateless in simple way.

Stateful is a behavior of a System or you can said application that holds bunch of retained data to supports its functionallity.

Stateless is a behavior of a System or you can said application that does not holds retained data to supports it functionallity.

What is Horizontal and Vertical Scalling?

Here is a illustration of Horizontal and Vertical Scalling in simple way.

Horizontal Scalling method is, Increase amount of Physical Device or Cloud Instance, where your service will be put on.

Verticall Scalling method is, Increase Compute Power of existing Physical Device or Cloud Instance, where your service will be put on.

Stateful and should have Load Balancer?

If yes, in this condition your service should not use Horizontal scaling, because every instances for Load Balancing purpose will be state inconsistent. And it will return different output while Load Balancing choose which service should be burdened next. If this condition inevitable you need to consider how retained data of each service synchronized around instances.

Stateful and should not have Load Balancer?

If yes, in this condition your service could use Vertical scaling, because when the service get stressed, instance will automatically increase computation power.

Stateless and should have Load Balancer?

If yes, in this condition your service could use Horizontal scaling, because there is no retained data inside each service on the instances, and each service will return consistent output everytime the Load Balancer moving to next service.

Stateless and should not have Load Balancer?

If yes, in this condition your service could use Vertical scaling, because there is no retained data inside each service on the instances, and when the service get stressed, instance will automatically increase computation power.

Data Source Strategy

Choosing which Data Source Engine should be used is a crucial decision. You need to address the problems your business might face, whether to prioritize performance over consistency or vice versa. In our development journey there are two Data Source Engine which are popular, there are NoSQL and SQL. Let's simplify, go with NoSQL to prioritize performance, otherwise go with SQL to embrace consistency. Or you can mixed it up for flexibility. After you have decided let's talk about data source strategy for your service.

One Database for All Service

In a microservices setup, "One Database for All Service", means that every service you have, uses the same central database to store information.

Advantages

Simplicity: Managing a single database can simplify the overall architecture, as there's no need to deal with distributed data management or service-specific databases.
Consistency: Since all services share the same database, maintaining data consistency and integrity can be easier compared to having multiple databases with potential data duplication or inconsistency.
Data Sharing: All services have access to the same dataset, enabling easier data sharing and communication between services.
Cost Savings: Managing a single database may be more cost-effective in terms of infrastructure and maintenance compared to multiple databases.

Challenges

Scaling: Scaling can become challenging as the system grows, as all services share the same database resources. Performance bottlenecks may arise if the database becomes a bottleneck.
Dependency: All services become tightly coupled to the shared database schema, making it difficult to make changes to the database schema without affecting multiple services.
Data Ownership: It can be unclear which service owns which data, leading to potential confusion and conflicts in data management responsibilities.
Security: Access control and security concerns become critical, as any security breach or data leakage in the shared database can affect multiple services.

One Database for Each Service

In a microservices setup, "One Database for Each Service" approach means each service you have, has its own database to handle its data.

Advantages

⁠Isolation: Each microservice has its own database, promoting isolation and independence. This isolation reduces the risk of data coupling between services and allows for more straightforward database schema changes without impacting other services.
⁠Scalability: Scaling becomes more granular and flexible since each service can scale its database independently based on its specific workload and performance requirements.
⁠Flexibility: Different services may have diverse data storage needs, such as relational, document-oriented, or key-value data stores. With separate databases per service, teams can select the most appropriate database technology for each service's requirements.
⁠ ⁠Security: Data isolation enhances security by limiting access to each microservice's data only to authorized users and services, reducing the risk of unauthorized access or data breaches affecting multiple services.

Challenges

Data Consistency: Ensuring data consistency across multiple microservices can be challenging when related data spans multiple databases. Implementing distributed transactions or eventual consistency mechanisms may be necessary to maintain data integrity.
⁠Complexity: Managing multiple databases introduces additional operational complexity, including backup and recovery, monitoring, and schema migrations. Teams must have the necessary expertise and tooling to manage these databases effectively.
⁠Resource Consumption: Running multiple databases increases resource consumption, including storage, memory, and CPU usage. Proper resource allocation and optimization are essential to avoid resource contention and performance degradation.

One Shared Database for Each Service

In a microservice setup, the "One Shared Database for Each Service" approach means every service you have has its own database, but other service also can access others service database.

Advantages

Isolation and Independence: Similar to the "One Database for Each Service" approach, each microservice retains its database, promoting isolation and independence. This separation allows for individual service teams to manage their data schemas and operations without affecting other services.
Resource Sharing: By sharing the same underlying database infrastructure, organizations can benefit from resource pooling and optimization. This can lead to more efficient resource utilization compared to managing entirely separate database clusters for each service.
⁠Consistency and Compatibility: Since all services interact with the same database schema, maintaining data consistency and ensuring compatibility between services becomes more straightforward. Changes to the database schema can be applied uniformly across all services, reducing the risk of schema mismatches or compatibility issues.
⁠Operational Efficiency: Managing a shared database infrastructure simplifies operational tasks such as backup and recovery, monitoring, and database upgrades. Organizations can leverage centralized tools and processes to streamline database management across all services.

Challenges

⁠Data Coupling: While each service has its database instance, they are still coupled at the database schema level. Changes to the database schema may impact multiple services, requiring coordination and communication between service teams to ensure smooth migrations and updates.
Performance Bottlenecks: Introducing a shared database infrastructure may lead to performance bottlenecks, especially if certain services generate high volumes of traffic or have complex data access patterns. Careful monitoring and optimization are necessary to prevent performance degradation.
⁠Security Concerns: Sharing a database infrastructure raises security concerns, as unauthorized access or data breaches in one service's database could potentially impact other services. Robust access control mechanisms and security measures must be implemented to mitigate these risks.

Communication Style

There are two popular communication styles: REST API and gRPC. Both have similar concepts but differ in output result. While REST API remains powerful and widely used, it may not be ideal for communication between services. It is more suitable for communicating with customer-facing systems due to its use of string-based responses, which are easy to serialize. On the other hand, gRPC is an excellent choice for communication between services because it uses bytes as the output, which is less CPU intensive to consume, aiming for better data exchange performance. However, gRPC may not be ideal for communication with customer-facing systems.

REST API

REST API, while still powerful and widely used, may not be the best choice for communication between services. It is more suited for interactions with customer-facing systems due to its string-based response format, which is easy to serialize. REST API offers you rapid development because of easy to implement and its popularity.

gRPC

gRPC is a great choice for communication between services, offering better data exchange performance by using bytes as the output. However, it may not be the ideal option for communicating with customer-facing systems. gRPC might cause development overwhelm because its tricky to implement and many steps to follow to make one data model.

Distributed Tracing

When decide moving to Microservice some of you might forget implement Distrubute Tracing to all service, and bottleneck becomes hard to find, without distributed tracing, and you need to research about how to use distributed tracing, for handle this problem, and you need to serve the information about your service to distributed tracer by adding event in every repository function and business logic function, to trace out the bottleneck, you can use whatever you want, OpenTelemetry is the one I recommended to you.

Contextual Logging and Logstash with Kibana

This one is a must, every activity from incoming request until the service response to a request, it should be stampled with "Request Id" or you can use other terms for that, this one called Contextual Logging where you can trace execution flow of a business logic request only by Request Id of a request, Logstash with Kibana will be very good choice for you to implement, it helps you to wrapping up all Logging Files content into Logstash and feeded to Kibana, and you just need to use the Query in the Kibana Dashboard, you will find execution footprint accross the service, if you implement it properly.

Conclusion

In conclusion, before making the move to Microservices, it's important to weigh whether your service is stateless or stateful, choose between horizontal or vertical scaling, and consider reducing maintenance costs through Domain Driven Design categorization. Additionally, setting up contextual logging and Logstash with Kibana, along with implementing Distributed Tracing, can greatly facilitate debugging across services. And for Data Source Strategy in this topic provides 3 strategies, you can define your own strategy with better consideration and best for your service, those are just example base on my experience. Communication Style, REST API or gRPC its depends on your goal, and rapidity of development phase, you can go with both of them, if you want separate gRPC for communicate between service and REST API for customer-facing system, and you can still use REST API for all communication. By addressing these considerations, you can ensure a smoother transition to Microservices and enhance the overall efficiency of your system.

My Thanks

Thank you for visiting! I hope you found it useful and enjoyable. Don't hesitate to reach out if you have any questions or feedback. Happy reading!

Go Test with Testify: Most Simple Way to Implement Unit Test

Synexis Microsystem — Sun, 05 May 2024 01:24:11 +0000

Preamble

Unit test is testing specific method using several test cases, that performed by the developer who create the method, to prove the method running as expected.

Preparation

Let’s say we have a method that generate OTP code that we can specify the length of the generated OTP code, and we want to test it, in this story I’ll using testify for assertion toolkit. Please take a look this library Testify Go.

Coding Time

OTP Generator Code

package generator

import (
    "crypto/rand"
    "errors"
    "fmt"
)

func OTP(length int) (string, error) {
    const otpChars = "1234567890"
    if length < 1 {
        return "", errors.New("can't generate code with less than 1")
    }
    buffer := make([]byte, length)
    _, err := rand.Read(buffer)
    if err != nil {
        return "", fmt.Errorf("OTP generator error : %+v", err)
    }
    otpCharsLength := len(otpChars)
    for i := 0; i < length; i++ {
        buffer[i] = otpChars[int(buffer[i])%otpCharsLength]
    }
    return string(buffer), nil
}

Let's Test it!

In golang test file should be inside the same package of the function which will be tested, for better understand your OTP generator method is inside otp.go and the test file should be otp_test.go, here is the sample structure of file.

Unit Testing Code for OTP Generator Code

package generator

import (
    "github.com/stretchr/testify/suite"
    "testing"
)

// otpTestSuite embedding suite.Suite of testify
type otpTestSuite struct {
    suite.Suite
}

// TestOTPTestSuite initialize test suite
func TestOTPTestSuite(t *testing.T) {
    suite.Run(t, new(otpTestSuite))
}

// TestOTP testing OTP() function using several test cases
func (s *otpTestSuite) TestOTP() {

    // make test cases
    tests := []struct {
        TestCaseName string
        Length       int
        WantError    bool
    }{
        {
            TestCaseName: "Test case generate otp with 6 digit and no error",
            Length:       6,
            WantError:    false,
        },
        {
            TestCaseName: "Test case generate otp with 12 digit and no error",
            Length:       12,
            WantError:    false,
        },
        {
            TestCaseName: "Test case generate otp with 0 digit and error",
            Length:       0,
            WantError:    true,
        },
    }

    // scan test cases
    for _, test := range tests {
        s.Run(test.TestCaseName, func() {
            // start test based on test case
            otp, err := OTP(test.Length)
            if test.WantError {
                s.Assert().Error(err)
            } else {
                s.Assert().NoError(err)
            }
            s.Assert().Len(otp, test.Length)
        })
    }

}

Run Through The Go Unit Test Files.

To run go test recursively change your current directory to root of project, then you can runing this command.

go test -v ./...

And the result should look like this.

Run Through The Go Unit Test Files with Coverage Details.

Still in side the root directory of your project. You can run this command.

go test ./... -coverprofile cover.out
go tool cover -html=cover.out

And automatically open browser and show the detail coverage of the OTP code statements.

Conclusion

So, to sum it up, unit testing in Golang is like having a trusty sidekick for your code. With tools like Testify, you can easily check if your functions are doing what they're supposed to, running tests and checking coverage is as simple as a couple of commands.

My Thanks

Thank you for visiting! This article was originally published on another website. You can find the original source here. I'm excited to bring it over here. I hope you found it useful and enjoyable. Don't hesitate to reach out if you have any questions or feedback. Happy reading!

Server-Sent Event: One Directed Data Transmission Server to Client

Synexis Microsystem — Sun, 05 May 2024 00:49:26 +0000

Preamble

SSE is unidirectional data transfer mechanism from the server to the client, that you need to perform GET request with Content-Type as text/event-stream, for tells the server you need to established server event channel and the server will keeps send you data changes, when you trigger an event.

Use Cases

Live Feeds, retrieve feeds data in real time.
Events, retrieve data when specific event was triggered.
Polling, retrieve data when there is a change.

Why not using WebSocket?

If you need bidirectional communication you can use websocket, even websocket can be used for unidirectional, but SSE is much simpler for communication with one direction (unidirectional).

Make SSE Server with Go

You can just copy this code into main.go for start learning how SSE works, after that you can read the code.

package main

import (
    "fmt"
    "log"
    "net/http"
)

// data variable
var messageChan chan string

func prepareHeaderForSSE(w http.ResponseWriter) {
    // prepare the header
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("Access-Control-Allow-Origin", "*")
}

func writeData(w http.ResponseWriter) (int, error) {
    // set data into response writer
    return fmt.Fprintf(w, "data: %s\n\n", <-messageChan)
}

func sseStream() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // call prepareHeaderForSSE for start endpoint as SSE server
        prepareHeaderForSSE(w)

        // initialize messageChan
        messageChan = make(chan string)

        // calling anonymous function that
        // closing messageChan channel and
        // set it to nil
        defer func() {
            close(messageChan)
            messageChan = nil
        }()

        // create http http.Flusher that allows
        // http handler to flush buffered data to
        // client until closed
        flusher, _ := w.(http.Flusher)
        for {
            write, err := writeData(w)
            if err != nil {
                log.Println(err)
            }
            log.Println(write)
            flusher.Flush()
        }
    }
}


// sendMessage used to write data into messageChan and flushed to client through sseStream
func sseMessage(message string) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if messageChan != nil {
            messageChan <- message
        }
    }
}

func main() {
    http.HandleFunc("/stream", sseStream())
    http.HandleFunc("/send", sseMessage("you can put json in here as the data"))
    http.HandleFunc("/right", sseMessage("right"))
    http.HandleFunc("/left", sseMessage("left"))
    log.Fatal("HTTP server error: ", http.ListenAndServe("localhost:8080", nil))
}

Once the server successfully running, you can test it via google chrome that will pretend like client side, and listen for data through stream endpoint that we have configured as SSE stream.

Using Google Chrome console for client

We use google chrome console, because its more representing server and client, (SSE) acting as the server & (google chrome console) acting as the client. you can start with this JavaScript code.

After that you can test sending data with /send, /left, /right endpoint, for seeing data will be printed using console.log.

Using cURL for Trigger the Event

Send event endpoint trigger

curl http://localhost:8080/send

Send event response after triggered

Left event endpoint trigger

curl http://localhost:8080/left

Left event response after triggered

Right event endpoint trigger

curl http://localhost:8080/right

Right event response after triggered

Conclusion

As you can see the value of data that google chrome console received from /stream endpoint is different every time we call 3 different endpoints, you can do the same like google chrome console does inside React.js framework using axios, or other programming language. And SSE server is only can handle one client and cannot be more, because the purpose of SSE is just to send data depends on event endpoints that we trigger through http request.

My Thanks

Redis Pipeline: The Way of Sending Redis Commands in One Shot

Synexis Microsystem — Sun, 05 May 2024 00:07:40 +0000

Preamble

Redis Pipeline is technique for improving performance by wrapping up bunch of Redis commands into a single process, without waiting for the response to each individual command.

Round-Trip Time (RTT)

Redis client-server using TCP connection to established, where the client send redis command to the server and client waiting for server to response, that’s how Redis command executed without Pipeline, it’s still ok if you only execute one command, when you execute multiple command it’s going to be faster if you use pipeline.

No Pipeline

Client ask to open connection to server, and execute one redis commands through one TCP request, and client do the same but with different redis command, until 3 times and connection closed.

Pipeline

Client ask to open connection to server, and execute several
redis commands through one TCP request, and server executing all commands that client sent, without response each command status to client, after server done, it will send response to client that pipeline already execute those commands.

Benchmark Result

This benchmark data generated by golang benchmark, I use golang as the redis client for prove which one is faster by comparing bulk insert 1024 data, using Non Pipeline & Pipeline.

Non Pipeline

Non Pipeline take more and more time, through data quantity from 1 to 1024, it’s going slower because Non pipeline execute insert command and waiting the response, and executed inside different TCP Request, it’s mean if the data quantity increase it goes slow, because the client perform 1024 TCP Request to server.

Pipeline

Pipeline time execution is almost linear, with pipeline the client perform multiple command that sent through one TCP Request, and server execute all the commands and response to client when all commands already execute, pipeline process 1024 command faster.

Conclusion

It's all depends on your need, don't over engineering simple task, if you need to perform multiple commands with faster, you can go with pipelined command, otherwise if you just perform one command you can go with non pipelined command.

DEV Community: Synexis Microsystem

Database Migration: Take Care of Your Database Changes

Preamble

Database Migration Directory

Database Server Setup

Install golang-migrate

Generate Migration Files

Users Table Migration Files

000001_create_users_table.up.sql

000001_create_users_table.down.sql

Profiles Table Migration Files

000001_create_profiles_table.up.sql

000001_create_profiles_table.down.sql

Implementation

connector.go

mlog.go

migration.go

How to Use

Standalone Program

UP

DOWN

Embedded to Application

Behavior

Conclusion

My Thanks

Contextual Logging: Keep Tracks Execution Flow

Preamble

Demonstration Project

Business Logic Flow

Log Files

Logger Library Setup

Logger logger/logger.go

Applications Code

Alpha alpha.go

Bravo bravo.go

Charlie charlie.go

Main main.go

Contextual Logging

Merging Log Files and Test It

Conclusion

My Thanks

Dependency Injection: A Straightforward Implementation in Golang

Preamble

Implementation

Use Case

Abstracting

Head

Body

Human

Inject Head and Body DependenciesHolder and *Human struct on init.go file

Load it from main.go

Conclusion

My Thanks

Microservice: Consideration Before Migrating

Preamble

Maintenance Cost

Scaling Factors

What is Stateful and Stateless Service?

What is Horizontal and Vertical Scalling?

Stateful and should have Load Balancer?

Stateful and should not have Load Balancer?

Stateless and should have Load Balancer?

Stateless and should not have Load Balancer?

Data Source Strategy

One Database for All Service

Advantages

Challenges

One Database for Each Service

Advantages

Challenges

One Shared Database for Each Service

Advantages

Challenges

Communication Style

REST API

gRPC

Distributed Tracing

Contextual Logging and Logstash with Kibana

Conclusion

My Thanks

`000001_create_users_table.up.sql`

`000001_create_users_table.down.sql`

`000001_create_profiles_table.up.sql`

`000001_create_profiles_table.down.sql`

`connector.go`

`mlog.go`

`migration.go`

Logger `logger/logger.go`

Alpha `alpha.go`

Bravo `bravo.go`

Charlie `charlie.go`

Main `main.go`

Inject Head and Body `DependenciesHolder` and `*Human` struct on `init.go` file

Load it from `main.go`