DEV Community: Firdavs Kasymov

Master DevOps career and interview

Firdavs Kasymov — Sun, 20 Nov 2022 18:41:45 +0000

DevOps is the intersection of Development and Operations, application release processes. When working on an application, the developer's main goal is to deliver the result to the end user, regardless of whether you use the Agile or Waterfall approach. The user must be able to use the app no matter what.
DevOps means making this continuous delivery process faster with minimal errors and preventing bugs. So, DevOps provides high-quality and well-tested improvements delivered to users.

This article accompanies the webinar which I had you can watch the full version.

What are the challenges that DevOps is supposed to solve?

The first and most important one is:

Miscommunications

Miscommunications and lack of collaboration between developers and operations. DevOps has to keep in mind that releasing an application includes the following:

Coding the app, deploying, and running it;
Developers are responsible for coding, and operations are responsible for running.

Developers usually face the problem of "I wrote an application, but I can not deploy and run it", while operations deal with the "I am running the app, but I don't know how it works". Developers code without considering where and how it will be deployed. And Operations try to deploy without understanding what and why they are deploying.

This usually results in miscommunications between these two. After finishing the code, developers often end up documenting the deployment guide too poorly, or some app's features might cause too many issues. As a result, the operations team struggles to deploy; they might even send it back with improvement suggestions. This kind of miscommunication could cause the release period to last for days, weeks, or even months.

So, there is no clearly defined automated process between the end of coding and the start of deployment. And even if there is one, it is based on complex checklists, with a constant need to get decisions approved by both teams.

Team cooperation

Traditionally, one of the teams is responsible for the development, while the other one has to deal with operations. These two have very different agendas, which makes it hard for them to cooperate properly. Developers want to push new features faster; the operations team, on the other hand, needs to make sure that those changes will not cause any issues since operations maintain the stability of production. They focus on making the app available, stable, safe, and so on.

This means that the app might take longer to be released, especially considering that the operations don't understand the code completely.

For example, a developer created a new feature that consumes too many resources in the production environment. Servers get overloaded and crush — now the operations team needs to fix it. Since it is the operation team that needs to put out the fire, developers often are not as careful as operations about every single change to the product and stability. Even though every employee's main goal must be delivering high-quality applications to end users fast. In practice, the developers do want to implement features and release them as quickly as possible, while operations focus on maintaining the system's stability. This is why they often resist new changes, creating a conflict of interest. So, this kind of setup naturally makes it difficult for these two teams to collaborate.

Security

Every company has to keep security their first priority when working on a new feature. The operations and security teams have to carefully evaluate any changes to make sure that nothing can affect system stability. In a traditional setup, this manual process as operations takes days, weeks, and months.

As mentioned before, DevOps is all about removing any roadblocks that slow down the process, including security issues. That is why the DevSecOps term was created — to highlight and remind teams about the importance of security.

Application Testing

Many projects hire a separate team or create new roles for testing applications on different levels, such as specific features, the complete app, different environments, and performance. Often these tests have to be done manually: teams can not always rely on their automated tests.

And only after manual testing is over can changes be released. Even though this may not be done by development or operation teams but rather by a separate tester role, this is an important part of the release process. It may also slow down the release process considerably.

Manual Work

Many tasks, such as testing, security checks, and deployments, used to be done manually during the release process. For example, operations would do most of their tasks manually by either directly executing commands on the servers to install tools, configure stuff, patches, or by creating a script or small program for execution.

But both cases include manual work — deploying applications, preparing the deployment environment, and configuring servers, user access, and permissions. This makes the process slower and more error-prone. With manual work, you have the disadvantage of knowledge sharing. It is complicated since people, who do all these tasks, have to document it, while others have to read it. It is also not a transparent, easy-to-trace process.

If infrastructure configurations are done manually, and something happens to infrastructure, recovering and replicating the exact same infrastructure state fast becomes impossible.

You have to remember exactly what was done to the servers. Again, the release process slows down due to new roadblocks.

When it comes to security and tests, a DevOps engineer has the right qualifications to go over the tasks of both the developer and operations teams. DevOps can remove all mentioned roadblocks that slow down the release process, whatever the problem might be.

Instead of manual, inefficient processes, DevOps can provide fully automated, streamlined ones that will make apps' releases easy and efficient.

How can a DevOps engineer help solve these issues?

DevOps is a combination of practices and tools that makes the releasing software fast and high quality.

DevOps allows developers and operations to work together more often. Different companies implement DevOps differently, so there is no standard way to become a DevOps engineer.

However, since the start of DevOps adoption, the process has taken on a more specific form with some common patterns in many companies, including creating an actual DevOps engineer role. The set of technologies used to implement the DevOps principles is called DevOps technologies which every DevOps engineer needs to learn. DevOps includes well-known CI (Continuous Integration) and CD (Continuous Delivery) processes.

How to become a DevOps engineer?

Let's see what makes up a CI/CD pipeline, what tools and concepts you need to learn to become a DevOps engineer, their tasks and responsibilities, and the line between DevOps and the development and operations teams.

It all starts with the application coded by developers that use specific technology stacks, different programming languages, and building tools. They will have a code repository to work on. One of the most popular ones is GIT. You, as a DevOps engineer, will not be programming the app. They need to understand how developers work, which GIT workflows they prefer, and how apps are configured with other services, such as databases, basic automated testing concepts, and so on.

DevOps has to create some infrastructure on-premise or cloud servers when the application is deployed in the server for users to access it. Again, as a DevOps engineer, you will be responsible for preparing the infrastructure to run the application. Since most of the servers and applications are running on Linux servers, basic knowledge of Linux (Linux shell commands and Linux File System), CLI, administering a server, and how to SSH into the server are required too.

Networking & Security

It would be best if you also learned the basics of networking and security, such as configuring firewalls to secure applications and opening ports to make them accessible from outside.

Other necessary skills include:

Learning how IP addresses Ports
DNS

However, DevOps engineers don't have to know advanced networking and security concepts and be able to administer the server from start to finish. System admins and network and security engineers usually specialize in these areas.

Your job as a DevOps engineer is to understand the concepts enough that you can prepare the server to run your application but not manage it and the whole infrastructure.

Containers

Nowadays, applications are run as so-called containers. This means you have to generally understand the virtualization and container concepts and manage containerized applications on the server. One of the most well-known containerized solutions today is Docker.

On the one hand, developers create new features and fix bugs. On another, we have an infrastructure and servers configured to run applications. The question is how to transfer new features and bug fixes from the
development team to the servers and make them available to the end users quickly and efficiently. This is the DevOps engineers' main goal.

Firstly, when the feature or bug fix is ready, we need to run tests and package the application as an artifact, such as a jar or zip file, so that we can deploy it. That is where the building and package manager tools come in (Maven Graden for the Java applications, and Npm for javascript applications).

The next step includes saving this artifact somewhere, for example, the image repository, Docker artifact repository on DockerHub, ECR (Amazon Elastic Container Registry). This means that DevOps engineers must understand how to create and manage artifact repositories by creating one pipeline that does all these in sequential steps. GitHub Actions, GitLab, or Jenkins automations can help you with that.

Connect pipelines with the Git repository to get the actual development code. This is part of the Continuous Integration (CI) process, when code changes in the Git repository are continuously tested. You only want to deploy new features and bug fixes to the server after they are tested, built, and packaged.

There could be more steps, for example, sending Slack notifications to
the team regarding the pipeline state or handling failed deployment. This flow represents the core of the CI/CD pipeline.

The CI/CD pipeline is at the core of all DevOps tasks and responsibilities. As a DevOps engineer, you should be able to configure it completely.

Nowadays, many companies use virtual infrastructure on the cloud (also known as infrastructure as a Service, such AWS, Google Cloud, and others) instead of creating their own physical infrastructure.

Doing your job would be impossible without learning the core concepts of at least one cloud infrastructure. These platforms manage lots of things for you. For example, using the UI admin portal of Cloud providers, you can create networks, firewalls, and all parts of your infrastructure through services.

For example, say your application runs on AWS. You will need to learn AWS cloud provider and its services (please, note that AWS is pretty complicated, but you don't have to know every single one of their services).

Container Orchestration (Kubernetes)

Our applications will run as containers. To manage them, Docker should be enough, but if you have a lot of containers and microservices, a more powerful container orchestration will be needed. The most popular one of them is Kubernetes.

Kubernetes is a powerful but very complex tool, so it requires a lot of effort to set up and manage multiple clusters for different teams.

Monitoring

Thousands of containers and hundreds of servers. How does one track performance of an individual application and infrastructure problems?

DevOps engineers set up and monitor applications, underlying Kubernetes clusters, and servers. Monitoring tools, such as Prometheus, usually help with that.

Infrastructure as a Code

Every project requires testing and development environment to properly prepare the application to deploy it. Creating and maintaining one infrastructure already takes a lot of time, and it is very error-prone. We don't want to do any testing manually.

As mentioned before, every DevOps engineer aims to automate as many processes as possible. So how do we automate creating infrastructure, configuring, and deploying? This can be done by two types of Infrastructure as Code tools:

Infrastructure provisioning tools (Terraform, Pulumi)
Configuration management tools (Ansible, Chef)

As a DevOps engineer, you must know at least one of these types to make.

Scripting Language

Since you would be working closely with developers and system administrators to automate some of the tasks, you will be writing scripts for them, maybe small applications, such as backups, system monitoring, cron jobs, or network management. In order to do that, you need to know a scripting language. This could be operating system-specific scripting languages, such as bash for Linux/Mac or Powershell for Windows, or more powerful and flexible languages (Golang or Python). These programming languages work regardless of what kind of operating system you have on servers or locally.

Golang is easy to learn, easy to read, and flexible. It has libraries for most of the databases, as well as different cloud platforms, such as AWS and Google Cloud.

Conclusion

You might be thinking right now: "How many of these tools do I need to learn? Do I need to learn multiple tools in each category? Which ones do I choose?"

You should learn the most popular and widely used tool in each category. Because once you start understanding one tool's concept, properly using alternative ones will become much easier.

It is important to learn these technologies all at once because this is what DevOps engineers do. It would be best if you started to use them on actual projects right away.

Restful API with Golang practical approach

Firdavs Kasymov — Mon, 17 Oct 2022 08:42:07 +0000

In this tutorial, we would be creating a Restful API with a practical approach of clean architecture and native Golang without any frameworks.

There are lots of tutorials out there, for Restful API but the issue they are facing is they don't have a clear structure and use various frameworks.

Those tutorials depend on the existence of some frameworks of Golang. With every external framework being used in the project, it comes with great responsibility. I don't have anything against these frameworks, but rather I prefer to minimize the use of external frameworks and stick with standard packages of Golang.

When a beginner starts learning Golang, most of them jump straight away to the famous external frameworks, such as Gin, Beego, Fiber and many more. Frameworks are good to implement something fast, but the main issue is here that beginners are using frameworks without understanding the native approach that Golang provides out of the box. This approach leads to the developer being dependent on the specific framework. If you are sticking with frameworks, what would you do if these frameworks are being deprecated, have bugs or are not being supported anymore?

All of these frameworks use the native Golang codes behind them with an extra layer which makes them frameworks.

So my two cents here is to try to learn standard Golang packages this way you will not get lost in the world of frameworks, as Golang provides most of the required things out of the box.

The initial requirement before we start our tutorial is to make sure that you have the following installed setup on your computer:
1) Golang (https://go.dev/doc/install)
2) IDE of your choice VSCode or GoLand JetBrains.
3) Postgresql (https://www.postgresql.org/download/)
4) Insomnia (https://insomnia.rest/)
5) Golang-Migrate (https://github.com/golang-migrate/migrate/blob/master/cmd/migrate/README.md)

For the purpose of this tutorial, we will not be using Docker as explaining Docker and it is function is out of the context of this tutorial.

Initial setup

Create your folder and initialize the app:



mkdir rest-api
cd rest-api
git init

Golang has a package manager which is handled by go.mod file. To generate this file, you must run the following command:
go mod init {Name Of Remote Repository}.

The convention to create a go.mod is to use the name of the remote repository of your app, in my case, it will be go mod init github.com/fir1/rest-api.

It is important to name it as a repository name because that is how Go manages dependencies when you execute the following command go get {PackageName}.

First, we need to create a project structure, before starting the actual implementation of Restful API.

I have already written a full blog regarding structuring a Golang application, you can read it in full by clicking here. (https://dev.to/firdavs_kasymov/a-practical-approach-to-structuring-golang-applications-1cc2)

So in this project, we would be following the same structure as it is explained in the previous blog.

Architecture

The architecture of our program will be looking like the following diagram:

As is clear from the diagram we have isolation, between Repository, Model, Service and Delivery layers.

This approach has been taken by the rules of the clean architecture of Uncle Bob more info at https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html

With this pattern we can have a clean and easy-to-follow logic, independent of any layer, so your business rules simply don't know anything at all about the outside world. The below flow explains how the layers access each other.

Delivery (REST, gRPC) -> Business Service -> Repository -> Model.

As you can see that we are not exposing the repository directly to the Delivery layer, instead only our business service has access to it. And once we have received an API call we just expose the business service to the delivery layer, which I think is quite logical to do and easier to follow.

Creating Restful endpoints

For the purpose of this tutorial we will be creating a simple REST API for the TO-DO list application which has a business logic of Creating, Reading, Updating and Deleting the todo item, known as (CRUD).

We will be using some essential libraries to make the REST API so please execute the following commands:



go get github.com/gorilla/mux

go get github.com/sirupsen/logrus

go get github.com/jmoiron/sqlx

go get github.com/lib/pq

go get github.com/asaskevich/govalidator

go get github.com/joho/godotenv

go get github.com/kelseyhightower/envconfig

Briefly, we will be explaining what each library does:

mux:
Package gorilla/mux implements a request router and dispatcher for matching incoming requests to their respective handler. Read in full from here (https://github.com/gorilla/mux)

logrus:
Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger. (https://github.com/sirupsen/logrus)

sqlx:
sqlx is a library which provides a set of extensions on go's standard database/sql library. (https://github.com/jmoiron/sqlx)

pq:
A pure Go postgres driver for Go's database/sql package (https://github.com/lib/pq)

govalidator:
A package of validators and sanitisers for strings, structs and collections (https://github.com/asaskevich/govalidator)

godotenv:
A library for loading .env config files

envconfig:
Library for managing configuration data from environment variables (https://github.com/joho/godotenv)

Config

In this project, we are going to use dependencies such as Postgres, as Postgres requires the client connection to provide credentials to be able to connect to the database.
So we have decided to create .env file in the root of project with the credentials for the database we are going to use:



DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USER=postgres
DATABASE_PASSWORD=password
DATABASE_NAME=rest
SERVER_PORT=80

The above credentials for Database might be different in your local computer, so make sure to adjust it accordingly.

Create a file in directory configs/config.go with the content:



package configs

import (
    "github.com/joho/godotenv"
    "github.com/kelseyhightower/envconfig"
)

type Config struct {
    Database   Database
    ServerPort int `envconfig:"SERVER_PORT" default:"80"`
}

type Database struct {
    Host     string `envconfig:"DATABASE_HOST" required:"true"`
    Port     int    `envconfig:"DATABASE_PORT" required:"true"`
    User     string `envconfig:"DATABASE_USER" required:"true"`
    Password string `envconfig:"DATABASE_PASSWORD" required:"true"`
    Name     string `envconfig:"DATABASE_NAME" required:"true"`
}

func NewParsedConfig() (Config, error) {
    _ = godotenv.Load(".env")
    cnf := Config{}
    err := envconfig.Process("", &cnf)
    return cnf, err
}

This code will be responsible for parsing .env files content to the struct Config.

Helper packages

In this section, we would be creating a helper package, which can be used within our project or can be imported to external projects.

Package: db
This package will have methods linked to the Database, such as connecting to the database, and error handling.

Create a folder in pkg/db which will include the files of db package.

Create the following associated files.

pkg/db/db.go



package db

import (
    "fmt"
    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"
)

type ConfingDB struct {
    Host     string
    Port     int
    User     string
    Password string
    Name     string
}

func Connect(cnf ConfingDB) (*sqlx.DB, error) {
    dsn := fmt.Sprintf(
        "host=%s port=%d user=%s password=%s dbname=%s sslmode=disable",
        cnf.Host,
        cnf.Port,
        cnf.User,
        cnf.Password,
        cnf.Name,
    )
    db, err := sqlx.Connect("postgres", dsn)
    return db, err
}

And the following file for error handling

pkg/db/error.go



package db

import (
    "database/sql"
    "errors"
    "fmt"
)

func HandleError(err error) error {
    if errors.Is(err, sql.ErrNoRows) {
        return ErrObjectNotFound{}
    }
    return err
}

// ErrObjectNotFound is used to indicate that selecting an individual object
// yielded no result. Declared as type, not value, for consistency reasons.
type ErrObjectNotFound struct{}

func (ErrObjectNotFound) Error() string {
    return "object not found"
}
func (ErrObjectNotFound) Unwrap() error {
    return fmt.Errorf("object not found")
}

Database migration

If you have followed the initial setup, you should have already "golang-migrate" installed locally.

So we use migration files all together to keep the track of database scheme changes.

For the purpose of this tutorial, we must create a table named todo for our database so for that we would be using golang-migrate tool.
First, create a folder migrations in the root directory.

Next, execute the following command from the root working directory of the project



migrate create -ext sql -dir migrations -seq create_todo_table

The command will automatically create two files inside migrations folder.

So we have now two files:

migrations/000001_create_todo_table.up.sql

We need to put all the changes which we would like to do in a database on this file, such as the creation of a new table or any changes related to the database.

Please put the following content into this file:



CREATE TABLE todo
(
    id SERIAL,
    name TEXT NOT NULL,
    description TEXT NOT NULL,
    status SMALLINT NOT NULL,
    created_on TIMESTAMP(0) WITHOUT TIME ZONE NOT NULL,
    updated_on TIMESTAMP(0) WITHOUT TIME ZONE,
    deleted_on  TIMESTAMP(0) WITHOUT TIME ZONE,

    PRIMARY KEY (id)
);

When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:



export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/rest?sslmode=disable'

Then run migration files by executing the command:



migrate -database ${POSTGRESQL_URL} -path migrations up

000001_create_todo_table.down.sql

In case of if you would like to roll back migrations, we will use .down.sql files, so in this tutorial rollback of migration would be to delete a created table.

So please put the following content into this file:



DROP TABLE IF EXISTS todo;

To rollback the previously executed migration run the following command:



migrate -database ${POSTGRESQL_URL} -path migrations down

Creating Layers

Model

We would be creating a separate folder for the model layer in the directory internal/todo/model/model.go inside the file please have the following content:



package model

import "time"

type Status int

const (
    StatusPending Status = iota + 1
    StatusInProgress
    StatusDone
)

func (s Status) IsValid() bool {
    switch s {
    case StatusPending:
        return true
    case StatusInProgress:
        return true
    case StatusDone:
        return true
    }
    return false
}

type ToDo struct {
    ID          int        `db:"id"`
    Name        string     `db:"name"`
    Description string     `db:"description"`
    Status      Status     `db:"status"`
    CreatedOn   time.Time  `db:"created_on"`
    UpdatedOn   *time.Time `db:"updated_on"`
    DeletedOn   *time.Time `db:"deleted_on"`
}

We would be using model purely to interact with the Database, to get the object from it and parse it to our model.

Repository

Let's create our repository which will be responsible for handling all the database interactions for the ToDo model.

Create a separate directory for the repository internal/todo/repository
and have file repository.go within:



package repository

import (
    "context"
    "fmt"
    "github.com/fir1/rest-api/internal/todo/model"
    "github.com/fir1/rest-api/pkg/db"
    "github.com/jmoiron/sqlx"
)

type Repository struct {
    Db *sqlx.DB
}

func NewRepository(db *sqlx.DB) Repository {
    return Repository{Db: db}
}

func (r Repository) Find(ctx context.Context, id int) (model.ToDo, error) {
    entity := model.ToDo{}
    query := fmt.Sprintf(
        "SELECT * FROM todo WHERE id = $1 AND deleted_on IS NULL",
    )
    err := r.Db.GetContext(ctx, &entity, query, id)
    return entity, db.HandleError(err)
}

func (r Repository) Create(ctx context.Context, entity *model.ToDo) error {
    query := `INSERT INTO todo (name, description, status, created_on, updated_on)
                VALUES (:name, :description, :status, :created_on, :updated_on) RETURNING id;`
    rows, err := r.Db.NamedQueryContext(ctx, query, entity)
    if err != nil {
        return db.HandleError(err)
    }

    for rows.Next() {
        err = rows.StructScan(entity)
        if err != nil {
            return db.HandleError(err)
        }
    }
    return db.HandleError(err)
}

func (r Repository) Update(ctx context.Context, entity model.ToDo) error {
    query := `UPDATE todo
                SET name = :name, 
                    description = :description, 
                    status = :status, 
                    created_on = :created_on, 
                    updated_on = :updated_on, 
                    deleted_on = :deleted_on
                WHERE id = :id;`
    _, err := r.Db.NamedExecContext(ctx, query, entity)
    return db.HandleError(err)
}

func (r Repository) FindAll(ctx context.Context) ([]model.ToDo, error) {
    var entities []model.ToDo
    query := fmt.Sprintf(
        "SELECT * FROM todo WHERE deleted_on IS NULL",
    )
    err := r.Db.SelectContext(ctx, &entities, query)
    return entities, db.HandleError(err)
}

The repository layer has only one purpose, it will make a SQL query to the Database. So in this case it provides the base CRUD operations for the ToDo model.

Create a separate directory for the service layer, so you can write the business logic here, so we would be creating a directory internal/todo/service and the files internal/todo/service/service.go and internal/todo/service/get.go within it:

Please have the following content on the file internal/todo/service/service.go

Service

We would be implementing the business logic for the CRUD operations. Please create the following folder in directory internal/todo/service and all the following files within the created directory

internal/todo/service/service.go



package service

import (
    "github.com/fir1/rest-api/internal/todo/repository"
)

type Service struct {
    repo repository.Repository
}

func NewService(r repository.Repository) Service {
    return Service{
        repo: r,
    }
}

It returns an instance of service, and this instance would be having all methods of our business logic.

So in this tutorial, the Delivery Layer, can only call the Service layer to get access to the business logic.

Create
The business logic will validate the parameters of the function and save the ToDo in the database.

internal/todo/service/create.go



package service

import (
    "context"
    "github.com/asaskevich/govalidator"
    "github.com/fir1/rest-api/internal/todo/model"
    "github.com/fir1/rest-api/pkg/erru"
    "time"
)

type CreateParams struct {
    Name        string       `valid:"required"`
    Description string       `valid:"required"`
    Status      model.Status `valid:"required"`
}

func (s Service) Create(ctx context.Context, params CreateParams) (int, error) {
    if _, err := govalidator.ValidateStruct(params); err != nil {
        return 0, erru.ErrArgument{Wrapped: err}
    }

    tx, err := s.repo.Db.BeginTxx(ctx, nil)
    if err != nil {
        return 0, err
    }
    // Defer a rollback in case anything fails.
    defer tx.Rollback()

    entity := model.ToDo{
        Name:        params.Name,
        Description: params.Description,
        Status:      params.Status,
        CreatedOn:   time.Now().UTC(),
    }
    err = s.repo.Create(ctx, &entity)
    if err != nil {
        return 0, err
    }

    err = tx.Commit()
    return entity.ID, err
}

It returns an instance of service, and this instance would be having all methods of our business logic. Please note that here we have a separate type CreateParams for taking arguments to function rather than using model.go. Because we would like to have clear isolation between each layer, so this way we would not clutter the tags of types and mandatory arguments to the function.

If we have used type ToDo from model.go as an argument to the function.



type ToDo struct {
    ID          int        `db:"id"`
    Name        string     `db:"name"`
    Description string     `db:"description"`
    Status      Status     `db:"status"`
    CreatedOn   time.Time  `db:"created_on"`
    UpdatedOn   *time.Time `db:"updated_on"`
    DeletedOn   *time.Time `db:"deleted_on"`
}

Which fields from ToDo model would be mandatory for the business logic?

As it is clear that it will mess up our architecture if we use the model.go types as arguments to the business layer. That is one of the reasons why I like to isolate the layers, so we have a clear understanding of mandatory fields for the functions.

Get

The business logic here is quite straightforward, we will have a function which will take id as a mandatory parameter then we would be looking up the database.

internal/todo/service/get.go




package service

import (
    "context"
    "errors"
    "github.com/fir1/rest-api/internal/todo/model"
    "github.com/fir1/rest-api/pkg/db"
    "github.com/fir1/rest-api/pkg/erru"
)

func (s Service) Get(ctx context.Context, id int) (model.ToDo, error) {
    todo, err := s.repo.Find(ctx, id)
    switch {
    case err == nil:
    case errors.As(err, &db.ErrObjectNotFound{}):
        return model.ToDo{}, erru.ErrArgument{errors.New("todo object not found")}
    default:
        return model.ToDo{}, err
    }
    return todo, nil
}

Update

The business logic here is that we will be taking id as a mandatory field, so we can look up the ToDo entity from the Database, then update the fields name, description, status optionally if it is provided.

internal/todo/service/update.go




package service

import (
    "context"
    "errors"
    "github.com/asaskevich/govalidator"
    "github.com/fir1/rest-api/internal/todo/model"
    "github.com/fir1/rest-api/pkg/erru"
)

type UpdateParams struct {
    ID          int `valid:"required"`
    Name        *string
    Description *string
    Status      *model.Status
}

func (s Service) Update(ctx context.Context, params UpdateParams) error {
    if _, err := govalidator.ValidateStruct(params); err != nil {
        return erru.ErrArgument{Wrapped: err}
    }

    // find todo object
    todo, err := s.Get(ctx, params.ID)
    if err != nil {
        return err
    }

    if params.Name != nil {
        todo.Name = *params.Name
    }
    if params.Description != nil {
        todo.Description = *params.Description
    }
    if params.Status != nil {
        if !params.Status.IsValid() {
            return erru.ErrArgument{Wrapped: errors.New("given status not valid")}
        }
        todo.Status = *params.Status
    }

    tx, err := s.repo.Db.BeginTxx(ctx, nil)
    if err != nil {
        return err
    }
    // Defer a rollback in case anything fails.
    defer tx.Rollback()

    err = s.repo.Update(ctx, todo)
    if err != nil {
        return err
    }

    err = tx.Commit()
    return err
}

Delete

Here we would be finding the ToDo entity from the database, if it is found then we soft deleted it by putting the current time for the field DeletedOn.
I choose to soft delete the records compared to hard delete from the database table, because this way we would be having records which we can use for various purposes, such as auditing and etc.

internal/todo/service/delete.go



package service

import (
    "context"
    "time"
)

func (s Service) Delete(ctx context.Context, id int) error {
    todo, err := s.Get(ctx, id)
    if err != nil {
        return err
    }

    tx, err := s.repo.Db.BeginTxx(ctx, nil)
    if err != nil {
        return err
    }
    // Defer a rollback in case anything fails.
    defer tx.Rollback()

    now := time.Now().UTC()
    todo.DeletedOn = &now
    err = s.repo.Update(ctx, todo)
    if err != nil {
        return err
    }

    err = tx.Commit()
    return err
}

Delivery

So please create a folder in the directory http/rest.

As we have described CRUD, so for each functionality of CRUD we would be having a separate handler which will receive external API calls, and then each handler will call it is relevant business service. We would be creating a separate handler and explaining the logic.

Initiate handler service

First we must initiate a handler service, so please create a file in http/rest/handlers/handler.go and put the following content:



package handlers

import (
    toDoRepo "github.com/fir1/rest-api/internal/todo/repository"
    toDoService "github.com/fir1/rest-api/internal/todo/service"
    "github.com/gorilla/mux"
    "github.com/jmoiron/sqlx"
    "github.com/sirupsen/logrus"
)

type service struct {
    logger      *logrus.Logger
    router      *mux.Router
    toDoService toDoService.Service
}

func newHandler(lg *logrus.Logger, db *sqlx.DB) service {
    return service{
        logger:      lg,
        toDoService: toDoService.NewService(toDoRepo.NewRepository(db)),
    }
}

The handler service will take business service as a dependency argument in our case the dependency is toDoService which is our internal business service layer.

Helpers

There is common logic which is being used by every handler, so I have decided to put that logic in a separate helper function, so we can reuse it rather than duplicate it.

Create a file in http/rest/handlers/helper.go and put the following content.



package handlers

import (
    "bytes"
    "encoding/json"
    "errors"
    "github.com/fir1/rest-api/pkg/erru"
    "io"
    "net/http"
)

/*
Don’t have to repeat yourself every time you respond to user, instead you can use some helper functions.
*/
func (s service) respond(w http.ResponseWriter, data interface{}, status int) {
    var respData interface{}
    switch v := data.(type) {
    case nil:
    case erru.ErrArgument:
        status = http.StatusBadRequest
        respData = ErrorResponse{ErrorMessage: v.Unwrap().Error()}
    case error:
        if http.StatusText(status) == "" {
            status = http.StatusInternalServerError
        } else {
            respData = ErrorResponse{ErrorMessage: v.Error()}
        }
    default:
        respData = data
    }

    w.WriteHeader(status)
    w.Header().Set("Content-Type", "application/json")
    if data != nil {
        err := json.NewEncoder(w).Encode(respData)
        if err != nil {
            http.Error(w, "Could not encode in json", http.StatusBadRequest)
            return
        }
    }
}

// it does not read to the memory, instead it will read it to the given 'v' interface.
func (s service) decode(r *http.Request, v interface{}) error {
    return json.NewDecoder(r.Body).Decode(v)
}

// it reads to the memory.
func (s service) readRequestBody(r *http.Request) ([]byte, error) {
    // Read the content
    var bodyBytes []byte
    var err error
    if r.Body != nil {
        bodyBytes, err = io.ReadAll(r.Body)
        if err != nil {
            err := errors.New("could not read request body")
            return nil, err
        }
    }
    return bodyBytes, nil
}

// will place the body bytes back to the request body which could be read in subsequent calls on Handlers
// for example, you have more than 1 middleware and each of them need to read the body. If the first middleware read the body
// the second one won't be able to read it, unless you put the request body back.
func (s service) restoreRequestBody(r *http.Request, bodyBytes []byte) {
    // Restore the io.ReadCloser to its original state
    r.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
}

Create
We will be creating handlers and the practical approach would be to have a struct for the request, response within the relevant handler, this way we have clear isolation between the data which is being required from the API call, and the data which is being inserted into the business layer, rather than using model everywhere within our application.

Some of you might not agree with me but I prefer clean code over a complicated one.
The main important thing for developers is to write the code clean, so the fellow developers can understand easily.

http/rest/handlers/create.go



package handlers

import (
    "github.com/fir1/rest-api/internal/todo/model"
    toDoService "github.com/fir1/rest-api/internal/todo/service"
    "net/http"
)

func (s service) Create() http.HandlerFunc {
    type request struct {
        Name        string       `json:"name"`
        Description string       `json:"description"`
        Status      model.Status `json:"status"`
    }

    type response struct {
        ID int `json:"id"`
    }

    return func(w http.ResponseWriter, r *http.Request) {
        req := request{}
        // Try to decode the request body into the struct. If there is an error,
        // respond to the client with the error message and a 400 status code.
        err := s.decode(r, &req)
        if err != nil {
            s.respond(w, err, 0)
            return
        }

        id, err := s.toDoService.Create(r.Context(), toDoService.CreateParams{
            Name:        req.Name,
            Description: req.Description,
            Status:      req.Status,
        })
        if err != nil {
            s.respond(w, err, 0)
            return
        }
        s.respond(w, response{ID: id}, http.StatusOK)
    }
}

As you can see the job of our handler is to receive an API call, and then call the relevant business service layer, to handle the rest of the logic (s.toDoService.Create - this is our business service layer).

It is a clean approach, as our handler knows only about the business service layer, and the handler does not know anything about the underlying logic of the business.

This approach is practical because imagine if you would like to expose your API with gRPC protocol, with the current approach we just add an additional delivery layer in the directory http/grpc then call our business service layer, without the need to write any business logic.

Get

Please create a file in http/rest/handlers/get.go and have the following content:



package handlers

import (
    "errors"
    "github.com/fir1/rest-api/internal/todo/model"
    "github.com/fir1/rest-api/pkg/erru"
    "github.com/gorilla/mux"
    "net/http"
    "strconv"
    "time"
)

func (s service) Get() http.HandlerFunc {
    type response struct {
        ID          int          `json:"id"`
        Name        string       `json:"name"`
        Description string       `json:"description"`
        Status      model.Status `json:"status"`
        CreatedOn   time.Time    `json:"created_on"`
        UpdatedOn   *time.Time   `json:"updated_on,omitempty"`
    }
    return func(w http.ResponseWriter, r *http.Request) {
        vars := mux.Vars(r)

        id, err := strconv.Atoi(vars["id"])
        if err != nil {
            s.respond(w, erru.ErrArgument{
                Wrapped: errors.New("valid id must provide in path"),
            }, 0)
            return
        }

        getResponse, err := s.toDoService.Get(r.Context(), id)
        if err != nil {
            s.respond(w, err, 0)
            return
        }
        s.respond(w, response{
            ID:          getResponse.ID,
            Name:        getResponse.Name,
            Description: getResponse.Description,
            Status:      getResponse.Status,
            CreatedOn:   getResponse.CreatedOn,
            UpdatedOn:   getResponse.UpdatedOn,
        }, http.StatusOK)
    }
}

Update

Please create a file in http/rest/handlers/update.go and have the following content:



package handlers

import (
    "errors"
    "github.com/fir1/rest-api/internal/todo/model"
    toDoService "github.com/fir1/rest-api/internal/todo/service"
    "github.com/fir1/rest-api/pkg/erru"
    "github.com/gorilla/mux"
    "net/http"
    "strconv"
)

func (s service) Update() http.HandlerFunc {
    type request struct {
        Name        *string       `json:"name"`
        Description *string       `json:"description"`
        Status      *model.Status `json:"status"`
    }

    type response struct {
        ID int `json:"id"`
    }

    return func(w http.ResponseWriter, r *http.Request) {
        vars := mux.Vars(r)
        id, err := strconv.Atoi(vars["id"])
        if err != nil {
            s.respond(w, erru.ErrArgument{
                Wrapped: errors.New("valid id must provide in path"),
            }, 0)
            return
        }

        req := request{}
        // Try to decode the request body into the struct. If there is an error,
        // respond to the client with the error message and a 400 status code.
        err = s.decode(r, &req)
        if err != nil {
            s.respond(w, err, 0)
            return
        }

        err = s.toDoService.Update(r.Context(), toDoService.UpdateParams{
            ID:          id,
            Name:        req.Name,
            Description: req.Description,
            Status:      req.Status,
        })
        if err != nil {
            s.respond(w, err, 0)
            return
        }
        s.respond(w, response{ID: id}, http.StatusOK)
    }
}

Delete
Please create a file in http/rest/handlers/delete.go and have the following content:



package handlers

import (
    "github.com/gorilla/mux"
    "net/http"
    "strconv"
)

func (s service) Delete() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        vars := mux.Vars(r)

        id, err := strconv.Atoi(vars["id"])
        if err != nil {
            s.respond(w, err, 0)
            return
        }

        err = s.toDoService.Delete(r.Context(), id)
        if err != nil {
            s.respond(w, err, 0)
            return
        }
        s.respond(w, nil, http.StatusOK)
    }
}

Middleware
As you might have noticed by this time that one of the most important parts of every application is being able to log, request and response to API calls.
Disclaimer, when you log requests and responses you must be careful not to log sensitive information such as credentials, usernames, passwords and any other data which violates privacy.

Please create a file in http/rest/middleware_logger.go and put the following content inside it:



package handlers

import (
    "fmt"
    "net/http"
    "time"
)

// responseWriter is a minimal wrapper for http.ResponseWriter that allows the
// written HTTP status code to be captured for logging. This type will implement http.ResponseWriter.
type responseWriter struct {
    http.ResponseWriter
    status      int
    body        []byte
    wroteHeader bool
    wroteBody   bool
}

func wrapResponseWriter(w http.ResponseWriter) *responseWriter {
    return &responseWriter{ResponseWriter: w}
}

func (rw *responseWriter) Status() int {
    return rw.status
}

func (rw *responseWriter) WriteHeader(code int) {
    if rw.wroteBody {
        return
    }
    rw.status = code
    rw.ResponseWriter.WriteHeader(code)
    rw.wroteHeader = true
}

func (rw *responseWriter) Write(body []byte) (int, error) {
    if rw.wroteBody {
        return 0, nil
    }
    i, err := rw.ResponseWriter.Write(body)
    if err != nil {
        return 0, err
    }
    rw.body = body
    return i, err
}

func (rw *responseWriter) Body() []byte {
    return rw.body
}

// middlewarMiddlewareLoggereLogger logs the incoming HTTP request and response. Enable it only for debug purpose disable it on production.
func (s service) MiddlewareLogger() func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        fn := func(w http.ResponseWriter, r *http.Request) {
            if r.URL.Path == "/healthz" {
                // Call the next handler don't log if it is internal request from health check of Kubernetes
                next.ServeHTTP(w, r)
                return
            }

            defer func() {
                if err := recover(); err != nil {
                    w.WriteHeader(http.StatusInternalServerError)
                }
            }()

            requestBody, err := s.readRequestBody(r)
            if err != nil {
                s.respond(w, err, 0)
                return
            }
            s.restoreRequestBody(r, requestBody)

            logMessage := fmt.Sprintf("path:%s, method: %s, requestBody: %v", r.URL.EscapedPath(), r.Method, string(requestBody))

            start := time.Now()
            wrapped := wrapResponseWriter(w)
            next.ServeHTTP(wrapped, r)

            logMessage = fmt.Sprintf("%s, responseStatus: %d, responseBody: %s", logMessage, wrapped.Status(), string(wrapped.Body()))
            s.logger.Infof("%s, duration: %v", logMessage, time.Since(start))
        }
        return http.HandlerFunc(fn)
    }
}

Register Handlers

As of now, we have all the handlers with the relevant handlers, but we need to register those handlers to the router with the customized Path.
We would be creating a file in http/rest/handlers/routes.go and have the following content:



package handlers

import (
    "github.com/gorilla/mux"
    "github.com/jmoiron/sqlx"
    "github.com/sirupsen/logrus"
    "net/http"
)

func Register(r *mux.Router, lg *logrus.Logger, db *sqlx.DB) {
    handler := newHandler(lg, db)
    // adding logger middleware
    r.Use(handler.MiddlewareLogger())
    r.HandleFunc("/healthz", handler.Health())
    r.HandleFunc("/todo", handler.Create()).Methods(http.MethodPost)
    r.HandleFunc("/todo/{id}", handler.Get()).Methods(http.MethodGet)
    r.HandleFunc("/todo/{id}", handler.Update()).Methods(http.MethodPut)
    r.HandleFunc("/todo/{id}", handler.Delete()).Methods(http.MethodDelete)
}

Creating Server

In this stage, we need to create an instance of an actual server which can receive API calls and handle those calls.

Please create a file in http/rest/logger.go and have the following content inside it:



package rest

import (
    "github.com/sirupsen/logrus"
    "os"
)

func NewLogger() *logrus.Logger {
    log := logrus.New()
    log.SetOutput(os.Stdout)
    log.SetLevel(logrus.InfoLevel)
    log.SetFormatter(&logrus.TextFormatter{
        ForceColors:     true,
        TimestampFormat: "2006-01-02 15:04:05.999999999",
        FullTimestamp:   true,
    })
    return log
}

Create a file in http/rest/server.go and have the following content inside it:



package rest

import (
    "context"
    "fmt"
    "github.com/fir1/rest-api/configs"
    "github.com/fir1/rest-api/http/rest/handlers"
    "github.com/fir1/rest-api/pkg/db"
    "github.com/gorilla/mux"
    "github.com/rs/cors"
    "github.com/sirupsen/logrus"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
)

type Server struct {
    logger *logrus.Logger
    router *mux.Router
    config configs.Config
}

func NewServer() (*Server, error) {
    cnf, err := configs.NewParsedConfig()
    if err != nil {
        return nil, err
    }

    database, err := db.Connect(db.ConfingDB{
        Host:     cnf.Database.Host,
        Port:     cnf.Database.Port,
        User:     cnf.Database.User,
        Password: cnf.Database.Password,
        Name:     cnf.Database.Name,
    })
    if err != nil {
        return nil, err
    }

    log := NewLogger()
    router := mux.NewRouter()
    handlers.Register(router, log, database)

    s := Server{
        logger: log,
        config: cnf,
        router: router,
    }
    return &s, nil
}

func (s *Server) Run(ctx context.Context) error {
    server := http.Server{
        Addr:    fmt.Sprintf(":%d", s.config.ServerPort),
        Handler: cors.Default().Handler(s.router),
    }

    stopServer := make(chan os.Signal, 1)
    signal.Notify(stopServer, syscall.SIGINT, syscall.SIGTERM)

    defer signal.Stop(stopServer)

    // channel to listen for errors coming from the listener.
    serverErrors := make(chan error, 1)
    var wg sync.WaitGroup
    wg.Add(1)
    go func(wg *sync.WaitGroup) {
        defer wg.Done()
        s.logger.Printf("REST API listening on port %d", s.config.ServerPort)
        serverErrors <- server.ListenAndServe()
    }(&wg)

    // blocking run and waiting for shutdown.
    select {
    case err := <-serverErrors:
        return fmt.Errorf("error: starting REST API server: %w", err)
    case <-stopServer:
        s.logger.Warn("server received STOP signal")
        // asking listener to shutdown
        err := server.Shutdown(ctx)
        if err != nil {
            return fmt.Errorf("graceful shutdown did not complete: %w", err)
        }
        wg.Wait()
        s.logger.Info("server was shut down gracefully")
    }
    return nil
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    s.router.ServeHTTP(w, r)
}

Run Server

At this point now we can run a Restful server, and test all our integrations.

Please create a file in cmd/app/main.go and have the following content inside it:



package main

import (
    "context"
    "github.com/fir1/rest-api/http/rest"
    "log"
)

func main() {
    if err := run(context.Background()); err != nil {
        log.Fatalf("%+v", err)
    }
}

func run(ctx context.Context) error {
    server, err := rest.NewServer()
    if err != nil {
        return err
    }
    err = server.Run(ctx)
    return err
}

Our main package has a few lines of code, which is an entry point to our application. I think the main package should be clean and must be used to start servers without any extra logic.

Tests

In this section, we are going to test our API server manually by making a Restful API call with the help of the Insomnia Tool.

Make sure that you are in the root directory and run the following command from the terminal go run cmd/app/*.go

Create

Make an API call to the



POST localhost:80/todo

Request Body

{
  "name": "Tutorial Restful pending",
  "description": "Still we are testing Restful api so not finished",
  "status": 1
}

Get

Make an API call to the



GET localhost:80/todo/4 (id of todo - 4 might be different in your case)

Update

Make an API call to the



PUT localhost:80/todo/4 (id of todo - 4 might be different in your case)

Request Body
{
  "name": "Tutorial Restful finished",
  "description": "Restful api testing finished",
  "status": 2
}

Get Updated ToDo

Make an API call to the



GET localhost:80/todo/4 (id of todo - 4 might be different in your case)

From the above screenshot, it is clear that the entity ToDo was updated.

Delete

Make an API call to the



DELETE localhost:80/todo/4 (id of todo - 4 might be different in your case)

The entity was deleted, so when we do GET request we will be getting an error, as the object does not exist anymore.

Enhancements

You can even enhance this project by adding more functionalities, such as:

Add todo list endpoint, that will return all existing ToDo's entity.
Extra functionalities, such as filtering the ToDo by status and keywords.
Add authentication

Once you have done these enhancements you can create a PR towards my repository github.com/fir1/rest-api

The end

Hopefully, this tutorial was useful and I would be more than happy to receive questions or suggestions about it.

The source code for this tutorial can be found here

That’s it for this blog. Please feel free to comment with your views on this blog. Thanks for your time reading this blog and hope it was useful.

Happy learning and sharing.

A practical approach to structuring Golang applications

Firdavs Kasymov — Mon, 03 Oct 2022 09:20:09 +0000

This article will describe the common approaches which are used for structuring Golang applications. Golang does not have an official guideline to structure the project, so it is an opinionated guideline to structure Go applications.

Go is an amazing language, it is simply easy to learn and gives you a lot of tooling out of the box. Most of the Gophers who started learning the Golang they have one thing in common, they struggle to figure out how to structure applications due to the lack of official guidelines. The main approach for structuring Golang applications is to start out simple, but with the flexibility to grow.

There are no strict rules for the directory structure or how to organize Go project files in a specific way. It is bad because it is easy to create a mess and good because the structure of the project can be built according to the taste of the developer.

The good structure is the one which is consistent across the project. There is nothing worse than coming to the codebase and seeing one part of code done in one way and another part done another way. Especially if you are not an original author and you just coming to the new codebase and you have no idea which structure you should follow. The good code structure must be easy to understand and navigate and it should make sense. Good code structure will make your life easy as a programmer and fun.

Go developers typically follow certain patterns in laying out the files and directories in their projects. They usually follow these patterns because it works for them and their fellow developers. In my opinion, the organisation should follow the same structure for all the projects they have for the Golang application, as this way it makes the developers more productive.

Structuring Project Files in Go

The better way to organize a Go project is to put relevant Go code files into a subdirectory under the main directory for the project so that other parts of the project are able to find the service APIs and use it.

Keeping all source files under the same directory is not a good idea, because it makes the code files cluttered, as a result, you spend most of the time looking for the correct file. Most experienced developers know about it.

For the purpose of demonstrating a project layout, we would be creating a demo project called "go-layout".

This is the tree structure of the sample project type. Below we would be explaining what each folder does:

/cmd

The folder will contain the main applications for this project.
There might be more than 1 application within cmd folder, and each application should have its own folder so the path would be cmd -> applicationNameFolder.

The directory name for each application should match the name of the executable you want to have, for example, cmd/myapp.

It is important that we don't put a lot of code in the application directory. If you believe that the code can be imported and used in other projects, it is better to place these codes in the pkg directory.

It is common practice to have a small main function that imports and invokes the code from the /internal and pkg directories.

Please check out some of the successful projects which are written in Golang.

Examples:

/api

This folder can have OpenAPI/Swagger specs, JSON schema files, and protocol definition files.
Example:

https://github.com/moby/moby/tree/master/api

/build

This folder can have folders/files which are relevant to the packaging and Continuous Integration files, such as Docker and scripts.

Put your CI configurations and scripts in the /build/ci directory.

Examples:

https://github.com/cockroachdb/cockroach/tree/master/build

/configs

It can have configuration file templates or default configs.

/examples

You can include examples of your applications and/or public libraries.

Examples:

/internal

Private application and library code. This is the code you don't want others importing into their applications or libraries. Note that this layout pattern is enforced by the Go compiler itself. Note that you are not limited to the top-level internal directory. You can have more than one internal directory at any level of your project tree.

You can optionally add a bit of extra structure to your internal packages to separate your shared and non-shared internal code. It's not required (especially for smaller projects), but it's nice to have visual clues showing the intended package use.

Your actual application code can go in the /internal/app directory (e.g., /internal/app/myapp) and the code shared by those apps in the /internal/pkg directory (e.g., /internal/pkg/myprivlib).

The folder /internal/pkg contains packages that are not specific to this application. These packages could theoretically be extracted into a standalone public package if they provide utility.

Examples:

https://github.com/hashicorp/terraform/tree/master/internal https://github.com/influxdata/influxdb/tree/master/internal

/pkg

Library codes which can be used by external applications (e.g. /pkg/mypubliclib). Other projects will import these libraries expecting them to work, so double-check before you put something here.

Note that the internal directory is a better way to ensure your private packages are not importable because it is enforced by Go.

it is ok not to use /pkg directory if your app project is really small and where an extra level of nesting does not add much value. But you need to think about it when your project is getting big, and your root directory gets busy.

Examples:

/third_party

External helper tools, forked code and other 3rd party utilities (e.g., Swagger UI).

/web

Web application-specific components: static web assets, server-side templates and SPAs.

/http

You might want to expose your application through several means of communication such as rest-api and grpc.

This way you can have a separate separation between each type of communication layer by creating a separate directory such as /http/rest or /http/grpc

You can download the full project layout template from my GitHub (https://github.com/fir1/go-layout).

The end

Hopefully, this tutorial gave you some insights on how to structure your next Golang applications.

That’s it for this blog. Please feel free to comment your views on this blog. Thanks for your time reading this blog and hope it was useful.

Happy learning and sharing.

Challenges of microservice in Kubernetes

Firdavs Kasymov — Sat, 01 Oct 2022 17:09:29 +0000

This article will describe in detail the common challenges of microservices in the Kubernetes world, and how we can solve those challenges.

Let's start listing some of the challenges:

Every single microservice needs to know the service address. Usually, it is injected within the microservice business logic as Config dependency.
Once the request is received in the Kubernetes Cluster, the communication is insecure as microservices talk to each other over HTTP. Every single microservice inside the cluster can talk to any other service freely. So, when it comes to security, there is no way to prevent attacks on the cluster.
Each microservice has a retry logic to make the whole application more robust. If one microservice is unreachable or you lose a connection, you naturally want to retry connecting. So, developers usually add retry logic to every service, and the same process is continued when the new microservices are added.
We want to be able to monitor how the service performs, what responses return, how many requests the microservice receives, the network latency, and tracing.
The development team may add monitoring logic to Prometheus using the client library, and collect tracing data using the tracing library, for example, Zipkin. As you might have noticed, the team of developers of each microservice needs to add all this logic to every service, which is quite time-consuming.
The developers are not integrating any business needs. Instead, they add logic for communication, security, and retry logic for each microservice. These will add complexity to the service instead of keeping it simple and straightforward.
Managing Canary deployment and backward compatibility of software, including websites and backend APIs in microservices, gets challenging without proper Service Mesh patterns.
In software engineering, canary deployment is the practice of making staged releases. We roll out software updates for a small part of the live users first, so they test it and provide feedback. Once the changes are accepted, the update will be rolled out for the rest of the users.

When it comes to Canary deployment, we split users into two groups. A small percentage of these groups will go to Canary, while the rest will stay in the stable version. Later we will decide whether to roll everyone to the Canary or roll back changes to the previous version. Alternatively, we can check the headers of requests to support Canary deployment for a specific set of users.

Some of the challenges I have faced are listed above. As a result, it would make sense to extract all non-business logic out of the microservice into the separate sidecar application that handles all of the above-mentioned issues. Also, the sidecar would act as a proxy. We can solve all these challenges with the “service mesh with Sidecar” pattern.

What is Service Mesh?

The main goals of a service mesh are to allow insight into previously invisible service communications layers and to gain full control of all microservices communication logic, like dynamic service discovery, load balancing, timeouts, fallbacks, retries, circuit breaking, distributed tracing, and security policy enforcement between services. The insights are provided by traffic audit and tracing features.

Kubernetes already has a very basic “service mesh” out-of-the-box; it’s the “service” resource. It provides service discovery by targeting the needed pods and a round-robin balancing of requests. A “service” works by managing iptables on each host in the cluster, allowing only a round-robin load balancing approach, with no retries and back off logic and no other features that we might expect a modern service mesh to handle. However, implementing one of the fully featured service mesh systems in your cluster, for example, Istio, Linkerd, or Conduit, will provide you with the following possibilities:

Allow services to talk plain HTTP and not bother about HTTPS on the application layer. The service mesh proxies will manage HTTPS encapsulation on the sender side and TLS termination on the receiver side, allowing the application components to use plain HTTP or gRPC and any other protocol without encryption in transit. Proxies will take care of the encryption.
Security policies enforcement: the proxy knows which services are allowed to access some other services and endpoints and will deny unauthorized traffic.
Circuit breaking: automatic back off in case of accessing an overloaded service or endpoint that would already have high latency. This is available to avoid hitting it with more and more requests which may cause that endpoint to fail under an excessive load.
Latency-aware load balancing: instead of using a round-robin style of balancing (which ignores the latency of each target), use smarter balancing according to the response times of each backend target. This is an incredibly crucial feature of a modern service mesh.
Queue depth load balancing: route new requests based on the least busy target by the current request processing amount. The service mesh knows where it has sent all previous requests and which ones are still being processed or were already completed. It will send new incoming requests based on that logic to a target with the lowest queue for processing.
Per request routing: route particular requests marked by selected HTTP header to specific targets behind the load balancer, allowing easy canary deployment testing and other creative use cases. One of the most powerful features a service mesh provides.
Metrics and tracing: reporting requests volume per target, latency metrics, success, and error rates.

We would be doing technical practice with the service mesh known as Istio. We will execute Canary Deployment with the Istio to cover the above-mentioned issues.

Istio

Service mesh is just a pattern, while Istio is one of its implementations. Istio architecture consists of a “Control Plane” component, also known as “Istiod”, which manages and injects proxy on each separate pod. Istio uses Envoy Proxy as the Proxy layer. The Envoy Proxy is an independent open source project which many other mesh providers use as a proxy sidecar.
You can learn more about Istio from its official documentation (https://istio.io/)

How to configure Istio in Kubernetes?

There is no need to adjust deployment and services in Kubernetes YAML files, as the Istio configuration is separate from the application configuration. Istio is configured with Kubernetes YAML files. It uses Kubernetes Customer Resource Definitions (CRD) by extending the Kubernetes API.

CRD is a custom Kubernetes component/object. For example, third-party technologies, like Istio, Prometheus, and many more, can be used like any other native Kubernetes objects.

Using a few Istio CRDs, we can configure different traffic routing between microservices, including which services can communicate with each other, traffic splitting, retry logic, and canary deployments.

Service Discovery
Istiod has a central registry for all the microservices. Instead of statically configuring endpoints for each microservice, when a new microservice gets deployed, it will be automatically registered in the service registry with no need for configuration from our side.

Envoy Proxy can query endpoints and send the traffic to the relevant services using this registry.

Certificate Management
Istiod also acts as a Certificate Authority. It generates certificates for all the cluster microservices, allowing secure TLS communication between microservices proxies.

Metrics and Tracing
Istiod receives metrics and tracing data from Envoy Proxies. It is gathered by monitoring servers, such as Prometheus, and tracing servers to have everything needed for microservice applications.

Istio Ingress Gateway
It is an entry point into the Kubernetes cluster, almost an alternative to Nginx Ingress Controller.

Istio gateway runs as Pod into the cluster and acts as load balancing by accepting incoming traffic. The gateway then will direct traffic to one of the microservices inside the cluster using VirtualService components.

VirtualService
The “VirtualService” links the gateway and destination pods of any request. Any “host” (DNS name or Kubernetes DNS name when services address each other inside the cluster) can be defined only in one VirtualService.

Istio traffic flow
A user will initiate a request to the web server microservice into the Kubernetes Cluster → then request hits to Istio Gateway because it is the entry point of the cluster → Gateway will evaluate the Virtual Service rules about how to route the traffic and send it to the WebServer microservice → Request reaches to the Envoy Proxy of web server microservice, it will evaluate the request and forward it to the actual web server container within the same container by using localhost.

For instance, webserver initiates another request to the Payment microservice so that the request will move out from Webserver container to the Envoy Proxy. Then, by applying the VIrtualService rules, the Destination rules configuration will communicate with the Envoy Proxy of Payment microservice using mTLS (mutual TLS).

Canary Deployment with Istio

Before we start the technical practice, make sure that the necessary tools which are listed below are installed on your computer:

Docker (https://docs.docker.com/engine/install/)
Kubectl (https://kubernetes.io/docs/tasks/tools/)
The Kubernetes command-line tool allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.
Minikube (https://minikube.sigs.k8s.io/docs/start/)
It is the most widely used local Kubernetes installer. It offers an easy-to-use guide on installing and running single Kubernetes environments across multiple operating systems. It deploys Kubernetes as a container, VM, or bare-metal and implements a Docker API endpoint that helps it push container images faster. It has advanced features like load balancing, filesystem mounts, and FeatureGates, making it the best for running Kubernetes locally. Run Minikube locally by executing the command - minikube start to create Kubernetes resources locally.

Istio Addons: additional elements that we can install with Istio. It will include all data visualisation about metrics, tracing, basically what your microservices are doing and how they perform. Below you can find a brief description of what each tool does, but we don't dive into details as it is out of the scope of this article.

We will install some of those add-ons into our local cluster. You can follow the official guide for installing these add-ons (https://istio.io/latest/docs/ops/integrations/). Please install the following add-ons: Kiali, Grafana, Jaeger, and Promethteus.

Prometheus is used for monitoring anything in the cluster. It can be the server itself, memory, cpu usage, as well as Kubernetes components, such as pods, services, and other components.

Grafana is a data visualization tool for metrics data.

Jaeger is a service for tracing microservice requests.

Kiali offers amazing data visualization features and configuring your microservice setup and communication.

We want to work in an isolated namespace for this example, so, please create a Kubernetes namespace, and save the following to namespace.ymal file.

apiVersion: v1
kind: Service
metadata:
  name: website
  namespace: website-istio-ns
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: website

Then, run kubectl create -f service.yaml to submit these resource definitions to the cluster.

Create, Istio Gateway and virtual service for the basic functionality of the service mesh ingress endpoint, so that we can access our application through the Istio Ingress load balancer that was created when you deployed Istio to the cluster.
Save the following definition to the gateway.yaml file:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: website-gateway
  namespace: website-istio-ns
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Then, run kubectl create -f gateway.yaml to submit these resource definitions to the cluster.

Save the following definition to the virtual-service.yaml file:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: website-virtual-service
  namespace: website-istio-ns
spec:
  hosts:
  - "*"
  gateways:
  - website-gateway
  http:
  - match:
      - headers:
          qa:
            exact: canary-test
    route:
      - destination:
          host: website
          subset: version-2

  - route:
    - destination:
        host: website
        subset: version-1

The host field in the “VirtualService” destination is a name of the Kubernetes “service” object for use. Destination can be divided into subsets, if we want to distinguish between our pods by label and address them in different scenarios separately (with URI path based or HTTP header based routing). In this case, we need to add a subset field like this:

http:
  - route:
    - destination:
        host: website
        subset: something-that-is-defined-in-DestinationRule

“Subsets” for routing are defined with “DestinationRule.” In addition to the label-based separation of target pods of service, we can apply a custom load-balancing policy, if, for example, we have a subset “version-1” and “version-2” defined below and saved in “destination-rule.yaml” file:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: website
  namespace: website-istio-ns
spec:
  host: website
  subsets:
  - name: version-1
    labels:
      version: website-version-1
  - name: version-2
    labels:
      version: website-version-2

Now deploy with kubectl create -f destination-rule.yaml. By doing that, we created a “subset” of the “website” destination, named “version-1” and “version-2”. That subset is any pod with “version: website-version-1” or “version: website-version-1” label and any other labels that are defined in the “website” service definition (which is “app: website”).

This will route to any pod with the “version” label that equals “website-version-1” or “website-version-2” when a “VirtualService” uses the destination service name. We defined it in the “spec.host” field (it is the Kubernetes “service” object name, the one we used in “spec.http.route.destination.host” field of “VirtualService”).

The next step is to prepare the service mesh for deploying the new website version. Let’s assume that this is a production cluster with real traffic flowing in, and “website-version-1” is active, and we have four pods that receive traffic. By creating a separate deployment, with version-2 of the website pod and the same label “app: website”, we will cause traffic to split and will impact existing users, which we want to avoid!

Let’s create a deployment pod with 2 different version of the website:

Save the below and name it as version-1-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-v1
  namespace: website-istio-ns
spec:
  replicas: 4
  selector:
    matchLabels:
      app: website
      version: website-version-1
  template:
    metadata:
      labels:
        app: website
        version: website-version-1
    spec:
      containers:
      - name: website-version-1
        image: hashicorp/http-echo:alpine
        args:
          - -listen=:80
          - --text="echo-v1"
        ports:
          - name: http
            protocol: TCP
            containerPort: 80
        imagePullPolicy: IfNotPresent

Deploy by executing command kubectl create -f version-1-deployment.yaml

For the "version-2" of website we need to have a separate deployment. Please create a file named as version-2-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-v2
  namespace: website-istio-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: website
      version: website-version-2
  template:
    metadata:
      labels:
        app: website
        version: website-version-2
    spec:
      containers:
      - name: website-version-2
        image: hashicorp/http-echo:alpine
        args:
          - -listen=:80
          - --text="echo-v2"
        ports:
          - name: http
            protocol: TCP
            containerPort: 80
        imagePullPolicy: IfNotPresent

Deploy by executing command kubectl create -f version-2-deployment.yaml

At this point we have four “version-1” pods and a “version-2” canary pod.

In order to access “istio ingress gateway” locally, we must tunnel it locally with the help of Minikube by executing the following command: minikube tunnel.

Finally, we can test the results by visiting the following browser address: “localhost:80”. As you can see on the screenshot, the “version-1” of our website responded as “echo-v1”. By default all visitors will land on the “version-1”.

The QA team, internal users, and a small percentage of real users can see the updated version of the website, known as the Canary version. Any HTTP request must include the “qa” header containing the “canary-test” value for viewing the Canary version.

Istio’s virtual service will check headers and values that match them, rerouting traffic to the “version-2” of the website.

To include the header we can use Postman, Insomnia API tools. The easiest way to exercise this would be by installing the ModHeader extension for Google Chrome (https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj?hl=en).

The below screenshot will demonstrate how the ModHeader extension would work in the browser.

Once you include the necessary headers, you can visit “localhost:80”. As you can see, after including the headers the Istio virtual service rerouted our request to the “version-2” of our website.

The proof of the headers being sent out to the server:

For the purpose of this article, I used a simplified version of the website. It will just echo text based on the different versions of the website. You can find the full working deployment codes from following GitHub link (https://github.com/fir1/istio-ingress).

This setup can be used as a guide for Canary Deployment in different projects of yours. Good luck!

Web3 and the future of finance.

Firdavs Kasymov — Sun, 15 May 2022 23:25:38 +0000

It makes sense to understand Web 3 by first understanding what goes before it. Web 1, the first online version, was released in the late 1990s and included a collection of links and home pages. The websites were not very interactive. Apart from reading and publishing basic content for others to read, there is not much you can do.

In a speech to the Congress of the United States in December 2021, Brian Brooks, CEO of Bitfury, summed it up: "When people remember their first AOL account, it was the ability to view 'walled garden' content that was not uncommon, but presented to you in -AOL, the way Time Magazine used it to show you the articles they wanted to see inside their magazine, only you can see it on screen."

Web 2 followed. Some refer to this as the "read/write" internet, referring to a computer code that allows you to open and edit files rather than just view them. This version of the Internet allowed people to create their own content and publish it on blogs like Tumblr, Internet forums, and marketplaces like Craigslist. Later, with the rise of social media platforms such as Facebook, Twitter, and Instagram, content sharing reached new heights.

After a while, the general public became aware of how tech giants were harvesting their personal data and using it to create tailored advertisements and marketing campaigns. Facebook, in particular, has been scrutinized numerous times for violating data privacy laws and was fined $5 billion in 2019 – the largest penalty ever imposed by the Federal Trade Commission (FTC.)

Although Web 2 offers amazing free services around the world, many people are tired of the new "walled gardens" created by these big technology companies and want more control over their data and content. This is where Web 3 comes into play.

Web 3 can be thought of as an online "read / write / write / write" section. Users can participate in the governance and implementation of agreements rather than simply accessing our data through free technology forums. This means that people can become shareholders rather than just customers.

Blockchains are tokens or cryptocurrencies used on Web 3 to represent the ownership of isolated networks. If you own enough of these tokens, you can exert influence on the network. Owners of administrative tokens, for example, can use their funds to vote on the future of a decentralized loan protocol.

The real takeaway here is that what happens on a diverse internet is determined by investors, whereas what happens on the big internet is determined by Twitter, Facebook, Google, and a few other companies.

Source: Fabric Ventures

Now that it is clear to you so let’s see the advantages and Disadvantages of Web3.

As data infrastructure improves people are increasingly demanding that data be accessible and retain control of their information. The default Blockchain data system allows data to be stored in such a way that it remains under the control of the owner, even if it is stored on a business server or under local government control.

Owners or governing bodies will not be able to access or edit data unless they have encryption keys that prove they are the rightful owners. Additionally, even if they shut down or delete their server, the data will still be accessible to one of the hundreds of other devices where it is stored.

With the launch of Web3.0, progress in the online world continues. Web3.0 is the third generation of the Internet, and it is growing rapidly.

Here are some of the advantages of Web3.0:

Improvements to the appearance of the semantic web
More efficient web browsing
Persuasive correspondence
Communication transformation
Search results that are more credible and valid
Working on the internet will become much easier as it becomes personalized.
Knowledge exchange is more accessible.

Here are some of the potential barriers to Web3.0:

Web3.0 will not be accessible to less advanced devices.
Web1.0 websites will appear more frequently than ever before.
Web3.0 is very difficult for newcomers to understand
The technology is not really ready yet
It is difficult to use
People will spend a lot of time online
Privacy rules will be needed to protect them.

How Web3 can be used in the Fintech Industry and how it has already Impacted the Industry?

The evolution of FinTech globally changed dramatically over time, becoming a roaring opportunity on Wall Street and Silicon Valley. FinTech's growing power from 1.0 to 3.0 has forced traditional banking systems and financial services providers to embark on a transition that will usher in a new era of the digital world. The remarkable transformation of the World Wide Web started with Web 3.0, which is backed by technologies such as machine-learning architecture and reshapes the financial services business with a more general, segmented work model.

The maturing Web 3.0 ecosystem, led by decentralized autonomous organizations, has fueled global demand for structured financial products enabled by superfast 5G data speeds, data formats, and software.

So, in this modern global economy, we'll take a deep dive into understanding the mutual relationship between Web 3.0 and the FinTech industry.

The first two waves further the momentum of FinTech 3.0, increasing the use of artificial intelligence (AI) and machine learning (ML) to flourish intelligent and independent processes. Many world powers are involved in developing and advancing a new wave of innovation, with the emergence of the Web playing an important role.

Web 1.0, the first phase of World Wide Web evolution, empowers users to share consistent information around the world by making democracy accessible to information. After the dot-com bubble, Web 2.0 transformed the digital age by emphasizing social media, user-generated content, and cloud computing, creating social media platforms such as LinkedIn, Facebook, Twitter, Instagram, and many more.

Finally, Web 3.0, also known as the decentralized web, is a vision of the internet's future in which computers and technologies deliver useful and relevant insights through intelligent data and transaction interpretation. FinTech 3.0's new cashless and virtual economy necessitates Web 3.0 transformation to lay the groundwork for how businesses deliver products and services to next-generation unicorns.

Now let’s see how web3 will benefit the Fintech Industry with some of the departments.

By disrupting the way individuals, businesses, and regulatory organizations collaborate and plan to collaborate, the third generation of internet services will enable a more connected and intelligent world for the financial services industry. Here are the top advantages of our new decentralized operating model and Web 3.0 convergence:

1. TRUSTWORTHINESS
Web 3.0 will make the onboarding process simpler and more user-friendly. Web 3.0 will ensure that end-users always have complete ownership and control over their online data by utilizing decentralized networks. Every subsequent evolution of the Web will ensure even greater trustworthiness for businesses by removing security concerns about storing any data.

2. IMPROVED CUSTOMER JOURNEY
Web 3.0 will help FinTech companies better understand the changing needs of their customers and their expectations. Businesses can automate customer visit mapping processes and allocate resources more efficiently with the Web 3.0 list of technology offerings to meet customer needs, facilitate better collaboration, and promote long-term reliability.

3. UNINTERRUPTED SERVICE
Because data will be stored in distributed locations due to fragmentation, Web 3.0 will significantly reduce account suspension and denial of distributed services. This will also help FinTech companies reduce hosting costs or server failures.

So we can finally say that the current rate of maturation of the FinTech industry has created an enormous demand for digital lending technology partners across businesses worldwide. Next-generation businesses are looking for a trusted partner who understands the current state of the FinTech market driven by Web 3.0 to unleash innovation to scale quickly. Partnerships like these provide an excellent opportunity to differentiate your company in this highly competitive world!

How realistic for the finance industry, specifically banks and fintech’s moving to web3 and DeFi.

What we can say right now is that banks really don’t like getting into Defi and Web3 or you can say into anything related to Cryptocurrencies. But obviously, the reasons differ by jurisdiction and bank; after all, the world is a huge place.

However, lawyers and banking professionals explain why some banks refuse to process crypto transactions and offer advice on what to do if your bank refuses to process a relevant transaction.

Many countries do not recognize cryptocurrencies, making it unlawful for banks to handle Bitcoin-related transactions. Banks in China and Bolivia, for example, will not execute Bitcoin transactions since it is illegal.

Even in areas where cryptocurrency is legal, some banks don't want to deal with irate clients asking for chargebacks from a crypto-related Ponzi scheme because they don't believe it's worth the effort. Some customers may not be aware of all the risks associated with cryptocurrency and banks don't want to pay for consumers who aren't well-informed—cheaper it's to avoid crypto entirely.

Banks do not want the difficulty of dealing with cryptocurrency-related crimes, whether their customers are perpetrators or victims. For many banks, the costs of dealing with their customers' cryptocurrency issues, as well as setting up cryptocurrency fraud or fraud prevention programs, are not worth the effort to work for what is still a small sector.

But these were the reasons for which Bank doesn’t want to go for adopting Defi and Web3 but now let’s discuss why they should get into Defi and Web3.

As recently, traditional banks made a mistake. Due to their precarious position and conservatism, the world's largest financial institutions have missed out on the rapid development of FinTech. Banks were forced to develop new solutions after the fact and quickly, which cost small FinTech firms market share. And now banks risk making the same mistake again, this time about Defi and Web3.

While cautious banks wait and watch, limiting bitcoin's future, companies like SEBA and Wirex are constructing a comprehensive ecosystem that combines fiat and Defi. These companies were among the first to connect the old financial system to the blockchain, and others should follow in their footsteps to avoid making the same mistake.

No amount of market growth or aggressive attempts by financial authorities to develop a common framework has been able to persuade banks of the legitimacy of the cryptocurrency and Defi industry. Extreme currency fluctuations and a lack of banking infrastructure in the emerging economy drive the growth of Decentralized Finance.

Defi has gained a lot in Mexico, India, and many African countries. The young people of these nations are eagerly learning new technologies, and Crypto trading is fast becoming the easiest way for them to send money. Banks considering expanding to emerging regions should keep in mind that Defi and Web 3 already have a place in the market.

So let’s finally find out that can we declare Web 3 the future of Finance?

As mentioned above in the article that the internet has undergone significant disruptions, transitioning from offering read-only, static content at its inception to a much more dynamic, interactive, and decentralized experience today. Web 3.0 is evolving as a result of innovations in decentralized finance (DeFi), blockchain, cryptocurrencies, and distributed ledgers. The total market capitalization of cryptocurrencies was $2.2 trillion in December 2021, while the total locked value in DeFi protocols increased by over 1000 percent year on year.

We can easily say that Web3 will be the future of Finance. You may have heard or seen that Elon Musk has bought Twitter and Binance’s CEO is also involved in it so they are maybe planning to adopt crypto for Twitter which will obviously open the ways of Web 3 and Defi in General And due to the launch of Crypto.Com’s CRO card and Binance’s Binance card for payments things are already not good for Banks and Financial sectors so let’s see what happens next.

Data Structures And Algorithms

Firdavs Kasymov — Sat, 07 May 2022 20:30:20 +0000

If you are a beginner or new to computer programming, some of the first things you will have to learn include data structures and algorithms. Data structures and algorithms are basics in programming that every developer and software engineer needs to know. Just like you learnt algebra in your first mathematics lessons, learning data structures and algorithms is crucial if you want to become a good programmer.

In today’s article, article, we will share with you everything you need to know about these two topics. Let’s jump right in!

What are data structures?

Data structures refer to the programmatic way of storing data so that it can be accessed and used effectively. It is through data structures that programmers are able to reference and manipulate data depending on the objective of the program being written. Every application uses various types of data structures in one way or another.

Having knowledge about the different data structures will help you learn how each of them works so that you choose the right one depending on the problem at hand.

Types of data structures

There are several types of data structures, but the ones every programmer needs to know are about six. These include the following;

Linear data structures

Arrays An array is the simplest type of data structure that contains elements of the same data type. That means one array cannot contain integers and strings at the same. It has to be only one data type. Arrays are also used to create more complex data types that I am about to share in the next paragraphs. Items in an array are indexed in order starting from 0.
Stacks This is the data structure type where items are stored using the LIFO (Last in First Out) principle. This simply means the items put last are removed first. Just like a pile of papers, the papers you put last are the ones will have to remove first.
Linked Lists Type of data structure where a sequence of elements are arranged in linear order and are all connected together. That means you have to access these items in order. Random access is not possible.
Queues This type is just like stacks, but instead of using the LIFO structure, it uses the FIFO (First In First Out) structure. That means the items added first are removed first and vice-versa.

Non-linear data structures

Graphs This is the type of data structure that consists of nodes that are usually referred to as vertices. Each vertex is connected to other vertices through edges to form a structure.
Tree This is another type of non-linear data structure which also consists of a collection of vertices and edges. Unlike in graphs, there can only be one edge between two vertices in tree data structures.

What is an algorithm?

In programming, an algorithm refers to a set of instructions that are written to accomplish a predefined task. An algorithm is not a complete program, but simply the core logic of the program. Algorithms are expressed either as informal high-level descriptions as pseudocode or using a flowchart.

The performance of an algorithm is measured based on two parameters; time complexity and space complexity. An efficient algorithm takes less time to execute and also takes up less space in computer RAM. Algorithms with more lines of code usually take up more RAM than those with fewer lines. As a programmer, your aim should be to create algorithms that take less time to execute and also take up less RAM.

Properties of an algorithm
Any algorithm must have the following properties;

Input: There should be zero or more inputs supplied to the algorithm
Output: The algorithm should bear at least one output
Definiteness: Every step of the algorithm needs to be clearly defined
Finiteness: Every algorithm should have a finite number of steps
Correctness: Every step of an algorithm must produce correct output

Types of Algorithms

There are various types of algorithms, but there are 6 main ones that every programmer needs to know;

Recursive Algorithm This refers to the type of algorithm that repeatedly calls itself until the problem is solved. A certain condition in the algorithm has to be met for the execution to stop.
Divide and conquer algorithms This is the type of algorithm where the algorithm to solve a certain problem is divided into two parts; the first part divides the problem into sub-problems of the same type. In the second part, these smaller subproblems are solved and then added together to form a solution for the overall problem.
Dynamic programming algorithms This is the type of algorithm that works by recalling the results from the previous run and using them to find the next solution. With these algorithms, a problem is also broken down into small problems that are solved once and their solutions stored for future use. Dynamic programming algorithms are usually used to solve optimization problems.
Greedy algorithm This refers to the type of problem-solving strategy that involves finding a locally optimum solution at each stage with the hope of using these solutions to find a globally optimum solution. This type of algorithm is normally used to solve optimization problems that require the maximum or minimum optimum results. Greedy algorithms are also pretty easy to implement. Some of the popular applications of greedy algorithms include; CPU Scheduling algorithms, Minimum spanning trees, Dijkstra shortest path algorithm, Fit algorithm in memory management, and Travelling salesman problem.
Brute force algorithms These are algorithms that involve interacting with all the possible solutions to search for one or more solutions that solve a certain function. You can think of this as trying all the possible combinations of numbers to find the passcode of a certain device. This type of algorithm is usually used when there is no other algorithm that can be used to speed up the process of solving a certain problem, so one has to check all the possible solutions to figure out which one solves the problem.
Backtracking algorithms These are algorithms where a problem is solved using an incremental approach. So, if a solution is not found at one stage, it is removed and we backtrack to find another solution. These algorithms are usually used to find a solution to decision problems.

How to boost the performance of a program by using the correct algorithms?

As you will see in your programming journey, some of the problems can be solved using any of the algorithm types we have just shared. However, there will be a difference in performance depending on the type of algorithm you choose to use. Let’s look at some of the factors you need to look at while choosing which algorithm to use.

Running Time Time complexity is one of the key factors you need to consider while choosing the appropriate algorithm to use. With all other factors constant, it is always best to choose an algorithm that solves the problem in less time. The time difference becomes significant when you have to use multiple algorithms to solve one big problem at hand.
Memory consumption The applications where these algorithms are implemented work with limited computing resources. That is why it is important to choose an algorithm type that takes up less memory space. Memory space usually depends on the size of input data needed for the algorithm to solve the problem at hand.
Parallel processing If the problem requires parallel processing for faster execution, then it is ideal to choose a type of algorithm that involves dividing the problem into smaller subproblems that can be solved independently and the solutions later combined to form the overall solution for the bigger problem. In such a case, divide and conquer algorithms are always the best choice.
Accuracy requirements You need to first look at the problem a figure out how accurate the output of an algorithm needs to be to solve the problem at hand. You should then choose the fastest algorithm that yields a solution within the acceptable range of accuracy.

Learning materials

A curated list of awesome places to learn and/or practice agorithms:
https://github.com/tayllan/awesome-algorithms
To practice your knowledge about algorithms and data structures you can use the below resources:
https://www.hackerrank.com
https://leetcode.com/

Final thoughts

There is a lot more you can learn about data structures and algorithms. Use the basics we have shared as a stepping stone to learning more about these two topics. You will have to dive deeper into the different types of data structures to figure out when to use which and the impact your choice will have on the performance of your program.

There are also more specific data structures and algorithm types that we haven’t covered in this article. However, the ones we have looked at are the common types that you will often encounter in your programming journey.

Monolith vs Micro-Services Everything you Need To Know

Firdavs Kasymov — Wed, 27 Apr 2022 15:22:42 +0000

One of the major choices you need to make before you start building an application is the structural architecture to use. Microservices and monolithic architectural designs are two of the most popular models used to create web applications. However, most of the web applications operating on a large scale now use the microservices architecture.

There are several other use cases where it makes more sense to choose microservice over monolithic architecture. In today’s article, we shall discuss all the major things you need to know about microservices, how they work, their pros and cons, best use cases, and a lot more. Let’s dive right in!

What are microservices?

Microservices is an architectural design where a big application is developed by building small autonomous services that work together to execute the various functionalities of the app. For instance, an eCommerce web application might have different microservices for the various functionalities including, registration, order management, Wishlist, search, etc.

Each of these microservices works independently, so a fault in one does not affect the operation of the others. With the monolithic architecture, an error in one section of the app will affect the functionality of the entire application. Some of the popular platforms that use the microservices approach include; Netflix, Amazon, Uber, eBay, and many more.

How do microservices work?

The way microservices work depends on the size and scale of the application. However, there are four main modules that all apps built using this architecture have. These include the API gateway, microservices, database, and Inter-microservices communication. Let me explain each of these in detail.

API Gateway: An API gateway is the entry point for the final users of the applications. The users of the app send the request to an API which then forwards it to the given microservice that was built to handle that task. For instance, when you click the sign-up button on a given web app, that request is sent to the registration microservice.
Database: The app developer can choose to have one database for all microservices. However, if you want all the services to be fully autonomous, it is best to have a database for each microservice.
Microservices: These are the small services that perform independent tasks such as payments, registration, search, order management, etc. The number of microservices may at times depend on how big the application is or how the developer decides to split up services. For instance, Netflix has over 1000 microservices.
Inter-microservices communication: Microservices communicate using different kinds of protocols such as REST and messaging. These enable the different microservice of an application to communicate whenever the need arises.

Advantages of using the microservices architecture

Easy fault isolation: With the microservices architecture it is much easier to isolate one service while the rest of the app is working. A similar fault could potentially lead to downtime of a monolithic application if the fault is severe enough.
Faster and more effective development of the app: Different teams can work on the various microservices without having to wait for another team to finish a specific section of the app.
Apps are more scalable: It is much easier to add new features and manage huge traffic. With microservices, each small service is built independently, so adding new features can be done much faster.
It is easier to use different technologies: With microservices, you can use different programming languages for each service without affecting the overall operation of the app. The procedure for using different languages for different sections of monolithic apps is more complicated.
Easier to understand: If there is anything to fix or changes to be made in a certain microservice, it is much easier to do such a task. This is because the size of code for the various microservices is small, so making changes can be done much faster even if the team working on this service is small.
Easier continuity: It is much easier for a new team member to understand the functionality and code of a microservices than the entire application. This makes continuity much easier.
Apps feel much faster on the user side: Apps using this architecture tend to load the various sections of the app much faster. This improves the overall user experience of the application.

Limitations of the microservices architecture

More complex deployment: Since a single application has multiple microservices, it is much harder to deploy on the web than monolithic applications. The communications infrastructure between the different services is also very sophisticated, hence making the app more complicated to deploy.
Requires more resources: Due to the complex networking involved to enable communication between the different services, apps built with the microservice architecture tend to take up a lot of computing resources than monolithic apps. This is an extra expense that the app developer needs to deal with.
Global testing and debugging are harder: It might be easier to test and detect errors for the individual microservices. However, testing the functionality of the entire app is much harder with this architecture.
Not ideal for small apps: Microservice architecture only makes sense for big apps that expect to get thousands or millions of users every day. For smaller apps, you are better off going for the monolithic architecture.

When to choose micro-service over monolith infrastructure

Yes, apps built with microservice architecture have several advantages over those built using monolithic architecture. However, this architecture is ideal for certain situations that you may need to know about before making the choice. Below are some of these scenarios.

When the main goal is to build a large-scale application that has the potential of attracting millions of users. However, if you are deploying an app that is to be used internally in your organization, it is best to use the monolithic architecture.
If the different functionalities of your app’s backend have to be coded in different languages. As we have earlier discussed, one of the benefits of using the microservices approach is it makes it easier to write the backend of every service using a different programming language.
If scalability is key to your application. Microservice architecture is ideal for apps that need constant revision and scaling up their user capacity.
When the app you intend to build has multiple functionalities and user journeys, it is best to develop it using the microservices architecture. For instance, an app that has several standalone business modules should be built using the microservice architecture.

How to handle microservices requests to other micro-services

One of the crucial factors that affect the performance of any web app built using the microservice architecture is how they communicate. Different microservices have to effectively communicate to execute a certain business task. There are three messaging patterns that microservices use to communicate with each. These include; Synchronous, asynchronous, and hybrid. Let’s explain each in detail.

Synchronous communication
This involves data moving to and from the endpoint in blocking iteration. That means the caller has to wait for a response from the receiver before proceeding to the next step. So, if microservice A sends a request to Microservice B, it has to wait for a response before sending any other requests

Asynchronous
With this communication protocol, a microservice sends messaging without having to wait for a response from the receiver.

Hybrid
A hybrid microservice supports both communication protocols. Most microservice apps use both protocols. So, the choice of which one to use depends on the type of communications that take place between the different microservices of a given application.

How database transactions can be handled on microservices

With monolithic apps, all data is in a single database, hence making it easier to achieve consistency in the various database transaction. However, with microservice apps, the different microservices have independent databases, so database transactions are distributed. This makes it more challenging to achieve consistency across all the databases in the network.

Monolithic apps databases handle ACID transactions to execute all-or-nothing transactions while ensuring consistency. ACID stands for;

Atomicity: All requests are executed successfully or everything fails .
Consistency: All data in the database is kept in a valid state as stated by the rules of the database.
Isolation: separated transactions do not interfere with each other .
Durability: After a transaction is complete, all changes are stored permanently.

As you may guess, the ACID transactions methodology does not work with microservice apps since the different microservices databases are distributed on a network. There are two main challenges faced in handling microservice database transactions;

How to keep the transactions atomic and
How to handle concurrent transactions

The solutions used to fix these two challenges are as follows;

Two-phase commit: This includes the prepare phase and the commit phase. In the prepare phase, all the relevant microservices for a given transaction inform the coordinator that they are ready for the transaction. During the commit phase, the coordinator either issues a commit or rollback command.
Eventual Consistency and Compensation: With this approach, all microservices publish an event whenever there is a change in its data. The rest of the services subscribe to this event and update their databases to ensure consistency.

Cloud computing – everything you need to know

Firdavs Kasymov — Thu, 14 Apr 2022 11:20:22 +0000

Cloud computing is one of the most popular buzzwords amongst technology enthusiasts and professionals. It is a revolutionary technology that has changed how applications are built, deployed, and used. I’m pretty sure everyone reading this article has used an app or service that takes advantage of cloud computing technology.

Despite being a popular topic, not so many people understand the fundamentals of cloud computing and how significant this technology is to this era where we rely on computers to do almost everything. In this article, we will discuss all the basics of cloud computing, including the cloud computing providers and how different categories of users can take full advantage of this technology. Let’s dive in!

What is cloud computing?

In simple terms, cloud computing refers to the delivery of computing services, including server hardware, storage, networking, analytics, software, and databases over the internet. Back in the days before cloud computing, organizations used to rely on their local servers to host their in-house applications and storage of data.

Cloud computing has changed all this; companies no longer have to spend thousands or even millions of dollars building local computing infrastructure. They can simply rent computing resources and storage they need from any of the cloud computing providers, including Amazon Web Services (AWS), Microsoft Azure, Google Cloud, Alibaba Cloud, and many more.

AWS is still the most popular cloud computing service provider with a market share of over 33%. For the most of this article, AWS will be our major focus since it is the most popular option and the one with the most services.

Source

Cloud computing services delivery models

1. Infrastructure as a Service (IaaS)

This is a cloud computing delivery model where providers such as AWS, Microsoft Azure, and Google Cloud offer users computing resources, storage, and application programming interfaces (APIs) that they can use for their various workloads. AWS has a wide range of services that users can choose from based on their computing and storage needs.

Some of the most of most used ones include Elastic Computer Cloud (EC2) for computing and Amazon Simple Storage Service (Amazon S3) for storage. With these services, users pay for what they need. For instance, EC2 gives you the option to choose the number of CPU cores and the amount of RAM you need for your application.

AWS has more than 200 services that developers and other professionals can choose from to get their work done.

2. Platform as a Service (PaaS)

PaaS is a cloud computing delivery model where the service providers host software development tools on their servers. These tools are mainly used for building and testing applications before they are deployed on the web for final consumers to use. Some of the popular PaaS platforms include Google App Engine, AWS Elastic Beanstalk, and Salesforce's Lightning Platform.

These platforms provide all the tools that developers need in their app development cycle. Many of these PaaS platforms also host the software after it is developed. However, a developer can choose to host their software on another platform of their choice.

3. Software as a Service (SaaS)

This refers to a cloud computing delivery model where providers deliver software applications via the internet. The target market for SaaS applications are final consumers. SaaS applications can be accessed by anyone via their mobile device or computer. Some of the popular SaaS platforms include Google WorkSpace, Microsoft 365, HubSpot, Salesforce, Shopify, and many more.

Benefits of using cloud computing services like AWS

1. Cost-effective

Cost is one of the reasons organizations are shifting from local computing to cloud computing. With cloud computing companies pay for the specific resources they need. Services such as Amazon’s EC2 give users the option to pay based on the exact computing resources they are using. That means costs only go up when your application is being used the most.

2. It is much easier to scale up

Cloud computing service providers like AWS allow you to get more resources for your application with just a few clicks. You also have the option of automating this process. Scaling up with local computing requires procuring and configuring new servers which is time-consuming and costly.

3. Deploying apps in different regions

Applications deployed on a server that is geographically closer to you run faster than those deployed further away. AWS has computing infrastructure in up to 26 geographical regions spread out across all the five continents, including North America, South America, Africa, Asia, Europe, and Australia. So, developers can deploy their apps in regions where they expect most of their users to be located.

4. Unlimited storage capacity

No, matter how much space you need, cloud services like Amazon’s S3 got you covered. As we shared earlier, you have the option of paying for the exact kind of space you need. So, you can always increase the size based on your storage demands.

5. Data security

AWS and other cloud computing providers spend billions of dollars on the security of their platforms. So, if you deploy your app on any of the top-rated platforms, you will be assured of top-notch security.

Final thoughts

Cloud computing is one of those fundamental technologies that is affecting almost everyone in a certain way. Enterprises and businesses now rely on IaaS platforms to deploy apps without having to incur the huge costs of buying and maintaining servers. On the other hand, developers and other professionals are using PaaS platforms to build, test, and deploy apps.

And finally, final consumers now use SaaS applications to do power-demanding tasks like photo editing without necessarily having to buy expensive and powerful computers. As long as you are using the internet, you will at some use a cloud computing service.