DEV Community: Georgii Kliukovkin

System design. Monitoring and Alerting System.

Georgii Kliukovkin — Sat, 31 Dec 2022 00:29:53 +0000

This article is more like tips to that question rather than system design itself. If you are looking for the system design try this amazing book.

Clarifying questions to establish the scope:

What metrics do we want to collect?

What are the supported alert channels?

How long should we keep the data?

Can we reduce the resolution of the metrics data for long-term storage?

Do we need to collect logs?

What is the scale of infrastructure we are monitoring with this system?

Is it a SaaS or just for internal purposes?

Fundamentals:

A metrics monitoring and alerting system contains 5 features:

1) Collect data from sources

2) Transfer data(what is different between 1 and 2)?

3) Store data

4) Alerting

5) Visualization

Data access pattern

We will persistently collect data and write it to the database. Heavy write load.

Visualization and alerting services would send queries to the databases, an access pattern will vary depending on graphs and alerts. So the read load will be spiky.

Data storage system

SQL - too complicated queries, bad performance under heavy writes

NoSQL - Cassandra and Bigtable can be used, they are optimized for heavy writes. But we can find a better solution

Time-Series databases - InfluxDB and Prometheus are the two most popular databases. They are designed to store large volumes of time series data and quickly perform real-time analysis on that data. Another feature is the efficient aggregation and analysis of a large amount of time-series data by labels. InfluxDB builds indexes on labels to facilitate the fast lookup of time-series by labels.

Pull or push model to collect metrics

Pull model

With such an approach we need to know the list of all services that we want to monitor. It is hard to maintain especially at a large scale when we need to add/remove services frequently. Of course, we can use some Service Discovery services like ZooKeeper where services will register their availability and our collector can be notified about any changes.

Push model

With that model, we need to install an agent on every monitored service. An agent is a long-running software that will collect metrics and send them to our collector.

Push or pull

Depends totally on the details of the system that we want to support. Large scale system probably needs to support both approaches cause it might uses serverless components and you will not be able to use an agent.

High level design with Kafka

Why Kafka needed?

avoid data loss in case if DB is unavailable
decoupling data collectors and data processors

Kth Largest Element in an Array - Quickselect Using Lomuto Partitioning Scheme.

Georgii Kliukovkin — Thu, 22 Dec 2022 01:38:52 +0000

According to Leetcode:

We highly recommend Kth Largest Element in an Array, which has been asked many times in an Amazon phone interview.

Task

Given an integer array nums and an integer k, return the kth largest element in the array.
Note that it is the kth largest element in the sorted order, not the kth distinct element.
You must solve it in O(n) time complexity.

Heap

There are several ways to solve this task. The most straightforward is to iterate through the array and keep the top Kth largest element while iterating. For that purpose, a heap is the best data structure.
The algorithm:

iterate through the array and put each element to the heap
if the heap size is greater than k - remove the smallest one(which will be at the beginning of our queue)
return the first element

class Solution {
    public int findKthLargest(int[] nums, int k) {
        PriorityQueue<Integer> heap = new PriorityQueue<>();
        for (int n: nums) {
          heap.add(n);
          if (heap.size() > k)
            heap.poll();
        }

        return heap.poll();        
  }
}

We iterate through the array and each time we insert the element in the heap of length K, so the time complexity is O(NlogK). Can we do better? :)

Quickselect time complexity

First, let’s discuss high-level design, and then we can dive deeper. Here is a link to the problem.
We need to find K-th largest element in the array. Since for us more comfortable sorting an array in non-decreasing order, let’s rephrase the task and say that we need to find the N-k smallest element in the array.

The idea behind the algorithm is to use a partitioning algorithm from quickselect.
We partition the whole array(O(n) time complexity).
Then through away half of it and continue with another half(O(n/2) time complexity).

Again through away half of it and continue with 1/4 of the original array (O(n/4) time complexity).
Continue doing so until we reach a single element. To summarize overall time complexity:

n + n/2 + n/4 + n/8 +... ~ 2n = n

You may be confused about time complexity thinking “wait, we are doing almost the same with binary search, it is logN time complexity, where is the logN part”. Well, you are right. The key point here is that not all partitions are doing the same amount of work.

This more precise analysis, which uses the fact that the work done keeps decreasing on each iteration, gives the O(n) runtime.
If you are still confused about time complexity take a look at this answer and this article.

Partition algorithm(Lomuto partitioning scheme)

One chooses a pivot and defines its position in a sorted array in a linear time using so-called partition algorithm.
The toughest thing in this algorithm is understanding how the partition works. Let’s imagine we have an array [2,6,3,4,7,1,8,5]. With the Lomuto partitioning scheme the algorithm will look like this:

choose the last element as a pivot
create 2 pointers, i and startIndex, both starting from the beginning of the target interval.
The first pointer i will scan the whole interval and check the condition - if the value at i is less than the pivot then swap it with the value at the pointer startIndex and increment startIndex
swap the pivot(which is placed at the end of the array) with the value at startIndex (since this is a place for our pivot value) and return startIndex . To achieve that we need a single for loop and dedicated variable startIndex which is equal to the index of the first element in the array(in our case it is 0).

Now we ended up with storeIndex = 3, which is actually a place for our pivot element. We can swap them and return this pivot index. The array looks like [2,3,4,5,7,6], pivot index is 3.

public void swap(int i, int j) {
  int temp = nums[i];
  nums[i] = nums[j];
  nums[j] = temp;
}

public int partition(int start, int end) {
  int pivot = nums[end];
  int storeIndex = start;
  for (int i = start; i <= end; i++) {
    if (nums[i] < pivot) {
      swap(i, storeIndex);
      storeIndex++;
    }
  }
  //don't forget to move pivot element from the end of the array to its position
  swap(end, storeIndex);
  return storeIndex;
}

Random pivot

The worst case of this algorithm will be O(n^2). Why is that so? The algorithm is sensitive to the pivot that is chosen. Imagine you have already sorted the array and each time you select the first element as a pivot. That means each partition will decrease the range of the elements only by 1. To avoid this, we need to select a random pivot each time:

Random random = new Random();
//asume that 'start' is an index of the first element in search interval of the array
//and 'end' is an index of the last element in that interval, then:
int pivot = left + random.nextInt(right - left);

Quickselect algorithm

At last, we need to implement a quickselect algorithm. The steps are:

select a random pivot
partition array with this pivot and return its new index
if the index is equal to N - k, then we find our value, otherwise select one of the two parts of the array, and repeat the algorithm Here is the source code:

class Solution {
    int[] nums;
    public void swap(int i, int j) {
      int temp = nums[i];
      nums[i] = nums[j];
      nums[j] = temp;
    }

    public int partition(int start, int end, int pivotIndex) {
      int pivot = nums[pivotIndex];
      //move pivot to the end of the array;
      swap(end, pivotIndex);
      int startIndex = start;
      for (int i = start; i <= end; i++) {
        if (nums[i] < pivot) {
          swap(i, startIndex);
          startIndex++;
        }
      }
      //don't forget to move pivot element from the end of the array to its position
      swap(end, startIndex);
      return startIndex;
    }
    public int findKthLargest(int[] nums, int k) {
        this.nums = nums;
        return quickselect(0, nums.length - 1, nums.length - k);
    }
    public int quickselect(int start, int end, int k) {
        if (start == end) {
            return nums[start];
        }
        Random random = new Random();
        int pivotIndex = start + random.nextInt(end - start);
        pivotIndex = partition(start, end, pivotIndex);
        if (pivotIndex == k) {
            return nums[pivotIndex];
        }
        if (pivotIndex < k) {
            return quickselect(pivotIndex + 1, end, k);
        }
        return quickselect(start, pivotIndex - 1, k);
    }
}

Inheritance vs Composition: Using a Role-Playing Game in JavaScript as an Example

Georgii Kliukovkin — Thu, 22 Dec 2022 00:38:35 +0000

Inheritance Problems

Duplication of code in children
Excessive complexity in the inheritance hierarchy
Changing the behavior of the parent can lead to errors in the children

In this article, we will look at what these problems are about and how we can solve them using composition.

the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. - Joe Armstrong, creator of Erlang

Role-Playing Game Inheritance

Consider the process of creating a hierarchy of role-playing game characters. Initially, two types of characters are required - Warrior and Mage, each of which has a certain amount of health and a name. These properties are public and can be moved to the parent Character class.

class Character { 
   constructor(name) { 
      this.name = name; 
      this.health = 100;
   } 
}

A warrior can strike, spending his stamina:

class Warrior extends Character { 
   constructor(name) { 
      super(name); 
      this.stamina = 100; 
   } 
   fight() { 
      console.log(`${this.name} takes a mighty swing!`); 
      this.stamina--; 
   } 
}

And a mage can cast spells that spend some amount of mana:

class Mage extends Character {
   constructor(name) { 
      super(name); 
      this.mana = 100; 
   } 
   cast() { 
      console.log(`${this.name} casts a fireball!`); 
   this.mana--; 
   } 
}

Paladin Class Problem

Now, let’s introduce a new class, Paladin. A Paladin can both fight and cast spells. How can we solve this? Here are a couple of solutions that share the same lack of elegance:

We can make Paladin a descendant of Character and implement both the fight() and cast() methods in it from scratch. In this case, the DRY principle is violated because each of the methods will be duplicated upon creation and will need constant synchronization with the methods of the Mage and Fighter classes to track changes.
The fight() and cast() methods can be implemented at the Character class level so that all three character types have them. This is a slightly better solution, but in this case, the developer must override the fight() method for the mage and the cast() method for the warrior, replacing them with empty methods or consoling an error.

Composition

These problems can be solved with a functional approach using composition. It is enough to start not from their types, but from their functions. Basically, we have two key features that determine the abilities of the characters - the ability to fight and the ability to cast spells.

These features can be set using factory functions that extend the state that defines the character:

const canCast = (state) => ({ 
   cast: (spell) => { 
      console.log(`${state.name} casts ${spell}!`); 
      state.mana--; 
   } 
}) 
const canFight = (state) => ({ 
   fight: () => { 
      console.log(`${state.name} slashes at the foe!`); 
      state.stamina--; 
   } 
})

Thus, a character is defined by a set of these abilities and initial properties, both general (name and health) and private (stamina and mana):

const fighter = (name) => { 
   let state = { name, health: 100, stamina: 100 } 
   return Object.assign(state, canFight(state)); 
} 
const mage = (name) => { 
   let state = { name, health: 100, mana: 100 } 
   return Object.assign(state, canCast(state)); 
} 
const paladin = (name) => { 
   let state = { name, health: 100, mana: 100, stamina: 100 } 
   return Object.assign(state, canCast(state), canFight(state)); 
}

Conclusion

With composition, you can avoid inheritance problems, and javascript is the perfect language to do so.

Integration tests with Go and testcontainers

Georgii Kliukovkin — Sat, 23 Jul 2022 12:35:00 +0000

Let's start by understanding what testing is. There is a testing pyramid (unit tests, integration and end-2-end testing)

Horizontal - the number of tests. Vertical - maintain cost.

Unit tests test the a+b function, where we describe the positive and negative scenario. There is a good video for unit tests. This topic is not in the context of our article.

E2e tests are usually written by testers, they are needed to test the full flow within a story. For example, we send a request to a raised server, the server processes it (goes to other services, goes to the database, to the radish, etc.). For us, the service is a black box. We are testing responses to our requests.

The most effective approach in practice, however, remains the following scheme (testing Trophy):

Integration tests - checking the integration of the tested service with other services. We fully check the flow of a particular service.

Integration tests become the foundation of all development testing. They are usually written by the developers themselves. The purpose of this article is to show you how easy and efficient it is to write integration tests.

Integration tests on the example of an http server that interacts with the database and goes to other services.

First, let's describe the service itself. Service in Github. The service uses gorilla/mux and Postgres. It implements a clean architecture and has such file structure:

❯ tree user_service
user_service
├── api
│   └── user.go
├── billing
│   ├── api.go
│   └── client.go
├── cmd
│   └── main.go
├── docker-compose.yml
├── domain
│   └── user.go
├── handler
│   └── handler.go
├── migrate
│   ├── migrate.go
│   └── migrations
│       └── 20220612163022_create_users.sql
├── server
│   └── server.go
├── storage
│   └── storage.go
└── use_case
    └── use_case.go

10 directories, 12 files

Step 1. Testing createUser

The method for creating a record about a new user in the database contains the main logic in the repository layer(we simply pass payload from handler through use_case to repository) and looks like this:

func (s *storage) CreateUser(ctx context.Context, name string) (domain.User, error) {
    query := `INSERT INTO users (name) VALUES ($1) RETURNING id, name, balance, created_at, updated_at`
    res, err := s.db.QueryxContext(ctx, query, name)
    if err != nil {
        return domain.User{}, err
    }

    defer res.Close()

    if !res.Next() {
        return domain.User{}, IncorrectQueryResponse
    }
    var resUser domain.User
    if err := res.StructScan(&resUser); err != nil {
        return domain.User{}, err
    }

    return resUser, nil
}

Now let's write a simple test. (test won't work)

func TestCreateUser(t *testing.T) {

    // copy from main
    repo, err := storage.New(dbDsn)
    require.NoError(t, err)
    useCase := use_case.New(repo, nil)
    h := handler.New(useCase)
    ///

    requestBody := `{"name": "test_name"}`

    // use httptest
    srv := httptest.NewServer(server.New("", h).Router)

    _, err = srv.Client().Post(srv.URL+"/users", "", bytes.NewBufferString(requestBody))
    require.NoError(t, err)
}

The test raises the service itself and calls request handler(using httptest)

We run the test and see that it does not work, since there is no connection to the database. So, we need to create a connection to the database. Moreover, we need to make sure that the database is automatically up and running when the test is launched and there is no need to do it manually. This is where the popular testContainers tool and its Go implementation can help.

Step 2. Create testcontainer with Postgres

Let's describe how the request to launch the database in the container will look like:

testcontainers

req := testcontainers.ContainerRequest{
    Env: map[string]string{
        "POSTGRES_USER":     "user",
        "POSTGRES_PASSWORD": "password",
        "POSTGRES_DB":       "postgres",
    },
    ExposedPorts: []string{"5432/tcp"},
    Image:        "postgres:14.3",
    WaitingFor: wait.ForExec([]string{"pg_isready"}).
        WithPollInterval(2 * time.Second).
        WithExitCodeMatcher(func(exitCode int) bool {
            return exitCode == 0
        }),
}

testcontainers.ContainerRequest describes what our Docker container will look like.

Let's take a look at our docker-compose.yml:

version: '3.8'
services:
  db:
    image: postgres:14.3
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=postgres
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready" ]
      interval: 2s

we can see that ContainerRequest almost completely repeats the description in docker-compose file:

Env - environment variables, in this case the user, password, and DB schema name. Same as docker-compose.
ExposedPorts - analogue to command docker run -p <port> , calling it, dockerd will map selected port <port> inside container with randomly selected available port on the host.
Image - Docker image and it's tag.
WaitingFor - launch waiting strategy, similar to healthcheck. With it, we check that the container is up and running.

container, err := testcontainers.GenericContainer(ctx, 
   testcontainers.GenericContainerRequest{
      ContainerRequest: req,
      Started:true,
   }
)

testcontainers.GenericContainer creates the container.

Started: true means we need to wait until a container will start. If you remove this parameter or set to false testcontainers won't wait for our condition in described in waitingFor field of the ContainerRequest and test will fail.

Finally let's describe the structure of a Postgres container itself:

type PostgreSQLContainer struct{
   testcontainers.Container
   MappedPort string
   Host       string
}

Besides testcontainers.Container in our structure will be stored external host an port of a Docker container with Postgres. Will can get them in a such way:

mappedPort, err := container.MappedPort(ctx, "5432")
host, err := container.Host(ctx)

Also let's writ helper function which will return real DSN address for the connection to Postgres:

// GetDSN returns DB connection URL.
func (c PostgreSQLContainer) GetDSN() string {
   return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable", "user", "password", c.Host, c.MappedPort, "postgres")
}

That is all for configuration and running our testcontainer. Whole code looks like:

package step_2

import (
    "context"
    "fmt"
    _ "github.com/jackc/pgx/v4/stdlib"
    _ "github.com/lib/pq"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/wait"
    "time"
)

type PostgreSQLContainer struct {
    testcontainers.Container
    MappedPort string
    Host       string
}

func (c PostgreSQLContainer) GetDSN() string {
    return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable", "user", "password", c.Host, c.MappedPort, "postgres_test")
}

func NewPostgreSQLContainer(ctx context.Context) (*PostgreSQLContainer, error) {
    req := testcontainers.ContainerRequest{
        Env: map[string]string{
            "POSTGRES_USER":     "user",
            "POSTGRES_PASSWORD": "password",
            "POSTGRES_DB":       "postgres_test",
        },
        ExposedPorts: []string{"5432/tcp"},
        Image:        "postgres:14.3",
        WaitingFor: wait.ForExec([]string{"pg_isready"}).
            WithPollInterval(1 * time.Second).
            WithExitCodeMatcher(func(exitCode int) bool {
                return exitCode == 0
            }),
    }
    container, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: req,
        Started:          true,
    })
    if err != nil {
        return nil, err
    }

    host, err := container.Host(ctx)
    if err != nil {
        return nil, err
    }

    mappedPort, err := container.MappedPort(ctx, "5432")
    if err != nil {
        return nil, err
    }

    return &PostgreSQLContainer{
        Container:  container,
        MappedPort: mappedPort.Port(),
        Host:       host,
    }, nil
}

Step 2.1. Refactoring

Let's not hardcode user and password and provide possibility to change configuration let's add structure for the config and use pattern options.

package step_2_1_improved_psql_container

import (
    "context"
    "fmt"

    "github.com/docker/go-connections/nat"
    _ "github.com/jackc/pgx/v4/stdlib"
    _ "github.com/lib/pq"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/wait"
)

type (
    PostgreSQLContainer struct {
        testcontainers.Container
        //add Config
        Config PostgreSQLContainerConfig
    }
    //also add options pattern method
    PostgreSQLContainerOption func(c *PostgreSQLContainerConfig)

    PostgreSQLContainerConfig struct {
        ImageTag   string
        User       string
        Password   string
        MappedPort string
        Database   string
        Host       string
    }
)

func (c PostgreSQLContainer) GetDSN() string {
    return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable", c.Config.User, c.Config.Password, c.Config.Host, c.Config.MappedPort, c.Config.Database)
}

func NewPostgreSQLContainer(ctx context.Context, opts ...PostgreSQLContainerOption) (*PostgreSQLContainer, error) {
    const (
        psqlImage = "postgres"
        psqlPort  = "5432"
    )

    config := PostgreSQLContainerConfig{
        ImageTag: "11.5",
        User:     "user",
        Password: "password",
        Database: "db_test",
    }
    //handle possible options
    for _, opt := range opts {
        opt(&config)
    }

    containerPort := psqlPort + "/tcp"

    req := testcontainers.GenericContainerRequest{
        ContainerRequest: testcontainers.ContainerRequest{
            Env: map[string]string{
                "POSTGRES_USER":     config.User,
                "POSTGRES_PASSWORD": config.Password,
                "POSTGRES_DB":       config.Database,
            },
            ExposedPorts: []string{
                containerPort,
            },
            Image:      fmt.Sprintf("%s:%s", psqlImage, config.ImageTag),
            WaitingFor: wait.ForListeningPort(nat.Port(containerPort)),
        },
        Started: true,
    }

    container, err := testcontainers.GenericContainer(ctx, req)
    if err != nil {
        return nil, fmt.Errorf("getting request provider: %w", err)
    }

    host, err := container.Host(ctx)
    if err != nil {
        return nil, fmt.Errorf("getting host for: %w", err)
    }

    mappedPort, err := container.MappedPort(ctx, nat.Port(containerPort))
    if err != nil {
        return nil, fmt.Errorf("getting mapped port for (%s): %w", containerPort, err)
    }
    config.MappedPort = mappedPort.Port()
    config.Host = host

    fmt.Println("Host:", config.Host, config.MappedPort)

    return &PostgreSQLContainer{
        Container: container,
        Config:    config,
    }, nil
}

Step 3. Migrations

Let's run test once again. We will see another error. DB is stand up and running, but insert is not working since there is no schema. To solve this issue let's run migration script. We will do it on each test run. Just copy code from cmd/main.go into our test.

// run migrations
err = migrate.Migrate(psqlContainer.GetDSN(), migrate.Migrations)
require.NoError(t, err)

Run test once again and see that it finally passed.

Step 4. Test getUser

Test for getUser will look like this:

func TestGetUser(t *testing.T) {
    //---------------- common part for all tests
    ctx, ctxCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer ctxCancel()

    psqlContainer, err := step2.NewPostgreSQLContainer(ctx)
    defer psqlContainer.Terminate(context.Background())
    require.NoError(t, err)

    err = migrate.Migrate(psqlContainer.GetDSN(), migrate.Migrations)
    require.NoError(t, err)

    repo, err := storage.New(psqlContainer.GetDSN())
    require.NoError(t, err)

    useCase := use_case.New(repo, nil)

    h := handler.New(useCase)

    srv := httptest.NewServer(server.New("", h).Router)
  //------------------------------------------------

    // test body of below ----------------------------
    res, err := srv.Client().Get(srv.URL + "/users/1")
    require.NoError(t, err)

    defer res.Body.Close()

    require.Equal(t, http.StatusOK, res.StatusCode)

    // check response
    response := api.GetUserResponse{}
    err = json.NewDecoder(res.Body).Decode(&response)
    require.NoError(t, err)

    // id maybe any
    // so we will check each field separately
    assert.Equal(t, 1, response.ID)
    assert.Equal(t, "test_name", response.Name)
    assert.Equal(t, "0", response.Balance.String())
}

The problem with testing getUser is that we need to have a record about that user in our DB. Of course we can solve it simply running getUser right after createUser sequentially. But this approach is an anti-pattern since each test should work in isolation and test only the requested functionality.

Step 5. Testfixtures

To solve the problem we will use testfixtures. Create a folders fixtures и fixtures/storage and put a file users.yaml inside:

- id: 1
  name: test_name
  balance: 0

Now install the library go get "github.com/go-testfixtures/testfixtures/v3" and add this code after a common part and before we call get.

db, err := sql.Open("postgres", psqlContainer.GetDSN())
require.NoError(t, err)

fixtures, err := testfixtures.New(
    testfixtures.Database(db),
    testfixtures.Dialect("postgres"),
    testfixtures.Directory("fixtures/storage"),
)
require.NoError(t, err)
require.NoError(t, fixtures.Load())

Run test once again and see that is passed.

Step 6. Testsuite

As you may notice each test has a common part. To optimise our code and avoid duplication we will use testsuites from the testify library. This tool helps us to describe actions we need to take before each test.

Let's create a structure for our TestSuite:

type TestSuite struct {
    suite.Suite
    psqlContainer *step2.PostgreSQLContainer
    server        *httptest.Server
}

Now let's describe special method SetupSuite() which will run before launching each of the test of this TestSuite. Move the common part inside of it:

func (s *TestSuite) SetupSuite() {
    // create db container
    ctx, ctxCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer ctxCancel()

    psqlContainer, err := step2.NewPostgreSQLContainer(ctx)
    s.Require().NoError(err)

    s.psqlContainer = psqlContainer
    //

    // run migrations
    err = migrate.Migrate(psqlContainer.GetDSN(), migrate.Migrations)
    s.Require().NoError(err)
    //

    // copy from main
    repo, err := storage.New(psqlContainer.GetDSN())
    s.Require().NoError(err)

    useCase := use_case.New(repo, nil)
    h := handler.New(useCase)
    ///

    // use httptest
    s.server = httptest.NewServer(server.New("", h).Router)
    //
}

Also let's describe TearDownSuite() method, which will be executed after all tests from the TestSuite will be done. To avoid memory leak let's terminate our container:

func (s *TestSuite) TearDownSuite() {
    ctx, ctxCancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer ctxCancel()

    s.Require().NoError(s.psqlContainer.Terminate(ctx))

    s.server.Close()
}

Last thing we need for our TestSuite is a test function which will take t *testing.T argument and inject it in the TestSuite:

func TestSuite_Run(t *testing.T) {
    suite.Run(t, new(TestSuite))
}

Awesome. TestSuite described and able to work. Now all our tests contain only testing methods, check of the return values and fixtures where needed:

func (s *TestSuite) TestCreateUser() {
    requestBody := `{"name": "test_name"}`

    res, err := s.server.Client().Post(s.server.URL+"/users", "", bytes.NewBufferString(requestBody))
    s.Require().NoError(err)

    defer res.Body.Close()

    s.Require().Equal(http.StatusOK, res.StatusCode)

    // check response
    response := api.CreateUserResponse{}
    err = json.NewDecoder(res.Body).Decode(&response)
    s.Require().NoError(err)

    // id maybe any
    // so we will check each field separately
    s.Assert().Equal("test_name", response.Name)
    s.Assert().Equal("0", response.Balance.String())
}

func (s *TestSuite) TestGetUser() {
    // create fixtures
    db, err := sql.Open("postgres", s.psqlContainer.GetDSN())
    s.Require().NoError(err)

    fixtures, err := testfixtures.New(
        testfixtures.Database(db),
        testfixtures.Dialect("postgres"),
        testfixtures.Directory("../step_5_add_testfixtures/fixtures/storage"),
    )
    s.Require().NoError(err)
    s.Require().NoError(fixtures.Load())
    //

    res, err := s.server.Client().Get(s.server.URL + "/users/1")
    s.Require().NoError(err)

    defer res.Body.Close()

    s.Require().Equal(http.StatusOK, res.StatusCode)

    // check response
    response := api.GetUserResponse{}
    err = json.NewDecoder(res.Body).Decode(&response)
    s.Require().NoError(err)

    // so we will check each field separately
    s.Assert().Equal(1, response.ID)
    s.Assert().Equal("test_name", response.Name)
    s.Assert().Equal("0", response.Balance.String())
}

Step 7. UpdateUserBalance test

Method updateUserBalance firstly calling external service Billing, request some information and based on it update the balance. Let's write test:

func (s *TestSuite) TestDepositBalance() {
    // create fixtures
    db, err := sql.Open("postgres", s.psqlContainer.GetDSN())
    s.Require().NoError(err)

    fixtures, err := testfixtures.New(
        testfixtures.Database(db),
        testfixtures.Dialect("postgres"),
        testfixtures.Directory("../step_5/fixtures/storage"),
    )
    s.Require().NoError(err)
    s.Require().NoError(fixtures.Load())
    //

    requestBody := `{"id": 1, "amount": "100"}`

    res, err := s.server.Client().Post(s.server.URL+"/users/deposit", "", bytes.NewBufferString(requestBody))
    s.Require().NoError(err)

    defer res.Body.Close()

    s.Require().Equal(http.StatusOK, res.StatusCode)

    // check response
    response := api.GetUserResponse{}
    err = json.NewDecoder(res.Body).Decode(&response)
    s.Require().NoError(err)

    s.Assert().Equal(1, response.ID)
    s.Assert().Equal("test_name", response.Name)
    s.Assert().Equal("100", response.Balance.String())
}

And this test won't work too :) The problem now is that we need to go to the external server. Because we are writing test for the certain handler in our User service we don't need to test the external service. The only thing we need is to supply integration with it API. In other words, we need to mock a call to the external service and provide a response.

Step 8. httpmock

We will use httpmock for this purpose. Inside function setupSuite(), where we created useCase and provided nil as a billingClient now we will pass mocked http client:

func (s *TestSuite) SetupSuite() {
    //...
    mockClient := &http.Client{}
    httpmock.ActivateNonDefault(mockClient)

    billingClient := billing.New(mockClient, billingAddr)
    useCase := use_case.New(repo, billingClient)
    //...
}

At the end of the function TearDownSuite() lets deactivate mocked httpClient:

func (s *TestSuite) TearDownSuite() {
    //...
    httpmock.DeactivateAndReset()
}

Now let's mock a call to the external service:

httpmock.RegisterResponder(
    http.MethodPost,
    billingAddr+"/deposit",
    httpmock.NewStringResponder(http.StatusOK, ""),
)

And now the test will finally passed.

Step 9. API fixtures

As we can see, all our tests come down to filling requests and checking responses. We can also optimize this by moving the request and response structures into separate files.

In the fixtures directory create a new one /api and a file fixtures.go:

package fixtures
import (
    "embed"
)

//go:embed fixtures
var Fixtures embed.FS

We will use go:embed FS, you can read more about it here or here. Long story short, it allows you to get path from the file it contains without In a nutshell, it allows you to get the path to the file in which the structure is located, without having to write the full path to the file, which is often problematic.

Also let's add a structure to this file:

type FixtureLoader struct {
    t           *testing.T
    currentPath fs.FS
}

func NewFixtureLoader(t *testing.T, fixturePath fs.FS) *FixtureLoader {
    return &FixtureLoader{
        t:           t,
        currentPath: fixturePath,
    }
}

Let's also add 2 methods to it. The first one reads the contents of the file and returns a string with the contents:

func (l *FixtureLoader) LoadString(path string) string {
    file, err := l.currentPath.Open(path)
    require.NoError(l.t, err)

    defer file.Close()

    data, err := io.ReadAll(file)
    require.NoError(l.t, err)

    return string(data)
}

The second method uses the first one to get the string and then parses the template using the standard library html/template:

func (l *FixtureLoader) LoadTemplate(path string, data any) string {
    tempData := l.LoadString(path)

    temp, err := template.New(path).Parse(tempData)
    require.NoError(l.t, err)

    buf := bytes.Buffer{}

    err = temp.Execute(&buf, data)
    require.NoError(l.t, err)

    return buf.String()
}

We will also create two files inside the folder /api:

create_user_request.json

{
  "name": "test_name"
}

create_user_response.json.temp

{
  "id": {{.id}},
  "name": "test_name",
  "balance": "0"
}

In the test itself, we will parse these files and compare actual with expected. To do this, we will create two helper functions that will help compare 2 sets of json data:

func JSONEq(t *testing.T, expected, actual any) bool {
    return assert.JSONEq(t, jsonMarshal(t, expected), jsonMarshal(t, actual))
}

func jsonMarshal(t *testing.T, data any) string {
    switch v := data.(type) {
    case string:
        return v
    case []byte:
        return string(v)
    case io.Reader:
        data, err := io.ReadAll(v)
        require.NoError(t, err)
        return string(data)
    default:
        res, err := json.Marshal(v)
        require.NoError(t, err)
        return string(res)
    }
}

And the test for createUser:

func (s *TestSuite) TestCreateUser() {
    requestBody := s.loader.LoadString("fixtures/api/create_user_request.json")

    res, err := s.server.Client().Post(s.server.URL+"/users", "", bytes.NewBufferString(requestBody))
    s.Require().NoError(err)
    defer res.Body.Close()

    s.Require().Equal(http.StatusOK, res.StatusCode)

    response := api.CreateUserResponse{}
    err = json.NewDecoder(res.Body).Decode(&response)
    s.Require().NoError(err)

    expected := s.loader.LoadTemplate("fixtures/api/create_user_response.json.temp", map[string]interface{}{
        "id": response.ID,
    })
    JSONEq(s.T(), expected, response)
}

As a result, the content of our tests has been greatly reduced.

To write new tests, we will only need to mock calls to external services, put data into the database if necessary and describe the request and response structures in separate files, while minimally changing the code of the tests themselves.

Conclusion

As a result, we see that writing integration tests has become quite simple and comparable to writing unit tests. But, there is much more practical benefit from them, since we test our functionality completely, not only individual calls, but also the conversion of entities, working with the database and interacting with the external API.

Co-author: Andrey Lukin

All code samples can be found here

Links:
https://www.sohamkamani.com/golang/options-pattern/

Writing server with http package. Understanding Mux and Handler concept by writing own RESTful Mux.

Georgii Kliukovkin — Tue, 19 Jul 2022 19:07:53 +0000

All examples of code can be found here.

In this article, we will try to understand two crucial Go concepts - mux and Handler. But before we start writing any code let us firstly understand what we need to run a simple web service.

First of all, we need a server itself that will run on some port, listening for the requests and providing responses for those requests.
The next thing we need is a router. This entity is responsible for routing requests to corresponding handlers. In the Go world, servemux is actually an analog of a router.
The last thing we need is a handler. It is responsible for processing a request, executing the business logic of your application, and providing a response for it.

Implementing a Handler Interface

Let’s start our journey with pure Go code, no libraries or frameworks. To run a server, we need to implement Handler interface:

type Handler interface{
   ServeHTTP(ResponseWriter, *Request)
}

To achieve that we need to create an empty struct and provide a method for it:

package main

import (
    "fmt"
    "net/http"
)

type handler struct{} 

func (t *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { 
    fmt.Fprintf(w, "ping %s\n", r.URL.Query().Get("name"))
}

func main() {
    h := &handler{} //1
    http.Handle("/", h) //3
    http.ListenAndServe(":8000", nil) //4

}

We creating an empty struct that implements http.Handler interface
http.Handle will register our handler for a given pattern, in our case, it is “/”. Why pattern and not URI? Because under the hood when your server is running and getting any request - it will find the closest pattern to the request path and dispatch the request to the corresponding handler. That means that if you will try to call 'http://localhost:8000/some/other/path?value=foo' it will still be dispatched to our registered handler, even if it is registered under the “/” pattern.
On the last line with http.ListenAndServe we starting server on the port 8000. Keep in mind the second argument, which is nil for now, but we will consider it in detail in a few moments.

Let us check how it works using curl:

❯ curl "localhost:8000?name=foo"
ping foo

Using handler function

The next example looks very similar to the previous one, but it is a little bit shorter and easier to develop. We don’t need to implement any interface, just to create a function with has a similar signature as Handler.ServeHTTP

package main

import ( 
    "fmt"
    "net/http" 
)

func handler(w http.ResponseWriter, r *http.Request) { 
    fmt.Fprintf(w, "ping %s\n", r.URL.Query().Get("name"))
}

func main() {
    http.HandleFunc("/", handler) 
    http.ListenAndServe(":8000", nil) 
}

Notice that under the hood it is just syntax sugar for avoiding the creation of Handler instances on each pattern. HandleFunc is an adapter that converts it to a struct with serveHTTP method. So instead of this:

type root struct{} 
func (t *root) ServeHTTP(w http.ResponseWriter, r *http.Request) {...}

type home struct{} 
func (t *home) ServeHTTP(w http.ResponseWriter, r *http.Request) {...}

type login struct{} 
func (t *login) ServeHTTP(w http.ResponseWriter, r *http.Request) {...}
//...
http.Handle("/", root)
http.Handle("/home", home)
http.Handle("/login", login)

We can just use this approach:

func root(w http.ResponseWriter, r *http.Request) {...}
func home(w http.ResponseWriter, r *http.Request) {...}
func login(w http.ResponseWriter, r *http.Request) {...}
...
http.HandleFunc("/", root)
http.HandleFunc("/home", home)
http.HandleFunc("/login", login)

Creating own ServeMux

Remember we passed nil to http.ListenAndServe? Well, under the hood http package will use default ServeMux and bind handlers to it with http.Handle and http.HandleFunc . In production, it is not a good pattern to use default serveMux because it is a global variable thus any package can access it and register a new router or something worse. So let’s create our own serveMux. To do this we will use http.NewServeMux function. It returns an instance of ServeMux which also got Handle and HandleFunc methods.

mux := http.NewServeMux()
mux.HandleFunc("/", handlerFunc)
http.ListenAndServe(":8000", mux)

One interesting thing here is that our mux is a Handler too. Signature of http.ListenAndServe waiting for a Handler as a second argument and after receiving a request our HTTP server will call serveHTTP method of our mux which in turn call serveHTTP method of registered handlers.

So, it is not obligate to provide a mux from http.NewServeMux() . To understand this let’s create our own instance of a router.

package custom_router

import (
    "fmt"
    "net/http"
)

type router struct{}

func (r *router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    switch req.URL.Path {
    case "/foo":
        fmt.Fprint(w, "here is /foo")
    case "/bar":
        fmt.Fprint(w, "here is /bar")
    case "/baz":
        fmt.Fprint(w, "here is /baz")
    default:
        http.Error(w, "404 Not Found", 404)
    }
}

func main() {
    var r router
    http.ListenAndServe(":8000", &r)
}

And let’s check this:

❯ curl "localhost:8000/foo"
here is /foo

Keep in mind that servemux, provided by Go will process each request in a separate goroutine. You may try to implement a custom router acting like a Go mux using goroutines per request.

Creating own server

What will happen if a customer will call our endpoint which needs to get info from the DB, but DB is not responding for a long time? Will the customer wait for the response? If so, probably it is not so user-friendly API. So, to avoid this situation and provide a response after some time of waiting we may instantiate http.Server, which has ListenAndServe function. In production, we often need to tune our server, e.g. provide a non-standard logger or set timeouts.

srv := &http.Server{
   Addr:":8000",
   Handler: mux,
     // ReadTimeout is the maximum duration for reading the entire
   // request, including the body.
     ReadTimeout: 5 * time.Second,
     // WriteTimeout is the maximum duration before timing out
   // writes of the response.
     WriteTimeout: 2 * time.Second,
}
srv.ListenAndServe()

Behaviour of timeouts might seems not obvious. What will happen when the timeout will be exceeded? Will the request be forced to quit or responded to? Let’s create a handler that will sleep for 2 seconds and take a look at how our server will behave with WriteTimeout for 1 second.

func handler(w http.ResponseWriter, r *http.Request) {
    time.Sleep(2 * time.Second)
    fmt.Fprintf(w, "ping %s\n", r.URL.Query().Get("name"))
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/", handler)
    srv := &http.Server{
        Addr:    ":8000",
        Handler: mux,
        WriteTimeout: 1 * time.Second,
    }
    srv.ListenAndServe()
}

Now let’s call curl with the help of the time util to measure how much time will take our request.

❯ time curl "http://localhost:8000?name=foo"
curl: (52) Empty reply from server
curl "http://localhost:8000?name=foo"  0.00s user 0.01s system 0% cpu 2.022 total

We see no reply from the server and request took 2 seconds instead of 1. This is because timeouts are just a mechanism that restricts specific actions after timeout was exceeded. In our case, writing anything to the response was restricted after 1 second has passed. I provided 2 links to awesome articles at the end.

And still, we have an open question: how to force our handler to quit after some period of time?

To achieve that we can simply use http method TimeoutHandler:

func TimeoutHandler(h Handler, dt time.Duration, msg string) Handler

Let’s rewrite our example with timeout handler. Don’t forget to increase time.sleep and timeout for +1 sec each, otherwise there will be still no response:

func handler(w http.ResponseWriter, r *http.Request) {
    time.Sleep(3 * time.Second)
    fmt.Fprintf(w, "ping %s\n", r.URL.Query().Get("name"))
}

func main() {
    mux := http.NewServeMux()
    mux.Handle("/", http.TimeoutHandler(http.HandlerFunc(handler), time.Second * 1, "Timeout"))
    srv := &http.Server{
        Addr:    ":8000",
        Handler: mux,
        WriteTimeout: 2 * time.Second,
    }
    srv.ListenAndServe()
}

Now our curl works exactly as we expected:

❯ time curl "http://localhost:8000?name=foo"
Timeoutcurl "http://localhost:8000?name=foo"  0.01s user 0.01s system 1% cpu 1.022 total

RESTful routing

Servemux provided by Go hasn’t some convenient way to support HTTP methods. Of course we always can achieve the same result using *http.Request, which contains all required information about the response, including HTTP method:

package main

import"net/http"

func createUser(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "Method Not Allowed", 405)
    }
    w.Write([]byte("New user has been created"))
}

func main() {
   mux := http.NewServeMux()
   mux.HandleFunc("/users", createUser)
   http.ListenAndServe(":3000", mux)
}

Custom RESTful router

Now let’s try to make something more interesting. We need our router to be able to register handlers using handleFunc method which will accept 3 parameters:

method string,
pattern string,
f func(w http.ResponseWriter, req *http.Request)

To achieve that we need to write a small bunch of code :)

Let’s start with types. First we need a router itself. It should have a map that will store registered URL pattern(e.g. /users ) and all information (or rules) that we want to apply to it:

type urlPattern string

type router struct {
    routes map[urlPattern]routeRules
}

func New() *router {
    return &router{routes: make(map[urlPattern]routeRules)}
}

Next let’s define what is a routeRules. In case of REST we want to store registered HTTP methods and related handlers:

type httpMethod string

type routeRules struct {
    methods map[httpMethod]http.Handler
}

Now we want our router to has a HandleFunc method:

/*
    method - string, e.g. POST, GET, PUT
    pattern - URL path for which we want to register a handler
    f - handler 
*/
func (r *router) HandleFunc(method httpMethod, pattern urlPattern, f func(w http.ResponseWriter, req *http.Request)) {
    rules, exists := r.routes[pattern]
    if !exists {
        rules = routeRules{methods: make(map[httpMethod]http.Handler)}
        r.routes[pattern] = rules
    }
    rules.methods[method] = http.HandlerFunc(f)
}

The last thing we need is that our router should implement Handler interface. So we need to implement ServeHTTP(w http.ResponseWriter, req *http.Request) method:

func (r *router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    // first we will try to find a registered URL pattern
    foundPattern, exists := r.routes[urlPattern(req.URL.Path)]
    if !exists {
        http.NotFound(w, req)
        return
    }
    // next we will try to check if such HTTP method was registered
    handler, exists := foundPattern.methods[httpMethod(req.Method)]
    if !exists {
        notAllowed(w, req, foundPattern)
        return
    }
    // finally we will call registered handler
    handler.ServeHTTP(w, req)
}

// small helper method
func notAllowed(w http.ResponseWriter, req *http.Request, r routeRules) {
    methods := make([]string, 1)
    for k := range r.methods {
        methods = append(methods, string(k))
    }
    w.Header().Set("Allow", strings.Join(methods, " "))
    http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
}

And that’s it. Now let’s register simple handler:

func handler(w http.ResponseWriter, req *http.Request) {
    w.Write([]byte("hello"))
}

func main() {
    r := New()
    r.HandleFunc(http.MethodGet, "/test", handler)
    http.ListenAndServe(":8000", r)
}

And check how it works:

❯ curl -X GET -i "http://localhost:8000/test"
HTTP/1.1 200 OK
Date: Wed, 13 Jul 2022 14:24:43 GMT
Content-Length: 5
Content-Type: text/plain; charset=utf-8

hello

❯ curl -X POST -i "http://localhost:8000/test"
HTTP/1.1 405 Method Not Allowed
Allow: GET
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Wed, 13 Jul 2022 14:24:14 GMT
Content-Length: 19

Method Not Allowed

Awesome. We just wrote a router that may easily register a handler for specific HTTP method. The full code looks like:

package main

import (
    "net/http"
    "strings"
)

type httpMethod string
type urlPattern string

type routeRules struct {
    methods map[httpMethod]http.Handler
}

type router struct {
    routes map[urlPattern]routeRules
}

func (r *router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    foundRoute, exists := r.routes[urlPattern(req.URL.Path)]
    if !exists {
        http.NotFound(w, req)
        return
    }
    handler, exists := foundRoute.methods[httpMethod(req.Method)]
    if !exists {
        notAllowed(w, req, foundRoute)
        return
    }
    handler.ServeHTTP(w, req)
}

func (r *router) HandleFunc(method httpMethod, pattern urlPattern, f func(w http.ResponseWriter, req *http.Request)) {
    rules, exists := r.routes[pattern]
    if !exists {
        rules = routeRules{methods: make(map[httpMethod]http.Handler)}
        r.routes[pattern] = rules
    }
    rules.methods[method] = http.HandlerFunc(f)
}

func notAllowed(w http.ResponseWriter, req *http.Request, r routeRules) {
    methods := make([]string, 1)
    for k := range r.methods {
        methods = append(methods, string(k))
    }
    w.Header().Set("Allow", strings.Join(methods, " "))
    http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
}

func New() *router {
    return &router{routes: make(map[urlPattern]routeRules)}
}

func handler(w http.ResponseWriter, req *http.Request) {
    w.Write([]byte("hello"))
}

func main() {
    r := New()
    r.HandleFunc(http.MethodGet, "/test", handler)
    http.ListenAndServe(":8000", r)
}

Why not use it in prod? Well, because there are several libraries that offer you such features and a bunch more! Restrictions by host header, handling paths with path parameters, query parameters, pattern matching(we implement only exact equals), and a lot more.

Gorilla mux

One of the most popular libraries for that is Gorilla/mux.

❯ go get "github.com/gorilla/mux"

Here is a simple example of a GET handler.

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello World!")
}

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/test", handler).Methods("GET")
    http.ListenAndServe(":8000", r)
}

Let’s check this endpoint and see the result:

❯ curl -X GET "http://localhost:8000/test"
Hello World!

And if we try to send the POST method we will get 405:

❯ curl -X POST -i "http://localhost:8000/test"
HTTP/1.1 405 Method Not Allowed
Date: Wed, 13 Jul 2022 12:54:22 GMT
Content-Length: 0

Gorilla mux:

https://github.com/gorilla/mux

Timeouts articles:

https://ieftimov.com/posts/make-resilient-golang-net-http-servers-using-timeouts-deadlines-context-cancellation/

https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/