DEV Community: joaosczip

Scale-up and Scale-out

joaosczip — Mon, 29 Apr 2024 13:22:01 +0000

Does your application fit your users' needs?

The problem

A common problem that companies are facing today, especially the big ones, is the growing number of concurrent users using their applications. If their systems are not prepared to handle all the requests they are receiving, it can be very problematic for the company as a whole.

Imagine a company whose business is an e-commerce platform, black-friday is coming and the engineering team hasn’t taken any actions to ensure the platform's availability and resilience on that day. When the day arrives, the number of simultaneous access has grown in a way that even the most optimistic executive wouldn’t imagine. What a great news! Not so great though… the platform becomes unavailable, and users are unable to even access it. Buying things? Not a change!

How harmful for the company this situation can be? There are a lot of e-commerce platforms waiting for black-friday every year. Sometimes they already forecasted the GMV for this day, and are only expecting Monday morning to collect the results of the whole weekend to celebrate.

Now, on the other hand, how this situation could have been if the business had an agreement with the engineering team? How the engineering team could help in this situation? How the whole platform could be prepared to face this highly growing peak?

About scaling

During peak loads, as the number of simultaneous access is growing on your system, the application needs to keep working as it was before, in a way that the users don’t end up being harmed by it.

Scaling is a process in which your software adapts to your users’ needs, growing and shrinking whenever it’s necessary. This process is responsible for responding to the different types of loads, ensuring its availability and resilience to keep working as expected by your users. It must be as transparent as possible, ensuring that not a single user will be impacted by it.

There are a lot of problems that can arise when the number of users accessing your application is high, and I’ll explain them below.

In simple terms, your system can operate in two different loads: normal and high, and it must be prepared to face both of them. Figure 1 demonstrates the differences between these loads.

In simple terms:

Normal load: the number of concurrent users using your system is under normal usage, it’s receiving a known number of requests and as a consequence, using nothing but an expected number of resources from your servers.
High load: the number of concurrent users is above normal, it’s receiving a lot of simultaneous requests, and as a consequence, using a lot of resources from the servers.

Given that context, in which case your application can be more harmful to the company if it isn’t capable of handling the situation as expected? In general, if you guess the last one, you’re right! However, the first one can be quite harmful if you aren’t prepared.

We have already explained above why not being prepared to face the high load situation may be very stressful. But why the system must be prepared to face the normal load as well? Imagine a scenario where your system has a few concurrent users, receiving nothing but a few requests, but your servers are prepared to handle a lot of requests, and you have provisioned a great amount of CPU and RAM to it. This CPU and RAM will cost you, especially if you are running your infrastructure on the cloud.

Provisioning many resources (CPU and RAM) when your application is under low-normal usage will end up being too expensive for the company.

With that being said, how to be prepared for both scenarios? Not underusing your resources and being capable of staying available?

There are two types of scaling to help you with that, and you will decide which to use depending on your application needs. They are vertical scaling and horizontal scaling.

Vertical scaling

Imagine a scenario where your application is running on a single server machine, with 4 GB of RAM and 2 CPU Cores, and it is under normal usage. But suddenly the number of users starts increasing, you are under a high load now and your application starts becoming slow and unavailable.

In that case, you have decided to upgrade your server resources. You add 4 more GB of RAM, and 2 more CPU Cores, now your machine has a total of 8 GB of RAM and 4 CPU Cores. With this upgrade your application becomes more stable, it isn’t presenting slowness and it’s more available now. The users are pretty happy.

This process of upgrading your machine (or choosing a more robust one) is called Vertical Scaling, or Scale-up. Figure 2 illustrates this process.

This approach can help in some cases, but it is definitely not suited for all situations.

The most common usage of Vertical Scaling is the situation where your load is more predictable. Where almost all of the time your load operates on a plateau and it does not change (increase or decrease) for a long period. And this is true because the process of scaling up your server isn’t quick. Your users will face a period of downtime while you are upgrading your resources.

In an on-premise situation, there isn’t a way to upgrade the server resources without shutting it down before. Or even in a cloud environment, if you change your VM, the users won’t be able to access it until the new machine is up and running with your application.

Pros

Implementation: As explained above, when scaling up your server on an on-premise environment, the process is upgrading the machine resources, using more CPU and RAM should do the trick. When handling this situation in a cloud environment, you can just choose a more robust VM.
Management: Running it on the cloud or on-premise, you’ll just need to manage a single machine. It’s easier than managing multiple machines (this will be explained below in more detail).
Costs: Having a single machine running is cheaper than having 5 or 10 for example.

Cons

Single point of failure: As you have just a single server running your application, what will happen if your server goes down? Besides planning the scaling, you must be prepared to face downtimes either.
Hardware limitations: There’s a hardware limit that each machine can run, you cannot add resources infinitely to your server. This limit can be a bottleneck if the application usage keeps growing.
Downtimes: When upgrading your server, the application will be off until the scaling process is finished. During this time, your users won’t be able to use your software. You must ensure that scaling your application won’t affect your current users.
Costs: In some cases, you’ll just need 10% more resources than your server currently has, however, depending on which cloud provider you are running your app, adding this 10% additional hardware can end up costing 50% more at the end of the month.

Horizontal Scaling

In contrast to the Vertical Scaling approach where more resources are added to a single machine, in the Horizontal Scaling (or Scale-out) approach, more machines are added to handle the application’s traffic.

Getting back to our black-friday example, the moment that the concurrent users start growing, copies of the same server are added and the traffic that was previously redirected to a single instance now is routed to each one of the new server instances. These copies contain the same specs as the original one, and now your application is prepared to handle an increased load.

When using this approach, normally the servers will have low hardware resources, instead of 8 GB of RAM, it’ll probably be less than 512 MB, for example. This happens because rather than running your application on a VM, you’ll use containers alongside a container management system like Kubernetes or AWS ECS. Containers are way smaller than VM instances, and they are built to scale horizontally, e.g. running a lot of instances.

Figure 3 illustrates the process of scaling out your system where the number of concurrent users is growing.

This process is all abstracted by the management system you choose. For Kubernetes for example, the HPA will take care of scaling your instances.

But adding more instances isn’t enough to fully scale your system horizontally, it also needs to know how to properly balance its load between all these instances.

When you have a single instance running it’s easier because all the traffic is routed to it, however, how to route the traffic when you have multiple instances running at the same time? R: Load Balancer.

Load balancing is the process of routing the requests to different application instances, this is done by a Load Balancer that sits in front of your instances and receives all the requests sent to your application. When receiving these requests, the Load Balancer knows to which instance redirects the requests.

There are a lot of different load balancing algorithms that your Load Balance can use, these algorithms not just route the requests, but they also take care to not overload a single container with too many requests, they try to balance the load as evenly as possible.

In Figure 4 below there’s an illustration of a Load Balancer sitting in front of the containers to receive and redirect its load.

The Horizontal Scaling approach is more suitable for cases where your load isn’t predictable, that is, it can increase and decrease many times during a single day.

This type of scaling allows more flexibility because it knows how to scale and shrink when necessary. You can configure different scaling thresholds, like RAM and CPU usage, HTTP metrics, or even asynchronous events like AWS SQS messages.

For example, you have set up your scaling based on HTTP metrics, once your app starts receiving a lot of concurrent HTTP requests, your management systems know that it needs to provision other instances to handle all the traffic. When the load decreases, the provisioned instances are killed to avoid unnecessary additional costs.

Pros

Redundancy: As you have multiple instances of your server running, if one of them fails, or gets killed for whatever reason, you’ll always have another instance to handle the traffic.
No downtime: Even if you have just a single instance running, if it fails, your management system will know it and will provision another instance for you. This new instance will start receiving the requests from the one that just died.
Fewer hardware limitations: The hardware limitations are smaller, but it does not mean that it doesn’t exist. When scaling out your containers, you can scale it as much as you need once the required resources are available within your cluster. The limitations are in the cluster, as bigger your cluster is, the fewer limitations you’ll face.
Performance: You can increase your system's parallel processing power by adding more instances to it.

Cons

Management complexity: Rather than managing a single instance, now you have N instances to manage. Usually, your container management system will do it for you, but it does not mean you won’t need to know how to use it. Depending on your scenario, company size, and some other factors, you’ll probably need someone specialized in those systems.
Costs forecasting: It’s harder to forecast the costs without knowing how much hardware you’ll use within a month. Is not impossible, but it’s trickier than when managing a single server.

Conclusion

As the software applications are evolving, the users are becoming more demanding about the products they are using. If your application does not fit your user’s needs, it’s more likely for them to choose a different vendor.

When projecting your software, the non-functional requirements are as important as the functional ones. Users start using your app for what it provides for them but stop using it for what it doesn’t.

Choose a Vertical Scaling approach when your app usage is more likely to remain the same almost all the time and a Horizontal Scaling when it is not predictable and is most likely to vary often.

Now that you know how to approach scaling, does your application fit your users’ needs?

Resources

What I learned in the Real Python testing courses

joaosczip — Thu, 01 Feb 2024 10:41:06 +0000

I recently completed the three main courses on automated testing from the Real Python platform. As I already use these tools in my daily work, I took the courses with the goal of understanding, in general, if there are better ways to write tests than how I do daily.

Below, you will find my overview of the three courses I completed:

Testing your code with pytest

pytest is a framework for writing automated tests in Python that emerged as a counter to the unittest framework, which is part of the language's standard library.

Unlike the latter, which has a very verbose approach where every test case must extend from a test class, pytest brings much greater simplicity, aligning much more closely with the concept of simplicity imposed by the language.

Writing tests

Below is an example of writing a test using unittest vs pytest.

def say_hello(to):
    return f'Hello, {to}'

# unittest
import unittest

class TestMyFunc(unittest.TestCase):
    def test_say_hello(self):
        self.assertEqual(say_hello('World'), 'Hello, World')

# pytest
def test_say_hello():
    assert say_hello('World') == 'Hello, World'

With the example above, it's possible to observe the simplicity of pytest in comparison to its alternative. We can write a test just by declaring a function with the test_ prefix and use the reserved word assert to validate that the comparison is true, making the test pass.

Conversely, it's also possible to do negative conditions just by changing the sign of the comparison:

def say_hello(to):
    return f'Hello, {to}'

# pytest
def test_say_hello():
    assert say_hello('World') != 'Hello! World'

We can perform comparisons between different objects, such as lists, tuples, dictionaries, etc.

def test_list_equal():
    assert [1, 2, 3] == [1, 2, 3]

def test_tuple_equal():
    assert (1, 2, 3) == (1, 2, 3)

def test_dict_equal():
    assert {'a': 1, 'b': 2} == {'a': 1, 'b': 2}

In summary, writing tests in pytest works as follows:

Write a function with the test_ prefix;
Use the word assert to ensure that the comparison is returning True.

It's that simple!

Using fixtures

In many cases, we want to write tests based on the same input argument. For example, given a list, we want to test different situations involving it.

One way would be to define this list in all the tests and use it, as in the examples below.

def test_list_pop():
    my_list = [1, 2, 3]
    my_list.pop(1)
    assert my_list == [1, 3]

def test_list_append():
    my_list = [1, 2, 3]
    my_list.append(4)
    assert my_list == [1, 2, 3, 4]

def test_list_reverse():
    my_list = [1, 2, 3]
    my_list.reverse()
    assert my_list == [3, 2, 1]

Although it works, writing tests in this manner brings some problems, especially related to difficulties in maintenance and evolution, since for each new test it will be necessary to define the same object. Besides, if there is any change in the object we are testing, it will need to be changed in all tests, leading to poor maintainability.

pytest provides the possibility to define fixtures that will be reused among tests, in order to solve the problems highlighted above. Using fixtures, we can define any type of object we want to reuse during the tests, making any changes to this object be made in just one place.

To do this, simply use the @pytest.fixture decorator and define the object we will work with.

import pytest

@pytest.fixture
def my_list():
    return [1, 2, 3]

def test_list_pop(my_list):
    my_list.pop(1)
    assert my_list == [1, 3]

def test_list_append(my_list):
    my_list.append(4)
    assert my_list == [1, 2, 3, 4]

def test_list_reverse(my_list):
    my_list.reverse()
    assert my_list == [3, 2, 1]

To declare a new fixture, simply add the decorator right above the declaration of a function that returns the desired object. Once declared, we need to provide the name of the fixture in the tests where we want to use it, and pytest takes care of loading it within the tests.

In addition to being able to define custom fixtures, pytest also provides a list of built-in fixtures that can be consulted here.

Parameterizing tests

Depending on the function you want to test, it's beneficial to test it with various different parameters in order to stress it as much as possible and ensure that it won't have side effects.

If, for example, we wanted to test a function that performs subtraction between two numbers, we could follow the approach below.

def subtract(a, b):
    return a - b

def test_subtract_2_and_1():
    assert subtract(2, 1) == 1

def test_subtract_5_and_2():
    assert subtract(5, 2) == 3

def test_subtract_10_and_5():
    assert subtract(10, 5) == 5

Although this approach works, we end up returning to the problem of maintainability, because working in this way makes our tests more difficult to maintain, since any change in the signature or return of the function would result in changes in more than one test, and consequently in a greater effort of refactoring.

To solve this problem, pytest provides us with another decorator responsible for supplying various parameters to the same test. Using this decorator, we can specify what the input parameters are and what output the function will have for the provided parameters.

def subtract(a, b):
    return a - b

import pytest

@pytest.mark.parametrize("param_a,param_b,expected_result", [
    (2, 1, 1),
    (5, 2, 3),
    (10, 5, 5),
])
def test_subtract(param_a, param_b, expected_result):
    assert subtract(param_a, param_b) == expected_result

The @pytest.mark.parametrize decorator takes two arguments:

A string representing the parameters that will be passed to the test function.
A list that represents each of the parameters that will be provided in each iteration of the test. When there is more than one parameter, as in the example above, we use a tuple to distinguish them within the list.

Course conclusion

The course is a good introductory content, as it covers all the fundamental topics of the tool. By the end of this course, the developer should be able to write their first tests using the framework.

Due to pytest being quite simple to use, and also because its documentation is very comprehensive, a course cannot stand out in such a way that it seems indispensable for learning. However, I missed some topics such as:

Installation and setup of pytest, including the configuration file and its parameters;
Test coverage collection;
Debugging;
Best practices for test writing;
Other commands and functionalities of the tool.

Generally speaking, I would say that you can learn everything the course shows just through the framework's documentation. Since it is a very simple tool to learn, the course could very well have covered the topics I mentioned above to ensure that the student not only knows how to use the tool itself but also how to write well-written and useful tests.

Improve Your Tests With the Python Mock Object Library

When working with automated tests, often it's necessary to "simulate" the behavior of some object or dependency so that the test can be implemented. This simulation occurs mainly in cases where your Subject Under Test (SUT) has external dependencies, such as HTTP calls, database interactions, etc.

In testing, we call this simulation a Mock. We mock a certain object when we can specify what its behavior will be during that specific test, determining the results of calls, the parameters used to call it, etc.

To clarify the necessity of this procedure, I provide an example of a method that needs to perform a GET request to another API. In this case, our test should not perform the call itself, as doing so would make it heavily dependent on the quality of the connection and the API being called. This can bring problems since if the API is not functioning correctly or is not available at the time of executing the test, it will fail. There's also the issue related to the requests themselves, as a large test load can impact the performance of the API operating in a production environment, bringing unexpected effects.

For these and other reasons, we use a Mock to simulate the behavior of the function making this GET call so that it does not make the call itself but only returns a predetermined result that makes sense for our test.

The Mock object

The Mock object is imported and used through the language's standard library (from version 3.3 onwards, for earlier versions it needs to be installed as a dependency), and it has a documentation that is very clear and comprehensive on how to use it.

The Mock class is quite flexible; once instantiated, we can make calls to methods that don't exist, and these methods will be attributed to the instance. Through this assignment, it is possible to carry out all the mocking operations we wish.

One of the functions of Mock is to take on the role of a real instance (production class or method) and make them return specific results, as in the example below.

from unittest.mock import Mock

class MyProductionClass:
    def some_method(self):
        pass

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.some_method.return_value = "it's not the original return"

print(my_instance.some_method() == "it's not the original return")

In addition to specifying the return of methods, it's also possible to perform asserts on the calls to the method in question.

from unittest.mock import Mock

class MyProductionClass:
    def some_method(self, a, b, c):
        return a + b + c

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.some_method.return_value = 15

assert my_instance.some_method(10, 15, 20) == 15
my_instance.some_method.assert_called_once()
my_instance.some_method.assert_called_once_with(10, 15, 20)

Mock has a list of methods that can be used to ensure that the calls were made in the way we want.

Every object to which a Mock instance is attributed inherits all its methods. Therefore, as demonstrated in the examples above, my_instance becomes an instance of Mock and no longer of MyProductionClass. When listing the methods of the my_instance instance, the result is as follows:

print(dir(my_instance))
"""
[
    'assert_any_call', 'assert_called', 'assert_called_once', 
    'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 
    'assert_not_called', 'attach_mock', 'call_args', 'call_args_list', 
    'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 
    'mock_calls', 'reset_mock', 'return_value', 'side_effect', 'some_method'
]
"""

The returned list are the methods that can be used from my_instance to test the behavior of the instance in question.

In addition to returning values, in some cases, we'll want to cover failure scenarios, or rather, scenarios of side effects. It's possible to throw exceptions to simulate these behaviors through the Mock object as well.

Returning to our example of a GET request, when we have an HTTP call, various errors can happen, such as timeout, service unavailable, etc. We can simulate a timeout in our GET call through the Mock object.

For example, we want to make a request to fetch all the holidays for the year 2024, and for this, we will use an API that returns all the holidays for a given year. We can force a timeout error once we mock the requests object.

import requests
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays_timeout():
    requests.get.side_effect = Timeout("too much time to complete")

    with pytest.raises(Timeout):
        get_holidays()

In this way, we simulate an undesirable behavior of the method and can work on that, without actually making the HTTP call.

Continuing with our example of the HTTP call, we can use the mock to return more complex objects, such as the response from this call.

import requests
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays():
    mocked_response = Mock()
    mocked_response.status_code = 200
    mocked_response.json.return_value = [{'Christmas': '25/12'}]
    requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

First, we instantiate a simulated response, mocked_response, and use the flexibility of the Mock class to specify the desired return.

The side_effect method is not only used to throw exceptions but also for cases where we make the call to the same method more than once and receive different results, or even the same result.

In the example below, the first request returned a timeout, due to the error we try again, and in the second request, we get the expected result.

import requests
from requests.exceptions import Timeout
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays_retry():
    mocked_response = Mock()
    mocked_response.status_code = 200
    mocked_response.json.return_value = [{'Christmas': '25/12'}]

    requests.get.side_effect = [Timeout, mocked_response]

    with pytest.raises(Timeout):
        get_holidays()

    resp = get_holidays()
    assert resp.json() == [{'Christmas': '25/12'}]

In the first call, a timeout occurs and the exception is thrown. The second call is successful, and we get the expected return.

This behavior can also be assigned at the moment of instantiating Mock, by providing the desired returns directly in its constructor.

def test_get_holidays_retry():
    mocked_response = Mock(status_code=200, json=Mock(return_value=[{'Christmas': '25/12'}]))
    requests = Mock(get=Mock(side_effect=[Timeout, mocked_response]))

    with pytest.raises(Timeout):
        get_holidays()

    resp = get_holidays()
    assert resp.json() == [{'Christmas': '25/12'}]

Patching

Assigning a Mock instance to the object whose behavior we wish to simulate works well in cases where the test module has access to the same dependencies as the production module. However, in many cases, both modules will be separated, and the test will hardly have access to the same instances that the production class has.

Therefore, we can use a technique called patching so that the Mock of the dependency is injected directly into our test.

In the above examples of the GET call, both the get_holidays function and its test test_get_holidays are in the same file, so the test has access to the same instance of requests that the function has. If this were not true, it would be necessary to use patching.

Patching can be used as a decorator or a context manager. Below are examples using both approaches.

# holidays.py
import requests

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

# test_holidays.py
from unittest import mock

@mock.patch('holidays.requests') # using as a decorator
def test_get_holidays(mocked_requests: mock.Mock):
    mocked_response = mock.Mock(
            status_code=200, 
            json=mock.Mock(return_value=[{'Christmas': '25/12'}]),
        )
    mocked_requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

When used as a decorator, the object will be injected into the list of arguments of the test function. This object will be a Mock instance, and all mocking operations can be performed normally from it.

# test_holidays.py
from unittest import mock

def test_get_holidays():
    mocked_response = mock.Mock(
            status_code=200, 
            json=mock.Mock(return_value=[{'Christmas': '25/12'}]),
        )

        # using as a context a manager  
        with mock.patch('holidays.requests') as mocked_requests:
          mocked_requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

Using it as a context manager is quite similar, with the major difference being that the instance is created within the test, instead of being injected.

In addition to mocking the complete object, we can mock just the method we want to test, through patch.object. In the example below, instead of mocking the entire requests object, we do this just for its get method.

from unittest import mock

@mock.patch.object(requests, 'get')
def test_get_holidays(mocked_get: mock.Mock):
    mocked_response = Mock(status_code=200, json=Mock(return_value=[{'Christmas': '25/12'}]))
    mocked_get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

This option is very useful when we want to maintain the behavior of the rest of the class, simulating the behavior of only specific methods.

Autospeccing

As we've seen, the Mock object is quite flexible, capable of "assuming the identity" of different types of objects. With this flexibility, some pitfalls can occur, such as changing the behavior of a non-existent method of the "mocked" class.

from unittest.mock import Mock

class MyProductionClass:
    def sum_2(self, num):
        return num + 2

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.sum2.return_value = 5

assert my_instance.sum2(3) == 5

In the example above, MyProductionClass has the sum_2 method, but when mocking, there was a typo, and the method used is a non-existent method in the class. However, due to the flexible nature of Mock, this operation was allowed, and even though the assert is returning true, the test is not simulating the behavior correctly because it is being used in a method that does not exist in the original class.

To prevent this type of problem, we use autospeccing, which is nothing more than forcing the Mock instance to follow the specification of its real object.

There are different ways to use it, one of which is represented in the example below, where we specify which specification (spec) the Mock should follow. In this case, it's the specification of the MyProductionClass class.

from unittest.mock import Mock

class MyProductionClass:
    def sum_2(self, num):
        return num + 2

my_instance = MyProductionClass()
my_instance = Mock(spec=MyProductionClass) # speccing the MyProductionClass
my_instance.sum2.return_value = 5

assert my_instance.sum2(3) == 5

"""
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'sum2'. Did you mean: 'sum_2'?
"""

In this case, an AttributeError is raised, specifying that the class does not have the sum2 method, but rather the sum_2, which is the true method.

The different ways to use autospeccing can be found in the documentation. This is a very useful way to prevent various errors during test writing.

Course conclusion

It's a much more extensive course than the first one, covering practically all topics related to the Mock object. Although all the course content can also be found in the documentation, I still found the course quite interesting for synthesizing various topics in a way that facilitates the understanding of the student.

However, I still missed the following two topics:

MagicMock
AsyncMock

It's a great course for those who are starting to write tests in Python.

Test-Driven Development With pytest

This was the course I was most excited about, as I work with TDD daily, and unfortunately, it was my biggest disappointment.

I won't dwell on the course content because TDD was never covered. The course is basically the instructor writing the test and then fully implementing the methods of the class without concern for the red-green-refactor cycle and other TDD-related practices. The content is quite shallow and covers pytest only superficially.

Although it is very short and you can finish it in just a few hours in front of the computer, it is not worth it if you are interested in learning only TDD because it is not covered here.

Overall Conclusion

The first two courses I presented are quite interesting for starting with automated testing in Python, especially the second one, which is quite comprehensive and covers a standard library of the language.

With just the first two courses, you can already navigate on your own in learning and improving your testing skills. Although they are good content, you can't rely solely on them; you need to research more and continue to specialize to become a good test writer.

As I mentioned earlier, the courses are quite technical, presenting the tools and how to use them. Although I felt the lack of topics on how to write good tests, I understand that it was not their intention, as their goal was to simply introduce the frameworks and not teach how to write good tests.

Regarding the last course, I was greatly disappointed. Not only because the content is weak, but also because people believe that TDD is just writing tests and then the complete production code, instead of the creative process that I believe it should be.

Apache Kafka: What is and how it works

joaosczip — Tue, 02 Mar 2021 02:33:10 +0000

Introduction

When we start to study microservices, many different concepts, patterns, protocols, and tools are presented to us, like Messaging, AMQP, RabbitMQ, Event-sourcing, gRPC, CQRS, and many others. Among so many different words, there is one that always caught my attention and I always wanted to start studying it, the Apache Kafka.

The problem is that people always have a misunderstanding about what Kafka really is and is very common to see people talking about it like it as if it were just another messaging system. In fact, you can do that, but use Kafka only for that may be a waste of resources.

For that, my goal with this article is to explain how Kafka works and why you may need to use it in your projects. I’ll start presenting the principal concepts, explain why it’s not just a messaging system, I hope that it may help people that are entering this world, as I’m.

What is Apache Kafka?

If you enter the Kafka website, you’ll find the definition of it right on the first page:

A distributed streaming platform

What is an “A distributed streaming platform”? First, we need to define what is a stream. For that, I have a definition that made me really understand it: Streams are just infinite data, data that never end. It just keeping arriving, and you can process it in real-time.

And distributed? Distributed means that Kafka works in a cluster, each node in the cluster is called Broker. Those brokers are just servers executing a copy of apache Kafka.

So, basically, Kafka is a set of machines working together to be able to handle and process real-time infinite data.

His distributed architecture is one of the reasons that made Kafka so popular. The Brokers is what makes it so resilient, reliable, scalable, and fault-tolerant. That’s why Kafka is so performer and secure.

But why there is this misconception that Kafka is just another messaging system? To respond to that answer, we need first to explain how messaging works.

Messaging

Messaging, very briefly, it’s just the act to send a message from one place to another. It has three principal actors:

Producer: Who produces and send the messages to one or more queues;
Queue: A buffer data structure, that receives (from the producers) and delivers messages (to the consumers) in a FIFO (First-In-First-Out) way. When a message is delivered, it’s removed forever from the queue, there’s no chance to get it back;
Consumer: Who is subscribed to one or more queues and receives their messages when published.

And that is it, this is how the messaging works (very briefly, there’s a lot more). As you can see there’s nothing about streams, real-time, cluster (depending on what tool you chose, you can use a cluster too, but it’s not native, like Kafka).

Kafka Architecture

Now that we know how messaging works, let’s get dive into the Kafka world. In Kafka, we also have the Producers and Consumers, they work in a very similar way that they work in the messaging, both producing and consuming messages.

https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html

As you can see it is very similar to what we’ve discussed about messaging, but here we don’t have the Queue concept, instead, we have the concept of Topics.

The Topic is a very specific type of data stream, it’s very similar to a Queue, it receives and delivers messages as well, but there are some concepts that we need to understand about topics:

A topic is divided into partitions, each topic can have one or more partitions and we need to specify that number when we're creating the topic. You can imagine the topic as a folder in the operation system and each folder inside her as a partition;
If we don’t give any key to a message when we’re producing it, by default, the producers will send the message in a round-robin way, each partition will receive a message (even if they are sent by the same producer). Because of that, we aren’t able to guarantee the delivery order at the partition level, if we want to send a message always to the same partition, we need to give a key to our messages. This will ensure that the message will always be sent to the same partition;
Each message will be stored in the broker disk and will receive an offset (unique identifier). This offset is unique at the partition level, each partition has its owns offsets. That is one more reason that makes Kafka so special, it stores the messages in the disk (like a database, and in fact, Kafka is a database too) to be able to recover them later if necessary. Different from a messaging system, that the message is deleted after being consumed;
The producers use the offset to read the messages, they read from the oldest to the newest. In case of consumer failure, when it recovers, will start reading from the last offset.

The partitions of a topic — https://d2h0cx97tjks2p.cloudfront.net/blogs/wp-content/uploads/sites/2/2018/07/Kafka-Topic-Partitions-Layout.png

Brokers

As said before, Kafka works in a distributed way. A Kafka cluster may contain many brokers as needed.

Each broker in a cluster is identified by an ID and contains at least one partition of a topic. To configure the number of the partitions in each broker, we need to configure something called Replication Factor when creating a topic. Let’s say that we have three brokers in our cluster, a topic with three partitions and a Replication Factor of three, in that case, each broker will be responsible for one partition of the topic.

The Replication Factor — https://fullcycle.com.br/apache-kafka-trabalhando-com-mensageria-e-real-time/

As you can see in the above image, Topic_1 has three partitions, each broker is responsible for a partition of the topic, so, the Replication Factor of the Topic_1 is three.

It’s very important that the number of the partitions match the number of the brokers, in this way, each broker will be responsible for a single partition of the topic.

To ensure the reliability of the cluster, Kafka enters with the concept of the Partition Leader. Each partition of a topic in a broker is the leader of the partition and can exist only one leader per partition. The leader is the only one that receives the messages, their replicas will just sync the data (they need to be in-sync to that). It will ensure that even if a broker goes down, his data won’t be lost, because of the replicas.

When a leader goes down, a replica will be automatically elected as a new leader by Zookeeper.

Partition Leader — https://www.educba.com/kafka-replication/

In the above image, Broker 1 is the leader of Partition 1 of Topic 1 and has a replica in Broker 2. Let’s say that Broker 1 dies, when it happens, Zookeeper will detect that change and will make Broker 2 the leader of Partition 1. This is what makes the distributed architecture of Kafka so powerful.

Producers

Just like in the messaging world, Producers in Kafka are the ones who produce and send the messages to the topics.

As said before, the messages are sent in a round-robin way. Ex: Message 01 goes to partition 0 of Topic 1, and message 02 to partition 1 of the same topic. It means that we can’t guarantee that messages produced by the same producer will always be delivered to the same topic. We need to specify a key when sending the message, Kafka will generate a hash based on that key and will know what partition to deliver that message.

That hash takes into consideration the number of the partitions of the topic, that’s why that number cannot be changed when the topic is already created.

When we are working with the concept of messages, there’s something called Acknowledgment (ack). The ack is basically a confirmation that the message was delivered. In Kafka, we can configure this ack when producing the messages. There are three different levels of configuration for that:

ack = 0: When we configure the ack = 0, we’re saying that we don’t want to receive the ack from Kafka. In case of broker failure, the message will be lost;
ack = 1: This is the default configuration, with that we’re saying that we want to receive an ack from the leader of the partition. The data will only be lost if the leader goes down (still there’s a chance);
ack = all: This is the most reliable configuration. We are saying that we want to not only receive a confirmation from the leader but from their replicas as well. This is the most secure configuration since there’s no data loss. Remembering that the replicas need to be in-sync (ISR). If a single replica isn’t, Kafka will wait for the sync to send back de ack.

ack = all — https://static.javatpoint.com/tutorial/kafka/images/apache-kafka-producer5.png

Consumers and Consumers Groups

Consumers are applications subscribed to one or more topics that will read messages from there. They can read from one or more partitions.

When a consumer reads from just one partition, we can ensure the order of the reading, but when a single consumer reads from two or more partitions, it will read in parallel, so, there’s no guarantee of the reading order. A message that came later can be read before another that came earlier, for example.

That’s why we need to be careful when choosing the number of partitions and when producing the messages.

Another important concept of Kafka is the Consumer Groups. It’s really important when we need to scale the messages reading.

It becomes very costly when a single consumer needs to read from many partitions, so, we need o load-balancing this charge between our consumers, this is when the consumer groups enter.

The data from a single topic will be load-balancing between the consumers, with that, we can guarantee that our consumers will be able to handle and process the data.

The ideal is to have the same number of consumers in a group that we have as partitions in a topic, in this way, every consumer read from only one. When adding consumers to a group, you need to be careful, if the number of consumers is greater than the number of partitions, some consumers will not read from any topic and will stay idle.

Consumer Groups — https://docs.cloudera.com/cdp-private-cloud-base/latest/kafka-developing-applications/topics/kafka-develop-groups-fetching.html

Conclusion

Kafka is one of the most completes and performer tools that we have today. After studying it and get their principal concepts, I realized that is much more than just a messaging system. Your distributed architecture and real-time processing are what make it so adaptable to very different scenarios and use cases.

With this article, I tried to dive deep into Kafka, presenting their principal concepts to make it easier to understand and bring their differences from the traditional messaging systems.

Because of the time, I left some important concepts out of this article, like idempotent producers, compression, transactions, etc, but I really hope that this helps you to study and explore more this world.

References

Build a file upload service with NodeJS, Typescript, Clean Architecture and AWS S3

joaosczip — Fri, 29 Jan 2021 16:19:19 +0000

In the old days, before that cloud thing become what it’s today, the way that the programs store their files (images, documents, etc) was a little different from now.

We had our local servers, running our applications locally and any type of file that was uploaded, was also stored in the same server (or not, but still locally) that the application was.

The problem with that architecture is that it can lead to several problems for our server, especially if the amount of the files that will be stored was meaningful (GB of files for example). The two main problems with that are the storage and the security. The storage could leave our server to become slow and the security, in case of loss of files, there’s no comment, especially if those files are confidential.

Client-Server architecture with local servers — https://www.sciencedirect.com/topics/computer-science/client-server-architecture

Today the scenario is a little different, instead of local servers, we have a lot of cloud providers who provide us several different services, including storage services. In this example, I’ll show you how to create an API with NodeJS, Express and Typescript, to handle and upload files to AWS S3 (Simple Storage Service), also using Clean Architecture, IoC, SOLID principles and multer to handle the form-data that we’ll send.

I’m assuming that you already have an AWS account and an IAM user with the right permissions to use S3.

API Structure

Let’s get started from our domain layer. In NodeJS environments, we don’t have the File Interface provided for us in the browser, so we need to create it on our own. So, inside the domain folder, let’s create the models/file.ts

This interface provides all the information that we need to handle and manipulate the incoming files. It’s very similar to the interface from the browsers.

And we also need an interface in our domain to represent the response from the file upload action. In our case, it’ll be very simple, just the path from the object in the storage service.

Now, we can start with our principal use case, the File Upload. Inside domain, let’s create the file usecases/file-upload.ts

Remembering that our use case is just the contract. The implementation will be in the application layer. As you can see, the imports are made with the tsconfig-paths, to be cleaner and flexible to changes.

When I have absolute knowledge of what I need to implement and what I’ll use to do that, I always start with the domain layer, because it’s the core of our service.

Getting started with our concrete implementation, in the application layer, let’s create a class that will implement the FIleUpload interface, this class will be responsible to group the received files from the controller and send them to the infra service that will communicate with the AWS SDK.

Before creating our class, let’s define the protocol for the infra service, we’ll call him FileUploader and place it in the application/protocols/file-uploader.ts

Now, we’ll be able to create our implementation to FileUpload (the use case, not the infra protocol). Our concrete class will be called RemoteFileUpload.

This is a very simple implementation, as you can see, we’re implementing the FileUpload and using tsyringe as IoC container to inject an implementation of FIleUploader (it’s not created yet).

We need to add the decorator @injectable to say that all the dependencies of the class will be injected when the application starts. The @inject decorator receives a token that we want to resolve, in our case, the FileUploader.

Now we need to create the class that will implement the FIleUploader and communicates with the aws-sdk. This class will be in the infra layer, so, let’s create the infra/aws-file-uploader.ts.

This class uses the aws-sdk library (don’t forget to install it) to communicate with the AWS S3 and make the uploads. It can send just a single or a list of files (one per time). Here, the implementation isn’t the most important, pay attention to the architecture details, the way I choose to implement was just what meets my needs.

As you can see, I’m using some config file to store the bucket name and the region, this file uses the env variables that I defined in my docker-compose (you can define in your .env if running locally).

Remember that in the actual version of the sdk there is no more need to fill the aws secret and key when instantiating any service, it’ll look for those values in your environment.

With our application and infra layer completes, it’s time to create the controller that will handle the requests and will call the use case with the files.

The controller also uses the IoC container to receive a FileUpload instance, which in your case will be the RemoteFileUpload. I won’t get deep into the details of the HttpRequest and HttpResponse types, those are just to abstract the Express response and request parameters. The interface that the controller was implementing, have just the method handle, that will receive the request and return a response.

With the controller created, let’s create our route and call our controller inside it. Another thing to notice about the route is that we need to use the multer to handle the files on the request (because we can’t send files on JSON).

Because multer has his own type of File, we created a middleware to map the multer file to our own File interface.

As you can see, we first added the upload middleware from multer and after that, our own fileHandler middleware, so, when passing the request to the controller, the files will be mapped to the format that we expect and inside the body.

Now, the rest is just configuration. We need to configure our service startup, to use express and also the IoC container with the map of contract => implementation.

Remember to put the import of the IoC inside your entrypoint.

I hope that this tutorial will help you clarify something, like handling files with express, IoC container, usage of AWS SDK and files upload.

The goal of this article was to explain how we can create a very robust and flexible architecture even in cases that we need to communicate with external services and/or use third-party libraries. The focus wasn’t on the Clean Architecture itself, but what we can do using the principles behind her and some SOLID principles.

I’ll leave the link to the repository here if you are interested to see how the project got.

Clean Architecture — A little introduction

joaosczip — Thu, 13 Aug 2020 22:05:48 +0000

Introduction

With the constant evolution in the software development process and the growing adoption of different frameworks, is becoming very common developers getting comfortable with the structures provided by these tools and leaving aside some principles of good software development. We need to keep in mind that these tools are just the path and not the end. Our software needs to be more “Domain Oriented Software” and less “Framework Oriented Software”.

That’s not a criticism with the frameworks and libraries adoptions, they really need to be used, there are a lot of wonderful projects that are here to help us, but we should use them in a way that these tools are adapted to our solution, and not the opposed.

Our domain does not need to know what web framework or what database system we are using, these things are just plugins that we may define later.

Thinking about this problem, many solutions have been created and one of these is the “Clean Architecture”, presented by Uncle Bob.

That post will give you a little introduction about Clean Architecture, your main concepts, and a way to implement it, giving us an example of an application build with ReactJS.

Clean Architecture

The main purpose of the Clean Architecture is the Dependency Rule, this rule is all about the direction that our dependencies should point to, that is, always to the high-level policies.

The high-level policies are defined as the core of our application, the components that are independent of any programming language or technology, the policies that only need to change when our domain changes, that is, only in very specific cases.

In contrast, the less specific the component is, the lower the level will be. Here we may put the repositories that connect to our database, the HTTP client that made requests, the presentation layer responsible for the UI, and some components that need to talk with third party libraries for example.

We can see that the low-level policies are responsible for things that aren’t specific to our domain but specific for our application, and the application is just the way we choose to solve our domain problem.

The figure below exemplifies how the Dependency Rule works.

As you may see, all the externals agents point in the same direction, and that pattern may give some benefits to us.

Our entities and use cases don’t have any external world dependency, the only concern that they have is about the domain itself.

So, if we need to change any external agent implementation like an HTTP client, we don’t need to change anything in our use cases, just in the HTTP client concrete class implementation.

That’s another crucial point in the Clean Architecture, all cross-layer communication is made through solid interfaces. Dependency Inversion has a crucial role in that design, as a matter of fact, we only can get the most of Clean Architecture with we know how to implement all the SOLID principles correctly.

Example

Let’s say we have a use case responsible to send a picture to the server. To accomplish that use case, we decided to build a front-end client application with React + Typescript.

We also decided to follow some implementation of the Clean Architecture structure proposal, so, our application will have the following layers:

Domain Layer: The Domain Layer is the most high-level policy that we have, is where we define the entities and the use cases. We don’t have any concrete class in that layer, only interfaces. All the use case implementations will be in the next layer, the Data.

Data Layer: Right behind the Domain Layer, this layer is responsible to implement all the use cases and to define protocols (interfaces) that use case needs. The protocols defined in that layer will be implemented on the next, the Infra.

Infra Layer: That Layer will implement the protocols defined in the Data, normally, those implementations are designed to the external world communications, such as database operations, HTTP requests, third-party libraries, and so on. If we think of a client application, like a React app, in that layer we have the HTTP clients, for example.

Looking with more attention to the last figure, we can see that the Infra Layer belongs to the outer circle, in that circle we do have concrete implementations of libraries and frameworks, it is the most low-level policy that we have, those that aren’t a domain-specific problem.

In that circle, we also have the Presentation Layer, responsible for the UI. Here, in our example, that layer will implement all the React code.

Separating the things in that way, we can see that our core code is completely independent of the library that we choose. We only see the React in one layer, only making what it is designed to do, leaving the business code to other layers. Isolating the app like that, we have many more benefits than coupling all the domain code in our UI.

Implementation

Following our example, first, we need a use case responsible to send images to a server. Let’s call him SendImage.

As you can see, our use case is just an interface that defines a method send and returns a Promise of a DetectedImage (a specific entity to our domain).

Now that we have our use case, we need an implementation for him, let’s create a RemoteSendImage in our Data Layer.

Some important things to notice about that implementation:

In the class constructor, via dependency injection, we are defining that this class needs an implementation of a HttpPostClient, and a URL.
Those dependencies are specific to that implementation. If we have another implementation that does not send the image over HTTP, the dependencies will be another.

Now, that we defined that we need a HttpPostClient protocol, let’s create in the Data Layer too.

Just like our use case, that protocol is just an interface that defines a post method. For that implementation, the axios library was chosen, now we create an AxiosHttpClient concrete class that implements the HttpPostClient.

An important thing to notice about that implementation is his interface. Why we didn’t create a more generic interface, instead of one that only knows POST request (HttpPostClient)?

At that moment, we only need to make POST requests, our use case implementation doesn’t tell not about any GET or PUT requests. So, following the Interface Segregation Principle, we make an extremely cohesive interface, that only knows the method that we need at the moment. If in the future we need to make a GET request for example, we define a protocol for that and make our AxiosHttpClient implement that interface too.

Presentation Layer

Now it’s the time that React comes in. In our Presentation Layer, we will define a functional component that receives a SendImage use case via dependency inversion in their props.

There are some important things to notice here:

A React functional component that only used hooks for your internal logic;
The fact that the component receives an implementation of SendImage in his props is one of the things that make the app structure more flexible;
The first advantage of this structure is that our UI component doesn’t know anything about sending an image to the server, he only needs someone that knows how to the that. Making your only concern about the UI. That’s the Single Responsibility Principle.
Another advantage is that if in the future we decide to send the images over another protocol, we only change the use case implementation, the UI will not even notice that change. With the help of the polymorphism, we can change the implementation easily (Liskov Substitution).

In the end, our folder structure will be like:

Conclusion

Unlike most of the examples that we can find on the internet, this implementation provides a very different approach, where the focus of our UI is only the presentation logic. We were able to create a component completely independent of our domain logic and vice versa.

If we have a use case that needs to send an image to the server, it doesn’t matter if we use a page created in React, Angular, or Vue, our domain is not concerned with that, it is only concerned with the core of the application. Our domain needs to work in the same way regardless of the framework or libraries we are using. If a more delicate exchange is ever needed, we can do it without having to invest a lot of time in it.

From the moment that we have a more flexible and robust architecture, in addition to being strongly abstracted, we are able to observe in practice the benefits of object orientation and its principles being fulfilled. At first, it may seem too much work, and it is indeed, because it is necessary to create many protocols and implementations, however, in the long run, it is a trade-off that is really worthwhile.

Important Links

React course with TDD and SOLID (PT-BR): https://www.udemy.com/course/react-com-mango/
Github repository for this article: https://github.com/joaosczip/clean_react

Applying Open-Closed Principle with Decorator Pattern in Typescript

joaosczip — Thu, 06 Aug 2020 16:17:21 +0000

That's the first time that I'm writing a post in english, so any feedback that you may give to me will be very important. I'll be grateful.

Preface

In the recent days some of my study focus are the SOLID principles, Clean Architecture and some releated topics like Design Patterns, and I need to confess that I was using the OOP techiniques in the wrong way since college, I realised that what I was writing was just procedural code separated in different files with classes and methods, there wasn't a single advantage by using OOP like that, so I decided to change.

Design Principles

Every paradigm has its own principles, in the OOP world those principles are the SOLID (there are a few more, but those are the most famous and used). So what this means, these SOLID keywords? They stand for:

Single Responsability;
Open-Closed;
Liskov Substitution;
Interface Segregation;
Dependency Inversion;

Basically, if you build your application following all those principles, your codebase will be more flexible, abstracted and maintainable, the evolution of the software will be less painful and costly, giving you more time to expend implementing new things.

Open-Closed Principle

The Open-Closed Principle says that our classes must be open for expansions and closed for changes. Basically, we must be able to change the implementation behavior on execution-time, and to reach that, we don't need to change the class codebase, we just need the help of composition.

There is a famous dictation that says that we need to prefer composition over inheritance, and this is really important. There are some issues when we decide to use inheritance, the first is that we're breaking the OOP foundation of encapsulation, because the children knows everything about the parent. The second is the static inheritance, we can't change the behavior of the children even in execution-time, we need to change the codebase itself to be able to change the behavior, breaking the Open-Closed Principle.

When we have composition, we no longer have the "is" relationship (Ex: SavingsAccount is an Account) and we passed to have the "has" relationship (Ex: AuthorizationClient has a HttpClient), so, following the example, AuthorizationClient behaves like a normal HttpClient, but he can change your default behavior, adding an authorization header for example.

Example

Imagine the following scenario, let's say we have a front-end client application written in React, we're consuming an API and we need to pass an authentication token (a jwt for example). We decide to create an interface responsible for send HTTP requests, so, in the data layer we create our HttpPostClient protocol (Only POST requests, following the Interface Segregation principle, subject for another post).

After that, we create an implementation for this protocol based in the axios library.

Now we have our protocol (HttpPostClient) and our implementation (AxiosHttpClient), why we can't just pass the authorization header normally in the method call? We need to think that this header needs to be passed in many requests and will be always the same: Capture the token from the localStorage or from another service and pass to the method that will do the request. If we just copy & paste this implementation, we will be breaking the DRY (Don't Repeat Yourself) principle, so we need to think a smart way to do that. That's where the Decorator Pattern comes in.

The Decorator Pattern is basically a wrapper for an object. This wrapper must have the same type of the wrapped object i.e. implement the same interface and because of that, they can be exchanged in a way that the client class don't notice that change (Liskov Substitution).

The goal of this pattern is to add a behavior to the decorated object.

Going back to our example, we need a Decorator that implements the HttpPostClient interface and add the desired behavior to our AxiosHttpClient without changing the class implementation.

This Decorator class will be called AuthorizationHttpPostClientDecorator.

Some important things to notice:

Our Decorator class has the same interface of the decorated object.
He receives the wrapped object on the constructor (Dependency Inversion principle)
Runs some logic and them calls the method from the decorated object

That's an example of how we can add behavior to an class without change your implementation code.

Conclusion

Whenever we choose a new program paradigm to work with, we need to be aware of his foundations and principles, only with that knowledge we will be able to deeply understand how to use that paradigm correctly and get the most of it.

In this example I tried to demonstrate the Open-Closed principle in an easy way, with an easy example, so that you can realize its importance. I choose the decorator pattern because your application of the Open-Closed is one of the most common, however I could also implement the Proxy Pattern, he is also an structural one and work in a very similar way.