What I learned in the Real Python testing courses

I recently completed the three main courses on automated testing from the Real Python platform. As I already use these tools in my daily work, I took the courses with the goal of understanding, in general, if there are better ways to write tests than how I do daily.

Below, you will find my overview of the three courses I completed:

• Testing your code with pytest
• Improve Your Tests With the Python Mock Object Library
• Test-Driven Development With pytest

Testing your code with pytest

pytest is a framework for writing automated tests in Python that emerged as a counter to the unittest framework, which is part of the language's standard library.

Unlike the latter, which has a very verbose approach where every test case must extend from a test class, pytest brings much greater simplicity, aligning much more closely with the concept of simplicity imposed by the language.

Writing tests

Below is an example of writing a test using unittest vs pytest.

def say_hello(to):
    return f'Hello, {to}'

# unittest
import unittest

class TestMyFunc(unittest.TestCase):
    def test_say_hello(self):
        self.assertEqual(say_hello('World'), 'Hello, World')

# pytest
def test_say_hello():
    assert say_hello('World') == 'Hello, World'

With the example above, it's possible to observe the simplicity of pytest in comparison to its alternative. We can write a test just by declaring a function with the test_ prefix and use the reserved word assert to validate that the comparison is true, making the test pass.

Conversely, it's also possible to do negative conditions just by changing the sign of the comparison:

def say_hello(to):
    return f'Hello, {to}'

# pytest
def test_say_hello():
    assert say_hello('World') != 'Hello! World'

We can perform comparisons between different objects, such as lists, tuples, dictionaries, etc.

def test_list_equal():
    assert [1, 2, 3] == [1, 2, 3]

def test_tuple_equal():
    assert (1, 2, 3) == (1, 2, 3)

def test_dict_equal():
    assert {'a': 1, 'b': 2} == {'a': 1, 'b': 2}

In summary, writing tests in pytest works as follows:

Write a function with the test_ prefix;
Use the word assert to ensure that the comparison is returning True.

It's that simple!

Using fixtures

In many cases, we want to write tests based on the same input argument. For example, given a list, we want to test different situations involving it.

One way would be to define this list in all the tests and use it, as in the examples below.

def test_list_pop():
    my_list = [1, 2, 3]
    my_list.pop(1)
    assert my_list == [1, 3]

def test_list_append():
    my_list = [1, 2, 3]
    my_list.append(4)
    assert my_list == [1, 2, 3, 4]

def test_list_reverse():
    my_list = [1, 2, 3]
    my_list.reverse()
    assert my_list == [3, 2, 1]

Although it works, writing tests in this manner brings some problems, especially related to difficulties in maintenance and evolution, since for each new test it will be necessary to define the same object. Besides, if there is any change in the object we are testing, it will need to be changed in all tests, leading to poor maintainability.

pytest provides the possibility to define fixtures that will be reused among tests, in order to solve the problems highlighted above. Using fixtures, we can define any type of object we want to reuse during the tests, making any changes to this object be made in just one place.

To do this, simply use the @pytest.fixture decorator and define the object we will work with.

import pytest

@pytest.fixture
def my_list():
    return [1, 2, 3]

def test_list_pop(my_list):
    my_list.pop(1)
    assert my_list == [1, 3]

def test_list_append(my_list):
    my_list.append(4)
    assert my_list == [1, 2, 3, 4]

def test_list_reverse(my_list):
    my_list.reverse()
    assert my_list == [3, 2, 1]

To declare a new fixture, simply add the decorator right above the declaration of a function that returns the desired object. Once declared, we need to provide the name of the fixture in the tests where we want to use it, and pytest takes care of loading it within the tests.

In addition to being able to define custom fixtures, pytest also provides a list of built-in fixtures that can be consulted here.

Parameterizing tests

Depending on the function you want to test, it's beneficial to test it with various different parameters in order to stress it as much as possible and ensure that it won't have side effects.

If, for example, we wanted to test a function that performs subtraction between two numbers, we could follow the approach below.

def subtract(a, b):
    return a - b

def test_subtract_2_and_1():
    assert subtract(2, 1) == 1

def test_subtract_5_and_2():
    assert subtract(5, 2) == 3

def test_subtract_10_and_5():
    assert subtract(10, 5) == 5

Although this approach works, we end up returning to the problem of maintainability, because working in this way makes our tests more difficult to maintain, since any change in the signature or return of the function would result in changes in more than one test, and consequently in a greater effort of refactoring.

To solve this problem, pytest provides us with another decorator responsible for supplying various parameters to the same test. Using this decorator, we can specify what the input parameters are and what output the function will have for the provided parameters.

def subtract(a, b):
    return a - b

import pytest

@pytest.mark.parametrize("param_a,param_b,expected_result", [
    (2, 1, 1),
    (5, 2, 3),
    (10, 5, 5),
])
def test_subtract(param_a, param_b, expected_result):
    assert subtract(param_a, param_b) == expected_result

The @pytest.mark.parametrize decorator takes two arguments:

A string representing the parameters that will be passed to the test function.
A list that represents each of the parameters that will be provided in each iteration of the test. When there is more than one parameter, as in the example above, we use a tuple to distinguish them within the list.

Course conclusion

The course is a good introductory content, as it covers all the fundamental topics of the tool. By the end of this course, the developer should be able to write their first tests using the framework.

Due to pytest being quite simple to use, and also because its documentation is very comprehensive, a course cannot stand out in such a way that it seems indispensable for learning. However, I missed some topics such as:

• Installation and setup of pytest, including the configuration file and its parameters;
• Test coverage collection;
• Debugging;
• Best practices for test writing;
• Other commands and functionalities of the tool.

Generally speaking, I would say that you can learn everything the course shows just through the framework's documentation. Since it is a very simple tool to learn, the course could very well have covered the topics I mentioned above to ensure that the student not only knows how to use the tool itself but also how to write well-written and useful tests.

Improve Your Tests With the Python Mock Object Library

When working with automated tests, often it's necessary to "simulate" the behavior of some object or dependency so that the test can be implemented. This simulation occurs mainly in cases where your Subject Under Test (SUT) has external dependencies, such as HTTP calls, database interactions, etc.

In testing, we call this simulation a Mock. We mock a certain object when we can specify what its behavior will be during that specific test, determining the results of calls, the parameters used to call it, etc.

To clarify the necessity of this procedure, I provide an example of a method that needs to perform a GET request to another API. In this case, our test should not perform the call itself, as doing so would make it heavily dependent on the quality of the connection and the API being called. This can bring problems since if the API is not functioning correctly or is not available at the time of executing the test, it will fail. There's also the issue related to the requests themselves, as a large test load can impact the performance of the API operating in a production environment, bringing unexpected effects.

For these and other reasons, we use a Mock to simulate the behavior of the function making this GET call so that it does not make the call itself but only returns a predetermined result that makes sense for our test.

The Mock object

The Mock object is imported and used through the language's standard library (from version 3.3 onwards, for earlier versions it needs to be installed as a dependency), and it has a documentation that is very clear and comprehensive on how to use it.

The Mock class is quite flexible; once instantiated, we can make calls to methods that don't exist, and these methods will be attributed to the instance. Through this assignment, it is possible to carry out all the mocking operations we wish.

One of the functions of Mock is to take on the role of a real instance (production class or method) and make them return specific results, as in the example below.

from unittest.mock import Mock

class MyProductionClass:
    def some_method(self):
        pass

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.some_method.return_value = "it's not the original return"

print(my_instance.some_method() == "it's not the original return")

In addition to specifying the return of methods, it's also possible to perform asserts on the calls to the method in question.

from unittest.mock import Mock

class MyProductionClass:
    def some_method(self, a, b, c):
        return a + b + c

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.some_method.return_value = 15

assert my_instance.some_method(10, 15, 20) == 15
my_instance.some_method.assert_called_once()
my_instance.some_method.assert_called_once_with(10, 15, 20)

Mock has a list of methods that can be used to ensure that the calls were made in the way we want.

Every object to which a Mock instance is attributed inherits all its methods. Therefore, as demonstrated in the examples above, my_instance becomes an instance of Mock and no longer of MyProductionClass. When listing the methods of the my_instance instance, the result is as follows:

print(dir(my_instance))
"""
[
    'assert_any_call', 'assert_called', 'assert_called_once', 
    'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 
    'assert_not_called', 'attach_mock', 'call_args', 'call_args_list', 
    'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 
    'mock_calls', 'reset_mock', 'return_value', 'side_effect', 'some_method'
]
"""

The returned list are the methods that can be used from my_instance to test the behavior of the instance in question.

In addition to returning values, in some cases, we'll want to cover failure scenarios, or rather, scenarios of side effects. It's possible to throw exceptions to simulate these behaviors through the Mock object as well.

Returning to our example of a GET request, when we have an HTTP call, various errors can happen, such as timeout, service unavailable, etc. We can simulate a timeout in our GET call through the Mock object.

For example, we want to make a request to fetch all the holidays for the year 2024, and for this, we will use an API that returns all the holidays for a given year. We can force a timeout error once we mock the requests object.

import requests
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays_timeout():
    requests.get.side_effect = Timeout("too much time to complete")

    with pytest.raises(Timeout):
        get_holidays()

In this way, we simulate an undesirable behavior of the method and can work on that, without actually making the HTTP call.

Continuing with our example of the HTTP call, we can use the mock to return more complex objects, such as the response from this call.

import requests
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays():
    mocked_response = Mock()
    mocked_response.status_code = 200
    mocked_response.json.return_value = [{'Christmas': '25/12'}]
    requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

First, we instantiate a simulated response, mocked_response, and use the flexibility of the Mock class to specify the desired return.

The side_effect method is not only used to throw exceptions but also for cases where we make the call to the same method more than once and receive different results, or even the same result.

In the example below, the first request returned a timeout, due to the error we try again, and in the second request, we get the expected result.

import requests
from requests.exceptions import Timeout
import pytest
from unittest.mock import Mock

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

requests = Mock()

def test_get_holidays_retry():
    mocked_response = Mock()
    mocked_response.status_code = 200
    mocked_response.json.return_value = [{'Christmas': '25/12'}]

    requests.get.side_effect = [Timeout, mocked_response]

    with pytest.raises(Timeout):
        get_holidays()

    resp = get_holidays()
    assert resp.json() == [{'Christmas': '25/12'}]

In the first call, a timeout occurs and the exception is thrown. The second call is successful, and we get the expected return.

This behavior can also be assigned at the moment of instantiating Mock, by providing the desired returns directly in its constructor.

def test_get_holidays_retry():
    mocked_response = Mock(status_code=200, json=Mock(return_value=[{'Christmas': '25/12'}]))
    requests = Mock(get=Mock(side_effect=[Timeout, mocked_response]))

    with pytest.raises(Timeout):
        get_holidays()

    resp = get_holidays()
    assert resp.json() == [{'Christmas': '25/12'}]

Patching

Assigning a Mock instance to the object whose behavior we wish to simulate works well in cases where the test module has access to the same dependencies as the production module. However, in many cases, both modules will be separated, and the test will hardly have access to the same instances that the production class has.

Therefore, we can use a technique called patching so that the Mock of the dependency is injected directly into our test.

In the above examples of the GET call, both the get_holidays function and its test test_get_holidays are in the same file, so the test has access to the same instance of requests that the function has. If this were not true, it would be necessary to use patching.

Patching can be used as a decorator or a context manager. Below are examples using both approaches.

# holidays.py
import requests

def get_holidays():
    r = requests.get('http://holidays.com/api/holidays?year=2024')
    if r.status_code == 200:
        return r.json()
    return None

# test_holidays.py
from unittest import mock

@mock.patch('holidays.requests') # using as a decorator
def test_get_holidays(mocked_requests: mock.Mock):
    mocked_response = mock.Mock(
            status_code=200, 
            json=mock.Mock(return_value=[{'Christmas': '25/12'}]),
        )
    mocked_requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

When used as a decorator, the object will be injected into the list of arguments of the test function. This object will be a Mock instance, and all mocking operations can be performed normally from it.

# test_holidays.py
from unittest import mock

def test_get_holidays():
    mocked_response = mock.Mock(
            status_code=200, 
            json=mock.Mock(return_value=[{'Christmas': '25/12'}]),
        )

        # using as a context a manager  
        with mock.patch('holidays.requests') as mocked_requests:
          mocked_requests.get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

Using it as a context manager is quite similar, with the major difference being that the instance is created within the test, instead of being injected.

In addition to mocking the complete object, we can mock just the method we want to test, through patch.object. In the example below, instead of mocking the entire requests object, we do this just for its get method.

from unittest import mock

@mock.patch.object(requests, 'get')
def test_get_holidays(mocked_get: mock.Mock):
    mocked_response = Mock(status_code=200, json=Mock(return_value=[{'Christmas': '25/12'}]))
    mocked_get.return_value = mocked_response

    assert get_holidays() == [{'Christmas': '25/12'}]

    mocked_response.json.assert_called_once()

This option is very useful when we want to maintain the behavior of the rest of the class, simulating the behavior of only specific methods.

Autospeccing

As we've seen, the Mock object is quite flexible, capable of "assuming the identity" of different types of objects. With this flexibility, some pitfalls can occur, such as changing the behavior of a non-existent method of the "mocked" class.

from unittest.mock import Mock

class MyProductionClass:
    def sum_2(self, num):
        return num + 2

my_instance = MyProductionClass()
my_instance = Mock()
my_instance.sum2.return_value = 5

assert my_instance.sum2(3) == 5

In the example above, MyProductionClass has the sum_2 method, but when mocking, there was a typo, and the method used is a non-existent method in the class. However, due to the flexible nature of Mock, this operation was allowed, and even though the assert is returning true, the test is not simulating the behavior correctly because it is being used in a method that does not exist in the original class.

To prevent this type of problem, we use autospeccing, which is nothing more than forcing the Mock instance to follow the specification of its real object.

There are different ways to use it, one of which is represented in the example below, where we specify which specification (spec) the Mock should follow. In this case, it's the specification of the MyProductionClass class.

from unittest.mock import Mock

class MyProductionClass:
    def sum_2(self, num):
        return num + 2

my_instance = MyProductionClass()
my_instance = Mock(spec=MyProductionClass) # speccing the MyProductionClass
my_instance.sum2.return_value = 5

assert my_instance.sum2(3) == 5

"""
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'sum2'. Did you mean: 'sum_2'?
"""

In this case, an AttributeError is raised, specifying that the class does not have the sum2 method, but rather the sum_2, which is the true method.

The different ways to use autospeccing can be found in the documentation. This is a very useful way to prevent various errors during test writing.

Course conclusion

It's a much more extensive course than the first one, covering practically all topics related to the Mock object. Although all the course content can also be found in the documentation, I still found the course quite interesting for synthesizing various topics in a way that facilitates the understanding of the student.

However, I still missed the following two topics:

• MagicMock
• AsyncMock

It's a great course for those who are starting to write tests in Python.

Test-Driven Development With pytest

This was the course I was most excited about, as I work with TDD daily, and unfortunately, it was my biggest disappointment.

I won't dwell on the course content because TDD was never covered. The course is basically the instructor writing the test and then fully implementing the methods of the class without concern for the red-green-refactor cycle and other TDD-related practices. The content is quite shallow and covers pytest only superficially.

Although it is very short and you can finish it in just a few hours in front of the computer, it is not worth it if you are interested in learning only TDD because it is not covered here.

Overall Conclusion

The first two courses I presented are quite interesting for starting with automated testing in Python, especially the second one, which is quite comprehensive and covers a standard library of the language.

With just the first two courses, you can already navigate on your own in learning and improving your testing skills. Although they are good content, you can't rely solely on them; you need to research more and continue to specialize to become a good test writer.

As I mentioned earlier, the courses are quite technical, presenting the tools and how to use them. Although I felt the lack of topics on how to write good tests, I understand that it was not their intention, as their goal was to simply introduce the frameworks and not teach how to write good tests.

Regarding the last course, I was greatly disappointed. Not only because the content is weak, but also because people believe that TDD is just writing tests and then the complete production code, instead of the creative process that I believe it should be.