This week for my AST Analyzer project I decided to do a deep dive on the testing framework pytest. When i started off the week I genuinely thought I already knew everything I needed, but man was I pleasantly surprised at how powerful that testing suite is. Today I'm going to go over a few new things I learned when implementing my tests suite, including:
- Assertion behind the scenes
- Fixtures
- Capsys and Caplog
- Parametrization
Like last week's post, most of these were learned with the combination of two resources:
Assertion
One of the first things I noticed when switching from unittest to pytest was how much simpler assertions are. In unittest, you need to use specific methods like assertEqual, assertTrue, assertIn, etc:
# unittest style
self.assertEqual(result, 5)
self.assertTrue(is_valid)
self.assertIn("error", message)
With pytest, you just use Python's built-in assert statement:
# pytest style
assert result == 5
assert is_valid
assert "error" in message
I was wondering how a 3rd party package could make assertion statements clearer than what's built in to Python, and while digging deeper I found that there's a whole process that goes on when the test fails.
Assert Rewriting with AST
Pytest uses the built in assert function because it raises an AssertionException when the case does not pass. This built-in assertion check allows Pytest to work with built-in functionality and expand functionality from there.
For example, once an AssertionException is raised pytest uses assertion rewriting to replace the default string. This is done at import time where pytest can grab the value of each variable and sub expressions. This new info is used to provide the logs with more details like:
- Provides a diff of what was given and what was expected
- The line the test failed at
- What was the failure (mismatch, missing item, etc)
By doing this at import time, it's able to place all code in an Abstract Syntax Tree (yes, the same AST we're working with in this project), finds all the assert statements, and rewrites them to capture intermediate values before the assertion runs. This is why when an assertion fails, pytest can show you exactly what each part of the expression evaluated to:
def test_string_comparison():
result = "hello world"
assert result == "hello pytest"
=========================== FAILURES ===========================
_________________ test_string_comparison _______________________
def test_string_comparison():
result = "hello world"
> assert result == "hello pytest"
E AssertionError: assert 'hello world' == 'hello pytest'
E
E - hello pytest
E + hello world
======================== 1 failed in 0.02s =====================
This works for complex expressions too:
def test_list_membership():
items = ["apple", "banana", "cherry"]
target = "grape"
assert target in items
E AssertionError: assert 'grape' in ['apple', 'banana', 'cherry']
Pytest captured both the value of target and items before the assertion ran, so it can show you exactly why it failed.
Fixtures
Coming from Node, one thing I appreciated about jest is being able to use beforeEach, beforeAll, afterEach, and afterAll in combination with describe scopes to setup and tear down test data in a straightforward manner. I was relieved to see that pytest gives us these features with the ability of fixtures that allow us to specify the same level of setup, teardown, and scope that's in jest.
Why Fixtures over beforeEach
While I was relieved to see familiar patterns, I quickly realized that fixtures are actually more powerful than Jest's approach in a few key ways:
- Composition - Fixtures can depend on other fixtures. We injected pytest's built-in tmp_path fixture into our own custom fixture - that's composition in action. In Jest, you'd have to nest your beforeEach blocks or manually call setup functions to achieve something similar.
- Scoping - Fixtures let you specify how often they run: function (default, runs for each test), class, module, or session. If I have an expensive setup like connecting to a database, I can scope it to session and it only runs once for the entire suite.
- Reusability via conftest.py - Any fixture defined in a conftest.py file is automatically available to all tests in that directory and subdirectories. No imports needed.
- Pay for what you use - This one was subtle but important. With Jest's beforeEach, the setup runs before every test in that scope whether you need it or not. With pytest, fixtures only run when a test actually requests them as a parameter. If I have 10 tests in a class but only 3 need the sample_code_file fixture, it only gets created 3 times. This keeps tests fast and avoids unnecessary setup.
Setup and Teardown
Part of testing the ASTAnalyzer was making sure that we had a file with data to parse throughout out tests. Initially I created a sample file in a tests/data document and tested against that, but then found that we can create one using a combination of our own fixture and one of pytest's built in fixtures:
@pytest.fixture
def sample_code_file(tmp_path):
"""Factory fixture to create temporary Python files with specified content."""
def _create_file(content, filename="test_file.py"):
file_path = tmp_path / filename
file_path.write_text(content)
return str(file_path)
return _create_file
def test_enter_opens_file(self, sample_code_file):
"""__enter__ opens the file and returns file object."""
filepath = sample_code_file("x = 1")
with Parser(filepath) as f:
assert f is not None
assert not f.closed
def test_context_manager_with_exception(self, sample_code_file):
"""File is closed even when exception occurs."""
filepath = sample_code_file("content")
file_ref = None
with pytest.raises(ValueError):
with Parser(filepath) as f:
file_ref = f
raise ValueError("test error")
assert file_ref.closed
As we can see, we've created a sample_code_file that takes in the content that we want to test against and automatically writes it to a file. This allows us to have a piece of reusable code that can be used to test all types of content inside of the file
Capsys and Caplog
One of the harder things I found while setting up my initial tests was making sure that my printing and logging decorators were being tested. Initially I was using unittest.patch to mimic this behavior like so:
@logger(logging.DEBUG)
def add(a, b):
return a + b
def test_ast_log_defaults():
with patch("ast_analyzer.decorators.logger.logger") as mock_logger:
add(3, 5)
mock_logger.debug.assert_called()
call_args = str(mock_logger.debug.call_args)
assert "add" in call_args
The test technically works, it checks that mock logger was called in the add() function, but it doesn't actually check what the contents of that log are. With pytest we can use caplog to grab the output and save it to a file that we can then read from:
@logger(logging.DEBUG)
def add(a, b):
return a + b
def test_ast_log_defaults(caplog):
with caplog.at_level(logging.DEBUG):
add(3, 5)
assert "DEBUG" in caplog.text
assert "add" in caplog.text
With this new functionality, we can look for specific strings of text inside of the log. We also have control over the logging level that we display, so we can write tests for DEBUG and INFO to check that the text in both of those logs are appearing properly. This can also be done with capsys for checking print statements:
def test_prints_timing_output(capsys):
"""Decorator should print timing information to stdout."""
factorial(3)
captured = capsys.readouterr()
assert "factorial" in captured.out
assert "->" in captured.out
In both formats we can see the ease of use that pytest gives us for accessing this text
Parametrization
Last but not least, there's parametrization. Parametrization is a technique that's used in python to make code more modular and reusable. In our specific case, we use parametrization for testing by iterating through multiple parameters. This is a great way to reduce the amount of code in your test suite without losing any functionality. To demo this, I'll show a before and after of a series of tests made in the repo.
Before Parametrization
When we create our custom ASTNode, one thing we do on initialization is see how many children the Node has. In order to test that our __init__ declaration for children is working, we set up the following tests:
def test_str_shows_children_count_one_child(self):
"""__str__ displays the number of children."""
tree = ast.parse("x = 1")
node = ASTNode(tree)
assert str(node) == f"AST Node | Children: 1"
def test_str_shows_children_count_mult_children(self):
"""__str__ displays the number of children."""
tree = ast.parse("x = 1\ny = 2\nz = 3")
node = ASTNode(tree)
assert str(node) == f"AST Node | Children: 3"
---
================ test session starts ================
platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/davidmoran/Sites/ai-bootcamp/projects/AST-Analyzer
configfile: pyproject.toml
plugins: cov-7.0.0
collected 2 items
tests/test_astnode.py .. [100%]
================= 2 passed in 0.03s =================
As you can see, the code is simple enough where having it be exactly the same isnt too bad, but its repetitive which is something that we want to avoid. When playing around with this I thought of combining this into a collection and testing that way:
def test_str_shows_children_count(self):
"""__str__ displays the number of children."""
tree = ast.parse("x = 1")
node = ASTNode(tree)
assert str(node) == "AST Node | Children: 1"
tree = ast.parse("x = 1\ny = 2\nz = 3")
node = ASTNode(tree)
assert str(node) == "AST Node | Children: 3"
---
================ test session starts =================
platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/davidmoran/Sites/ai-bootcamp/projects/AST-Analyzer
configfile: pyproject.toml
plugins: cov-7.0.0
collected 1 item
tests/test_astnode.py . [100%]
================= 1 passed in 0.02s ==================
While this is definitely a lot cleaner and lean, one thing I didn't appreciate is that I'm stuffing two test cases into one. In the future if one of these were to fail (in a larger test) it would be a bit annoying trying to figure out which one of these was the culprit
Introducing: Parametrization
Parametrization solves the above by creating a matrix of key values to test against and running them against 1 test. Unlike our second option above, using parametrization breaks the singular test out into the number of suites we specified above so that we can see which of the items failed a test
@pytest.mark.parametrize(
"code,expected_count",
[
("x = 1", 1),
("x = 1\ny = 2\nz = 3", 3),
],
)
def test_str_shows_children_count(self, code, expected_count):
"""__str__ displays the number of children."""
tree = ast.parse(code)
node = ASTNode(tree)
assert str(node) == f"AST Node | Children: {expected_count}"
---
================ test session starts =================
platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/davidmoran/Sites/ai-bootcamp/projects/AST-Analyzer
configfile: pyproject.toml
plugins: cov-7.0.0
collected 2 items
tests/test_astnode.py .. [100%]
================= 2 passed in 0.05s ==================
The way this works is pretty straightforward:
- We use the
@pytest.mark.parametrizeand pass in two arguments- A string of variable names separated by commas
- A tuple of values that you want each variable to represent on iteration
- For each tuple in our collection we passed, the test will iterate over and replace the variables with the values we provided
- Every time the test is run, it reports it as a separate test, meaning that we can get more insight into which parameter will fail in case of an error.
This feature of pytest works great when you want to test one specific thing against a number of start points. For our test, we just wanted to make sure that an ASTNode was created successfully based on the input, so it's a perfect candidate for parametrization. If I wanted to check the outputs or error handling of certain inputs, that is better handled as a separate test so that we can check on a number of items (type of exception if raised, state of ASTNode, log statements, etc).
Top comments (0)