In this post, I want to share my experience improving the automated testing process for web applications using Python, pytest, Selenium, and Playwright, and explain how I developed Testinel — a test sentinel, a guardian watching over your tests. My motivation for creating this tool was to reduce the time engineers spend fixing failures in automated tests and to lower the barrier to entry for test automation engineers.
Introduction
One of my favorite XKCD comics is “Programmers’ #1 Excuse for Legitimately Slacking Off” by Randall Munroe:
Manager:
— Hey! Get back to work!
Programmers:
— The code is compiling!
— Oh, okay then.
Almost 20 years have passed since it was published, but the comic is still relevant not only to large projects written in compiled languages such as C++, Rust, or Swift, but also to the process of automating web application testing.
— Get back to work!
— The tests are running!
— Oh, okay then.
What should a test automation engineer do while automated tests are running? When the entire test suite finishes within a few minutes, you can make yourself a cup of coffee or lean back in your chair and take a short break. But what if the tests take tens of minutes or even hours? There are several options:
- do nothing and relax;
- do nothing and stare at the console or the CI server page;
- work on something else: write new tests, communicate with the team, or perform manual testing.
If you have other options, tell me in the comments what you do when your tests take an entire hour to complete.
Two Problems with Monitoring Automated Test Execution
This is where I noticed the first problem in the testing process: while a long automated test suite is running, the engineer uses their working time inefficiently. They either do nothing — the XKCD option — or switch to other tasks, which reduces productivity because they lose context and stop actively monitoring the tests. Task switching itself also has a negative effect.
What do I mean by monitoring automated test execution? If all tests pass, which happens fairly often, that is good news and the engineer will not have much work to do. They only need to report to the manager and the team that the automated tests did not detect any problems.
If some tests fail, however, the difficult work of processing the report begins:
- identify which tests failed;
- determine the reason for each failure: a web application defect, invalid test data, an error in the automated test code, or a failure in the testing infrastructure;
- fix errors in the tests or failures in the testing infrastructure and rerun the tests;
- create or reopen a bug tracker ticket for defects in the web application.
Why does the engineer not monitor test execution from the very beginning? Because the pytest test runner is designed to generate a detailed test report only after all tests have completed. See the video below.
The engineer can see that a test has failed — FAILED — but pytest provides the reason for the failure, including the exception, message, and location in the code, only after every test has finished running.
This problem is even more noticeable on a CI server because various testing artifacts — logs, screenshots, videos, and so on — also become available only after the automated tests have completed.
Conclusion No. 1: during automated test execution, the engineer does not have all the information needed to determine the cause of a failure. Without this information, the engineer can neither create a high-quality bug tracker ticket nor properly fix the automated test scripts.
The second problem that makes engineers work inefficiently is that diagnostic information about test failures is poorly structured. Anyone who works with automated testing knows how painful it can be to investigate the failure of even a single test:
- the traceback can be very long;
- the exception message can be very long;
- the place where the exception occurs is not necessarily the place you should inspect, because you usually need to look at the automated test code rather than the library code;
- all of this is presented as one long wall of text.
Those who do not work with automated tests can watch the end of the video above or look at the screenshot below. And this is only part of the output, because a typical traceback does not fit on a single monitor screen.
Conclusion No. 2: engineers spend a significant amount of time reading large amounts of text to understand why a test failed and how to fix it.
How I Solved These Two Problems
Let me recap the problems I am trying to solve:
- pytest does not generate a detailed report until testing is complete.
- Diagnostic information about test failures is poorly structured.
I solved the problem of delayed reporting by developing the pytest-testinel plugin. It streams test events and diagnostic information to a cloud server while the tests are running. At the same time, it uploads screenshots, videos, and Playwright traces to cloud storage so that everything becomes available as soon as an individual test finishes, rather than only after the entire test suite has completed.
There is no need to rewrite, refactor, or annotate your tests in any special way. It is a drop-in plugin, which means you only need to install and configure it:
- Sign up, create a project, and obtain a
TESTINEL_DSN. - Run
pip install pytest-testinel. - Run
export TESTINEL_DSN=https://testinel.dev/ingest/your_secret_dsn/.
That is all!
This video demonstrates test events being streamed from Bitbucket Pipelines to the cloud server. Notice that test results appear progressively as the tests run on the CI server. You can begin investigating failures immediately instead of waiting for the entire test suite to finish.
I solved the problem of poorly structured diagnostic information by presenting it in a clear and organized way:
- I extracted the key information from messages and hid unnecessary noise;
- I separated each traceback frame from the others;
- I classified frames as test, page object, Selenium, and so on, and hid the code of library frames because engineers need to inspect them far less often than the test code;
- I added screenshots, videos, and Playwright traces directly to the report, so engineers no longer need to search for them on the CI server or in a reports directory;
- I added a little color to the code to make it easier on the eyes.
As a result, I got one big beautiful™ test report:
As a bonus, I also got a history of automated test runs. All the structured information was already stored in the database, so why not display it?
Anyone running tests with Python, pytest, Selenium, and Playwright can use this system for free: testinel.dev
Conclusions
I have been testing this solution with our test automation team for six months. On many occasions, I noticed that commits fixing errors in the automated tests appeared in Git before the test suite had even finished running.
This means that the value hypothesis behind real-time event streaming was confirmed not only by positive feedback from engineers, but also in practice.
Anyone interested in trying it with their own tests can do so for free: testinel.dev
“Volodymyr, you are several years too late. Humans do not read these stack traces anymore. An agent does not care what color the code is in the traceback!”
You’re absolutely right!™ I am late!
I came up with the idea for this kind of system long before the AI revolution, and I decided to bring it to production to finally close that gestalt.
Besides, agents do not solve the problems of real-time event streaming or cloud storage for diagnostic information. To make Testinel useful in this new environment, I am planning to develop a skill and a CLI utility that will make all of this diagnostic information available to agents.
I therefore believe that the tool will remain useful, provided that such an integration is implemented.
Vova, always close your gestalts.
Top comments (0)