Ok, thanks Miki for this nonsense, but what is the question then? Well, the right question is :
How much to automate?
With this article, I am starting a new series in which I am discussing testing theory topics with real-life examples. Before answering our question, let's look at why we need automated tests.
Why should you write automated tests?
In today's world, Continuous Delivery is a widely used practice. By using it you can make sure your software can be released anytime. In my team, we are using a GitHub pipeline where we automatically run tests and deploy the product to our development and staging server (from the appropriate branch) if the tests are successful.
Although at the beginning of the project manual testing might be adequate , as soon as the project gets bigger you won't be able to keep up with the pace of development using only manual testing. Especially as hopefully the number of relevant regression tests will increase in parallel with developing new features. In order to avoid it, it is highly recommended to start automating your tests from the beginning and continue with them in parallel with the product development.
After this introduction, let's get back to our question. In order to answer it, let me first introduce Mike Cohn's Test Automation Pyramid , which he described in his book: Succeeding with Agile: Software Development Using Scrum.
Mike Cohn's testing pyramid
As you can see, the pyramid consists of 3 layers: Automated Unit Tests, Automated Service Tests, Automated UI Tests. It shows the preferred distribution of different types of tests. Let's examine it in detail.
Bottom layer: Unit tests
Unit tests are the basic building blocks of every automated test portfolio. They are used to verify the code behaves as intended. A unit is the smallest testable component of the code. It can be a method, a class.
It is usually written by the developers , especially these days when Test Driven Development (TDD) is widely used. Writing unit tests is cheap , they run fast and they provide a safety net for refactoring. Whenever a unit test fails, you can easily identify the problem.
Using Java the 2 most popular unit testing frameworks are JUnit and TestNG.
Middle layer: Automated service tests
The service layer nestles between the presentation (UI) and the database layer. For further information, you can look at the MVC pattern.
By testing the service layer, you can test the business logic of your application. Talking about web applications means testing the APIs of your services. You can test many aspects like functionality, reliability, performance, security.
Writing service tests is much faster than writing UI tests , but of course slower than writing unit tests. The same can be told about their execution time.
Service tests can be written by both developers and testers. At my previous company, both developers and testers wrote API tests and at my current one, it is the test team's responsibility to write the automated service tests.
For API testing you can use frameworks/tools like RestAssured, Postman or Karate DSL.
Top layer: Automated UI tests
From the 3 levels, testing via the user interface simulates most precisely the user behaviour , but running automated UI tests can be much faster and less error-prone than manual testing, however, it is still way slower than API tests. From the 3 types of tests, UI tests are the easiest to break. Even if you use stable selectors, a small change in the user interface can fail your test. Usually, the frontend changes more often than the service layer , which also makes it harder to maintain UI tests. On the other hand, with a single E2E test you can achieve much higher coverage than with an API test. Usually, whenever you execute a UI test, it triggers many service calls in the background.
For user interface testing in Java, you can use frameworks like Selenide, Selenium WebDriver, Cypress.
Further versions of the Testing Pyramid
Mike Cohn's Testing Pyramid is more than 10 years old, which is quite a long time in our industry, so it is not surprising that the concept evolved further. There are many versions of it by now. One of them looks like this.
I won't write more about this version, you can google it in case you are interested, but I think Mike Cohn's testing pyramid is good enough to demonstrate the concept.
Avoid the ice cream cone
To close this topic, let's talk about an anti-pattern that you should avoid. It is called the Ice Cream Cone. It is the complete opposite of the testing pyramid. The ice cream represents manual tests and the code represents the same as the testing pyramid but in the opposite order. Having the most UI tests and having the least number of unit tests.
What about manual tests?
As you might found out by now you should have the least number of manual tests compared to other layers of testing. Most probably there will be scenarios always which cannot be automated (or does not worth the effort), so some manual testing is almost always needed.
One of my friends told me he has a friend who knows someone, who heard about a company that only has automated testing. In case you are working in such a company, please let us know , as it would be very valuable for all of us to hear about your experiences.
How do we do it?
Thanks to our management, our client is aware of the importance of quality assurance , so he is supporting us in continuously improving our test process.
The testing team's responsibility is to execute service and integration tests and also E2E tests. Our testing team is transitioning from mainly manual testing to automated testing. We have both automated API and UI tests integrated into our GitLab pipeline so tests are running quite frequently.
In the current situation, we have the most number of manual tests but we are working hard to automate them. We have several API tests and fewer UI tests. We are testing API using the Karate DSL framework which is using Gherkin language, you can read about it in an upcoming article series. For UI automation our choice is Selenide , which is a great framework built on top of Selenium WebDriver. You can get in-depth knowledge about it following my article series.
In case you are interested in the management side of our testing, I recommend my colleague's article: First steps of quality management in agile projects.
Main takeaways
Of course, the testing pyramid is just a suggestion , you can differ from it based on your special situation. To sum it up:
- You should have the most from unit tests , as they are cheap to write and run and maintain.
- You should have fewer automated service tests than unit tests.
- You should have fewer automated tests for the UI than service tests, as they are slow compared to the others and they break easily.
- You should have the least number of manual tests as they are slow to execute.
And now I will go and grab an ice cream. I advise you to do the same but have a real one, not a testing ice cream cone. 😊
P.S.: Support me by sharing this article in case you found it useful.
📚 Join the Selenide community on LinkedIn! ✌
.
Top comments (0)