The Test Pyramid

In the previous blog post, we touched on two different categories of automated tests, namely solitary and sociable tests. We also mentioned that in order to build maintainable and high-quality software systems, we need both solitary as well as sociable tests, but not in an equal amount. Although tests from both of these categories have their strong points and weaknesses, it’s more desirable to have several solitary tests combined with only a few accompanying sociable tests.

This is where the test pyramid comes in. Initially described by Mike Cohn in the book Succeeding with Agile, the test pyramid provides a visual representation of a beneficial balance of automated tests.

The concept of the test pyramid is made up from the following parts:

• At the base, we have solitary tests. These should make up the largest amount of automated tests in the software system.
• In the middle, we have sociable tests. These should definitely be part of the suite of automated tests, but in a significantly lesser amount than the number of sociable tests.
• At the top, we have a handful to a dozen broad system tests. This is usually in the form of UI or API tests that exercise the most important parts of the system. Try to resist the urge to create lots of these kind of tests as they can quickly turn against you and soon become a maintenance nightmare.

At the bottom of the test pyramid, we have the most isolation and also the fastest performance. This is where we get the most feedback about the design of the system. The more we move up the test pyramid, the more integration is employed and verified which results in slower tests and less feedback.

There are four primary reasons as to why we prefer that the majority of tests are solitary tests:

1. Sociable tests can be very slow and nondeterministic. This is due to the fact that they often cross their process boundary. They make failure diagnostics more difficult because they are further removed from the cause of the failure.
2. Sociable tests can therefore be overly verbose as they require a lot more code in order to setup parts of the system under test. It requires more effort and therefore takes more time to write.
3. Solitary tests apply a certain pressure on the design of the system. They clearly indicate the design flaws that might exist. Sociable tests on the other hand don’t provide such useful feedback about the design of the system because they are inherently farther removed from the details.
4. Sociable tests are highly susceptible to cascading failures. We’re going to ellaborate further on this issue in the next post.

Do not bet on only a single category of these tests! Applying all of these categories to the system is the best way to achieve a well-tested code-base. However, a surprisingly large number of development teams have made the mistake of applying a testing pyramid that is upside down. Such an “ice cream cone” shaped automated testing strategy implies that there are more sociable tests than solitary tests. This anti-pattern usually stems from an overreliance on manual testing and/or a lack of applying the Test-Driven Development process.

Do bear in mind that the exact number of tests in each layer of the test pyramid is not really that relevant. However, the test pyramid is a very useful model to think and reason about. Let it be your guide on all your test automation efforts.