Modern Test Pyramid

Valentina (Cupać) Jemuović — Fri, 27 Dec 2024 10:11:30 +0000

The Old Test Pyramid (Unit, Integration & E2E tests) - is dead. It's got fundamental problems. That's why we're introducing the Modern Test Pyramid, so you can practice effective Automated Testing.

The Old Test Pyramid is dead

Most developers see the Test Pyramid in the following way:

We previously outlined the problems with the Old Test Pyramid:

Overuse of E2E tests, often reaching the Inverted Test Pyramid
No consensus on the definition of Integration Tests, and over-reliance on large scoped integration tests makes it hard to maintain
No consensus on the definition of Unit Tests, and writing unit tests that are structurally coupled to code with over-mocking, thus fragile when refactoring
The Test Pyramid does not include Acceptance Tests needed for testing Acceptance Criteria, but instead it relies on fragile E2E Tests for that
The Test Pyramid doesn't provide us guidance for testing microservices, specifically it doesn't even mention Component Tests & Contract Tests which are crucial for microservice testing

Modern Test Pyramid

Now let's look at the Modern Test Pyramid:

We can also view it as a Test Spectrum where we illustrate the test spans:

Let's see the test types - why they are used, what they target and span:

Smoke Tests

We use Smoke Tests to verify that the System is up-and-running.
Smoke Tests target the System - they span the System and External System Stubs or External System Test Instances.

E2E Tests

We use E2E Tests to verify System behavior, however the scenarios are limited.

E2E Tests target the System - they span the System and External System Test Instances. They are executed on the E2E Environment.

Acceptance Tests

We use Acceptance Tests to verify behavior of the System. We have the possibility to execute any scenario.

Acceptance Tests target the System, and they span the System and External System Stubs.

External System Contract Tests

External System Contract Tests are used to verify that we can communicate with External System Test Instances and that their contract matches our expectations.

External System Contract Tests target (and span) the External System Test Instances.

Component Tests

We use Component Tests to verify the behavior of Components (e.g. verifying behavior of Frontend, Microservice #1, Microservice #2, Microservice #3)

Component Tests target the Components, and span Component Dependency Stubs (e.g. targeting Microservice #1 and Microservice #2 Stub and External System A Stub)

Contract Tests

We use Contract Tests to verify communication between a Component and its Dependencies.

Common example is Consumer Driven Contract Testing

Unit Tests

We use Unit Tests to verify application-side business behavior.

Narrow Integration Tests

We use Narrow Integration Tests to verify presentation behavior and infrastructure behavior within the Component.

The Modern Test Pyramid solves problems

The Modern Test Pyramid SOLVES the problems of the Old Test Pyramid in the following ways:

Problematic E2E tests

In the Old Test Pyramid, E2E Tests are problematic because they are limited regarding scenarios when testing System Behavior. The tests are often fragile and non-deterministic.

In the Modern Test Pyramid, we solve that problem through Acceptance Tests (together with External System Contract Tests) which enables us to fully test System Behavior in a deterministic, reliable way.
Problematic Integration tests

In the Old Test Pyramid, Integration Tests are problematic because developers generally interpreted them as Broad Integration Tests. They had similar characteristics as E2E Tests - limited scenarios and fragility, with the additional problem that it was hard to isolate where the fault is occurring.

That's why in the Modern Test Pyramid, we introduce Component Tests & Contract Tests to replace Broad Integration Tests. We can fully test each Component and its interactions with other Components in a reliable way.

No clue how to handle Microservice Architecture

In the Old Test Pyramid, there was no guidance how to handle Microservice Architecture. Often teams would use Broad Integration Testing for microservices, then waste time debugging which microservice was at fault, and cause a lot of dependencies between microservice teams.

The Modern Test Pyramid contains Component Tests, which are very effective for separately testing Frontend and each Microservice. It also contains Contract Tests, which are effective for testing communication with another microservice without having both of them running.

No clue how to test Presentation & Infrastructure Logic

In the Old Test Pyramid, there's no guidance on testing Presentation Logic and Infrastructure Logic, but only Business Logic. Sure, we'd use Unit Tests for Business Logic, but what kind of tests do we need for Presentation Logic & Infrastructure Logic? The Old Test Pyramid doesn't provide an answer.

In the Modern Test Pyramid, we solve that problem with Narrow Integration Tests.

Let's dive deeper - you can read the full article: Modern Test Pyramid (Optivem Journal).

The Old Test Pyramid is Dead

Valentina (Cupać) Jemuović — Thu, 26 Dec 2024 10:02:03 +0000

You’ve all heard of the Test Pyramid — E2E Tests, Integration Tests, and Unit Tests. It’s popular. But it has BIG problems — misinterpretations of test types and missing test types. It’s dangerous.

The Original Test Pyramid

The Test Pyramid was introduced by Mike Cohn in 2009 in his book Succeeding with Agile: Software Development Using Scrum. He has the following levels:

UI Tests
Service Tests
Unit Tests

UI Tests

Mike Cohn indicates that the minority of the tests should be UI tests. He notes their disadvantages — that they are: “Brittle”, “Expensive to write” and “Time consuming”.

“Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible.” — Mike Cohn, Succeeding with Agile

Service Tests

He places Service Tests in the middle, to avoid the situation that we have too many UI tests, instead we want to test at the layer below the UI.

“Service-level testing is about testing the services of an application separately from its user interface. So instead of running a dozen or so multiplication test cases through the calculator’s user interface, we instead perform those tests at the service level.” — Mike Cohn, Succeeding with Agile

However, it should also be noted that he isn’t too precise regarding the definition of service tests, that it doesn’t just mean SOA but can refer to anything that’s transforming inputs into outputs:

“Although I refer to the middle layer of the test automation pyramid as the service layer, I am not restricting us to using only a service-oriented architecture. All applications are made up of various services. In the way I’m using it, a service is something the application does in response to some input or set of inputs.” — Mike Cohn, Succeeding with Agile

Service Tests are often forgotten, so he wrote a blog post about the forgotten service level testing.

Unit Tests

Mike Cohn indicates that the majority of our tests should be unit tests. Unit tests are useful due to the fast feedback they provide, that we can easily localize regression bugs within code:

“At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer — there is a bug and it’s on line 47.” — Mike Cohn, Succeeding with Agile

The Popularized Test Pyramid

If you do a Google search for Test Pyramid images, most likely you’ll end up with an image that looks like this, with these layers:

E2E Tests
Integration Tests
Unit Tests

That’s THE way to do automated testing, right?

Wrong.

Using the “Popular” version of the Test Pyramid has led to the following problems:

Overuse of E2E tests, often reaching the Inverted Test Pyramid
No consensus on the definition of Integration Tests, and over-reliance on large scoped integration tests makes it hard to maintain
No consensus on the definition of Unit Tests, and writing unit tests that are structurally coupled to code with over-mocking, thus fragile when refactoring
The Test Pyramid does not include Acceptance Tests needed for testing Acceptance Criteria, but instead it relies on fragile E2E Tests for that
The Test Pyramid doesn’t provide us guidance for testing microservices, specifically it doesn’t even mention Component Tests & Contract Tests which are crucial for microservice testing

To dive deeper into the problems of the old Test Pyramid, read the full article on Optivem Journal.

DEV Community: Valentina (Cupać) Jemuović

Modern Test Pyramid

The Old Test Pyramid is dead

Modern Test Pyramid

Smoke Tests

E2E Tests

Acceptance Tests

External System Contract Tests

Component Tests

Contract Tests

Unit Tests

Narrow Integration Tests

The Modern Test Pyramid solves problems

Problematic E2E tests

No clue how to handle Microservice Architecture

No clue how to test Presentation & Infrastructure Logic

The Old Test Pyramid is Dead

The Original Test Pyramid

UI Tests

Service Tests

Unit Tests

The Popularized Test Pyramid