The difference in parallelization of each unit test frameworks in C#

One day, I investigated the difference in parallelization of each unit test framework in C#.

The unit test frameworks for .NET I investigated are as bellow.
Those are what I have been used to.

• xUnit.net
• MSTest
• NUnit

The project files that I used for my investigation are in the GitHub repository below.

• https://github.com/sample-by-jsakamoto/ParallelizableUnitTests

All of these unit test frameworks support running tests in parallel, but the parallelization "levels" of these frameworks are different from each other.

The comparison table that is the results of my investigation is below.

	xUnit.net	MSTest	NUnit
No parallelization	✔	✔	✔
Class scoped parallelization	✔	✔	✔
Method scoped parallelization		✔	✔
Parameter scoped parallelization			✔

No parallelization

"No parallelization" means run just a single test method at once.

MSTest and NUnit run tests in no parallelization when default configuration.

Class scoped parallelization

"Class scoped parallelization" means run tests each different test classes in parallel.

For example,

• When test class A has the test methods X and Y,
• and the test class B has the test method Z,
• In this case, the test methods X and Z run in parallel, because the test methods X and Z live in different test classes.

The test methods X and Y don't run in parallel because these test methods live in the same test class.

xUnit.net runs tests in class scoped parallelization when default configuration.

Method scoped parallelization

"Method scoped parallelization" means run tests any test methods in parallel whether those test methods live in the same test class or different test class.

For example,

• When test class A has the test methods X and Y,
• and the test class B has the test method Z,
• In this case, all of the test methods X, Y, and Z run in parallel.

xUnit.net doesn't support "Method scoped parallelization".

• See also: "Running Test in Parallel | xUnit.net"

Parameter scoped parallelization

"Parameter scoped parallelization" means runs test in parallel each parameter.

"Parameter scoped parallelization" is supported only NUnit, by annotating test classes and methods with the [Parallelizable] attribute.

• See also: "Parallelizable | NUnit Docs"

What does the "Run Tests in Parallel" checkbox on Visual Studio Test Explorer affect?

Visual Studio IDE user can see the "Run Tests in Parallel" checkbox in the Visual Studio Test Explorer settings menu.

This checkbox just controls runs multiple test projects in parallel or not, if the solution contains multiple test projects.

(Sorry, I'm not sure about Visual Studio for Mac.)

In my case...

I had the unit test project built on xUnit.net, which has some data-driven test methods with a lot of parameter data.
This project took 125 seconds to complete to all tests run.

I rewrite this project to built on NUnit and configured parameter scoped parallelization enabled.
After these changes, the duration of all unit tests complete was improved to 73 seconds.

Of course, parallelization does not always improve test performance.
Especially, If the testee is CPU centric code, parallelization might be slower.
We may have to consider thermal throttling.

That's all!
Happy testing! :)