DEV Community

loading...

The difference in parallelization of each unit test frameworks in C#

jsakamoto
Microsoft MVP for Visual Studio and Development Tech. (prefer C#, .NET Core, ASP.NET Core, Azure Web Apps, TypeScript, and Blazor WebAssembly App!)
・3 min read

One day, I investigated the difference in parallelization of each unit test framework in C#.

The unit test frameworks for .NET I investigated are as bellow.
Those are what I have been used to.

  • xUnit.net
  • MSTest
  • NUnit

The project files that I used for my investigation are in the GitHub repository below.

All of these unit test frameworks support running tests in parallel, but the parallelization "levels" of these frameworks are different from each other.

The comparison table that is the results of my investigation is below.

xUnit.net MSTest NUnit
No parallelization
Class scoped parallelization
Method scoped parallelization
Parameter scoped parallelization

No parallelization

"No parallelization" means run just a single test method at once.

fig002

MSTest and NUnit run tests in no parallelization when default configuration.

Class scoped parallelization

"Class scoped parallelization" means run tests each different test classes in parallel.

For example,

  • When test class A has the test methods X and Y,
  • and the test class B has the test method Z,
  • In this case, the test methods X and Z run in parallel, because the test methods X and Z live in different test classes.

fig003

The test methods X and Y don't run in parallel because these test methods live in the same test class.

xUnit.net runs tests in class scoped parallelization when default configuration.

Method scoped parallelization

"Method scoped parallelization" means run tests any test methods in parallel whether those test methods live in the same test class or different test class.

For example,

  • When test class A has the test methods X and Y,
  • and the test class B has the test method Z,
  • In this case, all of the test methods X, Y, and Z run in parallel.

fig004

xUnit.net doesn't support "Method scoped parallelization".

Parameter scoped parallelization

"Parameter scoped parallelization" means runs test in parallel each parameter.

fig005

"Parameter scoped parallelization" is supported only NUnit, by annotating test classes and methods with the [Parallelizable] attribute.

What does the "Run Tests in Parallel" checkbox on Visual Studio Test Explorer affect?

Visual Studio IDE user can see the "Run Tests in Parallel" checkbox in the Visual Studio Test Explorer settings menu.

Alt Text

This checkbox just controls runs multiple test projects in parallel or not, if the solution contains multiple test projects.

(Sorry, I'm not sure about Visual Studio for Mac.)

In my case...

I had the unit test project built on xUnit.net, which has some data-driven test methods with a lot of parameter data.
This project took 125 seconds to complete to all tests run.

I rewrite this project to built on NUnit and configured parameter scoped parallelization enabled.
After these changes, the duration of all unit tests complete was improved to 73 seconds.

Of course, parallelization does not always improve test performance.
Especially, If the testee is CPU centric code, parallelization might be slower.
We may have to consider thermal throttling.

That's all!
Happy testing! :)

Discussion (0)

Forem Open with the Forem app