DEV Community

jsakamoto
jsakamoto

Posted on

2

The difference in parallelization of each unit test frameworks in C#

One day, I investigated the difference in parallelization of each unit test framework in C#.

The unit test frameworks for .NET I investigated are as bellow.
Those are what I have been used to.

  • xUnit.net
  • MSTest
  • NUnit

The project files that I used for my investigation are in the GitHub repository below.

All of these unit test frameworks support running tests in parallel, but the parallelization "levels" of these frameworks are different from each other.

The comparison table that is the results of my investigation is below.

xUnit.net MSTest NUnit
No parallelization
Class scoped parallelization
Method scoped parallelization
Parameter scoped parallelization

No parallelization

"No parallelization" means run just a single test method at once.

fig002

MSTest and NUnit run tests in no parallelization when default configuration.

Class scoped parallelization

"Class scoped parallelization" means run tests each different test classes in parallel.

For example,

  • When test class A has the test methods X and Y,
  • and the test class B has the test method Z,
  • In this case, the test methods X and Z run in parallel, because the test methods X and Z live in different test classes.

fig003

The test methods X and Y don't run in parallel because these test methods live in the same test class.

xUnit.net runs tests in class scoped parallelization when default configuration.

Method scoped parallelization

"Method scoped parallelization" means run tests any test methods in parallel whether those test methods live in the same test class or different test class.

For example,

  • When test class A has the test methods X and Y,
  • and the test class B has the test method Z,
  • In this case, all of the test methods X, Y, and Z run in parallel.

fig004

xUnit.net doesn't support "Method scoped parallelization".

Parameter scoped parallelization

"Parameter scoped parallelization" means runs test in parallel each parameter.

fig005

"Parameter scoped parallelization" is supported only NUnit, by annotating test classes and methods with the [Parallelizable] attribute.

What does the "Run Tests in Parallel" checkbox on Visual Studio Test Explorer affect?

Visual Studio IDE user can see the "Run Tests in Parallel" checkbox in the Visual Studio Test Explorer settings menu.

Alt Text

This checkbox just controls runs multiple test projects in parallel or not, if the solution contains multiple test projects.

(Sorry, I'm not sure about Visual Studio for Mac.)

In my case...

I had the unit test project built on xUnit.net, which has some data-driven test methods with a lot of parameter data.
This project took 125 seconds to complete to all tests run.

I rewrite this project to built on NUnit and configured parameter scoped parallelization enabled.
After these changes, the duration of all unit tests complete was improved to 73 seconds.

Of course, parallelization does not always improve test performance.
Especially, If the testee is CPU centric code, parallelization might be slower.
We may have to consider thermal throttling.

That's all!
Happy testing! :)

Do your career a favor. Join DEV. (The website you're on right now)

It takes one minute and it's free.

Get started

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay