NET Core 2: Why xUnit and not NUnit or MSTest

With a recent new project using NET Core 2, my team and I looked at whether we should move to MS Test(Didn't consider MS Test 2 at that time), stick with NUnit or try xUnit. We had a spike, where I looked into whether we could still use NUnit in case we were not able to use xUnit, as we were not keen on MSTest as an alternative framework. In the end, we decided to give xUnit a go!

xUnit is pretty lean compared to NUnit and MsTest and has been written more recently. The .NET framework has evolved since NUnit was first created. XUnit leverage some of the new features to help developers write cleaner test, as tests should be kept clean and treated as first-class citizens. The authors wanted to codify some rules rather than repeating guidance about “do X” or “don’t do Y". You can read more about why xUnit was created here: https://xunit.github.io/docs/why-did-we-build-xunit-1.0.html.

Some of the reasons why we went with xUnit:

• NUnit was not fully compatible with .NET Core 2 at the time
• We wanted to move away from MS Test, as the team preferred the xUnit and NUnit way of writing tests
• xUnit is aimed at improving test isolation and trying to codify a set of rules to establish a testing standard.
• xUnit [Fact] and [Theory] attributes are extensible, so you can implement your own testing functionality.xUnit doesn’t use Test Lists and .vsmdi files to keep track of your tests.
• Microsoft is using xUnit internally, one of its creators is from Microsoft. xUnit was also created by one of the original authors of NUnit.
• There are no [Setup] and [Teardown] attributes, this is done using the test class’ constructor and an IDisposable. This encourages developers to write cleaner tests.
• xUnit allows us to write less code since its flexibility allows things like subspec which allow you to write only what you need to do. Using tools such as xBehave: http://xbehave.github.io/ and Xunit.Gherkin.Quick.
• xUnit is easier to read and uses intuitive terminology.

An interesting recent article from Uncle Bob on (unit) testing:
http://blog.cleancoder.com/uncle-bob/2017/05/05/TestDefinitions.html

A performance (with a new update to Visual Studio 2017) comparison was released a few months after picking our testing framework, comparing NUnit, xUnit and MS Test 2. You can find the blog post from Microsoft here.

References:

http://georgemauer.net/2015/05/01/why-not-mstest
https://seankilleen.com/2015/06/xUnit-vs-MSTest/
https://stackify.com/unit-test-frameworks-csharp/
https://xunit.github.io/docs/why-did-we-build-xunit-1.0.htmlhttp://blog.cleancoder.com/uncle-bob/2017/05/05/TestDefinitions.html

Comments

[Andrew Hill]

18 months later, how do you feel about your decision? I'll be making a similar one soon for a new project.

[Chris Mathurin]

Sorry for the late reply. I've moved on to another company, but we were happy with our choice. I am using MSTest, we are not looking into migrating to another framework for now 😅 I miss xUnit!

[Miguel Cudaihl]

My only gripe with Xunit is I can't print out to console from within my sut. The author's explanation on why is based on the purist's way of thinking. However, there are practical reasons why that feature would've been helpful.

All three are pretty much at parity feature wise.

[Tengiz Tutisani]

Besides xBehave, I think it's worth mentioning other BDD test frameworks (e.g., Xunit.Gherkin.Quick especially that it was written for Xunit with the same motivations). xBehave keeps both the Gherkin-like text and the code in the same place, which creates coupling. I believe that they are easier to maintain and use for the right purpose when they are separate (which is what Xunit.Gherkin.Quick allows).

Disclaimer - I am the author of the referenced BDD framework. I found this article because I was wondering whether there is a demand for nUnit with BDD too.

[Chris Mathurin]

Hi Tengiz, thanks for the feedback. I've only included xBehave as an example. I'll update to include Xunit.Gherkin.Quick. Cheers.

[Gaines Kergosien]

Thanks for sharing your experience evaluating unit test frameworks and providing links to useful resources. The insight was helpful as we face a similar decision.

[Chris Mathurin]

You are welcome :)

[Camilo Martinez]

There is no enough differences between theirs. Finally choose which one, but the better decision is that the team feel confortable using it.

For me xUnit and my team, choose xUnit because its part of .Net Foundations, I like his syntaxis and works like a charm with Test Explorer plugin with VSCode.

That's how we setup unit testing and code coverage. Maybe can help other.

.Net Core Unit Test and Code Coverage with Visual Studio Code

Camilo Martinez ・ Sep 28 '18 ・ 6 min read

#dotnetcore #productivity #developertool #vscode

[setagana]

Given what you know now with the performance benchmarks in the Microsoft blog, would you still make the same choice or would you have stuck with NUnit?

[Chris Mathurin]

I still stick with xUnit, as I think that the extra seconds that I will gain does not outweigh the benefits of xUnit or significant enough to make me switch from what a framework that will make me write better and cleaner tests. I am also hoping that this gap between the two will be smaller in the future.

[Raphaël Moreau]

xUnit was also created by one of the original authors of xUnit.

I guess you meant nUnit ? :)
Nice article, it answers to my question !

[Chris Mathurin]

Thanks Raphaël, i've now corrected this :)

[jarzynam]

Actually MS Tests supports parameterized tests since 2016 (and yes, that was quite surprise for me either).

[Chris Mathurin]

That's correct, I'll update the post. I have used MSTests with parameterised tests before, but I am not a fan of the implementation. I prefer the xUnit way compared to NUnit and MSTest.