Miguel Bernard

Posted on Apr 5, 2020 • Edited on Apr 13, 2020 • Originally published at blog.miguelbernard.com

What's wrong with the current state of unit testing

#testing #propertybasedtesting #dotnet #csharp

Unit testing nowadays is pretty much always done using a technique called example-based testing. The method is simple; we run a series of examples against an algorithm, and we validate that we get the expected result.

e.g.

[Fact]
public void GivenThree_WhenCalculate_ThenFiveIsReturned()
{
    // Given
    var input = 3;

    // When
    var result = Calculator.Calculate(input);

    // Then
    Assert.Equal(5, result);
}

This technique is relatively easy to understand and implement. However, it's far from being all rainbows and unicorns. In this article, we'll go over property-based testing, an alternative approach that shines where example-based testing tends to fall short.

The main problems with those tests are that:

They don't scale
They don't help you find edge cases
They are brittle
They don't explain the requirement
They tend to be too specific to the implementation

We all do this

If you are like most of us, mortals, you are probably terrible at using TDD, which means that in the best-case scenario, you add your tests after coding the actual method. Moreover, I bet that you write an empty test shell calling your method under test, take the output, and then write your assertion with that value. Let's be honest. The temptation to reach a better code coverage is just too strong that we have all done it at some point. Then, if you have a bug in your code, you effectively only coded a buggy test.

Property-based testing

Property-based testing is an alternative approach that can deal with all the shortcomings of example-based tests. However, it's by no means a full replacement, as example-based tests are still excellent in many cases. The best approach is to have the right mix of the two in your test suite.

What's a property

When we are talking about properties here, we don't mean properties of C# classes. We use the term property here in a more abstract way. Think of it as a characteristic, trait, attribute, feature, or quality of an algorithm.

Simple definition:

A property is a condition that will always be true for a set of valid inputs.

Testing method

The first thing to do when you want to create a property-based test is to understand and represent the relationship between inputs and outputs. Then, use randomly generated inputs to express deterministic expected results.

Said like that, it seems complex, but you'll see, it's more straightforward than it sounds.

Example

Let's try the technique with a simple example; the Add method. The signature looks like this.

int Add(int number1, int number2);

If you try to test this method with examples, you'll never be done for sure, as there's an infinite combination of numbers that you can add together. So how can we be sure that our Add method is behaving correctly in every situation?

First, try to think of what makes the addition different from the other mathematical operations like subtraction and multiplication.

Properties

The first property is called commutativity. For those of you that don't have a math major, here is a simple definition:

When I add two numbers, the result should not depend on the order of the parameters.

2+3 = 3+2  // True
2-3 != 3-2 // False
2*3 = 3*2  // True

This property is true for addition and multiplication, but not for subtraction. Translated to a test in C#:

// When
var res1 = Add(input1, input2);
var res2 = Add(input2, input1);

// Then
Assert.Equal(res1, res2);

The second property is called associativity.

When I add two numbers, the order of the operations doesn't matter.

(2+3)+4 = 2+(3+4)  // True
(2-3)-4 != 2-(3-4) // False
(2*3)*4 = 2*(3*4)  // True

Again, this property is true for addition and multiplication, but not for subtraction. Translated to a test in C#:

// When
var res1 = Add(Add(input1, input2), input3);
var res2 = Add(input1, Add(input2, input3));

// Then
Assert.Equal(res1, res2);

The third and last property of the addition that makes it unique is called identity.

Adding zero is the same as doing nothing.

2+0 = 2  // True
2-0 = 2  // True
2*0 != 2 // False

At last! We found something to differentiate the addition and the multiplication. Translated to a test in C#:

Assert.Equal(Add(input, 0), input);

Conclusion

With three simple tests, we've been able to cover an infinite number of examples. With these, it's impossible to write an implementation of the Add method that doesn't behave properly. That's precisely the way mathematicians define concepts. They describe the properties. If you look up the definition of the addition, you'll find something interesting. It's defined as commutativity, associativity, and identity.