Anti-pattern: manually iterating over test data
You probably have seen, or even coded yourself, a unit test that looks something like this:
[Test]
public void PokemonAreWaterType()
{
var pokemons = new List<string>()
{
"Vaporeon",
"Magikarp",
"Squirtle"
};
foreach (var pokemon in pokemons)
{
bool isWaterType = Pokedex.IsPokemonWaterType(pokemon);
Assert.IsTrue(isWaterType, $"Pokemon '{pokemon}' should be water type.");
}
}
You have a list of values that you want to test within the same test method, and you are looping over an array to perform the test. There are many problems with the code above:
- While the above test is very simple and we can understand what is happening, this will rarely happen in real life.
- We should also avoid nesting, the simpler the test, it will be easier to maintain.
- If more than two test cases are failing, you will only get the results of one, as the test short circuits the first time an assert fails. Sure, you can save all the results in a results array, then assert on this array at the end, but you are again adding complexity to what should be a simple test.
Solution: parametrized testing
All the commonly used C# test frameworks support parametrized tests:
- MSTest: https://learn.microsoft.com/en-us/visualstudio/test/how-to-create-a-data-driven-unit-test?view=vs-2022
- NUnit: https://docs.nunit.org/articles/nunit/writing-tests/attributes/testcasesource.html
- XUnit: https://xunit.net/docs/getting-started/netfx/visual-studio#write-first-theory
Data-driven tests, or parametrized tests, as you may wish to call them, take advantage of the test framework to simplify testing over a data source. This data source could be hardcoded values, collections, or any other data source.
We will create parametrized tests for all three common testing libraries in C#: MSTest, NUnit, and XUnit.
For the three test frameworks, we will showcase three unit test examples:
- Unit test manually iterating over the test data.
- Using attributes for each row of the test data.
- Using a static array as the test data source.
The code used in this post can be found at: https://github.com/rogeliogamez92/blog-data-driven-testing
Example application: Pokémon Factory
We will create a Pokémon factory that takes two strings, name, and type, and tries to create a valid Pokémon. If the type is not supported by our application, it will create a Pokémon with an invalid type.
Pokémon class:
public class Pokemon
{
public enum PokemonType
{
UnsupportedType,
Grass,
Fire,
Water
}
public readonly string Name;
public readonly PokemonType Type;
internal Pokemon(string name, PokemonType type)
{
Name = name;
Type = type;
}
}
Pokémon factory class:
public class PokemonFactory
{
public Pokemon CreatePokemon(string name, string type)
{
if (!Enum.TryParse(type, ignoreCase: true, out Pokemon.PokemonType pokemonType))
{
pokemonType = Pokemon.PokemonType.UnsupportedType;
}
return new Pokemon(name, pokemonType);
}
}
While this is not a true implementation of the factory method pattern, if you want to know more about it, I strongly recommend you to read: https://refactoring.guru/design-patterns/factory-method.
Parametrized tests in MSTest
Let's start manually iterating over the test data:
[TestMethod]
public void IteratingOverData()
{
/// Tuple(PokemonName, PokemonType, ExpectedPokemonType
var pokemonData = new List<Tuple<string, string, Pokemon.PokemonType>>()
{
new ("Bulbasaur", "Grass", Pokemon.PokemonType.Grass),
new ("Charmander", "Fire", Pokemon.PokemonType.Fire),
new ("Squirtle", "Water", Pokemon.PokemonType.Water),
new ("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType),
};
var pokemonFactory = new PokemonFactory();
foreach (var pd in pokemonData)
{
var pokemon = pokemonFactory.CreatePokemon(pd.Item1, pd.Item2);
Assert.AreEqual(pokemon.Type, pd.Item3);
}
}
We have our data in an array that we iterate and assert on each item. It works, but there is something off.
Let's see the same test using the DataRow attribute from MSTest:
[TestMethod]
[DataRow("Bulbasaur", "Grass", Pokemon.PokemonType.Grass)]
[DataRow("Charmander", "Fire", Pokemon.PokemonType.Fire)]
[DataRow("Squirtle", "Water", Pokemon.PokemonType.Water)]
[DataRow("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType)]
public void UsingDataRow(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.AreEqual(pokemon.Type, expectedPokemonType);
}
At first glance, we can see an aesthetic improvement. That ugly foreach is no more.
Now, what happens when there are errors in the Fire and Water test cases? We will append "Error" to both input strings for type.
[TestMethod]
public void IteratingOverData()
{
/// Tuple(PokemonName, PokemonType, ExpectedPokemonType
var pokemonData = new List<Tuple<string, string, Pokemon.PokemonType>>()
{
new ("Bulbasaur", "Grass", Pokemon.PokemonType.Grass),
new ("Charmander", "FireError", Pokemon.PokemonType.Fire), // ERROR
new ("Squirtle", "WaterError", Pokemon.PokemonType.Water), // ERROR
new ("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType),
};
///...
[TestMethod]
[DataRow("Bulbasaur", "Grass", Pokemon.PokemonType.Grass)]
[DataRow("Charmander", "FireError", Pokemon.PokemonType.Fire)] // ERROR
[DataRow("Squirtle", "WaterError", Pokemon.PokemonType.Water)] // ERROR
[DataRow("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType)]
public void UsingDataRow(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
///...
Let's see the difference in Visual Studio 2022 Test Explorer:
Using DataRow is so much better! If we manually iterate we would need to jump through hoops in code to try to give a similar report as the method using DataRow.
What if we want to reuse the test data? In MSTest we first declare the test data array, and reference it using the DynamicData attribute:
public static IEnumerable<object[]> PokemonData => new[]
{
new object[] { "Bulbasaur", "Grass", Pokemon.PokemonType.Grass },
new object[] { "Charmander", "Fire", Pokemon.PokemonType.Fire },
new object[] { "Squirtle", "Water", Pokemon.PokemonType.Water },
new object[] { "ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType },
};
[TestMethod]
[DynamicData(nameof(PokemonData))]
public void UsingDynamicData(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.AreEqual(expectedPokemonType, pokemon.Type);
}
Now we can reference that same array in as many tests as we want.
Parametrized tests in NUnit
For NUnit, we use the TestCase attribute instead of DataRow:
[TestCase("Bulbasaur", "Grass", Pokemon.PokemonType.Grass)]
[TestCase("Charmander", "Fire", Pokemon.PokemonType.Fire)]
[TestCase("Squirtle", "Water", Pokemon.PokemonType.Water)]
[TestCase("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType)]
public void UsingTestCase(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.AreEqual(expectedPokemonType, pokemon.Type);
}
And if we want to use an array, we use the TestCaseSource:
public static IEnumerable<object[]> PokemonData => new[]
{
new object[] { "Bulbasaur", "Grass", Pokemon.PokemonType.Grass },
new object[] { "Charmander", "Fire", Pokemon.PokemonType.Fire },
new object[] { "Squirtle", "Water", Pokemon.PokemonType.Water },
new object[] { "ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType },
};
[TestCaseSource(nameof(PokemonData))]
public void UsingTestCaseSource(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.AreEqual(expectedPokemonType, pokemon.Type);
}
Parametrized tests in XUnit
For XUnit, for parametrized tests, we use the Theory attribute instead of Fact.
Instead of DataRow we use the InlineData attribute:
[Theory]
[InlineData("Bulbasaur", "Grass", Pokemon.PokemonType.Grass)]
[InlineData("Charmander", "Fire", Pokemon.PokemonType.Fire)]
[InlineData("Squirtle", "Water", Pokemon.PokemonType.Water)]
[InlineData("ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType)]
public void UsingInlineData(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.Equal(expectedPokemonType, pokemon.Type);
}
And for a data array, the MemberData attribute:
public static IEnumerable<object[]> PokemonData => new[]
{
new object[] { "Bulbasaur", "Grass", Pokemon.PokemonType.Grass },
new object[] { "Charmander", "Fire", Pokemon.PokemonType.Fire },
new object[] { "Squirtle", "Water", Pokemon.PokemonType.Water },
new object[] { "ProfessorOak", "Human", Pokemon.PokemonType.UnsupportedType },
};
[Theory]
[MemberData(nameof(PokemonData))]
public void UsingMemberData(string pokemonName, string pokemonType, Pokemon.PokemonType expectedPokemonType)
{
var pokemonFactory = new PokemonFactory();
var pokemon = pokemonFactory.CreatePokemon(pokemonName, pokemonType);
Assert.Equal(expectedPokemonType, pokemon.Type);
}
Final thoughts
In this post, we went over all the three most common C# test frameworks and showed how we can write parametrized tests to avoid manually iterating over the test data.
Do not stop with just what I showed you today. It is cool being able to share the same test data array between several tests, but these frameworks are very complete. The three of them support other types of data sources such as files and services. I encourage you to go to their site and read the documentation of your framework of choice:
- MSTest: https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-mstest
- NUnit: https://docs.nunit.org/
- XUnit: https://xunit.net/#documentation
You can find the code of this post at: https://github.com/rogeliogamez92/blog-data-driven-testing
Top comments (1)
"This is the way!"