The Evolution of Assertions in Java Tests

As a developer, I certainly qualify as a testing freak. I absolutely love automated tests that produce meaningful output. Today, I want to focus on the history and state of the art in automated testing, more specifically: on the assertions.

Disclaimer: we are talking about assertions for Tests in this post. Other kinds of assertions (such as pre- and post-conditions and the assert keyword) are beyond the focus of this post.

Some Code to Test

An automated test is not a proper test without at least one assertion (except for smoke tests). For this article, we will test the following Person class:

public class Person {

    private String name;
    private Date birthDate;
    private final Set<String> hobbies = new HashSet<>();

    public Person(){}

    public String getName() { 
        return this.name; 
    }

    public void setName(String name){ 
        this.name = name; 
    }

    public Date getBirthDate() { 
        if(this.birthDate == null) { 
            return null; 
        }
        return new Date(this.birthDate.getTime()); 
    }

    public void setBirthDate(Date date) { 
        this.date = new Date(date.getTime()); 
    }

    public Set<String> getHobbies() { 
        return Collections.unmodifiableSet(this.hobbies); 
    }

    public void setHobbies(Set<String> hobbies) {
        this.hobbies.clear();
        if(hobbies != null) { 
            this.hobbies.addAll(hobbies); 
        }
    }

}

Disclaimer: I am well aware that testing a bean class and it's accessors is not something one would typically do. However, it serves as a nice, simple example that allows us to focus on the assertions themselves.

The Primordial Assertion

There most basic way to state an assertion in Java (without any frameworks) is the following:

if(someCondition){
    throw new AssertionError("Hey I didn't expect that!");
}

No matter how fancy your assertion framework is on the surface, in the end it will always boil down to this. There are a couple of ways in which assertion frameworks have improved over this initial assertion method:

• Make it more concise. Reduce the amount of code that needs to be written.
• Make it more fluent. Provide some kind of builder syntax.
• Auto-generate the message such that it never goes out of sync with the test.

Enter JUnit Assert

JUnit ships with a class which is simply called Assert. This class consists of a series of static methods that should help the user in writing concise assertions. Most of the methods have the following shape:

public static void assertXYZ(message, expectedValue, actualValue) { /* ... */ }

... where message is an optional string that is printed instead of the auto-generated message if the assertion fails. Tests in this fashion look like this:

import org.junit.*

public class PersonTest {

    @Test
    public void testWithAssert(){
        Person p = new Person();
        Assert.assertNull("A new Person should not have an initial name.", p.getName());
        p.setName("John Doe");
        Assert.assertEquals("John Doe", p.getName());
    }

}

In addition, one would typically use import static org.junit.Assert.* to statically import the assertion methods. This way, the Assert. in front of an assertion can be omitted.

This is already a big step forward from the initial if construction: it fits in one line. However, there are several problems with this approach:

• It is really REALLY easy to mess up the assertion call and swap expected and actual.
• As the message is always the first parameter, and this parameter also happens to be optional, it is not always immediately clear if the first passed string is the expected value or the message.
• The auto-generated assertion error messages were rather basic, and the number of available assertion methods was quite limited.

Hamcrest! Wait... what?

Several years after JUnit was released, a nifty little library by the (very odd) name of Hamcrest was released. Hamcrest was built around the idea that an assertion is essentially a match criterion, and as such it should be represented by a Matcher object. This allows for a greater variation of assertions with better error messages and a more fluent syntax:

@Test
public void hamcrestTest(){
    Person p = new Person();

    assertThat(p.getHobbies(), is(empty());
    assertThat(p.getName(), is(nullValue());

    p.setName("John Doe");
    p.setBirthDate(new Date());
    Set<String> newHobbies = new HashSet<>();
    newHobbies.add("coding");
    newHobbies.add("blogging");
    newHobbies.add("reading");
    p.setHobbies(newHobbies);

    assertThat(p.getName(), is("John Doe");
    assertThat(p.getHobbies(), not(hasItem("programming"));
    assertThat(p.getHobbies(), containsInAnyOrder("coding", "blogging", "programming");
    assertThat(p.getHobbies().size(), is(3));
    assertThat(p.getBirthDate().getTime(), is(greaterThan(0L));

}

As you can see, the assertThat(value, matcher) method is the entry point. Unlike assertEquals, it is perfectly clear which is the expected and which is the actual value, so that's a big plus right out of the gate. A downside is that, due to the fact that assertThat(...) has so many different overloads, you cannot use asserThat(p.getName(), is(null)), because null makes it ambiguous which override to use. Instead, you need to use nullValue(), which is essentially just a matcher that checks for equality with null.

Hamcrest also introduced negated conditions with not(...), easy numeric comparisons, as well as helpers for collections. All of these (in particular the collection helpers) generate quite useful error messages on their own right, so providing a custom error message (while possible) is usually not necessary anymore.

The downside of Hamcrest is that it relies heavily on static imports, which may even cause import conflicts (if you also use static methods a lot internally). Another drawback is that, while the assertion lines are now fluent to read, they are not actually very fluent to write:

// alright, let's test the name...
p.getName()|

// ... oh, I forgot the assertThat...
assertThat(|p.getName()

// ... moves carret all the way back again...
assertThat(p.getName(), |

// ok IDE, I need the "is" method, do your magic!
assertThat(p.getName(), is|

// (200 completion proposals show up)
// *sigh*
assertThat(p.getName(), is("John Doe"));|

// finally!

See what I mean? Luckily, some folks out there were hitting the same issues.

Truth be told!

Truth is a library similar to Hamcrest, except that it offers a fluent builder API:

@Test
public void testTruth(){
    Person p = new Person();

    assertThat(p.getName()).isNull();
    Set<String> newHobbies = new HashSet<>();
    newHobbies.add("coding");
    newHobbies.add("blogging");
    newHobbies.add("reading");
    p.setHobbies(newHobbies);

    assertThat(p).containsExactly("coding", "blogging", "reading");
}

This is now finally a fluent API for tests, and one that also doesn't require too many static imports. Just follow the code completion of your IDE and you will create powerful assertions in no time.

As always, there are some caveats here as well. My most pressing concern with this solution is that, in comparison to Hamcrest, it is very difficult to extend. The Truth library itself is not under your control (unless you fork it...), so you cannot simply add new methods to existing classes to do your custom assertions.

Testing how it shouldBe

We now leave the safe haven of Java and venture out into the wild. As it turns out, Kotlin lends itself well to build a testing mini-framework which consists of only two functions (and the Hamcrest library):

infix fun <T> T.shouldBe(other: T): Unit {
    assertThat(this, is(other));
}

infix fun <T> T.shouldBe(matcher: Matcher<T>): Unit {
    assertThat(this, matcher);
}

"How does this help" and "what in blazes is this" you may ask. Well, we are defining two extension functions that reside on Object (or, in Kotlin: Any). Which means: we can now call x.shouldBe(...) on anything, no matter what x is. In addition, it is an infix function, which means that we can drop the dot, the opening and the closing brace.

Check it out:

@Test
fun testKotlin(){
    Person p = Person()

    p.name shouldBe null
    p.name = "John Doe"
    p.name shouldBe "John Doe"

    p.hobbies = setOf("coding", "blogging", "reading")
    p.hobbies.size shouldBe 3

    p.birthDate = Date()
    p.birthDate!!.time shouldBe greaterThan(0L)

}

Now this is the kind of readability that I am looking for!

Further Reading

If you feeling adventurous, I also recommend taking a look at

• Spock (Groovy Testing)
• Cucumber (Origin of the Gherkin test syntax)

Closing Words

I hope you enjoyed this little excursion to assertion libraries. Feel free to share your experiences and provide recommendations on libraries and/or coding styles that I have missed!

Comments

[Casey Brooks]

I love how Kotlin opens up an entirely new way to tackle these problems that have seemed so solved for years. I recently found a new Kotlin assertion library based entirely on using extension functions to create assertions that you might be interested in, Strikt. It's still an early project and the API is likely to change through iteration, but its already my favorite one out there and I'm using it in nearly all my projects

[Martin Häusler]

Nice one, I didn't know about Strikt before. Definitly looks interesting!