DEV Community: sergio-sastre

Better unit tests with Property-Based testing

sergio-sastre — Sat, 14 Aug 2021 13:03:08 +0000

In the previous post, we saw how Parameterised Tests help write more focused, concise and scalable tests. However, Parameterised Tests belong to the so-called Example-Based Tests: the tests run only with the input and expected values we choose to pass as examples. And this arises the question: what if the tests would not pass with any other valid examples we have not picked?

Back to the Password Validator Example

One of the reasons to write tests is that they protect us against regression bugs. They serve as a tool to detect code changes that cause a feature that worked correctly to stop working. If that happens, the corresponding tests turn red and point out what got broken.

Coming back to the PasswordValidator of my previous post, the final Parameterised Test looked like this:

val passwordValidator = PasswordValidator(
     ContainsUpperCaseLetterValidator(),
     MinCharsValidator(6),
     ContainsDigitValidator(),
     ContainsLowerCaseLetterValidator(),
     NoBlanksValidator()
 )

@DisplayName("PasswordValidator for invalid passwords")
@ParameterizedTest(name = "When password is \"{0}\", the error contains \"{1}\"")
@CsvSource(
    "123456, no upper case letters",
    "ABCDEF, no digits",
    "HELLO, no lower case letters",
    "1234A, contains less than 6 chars",
    "12 3 456, contains blanks"
)
fun testPasswordValidatorRight(password: String?, expectedError: String?) {
    val actualError = passwordValidator.validate(password)
    assertThat(actualError).contains(expectedError)
}

Let's imagine now that you need to make some adjustments in the Validators. You finish it and push your code. However, It turns out that another teammate has also touched the Validators, and you are having some conflicts. You solve the conflicts and push the code again. Finally, your tests run green in your CI, so you are confident enough to merge your code.

But, before that happening, your teammate reviews the code and requires you to do some code changes. He found out that the ContainsUpperCaseLetterValidator implementation is incorrect.

So you take a look...

class ContainsUpperCaseLetterValidator : RequirementValidator {
    override val keywordOnError: String? = "no upper case letters"

    override fun isValid(password: String?): Boolean =
        password?.contains("[a-z]".toRegex()) == true
}

You can see the code snippet of this broken validator here

Yes, it turned out that it now validates lower case (i.e. "[a-z]") instead of upper case chars (i.e. "[A-Z]").
On the other hand, you feel confused. All tests ran green on the CI. What could be wrong? So you run the test again locally.

Still green. We know it should run red since we have detected the error in the implementation itself.
But take a closer look at the ContainsUpperCaseLetterValidator first Example-based Test.

There we input the password "123456", so passwordValidator.validate("123456") should return an error message, not only for ContainsUpperCaseValidator, but also for ContainsLowerCaseValidator as well, since it does contain neither upper case nor lower case chars, just digits.

In this case, we can fix the test (but not the implementation) by adding an extra example for ContainsUpperCaseValidator containing a lower case char in the password input, for instance "a23456". In doing so, the test runs red, as expected.

Click on this link to see the code of all the Parameterised Tests

This proves that choosing the right examples is very important to catch regression bugs, as well as tricky. How reliable is our test suite? How can we be sure that we have covered all the necessary cases with this extra example? what if we missed some edge cases? how many examples do we need to be confident enough?

Property-Based Tests to the rescue

Following the previous example, we could conclude that any password without at least 1 upper case char, must show the error message "does not contain upper case chars". This combination of preconditions and qualities that must be valid at every time is called property.

What if instead of just providing concrete valid examples through Parameterised Tests, we would also generate random valid values for the precondition (no upper case chars) that must always fulfil the quality (shows "does not contain upper case chars" in the error message)? This is called Property-Based Testing (PBT).

In my opinion, the most powerful PBT engine for JVM languages out there is Jqwik. I find it much more flexible than Kotest or QuickUnit when it comes to writing custom value generators. Moreover, it provides unique features such as collecting statistics for the generated inputs, what comes in handy to ensure that the generated random values are among those expected.
Therefore, all the examples of this article will be written with Jqwik.

Configuration

Jqwik is very easy to configure and can run together with Junit4 and Junit5. In an Android project you just need to add the following to your project gradle file.

android {
    ...
    testOptions {
        unitTests.all {
             useJUnitPlatform {
                  //this config supports Jqwik, Junit4 and Junit5
                  includeEngines 'jqwik', 'junit-jupiter', 'junit-vintage'
             }
             include '**/*Properties.class'
             include '**/*Test.class'
             include '**/*Tests.class'
        }
    }
}

dependencies {
     ...
     testImplementation "net.jqwik:jqwik-api:1.5.1"
     testImplementation "net.jqwik:jqwik-engine:1.5.1"
}

For other java/jvm projects you can check the Jqwik doc .

Our first Property-Based Test

Property-Based Tests require to provide a Generator that creates the constrained values programatically or through an annotation.
In Jqwik, you can create a Generator with @Provide, which must return a Jqwik Arbitrary<String> for our arbitrary password.
Our generator, which must generate random strings without upper case chars, is as follows

@Provide
fun noUpperCase(): Arbitrary<String> =
    Arbitraries.strings().ascii().filter { it.matches("[^A-Z]".toRegex()) }

Nevertheless, this does not ensure us that the generated input is among the values we expect. As I mentioned before, one of the special features of Jqwik is collecting statistics for the generated inputs. In doing so, we ensure the correctness of the generated inputs. I strongly recommend to always add them to your Property-Based Tests.

According to our "strong password" rules, makes sense to watch for digits, as well as lower and upper case chars in our tests. The corresponding methods might look like this in Jqwik.

fun statsUpperCase(password: String?){
    val withUpperCase = password?.contains("[A-Z]".toRegex())
    Statistics
        .label("Upper case")
        .collect(if (withUpperCase == true) "with upper case" else "without upper case")
}
...

fun collectPasswordStats(password: String?){
    statsUpperCase(password)
    statsLowerCase(password)
    statsDigits(password)
    ...
}

See the full implementation of the stat collectors here

By adding statistics on the generated inputs you feel more confident about testing the right thing. The reports will contain those statistics as you'll see later.

Once we have the generator and the statistics, our tests would look like this

@Label("If password without upper case chars," +
       "the error message contains 'no upper case letters'"
)
@Property
fun testPasswordValidatorRight(@ForAll("noUpperCase") password: String?) {
     collectPasswordStats(password)

     val actualError = passwordValidator.validate(password)
     Truth.assertThat(actualError).contains("no upper case letters")
}

You can find the full test suite with PBT examples for all the validators as well as the random value generators here

And if we run it, we get a report similar to the one below

Great! The test fails as expected, and 100% of the generated values are upper case free. As you can see, Jqwik Generators take edge cases into account as well (more about edge cases here ) by default.

Why flaky tests is not an issue in PBT

You might be thinking... the test fails, but the values are being generated randomly on every run... So if I run the test again, new values are generated, and could happen that the test does not fail for those new random values. Non-reproducible tests decrease reliability on your test suite and need to be avoided.

But do not worry.
First of all, Jqwik reuses the same seed to generate random values until a green run by default.
Secondly, if you need to reproduce the tests with the same values at any time, just add the seed shown in the previous report, at the very bottom, to the test

@Label("If password without upper case chars," +
       "the error message contains 'no upper case letters'"
)
@Property(seed = "-7293214509268013126")
fun testPasswordValidatorRight(@ForAll("noUpperCase") password: String?) {
     collectPasswordStats(password)

     val actualError = passwordValidator.validate(password)
     Truth.assertThat(actualError).contains("no upper case letters")
}

Now you can run the test continuously with the same values that made it fail.

Do not forget to remove the seed once fixed. That way the generators will continue generating new random values on every run. In doing, so you might find out other values for which the implementation is incorrect.

Congratulations, your tests have become more robust to regression bugs! You are officially ready to catch'm all!

Conclusion

Let's take a look at the Pros and Cons of Property-Based tests.

Pros

Tests become more robust, and give you more confidence on your code.
Tests are deterministic. It's possible to reuse the seed that made them fail, so that errors can be reproduced.
Explores edge cases by default, which we might forget otherwise.

Cons

Some properties are hard to find, especially while initiating in PBT. You can take a look at this interesting post by Scott Wlaschin on how to find properties.
Property-Based Tests run more slowly since they execute 1000 times by default. However, most engines allow you to change that value if desired.
Properties are not always sufficient to test correctness. Take a look at the wrong implementation of String?.reversed() and its test below

fun String?.reversed() : String? = this //forgot to reverse it?

@Label("Reversing a reversed String returns the original")
@Property
fun reverseStringTest(@ForAll originalString: String?) {
    assertTrue(
       originalString?.reversed()?.reversed() == originalString
    )
}

The test would pass though, giving us the false impression that the method functionality is correct. On the other hand, this is the kind of error that Example-Based Tests spot quickly.

Therefore, getting the best of both worlds with a combination of Example-Based and Property-Based Tests work the best in most cases.

All the code examples used in this article, as well as those of the first article of this series - Better Unit Tests with Parameterized Tests - are available under this repo.

Did you like this post?
You might also like to take a look at other articles I've written

Cover photo by Brett Jordan on Unsplash

Better unit tests with Parameterized testing

sergio-sastre — Tue, 27 Jul 2021 17:25:23 +0000

While writing Unit Tests, we often verify that given an input, the output of the method under test is the one expected. When you execute the same test with a different set of inputs and expected outputs several times, you come up with Data Driven Tests or Parameterized Tests.
In this first post I will explain how we can multiply not only the quantity, but also the quality of our tests with this methodology. The samples are in Junit5 and written in Kotlin but apply to Java as well.

*To configure Junit5 in Android projects (which I strongly recommend at least for JVM tests), you can find a complete guide here

A common test case: a password validator

Nowadays, most apps require the user to create an account with a "strong password". Let's assume we need to validate on the client side that the user provides a strong password, namely:

Contains at least 1 digit in lower case . Show "no digit" otherwise
Contains at least 1 char in upper case. Show "no upper case letters" otherwise
Contains at least 6 digits. Show "contains less than 6 chars" otherwise Although you can take another approach to implement it, I've used a Composite pattern: PasswordValidator (Composite) and Validators (Leaves) that accepts a vararg of Validators (Leaves) as argument in its constructor, each value validating one single requirement.

In order to test that our Password validator works, we write a single test with an assert for each requirement as follows

@Test
fun passwordValidatorTest() {
    val minCharsCount = 6
    val passwordValidator = PasswordValidator(
        ContainsUpperCaseLetterValidator(),
        MinCharsValidator(minCharsCount),
        ContainsDigitValidator()
    )

    assertThat(
        passwordValidator.validate("123456")
    ).contains("no upper case letters")

    assertThat(
        passwordValidator.validate("1234A")
    ).contains("contains less than $minCharsCount chars")

    assertThat(
        passwordValidator.validate("ABCDEF")
    ).contains("no digits")
}

If you are not new to Unit Tests, I'm sure you have heard that you should strive to have only one assert per test. In doing so, your tests are self-contained and clearly state what is being tested.

Moreover, asserts fail hard by default in Junit5. It means, whenever one assert fails, the test execution exits right away with the corresponding error. The subsequent asserts in the tests will not run and therefore no further errors will be reported. Soft Assertions would solve that problem though (assertAll in Junit5), but it does not scale well in this case. Tests would continue becoming longer and longer.

For instance, imagine that after a while, the requirements for a strong password get more strict and we also need to validated the following on password creation:

Contains at least 1 char in lower case. Show "no lower case letters" otherwise
Contains no blanks. Show "contains blanks" otherwise While adding the corresponding validators, our test becomes this:

@Test
fun advancedPasswordValidatorTest() {
    val minCharsCount = 6
    val passwordValidator = PasswordValidator(
        ContainsUpperCaseLetterValidator(),
        MinCharsValidator(minCharsCount),
        ContainsDigitValidator(),
        ContainsLowerCaseLetterValidator(),
        NoBlanksValidator()
    )

    assertThat(
         passwordValidator.validate("123456")
     ).contains("no upper case letters")

     assertThat(
          passwordValidator.validate("1234A")
     ).contains("contains less than $minCharsCount chars")

     assertThat(
          passwordValidator.validate("ABCDEF")
     ).contains("no digits")

     assertThat(
          passwordValidator.validate("HELLO")
     ).contains("no lower case letters")

     assertThat(
          passwordValidator.validate("12 3 456")
      ).contains("contains blanks")
}

*You can find the complete set of initial and advanced strong password samples with multiple asserts here

At a glance, we notice the following:

The test becomes longer: long code is generally more complicated, harder to read, and harder to maintain. The more requirements, the longer it becomes due to the extra asserts.
The test becomes less focused: the more password requirements, the more asserts added. The more asserts, the harder to see what the test is actually validating.

One test per requirement:

brief and focused tests! ...but repetitive

By creating one test per requirement we would solve both issues. Every test would assert only one thing. If a test fails, we know directly what is failing.

val passwordValidator = PasswordValidator(
     ContainsUpperCaseLetterValidator(),
     MinCharsValidator(6),
     ContainsDigitValidator(),
     ContainsLowerCaseLetterValidator(),
     NoBlanksValidator()
 )

@Test
fun passwordValidatorNoUpperCaseTest_showsError() {
    assertThat(
         passwordValidator.validate("123456")
     ).contains("no upper case letters")
}

@Test
fun passwordValidatorNoLowerCaseTest_showsError() {
     assertThat(
          passwordValidator.validate("HELLO")
     ).contains("no lower case letters")
}

@Test
fun passwordValidatorNoDigitsTest_showsError() {
     assertThat(
          passwordValidator.validate("ABCDEF")
     ).contains("no digits")
}
...

*You can find the full set of tests with one assert here

On the other hand, all tests look very repetitive. If we add more requirements, the new tests are basically a copy-paste of any of the previous tests* but with different input and expected output values. They **do not scale well.

We can do better. That's where Parameterized Tests come in handy...

Parameterized Tests:

brief and focused tests! ...and also concise and scalable!

Parameterized Tests enable us to execute a single test method multiple times with different parameters. In Junit5, we can use the @ParameterizedTest annotation for that purpose. In doing so, we can pass as input the password under test and its expected error message, solving the problems we were facing.

And the resulting Parameterized Test would look like this:

val passwordValidator = PasswordValidator(
     ContainsUpperCaseLetterValidator(),
     MinCharsValidator(6),
     ContainsDigitValidator(),
     ContainsLowerCaseLetterValidator(),
     NoBlanksValidator()
 )

@DisplayName("PasswordValidator for invalid passwords")
@ParameterizedTest(name = "When password is \"{0}\", the error contains \"{1}\"")
@CsvSource(
    "123456, no upper case letters",
    "ABCDEF, no digits",
    "HELLO, no lower case letters",
    "1234A, contains less than 6 chars",
    "12 3 456, contains blanks"
)
fun testPasswordValidatorRight(password: String?, expectedError: String?) {
    val actualError = passwordValidator.validate(password)
    assertThat(actualError).contains(expectedError)
}

*You can find the complete set of Parameterized Tests here

Apart from @CsvSource, Junit5 comes with other annotations to provide inputs, such as @CsvFileSource, @EnumSource or even @MethodSource for more complex input sets. You can take a deeper look at them here to find which one fits your needs better.

And if we run the test, we get the following result:

As you can see, parameterizing your tests has the following advantages:

Your tests are scalable: every new requirement involves adding one new pair of input and expected output to the parameters we inject into the tests. No more copy-pasting of tests. And what if you need to add an edge case you forgot? No need for extra asserts or extra tests. Again, simply add new data to your test.
Your tests are concise: the names of the tests become more concise since they can include the value of the parameters. What tells more about what we are actually testing? What tells the sheer truth? passwordValidatorNoDigitsTest_showsError or When password is "ABCDEF", the error contains "no digits"?

With Parameterized Tests, your tests are still brief and focused like before, but they are more concise and scale better! You have not only multiplied its quantity with ease, but also its quality!

Conclusion

As we've seen in the samples, whenever you find yourself writing a test with multiple asserts, consider whether it makes sense to create one test per assert instead. That way your tests will be shorter, more focused and more readable.

If after splitting the original tests into several, you find out that you can group the tests by extracting some common code into parameters, Parameterized Tests will also help your tests scale much better, while being brief and focused. They'll also become more concise by using its parameters to build the test name.

However, do not fall into the trap of excessive generalization. Do not write tests that take too many arguments as parameters. If you end up with Parameterized Tests whose names are too general or hard to understand, try to split them into separate Parameterized Tests (or even simple tests) with less parameters. Take a look at how I grouped the Parameterized Tests here instead of merging all of them into one.

Last but not least, remember that tests are also documentation. Therefore, they need to be understandable, you never know who will have to fix those tests if they fail!

*You can find the validators code and a more complete collection of the strong password test samples in this repo.

What comes next

In the following article of this series, we will explore Property-Based Tests and how they can help us make our Password validator tests more robust. Already available on the link below!

Better Unit Tests with Property-Based Tests

If you liked this article, you might also like other articles I wrote in Dev.to:

Styling dynamic strings directly in Android xml

Styling dynamic strings directly in Android xml with HTML markup

sergio-sastre — Fri, 09 Jul 2021 14:50:28 +0000

In Android, Strings are one of the most common Objects when working with views that show plain text. While implementing user interfaces, it happens often that the text requires some styling. For styled Strings, we are required to use CharSequences instead. Android supports some html tags out of the box, and those can be defined in the xml string resource, for instance

<string name="lorem_ipsum">
   This is <font color="red">red</font> and this
   <b><i>bold and italic (nested); </i>this just bold</b>,
   <u>underlined</u>
</string>

and resolved into a CharSequence by calling context.getText(stringRes: Int) which takes care of all the supported HTML tags to style the text without us needing to do anything else

However, sometimes we need to build a part of the string dynamically. In order to do this, we need to mark the dynamic parts in the xml with placeholders in the form of %{digit}${type}, for instance %1$s is a string passed as first vararg and %2$d is a decimal number passed as second vararg in String.format(text: String, vararg args: String), which is called to resolve the placeholders.

Nevertheless, things start getting complicated if we define HTML tags together with dynamic placeholders in xml string resources, for instance something like this

<string name="lorem_ipsum">
   This is <font color="red">red</font> and this
   <b><i>bold and italic (nested); </i>this just bold</b>, <u>underlined</u>
   and here the placeholder = %1s
</string>

If we take a look at the method signature of String.format(text: String, vararg args: String), its first argument requires a String instead of a CharSequence. This means, the dynamic text placeholders will be correctly replaced, but our CharSequence has to be converted to String, throwing away its styling.

In order to deal with HTML markup, Android provides HtmlCompat. It requires that the string resource encodes its opening unsafe characters, namely: '<' , which becomes '<'

<string name="lorem_ipsum">
   This is &lt;font color="red">red&lt;/font> 
   and this &lt;b>&lt;i>bold and italic (nested); &lt;/i>this just bold&lt;/b>,
   &lt;u>underlined&lt;/u> and here the placeholder = %1s
</string>

or alternatively, we can wrap the resource inside CDATASections instead to the xml as follows:

<string name="lorem_ipsum">
<![CDATA[This is <font color="red">red</font> and this <b><i>bold and italic (nested); </i>this just bold</b>, <u>underlined</u> and here the placeholder = %1s]]>
</string>

In any case, given our dynamic placeholder text to be "placeholder1", we can get the expected result by using HtmlCompat as follows:

val text = context.getString(R.string.lorem_ipsum)
val dynamicText = String.format(text, "placeholder1") 
val dynamicStyledText = HtmlCompat.fromHtml(
      dynamicText,
      HtmlCompat.FROM_HTML_MODE_COMPACT
)

textView.text = dynamicStyledText

Although the code above seems to work robustly, the result differs if the dynamic placeholder text contains at least one unescaped HTML character, for instance:<, >, &, \ or ", like in <placeholder1>, leading to the result below

Yes, the placeholder just disappears. That's because characters must be escaped before calling HtmlCompat.fromHtml(). We solve that by encoding the placeholders before using HtmlCompat, like this

val text = context.getString(R.string.lorem_ipsum)
val encodedPlaceholder = TextUtils.htmlEncode("<placeholder1>")
val dynamicText = String.format(text, encodedPlaceholder) 
val dynamicStyledText = HtmlCompat.fromHtml(
      dynamicText,
      HtmlCompat.FROM_HTML_MODE_COMPACT
)

textView.text = dynamicStyledText

Although it works and it is the recommended way according to the official documentation, I personally do not like any of the approaches before. Why?

You end up changing the xml string resource completely for the sake of using a dynamic text placeholder
You lose xml highlighting in the styled parts of the string resource and therefore, it is harder to read

A better approach would be to create a method that can handle the original xml string resource with HTML tags and placeholders. In doing so, it does not matter whether the string resource contains HTML markup or not, the method simply handles the placeholders while keeping the style defined by the (existing, if any) HTML tags ... no need to either replace opening unsafe characters or add CDATASections.

And yes, it is possible with a bit of hackery. Let's see how.

Digging into a better solution

We already know that using context.getText(R.string.lorem_ipsum) returns the string resource styled as a CharSequence. If the string resource has a placeholder, it will be shown the same as in the xml.

We also know that HtmlCompat.fromHtml() processes "some" HTML tags. Its inverse method exists and does exactly the opposite: takes a Spanned object and converts it to a string with the corresponding HTML tags. The flag we pass to the method also matters: HtmlCompat.TO_HTML_PARAGRAPH_LINES_INDIVIDUAL also adds a new line at the end of the HTML string and we have to account for that. Therefore, we can get the desired HTML string as follows

// step 2 - toHtml()
val spannedString = SpannedString(styledString)
val htmlString = HtmlCompat.toHtml(
        spannedString,
        HtmlCompat.TO_HTML_PARAGRAPH_LINES_INDIVIDUAL
    )
        .substringBeforeLast('>')
        .plus(">")

which results into

We've got the equivalent HTML string to the styled String of the first step so far. However, the final goal is to replace its placeholders with the corresponding values. As you might remember, I mentioned at the beginning of the article that we can use String.format(text: String, vararg args: String) for that. It would not work with a CharSequence, but that is why we converted it into its equivalent HTML string in the first place.

// step 3 - String.format()
val dynamicHtmlString = String.format(htmlString, args)

Just convert the HTML text into a CharSequence and we get the desired style. Remember to use HtmlCompat.FROM_HTML_MODE_COMPACT, since it is the inverse of the HtmlCompat.TO_HTML_PARAGRAPH_LINES_INDIVIDUAL we've previously used

// step 4 - fromHtml()
val result = HtmlCompat.fromHtml(
    dynamicStyledString,
    HtmlCompat.FROM_HTML_MODE_COMPACT
)
.removeSuffix("\n")   // fromHtml() adds a new line at the end

Well we are almost done... as we have seen at the beginning of this article, if the placeholders are Strings containing unsafe characters, they do not show up. Therefore, do not forget that we need to encode the string values that will substitute the placeholders. A Kotlin extension function following all the aforementioned steps would look like this

fun Context.getHtmlStyledText(
    @StringRes htmlStringRes: Int,
    vararg args: Any
): CharSequence {

    // step 0 - Encode string placeholders  
    val escapedArgs = args.map {
        if (it is String) TextUtils.htmlEncode(it) else it
    }.toTypedArray()

    // step 1 - getText()
    val styledString = Context.getText(htmlStringRes)

    // step 2 - toHtml()
    val spannedString = SpannedString(styledString)
    val htmlString = HtmlCompat.toHtml(
        spannedString,
        HtmlCompat.TO_HTML_PARAGRAPH_LINES_INDIVIDUAL
    )
        .substringBeforeLast('>')
        .plus(">")

    // step 3 - String.format()
    val dynamicStyledString = String.format(htmlString, *escapedArgs)

    // step 4 - fromHtml()
    return HtmlCompat.fromHtml(
        dynamicStyledString,
        HtmlCompat.FROM_HTML_MODE_COMPACT
    )
    .removeSuffix("\n")   //fromHtml() adds one new line at the end
}

BONUS

The same idea applies to plural resources. Simply replace

// step 1 - getText()
val styledString = context.getText(R.string.lorem_ipsum)

with

// step 1 - getText()
val styledString = context.resources.getQuantityText(R.plural.lorem_ipsum, quantity)

You can find the corresponding working gist for strings and plurals here

Cover photo by Markus Spiske on Unsplash