Property based testing in Go

What is property based testing?

According to Google:

Property based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test.

Not very straightforward, right? Let's make this statement more uncomplicated to understand:
Property based testing is a software testing methodology that involves testing a system against a set of properties or specifications that the system should satisfy. Unlike example-based testing, where specific test cases are manually created to verify individual behaviours or functionalities, property based testing focuses on describing general properties that the system should maintain across a wide range of inputs.

As software engineers, we create numerous unit tests for the components we work on. Writing the actual test code is straightforward, but deciding which test data to include to cover all critical scenarios can be more challenging.
Let's examine the following example:

package greetings

import (
    "testing"
    "regexp"
)

// TestHelloName calls greetings. Hello with a name, checking
// for a valid return value.
func TestHelloName(t *testing.T) {
    name := "Gladys"
    want := regexp.MustCompile(`\b`+name+`\b`)
    msg, err := Hello("Gladys")
    if !want.MatchString(msg) || err != nil {
        t.Fatalf(`Hello("Gladys") = %q, %v, want match for %#q, nil`, msg, err, want)
    }
}

// TestHelloEmpty calls greetings. Hello with an empty string,
// checking for an error.
func TestHelloEmpty(t *testing.T) {
    msg, err := Hello("")
    if msg != "" || err == nil {
        t.Fatalf(`Hello("") = %q, %v, want "", error`, msg, err)
    }
}

This is a test from golang's official documentation page. Let's break down each part of the code:

1. TestHelloName:
   • This function tests the f.Hello(...) function when called with a non-empty name ("Gladys").
   • It checks if the returned message matches the expected pattern.
2. TestHelloEmpty:
   • This function also tests the f.Hello(...) function, but now with an empty string.
   • It checks if the returned message is empty (msg != "").

We can see that both are hard-coded scenarios. We run both tests and they pass. Hurray!

$ go test
PASS
ok      example.com/greetings   0.364s

$ go test -v
=== RUN   TestHelloName
--- PASS: TestHelloName (0.00s)
=== RUN   TestHelloEmpty
--- PASS: TestHelloEmpty (0.00s)
PASS
ok      example.com/greetings   0.372s

It seemed like we covered all scenarios for the f.Hello(...) function. But in production, we found errors, like when name := "Ruben" caused a problem. We fixed that and added a new test. But later, a new v.name caused another error. Every new problem requires a new test, and it goes on endlessly.

Using randomly generated values for testing can help address the constant need for refactoring and adding more test cases. This is where the usefulness of Property based testing becomes clear.

How does property based testing achieve coverage of test cases during development?

Here's how property based testing typically works:

1. Define properties: Developers define a set of properties that the system should adhere to. These properties are essentially statements about the behaviour or characteristics of the system that should hold true under different conditions.
2. Generate random inputs: Instead of using predefined test cases, property based testing tools generate random inputs for the system. These inputs are usually generated within certain constraints to ensure they are valid and representative of typical usage scenarios.
3. Check properties: The generated inputs are then fed into the system, and the properties defined earlier are checked against the system's behaviour. If any property fails to hold true for a given input, it indicates a potential bug or violation of the system's specification.
4. Shrink failing inputs: When a property fails, property based testing tools often attempt to "shrink" the failing input. This means they try to find the simplest possible input that still causes the property to fail. This can help developers identify the root cause of the failure more easily.

The most interesting part is the Shrink failing inputs or simply Shrinking. This is a vital aspect of property based testing, which aims to simplify and pinpoint the root cause of test failures. When a property based test fails, it's crucial to find the smallest input that still triggers the failure. This allows developers to identify the exact conditions leading to the failure, making it easier to understand and fix the problem. By shrinking failing inputs, property based testing frameworks assist developers in isolating the specific circumstances causing test failures, thus accelerating the debugging process and enabling more effective problem resolution. This approach is especially helpful for identifying complex or obscure issues that traditional testing methods may struggle to reproduce or comprehend. Let's imagine that during testing, the test fails with a specific randomly generated input, say, "123". When this failure occurs, the property based testing library will attempt to shrink this input to find a minimal example that still triggers the failure. Here's how it might work:

1. Initial Failure: The test fails with the input "123".
2. Shrinking Process: The property based testing library starts the shrinking process. It tries to simplify the input while still reproducing the failure. It might try removing characters from the input or modifying it in other ways to see if the failure still occurs.
3. Candidate Inputs: During the shrinking process, the library might try inputs like "12", "13", "23", and eventually "1", "2", and "3". It checks if any of these simplified inputs still cause the failure.
4. Minimal Failing Input: Eventually, the library might determine that the minimal failing input is simply "1". This means that the failure can be reproduced with just the single character "1", indicating that this is the simplest case that exposes the issue.
5. Report: The property based testing library reports the minimal failing input, "1", along with the failure information. This helps developers identify the root cause of the failure more easily, as they now have a minimal example to analyze and debug.

Property based testing offers several advantages over traditional example-based testing:

• Coverage: Property based testing can explore a much larger range of input values, increasing the likelihood of uncovering edge cases and corner cases that might not be obvious from predefined test cases.
• Automation: Once properties are defined, property based testing can be largely automated. Developers don't need to manually create individual test cases for every possible scenario.
• Generality: Properties are often more general than individual test cases, capturing broader aspects of system behaviour. This can lead to more robust tests that are less likely to break when the system is modified.
• Bug detection: Property based testing can uncover unexpected behaviours or edge cases that might not be caught by traditional testing methods, helping to improve the overall reliability and correctness of the software.

Property based testing has a rich history, originating with the introduction of libraries for C. As its popularity grew, communities began developing their own versions of property based testing frameworks for different programming languages.

Rapid: property based testing framework for Go

Rapid is a Go library for property based testing. Rapid checks that properties you define hold for a large number of automatically generated test cases. If a failure is found, rapid automatically minimizes the failing test case before presenting it.

You can check the package here, in official GitHub repo. I recently found this amazing testing framework and now I can't imagine developing tests without it.
Let's try to refactor our previous example with this package:

package greeters  

import (  
    "regexp"  
    "testing"  
    "pgregory.net/rapid")  

func TestHello(t *testing.T) {  
    // Random string generator by the given pattern  
    // rapid.StringMatching creates a string generator matching the provided regular expression.    
    var genName = rapid.StringMatching("^[a-zA-Z0-9]*")  

    rapid.Check(t, func(t *rapid.T) {  
       // generate a random character sequence  
       // rapid.Draw produces a value from the generator.       
       name := genName.Draw(t, "randomly generated name")  

       want := regexp.MustCompile(`\b` + name + `\b`)  
       msg, err := Hello(name)  
       if !want.MatchString(msg) || err != nil {  
          t.Fatalf(`Hello("Gladys") = %q, %v, want match for %#q, nil`, msg, err, want)  
       }  
    })  
}

Let's break down what each part of the code does:

1. TestHello function:
   • Inside this function, a property based test is defined using the rapid.Check function provided by the rapid testing library.
2. Random string generation:
   • A generator (v.genName) is defined using rapid.StringMatching. This generator is used to produce random strings that match the provided regular expression pattern ("^[a-zA-Z0-9]*"), which matches any alphanumeric string.
3. Property based testing block:
   • Inside the rapid.Check function, a function literal (anonymous function) is provided. This function represents the property based test.
   • The rapid.T object (t) is passed to this function, which is used for test logging and failure reporting.
4. Generating random input:
   • Inside the property based test function, a random string (v.name) is generated using the v.genName.Draw function. This function draws a random value from the v.genName generator.

This property based test generates random input strings, calls the f.Hello(...) function with these inputs, and verifies that the output message matches the expected pattern. If the output message doesn't match the expected pattern or if there's an error, the test fails. This approach helps to verify the behaviour of the f.Hello(...) function across a wide range of input strings.

If we run this test, rapid.Check will run 100 times.

$ go test
=== RUN   TestHello
    greeter_test.go:15: [rapid] OK, passed 100 tests (1.127291ms) <-- this is the output message of the f.rapid.Check(...)
--- PASS: TestHello (0.00s)
PASS

By default, Rapid runs 100 times by generating a random input in every iteration and performers shrinking (to find the simplest possible input that still causes the property to fail). We can add a print statement fmt.Printf("genName.Draw() --> %s \n", name) to see what inputs are generated in every stage of execution:

...
genName.Draw() --> yz3sD 
genName.Draw() --> zeF0c 
genName.Draw() --> t609421631Zq0412d 
genName.Draw() -->  
genName.Draw() --> Ll30RqAk 
genName.Draw() --> f31HIXB15 
genName.Draw() --> so5WS 
genName.Draw() --> zBX 
genName.Draw() --> 7 
genName.Draw() --> N3D 
genName.Draw() --> 21 
genName.Draw() --> 7 
genName.Draw() --> 8 
genName.Draw() --> 1l 
genName.Draw() --> 9X5142g 
...

We see that the generator is producing a mix of alphanumeric strings as expected, but also some strings that do not match the pattern, such as empty strings or strings containing only numbers. There's a high chance that during these 100 test executions, at least one of the generated inputs will cause an error such as this one:

=== RUN   TestHello
    greeter_test.go:15: [rapid] failed after 2 tests: Hello("Gladys") = "", boom, error, want match for `\b0000\b`, nil
        To reproduce, specify -run="TestHello" -rapid.failfile="testdata/rapid/TestHello/TestHello-20240301112254-24744.fail" (or -rapid.seed=370674197420583485)
        Failed test output:
    greeter_test.go:18: [rapid] draw randomly generated name: "0000"
    greeter_test.go:22: Hello("Gladys") = "", boom, error, want match for `\b0000\b`, nil
--- FAIL: TestHello (0.01s)

FAIL

The line with Failed test output... provides additional information about the failure. It indicates that the randomly generated name for this specific test case was "0000" and it produced the test failure. Also, we can see that the property based test failed after running 2 tests: [rapid] failed after 2 tests:. The failure message provides comprehensive information about the test failure, including the context of the failure, details of the observed behaviour and reproduction instructions to aid in diagnosing and addressing the issue effectively.

To run the rapid, you just run go test as usual; it will also pick up all rapid tests.
I mentioned that Rapid, by default, runs 100 times, and this can be changed if needed. Run go test -args -h and look at the flags with the -rapid. prefix. You can then pass such flags as usual. For example:

go test -rapid.checks=10_000

It's that easy!

Conclusion

In conclusion, property based testing offers a powerful and efficient approach to software testing, enabling comprehensive test coverage, efficient bug detection, and guided debugging. By leveraging random input generation, defining properties, and systematically shrinking failing inputs, property based testing enhances the reliability, robustness, and overall quality of software systems.