Like Miracle Grow For Your Garden

In the last post I did about the Vegetables testing framework I developed, I talked about how much of a success it had been. Now I’ve cranked it up to 11. I’ve now got what is effectively QuickCheck level functionality implemented in a Fortran testing framework.

For anyone not familiar with QuickCheck, it is effectively a library for generating random values to use as examples for your tests. But the really useful part is that if it finds an example that causes your test to fail, it then attempts to find the simplest example that causes your test to fail, and reports that to you.

The simplest example I have is just that an integer should equal itself.

function checkPassForSameInteger(input) result(result_)
    use Vegetables_m, only: Result_t, assertEquals, assertThat, fail

    class(*), intent(in) :: input
   type(Result_t) :: result_

    type(Result_t) :: example_result

    select type (input)
    type is (integer)
        example_result = assertEquals(input, input)
        result_ = assertThat( &
                example_result%passed(),
                example_result%verboseDescription(.false.))
    class default
        result_ = fail("Expected to get an integer")
    end select
end function checkPassForSameInteger

Running just this test now gives the following result.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

All Passed
Test that
    assertEquals with integers
        passes with the same integer
            Passed after 100 examples
A total of 1 test cases containg a total of 1 assertions

Then, if I change the assert to assertEquals(input, input+1), I get the following.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

Failed
Test that
    assertEquals with integers
        passes with the same integer
            Fails with the simplest possible example
            Expected to be true
                User Message:
                    [Expected
                             [0]
                     but got
                             [1]]
1 of 1 cases failed
2 of 2 assertions failed

Or, if I change the assert to assertEquals(input, input/2), I get the following.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

Failed
Test that
    assertEquals with integers
        passes with the same integer
            Found simplest example causing failure
            Expected to be true
                User Message:
                    [Expected
                             [1]
                     but got
                             [0]]
1 of 1 cases failed
2 of 2 assertions failed

In order to use the QuickCheck functionality you need to create a derived type extended from the Generator_t type. This requires you to implement two functions for it. You need generate, which takes your generator type as an argument, and produces a random value wrapped in a Generated_t type. Then you need shrink, which takes one of your generated values as a class(*), and returns a “simpler” value than the one it was given, wrapped in either a ShrunkValue_t or a SimplestValue_t to indicate the value can’t be made any simpler.

I’ve implemented a couple of basic Generators and some helper functions for generating some random values, but for most things, you’ll probably need to create your own. But if you can, and you can specify some properties of your code that should hold for any random value, you can really stress test your code.

I think at this point I have one of the most feature-full testing frameworks I’ve ever used. And I think in fewer lines of code than any other testing framework I’ve ever seen. Functional programming principles for the win!