DEV Community: Sindre Røkenes Myren

Go generics beyond the playground

Sindre Røkenes Myren — Thu, 25 Mar 2021 20:34:26 +0000

While Go as of version 1.16 does not support Generics, there is an accepted language proposal for it, and we can probably expect it to arrive in Go 1.18 or later. But as it turns out, we don't need to wait for it to start experimenting with the new feature. We can start now to write small snippets of code on the go2go playground, or even full Go packages that you develop locally.

By far the easiest way to test out Go generics, is to use the playground. And there is nobody saying that the playground isn't awesome. However awesome though, there is clear limits to how much you can reasonably try out in the playground alone. What if you have enough code to start splitting into files? What if you want to write unit-tests? How would a full package look like with generics? In my view, the best way to try out a new feature, is to actually do something useful. And to do this with generics, we need to venture out of the safety and comfort of the playground.

My hope is that this will inspire you to do your own experiments with Go generics beyond the playground and write something potentially useful. Only then, can we truly see if generics itself, is going to be useful in Go.

In this article, I will go through how I re-wrote a test matching library from scratch with generics as part of the tool-box. My hope is that this will inspire you to do your own experiments with Go generics beyond the playground and write something potentially useful. Only then, can we truly see if generics itself, is going to be useful in Go. If you want, you could use the library in this article for testing your experiments; or you can extend the library and do a pull request.

Before we start: a warning. I will assume basic knowledge about the generics proposal in Go and what the most important design concepts are. If you don't have this knowledge, I would recommend that you first acquire it. E.g. by reading Go 2 generics in 5 minutes, by reading the updated design draft, or by doing your own experiments in the go2go playground first. In the meantime, keep this article on your TO-READ list. With that said, because we are trying to do something potentially useful, this article will be more about package design then about generics itself.

A problem to solve (again) with generics

Starting out with Go generics, in order to do something useful, we need a problem to solve. The problem I have picked for this article is one that I have tried to solve before when designing the test matcher/assertion library that we use to test the Clarify back-end at Searis. But first, you probably have a question: with all the great test matcher libraries we have in Go, why on earth would we want to write a new one? To answer that, it's worth having a closer look at at one of the existing matcher libraries. Does it solve it's mission in a useful way, and with the best possible package design?

In the Go world, by far the most popular Go matcher library still appear to be the assert package from the stretchr/testify repo. It's an old, good and stable library. However, because it's old, and because most (old) Go libraries strived to keep full backwards compatibility, it's also an excellent library to demonstrate some problems that can perhaps be solved by a different design. For this article, we will be content with considering the following code, testing an imaginary function mypkg.Sum:

func TestSum(t *testing.T) {
    a := mypkg.Vector{1,0,3}
    b := mypkg.Vector{0,1,-2}
    expect := mypkg.Vector{1,1,1}

    result, err := mypkg.Sum(a, b)

    assert.NoError(t, err, "unexpected error")
    assert.Equal(t, result, expect, "unexpected result")
}

At first glance, all of this might look fine. What's wrong with it, you might think. Before we get into that, there is one minor alteration I want to do to the code, and you would probably know this if you have ever used the assert library. Because the descriptive text in the assertion is optional, you would probably write them like this:

assert.NoError(t, err)
assert.Equal(t, result, expect)

Which may give the following output in the case of a failure:

--- FAIL: TestSum_assert (0.00s)
    /Users/smyrman/Code/blog/2021-03-generics-beyond-the-playground/mypkg/subtest_sum_test.go:35:
            Error Trace:    subtest_sum_test.go:35
            Error:          Not equal:
                            expected: mypkg.Vector{0, 1, -2}
                            actual  : mypkg.Vector{1, 1, 1}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,5 +1,5 @@
                             (mypkg.Vector) (len=3) {
                            - (float64) 0,
                              (float64) 1,
                            - (float64) -2
                            + (float64) 1,
                            + (float64) 1
                             }
            Test:           TestSum_assert
FAIL
FAIL    github.com/smyrman/blog/2021-03-generics-beyond-the-playground/mypkg    0.130s
FAIL

So, to the problems. Problems in design are often subtle ones. The ones you can't quite put your finger on. Once you know about them though, they are hard not to se. In this short snippet of code and failure output, there is actually as much as six problems that I want to bring to attention:

We have to pass in the t parameter. It's a minor inconvenience, but enough for Dave Cheney to write a quite interesting package for solving it. Not recommended for production use, I might add.
The "got" and "want" parameter order is hard to get right. If you are observant, you might have noticed that the code above gets it wrong. This again makes the failure output tell a lie. In my own experience, such lies can lead to significant confusion when debugging a problem. As a rule of thumb if you use the assert package a lot, and tend to continue doing so, the "expected" parameter almost always come first. This is of course not a general rule in Go. E.g. the standard library tend to report the "got" parameter first, and the "want" parameter second.
From simply starring at the code (which is what code reviewers generally do) it's not obviously clear what's meant by Equal. It's well enough documented, but from the name alone, it's not clear if it's going to use the equal comparison (==), the reflect.DeepEqual method, or if it can handel equal comparison of two time.Time instances with a different time-zone. At some point, subtile details in how equality is implemented, might come back and bite us. Especially so if any of these details don't match the comparison we usually do in our programs.
The descriptive text for what went wrong is optional; and thus easily omitted. This can make debugging failing tests hard, as there is little information to exactly what went wrong (other than two values did not compare equal). Especially so if we have multiple assert statements. Is it the result from the function under test that's wrong? Is this just a sanity check before running the main test?
For what the output potentially lack of useful information, it makes up for in redundant information. Why do we need a diff for finding the difference between these simple struct instances? Given the diff is printed, why do we also get the "actual" and "expected" one-liners? Why is the filename and line trace information repeated? Why is the test name repeated?
Many of the library assertions, assert.Equal included, lack compile-time type safety. Of course, one might say, but if type-safety is beneficial in the rest of our programs, would it not also be beneficial in tests?

In our test-matcher for Clarify (called subtest), we have managed to to solve many of these problems. In order to explain how, let's start by rewriting the test function for mypkg.Sum:

func TestSum(t *testing.T) {
    a := mypkg.Vector{1, 0, 3}
    b := mypkg.Vector{0, 1, -2}
    expect := mypkg.Vector{1, 1, 1}

    result, err := mypkg.Sum(a, b)

    t.Run("Expect no error", subtest.Value(err).NoError())
    t.Run("Expect correct sum", subtest.Value(result).DeepEqual(expect))
}

Which in case of a failing test, may give the output:

--- FAIL: TestSum (0.00s)
    --- FAIL: TestSum/Expect_correct_sum (0.00s)
        /Users/smyrman/Code/blog/2021-03-generics-beyond-the-playground/mypkg/subtest_sum_test.go:46: not deep equal
            got: mypkg.Vector
                [0 1 -2]
            want: mypkg.Vector
                [1 1 1]
FAIL
FAIL    github.com/smyrman/blog/2021-03-generics-beyond-the-playground/mypkg    0.106s
FAIL

So, let's go through which problems we managed to solve by this, and which one we didn't.

First out, you might note that we do not pass in a t parameter. That is because what subtest does, is to return a test function. This test function can then be run as a Go sub-test, omitting the need for the user to pass in a t parameter. Cleaver? I certainly thought so when I first had the idea.

Next up we solved the ordering issue of "got" and "want" by wrapping them in each their method call. We can see that we first call Value on the result. This actually return what we call a ValueFunc or value initializer. It is not important now to explain why it's a function, but we can perhaps get back to that. The important part now is that this type has methods that let's us initialize tests. In this particular instance, we call the method DeepEqual with the expect parameter. Here we have also attempted to solve the clarity issue of what Equal does, by giving the method a more descriptive name.

Next up, as we pass the func(*testing.T) instance returned by DeepEqual to t.Run, this method require us to supply a name. Thus a description for the check is required. In fact, as we are using Go sub-tests, there is now even a way for us to re-run a particular check in the test for better focus during debugging. This can be particular useful for cases when there are many failing sub-tests and many failing checks in a test.

test -name '^TestSum/Expect_correct_sum'

Next up, we kept our test-output brief without repeating any information. We should note that there definitively is some cases where we would not get enough information from this brief output format. While the library do allow some output customization, this is one of those trade-offs that could probably be improved.

As for the final problem though, compile time type-safety, subtest falls short. We can still pass pretty much anything into the subtest.Value and various test initializer methods, and it won't detect any type mismatches for us before the test run. Besides, does it really make sense to have the same test initializer methods available for all Go types? Should not a subtest.Value(time.Time{}) provide methods that are useful for comparing times, such as analogies to the time.Time methods Equal, Before and After?

If we do the count, we gather that subtest appear to solve five out of the six problems we identified with the assert library. At this point though, it's important to note that at the time when the assert package was designed, the sub-test feature in Go did not yet exist. Therefore it would have been impossible for that library to embed it into it's design. This is also true for when Gomega and Ginko where designed. If these test frameworks where created now, then most likely some parts of their design would have been done differently. What I am trying to say is that with even the slightest change in the Go language and standard library, completely new ways of designing programs become possible. Especially for new packages without any legacy use-cases to consider. And this brings us to generics.

With generics added to our tool-box, can we manage to solve all six problems by starting over?

But why do we even need a matcher library?

Before we start of re-designing a tool, it might be worth while asking ourselves the question, what exactly is the tool trying to solve? And why would anyone ever want to use it? In this case, why would anyone need a matcher library in Go?

And to be fair, you often don't. Go famously don't include any assert functionality, because the Go developers believe that it's better to do checks in tests the same way you do checks in your normal programs. E.g. if you do if err != nil in your program, that's the syntax for doing the same check in yout tests. This way reading Go tests, becomes no different then reading any other go code, and you are less likely to do mistakes.

I believe that perhaps a common misconception could be that a matcher library is needed because checking results is hard. But this simply isn't true. At least not for simple checks. We can prove this by rewriting our example test to not use a matcher library at all:

func TestSum(t *testing.T) {
    a := mypkg.Vector{1, 0, 3}
    b := mypkg.Vector{0, 1, -2}
    expect := mypkg.Vector{1, 1, 1}

    result, err := mypkg.Sum(a, b)

    if err != nil {
        t.Errorf("Expected no error, got: %v", err)
    }
    if !reflect.DeepEqual(result, expect) {
        t.Errorf("got: %v, want: %v", result, expect)
    }
}

As we can see, the check part is quite straight-forward. So why then would we need a matcher library?

My own opinion is that the part of writing tests that might make you consider a matcher library, is the disciplinary challenge of letting the output on failures be both useful and consistent. Especially so, if you have a project with multiple developers. How much team and review discipline do you need to consistently order the "got" and "want" parameter, for instance? Again, if you are observant, you might have noticed that the code above have already failed that test. While the test above won't tell lies, consistent wording, ordering and reporting of test failures, can be important tools in streamlining the debugging process. And while some teams, like the Go team, are extremely good at getting this right, other teams might prefer to just use a library.

To finish up, perhaps we can agree that the right mission statement for a matcher library, might be more about providing consistent and useful information on failure than it is about making complicated checks easy; although it might do that too.

Exploring the original design

We will soon move on to designing, or rather re-designing, our test matcher library with generics. But to do that we must understand a little bit more about the core design of the subtest library.

So, remember this code?

t.Run("Expect correct sum", subtest.Value(result).DeepEqual(expect))

We mentioned the code generates a test, and we mentioned something about lack of type-safety, but we didn't give to much detail around exactly what we mean by this. Actually this code is just a short-hand syntax. As it happens, the long syntax for the same operation reveals the underlying design much better.

t.Run("Expect correct sum", subtest.Value(result).Test(subtest.DeepEqual(expect)))

Or to break the code up even further:

// Step 1: create a "value function" a.k.a. a value initializer:
valueFunc := subtest.Value(result)

// Step 2: create a check:
check := subtest.DeepEqual(expect)

// Step 3: Combine the value initializer and the check into a
// test function:
testFunc := valueFunc.Test(check)

// Step 4: Run the test function as a Go sub-test:
t.Run("Expect correct sum", testFunc)

Obviously, the check portion here can be replaced. E.g. instead of subtest.DeepEqual, we could use subtest.NumericEqual, subtest.Before or even a user defined check. This is powered by a combination of Go interfaces and first-class function support.

While the subtest package has several checks and some formatting helpers, the core design can actually be summarized in very few lines of code.


// The interface for a check.
type Check interface{
    Check(ValueFunc) error
}

// An adapter that lets a normal function implement a check.
type CheckFunc func(interface{}) error

func (f checkFunc) Check(vf ValueFunc) error {
    v, err := vf()
    if err != nil {
        return fmt.Errorf("failed to initialize value: %w", err)
    }
    return f(V)
}

// A value initializer function.
type ValueFunc func() (interface{}, error)

// An adapter to convert a plain value to a value initializer
// function.
func Value(v interface{}) ValueFunc {
    return func() (interface{}, error) {
        return v, nil
    }
}

// A method that constructs and returns a Go (sub-)test function
// from the combination of a value initializer and a check.
func (vf) Test(c Check) func(*testing.T) {
    return func(t *testing.T) {
        if err := c.Check(vf); err != nil {
            t.Fatal(err.Error())
        }
    }
}

So comes the problem to solve with generics. Because what's the point of passing in a NumericEqual check (expecting some numeric value) or a Before check (expecting a time.Time) to a value initializer returning a mypkg.Vector type? Can we, by use of generics, make this fail compilation?

The generic re-design

Finally, the re-design. With generics, or type parameterization, which the proposed Go generics is more accurately called, we can enforce type-safety for the check and value initializer combination into tests. To demonstrate how, here is a re-write of the subtest core-design.

// The check function type.
type CheckFunc[T any]func(func() T) error

// An adapter for converting a value to a value initializer
// function.
func Value[T any](v T) func() T {
    return func() T{
        return v
    }
}

// A function for combining a compatible value initializer and a
// check function into a Go (sub-)test.
func Test[T any](vf func() T, cf CheckFunc[T]) func(t *testing.T) {
    return func(t *testing.T) {
        if err := c.Check(vf); err != nil {
            t.Fatal(err.Error())
        }
    }
}

Each individual check implementation, or check initializer implementation to be exact, can be declared with various degrees of type parameterization.

// A type parameterized check initializer that can be initialized
// for any type.
func DeepEqual[T any](w T) CheckFunc[T] {...}

// A type parameterized check initializer that can only be
// initialized with a comparable type.
func CompareEqual[T comparable](v T) CheckFunc[T] {...}

// A type parameterized check without a compare value.
func ReflectNil[T any]() CheckFunc[T]

// A check initializer that is not type parameterized.
func TimeBefore(t time.Time) CheckFunc[time.Time] {...}

The trick here is that the type used to initialize the check and the type returned by the value initializer needs to match. If we try to combine a subx.TimeBefore check with a mypkg.Vector value initializer, compilation would fail. If we try to combine a mypkg.CompareEqual with the same initializer, it will work if mypkg.Vector is implemented as a comparable type. E.g. if mypkg.Vector is implemented as a struct with three fields, that are all comparable, then it will work.

type Vector struct{
    X, Y, Z float64
}

If however it is implemented as a slice, then using CompareEqual would result in a compile time error.

type Vector []float64

For our rewrite, let's assume the slice implementation. In order to explain what's happening in the rewritten code, we will first show the code without any type inference.

func TestSum(t *testing.T) {
    a := Vector{1, 0, 3}
    b := Vector{0, 1, -2}
    expect := Vector{1, 1, 1}

    result, err := Sum(a, b)

    t.Run("Expect no error", subx.Test(
        subx.Value[error](err),
        subx.CompareEqual[error](nil),
    ))
    t.Run("Expect correct sum", subx.Test(
        subx.Value[Vector](result),
        subx.DeepEqual[Vector](expect),
    ))
}

Which might give the following output on failure:

-------- FAIL: TestSum (0.00s)
    --- FAIL: TestSum/Expect_correct_sum (0.00s)
        sum_test.go2:20: comparison failed:
            got: (mypkg.Vector)
                [2 2 2]
            want deep equal to: (mypkg.Vector)
                [1 1 1]
FAIL
exit status 1
FAIL    github.com/smyrman/subx/examples/vector_sum 0.299s

Looking at this code, you might argue that the subx.Test function has reintroduced the ordering issue of the "got" and "want" parameters. However, it has not. this is because ordering this parameters wrong would lead to a compile-time error. We mentioned that using CompareEqual would fail compilation for the slice implementation of mypkg.Vector. Actually, even subx.DeepEqual[*mypkg.Vector] will cause a compilation error. This type-safety can be a useful tool to prevent simple programming mistakes.

If you read the design proposal, you would know all about type inference, and when it can and cannot be used; or perhaps you discovered how through some trial and error in the playground. With the current type-inference implemented in the go2go tool, the [T] syntax can be omitted everywhere in our example except for when comparing an error to nil. This again makes the code more readable.

func TestSum(t *testing.T) {
    a := Vector{1, 0, 3}
    b := Vector{0, 1, -2}
    expect := Vector{1, 1, 1}

    result, err := Sum(a, b)

    t.Run("Expect no error", subx.Test(
        subx.Value(err),
        subx.CompareEqual[error](nil),
    ))
    t.Run("Expect correct sum", subx.Test(
        subx.Value(result),
        subx.DeepEqual(expect),
    ))
}

PS! Note that due to a limitation with the go2go experimental tool, tests had to be written as internal test, unlike the subtest example where we declared them as external tests. This means, instead of declaring the package as mypkg_test and do an import of .../mypkg, which would better demonstrate real usage of the package, we have to declare the package as mypkg and omit the import.

Some cool things we can do with subx

While not part of the core design, we define syntactic sugar that allows different short-hand methods to be placed on different value-initializer functions similar to subtest.

// Long syntax:
result := mypkg.Sum(2, 3)
t.Run("Expect correct sum", subtest.Test(
    subtest.Value(result),
    subtest.CompareEqual(5),
))

// Short-hand syntax:
result := mypkg.Sum(2, 3)
t.Run("Expect correct sum", subtest.Number(result).Equal(5))

If we have a function that isn't reliable, we can repeat a check.

vf := func() int {
    return mypkg.Sum(2, 2, 1)
}

t.Run("Expect stabler results", subx.Test(vf,
    subx.AllOf(subx.Repeat(1000, subx.CompareEqual(5))...),
))

If we want to run a check outside of a test, we can do that as well.

result := mypkg.Sum(2, 3)
cf := subx.CompareEqual(5)
fmt.Println("CHECK sum error:", cf(subx.Value(result)))

Challenge: Go beyond the playground

Now I have told you how I went beyond the playground to re-design a package with Go generics. Now, it's your turn!

I hereby challenge you to find something useful to do with generics. Re-design a package. Write an ORM with generics. What ever you find that you want to do, following the README instructions of subx should be enough to get you started.

Fair warning: As of the time of writing, writing anything with generics is a bit of a time-travel in terms of editor support. We are now quite spoiled by the features of gopls, gofmt, goimports and more. Not to mention, syntax highlighting. At the time of writing, none of this works for Go generics; at least not out of the box. You have to remove your own blank lines, insert your own imports, and align your own struct tags. Now that you are warned; Godspeed!

Test-With-Expect: A BDD-style Go naming pattern

Sindre Røkenes Myren — Wed, 10 Jun 2020 07:36:35 +0000

This article has received minor edited since it was first released, see the full history on GitHub.

TL;DR: This article demonstrate how to write GWT-inspired tests in plain Go, and how to name them. Skip to the Go TWE heading to see the result, or explore the summary article on Medium.

GWT, or "Given-When-Then", is a great naming convention for tests that comes from the BDD, or "Behavior-Driven-Development" paradigm. It makes it easy to plan tests as well as the behavior of your feature before you start the detailed implementation.

GWT is composed of three concepts or steps:

Given: A precondition for the test (context)
When: The action to perform (action)
Then: A result to expect (check)

Each of these steps can be nested, or sometimes skipped.

The challenge of this article is not to write GWT or BDD style tests in Go, this has been demonstrated many times before, but an exploration into how we can do this without a third-party test framework and DSL. There are also some benefits associated with relying on the the default test-runner as well as good-old (yes they are 4 years now!) subtests that we will look into.

Code under test

In this article, we will imagine that we are going to write a generic Sum function. Generic is a loaded term in the programming world. In our case, we plan to write a function that accepts any slice or array of numeric values and returns the sum. Because type parameterization is not (yet) possible in Go, we will allow the function to return an error when it receives invalid input.

Let's define the interface we want this function to have:

package mypkg

import (
)

// Sum accepts any kind of slice or array holding only
// numeric values, and returns the sum.
func Sum(v interface{}) (float64, error) {
    return -1, errors.New("NYI")
}

Following the sprit of BDD and TDD, we will wait with the actual implementation until after we are done with the tests. In fact, since this article is about writing and naming tests, we will leave the entire implementation as an exercise for the reader.

Planning our tests

For the scope of this article, let's hash out how we want this function to behave for integer slice ([]int) input in particular. By the power of GWT, we plan our tests in plain text first:

TestSum:
  Given a non-empty int slice S:
    When calling Sum(S):
      Then it should not fail
      Then it should return the correct sum
      Then S should be unchanged

  Given an empty int slice S:
    When calling Sum(S):
      Then it should not fail
      Then it should return zero
      Then S should be unchanged

Great, we have specified our tests, but how does it look once we turn it into Go code? And what kind of output can we expect?

Writing the tests in Go

Some popular BDD frameworks, such as Ginko, define their own test runner and re-implement Go sub-tests (although it predates them, to be fair) by structuring their library to implement a form of DSL (Domain Specific Language). This framework can render pretty output for tests, especially if run in a a terminal that supports color. However, if you wish to focus on a failing sub-test, you can not rely on go test -run to do it; you need to do it the "Ginko way". Because of this, you can also not rely on editor or IDE integrations in the same way you can with other Go tests.

Contrary to popular belief however, it is actually possible to write GWT-style tests in Go without using a BDD-style framework or DSL; GWT itself is just a naming convention, and we can use it with normal Go sub-tests. The most importantly benefit of doing this, is that your GWT-style tests will behave consistently to other Go tests, and can thus be treated equally by both humans and tools. This means you can spend less time in training humans, CIs, JUnit parsers, IDEs, etc., and more time writing tests. Especially so if your team is Go-centric anyways.

So let's write it!

func TestSum(t *testing.T) {
    t.Run("Given a non-empty int-slice S", func(t *testing.T) {
        s := []int{1, 2, 3}
        t.Run("When calling Sum(S)", func(t *testing.T) {
            i, err := mypkg.Sum(s)
            t.Run("Then it should not fail", subtest.Value(err).NoError())
            t.Run("Then it should return the correct sum", subtest.Value(i).NumericEqual(6))
            t.Run("Then S should be unchanged", subtest.Value(s).DeepEqual([]int{1, 2, 3}))
        })
    })
    t.Run("Given an empty int-slice S", func(t *testing.T) {
        s := []int{}
        t.Run("When calling Sum(S)", func(t *testing.T) {
            i, err := mypkg.Sum(s)
            t.Run("Then it should not fail", subtest.Value(err).NoError())
            t.Run("Then it should return zero", subtest.Value(i).NumericEqual(0))
            t.Run("Then S should be unchanged", subtest.Value(s).DeepEqual([]int{}))
        })
    })
}

To shorten the implementation somewhat, we are using our own experimental matching library searis/subtest. Subtest works like other matcher libraries, but instead of taking t *testing.T as a parameter, like testify/assert, or initialize a matcher instance, like Gomega can do, it returns a test-function (a.k.a. sub-test) that we can pass directly to t.Run. A convenient side-effect of this is that subtest allows focus on individual checks via go test -run '^ParentTestName/SubTestName/CheckName$'.

PS! I want to underline that I don't view a matcher library to be a test framework. library != framework. If you are not convinced, it is possible to write tests without a matcher library as well, and in fact, this what the Go team does. It's not difficult to do the checks, and there are good arguments for doing the checks yourself, but it requires a very high discipline and use of boiler-plate to ensure consistently styled failure output.

But we are diverging... The code reads well now, but there is a problem!

Long test-names and duplicated information

These are the full test names that was generated by our code above:

TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S)/Then_it_should_not_fail
TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S)/Then_it_should_return_the_correct_sum
TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S)/Then_S_should_be_unchanged
TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S)/Then_it_should_not_fail
TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S)/Then_it_should_return_zero
TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S)/Then_S_should_be_unchanged

Once you mange to grok them, they make sene, but they are long, and it stutters. If you find the names themselves intimidating, try scanning them quickly from the go test output:

$ go test github.com/smyrman/blog/2020-06-test-with-expect/mypkg -run ^(TestSum)$
--- FAIL: TestSum (0.00s)
    --- FAIL: TestSum/Given_a_non-empty_int-slice_S (0.00s)
        --- FAIL: TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S) (0.00s)
            --- FAIL: TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S)/Then_it_should_not_fail (0.00s)
                sum_gwt_test.go:16: error is not nil
                    got: *errors.errorString
                        NYI
            --- FAIL: TestSum/Given_a_non-empty_int-slice_S/When_calling_Sum(S)/Then_it_should_return_the_correct_sum (0.00s)
                sum_gwt_test.go:17: not numeric equal
                    got: float64
                        -1
                    want: float64
                        6
    --- FAIL: TestSum/Given_an_empty_int-slice_S (0.00s)
        --- FAIL: TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S) (0.00s)
            --- FAIL: TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S)/Then_it_should_not_fail (0.00s)
                sum_gwt_test.go:25: error is not nil
                    got: *errors.errorString
                        NYI
            --- FAIL: TestSum/Given_an_empty_int-slice_S/When_calling_Sum(S)/Then_it_should_return_zero (0.00s)
                sum_gwt_test.go:26: not numeric equal
                    got: float64
                        -1
                    want: float64
                        0
FAIL
FAIL    github.com/smyrman/blog/2020-06-test-with-expect/mypkg  0.350s
FAIL

To me, one of the most obvious problem with the names is that space () is replaced by underscore (_); this makes them hard to scan. The default go test runner also repeats the parent names when printing the sub-test name, contributing to the problem. But perhaps the biggest problem, is actually that the names themselves are too long. In fact, the names themselves are duplicating information. In particular "When_calling_Sum(S)" is information that can already by understood by reading the top-level test-name TestSum. We are testing Sum -- how can we do that without calling it?

Naming a test after the type, function or method that is under test is a pretty common Go convention for unit-tests. And even we are writing this test as BDD, this particular test is a unit-test. If we can keep following this convention, it does makes the test behave more like other Go tests.

Test-With-Expect

The fundamental concepts that GWT offers, are pretty cool, but the words themselves -- Given, When, Then -- is actually less important. We will look at an alternative wording that fit better for Go in particular, but it can of-course apply elsewhere.

Another aspect wi will attack, is that GWT names are written to be human readable, and as they form near complete "English-like" sentences using what BDD-guys call natural language, they are also naturally long. If there is one idiom that is important in Go though, it is that names should be short and precise, rather then long-winded. That's not my words. Here is an extract from Rus Cox's porclaimed and well-worded naming philosophy:

A name's length should not exceed its information content. (...) Global names must convey relatively more information, because they appear in a larger variety of contexts. Even so, a short, precise name can say more than a long-winded one: compare acquire and take_ownership. Make every name tell.

Other advice and information that relate to Go names, include:

Names in Go have semantic effect.
One should avoid redundancy in names, E.g. package names + global names.
Short and concise appear more important than grammatical correctness. E.g. a constant named StatusFail read just as well as StatusFailure.

The first restriction to note here, is that names have semantic effect. Relevant to our case, all test function names in Go have to start with the word Test. Taking the consequence of this, we might as well include that as our first word. The next two words follow relatively naturally from that restriction:

Test: Type or function to test (subject).
With: Configuration or input that are some-how passed to subject (configuration)
Expect: What to expect afterwards (check).

To sum it up (no pun intended), these are our new test-names:

TestSum/With_non-empty_int_slice/Expect_no_error
TestSum/With_non-empty_int_slice/Expect_correct_sum
TestSum/With_non-empty_int_slice/Expect_input_is_unchanged
TestSum/With_empty_int_slice/Expect_no_error
TestSum/With_empty_int_slice/Expect_zero
TestSum/With_empty_int_slice/Expect_input_is_unchanged

Notice that we write just Expect_correct_sum over the more correct Expect_the_correct_sum_to_be_returned. or the previous Then_it_should_return_the_correct_sum. This is just an application of the Go naming philosophy to BDD-style natural language. Keep it Short, Precise, and Happily Sacrifice Some English Grammar -- or KISPAHSSEG to make an abbreviation that you wil defiantly remember!

PS! KISPAHSSEG, is a very inclusive version of English, especially for the non-native speaker.

Go TWE

Finally, here is the code for our tests in Test-With-Expect format:

package mypkg_test

import (
  "github.com/smyrman/mypkg"
)

func TestSum(t *testing.T) {
    t.Run("With non-empty int slice", func(t *testing.T) {
        s := []int{1, 2, 3}
        i, err := mypkg.Sum(s)
        t.Run("Expect no error", subtest.Value(err).NoError())
        t.Run("Expect correct sum", subtest.Value(i).NumericEqual(6))
        t.Run("Expect input is unchanged", subtest.Value(s).DeepEqual([]int{1, 2, 3}))
    }
    t.Run("With empty int slice", func(t *testing.T) {
        s := []int{}
        i, err := mypkg.Sum(s)
        t.Run("Expect no error", subtest.Value(err).NoError())
        t.Run("Expect zero", subtest.Value(i).NumericEqual(0))
        t.Run("Expect input is unchanged", subtest.Value(s).DeepEqual([]int{}))
    }
}

And the co-responding test output:

$ go test github.com/smyrman/blog/2020-06-test-with-expect/mypkg -run ^(TestSum)$
--- FAIL: TestSum (0.00s)
    --- FAIL: TestSum/With_non-empty_int_slice (0.00s)
        --- FAIL: TestSum/With_non-empty_int_slice/Expect_no_error (0.00s)
            sum_twe_test.go:14: error is not nil
                got: *errors.errorString
                    NYI
        --- FAIL: TestSum/With_non-empty_int_slice/Expect_correct_sum (0.00s)
            sum_twe_test.go:15: not numeric equal
                got: float64
                    -1
                want: float64
                    6
    --- FAIL: TestSum/With_empty_int_slice (0.00s)
        --- FAIL: TestSum/With_empty_int_slice/Expect_no_error (0.00s)
            sum_twe_test.go:21: error is not nil
                got: *errors.errorString
                    NYI
        --- FAIL: TestSum/With_empty_int_slice/Expect_zero (0.00s)
            sum_twe_test.go:22: not numeric equal
                got: float64
                    -1
                want: float64
                    0
FAIL
FAIL    github.com/smyrman/blog/2020-06-test-with-expect/mypkg  0.064s
FAIL

Extensions

Test-With-Expect is our base, but these three words are not always enough. Maybe you need more words? One additional word I have used myself, is After. It is not useful in this example, but what if you are writing tests that utilize a for-loop and do a check for each iteration? Other starting words could be called for in specific contexts, but be sure to limit the number and usage to ensure consistency.

TestX/With_Y/After_N_repetitions/Expect_no_error

You can also include information for failing tests without putting information in the name; just make a call to t.Log/t.Logf. By default, this output only appear if your (sub-)test actually fails.

The following code shows both a log statement and a useful setup/teardown pattern:

func TestResourceFind(t *testing.T) {
    setup := func(t *testing.T, cnt int) (r Resource, teardown func()) {
        t.Logf("Resource R set-up with %d records", cnt)
        // setup r with records ...
        // setup teardown function ...
        return
    }
        t.Run("With Query={Limit:5,Offset:32}", func(t *testing.T) {
        r, teardown := setup(t, 1000)
        defer teardown()
        // test r.Find ...
    })
    // ...
}

Improving test-runner output

So to the problems that we have not resolved. Can we print space instead of underscore? Can we avoid printing the parent names? Can we add color?

To answer all of those question at once, I would like to quote Bob the Builder:

Can we fix it? Yes we can!

We can even fix it without writing or using a separate test-runner; All we need is a separate test-formatter that can handle the output of go test -json. But this is an exercise for another blog post.

Besides, having the full test name printed in-tact does hold value; it can be copy-pasted and inserted into go test -run to focus on individual tests or groups of tests. Perhaps in the future our IDEs and editors can also insert links to re-run them?

Conclusion

As for a final conclusion, I want to leave this up to the reader. Do you think Test-With-Expect is a meaningful way to name sub-tests in Go? Would you prefer to use sub-tests, or would you prefer the DSL from one of the main frameworks?