Test in Go: trick of Cleanup

My team is using testify as the testing framework for Go. We noticed that many methods provided by testify/suite can be easily achieved by the built-in testing package. For example, using Cleanup can do the same of AfterTest provided by testify/suite. It makes me wondering if we can just use Cleanup to replace as much methods from third-party packages as possible.

TL;DR

Cleanup can replace methods that are not relevant to subtests, which are AfterTest, TearDownAllSuite and TearDownTestSuite.

Cleanup can’t replace TearDownSubTest because there is a risk that a subtest pollute its parent test.

The experiment

I use a global variable externalResource as the external resource. It’s value is 1.

When preparing a subtest at SetupSubTest, externalResource is changed to 2. At the same time, a cleanup function is registered to restore externalResource to 1.

I expect that externalResource is set to 2 before the subtest and restored to 1 once the subtest is finished. So, from the point of parent test, externalResource is always 1. From the point of subtest, externalResource is always 2.

The code is

var externalResource = 1

type TestUnexpectedCleanupOrder struct {
    suite.Suite
}

func (s *TestUnexpectedCleanupOrder) TearDownTest() {
    fmt.Println("TearDownTest")
}

func (s *TestUnexpectedCleanupOrder) SetupSubTest() {
    externalResource = 2
    fmt.Printf("SetupSubTest. The externalResoure is: %d\n", externalResource)
    s.T().Cleanup(func() {
        externalResource = 1
        fmt.Printf("SetupSubTest. The externalResoure is: %d\n", externalResource)
    })
}

// Test case
func (s *TestUnexpectedCleanupOrder) TestCaseA() {
    fmt.Printf("Start TestCase A. The externalResoure is: %d\n", externalResource)

    s.Run("Sub-TestCase A", func() {
        fmt.Printf("Finish Sub-TestCase A. The externalResoure is: %d\n", externalResource)
    })
    fmt.Printf("Finish TestCase A. The externalResource is: %d\n", externalResource)
}

The result

The output of the above code is (with line number)

1 Start TestCase A. The externalResoure is: 1
2   SetupSubTest. The externalResoure is: 2
3   Finish Sub-TestCase A. The externalResoure is: 2
4 Back to TestCase A. The externalResource is: 2
5 TearDownTest
6   Cleanup subTest. The externalResoure is: 1

Notice line 4 and line 6. The externalResource is still 2 when the subtest is finished and the parent test is resuming! externalResource is not cleaned until the parent test is finished.

According to the documentation of Cleanup:

Cleanup registers a function to be called when the test (or subtest) and all its subtests complete. Cleanup functions will be called in last added, first called order.

It only guarantee that a registered function is called after something.

It does NOT guarantee when a registered function is called before something.

The solution

If we add a TearDownSubTest to reset externalResource, then its value is cleaned as we need

func (s *TestUnexpectedCleanupOrder) TearDownSubTest() {
    externalResource = 1
    fmt.Printf("\tTeardown subTest. The externalResoure is: %d\n", externalResource)
}

Start TestCase A. The externalResoure is: 1
    SetupSubTest. The externalResoure is: 2
    Finish Sub-TestCase A. The externalResoure is: 2
    Teardown subTest. The externalResoure is: 1
Back to TestCase A. The externalResource is: 1
TearDownTest
    Cleanup subTest. The externalResoure is: 1

Summary

We can use Cleanup to clean up resources for normal tests. However, we need to use TearDownSubTest to clean the resource held by subtests.

The source code

https://github.com/xuanyuwang/testify-examples/tree/main/unexpected-cleanup-order