Parallel Sub-tests in Go: A Cautionary Tale

By default Go executes tests sequentially, one test after another. In version 1.17 they added the ability to execute tests in parallel with a super easy and simple one line method t.Parallel().

func MyTest(t *testing.T) {
  t.Parallel()
}

Go will execute any tests that call this Parallel method, well, in parallel, by pausing each test and then resuming when all the sequential (tests that do NOT call the t.Parallel() method) have finished execution.

I won't dive too deep into that subject here, Jetbrains has a wonderful article already written you can read here that I used as a reference for this post.

What we're really here to discuss today is how t.Parallel() interacts with sub-tests in go.

But first, What is a sub-test and how do I use it?

A sub-test is a test function inside of a parent test function. You usually find them in situations where there is a common setup requiring a bunch of different conditions to be tested. Its a more readable and cleaner than throwing them all into a single HUGE function.

Below is a bare-bones example of a sequential test with a subtest:

import (
    "fmt"
    "testing"
)

func TestSequential(t *testing.T) {
    fmt.Println("Running main test...")

    t.Run("SubTest", func(t *testing.T) {
        subTest(t)
    })

    fmt.Println("Main test finished!")

}

func subTest(t *testing.T) {
    fmt.Println("Running Subtest!")
}

And here is the output of running that test. It's what you would expect:

Running main test...
Running Subtest!
Main test finished!

"Wow, that seems pretty useful!... wait, didn't you say this was a cautionary tale? What could go wrong with that?" You might be asking yourself.

Just like regular tests, you can also make sub-tests parallel by simply calling the t.Parallel() method! Pretty sweet right?

Well, you'd think so. But be warned. Parallel sub-tests play by their own rules!

Parallel sub-tests execute AFTER its main parent test has finished executing. And, its only after going slightly insane over a bug in one of our tests, that I stumbled upon this nugget of knowledge buried deep in a go dev blog in the Control of Parallelism section.

The below simple test set up demonstrates this issue:

func TestParallelSubTests(t *testing.T) {
    fmt.Println("Starting main test...")
    t.Run("SubTestOne", func(t *testing.T) {
        testOne(t)
    })

    t.Run("SubTestTwo", func(t *testing.T) {
        testTwo(t)
    })

    fmt.Println("Main test done!")
}

func testOne(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testOne!")
}

func testTwo(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testTwo!")
}

Based on our previous sub-test example, what do you think the output will be?

Well, as it happens. It actually executes these two sub-tests, testOne and testTwo AFTER the main test, TestParallelSubTests has already finished its execution.

Don't take my word for it. Here are the logs.

Starting main test...
Main test done!
Running testOne!
Running testTwo!

Things get even more confusing when you throw a defer statement in the mix

I've previously gone over defer statements in go and how they, like these parallel tests, also execute after the containing method has finished its execution. You can read more about that here.

So what happens when you mix the two, parallel sub-tests and defer statements?

They both execute after the main test has finished its execution, but it seems that defer statements will execute BEFORE your sub-tests.

Here is the test setup, it's the same as the previous setup, but with an added defer statement:

func TestParallelSubTests(t *testing.T) {
    fmt.Println("Starting main test...")
    t.Run("SubTestOne", func(t *testing.T) {
        testOne(t)
    })

    t.Run("SubTestTwo", func(t *testing.T) {
        testTwo(t)
    })

    defer deferredPrint()

    fmt.Println("Main test done!")
}

func testOne(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testOne!")
}

func testTwo(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testTwo!")
}

func deferredPrint() {
    fmt.Println("Deferred method!")
}

And here is its resulting output:

Starting main test...
Main test done!
Deferred method!
Running testOne!
Running testTwo!

So dear reader, BEWARE!

The parallel sub-test execution order can cause some serious stress induced headaches if you're unaware. Even more so when you're using it in conjunction with defer statements.

Its especially concerning when you consider that the main use case of these sub-tests is the common setup and teardown of testing data or infrastructure. If you're caught off guard you may be tearing down your test environment before running any tests!

Comments

[Seb C]

Hey,

You could use t.Cleanup() instead of defer.

[BlazingBits]

Thanks for the suggestion, that does the trick!

func TestParallelSubTests(t *testing.T) {
    fmt.Println("Starting main test...")
    t.Run("SubTestOne", func(t *testing.T) {
        testOne(t)
    })

    t.Run("SubTestTwo", func(t *testing.T) {
        testTwo(t)
    })

    defer deferredPrint()

    fmt.Println("Main test done!")

    t.Cleanup(func() {
        clean()
    })
}

func testOne(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testOne!")
}

func testTwo(t *testing.T) {
    t.Parallel()
    fmt.Println("Running testTwo!")
}

func deferredPrint() {
    fmt.Println("Deferred method!")
}

func clean() {
    fmt.Println("Cleaning!")
}

output:

Starting main test...
Main test done!
Deferred method!
Running testOne!
Running testTwo!
Cleaning