Tips for Terraform acceptance test: Disable test caching in Go

Today let's see a Go/Golang tips that can helps you ... in Terraform! 🙂

When you are developing, maintaining and updating a Terraform provider you certainly needs to execute a lot of time your acceptance tests.

Example with the OVHcloud Terraform provider:

Run the Kubernetes Node Pool main acceptance test:

$ make testacc TESTARGS="-run TestAccCloudProjectKubeNodePoolDataSource_basic"

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run TestAccCloudProjectKubeNodePoolDataSource_basic -timeout 600m -p 10
?       github.com/ovh/terraform-provider-ovh   [no test files]
?       github.com/ovh/terraform-provider-ovh/ovh/helpers   [no test files]
=== RUN   TestAccCloudProjectKubeNodePoolDataSource_basic
--- PASS: TestAccCloudProjectKubeNodePoolDataSource_basic (913.53s)
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh   913.963s
testing: warning: no tests to run
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh/helpers/hashcode  (cached) [no tests to run]

When you are modifying the code of the acceptance test, every time you run it, it is executed and ran.

But what happens if you want to run the test several time without modifying the code? The test is executed the first time and the test results are added in the cache to speed up the tests. The second time you execute it, the result are retrieved in the cache.

So if we execute the same acceptance test again:

$ make testacc TESTARGS="-run TestAccCloudProjectKubeNodePoolDataSource_basic"

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run TestAccCloudProjectKubeNodePoolDataSource_basic -timeout 600m -p 10
?       github.com/ovh/terraform-provider-ovh   [no test files]
?       github.com/ovh/terraform-provider-ovh/ovh/helpers   [no test files]
=== RUN   TestAccCloudProjectKubeNodePoolDataSource_basic
--- PASS: TestAccCloudProjectKubeNodePoolDataSource_basic (913.53s)
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh   (cached)
testing: warning: no tests to run
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh/helpers/hashcode  (cached) [no tests to run]

Let's zoom in on the line that interests us:

ok      github.com/ovh/terraform-provider-ovh/ovh   (cached)

Indeed, since Go 1.10 test results are automatically cached.

After reading the Go documentation, it seems that "the idiomatic way to disable test caching explicitly is to use -count=1" flag just after the go test command, so let's test it:

$ TF_ACC=1 go test -count=1 $(go list ./... |grep -v 'vendor') -v -run TestAccCloudProjectKubeNodePoolDataSource_basic -timeout 600m -p 10

?       github.com/ovh/terraform-provider-ovh   [no test files]
?       github.com/ovh/terraform-provider-ovh/ovh/helpers   [no test files]
=== RUN   TestAccCloudProjectKubeNodePoolDataSource_basic
--- PASS: TestAccCloudProjectKubeNodePoolDataSource_basic (965.96s)
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh   966.394s
testing: warning: no tests to run
PASS
ok      github.com/ovh/terraform-provider-ovh/ovh/helpers/hashcode  0.638s [no tests to run]

It's working, go test command bypass the cache!

The first intention of the count flag is to ask Go to run the test multiple times and benchmark the tests (so it omit the cache). So if you set the flag to count=1, Go will run the test only one time, without caching the result.

Conclusion

Even if Go seems to be an "easy to learn" language, it have a huge number of useful tips to know. Sometimes command flag, even undocumented ones, can help you ;-).