Writing parallelizable tests with the carlspring/idempotence framework for Java

Disclaimer

This is a re-post of my article on Medium.com with the hope of reaching more OSS developers as part of #Hacktoberfest.

Introduction

Have you ever had to deal with flaky parallelized tests that pass most of the time but then suddenly start failing with inconsistent random errors? Have you ever had to fix tests that share the same resource files and modify them in parallel to confuse the heck out of you and the outcomes of your other tests? Have you spent countless nights trying to refactor such code so that it can be properly parallelized and have reproducible and guaranteed results?

This is a complex topic that is not always straight-forward to resolve, especially in existing large code bases. However, following a simple set of rules can help you get there and the carlspring/idempotence framework aims to help with this.

What is test isolation?

In order for tests to be reproducible all of the time, you need to make sure that their resource files are contained and isolated just to them. What this means is that each test should own its test resources exclusively and other tests should not modify them.

What is test idempotence?

Test idempotence means that your tests will always return the same results. No matter how many times they have been exectued and no matter what other tests are running in parallel.

What is the carlspring/idempotence framework for Java

This is a leightweight framework which helps defining and copying test resource files in an isolated way for your JUnit5 tests. The test resources are defined using annotations and are copied in their own directories to help with the implementation of test resource separation and isolation.

How the Java carlspring/idempotence framework works

All common test resources are stored under the src/test/resources directory as usual. Each test method then defines the resources it needs by using the @TestResources annotation. The framework copies these resources to an isolated directory for each test method. This ensures that it has exclusive access to the resources it requires, preventing interference from other tests running in parallel, including other test methods in the same test class.

For each build tool, there is a separate dependency that contains path-related transformation logic for the specific directory layout of that tool. (As a very simplified example, among other things. Maven places the built code under target, whereas Gradle uses build for this purpose; resources are placed differently, etc). More on this will be explained below.

How to write tests using the Java carlspring/idempotence framework

Here are the step you will need to get started.

Define The Dependencies

You will need to define the respective dependency for your build tool. You can check what the latest released version is here.

• For Gradle (using Groovy DSL):

testImplementation "org.carlspring.testing.idempotence:idempotence-gradle:1.0.0-rc-3"

• Gradle (using Kotlin DSL):

testImplementation("org.carlspring.testing.idempotence:idempotence-gradle:1.0.0-rc-3")

• For Maven:

<dependency>
    <groupId>org.carlspring.testing.idempotence</groupId>
    <artifactId>idempotence-maven</artifactId>
    <version>1.0.0-rc-3</version>
    <scope>test</scope>
</dependency>

Add Annotations

Your test class will have to be annotated with the @ExtendWith(TestResourceExtension.class) annotation. This annotation is responsible for the actual copying of the resources.

You will also need to annotate your test methods with the @TestResources annotation to specify the resources that they need.

For example:

package com.foo;

import org.carlspring.testing.idempotence.annotation.TestResource;
import org.carlspring.testing.idempotence.annotation.TestResources;
import org.carlspring.testing.idempotence.extension.TestResourceExtension;

@ExtendWith(TestResourceExtension.class)
class MyTest {

    @Test
    @TestResources({ @TestResource(source = "classpath:/foo.txt"),
                     @TestResource(source = "classpath*:/**/bar.txt")} )
    void testFoo()
    {
        // Perform whatever checks you need using these resources
    }

}

For each test method a directory will be created using the following format:

• For a Gradle project, with a test called MyTest with a method testFoo, they will be placed under:

build/test-resources/MyTest-testFoo/nested/dir/foo.txt
build/test-resources/MyTest-testFoo/bar.txt

• For a Maven project, with a test called MyTest with a method testFoo, they will be placed under:

target/test-resources/MyTest-testFoo/nested/dir/foo.txt
target/test-resources/MyTest-testFoo/bar.txt

This way your tests will have the resource that they require copied into their own isolated directories. At this point you can modify these test resources from the test methods they belong to and your results should be idempotent, provided that they only depend on file based resources and not other types of shared resources (database, third-party services, etc).

Where to find the documentation

The documentation of the Idempotence project can be found here.

You can have a look at the Conceptual Overview for more detailed explanation of how things work.

How to contribute

This is a greenfield project with the core functionality and infrastructure in place, but help is always welcome.

Contributors who have experience with JUnit, Springframework, MkDocs could help shape the project with some great ideas and solutions. Early adopters who could provide feedback are also very welcome!

Issues labelled hacktoberfest or help wanted are up for grabs and should help you get started quickly. You can find them here.

Conclusion

One of the most important things when writing test cases is the test data that your tests will use and keeping it sane between runs. By following a set of simple rules to keep this data isolated between your tests, you can achieve idempotence and reliability of your results.

The carlspring/idempotence project provides and easy to use framework which is suitable for both greenfield projects and refactoring legacies.