An alternative to using Thread.sleep in your Java tests

At times the code you are testing runs in a separate thread than the thread executing the test itself. In this case, knowing when to perform the assertions can be troublesome as you must exercise some sort of synchronization between the threads.

A solution is to use Thread.sleep. While I personally do not not consider it wrong, using Thread.sleep has some drawbacks:

• sleep for too long and your test spends a lot of time doing nothing;
• not enough sleep and your test will fail; and
• sleep slight above the "necessary" and your tests might fail intermittently.

In this blog post I will show you an alternative to using Thread.sleep in your Java tests. This post assumes you can refactor the code you are testing.

The problem

To illustrate the problem let's jump right into our running example:

class AsyncCounter implements Runnable {
  static final int COUNT_TO = 1_000_000;

  private volatile int value;

  public int get() {
    return value;
  }

  @Override
  public void run() {
    while (value < COUNT_TO) {
      value++;
    }
  }

  public void startCounting() {
    var t = new Thread(this);

    t.start();
  }
}

It is a counter or incrementer that runs in a separate thread. You use it by invoking the startCounting() method; and you can query the current value by invoking the get() method. I know... it is of no practical use and has some warning signs, such as updating a volatile in a loop, but it should be enough as our example.

Let's write a test for it. The counter is fixed to count to one million. So, after it has finished its work, we expect the value to be one million.

Of course we could just invoke the run() method straight from our test. But bear with me, we want the counting to occur asynchronously.

An initial solution attempt could be:

public class AsyncCounterTest {
  @Test
  public void test() {
    var counter = new AsyncCounter();

    counter.startCounting();

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);
  }
}

But running this test fails:

FAILED: test
java.lang.AssertionError: expected [1000000] but found [0]

This is expected. Our assertion runs right after the thread starts and before the thread has a chance to do any work.

Using Thread.sleep

A solution could be to add a Thread.sleep after the counting has started and before the assertion, like so:

public class AsyncCounterTest {
  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter();

    counter.startCounting();

    Thread.sleep(1000);

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}

Apart from the Thread.sleep, I have also modified the test so it prints the total time (in ms) it took running the test. Running this version of the test in my machine gives:

totalTime=1003
PASSED: test

The question is: how much of the time sleeping is actual waiting for the counting to be over and how much is just sleeping and not doing much?

If we reduce the sleeping time from 1s to, say, 5ms and we ask TestNG to run the test three times the test fails (on my machine):

FAILED: test
java.lang.AssertionError: expected [1000000] but found [478484]

FAILED: test
java.lang.AssertionError: expected [1000000] but found [647833]

FAILED: test
java.lang.AssertionError: expected [1000000] but found [655242]

Finding an optimal waiting time is troublesome. Let's analyze some alternatives.

Alternative: use wait() and notify()

The first thing we need to do is refactor our AsyncCounter so it can send a signal when the counting is over. Let's create a listener interface with a single countingOver() method:

interface Listener {
  void countingOver();
}

And we invoke this method at the end of the run() execution: when the counting is over. The full modified code is listed below:

class AsyncCounter implements Runnable {
  static final int COUNT_TO = 1_000_000;

  private final Listener listener;

  private volatile int value;

  public AsyncCounter(Listener listener) {
    this.listener = listener;
  }

  public int get() {
    return value;
  }

  @Override
  public void run() {
    while (value < COUNT_TO) {
      value++;
    }

    listener.countingOver();
  }

  public void startCounting() {
    var t = new Thread(this);

    t.start();
  }

  interface Listener {
    void countingOver();
  }
}

Now we have to refactor our test as well.

After we start the counter we need to block the thread running the test until the counting is over. Remember, the counting happens in a different thread.

In Java we can do this by invoking the wait() method on a object that will be used for synchronization between threads. But before a thread can invoke wait() on an object, it must first acquire the lock on that object. We will use the test object itself for synchronization, like so:

counter.startCounting();

synchronized (this) {
  wait();
}

So the thread running the test:

• starts the counter and, therefore, starts the counting thread
• acquires the lock on the test object
• invokes wait() on the test object

Before it returns from the wait() method, in other words, inside the wait() method, the thread running the test:

• releases the lock on the test object
• blocks (stops executing code) and waits until notified by another thread

(to be exact the blocked thread can also stop waiting when interrupted. In this case the wait() throws an InterruptedException)

Next we have to ask the counting thread to notify the blocked thread when the counting is over. In Java we can do this by invoking the notify() (or notifyAll()) method on the object being used for synchronization. We are using the test object itself in our case.

So let's have our test class implement the Listener interface and let's code the countingOver() method:

@Override
public void countingOver() {
  synchronized (this) {
    notify();
  }
}

Remember, this method is invoked by the counting thread when the counting is over.

Just like the wait() method, before a thread can invoke the notify() method on an object, it must first acquire the lock on that object.

So the counting thread:

• acquires the lock on the test object. This lock is available. The testing thread released it as a consequence of invoking wait()
• invokes notify() on the test object. This notifies a single thread waiting on this object.
• releases the lock on the test object by exiting the synchronized block.

Notice that, since we know there is a single thread waiting, we invoke the notify() method and not the notifyAll() method.

Since now:

• the lock on the test object is available; and
• the testing thread has been notified by the counting thread

The testing thread will try to acquire the lock on the test object again. Once the lock is acquired, it will continue executing by exiting the wait() method.

The full version of the test is listed below:

public class AsyncCounterTest implements AsyncCounter.Listener {
  @Override
  public final void countingOver() {
    synchronized (this) {
      notify();
    }
  }

  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter(this);

    counter.startCounting();

    synchronized (this) {
      wait();
    }

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}

Running this test results:

totalTime=13
PASSED: test

Variation: use a Semaphore

I personally think the previous solution is enough. In particular because it is running in a testing environment.

Having said that, explicitly using both synchronized and Object.wait() has implications regarding virtual threads. Both will block the carrier thread of the virtual. But scheduler's behavior for each case is slightly different:

• a synchronized block (or method) causes the virtual thread to be pinned to its carrier. The scheduler does not compensate for that;
• on the other hand, Object.wait() is compensated by the scheduler: it may increase the number of carrier threads.

You can read more on the JEP 425 page.

The previous version is not wrong. We are not using virtual threads. Still, I think it would be nice to make the example more "loom-friendly". Or "forward compatible".

So let's write a variation of the alternative using a Semaphore instead. JEP 425 gives recommendations on how to mitigate the synchronized issue. It does not mention the Semaphore class explicitly. I am assuming the recommendations can be extended to it. I must stress though: this is an assumption.

First we need to get a Semaphore instance:

private final Semaphore semaphore = new Semaphore(0);

We initialized the semaphore to zero as we want the semaphore to block in the first acquire() invocation. So, right after the counting starts, we call acquire() on the semaphore:

counter.startCounting();

semaphore.acquire();

Once the counting is over, the counting thread must signal the waiting thread. It does so by calling release() on the semaphore:

@Override
public void countingOver() {
  semaphore.release();
}

The full test is listed below:

public class AsyncCounterTest implements AsyncCounter.Listener {
  private final Semaphore semaphore = new Semaphore(0);

  @Override
  public void countingOver() {
    semaphore.release();
  }

  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter(this);

    counter.startCounting();

    semaphore.acquire();

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}

Running it results in:

totalTime=12
PASSED: test

This variation is less verbose than the previous one.

It requires a new object instance and understanding of the Semaphore class though. In particular why it must be initialized with a zero value in this case.

Additionally, I must add that, apart from this blog post, I have never used a Semaphore before. I suspect this is not quite the use-case the designers of the class had in mind when creating it.

Test time-out

In both alternatives the testing thread will block until signaled by the counting thread. If for any reason the counting thread fails to signal then your test will block indefinitely.

Therefore I highly recommend that you add a time-out to your test. In other words, you should make sure your test fails if it does not complete after a specific amount of time.

How much time? Ironically, we got ourselves back to our initial problem, didn't we? Meaning that, if we decide on a high time-out value and in the case of a wrong implementation, our test suite will spend a lot of time doing nothing. Use a low time-out value and the test will fail before it can complete its work.

But, unlike the Thread.sleep situation, this will only happen when our implementation is wrong.

I am using TestNG. It provides a timeOut in its Test annotation. If the test does not complete in the specified value in milliseconds then TestNG fails the test.

@Test(timeOut = 100)
public void test() {
  // our test
}

I used 100 milliseconds. It is above the 12 milliseconds average I was getting running the tests.

Conclusion

In this blog post I presented two variations of an alternative to using Thread.sleep in your tests. It assumes you can refactor the code you are testing so a listener can be introduced.

All solutions (Thread.sleep included) have pros and cons. Which solution you choose depend on your requirements, personal preferences or comfort using one API or the other.

You can find the source code for all of the example in this GitHub repository.

Originally published at the Objectos Software Blog on May 23^th, 2022.

Follow me on twitter.