How to spot async trap in tests

Async is hard. Especially in tests.

Even though the code used in this post is EmberJS specific, the techniques apply to any test code.

Some definitions

Ember has a very convenient way to make sure all async events will be finished before advancing to the next step. It's called ember runloop. So it's simply a matter of putting await in front of your manually triggered event.

But for code that lives outside the runloop (animation libraries, 3rd party scripts, CSS transitions, ...) we have an active waiter waitUntil() that is part of ember-test-helpers. Basically what this does is that it regularly executes given callback function until the callback returns true. Example:

console.log('Starting journey');

waitUntil(() => donkey.areWeThereYet(), {timeout: 10000});

console.log('Finished journey');

The problem

await searchButton.click();
assert.equal(searchButton.isDisabled, true, 'button is disabled after search request execution');

await waitUntil(() => resultsTable.hasSomeData);
assert.equal(searchButton.isDisabled, false, 'button is enabled after search finished');

The problem with this innocent looking test code is that inherently has a hidden race condition. The bug is in the first assert() call because it can be executed after the search has finished.

The easy way to find out such a bug is to apply the following rule:

Any active waiting outside of the runloop should be moved up to the nearest await.

Applying that rule to our code we will get:

await searchButton.click();
await waitUntil(() => resultsTable.hasSomeData);

assert.equal(searchButton.isDisabled, true, 'button is disabled after search request execution');
assert.equal(searchButton.isDisabled, false, 'button is enabled after search finished');

And from this one can easily see that there is a bug in my testing logic. Can't assert two opposite values for searchButton.isDisabled right after each other.

Solution

Solution for this depends on your framework, testing framework and application code, but in general, you should be safe if you stick to the following:

Never put any assert() statements between the last await and your active waiter.