Beware silently skipped tests in Mocha

I'm a big believer in test-driven development, though I admit I'm a recent convert. My framework of choice is Mocha.

There are a couple of gotchas when using Mocha that can cause you to accidentally skip tests. The scary version of this results in Mocha telling you that all your tests have passed, when in fact some of your tests weren't run at all!

Here are the two causes of this that have hit me particularly hard, and that are both easy to fix:

1. Async errors thrown outside of test definitions are silently swallowed by Mocha.js.
2. If you frequently use the .only flag to limit test runs to a subset of tests, you might forget to remove that flag.

Silent, swallowed errors.

A while ago there was a CLI tool I was building that had been merrily passing all of its tests for weeks. When I finally made it available for the team, it was completely broken. Weirdly, it was broken in ways that I knew I had test coverage for. How could this be?

It turns out that errors thrown in async contexts can cause Mocha to exit early without registering all tests, all while swallowing the error that that caused it to happen! Observe:

describe('My test suite', async function(){

  // throw new Error("Bwahahaha! Tricked you!");

  it('can pass this test', async function(){
    // This will "pass", even without any code,
    // since Mocha tests pass unless an error is thrown.
  });

  it('cannot pass this test', async function(){
    throw new Error("OH NOOOOOO!");
  });
});

This test runs as expected, informing us that one test passed and one failed:

But what happens if we uncomment out that extra thrown error? Despite there now being two explicitly thrown errors in that little Mocha snippet, we get this:

Yeah, sure it says zero passed, which sounds like a failure. But that's in green, because Mocha saw zero tests! This is a success state, because Mocha doesn't care about things that pass (or that nothing passed), only things that fail. And nothing failed, according to Mocha.

When something fails, Mocha exits with a non-zero status. That non-zero exit would be used to inform downstream tools that something has gone wrong, preventing your automated pipelines from continuing when tests fail. But here we got a 0 status despite obvious test failures.

Even without the automation problem, this same bug can be hard to spot when doing things manually. Sure, in this case "0 tests passed" is quite obviously wrong. But this problem can cause a subset of tests to get skipped, so you might see "321 tests passed" when there should have been "351". If you hadn't memorized how many tests you had there'd be no way to realize you were skipping tests.

As a workaround, you can tell the Node process to catch such errors and force a non-zero exit status:

function onUncaught(err){
  console.log(err);
  process.exit(1);
}

process.on('unhandledRejection', onUncaught);

describe('My test suite', async function(){

  throw new Error("Bwahahaha! Tricked you!");
  // ...

And now we get:

NOTE: While you can, technically, use async callbacks in your describe()s, it very likely will not behave the way that you expect! If you remove the async in the example's describe, the thrown error is no longer swallowed. The protection mechanism shown here should be treated as a backup for accidentally making a describe() async!

Forgotten .only()

When you're actively working on a new feature, or debugging an existing one, the test-driven approach is to first write the tests, ensure they're failing where they should be, and then code until all tests are passing.

If you're doing this in the context of a large project, you probably don't want to be running all tests just to see if the current thing is working. Mocha provides a few mechanisms for dealing with that, the easiest one being to use .only to indicate that only that test (and any others similarly flagged) should be run:

describe('My test', function(){
  it.only('will run this test', function(){});
  it('will not run this test', function(){});
});

But what happens when you inevitably forget to remove that .only to ensure your other tests run again? You'll be bypassing tests that might be failing! I've done this countless times myself.

Mocha has a great solution for this: the --forbid-only flag.

When you add this flag to your CLI call, Mocha treats the mere existence of .only in any part of your test code as a test failure, and exits with a non-zero status.

For my build pipelines I always use this flag. It pairs nicely with --bail, which aborts as soon as a single test fails so that you don't waste time running other tests on a bad build.

This article was adapted from the DevChat newsletter.

Comments

[Juerg B.]

There is one fundamental mistake in your sample above, you can't use describe in an asynchronously way. So this code snipet is wrong: describe('My test suite', async function(){ ...
Only hooks and tests can have async callbacks.

[Adam Coster]

Ohhhhhhhhh, good catch! That makes sense and, in retrospect, explains a few problems I had in the past...