Testing in parallel with Mocha v8.0.0

With the release of Mocha v8.0.0, Mocha now supports running in parallel mode under Node.js. Running tests in parallel mode allows Mocha to take advantage of multi-core CPUs, resulting in significant speedups for large test suites.

Read about parallel testing in Mocha’s documentation.

Before v8.0.0, Mocha only ran tests in serial: one test must finish before moving on to the next. While this strategy is not without benefits — it's deterministic and snappy on smaller test suites — it can become a bottleneck when running a large number of tests.

Let's take a look at how to take advantage of parallel mode in Mocha by enabling it on a real-world project: Mocha itself!

Installation

Node.js v8.0.0 has reached End-of-Life; Mocha v8.0.0 requires Node.js v10, v12, or v14.

Mocha doesn’t need to install itself, but you might. You need Mocha v8.0.0 or newer, so:

npm i mocha@8 --save-dev

Moving right along...

Use the --parallel flag

In many cases, all you need to do to enable parallel mode is supply --parallel to the mocha executable. For example:

mocha --parallel test/*.spec.js

Alternatively, you can specify any command-line flag by using a Mocha configuration file. Mocha keeps its default configuration in a YAML file, .mocharc.yml. It looks something like this (trimmed for brevity):

# .mocharc.yml
require: 'test/setup'
ui: 'bdd'
timeout: 300

To enable parallel mode, I'm going to add parallel: true to this file:

# .mocharc.yml w/ parallel mode enabled
require: 'test/setup'
ui: 'bdd'
timeout: 300
parallel: true

Note: Examples below use --parallel and --no-parallel for the sake of clarity.

Let's run npm test and see what happens!

Spoiler: It didn't work the first time

Oops, I got a bunch of "timeout" exceptions in the unit tests, which use the default timeout value (300ms, as shown above). Look:

  2) Mocha
       "before each" hook for "should return the Mocha instance":
     Error: Timeout of 300ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/boneskull/projects/mochajs/mocha/test/node-unit/mocha.spec.js)
      at Hook.Runnable._timeoutError (lib/runnable.js:425:10)
      at done (lib/runnable.js:299:18)
      at callFn (lib/runnable.js:380:7)
      at Hook.Runnable.run (lib/runnable.js:345:5)
      at next (lib/runner.js:475:10)
      at Immediate._onImmediate (lib/runner.js:520:5)
      at processImmediate (internal/timers.js:456:21)

That’s weird. I run the tests a second time, and different tests throw "timeout" exceptions. Why?

Because of many variables — from Mocha to Node.js to the OS to the CPU itself — parallel mode exhibits a much wider range of timings for any given test. These timeout exceptions don't indicate a newfound performance issue; rather, they're a symptom of a naturally higher system load and nondeterministic execution order.

To resolve this, I'll increase Mocha's default test timeout from 300ms (0.3s) to 1000ms (1s):

# .mocharc.yml
# ...
timeout: 1000

Mocha's "timeout" functionality is not to be used as a benchmark; its intent is to catch code that takes an unexpectedly long time to execute. Since we now expect tests to potentially take longer, we can safely increase the timeout value.

Now that the tests pass, I'm going to try to make them pass more.

Optimizing parallel mode

By default, Mocha's maximum job count is n - 1, where n is the number of CPU cores on the machine. This default value will not be optimal for all projects. The job count also does not imply that "Mocha gets to use n - 1 CPU cores," because that's up to the operating system. It is, however, a default, and it does what defaults do.

When I say "maximum job count" I mean that Mocha could spawn this many worker processes if needed. It depends on the count and execution time of the test files.

To compare performance, I use the friendly benchmarking tool, hyperfine; I'll use this to get an idea of how various configurations will perform.

Regarding hyperfine usage:

In the below examples, I’m passing two options to hyperfine: -r 5 for "runs," which runs the command five (5) times. The default is ten (10), but this is slow, and I'm impatient. The second option I pass is --warmup 1, which performs a single "warmup" run. The result of this run is discarded. Warmup runs reduce the chance that the first k runs will be significantly slower than the subsequent runs, which may skew the final result. If this happens, hyperfine will even warn you about it, which is why I'm using it!

If you try this yourself, you need to replace bin/mocha with node_modules/.bin/mocha or mocha, depending on your environment; bin/mocha is the path to the mocha executable relative to the working copy root.

Mocha's integration tests (about 260 tests over 55 files) typically make assertions about the output of the mocha executable itself. They also need a longer timeout value than the unit tests; below, we use a timeout of ten (10) seconds.

I run the integration tests in serial. Nobody ever claimed they ran at ludicrous speed:

$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel --timeout \
10s test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     141.873 s ±  0.315 s    [User: 72.444 s, System: 14.836 s]
  Range (min … max):   141.447 s … 142.296 s    5 runs

That's over two (2) minutes. Let’s try it again in parallel mode. In my case, I have an eight-core CPU (n = 8), so by default, Mocha uses seven (7) worker processes:

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     65.235 s ±  0.191 s    [User: 78.302 s, System: 16.523 s]
  Range (min … max):   65.002 s … 65.450 s    5 runs

Using parallel mode shaves 76 seconds off of the run, down to just over a minute! That's almost a 53% speedup. But, can we do better?

I can use the --jobs/-j option to specify exactly how many worker processes Mocha will potentially use. Let's see what happens if I reduce this number to four (4):

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js
  Time (mean ± σ):     69.764 s ±  0.512 s    [User: 79.176 s, System: 16.774 s]
  Range (min … max):   69.290 s … 70.597 s    5 runs

Unfortunately, that's slower. What if I increased the number of jobs, instead?

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 12 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 12 --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     64.175 s ±  0.248 s    [User: 80.611 s, System: 17.109 s]
  Range (min … max):   63.809 s … 64.400 s    5 runs

Twelve (12) is ever-so-slightly faster than the default of seven (7). Remember, my CPU has eight (8) cores. Why does spawning more processes increase performance?

I speculate it's because these tests aren't CPU-bound. They mostly perform asynchronous I/O, so the CPU has some spare cycles waiting for tasks to complete. I could spend more time trying to squeeze another 500ms out of these tests, but for my purposes, it's not worth the bother. Perfect is the enemy of good, right? The point is to illustrate how you can apply this strategy to your own projects and arrive at a configuration you're happy with.

When to avoid parallel mode

Would you be shocked if I told you that running tests in parallel is not always appropriate? No, you would not be shocked.

It's important to understand two things:

1. Mocha does not run individual tests in parallel. Mocha runs test files in parallel.
2. Spawning worker processes is not free.

That means if you hand Mocha a single, lonely test file, it will spawn a single worker process, and that worker process will run the file. If you only have one test file, you'll be penalized for using parallel mode. Don't do that.

Other than the "lonely file" non-use-case, the unique characteristics of your tests and sources will impact the result. There's an inflection point below which running tests in parallel will be slower than running in serial.

In fact, Mocha's own unit tests (about 740 tests across 35 files) are a great example. Like good unit tests, they try to run quickly, in isolation, without I/O. I'll run Mocha's unit tests in serial, for the baseline:

$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.262 s ±  0.026 s    [User: 1.286 s, System: 0.145 s]
  Range (min … max):    1.239 s …  1.297 s    5 runs

Now I'll try running them in parallel. Despite my hopes, this is the result:

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.718 s ±  0.023 s    [User: 3.443 s, System: 0.619 s]
  Range (min … max):    1.686 s …  1.747 s    5 runs

Objectively, running Mocha's unit tests in parallel slows them down by about half a second. This is the overhead of spawning worker processes (and the requisite serialization for inter-process communication).

I'll go out on a limb and predict that many projects having very fast unit tests will see no benefit from running those tests in Mocha's parallel mode.

Remember my .mocharc.yml? I yanked that parallel: true out of there; instead, Mocha will use it only when running its integration tests.

In addition to being generally unsuitable for these types of tests, parallel mode has some other limitations; I'll discuss these next.

Caveats, disclaimers and gotchas, Oh my

Due to technical limitations (i.e., "reasons"), a handful of features aren't compatible with parallel mode. If you try, Mocha will throw an exception.

Check out the docs for more information and workarounds (if any).

Unsupported reporters

If you're using the markdown, progress, or json-stream reporters, you're out of luck for now. These reporters need to know how many tests we intend to execute up front, and parallel mode does not have that information.

This could change in the future, but would introduce breaking changes to these reporters.

Exclusive tests

Exclusive tests (.only()) do not work. If you try, Mocha runs tests (as if .only() was not used) until it encounters usage of .only(), at which point it aborts and fails.

Given that exclusive tests are typically used in a single file, parallel mode is also unsuitable for this situation.

Unsupported options

Incompatible options include --sort, --delay, and importantly, --file. In short, it's because we cannot run tests in any specific order.

Of these, --file likely impacts the greatest number of projects. Before Mocha v8.0.0, --file was recommended to define "root hooks." Root hooks are hooks (such as beforeEach(), after(), setup(), etc.) which all other test files will inherit. The idea is that you would define root hooks in, for example, hooks.js, and run Mocha like so:

mocha --file hooks.js "test/**/*.spec.js"

All --file parameters are considered test files and will be run in order and before any other test files (in this case, test/**/*.spec.js). Because of these guarantees, Mocha "bootstraps" with the hooks defined in hooks.js, and this affects all subsequent test files.

This still works in Mocha v8.0.0, but only in serial mode. But wait! Its use is now strongly discouraged (and will eventually be fully deprecated). In its place, Mocha has introduced Root Hook Plugins.

Root Hook Plugins

Root Hook Plugins are modules (CJS or ESM) which have a named export, mochaHooks, in which the user can freely define hooks. Root Hook Plugin modules are loaded via Mocha's --require option.

Read the docs on using Root Hook Plugins

The documentation (linked above) contains a thorough explanation and more examples, but here’s a straightforward one.

Say you have a project with root hooks loaded via --file hooks.js:

// hooks.js
beforeEach(function() {
  // do something before every test
  this.timeout(5000); // trivial example
});

To convert this to a Root Hook Plugin, change hooks.js to be:

// hooks.js
exports.mochaHooks = {
  beforeEach() {
    this.timeout(5000);
  }
};

Tip: This can be an ESM module, for example, hooks.mjs; use a named export of mochaHooks.

When calling the mocha executable, replace --file hooks.js with --require hooks.js. Nifty!

Troubleshooting parallel mode

While parallel mode should just work for many projects, if you're still having trouble, refer to this checklist to prepare your tests:

• ✅ Ensure you are using a supported reporter.
• ✅ Ensure you are not using other unsupported flags.
• ✅ Double-check your config file; options set in config files will be merged with any command-line option.
• ✅ Look for root hooks (they look like this) in your tests. Move them into a root hook plugin.
• ✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode.
• ✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via --timeout)
• ✅ Ensure your tests do not depend on being run in a specific order.
• ✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files.

What's next

Parallel mode is new and not perfect; there's room for improvement. But to do so, Mocha needs your help. Send the Mocha team your feedback! Please give Mocha v8.0.0 a try, enable parallel mode, use Root Hook Plugins, and share your thoughts.

Comments

[Anton Korzunov]

🥳. 🥳! 🥳! 🥳!

It's been 84 years, but I could stick with mocha for another enternity.