Vitest vs Jest benchmarks on a 5 year old real work SPA

#vitest #benchmark #unittests #jest

Intro

The following is about a 5½ year old production web app with 1,256 real world unit tests across 139 test suites. This production app has been the primary daily job of ~5 engineers since it's inception. Building it out, updating it, adding/demoing features, etc. It's a rare, real-world example of a largish webapp.

Motivation to switch to Vitest

Due to a bug that has existed in Jest since 2020, my team at work had to decide to either, fix the bug in Jest's repo.... or switch over to Vitest which doesn't have the bug. And since Jest's repo would require someone on our team to deal with a large Yarn mono repo, and TypeScript, it seemed like it might be worth it to avoid going that route.

Notably, we were not concerned with performance. So these benchmarks are more just personal curiosity than anything. But still worth sharing.

Also worth a note, these benchmarks were ran on the "worst case scenario" machine (the one with the slowest results to emphasize the biggest difference). I will also mention the results on the best case scenario (our CI).

Converting over the smaller repo first

Before taking on the multi-week task of converting the larger repo over to Vitest, it made sense to do a sort of "trial run" on the smaller repo.

Smaller Repo Stats:

Vue 2 + Vuex + Vue-Router + Vite + Element-UI
App Routes: 6
Test Suites: 14
Unit tests: 147
Snapshots: 64
From: Jest + JSDOM
To: Vitest + HappyDOM
Added 3 new dependencies, removed 7
Benchmarks on slowest machine:
- npm run unit (cold)
  - Jest - 35.79s
  - Vitest - 21.31s
- npm run unit (warm)
  - Jest - 27.77s
  - Vitest - 16.38s
- npm run unit-coverage
  - Jest - 25.39s
  - Vitest - 18.75s

So as you can see, Vitest actually outperformed Jest in all scenarios on the smaller repo. However, I think this had a lot more to do with switching to HappyDOM than Vitest itself.

Bigger Repo Stats:

Vue 2 + Vuex + Vue-Router + Vite
App Routes: 73
Test Suites: 139
Unit tests: 1,256
Snapshots: 517
From: Jest + JSDOM
To: Vitest + JSDOM
Added 3 new dependencies, removed 3
Benchmarks on slowest machine:

Tool	Coverage	Real-Time	Self-Time	Transform	Setup	Collect	Tests	Env	Prepare
Vitest			320.90s	9.71s	82.14s	442.08s	325.19s	232.20s	56.45s
Vitest			318.71s	9.73s	83.86s	447.80s	295.65s	240.87s	57.98s
Vitest		332.02s	327.54s	8.97s	82.09s	455.03s	328.10s	241.90s	56.88s
Vitest		415.32s	408.03s	9.80s	99.78s	568.28s	420.10s	294.90s	70.65s
Vitest	✔️		297.17s	9.47s	115.97s	638.88s	538.10s	370.47s	96.27s
Vitest	✔️		272.07s	9.37s	100.96s	576.24s	540.31s	328.53s	84.68s
Vitest	✔️	293.33s	287.45s	11.15s	111.67s	624.03s	551.18s	348.24s	92.34s
Vitest	✔️	272.23s	265.44s	9.17s	102.19s	580.97s	491.47s	320.93s	83.75s
Jest		213.68s	207.69s
Jest		201.76s	191.56s
Jest	✔️	226.06s	221.30s
Jest	✔️	273.54s	267.91s
Vitest --no-isolate			115.94s	12.73s	6.48s	53.88s	264.15s	7.65s	2.61s
Vitest --no-isolate	✔️		242.30s	10.53s	13.89s	121.78s	532.12s	33.38s	109.14s

Benchmark Results:

Note that "Real-time" was me timing the run with a stop watch and "Self-Time" is what the test runner reported as the total duration. The two are actually always within a few seconds of each other.

Average without coverage:

Vitest = 322s (+37.8% time) on average 122 seconds slower
Jest = 200s

Average with coverage:

Vitest = 281s (+12.8% time) on average 36 seconds slower
Jest = 245s

You'll note that this time around Vitest was actually a little slower than Jest. And very weirldy, Vitest with coverage reporting enabled is actually faster than with it off. I believe this is mostly due to not switching to HappyDOM. There were enough test failures that needed fixing that just swapping out Jest was enough work for ~2 weeks. So I decided to skip HappyDOM.

But how can we make it faster?

In my last Dev.to post, I documented many ways to make Vitest go faster:

https://dev.to/thejaredwilcurt/improving-vitest-performance-42c6

Here are some of the things from that post that I tried, and the results:

--single-thread and --no-threads didn't work. Basically no test would even run. This may just be the first test failing and the rest choked, but I didn't want to bother to find out.
Both css: false and useAtomics: true actually made it slower
With --no-isolate it was 2.8x faster than vitest and 1.7x faster than Jest, but 19 tests failed (see table above). Some people report issues with watch mode when using --no-isolate. So I decided to not pursue it any further. Once the vm module that Vitest relies on supports ESM, or when the amazingly named Shadow Realms are added to JavaScript, we will likely get this performance boost for free without the downsides.

What about on CI?

All of our repo's are "branch protected" on GitHub. They must pass ESLint, SassLint, Build, Unit tests, and E2E tests.

When running on GitHub Actions (GHA), we see average times of:

Jest = 128s
Vitest = 138s

Because it's only a 10 second difference to run the full suite on GHA, and we primarily just run specific test suites when working locally. The difference in time is negligible. Also, we know that eventually the bottlenecks preventing Vitest from going faster will be removed and it will then start easily out-performing Jest.

So we are sticking with Vitest going forward. At least... once this 239 file PR with +10,486, -36,461 diff gets reviewed 😰.

Wait did you say Vue 2?

Yes, these projects are using Vue 2. Vue 3 did not exist in 2017/2018 when these projects were started. We have plans to convert the larger repo to Vue 3 by the end of the year. And we will be sticking with the superior Options API in all repos.

Though newer repos we've started use this boilerplate. I would recommend it to everyone, for every project.

Photo Credit: