Vitest 4 to 5 on a 3,000-file test suite: what the upgrade bought us

We moved our test runner from Vitest 4.1.8 to Vitest 5.0.4 this week. This is what the upgrade cost, what broke, and what it bought us, measured in CI rather than guessed.

The setup

• About 3,000 test files.
• Three CI lanes (shards), merged into one coverage report at the end.
• Around 139 CI runs a day on self-hosted runners.
• Node 22, the forks pool, test isolation on.

That last line matters. The Vitest team published benchmarks for several setups, and the one closest to ours (forked workers, isolated files) showed a 19% speedup. I wanted to know whether that held on our suite.

How I measured

I timed the Vitest step inside each full-suite CI lane, not the whole job. Job time includes checkout, install and caching, which have nothing to do with the test runner.

	Lanes	Median	Mean
Vitest 4.1	243	730 s	781 s
Vitest 5.0	6	600 s	647 s
Change		−17.8%	−17.1%

That is about 18% faster, close to the published 19%.

Six samples is an early read. A 90% bootstrap interval for the change in median runs from −30% to +15%, so it still includes zero. I will post a follow-up after a week of main-branch runs.

What broke

Four things, all small:

1. An unawaited .resolves now fails the test. Vitest 5 treats a dangling expect(...).resolves as an error. Adding await fixed it, and the test was wrong before.
2. vi.mock inside a test body now throws. Mocks have to live at module scope. One of the two offenders mocked a file that does not exist. It had been dead code for months, and nothing noticed.
3. Blob reports moved to a new default folder. Our sharded CI merges blob reports from each lane. I pinned the old output path, so the pipeline needed no changes.
4. The coverage gate went red with no code change. Vitest 5's coverage remapping counts ?? fallbacks as branches. One file under a 100% branch threshold had an unreachable ?? 0. Switching match() to exec() removed it; behaviour is identical.

The new default that clears mocks between tests caused zero failures, which surprised me.

What I got wrong while measuring

1. My first baseline mixed run types. Some PRs only run tests related to changed files. Mixing those with full-suite runs gave a median of 394 seconds, which meant nothing. I split runs by type using the duration of the coverage merge step: related-only runs skip it.
2. My "Vitest 5" benchmark ran Vitest 4. I dispatched a scheduled coverage workflow on the upgrade branch, but that workflow always checks out main. The run looked like a v5 result and was not.

Lessons

• Time the step, not the job.
• Split runs by type before comparing them.
• Check which commit a benchmark actually ran on.
• Read the upstream benchmark, find the setup closest to yours, and use it as your prior.

Was it worth it?

If the 18% holds, that is about 8 runner-hours a day back across our CI volume. The wall-clock gain per pull request is smaller, about 2 minutes, because browser tests are often the slowest job and they did not change.

Migration took about half a day. For a suite this size, that pays back quickly.

I will update this post with the confirmed number.