Our tests now run 2-4x faster, and cost about $0.09 a run, down from $0.17-0.34

The result in one table. Same tests, same results, measured against where we started.

Per code change	Before (one old job)	Now (one machine)	Change
Wait	29–57 min	about 15 min	48–74% less
Machine time	29–57 min	about 15 min	48–74% less
Estimated cost	$0.17–0.34	about $0.09	48–74% less
Estimated cost over 100 runs	$17–34	about $9	$8–25 saved

The cost rows are estimates: job minutes times GitHub's published rate, not our bill.

Abstract

Our automated tests took 29 to 57 minutes on every code change. Now they take about 15, on a single machine (48–74% less). Three changes did it. First, I split the tests into six groups that run at the same time. That cut the wait to 15.1 minutes (48–74% less), but used more machine time (78 minutes per run, 37–169% more). Second, I found one import that made every test file load about 10,000 files it never used, and replaced it. That cut the wait to 7.9 minutes (48% less than six groups) and the machine time to 44 minutes per run (44% less). Third, I came back to an idea I had first set aside, sharing test workers, fixed what it broke, and folded the six groups back into one job. That takes about 15 minutes of one machine per run (66% less than the 44 before). At GitHub's published rate of $0.006 a minute, that is about 9 cents a run, against 17 to 34 cents for the old single job (48–74% less). These are estimates from job lengths, not our bill. This post covers where the time went, what broke along the way, the idea I set aside and later shipped, and what is left to do.

The problem is that heavy tests make every run slow

Our admin web app has about 7,000 test cases, spread over 960 test files. Every code change runs them on CI, the automated checks that run on each change. All of them ran on one small machine with 2 processor cores, one after another.

That took 29 to 57 minutes. Every code change waits that long. The tests are heavy. Most of the run goes to loading the app, not testing it.

A side effect. On a busy machine, some tests timed out and failed for no real reason. The fix below solved that too.

A first idea was to run six groups at the same time

Split the 960 files into six groups. Run all six at once, each on its own machine. The test runner calls a group a shard. A final job waits for all six and reports one answer.

Same tests, six machines instead of one. I also made every test clean up after itself (fake parts, shared variables, timers), so no test can affect the next one.

The first result was 15.1 minutes, down from 29 to 57 (a cut of 48% to 74%). The trade was more machine time. Each group still pays the loading cost, so charged minutes per run went from 29–57 to 78 (37% to 169% more). A later fix, covered below, brought that down to 44 (44% less) and the wait down to 7.9 minutes (48% less).

The time went on getting ready, not testing

Before splitting anything, I measured. Loading the app's code before each test file was the biggest cost, at about 2,500 s of 3,800 s in total (66%), on a quiet machine. So most of the run was not testing. It was getting ready to test.

CI proves it also. One group took 786 s. Loading took 445 s of it (57%). The tests took 90 s (11%). Everything else was overhead.

The loading has a culprit. It is react-admin, the library our admin screens are built on. Loading it pulls in about 12,800 code files and takes about 5 s. Any test that touches it pays that price. I first blamed MUI, the UI component library. Wrong. I blamed the wrong library, confidently. MUI comes along for the ride, because react-admin loads all of it. Trimming MUI imports could not reach even a 10% cut.

I set aside sharing the test workers, then came back to it

By default, the test runner (Vitest) starts every test file in a fresh worker, a separate copy of the program. So the app code loads again for every file. The obvious fix is to share workers and load once.

I tried it. Loading dropped from 2,500 s to 131 s. That is −95%.

Then 211 files failed. One test leaves a fake part or a global value behind, and the next test trips over it. The suite had been leaning on fresh workers to hide that. I set it aside for this round.

I needed a sure win now, one I could prove with confidence. Sharing workers saves the most, but I had only a partial test of it, and 211 files must be fixed before I can trust it. The icon fix was one change I could measure in the real pipeline. So I shipped the sure one first, and I came back to sharing workers afterwards, below.

The cleanup that came with six groups

The cleanup step broke 8 files (63 tests) at first, because they relied on leftovers from earlier tests. I fixed it in five files of fake parts and one test. Boring, and necessary. The shared-workers trial pointed at the same problem.

Running the tests on my laptop barely changed. The median was 352 s against 340 s before, under my 374 s limit. The three runs ranged from 313 to 541 s on a busy machine, so call it close, not proven.

Then I found the real culprit, and the wait halved again

Back to the library that loads 12,800 files. I counted them by package. 10,612 are icon files from MUI's icon set. One line in react-admin asks for the whole icon set so it can use two icons. Every test file paid for all of them.

The fix is a stand-in that loads each icon only when a test asks for it. Tests still get the real icons. Nothing else changes. On a sample of 25 test files, loading fell from 99.9 s to 24.5 s (−75%), and all 230 tests still passed.

In CI the same six groups went from 9–15 minutes each to 5–8. The slowest group fell from 15.0 to 7.9 minutes. Charged minutes per run fell from 74 to 44, which is about what the old single job cost.

Then I tried the obvious next step: one machine, no groups. It hit the 30-minute limit and was cancelled. The six groups add up to about 42 machine-minutes, and one machine needs those 42 minutes in a row. So the groups stayed, until loading got cheap in another way (next section).

I also looked for more imports like it. One more exists. It is a date library, pulled in by a react-admin feature we don't use. Replacing it gains 9% on the sample, about one or two charged minutes. Not worth the extra moving part, so I left it.

Then I came back to sharing workers, and one machine was enough

The icon fix made loading cheap, but every test file still started a fresh worker. So I went back to the idea I had set aside: reuse workers, so the app loads once per worker instead of once per file. This time I fixed what broke before turning it on.

Sharing workers shows every leftover a test leaves behind. These turned up, one by one:

• The style cache stayed tied to the first file's page. MUI draws its styles through a library called Emotion. It is like pinning your notes to the wall of a room that was then torn down: later files drew into a page that no longer existed. Now there is one copy of the library, and each file gets a fresh cache.
• The click-and-type helper remembered the first file's document. It now gets the current one.
• The pop-up message library kept a queue between files. It is emptied before each file.
• React Query remembered the first file's window. It now points at the current one.
• Some test helpers left an app mounted after the file ended. They are now unmounted.
• Two real bugs in the app itself. Both were lists missing a React key. One made an expired-invitation chip warn on every draw. The other let an open tooltip jump to the next attachment after you deleted the first. I fixed both, each with a failing test first.

Then the real pipeline. The first set of runs failed: 159 of 990 files. A leftover app fired a notification when its page no longer existed. The error stopped React from resetting its own state in that worker, so every later draw in it failed with "Should not already be working." It did not happen on my laptop, even with the same random order, because my laptop is too fast for the race. I did not retry. I removed the leftover and started three fresh runs.

All three passed: 990 of 990 files, 7,438 tests passed and 1 skipped, in 14 min 20 s, 15 min 52 s and 14 min 14 s. I then switched the pipeline to the one machine, and the three runs on that setup took 13 min 5 s, 14 min 59 s and 17 min 3 s.

To keep it honest, test files now run in a random order, and every run prints its seed. Tests inside a file keep their written order. A new test that leaks fails loudly instead of passing thanks to its neighbours, and the seed lets me replay the exact order.

One caution. The spread is wide, from 13 to 17 minutes, and the median sits right at my 15 minute goal. If the suite grows, the wait will pass 15 first. My rule is to open a ticket if two runs in a row push the median past 15 minutes, or any run goes past about 22.

Clock time fell about 2–4x, cost fell by about half or more, to roughly 9 cents a run, and two bars missed at first

The wait went from 29–57 minutes to 15.1 with six groups, then to 7.9 after the icon fix. My goal was 15. The final setup, one machine with shared workers, took 14 min 59 s at the median of three runs, and 13 min 5 s to 17 min 3 s across them.

I set five bars. The six-group run before the icon fix, against the other four (the next lines give the newer numbers).

Bar	Limit	Result	Verdict
Timeout failures, 3 runs	0	0	Met
Minutes charged per run	80 or less	78	Met
Local test run, median	374 s or less	352 s	Met
Slowest ÷ fastest group	1.5 or less	1.7	Missed

One machine changes the bars. Balance no longer applies, because there is one group. The others hold: no timeout failures in the six green full runs, about 15 charged minutes (about 9 cents) against a limit of 80, and a median wait of 14 min 59 s against 15. I read charged minutes from job lengths, not from the bill, and I did not measure the local run again.

Two bars missed at first, and the thresholds are only my guesses

• Clock time was 15.1 min against a goal of 15. Close, not a pass. After the icon fix it was 7.9 min, met. On one machine it is 14 min 59 s at the median, with runs from 13 min 5 s to 17 min 3 s: met, with almost no room.
• Balance was 1.7 (slowest ÷ fastest group) against my 1.5 bar. It was 1.4 the run before. The split is uneven and moves between runs. After the icon fix it was 1.5 (475 s over 320 s), which meets the bar. On one machine there is a single group, so this bar no longer applies.
• The thresholds are mine. 15 min, 80 charged minutes, 1.5×. I picked them. They are a call between speed and spend, not a law.
• The price is an estimate. GitHub publishes the rate for our runner: $0.006 per minute for a Linux 2-core machine, rounded up to whole minutes per job. I did not read our bill, so the real figure may differ. At that rate, the old single job's 29 to 57 minutes cost about $0.17 to $0.34, six groups' 78 about $0.47, the 44 after the icon fix about $0.26, and the one machine's 15 about $0.09. The lint check and the final gate add about $0.02 to each.

Where the numbers come from

Timings come from my own runs on a laptop and from the run logs of our CI pipeline (GitHub Actions, with Vitest as the test runner). The old 29–57 minute range is from our CI history before the change. The one-machine times are the job lengths GitHub reports, and the test counts come from the run logs. The dollar figures use GitHub's published runner rate (docs.github.com/en/billing/reference/actions-runner-pricing), applied to job lengths.

What I'd do again

• Measure first. I blamed the wrong library until the numbers said otherwise.
• No retries. A flaky test should look flaky.
• Clean up after every test. It is the price of running them side by side, and sharing workers found every one I had missed.
• Set up CI on a cold machine early. It found problems my laptop missed. The failure in the first try at sharing workers never showed up there, even with the same seed.