Most teams turn on parallelism in CircleCI, add circleci tests split, and assume the test-speed problem is solved. It usually isn't. The default split mode divides test files by name or count, which produces wildly uneven containers: one runs three seconds of smoke tests, another draws the 400-file integration suite that takes six minutes. Your slowest container sets the wall-clock time for the whole job — so an unbalanced split means you're paying for N containers but only getting the speedup of the busiest one.
CircleCI actually ships a fix for this: circleci tests split --split-by=timings, which balances containers by how long each test file actually took last time, not by file count or alphabetical order. Almost nobody uses it correctly, because the setup has a cold-start problem, a data-freshness problem, and a monorepo-namespacing problem the docs gloss over. Here's how to actually run it in production.
Why file-count splitting lies to you
Say you have parallelism: 4 and 200 test files, split alphabetically. Container 0 might get test_a.py through test_apple_pay.py — mostly checkout logic — while container 3 gets test_webhook_replay.py, test_worker_pool.py, and the rest of your slow integration coverage. CircleCI reports "4x parallelism," but your CI feedback loop is bottlenecked by whichever container drew the short straw. Teams "fix" this by manually moving files between glob patterns, which drifts out of sync within a month as the suite grows.
Splitting by timing data solves this directly: CircleCI redistributes files so each container's total historical runtime is roughly equal, not its file count.
Feeding CircleCI real timing data
Timing-based splitting isn't magic — it depends on JUnit XML test reports from a previous run of the same job name, uploaded via store_test_results. Without that step, --split-by=timings silently falls back to filename splitting, and you'll never know it happened.
# .circleci/config.yml
jobs:
test:
docker:
- image: cimg/node:20.11
parallelism: 8
steps:
- checkout
- run:
name: Install
command: npm ci
- run:
name: Run tests on this container's shard
command: |
TESTFILES=$(circleci tests glob "test/**/*.test.js" | \
circleci tests split --split-by=timings --timings-type=filename)
npx jest --ci --reporters=default --reporters=jest-junit $TESTFILES
- store_test_results:
path: reports/junit
The order matters: store_test_results has to run on every job execution, because CircleCI's Insights API is what --split-by=timings queries under the hood. Each successful run feeds the next run's split decision — it's a rolling feedback loop, not a one-time calibration.
The cold-start problem
The first time you turn this on, there's no timing history for the job name, so every file gets an equal default weight and you're back to a naive split. That's expected — don't panic and assume the flag is broken. What actually breaks things is doing this on a job you just renamed. CircleCI keys timing history by job name, so renaming test to test_unit resets your history to zero, and you'll silently get a bad split for a few runs while it re-learns.
If you can't tolerate a few uneven runs (e.g., a very large monorepo suite), seed the history manually: run the full suite once locally or in a throwaway pipeline, collect the JUnit XML, and push it through store_test_results on a dummy commit before switching real PRs onto the split job. It's a few minutes of setup that skips a week of your team wondering why "the fast CI feature" isn't fast yet.
New test files get the average, not zero
A file store_test_results has never seen (a test you added this morning) doesn't get scheduled first-come-first-served — CircleCI assigns it the average duration of all known files via --time-default=<seconds> (default: median of the observed set). This is usually fine, but it has a specific failure mode: if you add ten new, slow integration test files in one PR, they all land on the container with the most remaining capacity based on old data — and since CircleCI doesn't know they're slow yet, several new files can pile onto the same container and blow past your fastest container's time. This self-corrects after one run feeds the new timings back in, but if you're shipping a large batch of new slow tests, consider bumping parallelism for that one PR or manually spreading the new files across job invocations until the timing data catches up.
Namespacing timing data in a monorepo
If you run the same job definition for multiple services — say a shared test job template invoked once per service via matrix parameters — CircleCI's timing history is keyed by job name within the pipeline, not by which files you happened to pass in. Two services sharing a job name will pollute each other's timing data: the billing service's genuinely slow tests will skew the "average" applied to notifications' new files, because CircleCI can't tell the difference.
The fix is to make the job name service-specific rather than relying on parameters alone:
workflows:
test-all:
jobs:
- test:
name: test-billing
service: billing
- test:
name: test-notifications
service: notifications
Using name: on the job invocation (not just passing service as a parameter) gives each service its own timing bucket in Insights. This is a one-line change a lot of teams miss, and it's the difference between timing-based splitting actually working per-service versus quietly reverting to noisy, cross-contaminated averages.
Handling flaky-test reruns skewing your data
If you use CircleCI's automatic test re-run on failure, or a custom retry wrapper, be careful: a flaky test that gets retried three times before passing reports as three test executions in your JUnit XML, and depending on your test runner's reporter, that can inflate the recorded duration for that file by 3x. Over a few weeks, flaky tests silently become "slow" tests in the timing model and get over-weighted in the split, even though their real runtime (ignoring retries) is short. If your framework's JUnit reporter doesn't already dedupe reruns, filter the XML before store_test_results runs, keeping only the final successful attempt's timing per test case.
Rebalancing on a schedule, not just per-push
Timing-based splitting adapts continuously, but only across pushes that actually run the job — a service with low commit frequency can carry stale timing data for weeks. If your team merges to a long-lived branch daily but only occasionally touches a given service, add a lightweight nightly job (triggers: - schedule:) that runs the full suite for every service once a day purely to refresh the timing history, even if no one pushed code. It costs one extra full run a day and prevents the split from drifting badly out of balance for low-traffic services when a burst of commits finally does land.
Timing-based splitting isn't a switch you flip once — it's a small feedback system you have to feed correctly (store results every run), name correctly (per-service job names), and monitor for drift (flaky rerun inflation, cold starts after renames). Get those three right and parallelism: 8 actually means "roughly 8x faster," not "8 containers, one of which decides how long you wait."
Top comments (0)