Our monorepo pipeline took fifty-one minutes when it ran everything, so eighteen months ago we added path filters. Each service's test job carried an on.push.paths list, and a change under services/billing/** ran the billing tests and nothing else. Build time dropped to about seven minutes and everybody was pleased.
The failure arrived as a null pointer in production billing, from a change that had never touched the billing directory. Someone had edited libs/money/ to make a rounding helper return a nullable type for a legitimate reason in a different service. The billing tests would have caught it in four seconds. They did not run, because libs/money/** was not in billing's path list.
The filter lists had been written by hand once and then maintained by nobody. I diffed the declared paths against the real import graph and found eleven services whose tests could not be triggered by changes to code they depended on, plus three that ran on directories they had stopped importing two years earlier. The averages made this invisible: our green rate was 97 percent, our pipeline was fast, and the only symptom was that certain classes of bug reached production without ever meeting a red build.
The fix was to stop hand-writing the lists. A job now computes the changed files against the merge base, resolves them through the build tool's dependency graph, and emits the affected targets as a matrix. If libs/money changes, everything that transitively depends on it is in the matrix, because the graph says so and the graph is derived from the actual imports rather than from someone's memory in 2024. Anything outside the graph, CI config included, falls back to running the full suite.
I also added a small guard: if the computed set is empty and the diff is non-empty, the pipeline runs everything instead of passing instantly. That case used to be a silent green.
Selective CI is a good idea implemented as a cache invalidation problem, and hand-maintained lists are the worst possible cache key. If you skip tests, skip them because a dependency graph said so, and make the failure mode "ran too much", never "ran nothing".
– Sergey Shinder
Top comments (0)