Our pipeline took twenty nine minutes end to end and the test stage accounted for twenty six of them. The stage runs in twelve parallel containers, and the summary line on our dashboard said the average test job took three minutes and forty seconds. Both numbers were correct. Only one of them decided when anybody could merge.
We split the suite by dividing the list of test files into twelve equal parts, in whatever order the file system handed them over. Two thousand nine hundred tests across two hundred and fourteen files, eighteen files per container, and a shard boundary drawn without reference to what any of those files do. The container holding the four integration classes, each of which boots a database and a message broker before it asserts anything, ran for twenty six minutes. Nine of the twelve finished inside two minutes and then sat there, allocated, billed and doing nothing. The fastest took eighty seconds.
March had made it worse in a way nobody could have seen. A colleague consolidated about four hundred repetitive unit tests into one parameterised class, which is unambiguously better code and turned four hundred cheap files into one cheap file. Our splitter counts files. After that change the counts were not even a rough proxy for the work.
The fix is small. Every run writes a file of per class durations into the cache. The next run reads it and packs the classes into twelve bins by longest processing time first, which is a scheduling problem with a known greedy answer that is good enough. New classes with no history go to the shortest bin. The pipeline prints each container's duration as a bar, and the stage fails if the spread between the longest and the shortest is over twenty five percent, because that means the timings have gone stale and nobody would otherwise notice.
The test stage now finishes in six and a half minutes with the same tests on the same runners. Our runner bill for that repository fell by about a fifth as well, since we had been paying for nine idle containers to watch one work.
A parallel stage lasts as long as its unluckiest partition. Dividing the files evenly is easy and feels fair, and it quietly assumes every file costs the same, which is true of no test suite I have ever met.
– Sergey Shinder
Top comments (0)