Nobody ever complained about it directly. The pipeline took a little over forty minutes end to end, which had crept up gradually from about eight, and every individual addition had been justified. What I noticed instead were the second-order effects, and it took me an embarrassingly long time to connect them to the build time at all.
Pull requests got bigger, because if you are going to wait forty minutes you may as well bundle three changes into one trip. Reviews got shallower, because by the time the checks finished the reviewer had context-switched twice and just wanted it merged. People stopped pushing work-in-progress branches to get early signal, so integration problems surfaced later and larger. And the number of changes that went out on a Friday afternoon dropped to zero, not because of any policy, but because nobody wanted to start a forty-minute wait at four o'clock. Our release cadence had been quietly reshaped by a number that appeared on no dashboard anyone reviewed.
The worst habit it produced was speculative pushing. When feedback is slow, you stop reasoning carefully about whether a change is correct and start using the pipeline as a way of finding out, firing off a commit with a plausible guess and going to do something else. That doubles the load on the very system that is already too slow, which lengthens the queue, which makes people push even more speculatively. It is a feedback loop that only runs in one direction.
Fixing it was unglamorous work. We measured where the time actually went rather than where we assumed it went, and found roughly half of it in dependency resolution that could be cached and an integration suite that was re-running the same container build three times. We split the pipeline so that the fast, high-value checks report in under five minutes and the slower ones run afterwards. We deleted some tests that had never once failed for a real reason.
Build duration is not a developer convenience metric. It sets the size of the batches your organisation ships, the depth of your reviews and the honesty of your engineers' guesses, and it does all of that without ever appearing in a discussion about engineering quality.
– Serguey Shinder
Top comments (0)