Our main build took 41 minutes. Everybody hated it, nobody owned it, and every so often somebody would open a ticket titled "CI is slow" that went nowhere. I finally sat down with the raw timing logs instead of the vibes.
The first thing I found: we ran the full integration suite on every push to every branch, including branches that only touched the README. The second thing: we reinstalled the entire dependency tree from scratch every run because someone had disabled caching two years ago to debug a flake and never turned it back on. The third thing: we built the same Docker image three times in three different jobs.
None of that is clever. It's just accumulated debt that nobody had permission to look at.
So I stopped trying to make the slow steps fast and started making them not run. Path filters so doc-only changes skip the test matrix. A restored dependency cache keyed on the lockfile hash. Build the image once, push it to a scratch registry tag, and have downstream jobs pull it instead of rebuilding. I also split the integration suite so smoke tests gate the merge and the long tail runs post-merge on main, where a failure pages us but doesn't block fourteen people.
Build went from 41 minutes to 9. The number people actually felt, though, was the median PR: it dropped under 4 minutes because most PRs don't touch everything.
The lesson I keep relearning: the fastest pipeline step is the one that doesn't execute. Before you reach for a bigger runner or a fancier cache, ask what work is happening that didn't need to happen at all. Measure first — I wasted a full afternoon "optimizing" a step that turned out to be 30 seconds of the 41 minutes. The logs will tell you where the time goes if you actually read them instead of guessing.
Slow CI is rarely a hardware problem. It's usually a permission problem: nobody felt allowed to delete work.
– Sergey Shinder
Top comments (0)