For about a week, roughly one build in six failed on a test that had nothing to do with the change. Rerunning it usually worked. We had learned to rerun things, which is its own problem, and it took an afternoon of staring at two logs side by side before I noticed that the failing runs restored a dependency cache that was ninety megabytes larger than the passing ones.
Our cache key was the job name and the OS. That is it. No lockfile hash, no branch, no dependency manifest. So every job on every branch read and wrote the same cache entry. Someone working on a spike had added a heavyweight library, run their branch, and their post-job step had saved a node_modules tree containing it. Every subsequent build on every branch restored that tree. Where a transitive version differed from what the lockfile expected, the install step saw a populated directory, decided there was nothing to do, and the tests ran against a dependency graph nobody had asked for.
This is worse than flakiness. A shared writable cache means any branch can change the inputs of a build on main, which makes it a supply chain path with no review on it. We were not attacked. We were self-inflicted. But the mechanism is identical either way, and someone with commit access to a feature branch could have written whatever they liked into that directory.
The fix was three lines of YAML and a change of habit. The key now includes a hash of the lockfile, so a different dependency set is a different cache. Restore keys fall back only within the same branch and then to the default branch, never sideways between feature branches. The default branch writes cache; pull request builds restore it and do not write it. And we cache the package manager's download store rather than the installed tree, so the install step still runs and still reconciles against the lockfile.
Build times went up by about forty seconds. Nobody has noticed, because nobody is rerunning failed jobs any more.
If any branch can write your cache, your cache is an unreviewed input to production builds.
– Sergey Shinder
Top comments (0)