A rounding change in our shared money library went to production on a Thursday, and none of the twelve services that use it had been tested against it. The pipeline was green. Every job it ran passed, and it ran four jobs out of the forty in that repository.
Our monorepo builds by path filter. Each service declares the directories it cares about, the pipeline compares the changed files in a pull request against those globs, and skips everything that does not match. It is the reason a checkout change takes eleven minutes instead of fifty, and for two years it has been the single most popular thing about the build.
The library lives in libs/money. Four services list that path. The other eight reach it through an internal SDK that wraps it, and nothing in their filters mentions money, because nobody writing a filter thinks about the dependencies of their dependencies. Those filters were written by hand, once, by whoever set each service up, and they describe an import graph as it looked on the day they were typed.
So the change was reviewed by two people who knew the library well, tested against the four services that named the path, merged and released. The service that broke computes tax on invoices, and it broke in a way that was arithmetically small and commercially loud.
What I had wrong was thinking of a path filter as a performance feature. It is a statement about what depends on what, kept in a different file from the thing that actually says what depends on what, written in a different language, updated by a different person at a different time, and never checked against its subject.
We generate them now. The build asks the dependency graph which targets are affected by a set of changed files, and the filters come out of that rather than out of a YAML file somebody maintains by memory. A skipped job prints the reason and the paths it compared against, so a skip is visible rather than absent. And anything under libs is marked as widely used, which means the full suite, all forty jobs, fifty minutes, perhaps twice a month.
A green pipeline tells you what ran. It has never once told us what it decided not to run.
– Sergey Shinder
Top comments (0)