The pricing service panicked in production on a Thursday evening under the heaviest traffic of the week. The stack trace said concurrent map writes. That test suite had run roughly eleven thousand times over two years and had never once failed on it.
I copied the offending test onto a four core machine and ran it with the race detector enabled. It failed in six seconds. Then I looked at what our CI actually gave us: a container with one vCPU. A Go program on one core runs goroutines on one operating system thread and switches between them at a small number of well defined points, so the interleaving that corrupts a shared map is not impossible there, it is just rare enough that eleven thousand runs was not enough. The race detector, which would have caught it on any number of cores, had been in the pipeline once. It was removed in 2023 with a commit message saying the suite took too long, which was true.
So we had two safety nets and had quietly folded both of them up: the hardware could not produce the fault, and the tool that finds the fault without producing it was gone.
Test jobs now run on four vCPU runners, which cost us about forty pounds a month. The race detector runs on every pull request for the six packages that own shared state, and on the whole suite nightly with a randomised test order. The nightly run found two more races in its first week, one of them in a cache we had been trusting since 2021.
The broader exercise was more useful than the fix. We wrote down every way the CI environment differs from production: one core against four, sixteen gigabytes of memory against a five hundred and twelve megabyte pod limit, plaintext HTTP against TLS, a single instance against nine, a local filesystem against a network volume. Nine differences. We closed three, decided two were fine, and for the remaining four we now know which category of bug our pipeline structurally cannot see.
CI is not a smaller production. It is a differently shaped machine, and every difference in shape is a class of defect that will reach a customer before it reaches a test.
– Sergey Shinder
Top comments (0)