We promoted a release at ten past three on a Wednesday, after forty minutes of canary analysis with nothing to report, and watched our partner integrations start failing about ninety seconds later. Errors on that path went from none to a fifth of all calls. The canary had been serving the same code to real traffic and had been clean the whole time.
The canary takes five percent of traffic. What I had never asked is five percent of what, chosen how. Our edge routes by a consistent hash of the session cookie, which is the right thing for a browser, because a customer then stays on one side of the release for a whole visit instead of seeing a half migrated interface between two clicks. A request with no cookie hashes a constant instead, and a constant is one value, which lands on one side, which for the past two years has been the baseline.
Everything our partners send is server to server with no cookie. The canary had therefore never seen a partner request in its life, and the change we were shipping added a required header to exactly that path.
The sample was thin in a second way. We deploy in the early afternoon because it is quiet, and five percent of quiet is nine hundred requests across forty minutes spread over thirty endpoints. Several endpoints saw single figures. A clean error rate over a handful of requests is not evidence of anything, and our promotion gate read it as a pass because the number it compared was a total for the whole service.
Canary routing for endpoints with no session is per request now. The gate compares per endpoint class instead of in aggregate, and it will not promote until every class present in the baseline has been seen by the canary in enough volume to have shown the failure rate we care about. If that volume does not turn up, the canary waits, or we send traffic at it deliberately.
A canary is a sample, and a sample selected by a mechanism nobody has read is not a smaller copy of your traffic. It is a different population that happens to be smaller.
– Sergey Shinder
Top comments (0)