The first report arrived by phone: the application loads a blank page. By the time I had opened our dashboards I was ready to tell the customer it was his browser. Error rate zero. Latency normal. Throughput about two percent under the same time last week, which is well inside the noise we live in. Nothing was wrong anywhere in our monitoring, and roughly one user in six could not use the product.
Our web deploy uploads hashed asset bundles to object storage, invalidates the CDN, then publishes the new HTML. Somebody had made the upload and the invalidation run in parallel to save forty seconds. That morning the invalidation finished first. Edge nodes went looking for bundles that were not there yet, cached the 404s, and served them to everyone whose HTML referenced the new filenames.
The deploy bug took ten minutes to fix and is not the interesting part. The interesting part is that the whole incident happened in a region of the system we had no instruments in. Those failing requests were answered by the CDN out of its own cache. They never reached our origin, so they were never counted, never logged, never sampled into a trace. Our observability began at the load balancer. The user's experience began three hops earlier.
Three additions. A synthetic check runs every minute from three regions, loads the real page in a real browser, and asserts that a specific element renders; two consecutive failures page. Client side errors, including resource load failures, post to a collector and are graphed per release, which turned up two smaller breakages in the first fortnight. And CDN logs are now shipped and reduced to a handful of series, so edge status codes and origin request volume sit on the same dashboard as everything else.
The one that would have caught this fastest is also the cheapest. Nothing we owned was checking whether the front page worked.
Our monitoring was honest about everything it could see. The question nobody had asked was where our instruments stopped.
– Sergey Shinder
Top comments (0)