Our customer portal has a synthetic check that runs every two minutes. It loads the account page as a test customer and passes if the page returns 200 and contains the text Customer Portal. In September the account page was broken for nine days for a fifth of our customers, and the check was green for every one of them.
The broken part was ours: a deploy had changed how the page loaded saved addresses, and customers with more than one address got an error panel. The test customer had one address. That part is ordinary. The check would not have caught it anyway, and the reason was more interesting.
Five weeks earlier we had moved the portal to a new single sign on provider. The test customer's session cookie, which the check stored and reused, stopped being accepted. Requests for the account page now returned a redirect to the login page. The check's HTTP client followed redirects by default, the login page returned 200, and its title, in the header shared across every page of the portal, contained Customer Portal. The check had stopped testing the account page and was testing that the login page existed, and it had been passing that test perfectly for five weeks before the bug ever arrived.
Nobody noticed because nothing about the check changed. Its success rate and response times were steady. If anything it had become faster.
The check no longer follows redirects. A 3xx is a failure with its own label, so an expired login looks different from an outage. It signs in fresh on every run with an account whose password lives in our secret store, and a failure to sign in pages separately. It asserts on things only a signed in page can contain: the test customer's account number and the address list element. The test customer now has three addresses, a cancelled order and a saved card that has expired, because a test user with the simplest possible account exercises the simplest possible code. And every synthetic check now writes the final URL it reached to the log, which would have shown the login page on the first day.
A check is only as good as the evidence it asks for. We asked for a word that appeared on every page we own, including the one that tells you that you have not got in.
– Sergey Shinder
Top comments (0)