At 04:12 on a Sunday our settlement integration started failing every call with a TLS handshake error. Our code hadn't changed in six weeks. Their API was up, their status page was green, and our error said only that the peer certificate could not be verified.
The partner had rotated their server certificate to a new issuing CA. Perfectly normal operational hygiene on their side. On our side, someone years earlier had configured the client with an explicit trust store containing only that partner's old intermediate, rather than the system CA bundle, on the reasonable-sounding theory that pinning narrows the attack surface. It does. It also means the other party's routine maintenance becomes your outage, and they have no idea you did it, because nothing in the protocol tells them.
We fixed it in forty minutes by adding the new intermediate, which is fast only because the on-call engineer had seen this exact class of failure before. Without that, the error message points nowhere useful. "Unable to verify" reads like a network problem, and the first twenty minutes of most such incidents get spent on proxies and DNS.
What we changed afterwards was mostly about visibility. A daily job connects to every outbound integration endpoint, reads the presented certificate chain, and alerts at thirty, fourteen and seven days before expiry, on both their certificate and ours. That job also records the issuer, so a change of CA raises a warning the day it happens rather than at the next handshake failure. Where pinning is genuinely required by a contract, we now pin to the public key with a second, pre-provisioned backup pin, which is the whole point of pinning correctly and the part everyone skips.
We also stopped assuming client certificate expiry would announce itself. Our own mTLS cert for a different partner had eleven days left on it and no owner listed anywhere.
The broader lesson is that an integration's dependencies include the other party's operational calendar, and you have no visibility into it. Monitor the things they can change without asking you, because those are the ones that will wake you up.
– Sergey Shinder
Top comments (0)