On a Tuesday our carrier partner suspended our event subscription. Around fourteen thousand shipment updates over nine hours were never sent to us, and we learned about it from a customer asking why their tracking page had not moved since breakfast.
Their delivery platform grades the systems it delivers to. The rule is written in their documentation, in a section I had read four years ago as a paragraph about them: a receiver that takes longer than five seconds to answer is recorded as a failed delivery, and a subscription with more than a hundred consecutive failures is suspended until its owner switches it back on in their portal. All of that is sensible, and none of us had ever considered that our end of an integration carried a service level, measured by another company, against a number nobody here had looked up.
Our handler did everything inside the request. Check the event, load the shipment, update three rows, write an audit record. Two hundred milliseconds on an ordinary morning. That morning a long reporting query held a lock on one of those tables and our handler went to eleven seconds a call. We answered every single one of them successfully, just slowly, so our own instruments recorded nothing wrong at all. Zero errors. We were not failing, we were late, and late is a failure in somebody else's system rather than in ours.
The handler now does one thing. It confirms the event is a shape we expect, writes it to a durable queue and answers. Twelve milliseconds. The real work happens behind that, where it can take as long as it needs to without anyone else forming an opinion about us.
Two other things changed. Our receiver is monitored as a customer facing service with its own latency objective, because to them that is exactly what it is. And every hour we ask their API how many events they sent us in the previous hour and compare it with how many we stored, which would have told us in sixty minutes what a customer told us in nine hours.
Every integration makes you a supplier as well as a consumer. Somewhere in the other company's documentation is the standard you are held to and the penalty for missing it, and it is written as though it were about them.
– Sergey Shinder
Top comments (0)