For about a month our calls to a parcel carrier's API failed in bursts. Label requests came back with UNAVAILABLE a few dozen at a time, and the retries succeeded. When I lined the failures up against our own change log, every burst started within a minute of one of our deploys and ended about twenty minutes later.
The carrier's API is gRPC. Each of our pods holds one long lived connection to it, carrying label requests and a stream that pushes tracking events to us. Our pods reach the internet through a NAT gateway, and the gateway drops a connection after 350 seconds without traffic. Overnight, when no tracking events came for a while, the stream died that way, and neither side knew. So in the spring we had turned on client keepalive with a thirty second interval, and the overnight gaps went away.
gRPC servers police keepalive as well. The carrier's server used the library default, which permits a client ping at most once every five minutes and treats a client that pings more often as misbehaving. After a couple of early pings it sends GOAWAY with the code ENHANCE_YOUR_CALM and the debug message too_many_pings, and closes the connection, taking any calls in flight with it. Our client logged a warning nobody read and did what the client library is built to do: it doubled its keepalive interval for that channel and reconnected. Thirty seconds, one minute, two, four, eight. Once the interval passed five minutes the server stopped objecting, and everything was quiet until our next deploy started every channel at thirty seconds again. Four rounds of that took a little over twenty minutes.
The carrier agreed to lower their minimum to one minute on the endpoint their partners use, and we now ping every two minutes, comfortably inside both their limit and the gateway's. A GOAWAY from any partner is now counted as an error with its code as a label, and too_many_pings has an alert of its own, because it means our settings and theirs disagree and the next deploy will prove it. And our integration test environment includes a gRPC server with the same enforcement policy the carrier runs.
We had tuned keepalive for the network between us and forgotten the server at the far end of it. A ping is a request too, and the other side decides how often it will take one.
– Sergey Shinder
Top comments (0)