In June our customer service team noticed that disputes about missing parcels had doubled, and that in most of them we could not show the photo the driver takes at the door. Drivers were taking the photos. They were simply not arriving.
Our driver app uploads each proof of delivery photo to the delivery API, a JPEG of two to four megabytes straight from the phone camera. For three years the API sat behind a cloud load balancer that forwarded requests to our pods. In the first week of June we moved our public services onto ingress-nginx, to share one entry point across the cluster.
ingress-nginx limits the size of a request body, and the default is one megabyte, the same as nginx itself. A larger body is refused by the controller with 413 Request Entity Too Large. The setting is the proxy-body-size annotation on the Ingress, or a global value in the controller's ConfigMap, and the migration had copied neither, because the old load balancer had no such limit and nobody had a reason to write one down. Text requests were all far below a megabyte. Only the photos went over, and nearly all of them did.
The 413 came from nginx, so the request never reached our service. The delivery API's error rate was flat and its upload count dropped, which looked like a quiet week. The driver app kept failed uploads in a queue and retried them, and after five attempts it dropped them, with a line in a log on the phone.
We set proxy-body-size to fifteen megabytes on that one Ingress, not globally, and retrieved most of the photos still queued on phones. The app now resizes photos to about five hundred kilobytes before uploading, which is plenty to show a parcel on a doorstep. The controller's request metrics, labelled by status, feed an alert on any 413 rate above zero for a public host. The driver app reports a dropped upload to the API as an event of its own. And our migration checklist for any new proxy now lists its defaults for body size, timeouts and header size next to the old ones, before any traffic moves.
A proxy you put in front of a service brings its own idea of a reasonable request. Ours thought a photo was too big, and told only the phone.
– Sergey Shinder
Top comments (0)