Partners that book collections with us authenticate with OAuth client credentials. Our authorisation server issues a signed JWT, and to save every API call a database lookup, the token carries a claim listing every depot the partner may book into. For most partners that is a handful. The token travels in the Authorization header, through ingress-nginx, to the booking API.
nginx reads request headers into buffers sized by large_client_header_buffers, and ingress-nginx sets it to four buffers of eight kilobytes, the same as nginx itself. A single header longer than one buffer cannot be read. nginx answers with 400 and a small HTML page that says Request Header Or Cookie Too Large, and the request never reaches the application.
In October our largest logistics partner added a regional network to their account, taking them from three hundred and ten depots to five hundred and forty. After the RS256 signature, the other claims and the base64 encoding of the payload, there is room for about four hundred and seventy depot codes before the header passes eight kilobytes. Their next token was a little over nine.
Every call they made from then on failed with 400. Our API dashboards showed nothing, because the API never saw a request. The ingress logs counted them as client errors, which we did not alert on, since a bad request is normally the caller's problem. The partner saw 400 and drew the same conclusion, and spent two days checking their own payloads before they phoned us. About eleven hundred collections in that time were booked by phone instead.
We raised the buffers to sixteen kilobytes that afternoon, which bought room and fixed nothing. The token now carries a single access group, and the booking API resolves the depots from it through a cache. The authorisation server refuses to issue a token larger than four kilobytes and alerts when one passes three, so the next growing claim stops at our end, with our name on it. Ingress error rates are graphed per client ID, and a client whose responses go from almost all successful to almost all failed pages us, whatever the status code.
A 400 is meant to say the caller sent something wrong. This one meant we had issued them something we could not read.
– Sergey Shinder
Top comments (0)