DEV Community

Cover image for We Debugged "Random" 502 Errors for 2 Days. The Answer Was in One HTTP Header Nobody Looked At
Kamen
Kamen

Posted on Edited on Originally published at kamenivanov.substack.com

We Debugged "Random" 502 Errors for 2 Days. The Answer Was in One HTTP Header Nobody Looked At

The 502s hit about one login in fifty, and they looked completely random, which is the worst possible bug. Our application logs showed a clean 200 for every one of those requests. Our monitoring showed nothing unusual in CPU, garbage collection or database latency. The only evidence that anything had gone wrong was a user telling us they had been sent to an error page.

For three days we chased the usual suspects, because the usual suspects are usually right. We checked connection pool exhaustion, a long garbage collection pause, a node being recycled during a deployment, and the keep-alive timeout between the proxy and the application. Each one was a reasonable theory and each one fell apart against the same fact: the application believed it had answered successfully. A server that logs a 200 and a client that receives a 502 are not describing the same request, so the failure had to be happening somewhere between them.

The break came from the proxy's own error log, which we had not opened until then because the application logs had been so convincing. It contained one line, repeated at the same rate as the complaints: the upstream had sent too big a header while the proxy was reading the response header.

Header Buffer

The header in question was a custom X-User-Permissions that the login response used to hand the single-page frontend a list of the user's permissions, so it could hide the buttons they could not use. The header grew with the number of roles a user had, and the proxy buffers response headers in a fixed-size area, 8 KB in our configuration. A user with five roles produced a header of about 3 KB. A user with sixty produced one of about 12 KB. The proxy could not fit it, gave up on the upstream response and returned a 502 to the client, while the application, which had finished its work correctly, logged a 200 and moved on.

The pattern had been there all along, hidden by how we counted. The failures were not random across requests. They were concentrated in logins by long-tenured administrators with many roles. Plotted against role count instead of time, the errors began at around 35 roles, exactly where the header crossed the buffer.

Header size by role count

Raising the proxy buffer made the errors disappear within minutes, but that only moved the cliff further out. The real fix was to stop sending data that grows with the user in a response header. The permissions now travel in the response body, which is not subject to the small header buffers on every hop.

When a client sees an error that the server never recorded, the interesting component is the one in between, and the evidence is usually in the headers it was handed. We spent three days looking inside a healthy application to explain a failure that happened outside it.


More like this?
I write short production war stories like this one, plus a deep-dive architecture series. Subscribe here if you want both in your inbox.


Recommended Resources for Java & Spring Boot Engineers

If you are preparing for Senior/Lead Java interviews or looking to solidify your Spring Boot & Architecture skills, check out these highly-rated resources from the Javarevisited publication (Use promo code friends20 for an exclusive 20% discount automatically applied at checkout):

Top comments (0)