The 502 That Wasn't the Server

We spent over a year blaming the wrong things.

A 502 Bad Gateway kept hitting our login route — about one login in twenty, no pattern we could find. A 502 usually means the upstream is unhappy, so that's where we looked: restart counts, CPU, memory. All clean. The app was healthy the whole time. The failing requests never even reached it.

The symptom

It only happened on /api/login. It was intermittent — retry and it usually worked. The first ticket was opened in September 2024, and the error kept resurfacing in support channels for over a year: different customers, different browsers, different operating systems. Nobody could reproduce it on demand.

For most users it was a one-off. But one longtime customer hit it almost every single time. She contacted support again and again — calls, emails — for months. Support kept escalating, and every escalation died the same way: "cannot reproduce."

Two more details sat in those support threads, and in hindsight they were the whole answer. First: a private/incognito window fixed it for her, every time. Second: clearing cache didn't. We read both and moved on.

The wrong theories

The first guess was the obvious one: the upstream was flaking, nginx couldn't get a clean response, 502. Fair guess — it's the most common cause. But the upstream was healthy.

The second theory had more conviction behind it: a token-refresh bug. The symptoms rhymed — authentication route, intermittent, session-related. A fix was built, tested, and shipped. The customer hit the 502 again the same week. Whatever it was, it wasn't the refresh flow.

That failed fix bought us the insight that cracked the case: when we matched the failing requests against the app logs, they weren't there. The app never saw them. The requests were dying before the app, not because of it.

That's the turn. A 502 doesn't only mean "the backend errored." It also means "the proxy couldn't read the response back" — which can happen with a perfectly healthy backend if the response is too big for the proxy to hold.

The actual cause

Login used the OAuth authorization-code flow. Buried in the code that built the authorization request was one decision that aged badly: it concatenated the user's entire session cookie into the OAuth state object. The longer you'd been logged in, the bigger your cookie — and every byte of it rode through the redirect and came back inside a response header (a Location redirect plus its cookie).

nginx buffers upstream response headers into a fixed size (proxy_buffer_size). If they don't fit, it doesn't truncate — it drops the response and returns a 502, logging:

upstream sent too big header while reading response header from upstream

So: cookies grew with session age, the state grew with the cookie, the header occasionally crossed the buffer size, and nginx threw out a good response.

And suddenly every weird detail made sense:

• The randomness. Only oversized states broke; leaner ones slid under the limit. Same code, different cookie sizes, different outcomes.
• The one customer who always failed. Her long-lived session had accumulated the largest cookie — which meant the largest state, every time. It wasn't bad luck; her account was deterministically over the line.
• The incognito fix. A private window starts with no cookies: minimal state, lean header, fits the buffer. It was never "a browser thing." It was a size thing.
• Why cache-clearing didn't help. Clearing cache leaves cookies alone. The one thing support kept telling users to clear was the one thing that wasn't the problem.

The clues that looked like noise were the diagnosis. We just weren't reading them as data.

Worth noting: this was a 502, not a 400 or 414. Those are request-side (the client sending too much). Ours was response-side — a healthy upstream sending headers too big to read back. That's exactly why "the server's down" felt right and wasn't.

The check that would've saved a year

When a request fails but never shows up in your app logs, stop looking at the app. It's a thirty-second check and it cuts the search in half.

After that: read the proxy error log, not just the access log — "upstream sent too big header" points straight at buffer sizing. Then reproduce on purpose instead of retrying: build a login with a deliberately large state and the 502 shows up every time. Once you can toggle it with header size, you're done.

And treat support anecdotes as telemetry. "Incognito fixes it, cache-clearing doesn't" isn't a user quirk — it's a stack trace written in customer language. It told us the problem lived in session-accumulated data long before we knew where to look.

The fix

The tempting hotfix was raising proxy_buffer_size. We deliberately didn't. Larger header buffers mean the proxy will accept and hold bigger responses from anything upstream — expanding a limit designed to be a safety boundary, to accommodate a payload that had no business being that large. And it just moves the wall: if the state keeps growing, you hit it again at the new size, with the same year-long debugging story.

The real fix attacked the root cause: stop carrying the payload at all. We moved the session data to server-side storage (Redis) and the state went back to what the spec always intended — a small opaque reference. The redirect now carries a pointer, not the cargo, so no login can produce an unbounded header no matter how much a session has accumulated. The buffer stays where it is — the state just stopped testing it.

Proving it was actually fixed

A year of "cannot reproduce" earns you one obligation: when you finally ship the fix, you don't get to just believe it worked. Before the deploy, I set up an alert on the exact log signature — upstream sent too big header — so we had a live counter of the failure, before and after.

That did two things. It gave support an honest answer ("if it happens again, we'll see it before you do") instead of another "should be fixed now." And it turned the rollout into an experiment with a falsifiable result: either the counter drops to zero, or the root cause was wrong and we'd know within days — not after another year of tickets.

Takeaway

A 502 isn't always the server. Sometimes it's the proxy refusing a response it couldn't fit, from a backend that did nothing wrong. OAuth state grows quietly, redirects carry it in headers, and every proxy has a limit — and when that limit lands on your login route, you get a 502 that looks like the server's fault until you check whether the request ever reached the server at all.

Total cost: over a year of tickets nobody could reproduce, one shipped fix that fixed the wrong thing, and one very patient customer — for a root cause that a single line in the proxy error log had been stating plainly the entire time.

The fix shipped in July. The alert has been silent since.