nginx uses limit_req to cap the number of requests per second for each IP address. Requests over the limit get a 429 (Too Many Requests) through limit_req_status 429.
At some point, 504s (Gateway Timeout) start to increase. A 504 is the status nginx returns when it forwards a request to the application server behind it (an upstream, in nginx terms; called the upstream below), waits for the time set in proxy_read_timeout without receiving a response, and gives up. In the access log, $request_time (the time from when nginx receives the request until it finishes sending the response) on every 504 line equals the proxy_read_timeout value. Each time nginx gives up, error.log records the following string.
upstream timed out (110: Connection timed out) while reading response header from upstream
There are zero 429s, and the number of requests per IP address is the same as usual.
The upstream is not down, and its access log keeps flowing.
I reproduced this state and measured the conditions under which the upstream falls behind while traffic stays within the limit_req limit, how to tell the causes apart, and a remedy that needs only nginx configuration.
Impact
This article covers upstreams built this way: the upstream has a cap on how many requests it can process at the same time, and connections that arrive beyond that cap wait in an OS queue. Even after nginx closes the connection, the upstream finishes processing the request. Handlers that detect the disconnect and abort processing are out of scope.
With this design, if the upstream cannot keep up with even a fraction of the incoming requests and that continues, no response succeeds. Every user waits until the timeout and then receives a 504.
The upstream keeps processing at full capacity, and that capacity goes into building responses nobody will receive. Almost every request it finishes is one for which nginx has already returned a 504.
When 504s increase, nothing is recorded on the rate limiting side. 429s stay at zero, which leaves nothing to base a decision on: whether to lower the limit_req limit, and if so, by how much.
What limit_req looks at
limit_req records one excess value per key value. The key is the value given to limit_req_zone; here it is the IP address. Two settings determine the limit. rate is the average speed allowed for requests from one IP address. Requests that arrive faster than rate are recorded as excess (how many requests' worth above rate have arrived), and the excess decreases over time at the speed of rate. burst is how much excess is allowed. When the excess exceeds burst, that request gets a 429.1
Heavy and light requests alike, however many seconds they take on the upstream, each add the same one request's worth to the excess.
rate and burst are fixed when the configuration is loaded and do not change while nginx is running. Whether to return a 429 is decided before the request is forwarded to the upstream. When upstream responses slow down, or the number of requests in progress on the upstream grows, limit_req does not see the change.
In a zone keyed by IP address, rate and burst apply to requests from one IP address. If the number of IP addresses sending requests triples while requests per IP address stay the same, every IP address stays within the limit and the requests reaching the upstream triple.
Conditions under which the upstream falls behind within the limit
Setup
I ran nginx 1.30.5 and the upstream.
The upstream is a small HTTP server written in Go, built the way described in the Impact section. It processes up to 20 requests at a time, and connections beyond that wait in the queue where the OS holds connections until it hands them to the server (the accept queue). Processing takes 50ms per request on the light endpoint and 300ms on the heavy endpoint.
The upstream closes the connection after each response. In nginx 1.30.5, keepalive to the upstream is enabled by default, but because the upstream closes the connection, connections are not reused and one request is one connection.2 The handler processes each request to the end without watching for nginx disconnecting. In Go's net/http, the request context is canceled when the client disconnects, and this upstream does not check it.3
Load came from the load testing tool k6: 200 requests per second at even intervals, all to the light endpoint. I prepared 10 key values and assigned 20 requests per second to each.
limit_req was set to rate=50r/s and burst=20 nodelay. nodelay forwards the excess requests to the upstream immediately instead of holding them back, as long as the excess stays within burst.4 Each key value receives 20 requests per second, fewer than the 50 per second of rate, so every key value stays within the limit.
Because load came from a single machine, I used a request header value as the key in place of the IP address. limit_req treats the key as a string, so the behavior is the same whether the key is an IP address or a header value.
proxy_read_timeout was shortened to 1 second to keep the test short. The mechanism is the same with the default of 60 seconds.
At this starting point, about 10 requests are in progress on the upstream at a time (200 per second × 0.05 seconds), and the upstream can handle 400 per second (20 ÷ 0.05 seconds).
Load ran for 25 seconds, and one item was changed at the 8-second mark. I also ran a pass with nothing changed as the baseline.5
Results
In every condition, 429s stayed at zero, and the success rate was 0% from 6 seconds after the change onward. This is the same state as the symptom described at the start.
| Condition | Changed at 8 seconds | 429 | Success rate |
|---|---|---|---|
| Baseline | Nothing | 0 | 100% |
| More heavy requests | Share of heavy requests 0% → 40% | 0 | 0% |
| Longer processing time | Processing time of light requests 50ms → 120ms | 0 | 0% |
| Fewer concurrent slots | Requests the upstream can process at the same time 20 → 8 | 0 | 0% |
| More key values | Number of key values 10 → 30 (600 per second in total) | 0 | 0% |
The success rate is the fraction of requests k6 sent between 14 and 25 seconds (from 6 seconds after the change) that got a 200.
Reading the conditions
More heavy requests corresponds to heavy requests mixing in while the number of incoming requests stays the same. The number of requests the upstream can handle drops, and limit_req adds one request's worth to the excess regardless of weight.
Longer processing time corresponds to the upstream, or the DB or external API behind it, slowing down. The upstream can no longer keep up with the same number of requests, and limit_req does not see the change.
Fewer concurrent slots corresponds to a reduction in upstream nodes or workers. The number of requests the upstream can handle drops, while rate and burst stay at their configured values.
More key values corresponds to more IP addresses sending requests. Every key value stays within the limit while the total number of requests reaching the upstream exceeds its capacity.
Why no 200s come back
Mechanism
In each of the four conditions in the previous section, after the change, the number of requests the upstream can handle per second falls below the number arriving. The upstream processes at most 20 at a time, so the connections it cannot handle line up in the accept queue. While the amount reaching the upstream exceeds what it can handle, the number of connections in the accept queue keeps growing, and requests that arrive later wait longer.
sequenceDiagram
participant C as Client
participant N as nginx
participant Q as accept queue
participant U as upstream
C->>N: Request
N->>Q: Connects and sends the request
Note over Q,U: Waits while the upstream processes<br/>the connections ahead of this one, in order
Note over N: proxy_read_timeout elapses
N->>C: 504
N->>Q: Closes the connection
Q->>U: Hands over the connection once it reaches the head
Note over U: Processes it without noticing<br/>the connection was closed
U->>N: Response nobody receivesWhen the time waited in the accept queue plus the processing time on the upstream exceeds proxy_read_timeout, nginx gives up on that request, returns a 504, and closes the connection to the upstream. This is the symptom seen at the start, where $request_time on 504 lines equals the proxy_read_timeout value.
The upstream does not notice that nginx closed the connection. When the connection reaches the head of the accept queue, the upstream accepts it, processes it, and writes a response nobody receives. This is why records of finished processing keep appearing in the upstream access log.
The accept queue is processed from the front and keeps growing in the meantime, so everything that reaches the head is a request nginx has already given up on. All of the upstream's capacity goes to these, and by the time a request that is still waiting gets its turn, it too is past proxy_read_timeout.
In the logs
For the Longer processing time condition (light request processing time 50ms → 120ms, changed at 8 seconds), I counted the requests in each second from the start of the load, using the upstream access log and the nginx access log.
| Second | Requests the upstream finished | 200s returned by nginx | 504s returned by nginx |
|---|---|---|---|
| 9 | 167 | 167 | 0 |
| 10 | 167 | 167 | 0 |
| 11 | 162 | 162 | 0 |
| 12 | 166 | 166 | 0 |
| 13 | 168 | 52 | 139 |
| 14 | 164 | 0 | 200 |
| 15 | 165 | 0 | 200 |
| 16 | 167 | 0 | 200 |
Through second 12, the number of requests the upstream finished matches the number of 200s nginx returned. The difference from the 200 per second arriving accumulates in the accept queue.
Requests that arrive after 12.3 seconds spend more than the 1-second proxy_read_timeout on waiting plus processing, so 504s start at 13.3 seconds.
From second 14 on, the upstream keeps processing at the same speed, yet only 504s come back. nginx returns a 504 one second after a request arrives, so the count equals the 200 per second arriving.
Checking your own logs
When 504s increase and 429s are zero, first check whether the wait in the accept queue is growing. If it is, the upstream has fallen behind within the limit_req limit. Then identify which cause applies.
Check whether the accept queue wait is growing
The accept queue wait shows up as the difference between the upstream response time as seen by nginx and the processing time in the upstream access log. If this difference grows second by second, waits are building up in the accept queue. The difference starts growing before the 504s appear. On 504 lines $upstream_header_time is -, so look at the period before the 504s.
The upstream response time as seen by nginx is $upstream_header_time, which can be recorded in the access log. It is the time from when nginx starts connecting to the upstream until it receives the response header.6 It includes both the accept queue wait and the processing time on the upstream. The processing time in the upstream access log is the time from when the upstream accepts the connection until it returns the response, and does not include the accept queue wait.
To get the difference, take the per-second, per-endpoint median of $upstream_header_time and of the upstream processing time, then subtract one from the other. Record the nginx access log in the following format.
log_format rl '$msec $status $uri $remote_addr $upstream_addr $upstream_header_time';
Identifying the cause
Check in this order.
- In the upstream access log, look at processing time per endpoint. If it has grown, the upstream or the DB or external API behind it has slowed down (Longer processing time). However, an increase in heavy requests can put pressure on shared resources such as the DB and make even light endpoints slower. Even if processing time has grown, still check the share in the next step. In the test, the heavy processing is a sleep that does not affect other requests, so this interference was not verified.
- If it has not grown, look at each endpoint's (
$uri) share of the total count in the nginx access log. If the share of endpoints with long processing time has increased, heavy requests have increased (More heavy requests). - If the shares have not changed, look at the total requests per second in the nginx access log. If it has increased, more requests are arriving. Break it down by how many distinct key values there are (the number of distinct
$remote_addrvalues) and by requests per second per key value. If the number of distinct key values has increased, more IP addresses are sending requests (More key values). The case where requests per key value increase withinratewas not tested, but the total reaching the upstream increases in the same way. - If none of these apply, look at the number of upstream nodes or workers. If it is lower than usual, the number of requests that can be processed at the same time has dropped (Fewer concurrent slots).
Remedies
First stop the damage with max_conns, then fix the cause.
Cap concurrent requests with max_conns
Put max_conns on the upstream server line. Set it to the number of requests the upstream can process at the same time. zone is needed to share the concurrent connection count across nginx workers.7 If you run several nginx instances side by side, divide this value by the number of instances to get the per-instance value.
upstream app {
zone app 64k;
server app:8081 max_conns=20;
}
Connections nginx sends to the upstream at the same time no longer exceed max_conns. Requests over that are not sent to the upstream, and nginx returns a 502 (Bad Gateway) on the spot. nginx OSS has no mechanism to make them wait until a slot frees up. The upstream processes only requests nginx is still waiting on, and the accept queue does not grow.
In this test the upstream closes the connection, so keepalive connections are not reused. The nginx documentation states that with keepalive to the upstream, multiple workers, and zone, the total of active and idle connections can exceed max_conns.8 From nginx 1.29.7 on, keepalive is enabled by default, so if the upstream keeps connections open, you get this setup even without a keepalive directive. If the upstream limits concurrent processing by the number of connections, idle connections also take up slots. The case where keepalive connections are reused was not tested.
For each of the four conditions, I compared no remedy against max_conns=20. p99 is the value at the 99% position when the response times of successful responses are sorted from shortest to longest. p99 in the run with nothing changed was 51.6ms.
| Condition | Success rate (no remedy) | Success rate (max_conns=20) | p99 of successful responses (max_conns=20) |
|---|---|---|---|
| More heavy requests | 0% | 66% | 301ms |
| Longer processing time | 0% | 80% | 122ms |
| Fewer concurrent slots | 0% | 79% | 132ms |
| More key values | 0% | 64% | 52ms |
There were zero 504s in every condition. The success rate is roughly equal to what the upstream can handle divided by what arrives. Successful responses come back without waiting, and p99 matches the upstream processing time. In Fewer concurrent slots it was 132ms because the amount by which max_conns=20 exceeds the upstream's 8 (up to 12 requests) lines up in the accept queue, and that wait is added.
Even if the max_conns value is larger than the number of requests the upstream can actually process at the same time, the accept queue does not keep growing. In the Fewer concurrent slots run, max_conns stayed at 20 while the upstream's concurrent capacity dropped to 8. There were zero 504s, and the accept queue stopped at a maximum of 12. A connection is established when the OS accepts it, so the connections nginx counts include those waiting in the accept queue. The connections in the accept queue never exceed max_conns minus the upstream's concurrent capacity.
Users beyond what can be handled receive a 502 without waiting. You cannot choose which requests get the 502. This 502 is recorded in error.log with the following string.
no live upstreams while connecting to upstream
In the access log, $upstream_addr, which normally holds the upstream's IP address and port, becomes the upstream name (app here).9 502s returned without sending the request to the upstream can be counted by this value. The error.log message is the same as for a 502 when all upstream servers are down, but with a 502 from max_conns, 200s keep coming back in the same time window.
Options ruled out
Shortening proxy_read_timeout is ruled out. Changing it from 1 second to 500ms still gave a 0% success rate. nginx gives up sooner, and the accept queue grows at the same speed.
Stacking a second limit_req to cap the total of requests from all IP addresses is also ruled out. A zone keyed by a fixed string (limit_req_zone all zone=fixed:1m rate=160r/s;) is stacked on top of the IP address zone. rate=160r/s is set to match the 167 per second the upstream can handle under Longer processing time (20 ÷ 0.12 seconds). With this setting, the success rate under Longer processing time was 80%, and the accept queue did not grow. However, about 19% got a 429 even in normal operation, when the upstream had spare capacity. rate has to be fixed to match the number of requests per second the upstream can handle, and that number changes with processing time. max_conns caps the number of requests in progress at the same time, so when processing time grows, it automatically lets fewer requests through, and in normal operation it turns nothing into a 502.
Fixing the cause
max_conns only stops the damage. What the upstream can handle does not recover. Where to act for each cause:
- For Longer processing time, fix the DB or external API that slowed down
- For More heavy requests, move heavy processing to a separate upstream or to asynchronous processing
- For Fewer concurrent slots, restore the lost nodes or workers
- For More key values, add upstream capacity to match the increase
Summary
What limit_req caps is the number of requests arriving per IP address. What the upstream can handle is determined by processing time and the number of requests it can process at the same time, and changes in these do not reach limit_req.
What cannot be handled accumulates in the accept queue and turns into 504s once waiting plus processing exceeds proxy_read_timeout. The upstream keeps processing requests nginx has already given up on, so no response succeeds.
Protect the upstream with a cap on how many requests nginx sends to it at the same time (max_conns). Regardless of cause, only what cannot be handled becomes a 502, and the rest comes back at the usual speed.
This applies to upstreams that make connections arriving beyond their concurrent capacity wait in the accept queue, and whose handlers do not abort processing when nginx closes the connection.
-
In the nginx 1.30.5 source,
ngx_http_limit_req_lookup()makes this decision. ↩ -
Keepalive to the upstream has been enabled by default since nginx 1.29.7 (CHANGES; nginx documentation for
keepalive). ↩ -
Go documentation for
Request.Context. ↩ -
I ran it on a GitHub Actions runner (ubuntu-24.04). The results are here, and the code is in show-your-work. ↩
-
The nginx documentation for
$upstream_header_timedoes not say where the measurement starts. In the 1.30.5 source, it is computed fromu->start_time, which is recorded when nginx starts connecting to the upstream. ↩ -
nginx documentation for
max_conns. Withoutzone, the limit applies per worker. ↩ -
nginx documentation for
$upstream_addr. When no server can be selected, it holds the upstream name. ↩
Top comments (0)