A graph of requests per minute is useful, but it can hide a slow route or distributed crawling. Break traffic down by endpoint, response status, latency, and time window before deciding what the aggregate means.
Why per-IP averages are not enough
For a hypothetical example, 40 requests per minute spread across 400 addresses average 0.1 requests per address per minute. A per-IP threshold might miss that traffic. That arithmetic does not establish that the traffic is harmful or that any particular threshold will detect it.
Look at distinct paths visited, repeated paths, URL sequences, status patterns, and verified client identity where available. Crawlers can traverse many paths or concentrate on one expensive route; there is no single path-concentration metric that reliably separates them from people. An IP address or a user-agent string is not a trustworthy session identity.
Five questions worth asking
1. Which endpoints have slow tails?
Compute p50, p95, and p99 per route, along with request counts and a clear window. Small samples make high percentiles unstable. Compare traffic segments rather than assuming a user-agent filter identifies real users: bots can impersonate browsers, and legitimate automation can be important traffic. Access-log timing alone does not measure browser experience or prove why someone abandoned a page.
2. Where are the 4xx responses?
Requests for /.env or /wp-login.php may indicate generic probing on a site that does not serve those paths. Application-route failures can also reflect authentication, deployment, or client problems. Investigate context before blocking; neither path nor status proves a cause, and a sensitive file returning 200 deserves attention regardless of request rate.
3. How do route rates and path diversity change over time?
Measure requests per path and distinct paths per client/time window, as well as site-wide totals. Choose thresholds from observed baselines, endpoint cost, shared-IP behavior, and acceptable traffic. There is no universal rule that 30 requests per minute merits a block.
4. Which requests end with 499?
nginx's internal 499 status records a client closing the connection before nginx completes the response. The direct client may be a load balancer, not the person using the site. Cancellation, networking problems, and downstream timeouts can all contribute. A cluster of 499s is a useful signal, not proof of a slow application. Include it explicitly in dashboards: whether it appears in an error-rate metric depends on that metric's definition. A request time near a configured upstream timeout does not by itself establish a client cancellation.
5. How do total and upstream timings differ?
$request_time and $upstream_response_time measure different intervals. A gap can involve request-body upload, downstream transfer, buffering, or other processing; it does not prove the application is healthy. Upstream timing fields can be - or contain multiple values when multiple upstream attempts occur, so do not blindly subtract them as single numbers. See the upstream variable definitions.
Log structured data, not guessed field positions
The default combined format has no timing fields. For new analysis, JSON escaping avoids treating spaces inside user agents or requests as column separators:
# log_format belongs in the http context.
log_format detailed_json escape=json
'{"time":"$time_iso8601","remote_addr":"$remote_addr",'
'"host":"$host","method":"$request_method","uri":"$uri",'
'"status":"$status","bytes":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"upstream_response_time":"$upstream_response_time",'
'"user_agent":"$http_user_agent"}';
# Use this in the intended server/location (or http) context.
access_log /var/log/nginx/access-analysis.jsonl detailed_json;
The nginx log module documentation describes escape=json and request timing. All variable values above are quoted deliberately so missing values remain valid JSON strings. Validate configuration with nginx -t before reloading it through your normal deployment process.
$uri is normalized, excludes query arguments, and can change during internal redirects. It is useful for some grouping, but it is not necessarily the original URL or an application route template. /users/123 and /users/456 may need to map to /users/:id. Keep host and method when they distinguish different endpoints. Review sensitive-data exposure and retention before adding request parameters or identifiers.
Compute counts and percentiles locally
In a combined-style log, $11 is not reliably a URI: quoted referers and user agents can contain spaces, and the request itself spans whitespace fields. Scanning arbitrary fields for uri= can also match text inside those quoted values. Use a parser for the exact format, or parse the JSON format above by key.
This Python script accepts the JSON lines above, rejects malformed records, and computes a nearest-rank p95 for each host/method/URI bucket. It reports rejected-line counts instead of silently skipping them. It is an in-memory example: choose a bounded log window, and use a database or histogram-based aggregation for larger workloads.
import json
import math
from collections import defaultdict
samples = defaultdict(list)
rejected = 0
with open("access-analysis.jsonl", encoding="utf-8") as log:
for line in log:
try:
row = json.loads(line)
key = (row["host"], row["method"], row["uri"])
if not all(isinstance(part, str) for part in key):
raise ValueError("Invalid grouping key")
timing = float(row["request_time"])
if not math.isfinite(timing) or timing < 0:
raise ValueError("Invalid request_time")
except (ValueError, TypeError, KeyError):
rejected += 1
continue
samples[key].append(timing)
print("rejected lines:", rejected)
print("top endpoints by request count:")
for key, values in sorted(samples.items(), key=lambda item: len(item[1]), reverse=True)[:20]:
print(len(values), key)
results = []
for key, values in samples.items():
values.sort()
n = len(values)
p95 = values[math.ceil(0.95 * n) - 1]
results.append((p95, n, key))
print("endpoints by nearest-rank p95:")
for p95, n, key in sorted(results, reverse=True)[:20]:
print(f"p95={p95:.3f}s n={n}", key)
The percentile convention is explicit: ceil(0.95 * n) - 1 is a zero-based nearest-rank index. The earlier shortcut int(n * 0.95) rounds down and can select the wrong rank. If maintaining an awk parser for a known key-value format, reset extracted variables on every record, require both URI and a valid timing, and do not assume whitespace splitting understands quoting.
Avoid false confidence
Sampling can miss rare routes; it does not guarantee that they disappear. In a hypothetical independent 1% sample of 200 requests, the probability of retaining none is 0.99 ** 200, about 13.4%. Deterministic sampling may behave differently. Retain rare failures where practical and expose sampling policy alongside counts.
Group status classes with int(status / 100) or an equivalent mapping, not numerical rounding. Keep exact codes such as 499 available for investigation. Map dynamic paths to route templates where possible, without erasing distinctions needed to find a regression.
High tail latency, rising bandwidth, or repeated paths are investigation leads. Compare them with upstream traces, deployment history, verified crawler information, and business context rather than declaring a scanner or a performance incident from one number.
Related guide: What to actually look for in an nginx access log.
Working with messy spreadsheets? I build small browser-based tools for exactly these jobs — CSV merge, PII masking, date normalisation — all free to run locally, nothing uploaded.
If you'd rather not rebuild this from scratch, I packaged the exact scripts I use: LogDigest — turn an nginx log into a report in one command.
Top comments (0)