Mustafa ERBAY

Posted on Sep 12 Originally published at mustafaerbay.com.tr

Whose IP Is in Your Logs? set_real_ip_from and a 403

#nginx #cloudflare #security #linux

Today's access log on the server behind this blog has 12,389 lines, and every line starts with an IP address. Not one of them belongs to the machine that actually opened the TCP connection. The connection came from a Cloudflare edge node; nginx read a different address out of an HTTP header and swapped it in before writing the line. The swap is so quiet that in most setups nobody asks when it happens, under what condition it does not happen, or who could abuse it. I had not asked either. On 3 June, when the site returned 403 to everyone for about a minute, I had to.

My thesis: on any server behind a proxy, the "client IP" is not a measurement, it is a trust decision. nginx's set_real_ip_from directive is that decision; real_ip_header only says which header to read. Get the decision wrong and your logs become fiction, CrowdSec bans the wrong person, and your allow/deny lines either let everyone in or lock your own front door. In this article I first tell my own accident, then read the mechanism from the documentation and a container lab, and answer three questions: when to believe the header, which link to take when the chain gets long, and whether you can live without trusting a header at all.

I have written about Cloudflare here mostly from the angle of tunnel and reverse proxy setup. This article is one floor below that setup: what happens after the packet reaches nginx and before the first line hits the log.

That minute in June

The incident was simple. Ports 80 and 443 on the server were open to the world; anyone who skipped Cloudflare and came straight to the IP bypassed the WAF, the rate limits, everything. To close that, I wrote the best-known recipe into nginx: allow for the 22 prefixes Cloudflare publishes, then deny all. nginx -t was clean, reload was silent. A minute later the whole site was returning 403. All Cloudflare traffic included.

The cause was these two lines, which had been sitting on the same server for months:

# /etc/nginx/conf.d/cloudflare/cloudflare-ips.conf
set_real_ip_from 173.245.48.0/20;
# ... 22 prefixes ...
real_ip_header CF-Connecting-IP;

By the time nginx evaluated the allow/deny rules, $remote_addr was no longer Cloudflare's edge address; the realip module had already replaced it with the visitor's real address. The rule said "only Cloudflare may enter", and nginx looked at every request and saw "this one is from Bursa, this one from Frankfurt, this one from Singapore". All denied. I rolled it back, then built the lock in the kernel instead of nginx. The reasoning for that choice is at the end.

The nginx logs from that day have long rotated away; the evidence I have left is a cf-lock.service file dated 4 June, 08:04, and the notes I wrote that day. But the behaviour itself is unchanged, and I reproduced it in a container for this article.

At which moment does the module act?

nginx's development guide splits an HTTP request into eleven phases and writes a single sentence about the first one: ngx_http_realip_module registers its handler at NGX_HTTP_POST_READ_PHASE "to enable substitution of client addresses before any other module is invoked". ngx_http_access_module, which provides allow/deny, runs six phases later, at NGX_HTTP_ACCESS_PHASE. The distance is by design: realip exists so that everything after it (access control, rate limiting, logging, the application) sees the visitor, not the proxy. What I had done was write the one rule that needs to see the proxy into the layer that sees the visitor.

The module's documentation is short, but every sentence is a decision. set_real_ip_from "defines trusted addresses that are known to send correct replacement addresses". real_ip_header picks the header; the default is X-Real-IP, the options are X-Forwarded-For, any custom header name, and since 1.5.12 proxy_protocol. The substitution happens only if the connection's real source matches the trusted list. If it does not match, the header stays there, can be read, can be logged, but $remote_addr is not touched. The original address is not lost either: since 1.9.7 the $realip_remote_addr variable keeps the address that opened the connection. In June I had no idea that variable existed.

One small note: the module is not built by default; it needs --with-http_realip_module. The nginx.org packages and the official Docker image have it; if you build your own or use a minimal image, look for it in nginx -V. The 1.30.0 on the blog's server has both the HTTP and the stream variant.

Drawing the trust boundary in a container

On my Mac I started the nginx:1.28-alpine image with eight separate server blocks (8081 through 8088; 8088 only for the negative case of the geo lock), all using the same log format: $remote_addr, $realip_remote_addr, the raw X-Forwarded-For and $proxy_protocol_addr side by side. Requests came from a curl container on the same bridge network; that container's address was 172.17.0.4, inside the 172.16.0.0/12 block. As the trusted list I used either that block or, deliberately non-matching, 10.0.0.0/8.

These three tests show whom the header belongs to:

# trusted: 172.16.0.0/12, real_ip_header X-Forwarded-For
# request XFF: "203.0.113.9, 198.51.100.5, 172.17.0.99"
8081 recursive off -> remote=172.17.0.99   realip_remote=172.17.0.4
8082 recursive on  -> remote=198.51.100.5  realip_remote=172.17.0.4

# trusted: 10.0.0.0/8 (client 172.17.0.4 does not match)
# request XFF: "203.0.113.9"
8087               -> remote=172.17.0.4    xff="203.0.113.9"

Line 8087 is the important one. The header arrived, it was logged raw, but nginx did not recognise the source and left $remote_addr alone. That line shows the difference between "believe the header" and "believe the header from this source".

Then I made the mistake everyone makes once:

# trusted: 0.0.0.0/0  (i.e. everyone)
8083 -> remote=203.0.113.9   xff="203.0.113.9"

A single curl -H "X-Forwarded-For: 203.0.113.9" and in the logs I come from whatever country I like. That line means that in every setup with set_real_ip_from 0.0.0.0/0, the rate limit, the country block and the CrowdSec decisions are one header away. RFC 7239 says it for the Forwarded header in one sentence: the header cannot be relied upon to be correct, as it may be modified, mistakenly or maliciously, by every node on the way to the server, including the client making the request. X-Forwarded-For does not even have an RFC; the sentence applies all the more.

When the chain gets long, which link?

The definition of real_ip_recursive is two sentences, and both are easy to misread. When off (the default), a connection from a trusted source gets the last address in the header. When on, nginx walks right to left and takes the first non-trusted address. That is the 8081/8082 difference above: the chain is 203.0.113.9, 198.51.100.5, 172.17.0.99 and the connection comes from 172.17.0.4. Off, nginx takes the last link, 172.17.0.99, which is inside my trusted block, so it mistakes my own proxy for the visitor. On, it skips 172.17.0.99 as trusted and stops at 198.51.100.5.

Note: it never takes 203.0.113.9 in either mode. Recursive does not mean "the leftmost, i.e. the original client"; it means "the first stranger left after I peel off the links I can vouch for". I do not know who wrote 198.51.100.5 there; maybe a real intermediate proxy, maybe 203.0.113.9 made it up. It is the first address I cannot vouch for, so nginx is right to stop there. Every link you add to the list (Cloudflare plus your own load balancer plus a sidecar) makes turning recursive on mandatory; otherwise $remote_addr is always your own internal address. I also tried a chain that is trusted end to end (again on 8082): with the header 172.17.0.50, 172.17.0.99 the result was 172.17.0.50; nginx walked all the way left and returned the last trusted link.

Cloudflare's own behaviour makes this chain logic concrete. According to the documentation, if the request arrives at Cloudflare without an X-Forwarded-For header, the outgoing header is identical to CF-Connecting-IP; if one is present, Cloudflare appends the address of the proxy that connected to it. I tried this against the blog's live address:

$ curl -H "X-Forwarded-For: 203.0.113.9" https://mustafaerbay.com.tr/robots.txt
# in the nginx log:
xff="203.0.113.9,2a02:4e0:...:a53f"   cf-connecting-ip="2a02:4e0:...:a53f"

The address I invented is still in the header, but my real address was appended after it; CF-Connecting-IP carries only the truth. Using real_ip_header X-Forwarded-For behind Cloudflare is not wrong; if the trusted list is only Cloudflare's prefixes, you get the real address whether recursive is on or off. But make the list wider than necessary and turn recursive on, and nginx walks all the way left and takes that fake "203.0.113.9". CF-Connecting-IP is single-valued and never sets that trap. Of today's 12,389 lines, exactly one had a multi-hop X-Forwarded-For, and it was my test.

What if the client forges CF-Connecting-IP itself? I tried; the request never reached my server. The Cloudflare edge returned 403 with error code: 1000 in the body. Cloudflare's page for error 1000 lists the possible causes, and three of them are exactly this topic: the request includes a CF-Connecting-IP header, the request includes two X-Forwarded-For headers, the X-Forwarded-For value is longer than 100 characters. So the header exists only on edge-to-origin traffic, and when someone tries to fake it from outside, the edge shuts the door. That does not excuse a loose set_real_ip_from list; a request that never sees Cloudflare and comes straight to your IP does not pass through that check.

Who else on this server looks at that address?

The address realip substitutes is not only written to the log. On the blog's server CrowdSec reads mustafaerbay.access.log (its counter shows thirty-seven thousand and some lines since the last restart) and writes its decisions into nftables; there are around twenty active decisions right now. Every one of them rests on the $remote_addr column. Had realip been misconfigured, one of two things would have happened: the header is never processed, CrowdSec bans Cloudflare's edge addresses as "attackers" and I lock my own front door; or everyone is trusted, the attacker writes any address into the header and hands the ban to a neighbour. In the CrowdSec article I described the decision engine; that the column feeding the engine is correct was the unstated precondition of that article.

limit_req and limit_conn also run in the PREACCESS phase, that is after realip, and are usually keyed on $binary_remote_addr. Without realip, all Cloudflare traffic collapses onto a handful of edge addresses and the rate limit hits everyone at once. With realip trusting everyone, the attacker writes a different header on each request and strolls through the limit. With a trusted list that is exactly your proxy's addresses, neither happens.

Not trusting a header at all: PROXY protocol

The shared weakness of every header-based method is that it lives inside HTTP. HAProxy's PROXY protocol moves the address into the first bytes of the TCP connection: one line as soon as the connection opens, before HTTP starts. Version 1 is human-readable, version 2 is binary. In nginx you enable it with listen 443 ssl proxy_protocol;, available since 1.5.12, with version 2 support arriving in 1.13.11; the address lives in $proxy_protocol_addr and can be moved into $remote_addr with real_ip_header proxy_protocol.

In the lab I opened port 8084 in this mode and sent a hand-written line from a separate Python container (172.17.0.6; nginx was at 172.17.0.3):

PROXY TCP4 203.0.113.9 172.17.0.3 40000 8084\r\n
GET / HTTP/1.0\r\n...
-> 8084 remote=203.0.113.9 pp=203.0.113.9 realip_remote=172.17.0.6

Then I sent a plain HTTP request to the same port: curl said Empty reply from server, and nginx wrote broken header: "GET / HTTP/1.1" while reading PROXY protocol to its error log. The specification explicitly asks for this: the receiver "MUST be configured to only receive the protocol described in this specification and MUST not try to guess whether the protocol header is present or not"; otherwise untrusted parties get a door to spoof their addresses. The same text says that sharing one port between public and private access is therefore impossible.

What matters here is that PROXY protocol provides no security on its own. Any client could send the line I sent; nginx would still write 203.0.113.9. That is why the specification adds that the receiver "SHOULD ensure proper access filtering so that only trusted proxies are allowed to use this protocol". PROXY protocol does not remove the header's trust problem; it pushes it down from the HTTP layer to the TCP layer, to the question "who may connect to this port". And that question is answered by the firewall. To me that is the protocol's real value: instead of telling nginx "interpret the header", it says "everyone reaching this port is a proxy, the rest is the firewall's job", and puts the responsibility in the right layer. On the Cloudflare side this is a Spectrum matter: PROXY protocol v1 and v2 can be enabled for TCP applications, but the documentation says the feature requires an Enterprise plan; the ordinary HTTP proxy has no such option. So this section is mostly for people behind HAProxy, their own load balancer, or a cloud LB that forwards at the TCP level.

Who is the source behind Cloudflare Tunnel?

For tunnel users the trusted-list question changes shape. Cloudflare's tunnel FAQ is explicit: "Does Cloudflare Tunnel send visitor IPs to my origin? No. When using Cloudflare Tunnel, all requests to the origin are made internally between cloudflared and the origin." For the visitor address it points you back to the header-based "restoring original visitor IPs" page. In practice that means nginx sees the connection from 127.0.0.1 or from the cloudflared container's address; what belongs in set_real_ip_from is not Cloudflare's 22 prefixes but the address cloudflared connects from. If the same server has both a tunnel and the classic proxy, both. The blog's server does run such a tunnel, but HAProxy sits behind it, so this part comes from the documentation rather than from my own logs.

Lists go stale; where should the lock live?

Cloudflare's "restoring original visitor IPs" page puts a warning right under its nginx example: the prefix list needs to be updated regularly. The cloudflare-ips.conf on the blog's server is dated 11 May and holds 15 IPv4 and 7 IPv6 prefixes; today I compared it line by line with cloudflare.com/ips-v4 and ips-v6, and it is identical. Four months without change. But I never wired that to a timer, and that is a gap: the day a new block is added to the list, traffic from it shows up in the logs with the edge address, and in CrowdSec's eyes it becomes "ten thousand requests from one IP". While writing this I made a note to myself: the diff goes into a weekly cron and alerts on any difference. The same list lives in two places on the server, nginx's set_real_ip_from lines and the nft set; update one and not the other and nginx trusts while nft drops, or the reverse. Both lists should be generated from one source, and the nginx side does not change without nginx -t && nginx -s reload.

The real decision is the one I got wrong in June: who should reject a connection that does not come from Cloudflare? There are three options, in three different layers:

The kernel, layer 3. nftables ip saddr always sees the real TCP source; realip is not even involved. That is what runs on the server today: the inet cf_lock table drops new connections to 80/443 whose source is neither in the Cloudflare set nor loopback; there is an ip saddr @cf4 line for IPv4 and a separate ip6 saddr @cf6 line for IPv6, because in the inet family one does not cover the other and Cloudflare edges can reach the origin over IPv6 too. A direct curl -m 8 from my Mac to the server's IP times out after 8 seconds. It is the cheapest use of the toolset I described in the nftables migration runbook.
nginx, but with the right variable. allow/deny only looks at $remote_addr; you cannot change that. The geo module, however, has accepted an optional address variable since 0.7.27: geo $realip_remote_addr $edge_ok { default 0; 172.16.0.0/12 1; } and if ($edge_ok = 0) { return 403; }. In the lab, sending X-Forwarded-For: 203.0.113.9 to port 8086 returned remote=203.0.113.9 edge=172.17.0.4; the visitor address changed, the lock looked at the edge address. With the trusted list set to 10.0.0.0/8: 403. That was the working version of June's lock, and that day I had no idea the variable existed.
Identity, not IP. Cloudflare's Authenticated Origin Pulls presents a client certificate (mTLS) on the edge-to-origin connection; the documentation gives the reason in one sentence: "Without AOP, anyone who discovers your origin server's IP address can send requests directly, bypassing Cloudflare and all its protections." A prefix list can go stale; certificate verification does not. I have not enabled it on my own server yet; it is on the list.

On port 8085 in the lab I reproduced June exactly: set_real_ip_from 172.16.0.0/12 plus allow 172.16.0.0/12; deny all;. A request without the header: 200. With X-Forwarded-For: 203.0.113.9: 403, and in the error log access forbidden by rule, client: 203.0.113.9. Right rule, wrong variable. A small confession: on the first attempt I had written return 200 in the server block and could not get a 403; return runs in the rewrite phase and never lets the request reach the access phase. Serving a static file under a location made the behaviour appear. Phase order misleads not only realip users but also people setting up a lab.

Six questions for your own setup

Is the set_real_ip_from list exactly your proxy's addresses, or a wide block like 0.0.0.0/0 or 10.0.0.0/8 "to keep it simple"? Every machine in a wide block can invent IPs on your behalf.
If real_ip_header is a multi-valued header (X-Forwarded-For), is the real_ip_recursive setting deliberate? With more than one of your own proxies in the chain, on is mandatory.
In which layer is the rule that rejects connections from outside the proxy? If it is allow/deny, it looks at $remote_addr and runs after realip; move it to the firewall or use geo $realip_remote_addr.
Who updates the prefix list, and when? A diff cron is ten lines.
Who else reads the same $remote_addr? CrowdSec, fail2ban, limit_req, the application log, analytics. A change that breaks realip breaks all of them at once; when you test, look at all of them.
Is Pseudo IPv4 enabled in Cloudflare? According to the documentation, in "Overwrite Headers" mode CF-Connecting-IP and X-Forwarded-For are overwritten with a pseudo IPv4 address and the real IPv6 moves to the CF-Connecting-IPv6 header. With that on, anyone feeding CF-Connecting-IP to realip sees IPv4 addresses in their logs that do not exist, and CrowdSec bans them.

An address is not a measurement, it is a signature

A connection's TCP source is beyond argument; the packets came from there. An address in an HTTP header is someone's claim. set_real_ip_from is the line that says which signature you will accept that claim under, and nginx does not guess it for you; RFC 7239 and the PROXY protocol specification say the same thing: without a trusted-proxy list, the address field has no meaning. That minute in June taught me two things. First, what I called "the visitor IP" is really the last link of a trust chain, and every setting that changes the chain changes every rule that looks at it. Second, the lock must live in the layer where the address it needs to see lives. The edge address lives in TCP; a rule that wants to see it must live there too.

Behind each of the 12,389 addresses in today's logs there is a 22-line trust list that has not changed in four months. As long as the list is right, the addresses are real. And because I wrote the list, the responsibility for the word "real" is mine, not Cloudflare's.

DEV Community