Behind Cloudflare and nginx, All My Users Had the Same IP

My admin login page allows 5 wrong passwords per minute. After that, it should block you.

I tried 9. It never blocked me.

I checked the other rate limits in my app, and they did not work either. While I was fixing that, I found a second problem: my app saw every user in production as the same IP address. Nobody noticed, because the limiter was off. If I had only fixed the first bug, the limiter would have started to work with one counter for the whole internet.

Why the limiter did nothing

In ASP.NET Core, every request goes through a chain of middleware, one step after another, and the order matters.

One step is routing. It decides "this request is for the login endpoint". The rate limiter needs this answer, because every endpoint has its own limit.

In my app, the rate limiter was placed before routing. So it asked "which endpoint is this?" too early, got no answer, and let the request go. There was no error and nothing in the logs.

The fix was to move one line:

app.UseRouting();
app.UseRateLimiter(); // must come after routing

(If you never call UseRouting() yourself, ASP.NET adds it at the start for you, so you can't get this bug. I call it myself, later in the chain, and the limiter ended up above it.)

After this fix the limits started to work, and three more problems showed up. They were there all the time, but they could not hurt anyone while the limiter did nothing.

Every user had the same IP

My limits count requests per IP address. But my app sits behind Cloudflare and nginx, so the app never talks to the user directly. The user's IP comes in a header, X-Forwarded-For.

nginx builds this header like this:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

$proxy_add_x_forwarded_for takes the header it received and adds the address of whoever connected to nginx. In my setup that is cloudflared, running on the same machine. So every request arrived like this (the user IP here is an example):

X-Forwarded-For: 203.0.113.42, 127.0.0.1
                 real user      cloudflared, added by nginx

before the fix, my app used:  127.0.0.1
after the fix, my app uses:   203.0.113.42

By default, ASP.NET's forwarded headers middleware reads only one entry, the last one. And the last one was always 127.0.0.1. So my "3 guest sign-ups per IP" limit would have been 3 guest sign-ups for everybody.

There was a second problem in the same place. My config also trusted this header from anyone. In production the first problem hid it, but without that extra nginx hop, a client could write any IP in the header and get a fresh counter. I checked this in my test setup, and it worked.

This is the fix for both, a bit shortened:

builder.Services.Configure<ForwardedHeadersOptions>(o =>
{
    o.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    o.ForwardLimit = null; // read the whole list, not only the last entry

    // trust only my own hops: loopback and private networks
    o.KnownIPNetworks.Clear();
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("127.0.0.0/8"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("::1/128"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("10.0.0.0/8"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("172.16.0.0/12"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("192.168.0.0/16"));
});

Now the middleware reads the list from the end, skips the addresses it trusts (my own proxies), and stops at the first one it doesn't trust. That is the real user. If a client puts a fake IP into the header, it ends up further left, and the middleware never reaches it.

Ten wrong passwords would lock out everyone

My login limit was not per IP at all. It was one counter for the whole site, 10 tries per minute.

While the limiter did nothing, it did not matter. With the limiter on, anyone could send ten wrong passwords and close login for all users, and password reset too, because it used the same limit. I changed it to one counter per IP. An attacker still gets 10 tries, and everyone else can still log in.

My tests were green because of the bug

When the limits started to work, my integration tests started to fail. They send a lot of requests quickly, and they had passed only because nothing was ever blocked. I made the limits configurable, so CI can use higher numbers.

I also added two tests that check the limiter itself, not the endpoints:

• send one request more than the limit allows, and expect 429 Too Many Requests
• send requests as two different users through the same proxy, and expect two separate counters

Before this, I had never seen my limiter return 429. I think that was the real warning, and I missed it.

Check your own app

Send more requests than your limit allows. Only to your own app, of course:

for i in $(seq 1 11); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST https://your-app/login
done

You should see 429 at the end. If you see only 200, your limit does not work. Then try again with a different fake X-Forwarded-For header each time. If the 429 goes away, anyone can bypass your limit.

Have you ever checked that your rate limiter really returns 429 in production? I am curious if I am the only one who didn't.

---

Originally published on vasyl.blog.

I found this while building TextStack, a reader for technical books that explains hard terms in the book's context. The Android app is in closed testing, and I need a few more testers, so if you read on your phone, you can join the beta here.

Comments

[Harun - solo dev]

this is the kind of post that prevents real security incidents.

the x-forwarded-for header trap is brutal. i build koda on cloudflare workers + supabase, and i'm currently auditing my rate limiting strategy. your point about trusting only loopback/private networks in forwardedheadersoptions is exactly the kind of defense-in-depth most indie devs skip.

the scariest part: "my tests were green because of the bug." tests that never see a 429 are tests that don't actually validate the security boundary.

curious about your approach in edge compute environments (like cloudflare workers where there's no traditional nginx hop). do you rely purely on cf-connecting-ip, or do you still validate against a known proxy list?

massive respect for the curl test at the end. "if you see only 200, your limit does not work." that's going in my deployment checklist. 🐯🔒

[Vasyl]

thanks Harun! I never ran it on Workers, but as I understand there your code is already inside Cloudflare, so cf-connecting-ip comes from them and client can't fake it. no proxy list needed

in my case it's a home laptop behind CGNAT, so I use Cloudflare Tunnel. no open ports at all, so nobody can skip Cloudflare and talk to nginx directly

[Harun - solo dev]

vasyl, that makes perfect sense. running cloudflare tunnel behind cgnat is a brilliant way to handle the no-open-ports constraint.

since the origin only accepts connections from the tunnel, it completely eliminates the risk of someone bypassing the edge and hitting nginx directly. your origin is effectively invisible to the public internet.

thanks for sharing the real-world setup! it's a great reminder that sometimes the best security architecture comes from working around network constraints. 🐯

[shieldx]

Việc bị chặn nhầm cả một dải IP người dùng khi dùng Cloudflare là một lỗi cực kỳ đau đầu vì nó đánh sập trải nghiệm của khách hàng thật. Vấn đề này thường nằm ở chỗ cấu hình nginx chưa lấy đúng header X-Forwarded-For để nhận diện IP gốc của client thay vì IP của proxy. Mình từng gặp tình trạng tương tự khi thiết lập rate limiting dựa trên biến $remote_addr thay vì dùng module real_ip. Nếu không cấu hình set_real_ip_from cho đúng dải IP của Cloudflare thì hệ thống sẽ coi mọi request đi qua proxy là một nguồn duy nhất. Bạn nên kiểm tra lại xem module real_ip đã được kích hoạt và cấu hình đúng các dải IP của Cloudflare chưa trước khi áp dụng các luật chặn mạnh tay.

[IP99]

Classic Cloudflare/nginx trap: once TLS terminates upstream, your app sees the edge IP and every user collapses into one address. Same failure mode shows up behind mobile CGNAT and shared residential exits.

Worth logging the forwarded client address (and whether you trust that hop) separately from any IP risk feature. When you do keep an IP signal, treat sharedness and freshness as first-class — a busy exit is not the same as a dedicated one, and yesterday's label may already belong to someone else.

Soft second opinion if useful: curl ip99.com/v1/ip/8.8.8.8 returns usage_type plus a decaying risk score and evidence_state. Not a substitute for fixing X-Forwarded-For trust, just a clock on whether the address already carries proxy-class evidence.