Mustafa ERBAY

Posted on Sep 13 Originally published at mustafaerbay.com.tr

IP Lists Aren't Identity: Authenticated Origin Pulls

#cloudflare #nginx #mtls #security

I closed yesterday's real_ip article with the sentence "I haven't turned on Authenticated Origin Pulls on my own server yet; it's on the list." This post is the first item on that list. But first I measured where things stand today, because "my origin is locked down" is a sentence about my own infrastructure, and like every "all of it is X" sentence about my own infrastructure, it needs evidence:

$ curl -s -m 8 -o /dev/null -w "%{http_code} exit=%{exitcode}\n" -k https://178.18.252.209/
000 exit=28

$ curl -s -D - -o /dev/null "https://mustafaerbay.com.tr/feed.json?nocache=1" | grep -i "^HTTP\|cf-cache-status"
HTTP/2 200
cf-cache-status: MISS

The request aimed straight at the IP waits eight seconds and dies; the same request through Cloudflare reaches the origin and returns 200 (MISS, so from the server, not the cache). For three months an nft table has let only new connections from Cloudflare's 15 IPv4 and 7 IPv6 prefixes through to ports 80 and 443. Since port 443 to Cloudflare is open from the same network, the timeout is the lock's doing, not the network's. The lock works.

My thesis is still this: that lock answers the question "is this connection coming from Cloudflare", not the question "is this connection coming for my domain". The two look like the same question; they are not. And the default setup of the mechanism Cloudflare recommends for this, Authenticated Origin Pulls, answers exactly the same first question. To see the difference, I built an nginx lab with seven ports and eight server blocks.

Two side effects of a lock that has worked for three months

The lock's story is short. On June 3 I first tried nginx allow/deny; because of real_ip_header CF-Connecting-IP, nginx at layer 7 saw the visitor's real IP rather than Cloudflare's, and for one minute the entire site returned 403. I told that story yesterday. The second attempt went down to layer 3: a separate inet cf_lock table, priority -10, looking only at new connections to tcp dport {80, 443}, Cloudflare sets and loopback pass, everything else drops. Since that day every HTTPS request sent directly to the origin gets the 000 above.

Two side effects surfaced; I noticed both only months later. The first I described yesterday in the MTA-STS post: six mta-sts.* vhosts on the same server are unproxied in DNS, so Google's policy fetcher arrives at the raw IP and hits cf_lock. A layer-3 lock knows no hostnames; it drops the packet without seeing which domain is written inside it. The second is quieter: on the September 10 reboot, journalctl shows nginx coming up at 21:30:50 and cf-lock.service at 21:31:38. For forty-eight seconds the origin was open to everyone. Nobody noticed, because nobody was looking; but that is the price of a design where the lock lives outside nginx, as a separate service. The maintenance cost comes on top: the twenty-two-prefix list matches what Cloudflare publishes today, but on the day it changes, it is not clear who will tell me.

The question an IP list cannot answer

Cloudflare's own documentation describes the IP list as "shared by all proxied hostnames": the same egress addresses serve every Cloudflare customer. Which means: someone who finds my origin IP can open a zone in their own Cloudflare account, point an A record at that IP, and their request arrives at my server from Cloudflare's IPs, and cf_lock lets it in. There is nothing at layer 3 to tell them apart; the packet genuinely comes from Cloudflare.

At that point one barrier remains: what nginx does with a Host header it does not recognise. Not a lock, a habit. On the Free plan the attacker cannot rewrite Host with Origin Rules, the plan table shows Host override open only on Enterprise; but that is a price list, not a cryptographic guarantee. In this morning's HSTS post I had curled a random subdomain over plain HTTP and seen nginx's welcome page: the default_server on port 80 was not returning 444 for an unknown Host, it was returning a page. I have not measured the 443 side yet; but I have no evidence that a habit loose on 80 is tight on 443.

This is why Cloudflare's origin-protection guide rates an IP allowlist "Moderately secure" and Authenticated Origin Pulls and Tunnel "Very secure". The difference is this: the first asks "where did you come from", the second asks "who are you".

What the certificate says: inside the CA file

The Authenticated Origin Pulls (AOP) documentation defines three independent levels; each has its own certificate and its own on/off switch, and enabling one does not affect the others. The global level works with a client certificate Cloudflare provides. The zone level and the per-hostname level work with a certificate you upload; per-hostname overrides zone, zone overrides global.

I downloaded the global level's CA file and opened it:

$ openssl x509 -in authenticated_origin_pull_ca.pem -noout -subject -dates -ext basicConstraints
subject=C=US, O=CloudFlare, Inc., OU=Origin Pull, L=San Francisco, ST=California, CN=origin-pull.cloudflare.net
notBefore=Oct 10 18:45:00 2019 GMT
notAfter=Nov  1 17:00:00 2029 GMT
X509v3 Basic Constraints: critical
    CA:TRUE, pathlen:2

The documentation's sentence about this certificate is short and clear: "This certificate is shared across all Cloudflare accounts and guarantees that the request is coming from the Cloudflare network." So global AOP does exactly what cf_lock does, with a certificate instead of an IP. A request coming from another customer's zone carries the same client certificate (all that customer needs is to enable global AOP on their own zone); your origin waves it through as SUCCESS. Only the zone and per-hostname levels answer the second question, "did it come for me": there you upload a leaf certificate issued by your own CA (try to upload a root and the API returns missing leaf certificate), and you put that CA on your origin. From then on Cloudflare presents that certificate only for your zone.

Reading this, it felt like I was about to build a higher-end model of the nft table I had spent three months on. One click further, though, is something genuinely different. But I wanted to measure before touching production, because I was not sure whether several pieces of "knowledge" sitting in my memory about nginx client-certificate verification were true.

The lab: one nginx, seven ports, eight blocks

On the Mac, a single container running nginx:1.28-alpine (nginx 1.28.3, OpenSSL 3.5.5), seven ports, eight server blocks; each block is a separate scenario. The server certificate is one I generated myself with blog.lab and mta-sts.lab SANs; on the server side there are two trust anchors, Cloudflare's real CA file and my own root CA; on the client side three certificate families: a CN=mustafaerbay.com.tr leaf issued from my own root, a leaf with the same CN issued from "someone else's" CA, and a three-link root → intermediate → leaf chain. I have no Cloudflare-signed client certificate, nor could I; that key lives at Cloudflare. The core of the configuration:

log_format aop '$server_port $ssl_server_name "$ssl_client_verify" '
               's_dn="$ssl_client_s_dn" i_dn="$ssl_client_i_dn" -> $status';

# 9443: transition mode — request a cert, verify it if present, 403 unless SUCCESS
server {
  listen 9443 ssl; server_name blog.lab;
  ssl_verify_client optional;
  ssl_client_certificate /certs/cf-ca.pem;
  location / { if ($ssl_client_verify != SUCCESS) { return 403; } }
}
# 9444: enforced mode; mta-sts.lab on the same port without verification
server {
  listen 9444 ssl; server_name blog.lab;
  ssl_verify_client on;
  ssl_client_certificate /certs/cf-ca.pem;
}
server { listen 9444 ssl; server_name mta-sts.lab; }
# 9445: zone-level simulation — my own CA
server {
  listen 9445 ssl; server_name blog.lab;
  ssl_verify_client on;
  ssl_client_certificate /certs/my-ca.crt;
}
# 9446: transition — Cloudflare's CA and my own CA in one file
# 9447 / 9448: three-link chain, default depth and ssl_verify_depth 2
# 9080: plain HTTP

Then the matrix with curl. Left column the scenario, right column the HTTP code and what nginx wrote into the $ssl_client_verify variable:

9443 optional, no certificate               403   "NONE"
9443 optional, cert from a foreign CA       400   "FAILED:unable to verify the first certificate"
9443 optional, my CA (CF CA expected)       400   "FAILED:unable to verify the first certificate"
9444 on, no certificate                     400   "NONE"
9444 on, foreign CA                         400   "FAILED:unable to verify the first certificate"
9444 same port, mta-sts.lab, no cert        200   "NONE"
9445 my CA on, no certificate               400   "NONE"
9445 my CA on, my own leaf                  200   "SUCCESS"  s_dn="CN=mustafaerbay.com.tr"
9445 my CA on, foreign leaf (same CN)       400   "FAILED:unable to verify the first certificate"
9446 both CAs in one file, my own leaf      200   "SUCCESS"
9447 default depth, 3-link chain            200   "SUCCESS"  i_dn="CN=deep intermediate"
9448 depth 2, 3-link chain                  200   "SUCCESS"
9447 default depth, leaf alone              400   "FAILED:unable to verify the first certificate"
9080 plain HTTP                             200

Two of the three pieces of "knowledge" in my memory turned out wrong in this table; in order.

optional does not mean lenient

The first rows. With ssl_verify_client optional, a client that sends no certificate gets into nginx, $ssl_client_verify becomes "NONE", and my if line turns it into a 403. But a client that sends a certificate and fails verification never reaches my if line: nginx returns 400 and writes client SSL certificate verify error: (21:unable to verify the first certificate) to the error log. The documentation says it in one sentence, "requests the client certificate and verifies it if the certificate is present", and for years I had read that sentence as "checks if present, passes if absent". The truth: passes if absent, rejects with 400 if present and broken. The genuinely lenient mode is optional_no_ca: it requests the certificate but does not require it to be signed by a trusted CA; the documentation says it is meant for cases where "a service that is external to nginx performs the actual certificate verification". Not for AOP.

This matters for the transition plan. Cloudflare's recommended order is "first optional on the origin, then flip the switch in the dashboard, then on on the origin"; the purpose of the optional phase is to see in the log, as "SUCCESS", that Cloudflare really presents a certificate. If you installed the wrong CA file, in that phase you see 400 rather than "SUCCESS", and that is a good thing: you catch the error before saying on. But do not assume optional will protect you from a wrong CA; it only tells you.

Then the third row of 9445: a certificate that says CN=mustafaerbay.com.tr but was issued from a different root got 400. What counts is the signature, not the name. When moving to the zone level, there is no security value in fussing over "the certificate's CN should be my domain"; the value is in who holds the root.

The favour SNI does that nft could not

The third row of 9444 is the most valuable measurement in this post for me. Two server blocks on the same port; blog.lab requires a certificate, mta-sts.lab does not. A client with no certificate got 400 from blog.lab and 200 from mta-sts.lab. During the TLS handshake nginx reads the SNI first, picks the server block, and sends the client-certificate request according to that block's setting. Of the three beliefs in my memory this was the only one that held: ssl_verify_client can decide per hostname.

That was exactly the nut cf_lock could not crack in three months. When you say "80/443 Cloudflare only" at layer 3, mta-sts.* is caught inside that sentence; in the TLS handshake you can say "this vhost wants a certificate, that vhost does not". That is also why the documentation carries the warning "This block also applies for requests to unproxied DNS records": enable AOP at the http level and unproxied hosts get locked out too; enable it at the server level and each makes its own decision. The reboot race disappears too: because the lock lives in nginx's own configuration, there is no service that has to come up before or after nginx.

The ssl_verify_depth myth

The third piece of knowledge in my memory: "AOP needs ssl_verify_depth 2, otherwise the chain won't verify." I had read that somewhere, the Cloudflare root being issued with pathlen:2 made it sound plausible, and in April's management-API mTLS post on this blog I too wrote the line ssl_verify_depth 2; without measuring it. I measured: on 9447 the depth is left at the default (1), and the three-link chain (root → intermediate → leaf) got "SUCCESS". On 9448 with depth 2, the same result.

The reason is in the OpenSSL documentation: "a depth limit of 1 there can be one intermediate CA certificate between the trust anchor and the end-entity certificate". Depth does not count the leaf or the root; it counts the intermediates between them. Since nginx's default is 1, a chain with one intermediate already passes. I measured this with OpenSSL 3.5; the "set it to 2" advice in tutorials may be a trace of an older OpenSSL, or a copy that was never measured. I measured my share; the rest I don't know.

Let me also write down what I could not measure: I do not know whether Cloudflare's actual client certificate is issued directly from this root or through an intermediate, because only Cloudflare presents that certificate and I have not yet flipped the switch on the live zone. The last row of 9447 shows why that uncertainty matters: if the intermediate is not sent in the chain, the leaf alone gets 400. If Cloudflare uses an intermediate, either it sends it in the chain or you have to add it to your CA file. This is the real reason to log $ssl_client_verify and $ssl_client_i_dn in the optional phase: you see the difference between "FAILED:unable to verify the first certificate" and "SUCCESS" in production, in one line.

Small traps

Cloudflare's PEM file arrives without a newline at the end of its last line. When I concatenated it with my own CA into one file using cat, nginx said cannot load certificate ... bad end line and never started. The -----END CERTIFICATE----- sticks to the next -----BEGIN on the same line. An echo between them fixed it; it is not in the documentation; it came out in the lab.

Connecting to 9444 with openssl s_client, the handshake shows CN=origin-pull.cloudflare.net under the heading "Acceptable client certificate CA names". The nginx documentation says this too: the list given with ssl_client_certificate is sent to clients. So an origin with AOP enabled tells everyone who knocks "I am expecting a Cloudflare certificate"; when you install your own CA, it tells them its name. Not a vulnerability, but know that it is not secret; for those who mind, the documentation's remedy is ssl_trusted_certificate, which does not send that list to clients. nginx also has two internal error codes: 495 for a verification error, 496 for a missing certificate; with error_page you can give your own response to either instead of 400.

File confusion is common too: this CA file is not the same thing as the Origin CA that Cloudflare uses to issue server certificates for your origin; the documentation warns about it in bold on the same page. And a date: the root expires on November 1, 2029, and on that day every origin using global AOP will want a new file.

The 9080 row: plain HTTP returned 200. AOP is a TLS mechanism; there is no such thing as a client certificate on port 80. On the origin you either close 80 or leave it redirect-only; I will keep the nft table precisely for this. Two layers, two separate questions: nft is cheap, it drops the packet before a handshake even happens; the certificate is expensive but it tells you who.

For Tunnel users the documentation's note is explicit: AOP is not compatible with Cloudflare Tunnel, because with a tunnel there is no listener for Cloudflare to present a certificate to; the tunnel already does its own authentication. In the post where I weighed Tunnel I did not write this item; now it is missing there too.

When you upload your own certificate, its expiry becomes your problem. Cloudflare can send notifications 30 and 14 days ahead; it does not delete an expired certificate, and if your origin accepts only valid certificates, every request drops that day. If two certificates exist on the same zone, both show as "active" but the most recently uploaded one is used. The zero-downtime sequence: upload the new one, see it become "active", delete the old one.

How I would migrate

Here is the order in which I will take the idea in this post's title to production. First optional and logging: $ssl_client_verify, $ssl_client_s_dn, $ssl_client_i_dn in a log_format. Then the global switch in the dashboard; in the API that is the tls_client_auth zone setting, the zone level has its own separate endpoint and the documentation specifically warns not to mix the two. A day of looking at "SUCCESS" lines; if I see "FAILED", the intermediate question is answered. Then on. Then issue a leaf from my own CA, upload it at the zone level, and replace ssl_client_certificate with my own root; keeping both CAs in one file during the transition worked in the lab (the 9446 scenario, "SUCCESS"). Leave the mta-sts.* blocks alone. Keep the nft table in place for port 80 and for the cheap drop. Also write Before=nginx.service into the cf-lock.service unit; though that only orders them, and if cf-lock fails nginx still comes up, so the clean fix is to move the rule into nftables.service's own configuration. With AOP on, the cost of those forty-eight seconds drops but does not reach zero: port 80 is still there.

Today, from the network I was on, SSH to the server would not open; just as well. A Sunday afternoon is not the right hour to switch on mTLS on a live site. That is what the lab was for.

Checklist before switching it on

Is the zone's SSL/TLS mode Full or Full (strict)? On Off and Flexible, AOP never engages.
Is there anything that has to reach the origin from outside Cloudflare: unproxied DNS records, monitoring services, ACME HTTP-01? A separate server block or a separate door for each.
Does the CA file end with a newline? Check with tail -c1 file | od -c.
Do not say on before seeing "SUCCESS" in the log during the optional phase; if you see "FAILED", read $ssl_client_i_dn.
If you settle for the global certificate, settle knowing it answers the same question as cf_lock; the "my zone" guarantee comes only with your own certificate.
Is an expiry notification set up for your own certificate? Cloudflare does not delete an expired one; your origin rejects it.
What is port 80 doing? AOP does not look there.
Is the file you put on the origin the AOP CA or the Origin CA? Two separate files, both from Cloudflare.
What does default_server return for an unknown Host? If not 444, you are at the same weak spot as the IP list.
If you use Tunnel, close this list; AOP does nothing there.

The difference between where and who

The lock I built three months ago was the right lock and it is still in place; but the question it asked was "where did you come from". The question I asked in this post is "who are you". The IP list and Cloudflare's shared certificate give the same answer in two languages: from Cloudflare. A certificate issued from your own CA gives a different answer: for you. The difference is not climbing one layer up; it is changing whose ledger the identity check is written in. In the first, the ledger sits in Cloudflare's IP publication and is shared with all its customers; in the second, it sits on your disk and is yours alone. Whatever the layer, before saying "all of it is safe", you have to ask who holds that ledger. Yesterday I said "it's on the list"; today I at least know what the list lists.

DEV Community