DEV Community

Jonas Francisco de Lima Filho
Jonas Francisco de Lima Filho

Posted on AI-assisted

One User-Agent string got a 403. curl, wget and Go went straight through.

Where the block actually happens, measured with eight clients on the same URL.

I publish everything I build to a small private ops portal. It sits behind basic auth, and in front of that nginx there is a Cloudflare zone for the domain. Today I sent it a request from a Python client with the library's default User-Agent and got back a 403 whose body was 17 bytes long: error code: 1010. The two explanations that come first are the credential file and a forgotten firewall rule. Neither was true. The site answered normally to every other client I tried, and the only difference between the request that failed and the ones that worked was the string the client used to introduce itself.

Eight strings, one blocked

I sent the same GET to the same host, in the same minute, changing one header:

User-Agent Status
Python-urllib/3.14 403
Python-urllib/3.12 403
python-requests/2.32.3 401
curl/8.5.0 401
Wget/1.21.4 401
Go-http-client/2.0 401
my-app/1.0 401
Mozilla/5.0 (X11; Linux x86_64) Chrome/126.0 401
(no User-Agent header at all) 401

Two strings failed and both were the same token with a different version number. Everything else reached nginx and got the 401 that basic auth is supposed to produce. That rules out the explanations that come to mind first. A rate limit would have cut the run short, not blocked two specific strings. A broken credential would have produced the same status for all of them. An outage would not care which version of Python the client claims to be. And the blocked clients are not the automated ones: wget and the Go standard library look nothing like a browser, and both were allowed through.

The block happens before your server sees anything

The 403 came with headers I had never looked at closely:

HTTP/2 403
content-type: text/plain; charset=UTF-8
content-length: 17
cache-control: private, max-age=0, no-store
server: cloudflare
cf-ray: a5da0ee3dee9dd6b-DFW
Enter fullscreen mode Exit fullscreen mode

No www-authenticate, no content-type: text/html, no nginx version. The body in hex is error code: 1010 plus a newline, 17 bytes, exactly what content-length announces. That is not a page my server generated.

Cloudflare's documentation for error 1010 describes it as access denied based on the browser's signature, and lists among the common causes "the default Python urllib user agent" as an example of what Browser Integrity Check treats as a non-browser client (developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1010/, read on 2026-11-20). The check is part of the zone configuration, not of my application, which explains why nothing in my nginx logs would ever show this request.

The origin answers in 34 milliseconds

To confirm the server was innocent I kept the hostname and TLS handshake and pointed only the connection at loopback:

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  --resolve portal.example.com:443:127.0.0.1 \
  -A "Python-urllib/3.14" https://portal.example.com/
Enter fullscreen mode Exit fullscreen mode

The answer was 401 0.034. Three repetitions gave 0.036, 0.034 and 0.034 seconds. The same request through the public hostname took about 130 to 160 milliseconds and returned 403. --resolve keeps the certificate validation and the virtual host match, so this is the same request as far as nginx is concerned, and nginx answered with the auth challenge, meaning it never considered the User-Agent at all. The shape matters more than the milliseconds: the origin is a local round trip, the edge adds a network round trip and a WAF decision.

That single command splits the two halves of the problem. If the origin fails too, your configuration is the suspect. If the origin is fine and the public name is not, the difference is in whatever sits between them, and no amount of reading the server config will show it.

A credential cannot fix a request the edge rejected

The part that cost me the most time: the 403 came back even with valid credentials attached.

source ~/.site_auth
curl -s -o /dev/null -w '%{http_code}\n' -u "$SITE_AUTH" \
  -A "Python-urllib/3.14" https://portal.example.com/    # 403
curl -s -o /dev/null -w '%{http_code}\n' -u "$SITE_AUTH" \
  -A "Mozilla/5.0 (X11; Linux x86_64) Chrome/126.0" https://portal.example.com/    # 200
Enter fullscreen mode Exit fullscreen mode

Same user, same password, same URL, one header different. The WAF evaluates the request before nginx gets a chance to read the Authorization header, so the credential was never examined. That is the practical rule I took from it: when a URL that normally answers 401 starts answering 403, the password was not looked at. Rereading the credential file, rehashing it, or trying another user is work in the wrong layer.

Removing the header is not the fix

The absence of a User-Agent also passed, with a 401. That makes "delete the header" look like a solution, and it is a bad one for two reasons. The rule matches a signature, so a client that stops signing itself is still an unnamed client, and the log entry it leaves is no more useful than before. And the monitor that only alarms on 403 would go green while the request still does not identify itself.

The documented fix is the simpler one: send a descriptive User-Agent. I tested two, my-app/1.0 and jonas-check/1.0, and both passed the edge with a 401. No browser impersonation needed.

That matters to me because of what my own scripts do. Several of them set a constant at the top of the file, and the constant is a full Chrome string:

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
Enter fullscreen mode Exit fullscreen mode

A scraper that calls itself Chrome is lying to every server it talks to, including servers I own, and it makes the access logs of those servers useless for telling a real browser from a script. my-app/1.0 is one line, passes the check that blocked the original request, and tells anyone reading the log what is happening.

Which layer my smoke test is allowed to judge

My deploy script runs a set of checks after each publish. Every one of them goes through this function:

check() {
  local url="$1" expect="$2" desc="$3"
  local code
  # --resolve aponta o hostname do cert para o loopback local, para nao
  # depender de DNS/Cloudflare no smoke test
  code="$(auth_curl -o /dev/null -w '%{http_code}' \
    --resolve "portal.example.com:443:127.0.0.1" "${BASE}${url}" || echo 000)"
  [[ "$code" == "$expect" ]] && echo "    OK   ${desc} (${code})" \
    || { echo "    FAIL ${desc} esperado=${expect} obtido=${code}" >&2; fail=1; }
}

check "/"                            "200" "index"
check "/health"                      "200" "health"
check "/css/style.css?v=${ASSET_SHA}" "200" "css"
check "/js/app.js"                   "200" "js"
Enter fullscreen mode Exit fullscreen mode

The comment in the middle is the decision. The smoke test deliberately bypasses DNS and Cloudflare, because a deploy should not fail for a reason that has nothing to do with the files it just wrote. The price of that choice is the corollary: a green deploy proves nginx has the files, and proves nothing about a client reaching them through the edge.

Both checks are worth having, and they answer different questions. If I pointed this one at the public hostname, a 403 from the edge would turn every deploy red for a reason unrelated to the release, and a 401 would pass while proving nothing about the content. The arm that goes through the edge needs its own client with a known User-Agent, its own expected status, and its own message.

When one client fails and the others do not

The sequence that worked, in the order that saved the most time:

  • Reproduce with the exact client that failed. A browser standing in for a script hides the whole problem.
  • Change one header at a time and keep URL, method and credentials fixed.
  • Read the body and the server header before reading the status code. Seventeen bytes of plain text is not your application.
  • Test the origin with --resolve to separate edge from server in one command.
  • Treat 403 on a URL that normally answers 401 as "the application never saw this request".
  • Give every script a fixed, descriptive User-Agent, written in the code, not passed through the environment.

Quem mantém painel privado atrás de Cloudflare já levou um 403 desses e sabe o tempo que se perde procurando no lugar errado. Se aconteceu com você, conta nos comentários qual foi o cliente e o que apareceu no corpo da resposta.

Top comments (1)

Collapse
 
nexusverify profile image
Nexus Verify •

That 17-byte error code: 1010 body is the giveaway — Cloudflare's Browser Integrity Check rejected Python-urllib/3.14 before your origin or basic auth ever saw it. One check worth adding: run a second post-deploy probe through the public hostname with urllib's default User-Agent and expect a 401, not a 403, because your current --resolve smoke test bypasses Cloudflare, so it would pass while the real client is still blocked.