DEV Community: North Dev

Stop AI scrapers without blocking Google: verified crawlers + a tarpit

North Dev — Mon, 26 Oct 2026 14:14:29 +0000

AI crawlers are hammering your site, ignoring robots.txt, and running up your bandwidth. The tempting fix — block by User-Agent — also blocks the real Googlebot you want, because anyone can claim any UA. Here's the approach that separates the two.

User-Agent is a claim, not an identity

User-Agent: Googlebot means nothing — a scraper sets it in one line. So a UA allowlist lets impostors in, and a UA blocklist eventually catches a real crawler you cared about. You need to verify.

Forward-confirmed reverse DNS

The one check that actually works for the big crawlers: FCrDNS.

Take the client IP, do a reverse DNS (PTR) lookup → a hostname.
Check the hostname ends in the crawler's domain (.googlebot.com, .search.msn.com).
Resolve that hostname forward (A/AAAA) and confirm it maps back to the original IP.

A spoofer can set a UA but can't make 66.249.x.x reverse to crawl-*.googlebot.com and forward back. Verified crawlers pass through untouched. (Range-only crawlers like DuckDuckBot need a published IP-range list instead.)

Tarpit the unverified — without hurting humans

For traffic that fails verification and ignores your rules, serve a tarpit: a cheap, bounded maze of generated links that wastes a scraper's crawl budget. Keep it honest:

Advertise the trap paths as Disallow in robots.txt. Compliant bots self-exclude; only bots that ignore robots fall in. That's what makes it ethical rather than hostile.
Fail open for humans. A normal visitor who stumbles onto a trap path shouldn't get mazed — gate on behavior, not on being unverified.
Non-destructive, synthetic content. No real data, size-capped, rate-limited.

Honest limits

This raises a scraper's cost; it isn't a wall. A determined operator rotating residential IPs defeats per-IP logic. The point is to make your site not worth the trouble.

I packaged the verify-good-bots + tarpit-the-rest logic into one Go binary, crawlgate — reverse proxy or middleware, forward-confirmed verification for Google/Bing/Yandex, rate limiting, and a non-destructive tarpit. Respects robots.txt and real crawlers.

How to verify a real Googlebot (User-Agent isn't enough)

North Dev — Mon, 26 Oct 2026 14:09:45 +0000

Your access log is full of Googlebot. Some of it is Google. A lot of it isn't. Here's how to tell them apart — because the User-Agent string is worthless for this, and blocking the fakes the wrong way will deindex you.

The User-Agent is a claim anyone can make

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) is a header. A scraper sets it in one line to inherit Googlebot's reputation (and your allowlist). So:

A UA allowlist lets every impostor straight in.
A UA blocklist eventually catches the real Googlebot — and now you're not in the index.

You have to verify identity, not read a label.

Forward-confirmed reverse DNS (the method Google itself documents)

Take the client IP. Do a reverse DNS (PTR) lookup → a hostname.
Check the hostname is under Google's domain: it must end in .googlebot.com or .google.com.
Resolve that hostname forward (A/AAAA) and confirm it resolves back to the original IP.

All three must hold. A spoofer can forge the UA but can't make 66.249.66.1 reverse to crawl-*.googlebot.com and forward-resolve back to 66.249.66.1.

# client IP 66.249.66.1
dig -x 66.249.66.1 +short        # -> crawl-66-249-66-1.googlebot.com.
dig crawl-66-249-66-1.googlebot.com +short   # -> 66.249.66.1  ✓ match

The suffix check in step 2 is not optional — a PTR record is controlled by whoever owns the IP's reverse zone, so evil-googlebot.com.attacker.net or a lookalike must fail. Confirm the dot-bounded suffix, then forward-confirm.

Bing, and the range-only crawlers

Bing (.search.msn.com), Apple, and Yandex support the same FCrDNS check. Some crawlers (e.g. DuckDuckBot) don't publish reverse DNS and instead publish an IP range list — for those, verify the IP is in the official, published ranges, and refresh that list.

Don't punish humans on the way

Once you can verify good bots, the temptation is to tarpit or block everything unverified. Keep it honest: a real person hitting your site isn't a verified crawler either. Gate on behavior (ignoring robots, abusive rate) — not on "failed bot verification" — so you slow abusive scrapers without breaking normal visitors.

I put forward-confirmed verification (Google/Bing/Yandex/Apple), rate limiting, and an ethical tarpit for the unverified-and-abusive into one Go binary, crawlgate — reverse proxy or middleware, respects robots.txt and real crawlers.

The nginx security headers checklist (2026), and how to verify them

North Dev — Mon, 26 Oct 2026 14:09:08 +0000

Most nginx configs are one add_header away from leaking more than they should. Here's a 2026 security-header checklist you can actually verify — plus the inheritance gotcha that silently disables half of them.

The headers that matter

Strict-Transport-Security — force HTTPS for future visits. max-age=63072000; includeSubDomains; preload.
X-Content-Type-Options: nosniff — stop MIME sniffing.
Content-Security-Policy — the big one. Even a minimal frame-ancestors 'self' (clickjacking) and a scoped default-src beats nothing. Build it per app.
Referrer-Policy: strict-origin-when-cross-origin — stop leaking full URLs.
X-Frame-Options: SAMEORIGIN — legacy clickjacking guard (CSP frame-ancestors supersedes it, but harmless to keep).
Drop server_tokens off to stop advertising your exact nginx version.

The gotcha that breaks everything

add_header does not inherit into a block that defines its own add_header. If you set security headers at server level and then add a single add_header inside a location (say, a cache header), every security header vanishes for that location — silently, no error.

Fix: define headers once at the broadest scope and don't re-declare add_header in child blocks, or repeat them (an include snippet of headers is the clean pattern).

Don't serve your dotfiles

A classic exposure — .git/, .env, .htpasswd reachable over HTTP:

location ~ /\.(?!well-known) { deny all; return 404; }

Verify, don't assume

Check what you actually send:

curl -sI https://example.com | grep -iE 'strict-transport|content-security|x-content-type|referrer-policy'

If a header you configured isn't there, you probably hit the add_header inheritance trap.

I put these rules (plus TLS protocol/cipher and server_name checks) into nginx-doctor — point it at your config and it prints each finding with the fix. Zero-config Go binary, read-only.

Stop AI scrapers without blocking Google: verified crawlers + a tarpit

North Dev — Mon, 26 Oct 2026 14:01:44 +0000

User-Agent is a claim, not an identity

Forward-confirmed reverse DNS

The one check that actually works for the big crawlers: FCrDNS.

Take the client IP, do a reverse DNS (PTR) lookup → a hostname.
Check the hostname ends in the crawler's domain (.googlebot.com, .search.msn.com).
Resolve that hostname forward (A/AAAA) and confirm it maps back to the original IP.

Tarpit the unverified — without hurting humans

For traffic that fails verification and ignores your rules, serve a tarpit: a cheap, bounded maze of generated links that wastes a scraper's crawl budget. Keep it honest:

Advertise the trap paths as Disallow in robots.txt. Compliant bots self-exclude; only bots that ignore robots fall in. That's what makes it ethical rather than hostile.
Fail open for humans. A normal visitor who stumbles onto a trap path shouldn't get mazed — gate on behavior, not on being unverified.
Non-destructive, synthetic content. No real data, size-capped, rate-limited.

Honest limits

This raises a scraper's cost; it isn't a wall. A determined operator rotating residential IPs defeats per-IP logic. The point is to make your site not worth the trouble.

The nginx security headers checklist (2026), and how to verify them

North Dev — Mon, 26 Oct 2026 14:00:01 +0000

The headers that matter

Strict-Transport-Security — force HTTPS for future visits. max-age=63072000; includeSubDomains; preload.
X-Content-Type-Options: nosniff — stop MIME sniffing.
Content-Security-Policy — the big one. Even a minimal frame-ancestors 'self' (clickjacking) and a scoped default-src beats nothing. Build it per app.
Referrer-Policy: strict-origin-when-cross-origin — stop leaking full URLs.
X-Frame-Options: SAMEORIGIN — legacy clickjacking guard (CSP frame-ancestors supersedes it, but harmless to keep).
Drop server_tokens off to stop advertising your exact nginx version.

The gotcha that breaks everything

Fix: define headers once at the broadest scope and don't re-declare add_header in child blocks, or repeat them (an include snippet of headers is the clean pattern).

Don't serve your dotfiles

A classic exposure — .git/, .env, .htpasswd reachable over HTTP:

location ~ /\.(?!well-known) { deny all; return 404; }

Verify, don't assume

Check what you actually send:

curl -sI https://example.com | grep -iE 'strict-transport|content-security|x-content-type|referrer-policy'

If a header you configured isn't there, you probably hit the add_header inheritance trap.

I put these rules (plus TLS protocol/cipher and server_name checks) into nginx-doctor — point it at your config and it prints each finding with the fix. Zero-config Go binary, read-only.

Why your PHP-FPM throws 502 under load (pm.max_children, explained)

North Dev — Mon, 26 Oct 2026 13:54:54 +0000

Your PHP site runs fine, then traffic spikes and nginx starts throwing 502 Bad Gateway. Logs blame the upstream. The cause is almost always one number in your PHP-FPM pool: pm.max_children.

What actually happens

Nginx hands PHP requests to PHP-FPM. FPM has a fixed ceiling of worker processes — pm.max_children. When every worker is busy and the listen backlog fills, FPM can't accept the connection, nginx's fastcgi_pass fails, and the user gets a 502. Under pm = dynamic, you'll also see this in the FPM status page:

max children reached: 37
listen queue: 120

That counter is the smoking gun: you've hit the ceiling 37 times.

How to size max_children (stop guessing)

The honest formula:

max_children = (RAM budget for FPM) / (average worker RSS)

Measure a real worker's resident memory (RSS) under your actual app, not memory_limit (that's a cap, not usage).
Pick a RAM budget — total RAM minus what the DB, cache, and OS need. Leave headroom.
Divide. That's your ceiling. Set pm.max_children to it, not to a number you copied from a blog.

Too high and a traffic spike spawns more workers than RAM allows → the OOM killer starts reaping processes (worse than a 502). Too low and you 502 under load you could have served.

The shared-pool 502 storm

The nastiest version: multiple sites share one FPM master and one max_children. One busy site starves the others, and they all 502 — even the idle ones. Fix: give each site its own pool with its own ceiling, sized from its own RSS.

Also worth setting

pm.max_requests (recycle workers to bound memory leaks).
Watch slow requests in the status page and the slowlog — a few slow endpoints hold workers hostage and fake a capacity problem.

I turned this into a read-only CLI, fpm-doctor — it reads your pool configs + live status, measures real worker RSS, and recommends max_children from your actual memory budget, flags the shared-pool pattern, and ranks your slowlog. Composer + .phar, never writes your config.

SPF "too many DNS lookups" (PermError): what it means and how to fix it

North Dev — Mon, 26 Oct 2026 13:54:18 +0000

Your mail quietly started landing in spam, and when you checked SPF you found this: PermError: too many DNS lookups. Here's exactly what that means and how to fix it without guessing.

The rule nobody tells you about

SPF (RFC 7208) caps the number of DNS-lookup-causing terms in your record at 10. Not 10 records — 10 lookups, counted recursively. Blow past it and a correct receiver returns PermError, which many treat as "SPF failed" — so your legitimate mail gets downgraded or junked.

What costs a lookup:

include: — 1 each, and you recurse into it (its own includes count too).
a, mx, ptr, exists — 1 each.
redirect= — 1 (when there's no all).
ip4: / ip6: / all — 0. These are free.

The trap is include:. One include:_spf.google.com can itself expand into several more includes. Add a few SaaS senders (include:mailgun.org, include:servers.mcsv.net, include:_spf.salesforce.com…) and you silently cross 10.

Diagnose it

Resolve your TXT record and walk the includes, adding up the lookup-causing terms across the whole tree. If the total is >10, that's your bug.

dig +short TXT example.com | grep spf1

Then follow every include:/redirect= and count. (This is tedious by hand, which is the point — tooling should do it.)

Fix it

Remove senders you no longer use. Old ESPs linger in SPF for years.
Flatten a stable include into its ip4:/ip6: ranges (free lookups) — but only for senders whose IPs rarely change, and re-check periodically, or you'll authorize stale IPs.
Delegate by subdomain. Send bulk/marketing from mail.example.com with its own SPF, keeping your root record lean.
Prefer -all (hard fail) once you're confident; ~all (soft fail) while testing.

Don't stop at SPF

SPF alone doesn't survive forwarding. Pair it with DKIM (a signature that survives relays) and a DMARC policy that moves from p=none (monitoring) to p=quarantine/p=reject once your reports are clean. All three is what actually keeps you out of spam.

I bundled the SPF lookup-counter (and DKIM/DMARC/TLS checks) into a zero-config CLI, domain-doctor — domain-doctor example.com prints a graded report with the fix for each problem. MIT, single Go binary.

Why Chrome's JA3 is useless now (and what to use instead)

North Dev — Mon, 26 Oct 2026 13:39:48 +0000

If you've ever tried to make an HTTP client "look like Chrome" to get past a WAF, you've probably copied a Chrome JA3 hash from somewhere, hard-coded it, verified it once against a reflector, and shipped. And then, weeks later, you got blocked anyway — with the exact fingerprint you "matched."

Here's what happened, and what to do about it.

Your TLS handshake talks before your HTTP does

Before a single byte of your HTTP request goes out, your TLS ClientHello is already on the wire: the version you offer, your cipher list, your extensions, your curves and signature algorithms. A CDN reads that and forms a fingerprint. If it says "Go stdlib" or "python-requests" instead of "Chrome," you can send the most perfect browser headers in the world and still eat a 403.

JA3 (Salesforce, 2017): an MD5 over five ClientHello fields.
JA4 (FoxIO, the modern successor): a structured, human-readable fingerprint — and it sorts the extension list before hashing.

That sorting is the whole story.

Chrome's JA3 is non-deterministic on purpose

Since Chrome 110 (2023), Chrome randomizes its TLS extension order on every connection — a deliberate anti-ossification measure. The side effect: Chrome's JA3 changes every connection. There is no "the" Chrome JA3. Hard-code one and you've pinned a single permutation real Chrome almost never reproduces — so you look less like Chrome.

Same client, three consecutive handshakes:

run 1: JA3 0a9d85cc...   JA4 t13d1516h2_8daaf6152771_d8a2da3f94cd
run 2: JA3 b7ca61f3...   JA4 t13d1516h2_8daaf6152771_d8a2da3f94cd
run 3: JA3 6910cc00...   JA4 t13d1516h2_8daaf6152771_d8a2da3f94cd

JA3 is all over the place. JA4 is constant — the sort washes out the shuffle. Match JA4, not JA3. (Firefox/Safari/iOS don't shuffle the same way, so their JA3 stays stable — this is a Chrome/Chromium thing.)

The third fingerprint most people miss

Even with byte-perfect TLS, you get flagged by your HTTP/2 fingerprint (the "Akamai h2" string): SETTINGS values, WINDOW_UPDATE, stream priority, and pseudo-header order. Most HTTP libraries never think about frame order, so this is where "I set the right ciphers" setups quietly fail. The real checklist is three numbers: JA4 (TLS), Akamai h2 (HTTP/2), and matching headers — not just a User-Agent.

Matching them in practice

I got tired of re-solving this from both ends, so I put it in one small Go binary, fp (fpkit, MIT):

fp get <url> --profile chrome — fetch with a byte-exact Chrome ClientHello + HTTP/2 settings. No headless browser, no CGo.
fp inspect — a reflector showing a caller's JA3/JA4/JA4H/Akamai-h2 with a plain-English verdict.
fp check <url> --profile chrome — a CI guard that fails when your client stops looking like a browser.
fp db ja4 <value> — reverse-look-up a JA4 against bundled browser + known-client fingerprints.

Try the reflector with no install:

curl --http2 https://fp.shshbshsnang.shop/api/all

or open https://fp.shshbshsnang.shop/ and it'll tell you whether you read as a browser or a bot.

Takeaways

Don't hard-code a Chrome JA3 — it's non-deterministic by design. Match JA4.
TLS isn't enough; match the HTTP/2 (Akamai) fingerprint too.
Measure against a reflector, and re-measure — fingerprints drift with every browser release.

Honest limit: a careful spoofer can forge any JA4, so an inbound JA4 guard raises cost, it isn't a wall. If you've seen a profile drift, or a CDN keying off something I haven't documented, I'd like to hear it.