Half of a Magento store's traffic are bots, and today's scrapers solve puzzles and rent home internet. Here is the layered strategy we run, with real numbers.
Every Magento store I run has more visitors that are bots than human. I used to say that as a joke. Then I counted.
Across multiple production stores over a given period, only 19% of page requests came from a clean, ordinary browser. Another 31% were verified search engines, which we want. The remaining half was everything else: price scrapers, stock checkers, SEO tools, AI crawlers hoovering up product copy, credential stuffers, and a steady drizzle of vulnerability scanners.
Most of it is not an attack. All of it costs money. A bot walking your layered navigation burns the same PHP and database time as a shopper, and the shopper is the only one who ever pays you back.
On a managed platform with a giant CDN in front, somebody else fights this for you. Self-hosted, congratulations: you are the CDN. That is why I think anti-bot is the hardest part of running Magento yourself. Harder than caching. Harder than upgrades. And constantly eating server resources.
This post is the overview of how we handle it on every store we operate. Each layer gets a short section, and four follow-up posts go deeper. I will explain how each piece works. I will not publish the numbers, weights or exceptions, for reasons that become obvious by the end.
Scrapers in 2026 are not your cousin's curl script
Let me set expectations, because the internet still thinks a bot is a Python script with a funny user agent. Some are. The ones that matter are not.
**They solve proof of work. **One swarm that hit us solved our challenge every single time it was asked: 2,490 challenges, 2,490 solutions. Politely. Instantly.
**They drive real browsers. **Detecting headless Chrome was a fun hobby in 2020. In February 2026 we added headless detection to our challenge page and removed it the same evening, because stealth plugins walked straight past every check. A real browser, driven by automation, is still a real browser.
**They rent your customers' internet. **Residential proxy networks route scraper traffic through real home and mobile connections. A scraper on one of those looks exactly like a shopper on a VPN, or on a mobile carrier doing something creative with its traffic. Block the address and you block the buyer.
**They spread thin. **One hour on a day in September, one store: 32,462 requests from 31,201 different addresses. That is roughly one visit per address, then gone. Any rule that counts requests per address sees 31,201 perfectly polite visitors.
**They lie about who they are. **Most of them claim to be Chrome. Many claim to be Googlebot. The claiming is cheap; making everything else agree with the claim is not, which is where most of our catching happens.
**And fingerprints have flaws too. **In August, a swarm ran genuine Chrome. Its fingerprint was shared with 465 addresses of real customers on the same store. We had a wall that armed itself on fingerprints when a store was under load. That day it turned away real shoppers and broke checkout. We took it down for good. Lesson paid for in full.
There is no one thing
If you have read ZDNet's piece on Anubis, the open-source bot blocker that makes every visitor solve a proof-of-work puzzle, you know the idea is good. It is the same family of idea we use. Its own README calls it a bit of a nuclear response, and it is honest about why.
Then, in August 2025, Codeberg reported that AI crawlers had learned to solve the Anubis challenges. Not because Anubis is bad: any single mechanism, once popular, gets optimised against.
Cloudflare says the same thing about fingerprints in its JA4 signals write-up: fingerprints can be easily spoofed, they change frequently, and traffic keeps evolving. Their answer is to combine them with other signals. So is ours.
So the strategy is not a wall. It is a lot of small, cheap questions, each of which a bot can pass, and very few bots can pass all of them at once while still looking like the same visitor.
The edge: OpenResty instead of plain NGINX
Every store sits behind OpenResty: NGINX with a Lua runtime inside, so every stage of a request, from the encryption handshake to the log line, can run our own logic before a single line of PHP wakes up.
Plain NGINX can match paths and count requests. It cannot inspect a connection, keep a score, or talk to a challenge service. Part 1 covers why OpenResty and not Caddy, Traefik, APISIX or the forks: OpenResty vs NGINX, Caddy and Traefik.
Detect first, decide later
The most important design rule is boring: no single signal is allowed to block anyone.
Every request runs through every detector, and each one only adds to a score. Only when all of them are done does the edge look at the total. Low score, the visitor passes untouched. Middling, a challenge runs in the background. High, they are refused.
In our week of data, 82% of requests passed untouched, 3.4% got an invisible challenge, and 9.7% were stopped. Part 3 opens up the scoreboard: How to Build a Bot Score.
JA4 and HTTP/2: fingerprinting the connection
Every bot claims to be Chrome, and many copy Chrome's headers perfectly. What they rarely copy is how Chrome connects. The encryption handshake is as distinctive as handwriting, and JA4 turns it into a short fingerprint. HTTP/2 adds a second one: the settings a client announces when it opens the connection, baked deep into its network stack. Stock NGINX exposes neither, so we patched OpenResty from source to read them.
We do not care which browser you use. We care whether your connection matches the browser you claim to be. In our week of data, 174,920 requests claiming to be a browser had the connection of something else. Only 86 of them were caught by both fingerprints. Each caught what the other missed. That is layering in one number.
Part 2 goes deep on this, including the bugs: How to Spot a Fake Chrome.
Reverse DNS against the fake Googlebot
The oldest trick in the book: call yourself Googlebot. Stores are terrified of blocking Google, so plenty of setups wave that name straight through.
Google publishes how to verify its crawler: look up the hostname behind the address, check it is Google's, then confirm that hostname points back to the same address. We do that for anything claiming to be a search crawler.
The result made me laugh out loud: 210 of the 408 addresses claiming to be Googlebot were fake. The real Googlebot is never challenged, because we want to be indexed. The impostors were blocked 97% of the time.
Behaviour and reputation
A real browser viewing a product page also downloads its images. A scraper that only wants the HTML usually does not. Page after page with no pictures feeds the score quietly.
CrowdSec adds memory: a community blocklist of about 22,000 to 25,000 addresses per store, dropped at the firewall before they reach the edge. Addresses that merely behaved oddly get a score bump instead of a ban, because a shared office network should not be punished for one noisy laptop.
The Turnstile strategy, without the Cloudflare
Cloudflare Turnstile changed how people think about captchas: most visitors should never see a puzzle. The browser proves itself quietly in the background, and only the doubtful ones see anything.
We loved the strategy and did not want the dependency: third-party JavaScript on every page, a vendor in the checkout path, visitor data leaving the store. So we built the same pattern ourselves, self-hosted, and it took three attempts to get it right.
There is a boring reason too, and it might be the best one: time. Turnstile is set up per site, by a person: an account, keys, a widget wired into the store, domains to register. Ours ships with the store. It is installed automatically when a store is first provisioned, so every store has the same protection from day one, and nobody has to remember to go and set up a captcha. Part 4 is the story: Self-Hosted Proof of Work with ALTCHA.
Proof of work and ALTCHA
What the browser solves is a proof-of-work puzzle: search for a number that makes a deliberately slow hash come out right, which the server checks in one go. For one shopper that is a moment of CPU they never notice. For a scraper opening thousands of fresh sessions an hour, it is a compute bill, every session, forever. Proof of work does not prove you are human. It makes being a bot at scale expensive.
The engine is ALTCHA, an open-source, self-hosted, proof-of-work captcha running inside the edge. No image grids, no traffic lights, no data leaving the store. We also use it inside Magento, replacing reCAPTCHA on the storefront forms.
Does it work? Of the 404,149 visitors who were shown the full challenge page in our week, 85.6% never solved it. Some of the ones who did were bots too, which is fine: they paid for the privilege, and the other layers were still watching.
So when do we actually show the challenge?
Putting every request through proof of work is the Anubis approach, and it is a bad experience for a shop. Shoppers are impatient, phones are slow, and every half second on a product page is conversion you will never see. So we are picky about it:
**First visit from a browser: **an invisible check in the background, on the real page. The shopper reads the product title; the browser does the maths.
**A doubtful score: **the same invisible check, with the page still served.
**A high score: **a short interstitial that solves itself. A script that cannot run JavaScript is stuck at the door.
**A cold hit on an expensive page: **catalogue search, deeply filtered categories. These cost the database real work, so a visitor arriving there with no history on the store proves itself before the query runs. That one-hour swarm of 31,201 addresses? Almost all of it was aimed at site search, and it hit this gate.
**A store under pressure: **the puzzle gets harder for everyone who has not already earned trust. The cost follows the abuse, not the address, so rotating addresses does not help.
Verified search engines never see a challenge. A returning shopper with a valid pass does not see it again for a while. And software that talks to the store through its own machine channels, like payment providers and stock systems, is judged by different rules, because it cannot solve a browser puzzle and should not have to. We wrote about how that played out during a real zero-day in We Survived StyleSmuggler.
Where the bots actually go
Of everything we stopped, 46% was aimed at site search, 21% at layered-navigation filters, 19% at product, category and content pages, and 9% was exploit probes. Checkout and login together: under half a percent.
Read that again. The bots are not after your checkout (some does for skimming). They are after your catalogue, and specifically the pages that are most expensive for you to render. Anti-bot on a store is as much a performance feature as a security one.
Behind the edge
Anti-bot is the front line, not the whole defence. Behind it sit a web application firewall, a Varnish cache and a hardened, watched host. Every decision the edge makes is logged centrally, off the store, which is how every number in this post exists.
What I deliberately left out
There is not a single threshold, weight, cookie lifetime or exception list in this series. That is not modesty. Every number I publish is a number someone can tune against, and a list of what we skip is a list of where to aim.
I am also not going to pretend it is perfect. A patient human in a real browser on a real home connection is, correctly, indistinguishable from a customer. The goal was never to stop everyone. It was to make automated abuse slow, expensive and noisy, while a real shopper notices nothing at all.
Many small questions instead of one big wall. Detect first, decide later. Make bots pay in CPU instead of making people pay in patience.
Numbers in this post come from the edge logs of selected production stores we operate, over a given period — about 8.5 million requests. Store names are left out on purpose.
The anti-bot series: *Overview** · Part 1: OpenResty vs NGINX, Caddy and Traefik · Part 2: How to Spot a Fake Chrome · Part 3: How to Build a Bot Score · Part 4: Self-Hosted Proof of Work with ALTCHA.*
StoreFrame runs Magento on your own cloud with this edge in front of every store: an OpenResty anti-bot layer, self-hosted proof-of-work challenges, a firewall, hardened containers and central logs. It is not an add-on; it is the platform.
See how StoreFrame protects Magento stores
Top comments (0)