A few weeks ago I posted a milestone on LinkedIn: NessFlow had crossed 50 million server log lines analyzed in private beta. Most of the replies weren't about the number. They were about one line:
Sites that traditional crawlers fail to audit: WAFs, firewalls, adaptive rate limiting, 10k-SKU e-commerce catalogs, media.
Several people asked the same thing, in different words: which sites, and what exactly fails?
Here is one run, and the finding at the end of it, which had nothing to do with crawling.
The site
One of Carrefour's regional storefronts. Roughly 30,000 crawlable pages, on a PWA front end: the initial HTML response is a shell, and product, category and facet content is assembled client-side.
An SEO consultant had already tried to audit it with Screaming Frog. The crawl never produced a usable dataset.
Could Screaming Frog have crawled it?
I want to answer this honestly. The tempting version of this story, "their tool failed, ours didn't", is false, and any competent reader would take it apart in the comments.
Yes. Screaming Frog could have crawled this site. Let's be precise about why it didn't.
Scale was never the problem. 30,000 URLs is small. Screaming Frog's memory storage mode is recommended up to 500k URLs, handling a couple hundred thousand on 8GB of RAM; database storage mode goes into the millions. Anyone claiming 30k pages broke the tool is describing a misconfiguration, not a limit.
Rendering wasn't the problem either. The SEO Spider has Chromium-based JS rendering. It's expensive: you must keep resource crawling enabled, and Screaming Frog's own guidance notes most servers don't want to be crawled faster than about 5 URLs/second. So 30k rendered URLs is a multi-hour job on a laptop. Painful, not impossible.
The URL space was a real problem, but a solvable one. A grocery catalog with facets doesn't have 30,000 URLs, it has a combinatorial space. Screaming Frog handles this with exclude regex, if you already know the facet patterns. On an unfamiliar site you don't, so the first crawl diverges, and you find out at hour six.
Access was the actual blocker. One desktop, one IP, one user agent, sustained concurrency, against adaptive rate limiting and WAF edge rules. And here's the part that matters: the fixes are mostly not client-side. You can slow down and change your UA, but the reliable fix is having the client allowlist your IP at the edge, which is a permissions and relationship problem, not a software feature.
So the honest framing isn't "we crawl harder." It's: we start from the server side, where you don't need to be let in, because you're already inside.
The question a crawl doesn't answer
Here's where technical audits usually stop and where the commercial work starts.
Suppose you do get a clean crawl of 30,000 pages. You now have 30,000 rows of status codes, titles, word counts and canonicals. Almost none of that is a business finding, because in grocery e-commerce those 30,000 pages are not equal in any way that matters.
Search demand in grocery is brutally concentrated. Nobody types "plain yogurt 4×125g, brand X, reference 384920." They type "grocery delivery [city]", "buy basmati rice online", "organic products cheap". A 30,000-SKU catalog rarely carries more than a few hundred URLs with meaningful non-branded demand, and they're almost all category, subcategory and brand pages, not product detail pages.
That single fact reorganizes the audit:
- Crawl budget spent on PDPs is spent on pages that will never rank and rarely convert from organic. They still need to be indexable for long-tail and structured data, but they aren't where the money is.
- Facets are not uniformly bad. Most combinations are index bloat and should be closed. A small subset (organic, gluten-free, halal, brand-within-category) carries genuine standalone demand and deserves promotion to real landing pages with its own copy. Deciding which is an SEO judgement call, not a crawl setting.
- SKU churn is the silent killer. Grocery rotates constantly: seasonal lines, delisted references, permanent out-of-stock. Thousands of PDPs die every month. Whether they 404, 410, redirect to their category, or stay live with alternatives changes both your index footprint and your revenue, and most storefronts have never made that decision deliberately.
- Expired promotion pages accumulate. Weekly flyers and campaign pages go stale, become orphans and soft 404s, and they're often the pages that earned the site's few real backlinks.
- For a chain, store pages are frequently the strongest organic asset and the most neglected: thin, templated, duplicated across hundreds of locations, competing with each other.
So the first job isn't "find broken pages." It's segment by template, then ask which templates carry demand. Everything else is triage.
Your platform picked your no-JS ratio before you did
Once you segment by template, a pattern shows up across projects that has nothing to do with any individual site. The commerce platform and theme you chose already determined how much of your catalog exists in the initial HTML.
Every major platform now offers two paths:
| Platform | Server-rendered path | Headless / React path |
|---|---|---|
| Magento | Hyvä: PHP templates, Tailwind, Alpine.js | PWA Studio: React, Apollo, GraphQL |
| PrestaShop | Warehouse: Smarty templates | custom headless front end |
| WooCommerce | Astra: PHP/WordPress | headless WP + Next.js |
| Shopify | Online Store 2.0: Liquid + JSON templates, Dawn | Hydrogen: React |
| BigCommerce | Stencil: Handlebars | Catalyst: Next.js, React, GraphQL |
The left column ships fully-formed HTML. Hyvä is the clearest case: PHP composes the page on the server, Tailwind styles it inline, Alpine.js adds reactivity to specific components. Crawlers get the content and the structured data immediately, with no hydration gap. Warehouse, Astra, Liquid and Stencil work the same way for the same reason: they never stopped being server-rendered.
The right column is not "bad for SEO," and I'd push back on anyone who says so. Hydrogen and Catalyst both server-render. The leak is narrower and far more specific:
The shell renders. Then facets, pagination, variant selection, stock status and price are fetched client-side after hydration.
Which means the page a non-rendering client sees is a category page with no products in it. The template is fine. The catalog isn't there. And because the header, nav and footer render normally, a naive word count says the page has content, which is exactly why a raw word count is a useless test and a ratio isn't.
You can check any site in one command:
# words in the raw HTML
curl -sA "Mozilla/5.0 (compatible; Googlebot/2.1)" "$URL" \
| sed -e 's/<[^>]*>/ /g' | wc -w
Compare that to the word count of the rendered DOM. If the first number is a small fraction of the second on your category templates, the rest of this article is about you.
Two measurements
The control probe
Before crawling anything, NessFlow sends reachability probes to a handful of URLs with different identities, and one of them is deliberately fake:
$probes = [
'googlebot' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
'gptbot' => 'Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)',
'browser' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 …',
// the control: a user agent that exists nowhere,
// matches no rule, and is on nobody's allowlist
'control' => 'NessFlowReachabilityProbe/1.0',
];
The control is the whole point. On its own, "Googlebot got challenged" tells you nothing, because maybe the edge challenges everyone. Compared against a user agent no rule could possibly target, it becomes a diagnosis:
| Googlebot UA | Control UA | What it means |
|---|---|---|
| passes | challenged | Verified-bot allowlist. No third-party crawler will ever see what Google sees. |
| challenged | passes | UA-based blocking of known bots. Crude, and usually accidental. |
| challenged | challenged | Global edge rule. The site is closed to every automated client, Google included. |
| passes | passes | Open. Thin pages are a rendering problem, not an access problem. |
Note what the probe does not do: work around the answer. It reports the wall, it doesn't climb it. When the wall is real, the audit moves server-side.
The no-JS ratio
For every page we keep both views, raw HTML and rendered DOM, and compare word counts:
ratio = words(raw_html) / words(rendered_dom)
flag JavaScript-Only Content when:
ratio < 0.3
AND words(rendered_dom) >= 200
The 200-word floor matters more than the threshold. Without it, every legitimately short page (a thin category, a sold-out product, a redirect stub) trips the rule and the report becomes noise. With it, the flag means something specific: this page has real content, and a client that doesn't execute JavaScript sees almost none of it.
Run it per template, not per page. The flag almost never lights up uniformly. In practice it concentrates exactly where client-side filtering lives, in category and facet templates, which is also, per the section above, where the demand is. That coincidence is the finding. A JS-only PDP is a long-tail annoyance. A JS-only category template is the head of your demand curve.
The part that isn't technical
Googlebot renders JavaScript. It has for years, on a delay, and a well-built PWA can rank perfectly well. That fact has been used for a decade to close the conversation: "Google handles it, move on."
The crawlers that feed AI answers don't. Per Vercel's crawler analysis, GPTBot fetched JavaScript files in a minority of requests and never executed them; ClaudeBot downloaded JS and never executed it either. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot parse the HTML they first receive, take what's there, and move on. Bingbot renders partially and unreliably at scale.
Put those two facts side by side and a PWA storefront lands somewhere very specific:
It ranks in Google, and it does not exist in ChatGPT, Claude or Perplexity.
Not "ranks poorly." Does not exist. When someone asks an assistant "where can I buy organic rice online in [country]", the retrieval layer reads a shell with a nav bar and no products. The competitor on a server-rendered theme gets cited instead, and the gap is invisible in every rank tracker and every Search Console report, because neither measures retrieval by AI crawlers.
That reframes the no-JS ratio entirely. It isn't technical hygiene. It's the answer to "are we in the answer set?", and for a grocery chain the queries at stake are the high-intent ones: where to buy, delivery, availability, price.
One caveat I'd insist on: don't take Vercel's numbers, or mine, as your own. Crawler behavior changes quietly and varies per site. That's the argument for having logs at all. You can measure it directly on your own domain: which AI user agents arrived, what they requested, whether they ever fetched the JS bundles, whether they came back. We track 19 AI crawlers with their declared purpose (training / search / user-request) and whether they respect robots.txt, because several don't, by design.
On log analysis, and giving credit
Screaming Frog ships a Log File Analyser, and it's a good tool. It verifies bots against spoofed user agents, handles large files, and has done so for years. Claiming log analysis as a novelty would be silly.
The differences that made it worth building our own are narrow and specific:
- It's a desktop application, project-based, disk-bound: you import a file and look at it. We needed a continuous server-side pipeline, because crawl behavior is a time series, not a snapshot.
- No Search Console join. Ours joins crawl, GSC and logs on the same canonical URL hash, the registry from Part 4, which is what turns three separate reports into one answer about where the budget actually goes.
- No AI-crawler taxonomy. In 2026 that's the half of the log file that decides whether you're in the answer set.
Our parser handles 13 log formats at roughly 52,000 lines/second on a single node, parsing, normalizing and joining to canonical URLs, with no IP ever persisted. IPs are used in memory for reverse-DNS bot verification and discarded; they never reach an analysis table, and never reach a model prompt.
Three things I'd keep
- When a crawler stalls, name the wall: access, rendering, or URL space. They fail identically from the outside and the fix for each makes at least one of the others worse. The control probe is four HTTP requests and it tells you which one you're facing.
- A crawl of 30,000 pages is not an audit. Segment by template, find the few hundred URLs that carry demand, and spend the analysis there. In grocery that means categories, brands and store pages, not the catalog.
- "Google renders JavaScript" stopped being a sufficient answer. Check the ratio on your category templates, then check your logs for who actually came and what they got.
NessFlow is in private beta: crawl, Search Console and server logs joined on one canonical URL, with AI-crawler visibility built in. If you run a storefront that breaks crawlers, that's the profile I'm looking for. Reach out on LinkedIn.
Top comments (0)