DEV Community: Tony Wang

AI vs Traditional Web Scraping: Which Wins, When

Tony Wang — Fri, 10 Jul 2026 11:10:39 +0000

Key takeaways

Traditional scraping (CSS/XPath selectors) is fast, cheap, and near-100% accurate on stable pages — but brittle: by one industry estimate, 10–15% of crawlers need maintenance every week as layouts shift.
‘AI web scraping’ is really three methods — AI-generated code, full-page LLM extraction, and vision (screenshots) — with very different cost, speed, and accuracy.
In a McGill benchmark of 3,000 pages, every LLM method scored above 98% accuracy and AI-generated code hit 100% — but full-page LLM parsing adds ~30s latency per page and a token tax (raw HTML is a median ~7.4× the tokens of the text you actually want).
Neither approach solves the hard part: you still have to fetch the page past proxies, rendering, and anti-bot before anything parses it.
The 2026 consensus is hybrid — deterministic extraction (or a structured API) for known, high-volume sources; AI for the unknown long tail.

AI web scraping and traditional web scraping solve the same goal — turning a web page into usable data — in very different ways, and each wins in different situations. Traditional scraping parses HTML with rules you maintain; AI scraping asks a model to read the page; a structured data API skips the page entirely for known sources. Here’s how they actually differ, with the 2026 numbers, and when to use each.

Traditional web scraping (selectors)

You fetch the HTML and extract fields with CSS/XPath selectors or a library like BeautifulSoup or Scrapy:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/product/123").text
soup = BeautifulSoup(html, "html.parser")
price = soup.select_one(".price").get_text(strip=True)

Strengths: fast, cheap, deterministic, and near-100% accurate when the structure is stable.
Weaknesses: every site needs its own parser, and selectors break the moment the layout changes. The real cost is maintenance — by one industry estimate roughly 10–15% of crawlers need attention every week, and that climbs on JavaScript-heavy sites — plus the proxies and rendering needed just to fetch the page.

"AI web scraping" is really three methods

People say "AI scraping" as if it’s one thing. In practice a 2025 McGill study that benchmarked AI extraction across 3,000 pages identifies three distinct approaches — and they behave very differently.

1. AI-generated code. You hand an LLM a sample of the page’s HTML and a description of what you want; it writes the scraper (selectors and parsing logic). You review it once, then run it deterministically — so there’s no per-page model cost, execution is near-instant, and in the benchmark it reached 100% accuracy, on par with a hand-written scraper. The catch is the same as traditional scraping: it breaks when the layout changes — unless you regenerate it (the "self-healing selector" pattern). This is the method that quietly blurs the AI-vs-traditional line.

2. Full-page LLM extraction. You send the page’s (ideally cleaned) HTML plus a prompt or JSON schema, and the model returns structured data. No selectors to write, one prompt can cover many layouts, and it’s resilient to redesigns. The trade-offs are real: it pays a token tax (see below), adds latency — the McGill run averaged ~30 seconds per page — and can occasionally mislabel a field.

3. Vision (screenshots). A vision-capable model reads a screenshot of the rendered page. It handles visually complex or dynamic layouts and has a fixed cost — about $0.0004 per page in the benchmark, regardless of page complexity — at the price of slower processing and a higher hallucination risk.

# Full-page LLM extraction — a prompt, not a selector:
"From this page, return JSON with: product_name, price, rating."

Across all three, the McGill benchmark put accuracy above 98%, with AI-generated code at 100% — so AI extraction is genuinely reliable now. The differences are in cost, latency, and how each fails.

The token tax nobody mentions

Full-page LLM extraction has a cost that demos hide: HTML is mostly scaffolding. One developer measured it on 10 real pages and found raw HTML cost a median 7.4× the tokens of the text you actually want — with a spread from 1.1× on a minimal page to 47.8× on a news homepage (112,721 tokens of HTML wrapping just 2,356 tokens of text — 98% scripts, nav, and tracking).

Two things follow. First, clean the page before the model sees it — converting to markdown or stripping script/nav/footer is where most of the savings are, not the model. Second, that multiplier hits every scheduled run: a per-page cost that feels like a rounding error becomes a real line item once you multiply it by your page count and your daily cron. A structured API sidesteps this entirely by never sending HTML to a model.

The catch: AI doesn’t get you the page

The most common misconception is that AI scraping solves blocking. It doesn’t. Every method above still has to fetch the page first — past rate limits, IP bans, CAPTCHAs, and JavaScript rendering. An LLM is great at reading a page you already retrieved; it does nothing about residential proxies, headless browsers, or anti-bot defenses. AI changes parsing, not fetching — and on protected sites, fetching is the hard part.

A structured data API (skip the page)

For known platforms — search, maps, marketplaces, social, finance — a structured data API returns documented, normalized JSON, so there’s no HTML to parse with selectors or a model, and no token tax:

curl -s "https://api.crawlora.net/api/v1/amazon/product/B0DGJ736JM" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Strengths: no parser, no per-page model cost, predictable schema, anti-bot handling behind the endpoint, and a hosted MCP server so agents can call it as a tool.
Weaknesses: only covers supported platforms — for an arbitrary unknown page you still want AI extraction or a crawler.

Side by side

	Traditional (selectors)	AI extraction (LLM)	Structured API
Setup per site	Write a parser	Write a prompt	None (documented endpoint)
Handles layout changes	No — breaks	Yes — adapts (or self-heals)	N/A — no page parsed
Cost per page	Lowest	Token tax + latency	Per credit, predictable
Speed	Milliseconds	~17–30s (parse / vision)	Fast
Accuracy (McGill, 3k pages)	~100% when stable	Above 98%	High for supported fields
Maintenance	High (selectors rot)	Lower (semantic)	None (managed)
Solves blocking?	No	No	Yes (behind the API)
Best for	Stable, high-volume targets	Unknown / long-tail pages	Known platforms at scale

So which should you use?

Known platform, repeatable, at scale → a structured API (Crawlora). No parser, no per-page model cost, agent-ready, and the anti-bot problem is handled for you.
Arbitrary or unknown page, low volume → AI extraction. It adapts without selectors. Use AI-generated code when you’ll re-run the same site often (write once, run cheap); use full-page or vision extraction for one-offs and messy layouts.
Stable target, very high volume, cost-critical → traditional selectors can still be cheapest — if you’ll maintain them, or let AI regenerate them when they break.
Whole site into a RAG index → a crawl-to-markdown tool like Firecrawl.

In practice, most production stacks are hybrid: deterministic extraction (or a structured API) for the sources they hit constantly, AI for the long tail — which is exactly what the 2026 buyer’s guides converge on. Whichever you choose, collect only public data and respect each source’s terms — see is web scraping legal in 2026.

Sources

Sources

Next steps

Compare tools in best AI web scraping tools in 2026, see how data feeds models in web scraping for AI training data, and try the AI Web Scraping API in the Playground.

Frequently asked questions

What is the difference between AI and traditional web scraping?

Traditional scraping fetches HTML and parses it with CSS or XPath selectors you maintain per site. AI web scraping hands the page to an LLM that returns fields from a prompt, adapting to layout changes. A structured data API skips parsing entirely for known platforms by returning documented JSON.

What are the types of AI web scraping?

Three main methods: AI-generated code, where a model writes the scraper once and you run it deterministically; full-page LLM extraction, where you send the page and a prompt and the model returns JSON; and vision-based extraction, where a model reads a screenshot of the rendered page. They differ in cost, speed, and accuracy.

Is AI web scraping more accurate than traditional scraping?

Both can be very accurate. In a McGill benchmark of 3,000 pages, LLM methods scored above 98% and AI-generated code reached 100%, on par with hand-written scrapers. AI is more resilient when layouts change; traditional selectors are near-perfect on stable pages but break on redesigns.

How much does AI web scraping cost per page?

It depends on the method. Full-page LLM extraction pays a token tax — raw HTML is a median of about 7.4 times the tokens of the text you want, and far more on bloated pages. Vision extraction is a fixed fraction of a cent per page. AI-generated code has no per-page model cost once written. A structured API charges a flat credit and sends no HTML to a model.

Does AI web scraping avoid getting blocked?

No. AI helps parse a page you already fetched; it does nothing about proxies, browser rendering, CAPTCHAs, or anti-bot defenses. You still need to retrieve the page before any model can read it — and on protected sites, fetching is the hard part.

When should I use a structured API instead?

When the source is a known platform — search, maps, marketplaces, social, finance — you call repeatedly, and you want clean JSON for an agent or pipeline without maintaining parsers or paying a per-page token tax.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Best SERP APIs in 2026 for Rank Tracking and Search Data

Tony Wang — Fri, 10 Jul 2026 11:10:19 +0000

Key takeaways

Google has no official SERP API — the Custom Search JSON API closed to new customers in 2025 and shuts down January 1, 2027 — so production rank tracking runs on third-party SERP APIs.
Compare on engine coverage, structured output, SERP-feature parsing (AI Overviews now appear on ~48% of tracked queries), localization, and cost per successful query — not headline price.
For multi-engine rank tracking with the least parsing, a structured search API across Google, Bing, and Brave is the fastest path; pick SerpApi for feature breadth, DataForSEO for bundled SEO datasets, or Serper/Scrapingdog for the lowest cost.
After Google's December 2025 lawsuit against SerpApi, keep a tested second provider and normalize on a schema you control.

A SERP API turns a search engine results page into structured data — positions, URLs, titles, snippets, and SERP features — so you can track rankings, monitor competitors, and feed search data into products. This guide covers why you need one, what to compare in 2026, how the main options differ, and how to turn whichever you pick into a working rank tracker.

Why there's no official Google SERP API

Google deprecated its first Web Search API back in 2011. The closest first-party option, the Programmable Search Engine / Custom Search JSON API, was closed to new customers in 2025 and is scheduled to shut down on January 1, 2027 — and even at its best it was limited to 100 free queries/day and a configured engine rather than open-web parity. Bing's Search API was also retired in August 2025 (see Bing Search API alternatives). That leaves third-party SERP APIs as the practical way to collect search results at production scale — which is why this comparison matters.

What to compare in a SERP API

Headline price is the wrong starting point. Compare on:

Engine coverage — Google only, or also Bing, Brave, and others? Rankings differ by engine, so cross-engine visibility matters.
Structured output — clean, normalized fields (position, URL, snippet, modules) or HTML you parse yourself?
SERP-feature parsing — organic results, related queries, people-also-ask, news, video, local pack — and AI Overviews, which now appear on roughly 48% of tracked queries; an API that can't parse them returns incomplete rankings.
Localization — country and language parameters.
Free tier — to benchmark on your real keywords before committing.
Cost per successful query at your real volume, including retries (see below).

The best SERP APIs in 2026

There is no single winner — pick by whether you want multi-engine breadth, the deepest SERP features, bundled SEO datasets, or the lowest cost. Here is the landscape, then a closer look.

Provider	Engines	Output	Notable	Best for
Crawlora	Google, Bing, Brave (+ Trends)	Normalized JSON, identical shape per engine	2,000 free credits/mo; one schema across engines	Multi-engine rank tracking, minimal parsing
SerpApi	Many engines + deep SERP features	Structured JSON	Broadest feature/engine coverage; priciest	Feature breadth across engines
DataForSEO	Google + others	Structured JSON + datasets	Keyword volume, difficulty, backlinks; cheapest at scale	SEO platforms needing datasets
Serper	Google	Clean JSON	Among the cheapest (~$0.30–$1/1K), fast	Low-cost, high-volume Google
Scrapingdog	Google + more	JSON	Fastest in benchmarks, low cost at scale	Budget high-volume scraping
Bright Data	Google, Bing, Yahoo, Yandex, DuckDuckGo, Baidu	Structured JSON	Flat pricing, enterprise infra	Enterprise scale across engines
ScrapingBee / SearchApi / Decodo	Google	JSON	Mid-tier value, simple integration	General-purpose SERP

1. Crawlora — multi-engine structured JSON

Crawlora exposes a search endpoint per engine — Google Search, Bing, and Brave — returning normalized JSON with an identical shape across engines, so the same code snapshots all three:

curl -s -X POST "https://api.crawlora.net/api/v1/google/search" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keyword": "best serp api", "language": "en", "country": "us", "limit": 10}'

import os
import requests

resp = requests.post(
    "https://api.crawlora.net/api/v1/google/search",
    headers={"x-api-key": os.environ["CRAWLORA_API_KEY"]},
    json={"keyword": "best serp api", "language": "en", "country": "us", "limit": 10},
)
for row in resp.json()["data"]["result"]:
    print(row["position"], row["title"], row["link"])

Switch the endpoint to /bing/search or /brave/search for the same shape, and pair it with the Google Trends API to connect rankings with demand. When to choose it: multi-engine rank tracking with the least parsing and single-source resilience, on a free tier (2,000 credits/month, no card). See Crawlora vs SerpApi and vs DataForSEO. The trade-off: SerpApi documents more niche Google surfaces.

2. SerpApi — feature-rich, but the priciest

SerpApi is mature and SERP-focused with the broadest engine and SERP-feature coverage (including niche Google surfaces like Flights, Hotels, and Scholar). It is also the most expensive — roughly $9–$15 per 1,000 searches depending on tier, with hard plan caps and no pay-as-you-go.

When to choose it: you need deep SERP-feature breadth and niche Google surfaces, and the cost fits. See SerpApi alternatives for cheaper options.

3. DataForSEO — cheapest at scale, with SEO datasets

DataForSEO pairs a SERP API with keyword volume, difficulty, and backlink datasets, and its queued pricing is among the cheapest at scale (fractions of a cent per SERP). The trade-off is a more involved API and a higher minimum spend.

When to choose it: you are building an SEO platform that needs SERPs and the supporting datasets from one vendor.

4. Serper & Scrapingdog — lowest cost, fast

Serper is among the cheapest Google SERP APIs (around $0.30–$1 per 1,000 at volume) with clean JSON and low latency; Scrapingdog is similarly budget-friendly and fast, with dedicated endpoints. Both trade some feature depth for price and speed.

When to choose it: high-volume Google rank tracking where cost-per-query and speed matter most.

5. Bright Data — enterprise scale across engines

Bright Data ships a SERP API across Google, Bing, Yahoo, Yandex, DuckDuckGo, and Baidu with flat per-request pricing on enterprise-grade infrastructure (SSO, audit logs, support).

When to choose it: rank tracking that spans many engines at scale where reliability and enterprise controls outweigh per-request cost. The trade-off: no ongoing free tier and heavier procurement.

6. ScrapingBee, SearchApi, Decodo — mid-tier general-purpose

ScrapingBee, SearchApi, and Decodo (formerly Smartproxy) are solid mid-tier Google SERP options with simple integration and reasonable pricing — good when you want a dependable general-purpose SERP API without enterprise overhead.

7. Generic scrapers and proxies — DIY SERP parsing

Generic scraping APIs and proxy networks can fetch search pages, but you handle parsing and SERP structure yourself — more flexible, more work for rank tracking specifically. See ScraperAPI alternatives and the category overview in Best Web Scraping APIs in 2026.

Track SERP features, not just positions

Rankings in 2026 are more than ten blue links. AI Overviews now appear on roughly 48% of tracked queries, alongside people-also-ask, local packs, video carousels, and news. A position-only tracker that ignores these undercounts your real visibility — an organic #1 sitting below a large AI Overview is not the same #1 it was a year ago. When you compare SERP APIs, check which features each one actually parses on your queries, not just whether it returns organic positions.

Reliability: the 2025 SerpApi lawsuit

In December 2025, Google filed a lawsuit against SerpApi over scraping Google Search results; SerpApi has moved to dismiss and the case is unresolved. It does not mean any one provider is disappearing, but it is a reminder to treat SERP data like any critical dependency: keep a tested second provider and normalize on a schema you control, so swapping sources is a config change rather than a rewrite. More in SerpApi alternatives.

Cost per successful query, not sticker price

Benchmarked Google SERP rates span a wide range — roughly $0.30 to $15 per 1,000 queries — but the rate alone is misleading. Retries: a query that hits a challenge, returns partial data, or times out still costs you, and you pay again on the retry. Parsing maintenance: even a cheap fetch carries the engineering time to keep selectors alive as layouts shift. A structured API that returns normalized JSON on the first call and absorbs parser upkeep can be cheaper per usable record even at a higher sticker price — and far more predictable. Use the free tiers (Crawlora 2,000 credits/month, plus free credits from most providers) to benchmark on your real keywords first.

From SERP API to rank tracker

Whichever you choose, the rank-tracking pattern is the same:

Define keywords by country and language.
Snapshot the SERP on a schedule.
Store one row per keyword per run — position, URL, snippet, and the SERP features present.
Diff over time to surface movement.

The SERP monitoring use case walks through this, and How SERP Monitoring APIs Work builds the tracker step by step. The SERP tracker API, SERP tracking API, SERP rank checker API, and white-label SERP tracking API pages cover the variants.

Recommendation

For structured, multi-engine SERP data that drops into a rank tracker with minimal parsing, a structured search API across Google, Bing, and Brave is the fastest path. If you need deep SERP-feature coverage choose SerpApi; for bundled SEO datasets, DataForSEO; for the lowest cost at volume, Serper or Scrapingdog; for enterprise multi-engine scale, Bright Data. Compare on coverage, SERP-feature parsing, and cost per successful query — not headline price.

Next steps

Test a search endpoint in the Playground, read the shapes in the API docs, and check pricing. The comparison index has every head-to-head.

Sources

Sources

Frequently asked questions

What is a SERP API?

Is there an official Google SERP API?

No. Google deprecated its Web Search API in 2011, and the closest first-party option — the Custom Search JSON API — was closed to new customers in 2025 and is scheduled to shut down on January 1, 2027. Production rank tracking therefore relies on third-party SERP APIs.

What is the best SERP API for rank tracking?

For multi-engine rank tracking with minimal parsing, a structured search API across Google, Bing, and Brave (such as Crawlora) is the fastest path. For deep SERP-feature breadth choose SerpApi; for bundled keyword and backlink datasets, DataForSEO; for the lowest cost at volume, Serper or Scrapingdog; for enterprise multi-engine scale, Bright Data.

How much does a SERP API cost?

Benchmarked Google SERP rates span roughly $0.30 to $15 per 1,000 queries. SerpApi sits at the high end (about $9–$15/1K), while Serper and Scrapingdog are among the cheapest at scale. Compare cost per successful query — after retries and the SERP features you actually need — not the headline rate.

Do I need to track AI Overviews?

Increasingly, yes. AI Overviews now appear on roughly 48% of tracked queries, and a position-only tracker that ignores them undercounts real visibility. Check which SERP features (AI Overview, PAA, local pack, video) each API parses on your queries, not just whether it returns organic positions.

Is there a free SERP API?

Crawlora includes 2,000 free credits per month (no card) covering Google, Bing, and Brave search endpoints. Most SERP-specific APIs also offer a small free tier or credits (for example SerpApi's 250 searches/month) for evaluation.

Do I need Google, Bing, and Brave?

Rankings differ by engine and Bing and Brave feed some AI answers, so cross-engine tracking catches visibility shifts you would miss on Google alone. Because the response shape is identical across engines, adding them is cheap.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Proxies for Web Scraping, Explained (2026)

Tony Wang — Fri, 10 Jul 2026 11:10:08 +0000

Key takeaways

Proxies spread scraping traffic across many IPs to avoid rate limits and bans, and let you appear from a specific country — but they do not exempt you from a site's rules.
Trust ladder: datacenter (cheap, detectable) → residential → ISP (static residential) → mobile (costly, hardest to block). Start cheapest, move up only when blocked.
Rotating proxies suit many independent requests; sticky sessions hold one IP for multi-step flows like logins and carts.
Proxies are one layer — modern anti-bot also fingerprints TLS, headers, and behavior; a managed API hides proxies entirely for supported sources.

A proxy is an intermediary server that forwards your request to a target site and returns the response, so the site sees the proxy's IP address instead of yours. For web scraping, proxies spread requests across many IPs to avoid rate limits and IP bans. This guide explains the main proxy types, how rotation works, when you actually need proxies, and when a managed API makes them someone else's problem.

Why scraping needs proxies at all

Send a few hundred requests from one IP and most sites will throttle, challenge, or block it. Proxies solve that by distributing traffic across a pool of addresses, so no single IP looks abusive. They also let you appear to come from a specific country, which matters for geo-targeted content like local search results, prices, or availability.

Proxies are not a license to ignore a site's rules. They are an infrastructure tool; you still need to respect rate limits, terms, and the law. See is web scraping legal.

The main proxy types

Type	Detectability	Cost	Best for
Datacenter	Easy to block (known IP ranges)	Cheapest	Lenient targets, high throughput on a budget
Residential	Hard to block (real home IPs)	Higher, usage-based	Defended targets
ISP (static residential)	Hard to block	Mid–high	Residential trust + datacenter speed and stable IPs
Mobile (4G/5G)	Hardest to block (shared carrier IPs)	Most expensive	The most aggressively defended targets

Rule of thumb: start with the cheapest tier that works for your target and only move up the trust ladder when you get blocked.

Rotating vs sticky sessions

Rotating proxies give you a different IP on each request (or every few minutes) from a large pool. Best for spreading many independent requests — search results, product listings, broad collection.
Sticky (session) proxies hold the same IP for a set duration. Best when a workflow must keep one identity across steps — logins, multi-page flows, carts, or anything that breaks if the IP changes mid-session.

How proxies fit anti-bot defenses

Proxies are only one layer. Modern anti-bot systems also fingerprint TLS, headers, and browser behavior, and serve CAPTCHAs or JavaScript challenges. A clean IP with a sloppy browser fingerprint still gets blocked. Reliable collection usually needs proxies plus realistic headers, a real browser engine where pages render client-side, retries, and challenge handling — which is why DIY proxy management rarely stays simple.

When you don't need to manage proxies

If your target is a supported public platform, a structured API can hide proxies entirely. With Crawlora, proxy routing, browser rendering, and retries run behind the endpoint — you send a request and get normalized JSON back, with no pool to rent, rotate, or maintain:

curl -X POST https://api.crawlora.net/api/v1/google/map/search \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "coffee shops in Austin, TX", "limit": 20}'

You manage your own proxies when you crawl arbitrary or unsupported sites; you skip them when a documented endpoint already covers the source.

Quick decision guide

Lenient target, high volume, tight budget → datacenter proxies.
Defended target, need to look like a real user → residential or ISP proxies.
Most aggressively defended target → mobile proxies.
Supported public platform → a structured API so you skip proxy management.

Sources

Sources

Start collecting

Skip proxy management for supported sources: test an endpoint in the Playground, browse the API docs, and review pricing. See also how to choose a web scraping API and is web scraping legal.

Frequently asked questions

Are residential proxies legal?

Using proxies is generally legal, but how you source and use them matters. Use reputable providers with consent-based IP pools, scrape public data, and respect target rules and the law.

Datacenter or residential proxies for scraping?

Start with datacenter for lenient targets and speed; move to residential or ISP when you hit blocks on defended sites.

Do I always need proxies?

No. Small, polite workloads against lenient sites may not, and a managed API handles proxies for supported platforms so you don't rent or rotate them yourself.

Do proxies stop CAPTCHAs?

Not by themselves. Proxies address IP-based blocking; CAPTCHAs and fingerprinting are separate layers that need browser realism and challenge handling.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

World Cup 2026 TikTok Creators by the Numbers

Tony Wang — Mon, 29 Jun 2026 06:03:08 +0000

Key takeaways

The 2026 World Cup is enormous on TikTok — the #worldcup2026 hashtag alone has 25.3 billion views across 1.98 million videos. We went looking for the creators behind that wave across all 48 qualified nations, and the map of where the big ones live is not the one you'd draw.
The biggest TikTok creators aren't in the USA, Brazil or footballing Europe. Among World Cup nations the typical creator is largest in Saudi Arabia (median 45k followers), Egypt (42k), Colombia (41k), Ecuador (41k) and South Korea (39k) — top-heavy creator economies — while Germany, Spain and the US sit closer to 32–33k.
Star density says the same thing: about 30% of Egyptian and Saudi creators clear 100k followers, and Egypt has the highest share of million-follower accounts (4.0%). The footballing aristocracy fields more creators, but smaller ones.
The USA 'dominates TikTok' only at the very top — it holds 11 of the 15 most-followed accounts on earth. But measured by the typical creator, it's mid-table among World Cup nations. Megastars and a big creator base are two different things.
The twist: the nations with the biggest creators are the hardest to actually reach. In Saudi Arabia, Egypt, Ecuador and Argentina, only 1–3% of creators list any way to contact them, versus 10–15% in England, Canada and the Netherlands. Giant audiences, closed inboxes.
Coverage caveat: a creator's country comes from TikTok's own account region (with a best-effort flag-emoji/place-name fallback) and is present on only ~7.7% of creators — so it reflects account region, not nationality. 26 of the 48 qualified nations have enough data to rank; 22 — including Uruguay, Croatia, Senegal and Ghana — do not. Every figure reads as 'among established creators with a known country.'
Aggregate-only — counts and rates by nation, never individual creators — and the underlying table is open (CC BY 4.0) and reproducible.

The 2026 World Cup is the first 48-team tournament, played across 16 cities in the United States, Canada and Mexico — and on TikTok it is already vast. The #worldcup2026 hashtag alone has 25.3 billion views across 1.98 million videos, and that is before a single knockout match.

Conventional wisdom says the United States rules TikTok — and for the very top of the leaderboard, it does: 11 of the 15 most-followed accounts on the planet are American. But zoom out from that handful of mega-celebrities to the typical creator in each nation, and a stranger map appears. We pulled every World Cup nation's creators from Crawlora's 3.33-million-creator dataset and asked two simple questions: whose creators are biggest — and can you actually reach them?

Methodology & definitions at a glance

What we measured: Crawlora's creators_search dataset — 3,327,485 discoverable TikTok creators — filtered to the 48 nations qualified for the 2026 World Cup, June 2026 snapshot. Per nation we computed the median follower count, the share at brand-tier reach (100k+ and 1M+ followers), the share that is verified, and the share that lists any public contact.

How we assign a country: primarily TikTok's own account region, with a best-effort bio fallback (a flag emoji like 🇧🇷, then a place name) when region is missing — so it tracks account region, not nationality. It is present on only 7.7% of the dataset, so we measure the 170,551 qualified-nation creators who carry one. As a sanity check, posting language lines up with the assigned country (Spanish dominates Mexico and Argentina, Portuguese Brazil, Arabic Saudi Arabia). A nation needs at least 300 such creators to rank; 26 of 48 clear that bar.

What we did NOT measure: all of TikTok, or fan-account volume. This is the set of established, discoverable creators — not a census. England and Scotland both fall under ISO gb, which does not split UK home nations.

Aggregate only: we publish counts and rates by nation, never individual creators. Every figure is reproducible from the open dataset (CC BY 4.0).

The World Cup's biggest TikTok stars aren't where you'd think

Rank the 26 nations by the size of their typical creator — the median follower count — and the football map turns upside down. The biggest creators belong to Saudi Arabia, Egypt, Colombia, Ecuador, South Korea and South Africa, not to the tournament favourites:

Mapped, the pattern is unmistakable — the brightest creator economies ring the Gulf, North Africa and the Andes, not Western Europe:

Read down that list and it is almost the inverse of the form guide. The nations with the biggest creators — Saudi Arabia, Egypt, Ecuador, Colombia — are mostly tournament outsiders, while the powerhouses expected to contest the latter rounds (Germany, Spain, France) field smaller, more numerous creators. Star density tells the same story: roughly 30% of Egyptian and Saudi creators clear 100,000 followers, against ~22% in the US and 19% in Germany, and Egypt has the single highest share of million-follower accounts (4.0%). These are top-heavy creator economies — a smaller number of very large stars — whereas the European football heartlands run on a long tail of mid-sized creators.

It also reframes the "USA dominates TikTok" cliché. America genuinely owns the celebrity tier — Charli D'Amelio, MrBeast, 11 of the global top 15. But its median World Cup creator (33,338 followers) is squarely mid-table, smaller than eight other nations here. Megastars and a deep, large creator base are not the same thing.

Giant audiences, closed inboxes

Here is the part that surprised us most. The nations with the biggest creators are the hardest to actually reach. If you ranked nations by raw star power you would get one list; rank them by the share of creators who publish any way to contact them and you get almost the opposite. Plot the two together and the gap is the story:

In Egypt and Saudi Arabia, a creator is more than ten times as likely to clear 100,000 followers as to list a contact. In Ecuador the ratio is 24 to 1. The big-creator nations keep their audiences vast and their inboxes shut — most reach a manager only through a DM lottery. The English-speaking and Northern-European pools are the mirror image: smaller stars, but a contact listed 4 to 10 times more often. South Africa is the one nation that refuses the trade-off — top-six on reach and on openness.

The verified elite — and where sport sits

Verification is rarer than reach: only 2.73% of all 3.33M creators carry a badge. Among World Cup nations, Sweden (5.8%), Egypt (4.6%), Australia (4.5%) and England (4.3%) lead.

One cut is worth isolating for a football tournament: the sports niche. Across the dataset, sports creators are a verified elite — 5.9% carry a badge, 2.2× the 2.73% baseline. They are who they say they are far more often than the average creator — though, tellingly, no easier to reach: their contact rate (3.2%) sits below the all-creator average. A verified badge signals legitimacy, not availability.

Full table — all 26 measurable nations

World Cup 2026 TikTok creators by nation

Nation	Confed.	Creators	Median followers	100k+ reach	1M+ share	Verified	Lists a contact
Saudi Arabia	AFC	3,278	45,435	30.0%	2.81%	3.1%	2.3%
Egypt	CAF	1,843	42,200	30.4%	3.96%	4.6%	2.4%
Colombia	CONMEBOL	7,931	41,181	27.5%	3.16%	2.3%	2.7%
Ecuador	CONMEBOL	5,254	40,677	26.3%	2.04%	1.3%	1.1%
South Africa	CAF	2,028	39,664	26.5%	2.51%	3.6%	11.8%
Korea Republic	AFC	4,774	39,027	26.1%	3.35%	3.6%	6.3%
Japan	AFC	3,756	37,595	25.7%	2.26%	3.9%	3.6%
Morocco	CAF	4,050	37,104	24.3%	2.12%	1.7%	5.6%
Brazil	CONMEBOL	15,541	37,000	24.6%	2.54%	2.5%	3.0%
England (GB)	UEFA	13,717	34,179	23.3%	2.25%	4.3%	15.4%
Argentina	CONMEBOL	8,375	33,848	22.0%	2.10%	1.9%	1.8%
France	UEFA	7,061	33,603	24.4%	2.22%	4.0%	8.8%
United States	Host	22,636	33,338	22.5%	1.94%	2.9%	9.7%
Germany	UEFA	6,939	32,116	21.9%	2.16%	3.6%	6.9%
Spain	UEFA	6,670	31,722	22.3%	2.43%	2.7%	7.5%
Belgium	UEFA	1,014	31,550	22.1%	1.58%	2.7%	8.8%
Sweden	UEFA	895	31,400	23.8%	2.01%	5.8%	8.0%
Mexico	Host	25,173	31,113	20.7%	2.00%	1.7%	3.7%
Australia	AFC	6,023	29,344	20.5%	1.76%	4.5%	8.2%
Canada	Host	11,779	29,286	20.1%	1.71%	2.8%	11.6%
Switzerland	UEFA	1,101	29,000	22.1%	2.45%	3.1%	9.3%
Austria	UEFA	971	29,000	18.8%	1.13%	3.4%	3.9%
Netherlands	UEFA	1,960	28,600	18.6%	1.48%	4.0%	10.7%
Norway	UEFA	742	27,500	21.2%	1.21%	3.0%	11.5%
Portugal	UEFA	2,395	27,462	16.2%	1.13%	2.3%	5.0%
Türkiye	UEFA	4,645	26,164	18.8%	1.40%	3.2%	2.6%

TikTok creators by World Cup 2026 nation, sorted by median follower count. n=170,551 creators with a detected country, June 2026. Source: Crawlora creators_search.

What it tells us

Fans: the World Cup you scroll on TikTok is shaped by where the big creators actually are — and that is as much the Gulf, North Africa and the Andes as it is Europe or Brazil. Saudi Arabia and Egypt punch far above their football weight on the For You page.
On the 'USA dominates' myth: true for celebrities, not for the median creator. America owns the megastar tier and a huge creator base, but its typical creator is mid-table — a reminder that 'most-followed accounts' and 'biggest creators overall' are different leaderboards.
Creators: a giant following and a way to reach you almost never travel together. In most nations, simply listing a contact puts you in a small, valuable minority — nowhere more so than in the big-creator markets of the Gulf and South America.
Analysts & journalists: rank creator economies by the typical creator, not the top 10. Reach and reachability are weakly — often inversely — correlated across these 26 nations, so a 'star power' map and a 'who can you actually work with' map look nothing alike.

A 48-team World Cup will mint thousands of overnight creator moments. Where the biggest of them already live — and which of them you could ever actually reach — turns out to be one of the more surprising maps of the tournament.

Sources

Sources

Frequently asked questions

How big is the 2026 World Cup on TikTok?

Enormous. The #worldcup2026 hashtag alone has 25.3 billion views across 1.98 million videos as of June 2026 — before the knockout rounds — and that is one of many World Cup tags. The tournament is the first 48-team edition, hosted across the United States, Canada and Mexico.

Which World Cup country has the biggest TikTok creators?

Measured by median follower count, the typical creator is largest in Saudi Arabia (about 45,000 followers), Egypt (42,000), Colombia (41,000), Ecuador (41,000) and South Korea (39,000) — top-heavy creator economies. The footballing and creator giants are smaller: Germany and Spain about 32,000, the United States 33,000, England 34,000. Egypt and Saudi Arabia also have the highest share of creators above 100,000 followers (about 30%).

Does the USA dominate TikTok?

Only at the very top. The United States holds 11 of the 15 most-followed TikTok accounts in the world, but among World Cup nations its typical creator (median about 33,000 followers) is mid-table — smaller than Saudi Arabia, Egypt, Colombia, Ecuador, South Korea, South Africa, Japan, Morocco and Brazil. The US wins on celebrity megastars, not on the size of the average creator.

Are the biggest TikTok creators the easiest to reach?

No — it's almost the opposite. The nations with the biggest creators are the hardest to contact: in Saudi Arabia, Egypt, Ecuador and Argentina only 1–3% of creators list any public contact, versus 10–15% in England, Canada, Norway and the Netherlands. In Ecuador a creator is about 24 times more likely to clear 100,000 followers than to list a way to reach them. Giant audiences, closed inboxes — South Africa is the rare nation strong on both.

How was this World Cup creator study measured?

We filtered Crawlora's creators_search dataset (3,327,485 discoverable TikTok creators, June 2026 snapshot) to the 48 nations qualified for the 2026 World Cup and rolled up, per nation, the median follower count, brand-tier reach (100k+/1M+), the verified share and the share listing a public contact. Country is bio/locale-derived and present on about 7.7% of creators, so 26 of the 48 nations have enough data to rank (170,551 creators); the figures are aggregate-only and the dataset is open and reproducible.

Which World Cup nations could not be measured?

22 of the 48 qualified nations — including Uruguay, Croatia, Senegal, Ghana, Tunisia, Paraguay and debutants like Curaçao and Cabo Verde — have too few creators with a detected country to rank reliably. Country detection in the dataset skews toward larger English, Spanish and Portuguese-speaking markets, so smaller or less-represented nations fall below the 300-creator floor we require.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

The TikTok World Cup: Who the Internet Actually Talks About

Tony Wang — Mon, 29 Jun 2026 06:02:03 +0000

Key takeaways

The 2026 World Cup is a TikTok event before it is a stadium one — #worldcup2026 already has 25.4 billion views. So we ranked who the platform actually talks about, by hashtag views and by how many creators reference them — not by follower counts.
On TikTok, Messi beats Ronaldo. #messi has 700.7 billion views to #ronaldo's 580.6B — even though Ronaldo crushes Messi on Instagram followers (665M vs 506M). The internet makes more about Messi than about anyone alive.
A 17-year-old is rewriting the order. #lamineyamal (147.5B views) already outranks Haaland, Bellingham, Salah and Modrić — and Yamal is TikTok's single most-followed footballer (38.3M). TikTok's football attention is generational.
Argentina is the TikTok World Cup's biggest nation — #argentina has 631B football-tag views, ahead of France (576B), and far ahead of Brazil and England (~197B each). (We excluded #mexico and #japan: they're dominated by non-football culture.)
The fan-identity social graph has one giant edge: 161 creators name BOTH Messi and Ronaldo in their bio — 2.5× any other link. The GOAT debate is the gravitational centre. Players fuse with their nation (Messi–Argentina, Neymar–Brazil), and the viral newcomers aren't woven in yet.
Why TikTok's map differs from Instagram's: the algorithm rewards meme-ability, youth and authenticity over stature — follower count isn't even a ranking factor — so flair players (Yamal, Neymar, Vini) lead, not the all-platform giants. FIFA made TikTok its official video partner for 2026.
Aggregate-only — views and counts by player and nation, never individual creators — and the underlying data is open (CC BY 4.0) and refreshable.

Every World Cup now has two tournaments. One is played on grass. The other plays out on a billion phones, in fifteen-second clips, and it has its own table — one that does not always agree with the FIFA rankings or even with Instagram. We went to measure it: across the 48 qualified nations, which players and teams does TikTok actually talk about?

Not "who has the most followers" — that list is published every year and it is boring. We measured two things instead: how many views each player's and nation's hashtag has racked up (the size of the conversation), and how many of the platform's 3.33 million catalogued creators reference each one in their bio (the depth of the fandom). The gap between those two is where the story lives.

Methodology & definitions at a glance

Hashtag views: pulled live from TikTok via Crawlora's hashtag endpoint (June 2026), for a curated, football-specific tag per player and nation. These are cumulative, all-time view totals — the size of each entity's TikTok footprint.

Creator-mention affinity & the social graph: from Crawlora's creators_search dataset (3,327,485 discoverable creators; a bio is present on 92.5% of them — far higher coverage than profile-country). We count how many creators name each player/nation in their bio or display name, and how often two are named together (the co-mention edge).

Caveats we designed around: nation tags double as country-culture tags — #mexico (1.06T) and #japan (355B) are dominated by travel/food/anime content, not football, so we exclude them from the football ranking. #salah is inflated because "salah" also means "wrong" in Indonesian. View totals are cumulative, not tournament-only.

Aggregate only: we publish per-entity totals and edge weights, never individual creators. The data is open (CC BY 4.0) and refreshable.

The players TikTok can't stop posting about

Ranked by hashtag views, the top of the board is a Messi–Ronaldo–Neymar wall — and then a teenager:

Two things jump out. First, a 17-year-old (Lamine Yamal) sits fifth — ahead of Haaland, Bellingham, Salah and Modrić combined-adjacent — and he is, separately, the most-followed footballer on TikTok at 38.3M, ahead of both Ronaldo and Messi. The platform's centre of gravity is visibly young. Second, this is not the Instagram order: there, Ronaldo (665M followers) towers over everyone, Salah and Bellingham rank high on followers, and a teenager would not be fifth.

Messi beats Ronaldo — the one place he does

The Ronaldo-vs-Messi debate has a clean, surprising answer on TikTok, and it is the opposite of the Instagram answer:

It fits everything else about the platform. Ronaldo's dominance is built on followers — an audience that subscribes to him and his lifestyle empire. Messi's TikTok lead is built on being made about — the 2022 trophy lift, the Miami move, the GOAT edits, the memes. TikTok rewards the second thing, and it is why the #messi tag, fragmented as it is across #messi and #leomessi (another 72B), runs ahead.

The nations: Argentina runs the table

Among football-specific national tags, the reigning champions lead — and the order again ignores the bookmakers:

Argentina (631B) sits clear at the top, fittingly for the holders, with Morocco (163B) the standout overperformer — the Global-South wave from its 2022 semi-final run still moves on TikTok, level with Saudi Arabia and ahead of Croatia and the Netherlands. Brazil and England, two of the loudest football cultures on earth, land mid-table on the nation tag precisely because their TikTok energy flows through players (Neymar, Vini, Bellingham) rather than the country hashtag.

The social graph: who gets talked about together

We mapped which players creators name together in their bios — the fan-identity network. It has one overwhelming feature:

The graph is a solar system. Messi and Ronaldo are the twin suns — 161 creators build their identity around both, the GOAT debate made flesh. Neymar is the third body (63 links to Messi, 21 to Ronaldo), and Mbappé and Haaland orbit the edges. Two more patterns:

Players fuse with their nation. The cleanest one-way ties are Messi → Argentina (52 creators name both), Neymar → Brazil (27) and Ronaldo → Portugal (5). Fans don't just love the player — they wear the flag with him.
The viral newcomers aren't in the graph yet. Lamine Yamal pulls 147.5B views but only ~6 co-mention links and 32 bio-identity accounts; Vinícius, Endrick and Haaland are near-isolates. They're watched, not yet woven in — devotion lags reach by a generation.

Viral isn't the same as beloved

That last point is the whole study in one line. Put the two metrics side by side and they diverge hard. Messi has 700.7B views and 1,705 creators who put him in their bio — reach and devotion. Lamine Yamal has nearly a sixth of Messi's views (147.5B) but 32 identity accounts. Ronaldo: 1,112. Neymar: 911. The legends own the deep fandom; the new stars own the feed.

Full table — players by TikTok views and creator devotion

2026 World Cup footballers by TikTok hashtag views and creator-mention affinity

Player	Nation	TikTok views	Videos	Creators naming them
Lionel Messi	Argentina	700.7B	24.1M	1,705
Cristiano Ronaldo	Portugal	580.6B	21.4M	1,112
Neymar Jr	Brazil	401.6B	16.7M	911
Kylian Mbappé	France	269.0B	6.2M	102
Lamine Yamal	Spain	147.5B	4.4M	32
Erling Haaland	Norway	59.4B	1.2M	51
Vinícius Jr	Brazil	54.4B	1.4M	158
Jude Bellingham	England	51.2B	1.3M	42
Mohamed Salah	Egypt	37.0B	1.7M	316
Ousmane Dembélé	France	23.8B	0.6M	6
Endrick	Brazil	15.6B	0.3M	3
Antoine Griezmann	France	12.7B	0.3M	20

Players ranked by cumulative TikTok hashtag views, with the count of creators naming them in their bio/handle. June 2026. Source: Crawlora tiktok_challenge + creators_search.

Why TikTok's football map looks nothing like Instagram's

The order keeps surprising because TikTok is a fundamentally different machine. On Instagram, reach follows the follower graph — big accounts get seen, so Ronaldo's 665M compounds. On TikTok, follower count isn't a ranking factor at all; the algorithm serves clips by engagement, so an unknown's celebration can out-travel a superstar's polished post. That is why the platform's leaderboard skews young, Brazilian and meme-friendly — Yamal's "everyday moments", Vini's touchline dances remixed into multilingual edits, Neymar's everything-at-once persona — rather than toward stature.

It is also why this is the World Cup that institutionalised it: FIFA named TikTok its official video content partner for 2026, with a 30-creator correspondent programme and in-app match hubs. The conversation we measured isn't a sideshow to the tournament — increasingly, for a billion young fans, it is the tournament.

What it tells us

Fans & journalists: 'the most popular footballer' depends entirely on the platform. Ronaldo wins Instagram followers; Messi wins TikTok views; Lamine Yamal wins TikTok followers. There is no single leaderboard — there are three, and they disagree.
The generation has turned. A 17-year-old out-hashtags every veteran except the four all-time giants, and is the most-followed footballer on the app. Watch the gap between his reach (huge) and his fan-identity depth (still tiny) close over the next few years.
Nations live through their players. Brazil and England under-index on the country tag because their TikTok energy is carried by Neymar, Vini and Bellingham. Argentina and Morocco over-index because the nation itself became the story (2022 title; 2022 semi-final run).
Data buyers: 'who's followed' and 'who's talked about' are different products. Follower counts are a vanity census; hashtag views and creator co-mentions measure cultural footprint — which is what actually predicts a viral tournament moment.

When the first ball is kicked in June 2026, the grass tournament and the phone tournament will run in parallel — and they will crown different winners. On the pitch, anyone can win. On TikTok, the table is already drawn: Messi on top, a teenager rising fast, and a billion creators arguing about both.

Sources

Sources

Frequently asked questions

Who is the most popular footballer on TikTok in 2026?

It depends what you measure. By hashtag views, #messi leads with 700.7 billion, ahead of #ronaldo (580.6B) and #neymar (401.6B). By follower count, the most-followed footballer on TikTok is 17-year-old Lamine Yamal (38.3M), ahead of both Ronaldo and Messi. And on Instagram, Ronaldo dominates with about 665 million followers. There is no single leaderboard — the three platforms and metrics crown different winners.

Does Messi or Ronaldo dominate TikTok?

Messi, by the size of the conversation. The #messi hashtag has roughly 700.7 billion views to #ronaldo's 580.6 billion — about 120 billion more — even though Ronaldo has more Instagram followers (665M vs 506M). TikTok talks more about Messi; Instagram follows more of Ronaldo.

Which World Cup nation is biggest on TikTok?

Among football-specific national hashtags, #argentina leads with about 631 billion views (fitting for the reigning champions), ahead of France (576B), and well ahead of Brazil and England (~197B each). Morocco overperforms at 163B, a lingering effect of its 2022 semi-final run. We exclude #mexico (1.06 trillion) and #japan (355B) because those tags are dominated by non-football culture.

How popular is Lamine Yamal on TikTok?

Very — disproportionately so for his age. At 17, #lamineyamal has 147.5 billion hashtag views, fifth among all footballers and ahead of Haaland, Bellingham, Salah and Modrić, and he is the single most-followed footballer on TikTok at 38.3 million. But his fan-identity depth is still small: only about 32 creators name him in their bio, versus 1,705 for Messi — he is watched, not yet woven in.

How was the TikTok World Cup study measured?

Two ways. TikTok hashtag view and video counts came live from Crawlora's tiktok_challenge endpoint (platform-wide, cumulative) for a curated football-specific tag per player and nation. The creator-mention affinity and the co-mention social graph came from Crawlora's creators_search dataset of 3.33 million creators (a bio is present on 92.5%), counting how many name each entity and how often two are named together. Figures are aggregate-only and the dataset is open and reproducible.

Why is TikTok's football ranking so different from Instagram's?

Because TikTok's algorithm serves clips by engagement, not by follower count — follower count is not a ranking factor — so an unknown's celebration can out-travel a superstar's polished post. That structurally favours young, flair, meme-friendly players (Lamine Yamal, Neymar, Vinícius) over all-platform giants, which is why TikTok's leaderboard looks nothing like Instagram's follower order.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Product Hunt Trends 2013–2026: How AI Agents Took Over Startup Launches

Tony Wang — Mon, 29 Jun 2026 06:01:58 +0000

Key takeaways

We read every Product Hunt annual and monthly leaderboard from 2013 to 2026 — one of the few public, longitudinal records of what builders actually shipped and what other builders upvoted.
AI didn't trend — it took over: on a deep ~190-product-per-year sample, the 'Artificial Intelligence' tag went from under 4% of the board before 2022 to ~70% by 2025, with the post-ChatGPT 2022→2023 jump the sharpest single-year move in the dataset.
The deepest shift is the verb: from tools you operate to agents that do the job for you — 'the AI to-do list that does itself', 'a team of AI agents that runs your stores'.
Trends now churn monthly inside a stable AI-agent theme — and winning a launch isn't lasting: of 28 era-defining winners, ~3 in 10 are dead or pivoted away, and the survivors are trend-independent utilities and infrastructure.

The headline numbers

AI's share of the board: under 4% before 2022 → ~12% (2022) → 42% (2023) → ~70% by 2025 (deep sample of ~190 products/year); by the annual Top 10 alone, 0 → 9 of 10.

2023 is the inflection year — AI's board share roughly quadrupled in a single post-ChatGPT year, the sharpest move in the dataset.

In 2026, monthly product churn is ~100% (almost no launch repeats month to month) — but the theme stays AI agents; only the sub-flavor rotates.

28 era-defining winners re-checked: roughly 3 in 10 are dead or pivoted away from the idea that won, and half the 2020–22 cohort is gone.

Most "state of tech" takes are vibes. Product Hunt's leaderboard is one of the few public, longitudinal records of what people actually shipped and what other builders actually upvoted — every day since 2013. So instead of guessing, we read the boards: every annual leaderboard from 2013 to 2026, plus a stack of monthly ones, and tallied what each year was about.

Ryan Hoover launched Product Hunt on November 6, 2013 as a Thanksgiving-break email list, so the board is a ~12.5-year record — and that's the more dramatic span, because the entire "AI everything" earthquake is compressed into its back third.

A method note, with some irony: we pulled the leaderboards programmatically through a structured web-data API. Trying to read the same pages in a plain browser hits a Cloudflare "verifying you are not a bot" wall — which is exactly why reliable, structured access beats hand-scraping HTML. (Caveats are at the end; the short version: the annual board is a curated top-~20, scores are cumulative upvotes, and 2026 is year-to-date.)

Monitor Product Hunt as structured data

Want to track Product Hunt launches, categories, and competitor traction as clean JSON instead of fighting the Cloudflare wall? It's the same tooling that built this study — Crawlora's Product Hunt API turns blocked leaderboard pages into normalized API responses (leaderboards, categories, makers, reviews, and more).

Here's the whole twelve years in one shot — Product Hunt's five leading product categories, racing for the top each year. Watch Artificial Intelligence climb from a sliver to first place, overtaking Design Tools and Productivity:

From mobile apps to an AI monoculture: the 12-year arc

The board's center of gravity moved in clear eras:

2013–2014 · Mobile app discovery. Almost everything is tagged generically "Tech," with a heavy iOS lean. 2013's top product is Sunrise (a calendar app later bought by Microsoft); 2014's #1 is literally Product Hunt's own iOS app. The platform is a place to find apps.
2015–2016 · The maker era. Curated directories and founder toolkits dominate — 2015's #1 is Startup Stash (400 startup tools), 2016's is Startup Pitch Decks. AI first pokes through as novelty (Prisma, Amazon Go).
2017–2019 · Design tools & productivity. The two most durable topics in the whole dataset. AutoDraw (2017), Notion 2.0 and remove.bg (2018), Checklist Design (2019). AI is still toys (AutoDraw, the "Not Hotdog" meme app, Google Duplex).
2020 · The remote-work pivot. HEY tops a board full of video-chat and remote tooling; the community's Golden Kitty Product of the Year is Clubhouse.
2021 · The no-code peak — the calm before the storm. 2021's #1 is Cal.com, surrounded by no-code site builders. AI products in the Top 10: zero.
2022–2023 · The AI spark, then the breakout. ChatGPT debuts on the 2022 board with the single highest score; by 2023 "Artificial Intelligence" is the #1 topic and the Golden Kitty goes to ChatGPT itself.
2024–2026 · The agent era. 2024's #1 is Wordware — a tool for building AI agents. By 2025, 9 of the Top 10 are AI; in 2026 the incumbents arrive (Anthropic's Cowork, ChatGPT Health) and the unit of competition becomes the agent, not the app.

Year	Most-upvoted #1	Top topics
2013*	Sunrise (calendar)	Tech, iOS
2014	Product Hunt for iOS	Tech, iOS, Web App
2015	Startup Stash	Web App, Design Tools
2016	Startup Pitch Decks	Web App, iOS, Tech
2017	AutoDraw (Google)	Tech, Design Tools, Dev Tools
2018	Notion 2.0	Design Tools, Productivity
2019	Checklist Design	Design Tools, UX, Productivity
2020	HEY (email)	Productivity, Design Tools
2021	Cal.com	Productivity, Design Tools
2022	ChatGPT (top score)	Productivity, AI, SaaS
2023	Chat by Copy.ai	AI, Productivity, Web3
2024	Wordware	AI, Productivity, Dev Tools
2025	Screen Studio 3.0	AI, Productivity, Design
2026†	Cowork (Anthropic)	AI, Productivity, Dev Tools

* 2013 is a partial first ~2 months. † 2026 is year-to-date.

AI didn't trend — it took over

Watch the whole board's category mix reshape itself. Each band below is one of Product Hunt's five leading product categories (deep sample, ~180–200 products/year); Design Tools and Productivity own the early years, then the orange Artificial Intelligence band — barely a sliver before 2022 — swells into the widest by 2024:

Plot AI against Productivity — the most durable old-guard category — as a share of each year's ~190 top products, and you can watch the handover happen. Productivity is the steady incumbent (~30–47%); AI goes from a rounding error to dominant, crossing it in 2023:

The spark was the back half of 2022, when ChatGPT launched and immediately posted the top score on the board. From there AI's share of the board roughly quadrupled in a single year — ~12% in 2022 to ~42% in 2023 — then climbed to ~53% (2024) and ~70% (2025). Just as telling is how the flavor of AI kept mutating: chat/LLM wrappers (2023) → AI agents and vertical apps (2024) → agentic workflows, AI dev-teams and "vibe coding" (2025) → incumbent agents and MCP-native tooling (2026). The category didn't just grow; it kept reinventing what "an AI product" even is.

2026 so far

The frontier shifted from startups to incumbents shipping agents — Anthropic's Cowork ("turn Claude into your digital coworker"), ChatGPT Health, Gmail in the Gemini era — plus two rising sub-themes: MCP-native tooling (turning any site or app into an agent tool) and GEO (getting your brand cited by AI assistants).

A single month, four eras

Zoom from years to months and a paradox appears. At the product level, the monthly Top set turns over almost completely — across the six monthly boards of 2026 so far, essentially no individual product repeats in the Top 8. But at the theme level, almost nothing changes: every 2026 month is ~100% AI/agent products. What recurs is brand families, not entries — Anthropic's Claude charts nearly every month via a different launch; the "…Claw" agent family propagates across the year.

Take one month per era and the trajectory is stark:

So the thing that "changes fast" in 2026 isn't the category — it's the sub-flavor of AI, month to month: January was agent runtimes and code review; March, AI design and agent marketplaces; April, meeting-notes and evals; May, agent-commerce and voice-calling agents; June, fundraising and GTM agents. The trend clock compressed from a theme per year to a sub-theme per month.

The platform grew, then the clock sped up

Product Hunt itself scaled fast and then plateaued — which makes the acceleration of trends (not volume) the real story:

What actually changed in 12 years

From tools you use to agents that do it for you. 2013–2021 winners are utilities a human operates; 2024–2026 winners are described as employees — "the AI to-do list that does itself", "a team of AI agents that runs your stores". The verb moved from use to delegate.
Topic diversity collapsed into a monoculture. Through 2021 the top topics were a genuine mix (Design, Productivity, Dev Tools, No-Code, Hardware, Crypto). Since 2024, "AI" is on nearly everything; the other topics are just the domain the AI points at.
Durable winners gave way to franchise churn. Early hits stayed relevant for years; the 2026 board is version bumps and brand families cycling through the top spots monthly.
Each wave left a fingerprint, then receded — except one. Hardware, the maker boom, crypto/Web3, COVID/remote, no-code — each surged and faded. AI is the first wave that, instead of receding, became the substrate the next waves run on.

The takeaway for anyone shipping: the leaderboard rewards whatever frame is ascendant. In 2019 that was a polished design resource; in 2026 it's an autonomous agent with a clear "it does the job for you" promise. The same capability launched in the year's language outperforms one launched in last year's.

How the board itself changed — not just the topics

The topics flipped to AI, but three quieter shifts in the leaderboard's own mechanics are just as telling.

Winners now keep gaining votes after launch day. Comparing each year's winners' current upvotes to their launch-day score: early-era winners were launch-day spikes that bled votes afterward (ratio below 1); since 2025 they keep climbing (above 1) — the board rewards sustained momentum over a one-day burst.

Annual winners cluster in Q1. Across 2018–2025, a disproportionate share of year-topping products launched early in the year — February is the single peak and Q1 holds ~37% of winners. Early launches have the most time to accumulate the cumulative votes that decide annual rank.

AI didn't just grow — it fused with Productivity. Among each year's top winners, the number carrying both the AI and Productivity tags went from ~zero to 9 of 17 in 2025 — "AI + Productivity" became the board's defining combination, and the old Web3/blockchain pairings vanished after 2023.

Where are Product Hunt's winners now?

A leaderboard tells you what launched, not what lasted. So we ran the winners back through the same API: of the products that topped Product Hunt, how many are still alive today? We fetched the homepage of 28 era-defining winners and pulled each one's current traffic.

The catch — and it's the whole point — is that a domain returning a page is not a living product. Four "winners" still answer with 200 OK but serve something else entirely: Station's domain is now a Thai casino, Sunrise's is an SEO content farm, Rewind's is a generic AI-tools page, and Polywork's is a different site builder. Traffic estimators even keep serving the old title, so they'd happily score these as alive. Only fetching the live page and cross-checking the traffic — the same fetch-and-verify discipline behind our anti-bot index — reveals that the product is gone.

By that bar, here's how the 28 winners split:

Roughly seven in ten still run the product that won — but about three in ten are dead or have pivoted away from the thing that put them on the board. A #1 badge was no insurance: Sunrise (2013's beloved calendar, bought by Microsoft and shut down) and Wordware (2024's literal #1 — "$30M, the largest seed in YC history" — now replaced by a different product) both topped Product Hunt and are effectively gone.

And the deaths aren't evenly spread — they cluster hard in one era:

Put both dimensions together — which era each winner launched in, and where it ended up — and the whole story sits in one flow:

Half of the 2020–22 cohort is already gone — the pandemic-and-social-novelty winners (mmhmm, Polywork, Typedream, Rewind, plus a much-diminished Clubhouse). The survivors, in every era, are the trend-independent ones: single-purpose utilities (Workflowy, Coolors, Hunter, remove.bg) and infrastructure (Notion, Supabase, Cal.com, Resend). Winning a launch is a moment; outliving the trend that powered it is the hard part.

And outliving the trend still isn't the same as winning the market. Among the survivors, monthly traffic is brutally top-heavy:

Winning a Product Hunt launch gets you on the board; becoming Notion is a different game entirely.

How to use this before you launch

The leaderboard rewards trend-fit, but the data above says trend-fit and durability are different games. If you're timing a launch, treat these as the operating rules:

Don't launch as generic "AI" — launch as this month's sub-flavor. "An AI tool" is invisible on a board that's already ~100% AI. The launches that break out name the specific job that's cresting right now — code review, meeting notes, agent commerce, GTM — so match the sub-theme that's ascendant the month you ship.
Phrase the promise as delegation, not tooling. The winning verb moved from use to delegate: "does X for you," not "a tool for X." "The AI to-do list that does itself" beats "a smarter to-do list."
Use trend-fit for launch velocity, but ship a utility for survival. Trend-of-the-moment framing wins the day; the products still alive years later are single-purpose utilities and infrastructure (Notion, Supabase, Cal.com, remove.bg). Ride the wave to get seen — but build something people still need after it breaks.
Time it: track the monthly sub-theme rotation, and aim for Q1. The AI sub-flavor shifts roughly monthly, so launch into the one that's cresting. And if you're chasing the year-end lists, launch early — February is the single peak month for annual winners, because early launches have the longest to accumulate the cumulative votes that decide annual rank.
Treat Product Hunt as an awareness and positioning signal — not proof of retention or market leadership. A #1 badge is a moment of attention, not evidence of a durable business; among survivors, traffic is brutally top-heavy. Winning the launch and becoming Notion are different games.

How we did this (and the caveats)

We read Product Hunt's annual and monthly leaderboards directly via a structured API, then cross-checked the platform facts — launch date, launch volumes, and Golden Kitty winners — against public sources. Caveats worth stating plainly: the annual board is a curated top-~20, not an exhaustive vote ranking; scores are cumulative upvotes, so they're directional, not exact annual totals; 2026 is year-to-date; 2013 is a partial ~2 months; and "AI in the Top 10" counts are by listed topic, so borderline cases are judgment calls — but the direction is robust regardless. For the topic charts (the race, streamgraph, and AI-vs-Productivity line) we went deeper than the annual top-20: we aggregated all twelve monthly leaderboards per year and de-duplicated by product, giving a ~180–200-product sample each year (vs ~17 at the very top) — which is why the AI share here (e.g. ~70% in 2025) is a touch lower and far smoother than a top-of-board figure would be. The yearly endpoint's deep pagination is broken, so the survival, vote-accumulation, seasonality, and co-occurrence cuts use the top-~17/year set. For the "where are they now" check, we fetched each winner's homepage and pulled its current SimilarWeb traffic, counting a product as gone only when the live page no longer serves it — a domain that returns 200 with unrelated content (a parked page, a casino, a different product) is dead, not alive. That liveness snapshot is June 2026, and "gone" includes products that pivoted away from the winning idea, not only outright shutdowns.

The datasets behind the piece, at a glance:

Dataset	Sample size	Used for	Caveats
Deep monthly-aggregated sample	~180–200 products/year (all 12 monthly boards, de-duplicated)	The topic-category charts: race, streamgraph, AI-vs-Productivity line	Smoother and a touch lower than a top-of-board figure (AI is ~70% in 2025, not 90%); "AI" tagging is a judgment call on borderline products
Annual top boards	~17 products/year	Survival, vote-accumulation, Q1 seasonality, AI+Productivity co-occurrence	Curated top list, not a full vote ranking; cumulative-upvote scores; the deep-pagination endpoint is broken, so these cuts cap at the top ~17
Monthly boards (era sample)	Top 8 per sampled month	Sub-theme rotation and product-level churn	Sampled months, not every month; 2026 is year-to-date
Winner survival + traffic	28 era-defining winners	Liveness, pivots, and surviving-product traffic	Live homepage fetch + SimilarWeb (May–June 2026); a 200 with unrelated content counts as gone; "gone" includes pivots

If you're wondering how we read pages that block bots: that's the day job. Crawlora is a web-data API for AI agents and pipelines that returns normalized JSON for dozens of platforms, handles the anti-bot layer, and bills pay-on-success — you only pay when it actually returns your data. The same approach powers our anti-bot adoption study, the streaming fragmentation study, and the Google-vs-Bing-vs-Brave SERP study.

Sources

Frequently asked questions

When did Product Hunt launch?

Product Hunt launched on November 6, 2013, built by Ryan Hoover as a Thanksgiving-break email list. Its leaderboard is therefore about a 12.5-year record (2014–2026) — and the entire AI wave is compressed into the back third of it.

How have Product Hunt trends changed over time?

The board moved through clear eras: generic mobile apps (2013–14), startup and maker resource directories (2015–16), a design-tools and productivity golden age (2017–19), a remote-work and COVID pivot (2020), a no-code peak (2021), the AI spark and breakout (2022–23), and an AI-agent monoculture (2024–26).

When did AI take over Product Hunt?

2023 is the inflection year — the first year 'Artificial Intelligence' is the single most common topic in the annual Top 10. AI products in the Top 10 grew 0 → 3 → 5 → 7 → 9 across 2021 to 2025, sparked by ChatGPT's debut in late 2022.

Do Product Hunt trends change month to month?

At the product level, yes — in 2026 the monthly Top 8 turns over almost completely each month, with essentially no repeat products. But the theme is stable: nearly every entry is an AI agent. What rotates month to month is the sub-flavor — coding, then design, then meetings, then commerce, then fundraising.

What is trending on Product Hunt in 2026?

AI agents. Almost every top launch in 2026 is an autonomous agent that does a job for you rather than a tool you operate, so the dominant theme barely moves — but the sub-flavor rotates roughly monthly: agent runtimes and code review, then AI design and agent marketplaces, then meeting-notes and evals, then agent-commerce and voice agents, then fundraising and GTM agents, plus a rising MCP-native tool-layer cohort that turns any site or app into an agent tool.

How was this Product Hunt trend analysis done?

We read Product Hunt's annual and monthly leaderboards directly through a structured web-data API and cross-checked platform facts against public sources. The annual board is a curated top-~20, scores are cumulative upvotes, and 2026 is year-to-date, so the figures are directional rather than exact.

Do Product Hunt winners survive?

Not reliably. Re-checking 28 era-defining Product Hunt winners in June 2026, about 7 in 10 still run the product that won, but roughly 3 in 10 are dead or have pivoted away — and a #1 badge was no guarantee (2013's Sunrise and 2024's #1 Wordware are both effectively gone). The deaths cluster in the 2020–22 cohort (~50%), the pandemic and social-novelty bets; trend-independent utilities and infrastructure survive best.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

14% of the Web Is Actually Dead — But Not How You Think (We Scanned 10M Domains)

Tony Wang — Thu, 18 Jun 2026 01:33:38 +0000

Key takeaways

We probed 9,992,781 of the top 10 million domains in June 2026. 14.2% are genuinely dead — no DNS, no connection, nothing answers — not the 27.6% a naive crawl of the same list reports.
Nearly half of 'dead' was never dead. 8.9% of the top web (891,672 sites) answers but blocks an automated client (403/429/anti-bot), and another ~4% serves a 404 or 5xx from a live server. Naive crawls count all of that as death.
The genuinely dead web is mostly DNS that no longer resolves: 1,077,715 domains — 76% of all dead — have left DNS entirely. The rest refuse or reset the connection. A 404 page is not death; a missing DNS record is.
Death is uneven by TLD. .cn (33%), .info (28%), .in and .gov (26%), and .edu (22%) rot fastest — institutional and cheap-registration domains lead, echoing Pew's finding that government and reference pages suffer the worst link rot. .com sits near the 14% line.
This is not 'link rot' or 'dead internet theory.' We measure whether the domain itself still resolves and answers — a different question from broken links inside pages (Pew, Ahrefs) or AI-generated content flooding the web.

You have probably seen the stat: 27.6% of the web is dead. It comes from a 2024 crawl of the top 10 million domains, and it gets repeated because it is striking and a little bleak. We ran that study. And when we re-scanned the same 10-million-domain list in 2026 — this time separating a domain that is genuinely gone from one that is merely refusing a bot — the real number came out at 14.2%.

The web didn't suddenly heal. The original number was counting the wrong things. A naive crawler can't tell a dead domain from a live one hiding behind Cloudflare, and it counts a server that politely returns "404 Not Found" the same as one that never answers at all. Fix the classification and roughly half of "dead" turns out to be alive — it just wasn't talking to a bot. Here is the full picture, from 9,992,781 probed domains.

What the outcome of a 10-million-domain scan actually looks like

Every domain gets one of four labels. alive means it answered (a 2xx, or even a 404/5xx — the server is up). blocked means it answered but refused our automated client (a 403, 429, or anti-bot challenge). redirect means it bounced somewhere we couldn't resolve. dead means it never answered at all — no DNS record, or nothing accepts a connection.

Three-quarters of the top web is alive and answering. The interesting part is the bottom 23% — the slice everyone argues about — and how you split it.

The real number: 14% dead, not 27.6%

Same list, same scale, one difference: in 2026 we refuse to call a domain dead just because our bot couldn't read it. A genuinely dead domain fails early — DNS returns nothing, or the connection is refused. A live-but-defended domain fails late, with a 403 or a challenge page, which is a completely different signal. Counting honestly moves the headline from 27.6% to 14.2%.

Where do the missing ~13 points go? Almost all of it is two things a naive crawl mislabels:

8.9% (891,672 sites) answer but block bots. A 403, a 429, or a Cloudflare "Just a moment" challenge to a datacenter IP. These are some of the most alive sites on the web — they run active defenses precisely because people want their data.
~4% serve a 404 or 5xx from a live server. A "404 Not Found" or a "503 Service Unavailable" is proof the host answered. The original crawl counted them as dead; a server that returns an error is the opposite of gone.

The remainder is a 2024 measurement artifact: that crawl resolved each domain through a single DNS resolver, and a flaky lookup falsely marked resolvable domains dead. We now cross-check across resolvers before declaring a DNS failure.

Dead means unreachable, not 'returned an error.'

The whole correction rests on one rule: a server that answers anything — even a 404, a 500, or a 403 — is up, so it isn't dead. Only a domain that no DNS resolver can find, or that refuses and resets every connection, is dead. Most "dead web" counts skip this and inflate the number by half.

What a no-follow crawler gets wrong

The gap between 27.6% and 14.2% is largely a measurement choice: whether you follow redirects and read what the server actually says. A crawler that stops at the first response sees only 45.9% return a clean 200 and writes off the rest. Follow the redirects and read the bodies, and 71.9% are alive. Here is where every first response actually ends up:

The big rivers carry the point: a 301 is not a dead end — 87% of redirects resolve to a live page, and a 403 or 429 is a live site refusing a bot, not a corpse. The only response that reliably means dead is no response at all — and that single No response → Dead band is almost the entire dead web.

The genuinely dead web is mostly DNS that's gone

So what is the 14.2%? Overwhelmingly, it's domains that have left DNS entirely. Of the 1,414,788 genuinely dead domains, 1,077,715 — about 76% — no longer resolve to any IP at all. The registration lapsed, the zone was deleted, the project was abandoned. The rest refuse or reset every connection, or fail TLS to a host that is truly down. A dead domain almost never answers and errors — it simply isn't there.

This matters if you build anything that follows links or crawls a list: the failures you'll actually hit are split between "this domain is gone" (retry never helps) and "this site is blocking me" (a different request gets in). Treating them the same is the single most common way web-health numbers get inflated — and the most common way a scraper wastes a budget retrying domains that will never answer.

The famous dead

Aggregate percentages are abstract. So we sorted the genuinely-dead domains by popularity rank and went looking for names you'd recognise — and the graveyard is remarkable. The single highest-ranked dead domain in the entire top 10 million makes the point on its own.

At #568 sits fanlink.to, the music "smart-link" service artists and labels used for pre-save and streaming links. In March 2024 its parent — Eventbrite's ToneDen — lost control of the .to domain and never recovered it, instantly breaking millions of links sitting in artist bios, ads, and press releases.

Which raises the obvious question: how is a dead domain the 568th most popular on the web? Because the web never stopped knocking. Every un-updated link, embed, and bookmark keeps firing requests at an address that no longer answers — the rank is a fossil of past popularity. That is precisely why a popularity-ranked list is full of corpses at all.

Music & video

fanlink.to († 2024) — Music smart-links · ToneDen / Eventbrite. The single highest-ranked dead domain in the whole top 10M (#568). In March 2024 Eventbrite lost control of the .to domain overnight, instantly breaking millions of artists' pre-save and streaming links sitting in bios, ads, and press releases. Wayback
grooveshark.com († 2015) — Free music streaming · ~20M users. Forced shut by the major labels' copyright suit (willful infringement, ~$700M of exposure). The entire catalogue was wiped the day the settlement landed; a co-founder died months later at 28. Wayback
rdio.com († 2015) — Music subscription service. Bankrupt after burning ~$2M a month. Pandora bought the technology for $75M and shut the service down the day before the sale closed. Wayback
gfycat.com († 2023) — GIF host for Reddit & Discord · ~220M users. Bought by Snap in 2020, then switched off as a non-core asset — one of the largest single link-rot events ever, breaking millions of embedded GIFs across the web. Wayback
veoh.com († 2024) — Video-sharing site. Won a landmark DMCA case that helped protect every YouTube-style site, limped on for years under Japan's FC2, and finally went dark in November 2024. Wayback
metacafe.com († 2021) — Top-3 video site of 2006. One of YouTube's first serious rivals — it simply went offline one day in 2021 with no announcement at all. Wayback

The social web

del.icio.us († 2017) — Delicious · invented social bookmarking. The site that coined web-scale tagging. Passed through five owners (Yahoo → AVOS → Science → Delicious Media → Pinboard for $35,000) before going read-only. Wayback
dmoz.org († 2017) — The Open Directory · a human-curated map of the web. 91,000 volunteers cataloguing 3.8M sites — once a near-prerequisite for SEO, then made obsolete by Google's algorithm. Lives on as the community fork Curlie. Wayback
pipes.yahoo.com († 2015) — Yahoo Pipes · visual no-code data mashups. The “Zapier of 2007.” Killed in a Yahoo cost-cut; thousands of live RSS and data pipelines broke on the same day. Wayback
topsy.com († 2015) — The only full historical Twitter search. Indexed hundreds of billions of tweets back to 2006. Apple bought it for ~$200M and quietly switched it off two years later; the searchable archive simply vanished. Wayback
aviary.com († 2018) — Photo-editing SDK embedded in 7,000+ apps. Powered in-app photo editing across the mobile economy (10B edits). Adobe acquired it, folded the tech into Creative Cloud, then sunset the free SDK. Wayback

The developer web

s7.addthis.com († 2023) — Share buttons + tracking on 15M websites. Oracle bought it for the behavioural data, then killed it under GDPR pressure — a single shutdown darkened share widgets across millions of sites at once. Wayback
programmableweb.com († 2023) — The public directory of ~19,000 web APIs. The index of the “API economy” for 17 years. Salesforce / MuleSoft erased the whole thing with no archive. Wayback
securityfocus.com († 2021) — Home of the Bugtraq disclosure list (since 1993). The security world's noticeboard for nearly 30 years. Symantec → Broadcom → Accenture let it freeze; the Bugtraq archive survives only at seclists.org. Wayback
opensolaris.org († 2013) — Sun's open-source operating system. Oracle froze it the moment it bought Sun and pulled the domain in 2013. The community kept the code alive as the illumos fork. Wayback
sorbs.net († 2024) — Spam blocklist covering 512M IP addresses. A DNS blocklist that mail servers queried for over two decades. Proofpoint pulled the plug in 2024; servers worldwide still query a list that no longer answers. Wayback

Government & institutions

patft.uspto.gov († 2022) — US patent full-text search (1790–present). Retired for a new search tool — breaking decades of direct patent links embedded in academic papers, legal briefs, and analysis tools. Wayback
petitions.whitehouse.gov († 2021) — Obama's “We the People” e-petitions. A petition once topped a million signatures. The platform was quietly discontinued on Inauguration Day 2021 and never revived. Wayback
weblogs.com († ~2009) — Dave Winer's blog-ping server · the early blogosphere's heartbeat. Every new blog post once pinged this host; VeriSign paid $2.3M for it. It faded after 2009 — yet old WordPress installs still ping the dead address to this day. Wayback
europa.eu.int († 2006) — The European Union's original web address. The canonical home of EU law and institutions for over a decade. Migrated to europa.eu on Europe Day 2006, stranding a generation of links. Wayback

Read those twenty obituaries back-to-back and one cause of death stands out: being acquired. Seven of the twenty were bought by a bigger company that then switched them off — Snap killed Gfycat, Apple killed Topsy, Oracle killed both AddThis and OpenSolaris, Adobe killed Aviary, Salesforce killed ProgrammableWeb, Broadcom let SecurityFocus rot. "Acqui-killed" beats bankruptcy, lawsuits, and neglect combined.

Twenty headliners can't show the shape of the whole graveyard. So we widened the lens — pulling ~100 widely-recognised, verifiable shutdowns (from this scan's dead domains and the public record), dating each to the year its service ended and sorting them into six corners of the web. Stacked by year, two decades of the dying web look like this:

Two things stand out. Social platforms and developer tools are the bulk of the dead web — the social graveyard (Friendster, Orkut, Bebo, Google+, Path, Yik Yak, Ello, Digg…) and the dev-tools column (Google Code, Parse, Google Wave, Gitorious, Sunrise, Mailbox…) are dead even, and together they're more than half of everything here. And the deaths cluster: a first swell in 2012–2017 as the Web 2.0 and check-in/anonymous-app generation collapsed, then a second from 2020 as pandemic-era and big-tech bets were cut (Quibi, Mixer, CNN+, Stadia, Google Play Music). Before 2009 the stream barely exists — most of the web simply wasn't old enough to have died yet.

One honest caveat, and the reason we re-checked every domain by hand: a dead domain is not always a dead thing. Some only look dead because the service rebranded or moved — money.yandex.ru became YooMoney, the old suicidepreventionlifeline.org host gave way to 988lifeline.org, the EU's europa.eu.int simply became europa.eu. We re-probed every domain above against live DNS in June 2026 and dropped the false positives (nrel.gov and angelfire.com still resolve fine). What remains genuinely no longer answers.

Death is uneven: which TLDs rot fastest

Dead rate is not evenly spread. Split the 10 million by top-level domain and a clear gradient appears — cheap-registration and institutional TLDs rot far faster than the .com baseline.

The standouts tell two stories. .cn, .info, and .in lead because they are cheap and heavily registered for short-lived or speculative sites that lapse quickly. But .gov (26%) and .edu (22%) near the top is the more striking finding: institutional domains rot badly because content is reorganized, departments are dissolved, and old project sites are simply switched off — exactly the digital decay Pew Research documented in 2024, where government and reference pages had some of the worst link rot. The web's most authoritative corners are some of its least permanent.

The geography of the dead web

Group the country-code domains by country and the decay draws a map. The emerging-market registration booms of the last decade left the biggest graveyards — China's .cn leads at a third dead — while German-speaking Europe runs the most durable web on earth.

A domain in China's .cn space is more than four times as likely to be dead as one in Germany's .de. Fast, cheap, speculative registration — and, for .cn, a churn-heavy market behind the Great Firewall — leaves more abandoned domains behind; the mature, costlier-to-register German-speaking TLDs barely rot at all.

What the top 10 million is even made of

For context, here's the shape of the corpus itself. .com is not just first — it is nearly half of the entire top 10 million, larger than every country-code and new-gTLD combined.

Two details worth flagging: .io (3.6%) has quietly become the third-largest TLD on the popular web — the developer/startup default — and the AI-era .ai (0.30%, ~30,000 domains) has already overtaken established country domains like .fi, .no, and .tw in the top 10 million.

The dead web is the long tail nobody visits

Death is not spread evenly through the ranking. Split the 10 million by popularity and the dead rate climbs more than 20× — from 0.8% in the top 1,000 to 16.1% past rank 5 million. blocked runs the other way: the most-trafficked sites wall bots hardest, then the defenses thin out down the tail.

That gradient reframes the headline. The 14% is real by domain count — but those dead domains are almost all in the part of the web nobody visits. 99.8% of dead domains sit below rank 100,000, and the popular top-100K — where the overwhelming majority of web traffic lives — is only 2.2% dead. Weighted by attention instead of raw count, the dead web nearly disappears:

"Dead web" is not "link rot" — and definitely not "dead internet theory"

Three different things get blurred together. Keeping them separate is the whole point:

This study (dead domains): does the domain still resolve and answer? We find 14.2% of the top 10M do not.
Link rot (Pew, Ahrefs): are the links inside living pages still good? Pew Research found 25% of pages from 2013–2023 are gone and 38% of 2013 pages have vanished; Ahrefs found 66.5% of tracked links have rotted. Those measure decay within the living web — a complement to this, not the same number.
Dead internet theory: the claim that AI-generated content and bots have displaced human activity online. That is about what's on the living web, not whether domains are reachable. It is a separate conversation, and conflating it with link rot is how bad statistics spread.

If you only remember one distinction: link rot is about the pages that are still up; the dead web is about the domains that aren't.

What this means if you're building a scraper or a data pipeline

The practical takeaway is the 8.9% blocked slice, because it is the part most likely to break your project. When a request fails, the reason dictates the fix, and they are nothing alike:

A dead domain (no DNS, refused) will never answer. Retrying, rotating proxies, or switching to a browser does nothing. Drop it and move on.
A blocked domain is alive and reachable — it just refused your client. A matched browser TLS/JA3 fingerprint or a residential IP gets in where a datacenter bot gets a 403. This is a transport problem, not a dead site.

This isn't theoretical. Probing every domain a second time with a real Chrome TLS/JA3 fingerprint recovered ~72,000 of the ~890,000 sites the polite bot was blocked from — enough to pull the blocked rate from 8.9% down to 8.2%. Every one of those is a live site reachable with the right client, not a dead end.

The blocked web is the web you actually want.

We cross-checked a sample of these results against Similarweb traffic, and the blocked sites are by far the valuable ones. The blocked domains in our top-ranked sample pull a median of roughly 150 million monthly visits — Reddit (4.4 billion), Canva (975 million), Quora (313 million), Claude.ai (952 million). The dead ones record under 5,000 visits each, and most register zero — a four-to-five-orders-of-magnitude gap. Sites run a wall precisely because their data is worth taking, so the 8.9% blocked slice isn't noise; it is the most valuable 8.9% of the web.

Naive crawlers can't tell these apart, so they either give up on reachable sites or burn a budget retrying gone ones. The cost-efficient pattern is to escalate only as far as a site forces you to — which is exactly how Crawlora's anti-bot unblocker works, and why it bills on success rather than per attempt. If you want to know which bucket a specific URL is in before you build, the free anti-bot checker tells you in about 30 seconds, and our companion Anti-Bot Adoption Index measures how much of the live web runs a wall at all.

Two more things the scan turned up

The web is a maze of redirects. Only 69% of domains serve their final page directly; 31% bounce through at least one redirect — and a stubborn sliver loops until our 10-hop cap. That is exactly why a crawler that doesn't follow redirects sees a web that looks half-broken.

The dead web is stuck on HTTP. A decade into the HTTPS transition, the living web is ~78% encrypted — but dead and bot-blocked domains are barely half, abandoned before they ever got a certificate.

How we measured it

No magic — a deliberately simple, reproducible probe, run at 10-million scale.

The list. The full top 10 million domains (a DomCop/Tranco-style popularity ranking). We reached 9,992,781 of them — 99.95% coverage.

The probe. Each domain is fetched HTTPS-first from a datacenter IP, following redirects, with a short timeout and a cross-resolver DNS retry before any "DNS failed" verdict. We never submit a form, solve a CAPTCHA, log in, or fetch anything behind a wall. Every domain is probed twice — once as an honest bot, and once as a browser-like client with a real Chrome TLS/JA3 fingerprint — so we can separate "nobody's home" from "the bot wasn't let in."

The classification. A final 2xx, or a served 404/5xx (the host answered), is alive. A 403/429 or anti-bot challenge is blocked. A 3xx we can't resolve is redirect. Only no DNS, a refused/reset connection, or nothing accepting a connection is dead. That single rule — a server that answers anything is up — is the entire difference between 14.2% and 27.6%.

Limits. This is homepage-level reachability from a datacenter vantage, so it is a lower bound: a domain that blocks a datacenter bot may open for a residential browser, and a deep page can be deader (or more defended) than the homepage. Snapshot: June 2026. The full per-domain dataset — every domain, every arm — is open, and the live, searchable version is the Dead-Web Index.

Sources

Frequently asked questions

How many of the world's top websites are dead?

14.2% of the top 10 million domains are genuinely dead — about 1.41 million sites that no longer resolve in DNS or refuse every connection. That is far below the often-quoted 27.6%, which counted anti-bot blocks and answered errors as death.

What's the difference between a dead website and a blocked one?

A dead site never answers — no DNS record, or nothing accepts a TCP connection. A blocked site is alive and answering, it just refuses an automated client (a 403, 429, or anti-bot challenge). 8.9% of the top web — 891,672 sites — is blocked, not dead, a distinction naive crawlers miss.

Is the dead web the same as the dead internet theory?

No. The dead internet theory is a claim that AI-generated content and bots have replaced human activity on the living web. This study measures the opposite, concrete thing: how many domains have gone completely dark and unreachable — DNS gone, connection refused, server gone.

Why is this lower than the 27.6% dead-web figure?

Earlier top-10M crawls counted three non-dead things as dead: anti-bot 403/429 blocks, 404/5xx pages served by a live server, and domains a single flaky DNS resolver failed to look up. Classifying honestly — dead means genuinely unreachable — brings the real figure to 14.2%.

Which TLD has the most dead domains?

.cn has the highest death rate among common TLDs at 33%. Institutional TLDs like .gov (26%) and .edu (22%) also rank high — matching Pew Research's finding that government and reference pages suffer the worst link rot.

Why does a site look dead to a scraper but load fine in my browser?

Anti-bot systems serve a 403 or a challenge to a datacenter IP while letting a real browser through. A matched browser TLS/JA3 fingerprint reaches the site where a naive bot is blocked — which is why this study probes every domain twice, as a polite bot and as a browser-like client.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Why Reddit Blocked Unauthenticated JSON in 2026 (and How to Still Get Reddit Data)

Tony Wang — Mon, 15 Jun 2026 05:43:49 +0000

Key takeaways

On May 28, 2026, Reddit announced it is deprecating unauthenticated .json endpoints — within days, appending .json to a URL started returning 403, silently breaking most open-source Reddit scrapers.
The real driver is AI and money: Reddit's two decades of human conversation became a licensed AI-training asset (~$130M in 2024 from deals with Google and OpenAI), and free scraping undercut it — so Reddit is gating the data and suing those who take it without paying.
Reddit's stated reason is scraping 'without accountability,' bot and agentic abuse, and a clarified Rule 8; it is steering developers to authenticated access and Devvit — and has flagged RSS as the next surface to close.
You can still get public Reddit data compliantly — the official (paid) API, authenticated access, or a managed API that keeps the access path working and returns normalized JSON — but the free append-.json era is over.

For years, the simplest way to get structured data out of Reddit was a trick everyone knew: append .json to any Reddit URL and get clean JSON back — no API key, no OAuth, no account. It quietly powered most open-source Reddit scrapers, research scripts, bots, and data pipelines.

That door is now closed. On May 28, 2026, Reddit posted Protecting communities from scrapers and platform abuse to r/modnews, announcing it would shut down unauthenticated .json access. Within days, requests started coming back 403 Forbidden — with no deprecation window. If your scraper "still runs" but returns nothing, this is why.

This post explains why Reddit did it — the answer is mostly AI and money — and the compliant ways to still get Reddit data in 2026.

What actually broke

In Reddit's own words: "Deprecating unauthenticated JSON access: We'll also be shutting down unauthenticated .json endpoints. These endpoints can be used to scrape Reddit without accountability. Logged-in and authenticated access won't be impacted."

So:

Anonymous .json requests now 403. https://www.reddit.com/r/<sub>/top.json and friends no longer return data without authentication.
It fails silently in a lot of tools. Many scrapers get a 403 (or an empty/redirect response) but appear to "succeed," so pipelines quietly go dark instead of erroring loudly.
Authenticated access still works. Logged-in sessions and the official OAuth API are unaffected — that is the entire point of the change.
RSS is next. In the same post Reddit called RSS "another common surface for scraping," so feed-based access is on notice too.

Why Reddit did it

The technical change is small. The motivation behind it is the bigger story — and yes, it is largely about AI chatbots and bot traffic.

Reddit's data became an AI goldmine — and a product

Reddit is two decades of real human questions, answers, and opinions — exactly the text that makes large language models useful, and one of the most-cited sources in AI answers. Once that became obvious, Reddit turned its archive into a licensed product:

A ~$60M/year licensing deal with Google (February 2024) to train Gemini on Reddit data.
A licensing deal with OpenAI (May 2024) for ChatGPT.
~$130M in data-licensing revenue in 2024 — roughly 10% of Reddit's total revenue.

When the data is the product, the free append-.json endpoint is a leak: it let anyone — especially AI companies — take the same data for nothing, undercutting the paid deals.

AI bots were taking it for free — "without accountability"

This is the part most people's instinct gets right. The explosion of AI training crawlers and live "grounding" agents (assistants that fetch Reddit threads at answer time) created enormous automated traffic against the exact endpoints that required no identity. Reddit's framing names it directly: "large-scale scraping, spam networks, agentic account creation, and automated abuse." The unauthenticated .json route was the anonymous front door for all of it — data taken with no key to rate-limit, bill, or ban.

So Reddit started enforcing — in court

Killing .json is the technical half of a broader campaign:

Reddit sued Anthropic (June 2025), alleging its bots crawled Reddit 100,000+ times and bypassed robots.txt after declining to license.
Reddit then sued Perplexity and three scraping firms — SerpApi, Oxylabs, and AWM Proxy (October 2025).
Reddit blocked the Internet Archive's Wayback Machine (August 2025) over AI-scraping concerns.

Cutting off anonymous .json is how you enforce "license it or don't take it" at the protocol level.

It's part of the bigger "closing web"

Reddit is the highest-profile example of a wider shift: as AI made web data commercially valuable, the open, anonymous, append-.json web is closing. Sites are gating and monetizing data, Cloudflare now blocks AI crawlers by default for many customers, and "pay-per-crawl" is becoming real. The era of casual anonymous public-data access is ending.

Why your scraper gets 403 now (it is not your credentials)

Teams hitting this assume it is an auth or rate-limit bug. It usually is not. Reddit's 2026 enforcement also leans on:

TLS fingerprinting — generic clients (requests, wget, default curl) are identified by their TLS handshake and blocked, even with perfect headers.
IP reputation — datacenter and cloud IPs (GitHub Actions, Vercel, common hosts) are heavily flagged; the same request often works from a residential browser and 403s from a server.
No anonymous fallback — the .json path that used to absorb all this is gone.

That is why "add a User-Agent" or "back off the rate" no longer fixes it — the block is at the access-policy and fingerprint layer, not the request rate.

How to get Reddit data in 2026 (compliant options)

The free anonymous path is over, but public Reddit data is still reachable through sanctioned routes. Ranked:

1. The official Reddit Data API / Devvit

Reddit points developers to its authenticated Data API (OAuth) and the Devvit developer platform — the sanctioned path:

Free for non-commercial use, capped at ~100 requests/minute.
Commercial access runs about $0.24 per 1,000 requests; enterprise agreements start near $12,000/year.

Best when you can register an app, do the OAuth dance, and your use fits Reddit's terms.

2. Authenticated / session-based access

A logged-in browser session (cookies, a real browser via Playwright) still works, because authenticated access is unaffected. It is viable for small, careful jobs — but it is fragile (sessions expire, fingerprints get flagged) and you own all the maintenance and the terms-of-service risk.

3. A managed Reddit API (Crawlora)

If you want structured Reddit data without maintaining auth, proxies, and fingerprints — or rewriting your scraper every time Reddit changes the rules — a managed API does that for you. Crawlora's Reddit API returns normalized JSON for search, posts, comment threads, and subreddit feeds from one key, and maintains the access path as Reddit tightens it:

curl -G "https://api.crawlora.net/api/v1/reddit/subreddit/webdev/posts" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  --data-urlencode "sort=hot" \
  --data-urlencode "limit=25"

import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/reddit/search",
    headers={"x-api-key": "YOUR_API_KEY"},
    params={"q": "web scraping", "sort": "top", "limit": 25},
)
for post in resp.json()["data"]["posts"]:
    print(post["score"], post["subreddit"], post["title"])

You get posts, comments, and feeds as clean JSON, and you stop chasing Reddit's changes — that is the trade you are buying.

A note on compliance

Reddit's updated Data API terms and Rule 8 now explicitly cover automated abuse and unauthorized scraping, and the May 2026 change makes Reddit's stance clear. Whatever route you choose:

Collect only public posts, comments, and subreddits — never private, quarantined, or personal data.
Treat usernames and comment text as personal data (GDPR/CCPA) — minimize what you store and have a lawful basis, especially for AI-training use.
Prefer the official API or a licensed/managed path, and review Reddit's terms and your local law before commercial or AI use.

This is not legal advice — see Is web scraping legal in 2026? for the public-vs-personal-data detail.

Sources

Sources

Where this fits

The append-.json era is over, but Reddit remains one of the richest sources for community research, brand and product sentiment, and grounding data for AI. For the practical how-to (search, posts, comments, subreddit feeds, pagination), see how to scrape Reddit in 2026; to feed threads into a retrieval pipeline or agent, see the MCP integration and the AI-agent web data workflow.

Try it first, free: test the endpoint in the Playground, read the schema in the API docs, and review credit costs on the pricing page.

Frequently asked questions

Why did Reddit block unauthenticated .json endpoints?

On May 28, 2026 Reddit announced it was deprecating unauthenticated .json access to stop scraping 'without accountability' and curb bot and agentic abuse. The bigger driver is commercial: Reddit's data is now a licensed AI-training asset (deals with Google and OpenAI worth ~$130M in 2024), and the free .json path let anyone — especially AI companies — take that data without paying.

Are Reddit .json URLs still working in 2026?

No. Since late May 2026, appending .json to a Reddit URL returns 403 Forbidden for unauthenticated requests. Logged-in sessions and the official OAuth API still work, and Reddit has flagged RSS as the next surface it may close.

Why does my Reddit scraper get 403 even with a User-Agent?

Because the block is no longer about rate or headers. Reddit uses TLS fingerprinting and IP-reputation checks, so generic clients (requests, wget, default curl) and datacenter or cloud IPs get 403 even with a valid User-Agent. The anonymous .json fallback that used to absorb this is gone.

What is the official way to get Reddit data now?

Reddit's authenticated Data API (OAuth) and the Devvit developer platform. It is free for non-commercial use at about 100 requests/minute; commercial access is roughly $0.24 per 1,000 requests, with enterprise agreements starting near $12,000/year.

Is scraping Reddit legal or allowed in 2026?

Reddit's updated Rule 8 and Data API terms restrict unauthorized scraping. Public data is generally accessible, but collect only public content, treat usernames and comments as personal data, and prefer the official API or a licensed/managed path — review Reddit's terms and your local law before commercial or AI use. This is not legal advice.

How can I still get Reddit data without maintaining a scraper?

A managed API like Crawlora returns normalized JSON for Reddit search, posts, comment threads, and subreddit feeds from one key, and maintains the access path as Reddit tightens it — so you avoid auth, proxies, fingerprinting, and constant breakage.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Best AI Web Scraping Tools in 2026: How to Choose

Tony Wang — Sun, 14 Jun 2026 18:02:08 +0000

Key takeaways

‘AI web scraping’ means two different things: AI-native extractors that read an arbitrary page with an LLM, and structured data APIs that hand AI clean JSON for known sources. Pick by which problem you have.
AI-native extractors (Firecrawl, ScrapeGraphAI, Diffbot, Browse AI, Kadoa) shine on unknown, one-off pages — but in hands-on tests several still can't paginate natively and lack anti-blocking, and AI extraction runs roughly $0.004–$0.02 per page.
For repeatable pipelines that feed agents or RAG, a structured API like Crawlora returns documented JSON for supported platforms with no per-site parser, no token tax, and a hosted MCP server.
Nearly every tool has a free tier — so benchmark accuracy on YOUR pages and compare cost per successful result, not the vendor demo.

The best AI web scraping tool depends on the job: extracting fields from an arbitrary page you’ve never seen, or feeding an AI agent clean, structured data from known sources at scale. Those are different problems, and the tools that win each are different. This guide splits the landscape into categories, ranks the main options with real 2026 pricing and benchmark data, and shows how to compare them on cost.

"AI web scraping" is two categories, not one

AI-native extractors — point a model at a page and ask for fields in plain English. They handle unknown layouts and need no selectors, which is great for one-off or long-tail pages. The trade-offs: a per-page model cost, variable accuracy, and drift when sites change.
Structured data APIs — documented endpoints that return normalized JSON for known platforms (search, maps, marketplaces, social, finance). No parser to maintain, predictable schemas, no token tax, and easy to hand to an agent or a RAG pipeline. This is Crawlora’s category.

Most teams end up using both: a structured API for the platforms they hit constantly, and an AI-native extractor for the arbitrary pages in the tail.

What to evaluate

Accuracy on YOUR target pages — run a real sample, not the vendor demo.
Output: clean JSON you can store directly vs. text you must validate.
Anti-bot handling: proxies, browser rendering, and CAPTCHAs behind the tool, or your problem.
Pagination: does it follow ‘next page’ on its own, or stop at page one?
Repeatability: does it hold up on a schedule, or drift when the page changes?
Agent fit: REST + a hosted MCP server so agents can call it as a tool.
Cost per successful result at your volume — after retries and per-page model costs.
Compliance: public data only; review each source's terms.

The best AI web scraping tools in 2026

No single winner — match the tool to the problem. Pricing below is the published rate as of mid-2026; always re-check before you commit.

Tool	Category	Free tier	From (paid)	Best for
Crawlora	Structured API + hosted MCP	2,000 credits/mo	Credit-based	Repeatable pipelines + agents over known platforms
Firecrawl	Crawl-to-markdown for LLMs	500 one-time credits	Usage-based	Whole sites into LLM-ready text / RAG
ScrapeGraphAI	AI extraction (open source + cloud)	Open source	~$0.02/page (cloud)	Prompt-defined extraction with self-hosted control
Crawl4AI	AI crawler (open source)	Free (self-host)	$0 self-host	Developers who want a free, self-hosted AI crawler
Diffbot	AI extraction + Knowledge Graph	10,000 credits/mo	$299/mo	Article / product / entity extraction at scale
Browse AI	No-code AI robots	Yes	~$19/mo	Point-and-click monitoring of specific pages
Kadoa	No-code AI + self-healing	Yes	~$39/mo	Hands-off no-code extraction
Apify (AI Web Scraper)	Platform + AI Actor	Yes	$35 / 1,000 pages	Prebuilt scrapers and pipelines
Octoparse	No-code visual + AI assist	Yes	Tiered	Visual scraping for non-developers

1. Crawlora — structured JSON for agents, no parser

For data you call repeatedly, Crawlora returns normalized JSON by endpoint for dozens of platforms — search, maps, marketplaces, social, finance — so your model spends tokens on reasoning, not on cleaning HTML:

curl -s "https://api.crawlora.net/api/v1/google-search/search?keyword=ai%20web%20scraping&country=us" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Because it ships a hosted MCP server, an agent in Claude, Cursor, or your own stack can call these as tools directly, and there’s no HTML sent to a model (so no token tax). Free tier is 2,000 credits/month, no card. When to choose it: the sources you need are supported platforms, you want documented JSON without parser upkeep, and you’re feeding agents or RAG. The trade-off: for an arbitrary page on an unknown site, an AI-native extractor or a crawler fits better.

2. Firecrawl — whole sites to LLM-ready markdown

Firecrawl crawls a site and returns clean markdown or JSON built for LLMs — ideal for ingesting an entire docs site or blog into a RAG index. It’s the most adopted tool in this category (over 125,000 GitHub stars), with a 500-credit one-time free trial and AI extraction around $0.004 per page. A useful reality check: on Firecrawl’s own public 1,000-URL benchmark it reported ~87.7% scrape success and ~63.7% content truth-recall — even the leading tool doesn’t capture everything. When to choose it: turning arbitrary websites into text for retrieval. It’s a different shape from a structured platform API — you point it at URLs rather than calling typed endpoints.

3. ScrapeGraphAI — prompt-defined extraction, open source

ScrapeGraphAI uses LLMs to extract structured data from a page based on a prompt, with an open-source core and a managed cloud. It’s model-agnostic — OpenAI, Anthropic, Gemini, Azure, Groq, and local models via Ollama — so you control the engine. Cloud SmartScraper runs around $0.02 per page (a published comparison put it at roughly 5× Firecrawl’s per-page cost), the trade-off for prompt flexibility. When to choose it: developers who want AI extraction from arbitrary pages and either self-hosted control or a specific LLM.

4. Crawl4AI — free, self-hosted AI crawler

Crawl4AI is a fully open-source, self-hosted crawler built for LLM pipelines, with markdown output and adaptive crawling that auto-learns selectors — third-party testing found it cut crawl times by roughly 40% on structured sites. When to choose it: developers comfortable running their own infrastructure who want no per-page vendor fees. You own the proxies, scaling, and anti-bot handling.

5. Diffbot — AI extraction with a Knowledge Graph

Diffbot applies computer vision and NLP to classify and extract articles, products, and discussions semantically rather than by selector, and exposes a Knowledge Graph for entity context. It has the most generous free tier here (10,000 credits/month), with paid plans from $299/month (250K credits) to $899/month (1M credits). When to choose it: large-scale article/product extraction and entity data.

6. Browse AI, Kadoa & Parsera — no-code AI extractors

Browse AI records point-and-click “robots” that monitor specific pages (free tier; paid from about $19/month) and, unlike most, supports pagination. Kadoa turns natural-language workflows into self-healing extractors that adapt to layout changes (free tier; from about $39/month) but lacks strong anti-blocking out of the box. Parsera infers selectors from a URL with self-healing agents and stealth proxies (free tier; from about $25/month). When to choose them: business users monitoring a handful of pages without code. In Apify’s hands-on test, all of these adapted to layout changes — but several couldn’t paginate natively and struggled on protected sites.

7. Octoparse & Apify — visual scraping and prebuilt Actors

Octoparse is a visual, no-code scraper with AI assist for non-developers. Apify is a platform of prebuilt “Actors” with scheduling, storage, proxies, and an MCP server; its AI Web Scraper Actor extracts structured data from any URL with a plain-English prompt (AI tokens included) at $35 per 1,000 pages — though it doesn’t paginate natively yet. When to choose them: off-the-shelf scrapers and a pipeline platform rather than a typed API.

What the hands-on tests reveal

Two patterns show up across the 2026 reviews and benchmarks, and they matter more than any feature list:

AI removes selectors, not the hard part. These tools genuinely drop the need to write CSS/XPath — but in Apify’s four-tool test, several still couldn’t follow pagination on their own and lacked robust anti-blocking. Getting the page (proxies, rendering, CAPTCHAs) is still where most failures happen. See AI vs traditional web scraping for why fetching, not parsing, is the bottleneck.
No tool hits 100% recall. Even Firecrawl’s own benchmark lands near 88% scrape success — so whatever you pick, run a real sample of your pages and measure accuracy and cost per successful result, not the demo.

How to choose in four questions

Are you extracting from arbitrary unknown pages, or calling known platforms repeatedly?
Do you need clean JSON you can store directly, or text you’ll validate?
Will an agent call it — i.e. do you need REST plus a hosted MCP server?
What’s the cost per successful result at your volume, after retries and per-page model costs?

If you’re feeding agents or pipelines from supported platforms, a structured API like Crawlora fits; for whole sites into RAG, Firecrawl or Crawl4AI; for arbitrary one-off pages, an AI-native extractor. Many teams use both. Whatever you choose, collect only public data — see is web scraping legal in 2026.

Sources

Sources

Next steps

Try it first, free: turn any URL into clean Markdown with the Free Web Scraper — no signup, no API key.

Read AI vs traditional web scraping and web scraping for AI training data, see the AI Web Scraping API, connect the hosted MCP server, and test a call in the Playground. For the broader market, see how to choose a web scraping API.

Frequently asked questions

What is the best AI web scraping tool?

There is no single winner — it depends on the job. For repeatable pipelines and agents over known platforms, a structured data API like Crawlora fits; for whole sites into LLM-ready text, Firecrawl; for prompt-defined extraction from arbitrary pages, ScrapeGraphAI or Diffbot; for no-code monitoring of specific pages, Browse AI or Octoparse.

What does 'AI web scraping' actually mean?

Two things: AI-native extractors that read an arbitrary page with an LLM and return fields from a prompt, and structured data APIs that hand AI clean JSON for known sources. They solve different problems, and many teams use both.

Are AI web scrapers better than traditional scrapers?

Not universally. AI extraction adapts to unknown layouts without selectors, but costs more per page and can drift; traditional selectors are cheap and precise on stable pages; a structured API skips parsing entirely for supported platforms. See our AI vs traditional web scraping guide.

Is there a free AI web scraping tool?

Several offer free tiers or credits. Crawlora includes 2,000 credits per month with no card, and tools like ScrapeGraphAI are open source. Benchmark a few on your real target pages before committing.

Can AI web scraping feed an AI agent directly?

Yes, if the tool exposes a tool interface. Crawlora ships a hosted MCP server, so agents in Claude, Cursor, or your own stack can call its structured web-data endpoints as tools.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

How Paywalls Actually Work: The Engineering Behind Them

Tony Wang — Thu, 11 Jun 2026 12:12:16 +0000

A paywall is one of the more interesting engineering problems on the web, because the publisher has to satisfy two goals that pull in opposite directions. It needs Google to index the article so people can find it and click through — which means a search crawler has to see the full text. But it also needs to withhold that same text from a logged-out reader so there's a reason to subscribe. Reconciling "show the bot everything" with "show the human almost nothing," without getting penalized for it, is the whole game. How a publisher resolves that tension decides whether its paywall is a bank vault or a velvet rope you can step around.

This guide explains the machinery from an engineer's point of view: the kinds of paywall, where the content actually lives, the structured-data contract that lets publishers serve crawlers and readers different things on purpose, and why some of these walls are trivial to read past while others are effectively sealed.

Key takeaways

Paywalls come in four flavors — hard, soft/freemium, metered, and dynamic — and each is enforced differently.
The single most important fact is where the content is hidden: client-side paywalls ship the full article to the browser and then hide it (often readable), while server-side paywalls never send it (effectively not).
Publishers declare gated sections to Google with isAccessibleForFree JSON-LD and grant Googlebot full, IP-validated access — which is exactly why 'pretend to be Googlebot' sometimes works and is usually blocked.
Reading content behind a paywall is the highest-risk category of access (DMCA §1201, CFAA, terms of service). The defensible path is public data, official APIs, and the structured data publishers already expose.

What this guide is — and isn't

This is a technical explainer for engineers, SEOs, and publishers who want to understand the machinery. It is not a how-to for reading paid articles without paying. Bypassing a paywall to reach gated content is a real legal risk (covered below), and it is explicitly not what Crawlora is for — we build for public web data.

The four kinds of paywall

"Paywall" is a single word for several very different mechanisms. Knowing which one you're looking at tells you almost everything about how it behaves and how robust it is.

Type	What the reader gets	How it's enforced
Hard	Nothing without a subscription	The article body is withheld outright; you see a headline, a deck, and a subscribe prompt
Soft / freemium	Some articles free, some "premium"	A per-article flag decides whether the full body is served at all
Metered	N free articles per period	A counter (cookie, local storage, device fingerprint, or server-side account) tracks views and gates after the limit
Dynamic / propensity	Varies per visitor	A model scores how likely you are to subscribe and shows a harder or softer wall accordingly

Hard paywalls are the simplest and the strongest: the body never ships to a non-subscriber, so there's nothing to recover. The Financial Times and parts of the Wall Street Journal run close to this model. The tradeoff is reach — a hard wall sacrifices the casual reader and some SEO surface to protect revenue.

Soft/freemium walls flag certain articles as premium and leave the rest open. The decision is per-article, made on the server, so a "premium" piece behaves like a hard wall while a "free" piece is fully open.

Metered paywalls are the most common on large news sites because they thread the needle: a handful of free articles per month drive subscriptions, social sharing, and search traffic, while heavy readers eventually hit the wall. The catch is that metering has to count, and where it counts is the whole story (more on that below).

Dynamic / propensity paywalls are the modern evolution. Instead of a fixed meter, a model looks at signals — how often you visit, what you read, where you came from, whether you look like a likely subscriber — and decides in real time whether to show you a hard wall, a soft nudge, or nothing at all. Two readers can hit the same URL and see completely different walls. That variability is deliberate: it makes the wall harder to reason about and harder to defeat with a single static trick.

The one distinction that explains everything: client-side vs server-side

Forget the marketing names for a second. The question that actually determines whether a paywall is robust is brutally simple: does the full article text reach the browser at all?

CLIENT-SIDE (leaky)                  SERVER-SIDE (sealed)

  origin ──[ full article ]──▶ browser   origin ──[ teaser only ]──▶ browser
                 │                                   ▲
        JS / CSS hides the body            access check runs at the origin,
        (overlay, truncation, fade)        BEFORE the body is ever sent
                 │                                   │
   the bytes are already on the         there is nothing on the page
   page  →  "un-hideable"               to un-hide  →  sealed

Client-side paywalls send the complete article in the HTML or in a JSON blob the page hydrates from, then use JavaScript and CSS to hide most of it — an overlay, a display:none, a truncated container, or a gradient "fade to subscribe." The content is already on the page; the wall is cosmetic. This is why the classic tricks (disable JavaScript, view source, use a browser's reader mode) sometimes reveal the whole article: the bytes were delivered before the wall was painted.
Server-side paywalls make the access decision on the server and simply never include the gated text in the response. A non-subscriber receives a teaser — headline, a paragraph or two, structured metadata — and nothing else. There is nothing to un-hide because the body was never sent.

Google says exactly this to publishers in its own documentation: "If you don't want the content to be accessible to the browser at the time of serving, choose a paywall implementation that doesn't supply the paywalled content to the browser." In plain terms, Google is openly telling publishers that client-side gating is leaky and server-side gating is not.

So why does anyone still ship client-side? Because it's cheaper and more flexible. Rendering the full page and gating it in the browser plays nicely with ad tech, A/B testing, personalization, and CDN caching (one cached page serves everyone; the JS decides what to show). Server-side entitlement checks mean per-request rendering, a harder caching story, and more backend work. Plenty of publishers knowingly trade a little leakiness for a lot of operational convenience — which is why the web is full of client-side walls a reader can see straight through.

How metering actually counts you

Metered paywalls deserve their own look, because "you've read 5 of 5 free articles" has to be stored somewhere, and where decides how sturdy the meter is.

Cookies / local storage. The cheapest meter increments a counter in your browser. It's also the weakest: clearing site data, or opening a private/incognito window (which starts with empty storage), resets the count. This is the single reason "open it in incognito" works on so many sites — you're not breaking anything, you're just presenting as a brand-new visitor.
Device fingerprinting. Sturdier meters derive a semi-stable id from your browser and device characteristics, so a fresh incognito window still looks like the same device. Harder to reset, but probabilistic and privacy-fraught.
IP address. Some meters count per IP. Effective against casual evasion, but blunt — it can wrongly gate everyone behind a shared office or campus network.
Server-side accounts. The sturdiest meter ties consumption to a logged-in identity. There's nothing client-side to clear, because the count lives in the publisher's database. This is where metering converges with a hard wall.

The pattern to notice: the more robust the meter, the more it moves off the client and onto the server — the same migration we just saw with rendering. Anything enforced in the browser can be undone in the browser.

The Googlebot contract: how publishers show bots what they hide from you

Here's the part most explanations skip, and it's the most important. A publisher who hides the article from readers but serves the full text to Googlebot is, on its face, doing cloaking — showing crawlers something different from what users get. Cloaking is a search-spam violation that gets a site demoted or removed from the index. So how do paywalled articles rank at all?

Google built a sanctioned exception. It evolved out of the old "first click free" policy (drop the wall for visitors arriving from Google) and became, in 2017, flexible sampling plus a structured-data declaration. Publishers mark their paywalled sections with schema.org markup — isAccessibleForFree: false plus a hasPart block whose cssSelector points at the gated element:

{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "Article headline",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywall"
  }
}

That declaration is the contract. It tells Google: "this .paywall section is gated, and any difference between what Googlebot sees and what a logged-out human sees is intentional, not cloaking." In return, the publisher grants Googlebot (and Googlebot-News) full access to the body so the article can be indexed and ranked.

                    ┌──────────────────────────────┐
   Googlebot  ────▶ │  Publisher origin            │ ──▶  FULL article
 (verified by       │   isAccessibleForFree: false │      (so it can be indexed)
  reverse DNS)      │   hasPart → ".paywall"       │
   Logged-out  ───▶ │                              │ ──▶  teaser + subscribe wall
   reader           └──────────────────────────────┘
      The JSON-LD declares the gap on purpose, so serving the
      bot more than the human is treated as policy — not cloaking.

Two consequences fall out of this, and they explain a lot of real-world behavior:

Publishers verify that Googlebot is really Googlebot. Because crawler access is a privilege, sites confirm it by reverse-DNS and IP against Google's published ranges — not by trusting the User-Agent header. That's why simply sending User-Agent: Googlebot from an ordinary server gets you an HTTP 403: the request's IP doesn't belong to Google. The user-agent trick only ever worked on sites that didn't bother validating, and the big publishers all validate.
The markup hands out a map of the wall. The cssSelector: ".paywall" is, quite literally, the selector of the overlay element. A declaration intended to help search engines also tells anyone reading the page source exactly which node is the gate — which is why client-side "un-hide" tools target that same selector.

The same logic extends to AMP: Google requires a publisher's bot-access policy to match across AMP and non-AMP pages (via amp-subscriptions), or Search Console flags a content mismatch. That parity requirement is why AMP versions of articles are sometimes less aggressively gated than their canonical pages — the publisher had to keep the two consistent for the crawler.

How paywall "bypass" tools actually work

Open-source paywall removers — the best known being Bypass Paywalls Clean, plus web tools like 12ft and archives like archive.today — are essentially a catalogue of per-site rules, each exploiting one of the weaknesses above. Understanding what they do is useful for reasoning about how robust a given paywall is. It is not an endorsement: several have been removed from extension stores under legal pressure, which is the subject of the next section.

Technique	Which paywall design it targets	Why it fails on hardened sites
Crawler user-agent (Googlebot/Bingbot)	Sites that serve crawlers the full body	Blocked by IP / reverse-DNS validation of the bot
Referer spoofing (Google / social)	"First-click-free"-style allowances	Most publishers dropped first-click-free; ignored on server-side gates
Clearing cookies / storage	Metered counters tracked client-side	Useless against server-side, account-based, or fingerprinted meters
Blocking the paywall script (Piano/Tinypass, Poool, etc.)	Client-side JS enforcement	Nothing to block when the gate is server-side
AMP / reader-mode / view-source	Content shipped-then-hidden	The body simply isn't in the response on server-side pages
Reading embedded JSON (`articleBody`, framework state)	Sites that ship full text for their own SPA/SEO	The text isn't embedded when rendered server-side per entitlement
Web archives (archive.today)	Anything someone already archived	Depends on a third-party copy existing; raises its own copyright questions

Walk down the column and a single pattern emerges. Crawler-UA and referer tricks exploit the indexing contract — they try to look like the privileged visitor the publisher serves in full. Cookie-clearing exploits client-side metering. Script-blocking, reader-mode, and view-source exploit client-side rendering. Reading embedded JSON exploits the fact that a single-page app or an SEO setup often ships the whole article as data even when the visible DOM is truncated. Archives sidestep the live site entirely by reading a copy someone else already saved.

The throughline: every one of these works only because the content already left the publisher's server. Server-side rendering plus IP-validated bot access closes the entire column at once — there is no header to spoof into a privilege, no counter in the browser to reset, no hidden body to un-hide, and no embedded JSON because the body was never serialized to the client.

Why the arms race now favors publishers

A decade ago, "disable JavaScript" beat most paywalls. Today it rarely does, for a few converging reasons:

Server-side rendering keeps the body off the wire until entitlement is checked. The leak closes at the source.
Dynamic / propensity models change the wall per visit, so a single static rule breaks the moment the model decides you look different.
Bot validation — reverse DNS for Googlebot, plus commercial anti-bot vendors like Cloudflare and DataDome at the edge — makes crawler impersonation and naive automated access expensive and unreliable. A spoofed user-agent now meets a fingerprinting challenge, not a free pass.
Edge enforcement means the gate is applied at the CDN, before a request ever reaches the origin app. The decision happens in front of the content, not inside it.

The net effect is that the cheap, client-side techniques are dying off, and what remains is either legally fraught (archives, account sharing) or simply doesn't work against a modern server-side, dynamically gated, edge-protected site.

The legal reality: paywalls are the highest-risk category

This is the part that matters most, and it's why Crawlora's position is unambiguous: don't bypass paywalls. It's consistent with everything in our guide on whether web scraping is legal in 2026 — the rules depend on the data, the method, and what you do with the results.

Access risk stratifies cleanly:

Tier 1 — public, non-gated pages. The lowest risk. In the US, hiQ Labs v. LinkedIn and the Supreme Court's narrowing of the CFAA in Van Buren v. United States support the view that accessing data available to the public without authentication is not "unauthorized access."
Tier 2 — login-gated content. A step riskier: you're now past an authentication boundary, and terms of service are squarely in play.
Tier 3 — paywalled content. The top of the risk stack. Engineering a workaround around a technological access control can implicate the DMCA's anti-circumvention rule (§1201) — which targets circumventing a measure that controls access to a work, separate from copyright infringement itself — and the CFAA, on top of breaching the site's terms of service.

The case law is moving in the publishers' direction. Reddit v. Perplexity alleges circumvention of rate limits and anti-bot systems; Google sued SerpApi in late 2025 citing the DMCA and copyright. And the open-source paywall removers themselves have been pulled from the Chrome and Firefox stores under the DMCA — the clearest signal of where the legal line sits.

Public, non-gated pages are the defensible tier; logins and paywalls escalate risk sharply.
Circumventing a technological access control — a paywall, login, or anti-bot system — is a distinct legal exposure under DMCA §1201, separate from reading a public page.
Terms of service can prohibit automated access even to public content; that's a contract risk on top of everything else.
If you need a specific publisher's articles at scale, the right path is a licensing or syndication deal — not a workaround.

The right way to get article content at scale

If your project genuinely needs article text, there are legitimate routes, in rough order of preference:

Official content APIs and licensing. Many publishers and wire services license full text, and a syndication or licensing agreement is the durable answer for a specific outlet's articles at scale. Several large publishers also expose documented developer APIs for metadata.
The structured data publishers already expose. Headlines, descriptions, authors, dates, sections, and tags are published for crawlers in JSON-LD — that's fair game and machine-readable by design. You can get a lot of value from the metadata layer without touching gated bodies.
Public, non-gated pages. For the large universe of web content that isn't paywalled at all, a compliant scraping API that respects robots.txt, rate limits, and terms is the clean way to get structured content without running your own browser fleet.

That last one is where Crawlora fits. Our web scraping API and the /web/scrape endpoint turn public URLs into clean Markdown and structured metadata, with managed rendering and proxies — built for public web data, not for circumventing paid content. If you want to know how hard a given public page is to fetch before you start, the anti-bot checker gives you a difficulty read on the exact URL, and the proxies explainer covers responsible pacing.

The takeaway

A paywall is just an answer to one question — where does the content live when a non-subscriber asks for it? Keep it in the browser and hide it, and the wall is cosmetic. Keep it on the server and never send it, and the wall is real. The structured-data contract with Google explains the strange middle ground where bots see everything and humans see a teaser, and the steady migration of every defense — rendering, metering, bot checks — from the client to the server and the edge is why the easy tricks keep dying. The robust, lawful way to work with article content at scale isn't to fight that trend; it's to use the public data, the structured metadata, and the licensing the open web already provides.

Sources

Frequently asked questions

How do paywalls work?

A paywall withholds an article from non-subscribers, but the implementation varies. Hard paywalls serve no body at all; metered paywalls track your free-article count with a cookie, device fingerprint, or account; dynamic paywalls vary the wall per visitor. The key technical difference is whether the full text is sent to your browser and then hidden (client-side) or never sent at all (server-side).

Why can I read some paywalled articles in incognito mode but not others?

Incognito clears cookies and local storage, which resets a client-side metered counter that tracks how many free articles you've read — so metered paywalls often reopen in a fresh private window. It does nothing against hard or server-side paywalls, where the article body is never delivered to the browser in the first place.

What is the difference between a client-side and server-side paywall?

A client-side paywall sends the full article to the browser and hides it with JavaScript/CSS (an overlay or truncation), so the content technically reached your device. A server-side paywall decides access on the server and never includes the gated text in the response. Client-side gates are far easier to circumvent; server-side gates are, in Google's own words, almost impossible to get around.

Is it legal to bypass a paywall?

Bypassing a paywall is the highest-risk category of web access. Circumventing a technological access control can implicate the DMCA's anti-circumvention rules (§1201) and the CFAA, on top of breaching the site's terms of service. Reading public, non-gated pages is far more defensible, and for a specific publisher's full articles at scale, licensing is the right path — not a workaround. This is not legal advice.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

Give Your AI Agent Live Web Data with MCP

Tony Wang — Mon, 08 Jun 2026 09:51:45 +0000

Key takeaways

Give an AI agent live web data by connecting it to Crawlora's hosted MCP endpoint — it calls documented tools (search, maps, commerce, social, finance) and gets normalized JSON back, with no scraping code or proxies to run.

MCP (Model Context Protocol) is an open standard: agents discover and call tools through one interface instead of a bespoke integration per data source.

Connect over Streamable HTTP at https://mcp.crawlora.net/mcp with your API key — about three minutes in Claude, Cursor, Cline, Windsurf, or any MCP client.

One connection exposes 319 tools across 33 platforms (393 REST endpoints underneath): Google/Bing/Brave search, Google Maps, Amazon, YouTube, TikTok, Yahoo Finance, CoinGecko, and more.

You pay only on a successful (2xx) response — failed calls are free — and the free tier includes 2,000 credits a month with no card.

Versus writing your own scrapers: no per-source glue code, normalized JSON instead of HTML, and proxy routing, rendering, and retries handled behind the endpoint.

You can give an AI agent live web data by connecting it to a hosted MCP endpoint: your agent calls documented tools — search, maps, e-commerce, app stores, social, finance, and more — and gets back normalized JSON, with no scraping code to write or proxies to run. This guide explains what MCP is, what data you can pull, how to connect in about three minutes, and what a real tool call and its response look like.

Most LLMs are frozen at their training cutoff and can't see the live web. The usual fix — writing a scraper per source, then maintaining proxies, headless browsers, and parsers — is exactly the work teams don't want to own. MCP plus a hosted data server removes it: the model gets a stable set of tools, and the fetching lives behind an endpoint.

What is MCP, and why does it matter for agents?

The Model Context Protocol (MCP) is an open standard that lets an AI agent call external tools through one consistent interface. Instead of wiring a bespoke integration for every data source, the agent connects to an MCP server, discovers the tools it exposes, and calls them during a task.

An MCP server can expose three kinds of primitives: tools (functions the model can call, like google_map_search), resources (read-only data), and prompts (reusable templates). For live web data, tools are what matter — each one is a documented action with typed inputs and a predictable output.

Why this beats a pile of one-off integrations:

One interface, many sources. Add a data source, swap a search engine, or pull a new platform without touching your agent's wiring — it's a tool call, not a rewrite.
Self-describing. The agent reads each tool's schema, so it knows what arguments to pass and what shape comes back.
Portable. The same server works across Claude, Cursor, Cline, Windsurf, n8n, and any MCP-compatible client.

Who should use this

Claude Code, Cursor, Cline, and Windsurf users who want their editor or agent to read live web, SERP, commerce, or finance data while coding or researching.
Agent builders wiring tools into LangChain, n8n, or a custom framework who need a reliable web-data layer instead of bespoke scrapers.
RAG and data teams that need fresh, structured records — places, products, reviews, prices, quotes — rather than raw HTML to parse.
Anyone moving an agent from prototype to production who doesn't want to run proxies, browsers, and parser maintenance.

What your agent can pull: the tool catalog

Crawlora's hosted MCP server exposes 319 tools across 33 platforms, backed by 393 documented REST endpoints. One connection covers a wide slice of the public web, each tool returning the same JSON fields every time:

Category	Platforms	Example tools
Search & SERP	Google, Bing, Brave (web, news, images, suggest)	`google_search`, `bing_search`, `brave_search`
Maps & local	Google Maps (places, search, reviews)	`google_map_search`, `google_map_place`
E-commerce	Amazon, eBay, Shopify, Shop.app	`amazon_search`, `ebay_search`, `shopify_products`
App stores	Apple App Store, Google Play	`appstore_search`, `googleplay_reviews`
Social & creator	TikTok, YouTube, Instagram, Reddit	`tiktok_search`, `youtube_search`, `reddit_search`
Reviews & travel	Trustpilot, Tripadvisor	`trustpilot_business_reviews`, `tripadvisor_search`
Finance & crypto	Yahoo Finance, Google Finance, CoinGecko	`yahoo_finance_ticker_quote`, `coingecko_coin`

The deepest groups carry dozens of tools each — Yahoo Finance (39), Spotify (30), TikTok (24), CoinGecko (21), JustWatch (21), Google Finance (20) — so an agent can do real work on one platform without leaving the server.

Connect the hosted MCP endpoint in about three minutes

Crawlora runs a hosted MCP endpoint over Streamable HTTP at https://mcp.crawlora.net/mcp. There's nothing to install or host — you point your client at the URL and authenticate with your API key, either as an x-api-key header or an Authorization: Bearer token. Get a free key (2,000 credits/month, no card) first.

Claude Desktop / Claude Code, Cursor, Windsurf — add the server to your client's MCP config:

{
  "mcpServers": {
    "crawlora": {
      "url": "https://mcp.crawlora.net/mcp",
      "headers": { "x-api-key": "YOUR_API_KEY" }
    }
  }
}

Cline (VS Code) — open the MCP Servers panel, choose Remote, and use the same URL and header. The tools appear in the agent's tool list once connected.

A stdio bridge — if your client only speaks stdio rather than a remote URL, wrap the endpoint with a proxy and pass the key as an environment variable:

npx mcp-remote https://mcp.crawlora.net/mcp \
  --bearer-token-env-var CRAWLORA_API_KEY

The MCP docs have the current connection details and a server card listing the full tool catalog. After connecting, ask your agent to "list available tools" to confirm the tools are visible.

A worked example: from one prompt to clean JSON

Once connected, the agent calls tools and reasons over the normalized JSON they return. Ask:

"Find the top-rated coffee shops in Austin and summarize what reviewers like."

The agent picks the maps tool and calls it:

{
  "tool": "google_map_search",
  "arguments": { "query": "coffee shops in Austin, TX", "limit": 5 }
}

It gets back structured records — not HTML to parse — that look like this (trimmed for the example):

{
  "results": [
    {
      "name": "Houndstooth Coffee",
      "rating": 4.6,
      "reviews": 1284,
      "address": "401 Congress Ave, Austin, TX 78701",
      "category": "Coffee shop"
    },
    {
      "name": "Cuvée Coffee Bar",
      "rating": 4.5,
      "reviews": 932,
      "address": "2000 E 6th St, Austin, TX 78702",
      "category": "Coffee shop"
    }
  ]
}

From there the agent ranks by rating and review count and writes the summary. The same pattern works for any platform: search a marketplace with amazon_search, pull a stock quote with yahoo_finance_ticker_quote, or read app reviews with googleplay_reviews. The data layer is Crawlora; the orchestration is your agent framework.

MCP vs. writing your own scrapers

The shortcut is real, but it helps to see exactly what you trade away by not building the plumbing yourself:

	Crawlora MCP	DIY scrapers
Integration	One interface; tools discovered automatically	Bespoke glue code per source
Output	Normalized JSON with a documented schema	HTML you parse and re-parse
Fetching	Proxy routing, JS rendering, retries handled	You run proxies and headless browsers
Maintenance	None — the endpoint owns the schema	Parsers break when a page's layout shifts
Coverage	319 tools across 33 platforms, one key	One scraper per source you build
Cost model	Pay on success (2xx only); free tier	Infra + engineering time, paid regardless

The two aren't mutually exclusive. For arbitrary, unpredictable pages — docs sites, blogs, long-tail URLs — an AI-native crawler that returns markdown is the better fit. For known platforms where you want stable records to sort, join, and chart, documented endpoints win because there's no parser to maintain. Many teams run both.

Best practices for agents that call web data

Authenticate with a header, not a query string: send the key as x-api-key or Authorization: Bearer so it never lands in logs or URLs.
Let the model read the tool schemas before calling — discovery is the point of MCP; don't hard-code arguments your agent could infer.
Handle the 2xx-only billing model in your logic: a failed call costs nothing, so retries are cheap, but check status before treating a response as data.
Start narrow. Point the agent at the few tools a task needs rather than all of them, so its tool-selection stays accurate.
Cache results you'll reuse within a task to save credits and latency — live data doesn't mean re-fetching the same page twice.
Prototype on the free tier, then watch the credits dashboard before you scale a multi-step agent that fans out calls.

Pricing, credits, and limits

Crawlora bills on a pay-on-success model: each call costs 1–8 credits and is charged only on a successful (2xx) response — 4xx and 5xx responses are free, so an agent that retries or probes doesn't run up a bill for failures. The free tier includes 2,000 credits per month with no card, which is enough to build and test a real agent before upgrading. There's also a public Playground to run any endpoint and inspect the JSON before you wire it into a tool call.

Frequently asked questions

Do I need to host anything to use Crawlora's MCP server?

No. It's a hosted, remote MCP server over Streamable HTTP — you point your client at https://mcp.crawlora.net/mcp and add your API key. There's no server to install, no proxies to rotate, and no browsers to run.

Which clients work with it?

Any MCP-compatible client: Claude Desktop and Claude Code, Cursor, Cline, Windsurf, and agent frameworks like n8n or LangChain via an MCP adapter. The same remote URL and header work everywhere.

How is this different from a general web-scraping or crawler MCP?

Crawler-style servers fetch an arbitrary URL and return its page content as markdown — great for unstructured pages. Crawlora exposes documented tools for known platforms, so a Google Maps place or an Amazon product comes back as the same JSON fields every time, with no extraction prompt or parser to maintain.

What data formats does it return?

Normalized JSON per tool, with a documented schema. You get records — places, products, reviews, prices, quotes, posts — not raw HTML, so your agent can use the response immediately.

How does authentication work?

Send your Crawlora API key as an x-api-key header or an Authorization: Bearer token on the MCP connection. The same key authenticates every tool the server exposes.

Can I try it for free?

Yes — the free tier is 2,000 credits a month with no card, and you only spend credits on successful responses. Get a key and connect in about three minutes.

Give your agent live web data in three minutes — a hosted MCP server, 319 documented tools across search, maps, commerce, social, and finance, normalized JSON, and managed proxies and retries. 2,000 free credits a month, no card. → Read the MCP docs · Try the Playground

Where this fits

See the AI agent web data use case for the broader pattern, and the LangChain integration if you're wiring tools through a framework rather than a native MCP client. For the web-data fundamentals behind the tools, see how to choose a web scraping API.

Sources

27.6% of the Top 10 Million Sites are Dead

Tony Wang — Wed, 30 Oct 2024 08:48:52 +0000

The internet, in many ways, has a memory. From archived versions of old websites to search engine caches, there's often a way to dig into the past and uncover information—even for websites that are no longer active. You may have heard of the Internet Archive, a popular tool for exploring the history of the web, which has experienced outages lately due to hacks and other challenges. But what if there was no Internet Archive? Does the internet still "remember" these sites?

In this article, we'll dive into a study of the top 10 million domains and reveal a surprising finding: over a quarter of them—27.6%—are effectively dead. Below, I'll walk you through the steps and infrastructure involved in analyzing these domains, along with the system requirements, code snippets, and statistical results of this research.

The Challenge: Analyzing 10 Million Domains

Thanks to resources like DomCop, we can access a list of the top 10 million domains, which serves as our starting point. Processing such a large volume of URLs requires significant computing resources, parallel processing, and optimized handling of HTTP requests.

To get accurate results quickly, we needed a well-designed scraper capable of handling millions of requests in minutes. Here’s a breakdown of our approach and the system design.

System Design for High-Volume Domain Scraping

To analyze 10 million domains in a reasonable timeframe, we set a target of completing the task in 10 minutes. This required a system that could process approximately 16,667 requests per second. By splitting the load across 100 workers, each would need to handle around 167 requests per second.

1. Efficient Queue Management with Redis

Redis, with its capability of handling over 10,000 requests per second easily, played a key role in managing the job queue. However, even with Redis, tracking status codes from millions of domains can overload the system. To prevent this, we utilized Redis pipelines, allowing multiple jobs to be processed simultaneously and reducing the load on our Redis cluster.

// SPopN retrieves multiple items from a Redis set efficiently.
func SPopN(key string, n int) []string {
    pipe := Redis.Pipeline()
    for i := 0; i < n; i++ {
        pipe.SPop(ctx, key)
    }
    cmders, err := pipe.Exec(ctx)
    if err != nil { return nil }

    results := make([]string, 0, n)
    for _, cmder := range cmders {
        if spopCmd, ok := cmder.(*redis.StringCmd); ok {
            val, err := spopCmd.Result()
            if err == nil && val != "" { results = append(results, val) }
        }
    }
    return results
}

Using this method, we could pull large batches from Redis with minimal impact on performance, fetching up to 100 jobs at a time.

func (w *Worker) fetchJobs() {
    for {
        if len(w.Jobs) > 100 {
            time.Sleep(time.Second)
            continue
        }
        jobs := SPopN(w.Name+jobQueue, 100)
        for _, job := range jobs {
            w.AddJob(job)
        }
    }
}

2. Optimizing DNS Requests

To resolve domains efficiently, we used multiple public DNS servers (e.g., Google DNS, Cloudflare) and handled up to 16,667 requests per second. Public DNS servers typically throttle large volumes of requests, so we implemented error handling and retries for DNS timeouts and throttling errors.

var dnsServers = []string{
    "8.8.8.8", "8.8.4.4", "1.1.1.1", "1.0.0.1", "208.67.222.222", "208.67.220.220",
}

By balancing the load across multiple servers, we could avoid rate limits imposed by individual DNS providers.

3. HTTP Request Handling

To check domain statuses, we attempted direct HTTP/HTTPS requests to each IP address. The following code retries with HTTPS if the HTTP request encounters a protocol error.

func (w *Worker) worker(job string) {
    var ips []net.IPAddr
    var err error
    var customDNSServer string
    for retry := 0; retry < 5; retry++ {
        customDNSServer = dnsServers[rand.Intn(len(dnsServers))]
        resolver := &net.Resolver{
            PreferGo: true,
            Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
                d := net.Dialer{}
                return d.DialContext(ctx, "udp", customDNSServer+":53")
            },
        }

        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()

        ips, err = resolver.LookupIPAddr(ctx, job)
        if err == nil && len(ips) > 0 {
            break
        }

        log.Printf("Retry %d: Failed to resolve %s on DNS server: %s, error: %v", retry+1, job, customDNSServer, err)
    }

    if err != nil || len(ips) == 0 {
        log.Printf("Failed to resolve %s on DNS server: %s after retries, error: %v", job, customDNSServer, err)
        w.updateStats(1000)
        return
    }

    customDialer := &net.Dialer{
        Timeout: 10 * time.Second,
    }
    customTransport := &http.Transport{
        DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
            port := "80"
            if strings.HasPrefix(addr, "https://") {
                port = "443"
            }
            return customDialer.DialContext(ctx, network, ips[0].String()+":"+port)
        },
    }
    client := &http.Client{
        Timeout:   10 * time.Second,
        Transport: customTransport,
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }

    req, err := http.NewRequestWithContext(context.Background(), "GET", "http://"+job, nil)
    if err != nil {
        log.Printf("Failed to create request: %v", err)
        w.updateStats(0)
        return
    }
    req.Header.Set("User-Agent", userAgent)

    resp, err := client.Do(req)
    if err != nil {
        if urlErr, ok := err.(*url.Error); ok && strings.Contains(urlErr.Err.Error(), "http: server gave HTTP response to HTTPS client") {
            log.Printf("Request failed due to HTTP response to HTTPS client: %v", err)
            // Retry with HTTPS
            req.URL.Scheme = "https"
            customTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
                return customDialer.DialContext(ctx, network, ips[0].String()+":443")
            }
            resp, err = client.Do(req)
            if err != nil {
                log.Printf("HTTPS request failed: %v", err)
                w.updateStats(0)
                return
            }
        } else {
            log.Printf("Request failed: %v", err)
            w.updateStats(0)
            return
        }
    }
    defer resp.Body.Close()

    log.Printf("Received response from %s: %s", job, resp.Status)
    w.updateStats(resp.StatusCode)
}

Deployment Strategy

Our scraping deployment consisted of 400 worker replicas, each handling 200 concurrent requests. This configuration required 20 instances, 160 vCPUs, and 450GB of memory. With CPU usage at only around 30%, the setup was efficient and cost-effective, as shown below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 400
  ...
  containers:
    - name: worker
      image: ghcr.io/tonywangcn/ten-million-domains:20241028150232
      resources:
        limits:
          memory: "2Gi"
          cpu: "1000m"
        requests:
          memory: "300Mi"
          cpu: "300m"

The approximate cost for this setup was around $0.0116 per 10 million requests, totaling less than $1 for the entire analysis.

Data Analysis: How Many Sites Are Actually Accessible?

The status code data from the scraper allowed us to classify domains as "accessible" or "inaccessible." Here’s the criteria used:

Accessible: Status codes other than 1000 (DNS not found), 0 (timeout), 404 (not found), or 5xx (server error).
Inaccessible: Domains with the status codes above, indicating they are either unreachable or no longer in service.

accessible_condition = (
    (df["status_code"] != 1000) &
    (df["status_code"] != 0) &
    (df["status_code"] != 404) &
    ~df["status_code"].between(500, 599)
)
inaccessible_condition = ~accessible_condition

After aggregating the results, we found that 27.6% of the domains were either inactive or inaccessible. This meant that over 2.75 million domains from the top 10 million were dead.

| Status Code | Count     | Rate |
| ----------- | --------- | ---- |
| 301         | 4,989,491 | 50%  |
| 1000        | 1,883,063 | 19%  |
| 200         | 1,087,516 | 11%  |
| 302         | 659,791   | 7%   |
| 0           | 522,221   | 5%   |

Conclusion

With a dataset as large as 10 million domains, there are bound to be formatting inconsistencies that affect accuracy. For example, domains with a www prefix should ideally be treated the same as those without, yet variations in how URLs are constructed can lead to mismatches. Additionally, some domains serve specific functions, like content delivery networks (CDNs) or API endpoints, which may not have a traditional homepage or may return a 404 status by design. This adds a layer of complexity when interpreting accessibility.

Achieving complete data cleanliness and uniform formatting would require substantial additional processing time. However, with the large volume of data, minor inconsistencies likely constitute around 1% or less of the overall dataset, meaning they don’t significantly affect the final result: more than a quarter of the top 10 million domains are no longer accessible. This suggests that as time passes, your history and contributions on the internet could gradually disappear.

While the scraper itself completes the task in around 10 minutes, the research, development, and testing required to reach this point took days or even weeks of effort.

If this research resonates with you, please consider supporting more work like this by sponsoring me on Patreon. Your support fuels the creation of articles and research projects, helping to keep these insights accessible to everyone. Additionally, if you have questions or projects where you could use consultation, feel free to reach out via email.

The source code for this project is available on GitHub. Please use it responsibly—this is meant for ethical and constructive use, not for overwhelming or abusing servers.

Thank you for reading, and I hope this research inspires a deeper appreciation for the impermanence of the internet.

DEV Community: Tony Wang

AI vs Traditional Web Scraping: Which Wins, When

Traditional web scraping (selectors)

"AI web scraping" is really three methods

The token tax nobody mentions

The catch: AI doesn’t get you the page

A structured data API (skip the page)

Side by side

So which should you use?

Sources

Next steps

Frequently asked questions

What is the difference between AI and traditional web scraping?

What are the types of AI web scraping?

Is AI web scraping more accurate than traditional scraping?

How much does AI web scraping cost per page?

Does AI web scraping avoid getting blocked?

When should I use a structured API instead?

Best SERP APIs in 2026 for Rank Tracking and Search Data

Why there's no official Google SERP API

What to compare in a SERP API

The best SERP APIs in 2026

1. Crawlora — multi-engine structured JSON

2. SerpApi — feature-rich, but the priciest

3. DataForSEO — cheapest at scale, with SEO datasets

4. Serper & Scrapingdog — lowest cost, fast

5. Bright Data — enterprise scale across engines

6. ScrapingBee, SearchApi, Decodo — mid-tier general-purpose

7. Generic scrapers and proxies — DIY SERP parsing

Track SERP features, not just positions

Reliability: the 2025 SerpApi lawsuit

Cost per successful query, not sticker price

From SERP API to rank tracker

Recommendation

Next steps

Sources

Related reading

Frequently asked questions

What is a SERP API?

Is there an official Google SERP API?

What is the best SERP API for rank tracking?

How much does a SERP API cost?

Do I need to track AI Overviews?

Is there a free SERP API?

Do I need Google, Bing, and Brave?

Proxies for Web Scraping, Explained (2026)

Why scraping needs proxies at all

The main proxy types

Rotating vs sticky sessions

How proxies fit anti-bot defenses

When you don't need to manage proxies

Quick decision guide

Sources

Start collecting

Frequently asked questions

Are residential proxies legal?

Datacenter or residential proxies for scraping?

Do I always need proxies?

Do proxies stop CAPTCHAs?

World Cup 2026 TikTok Creators by the Numbers

The World Cup's biggest TikTok stars aren't where you'd think

Giant audiences, closed inboxes

The verified elite — and where sport sits

What it tells us

Sources

Frequently asked questions

How big is the 2026 World Cup on TikTok?

Which World Cup country has the biggest TikTok creators?

Does the USA dominate TikTok?

Are the biggest TikTok creators the easiest to reach?

How was this World Cup creator study measured?

Which World Cup nations could not be measured?

The TikTok World Cup: Who the Internet Actually Talks About

The players TikTok can't stop posting about

Messi beats Ronaldo — the one place he does

The nations: Argentina runs the table

The social graph: who gets talked about together

Viral isn't the same as beloved

Why TikTok's football map looks nothing like Instagram's

What it tells us