DEV Community: Tony Wang

How Much of the Web Runs Anti-Bot? We Scanned the Top 1,000,000 Sites

Tony Wang — Tue, 11 Aug 2026 08:51:49 +0000

Key takeaways

We scanned all ~1,000,000 sites on the Tranco research ranking. Of the 818,614 we reached, 53.5% run a detectable managed anti-bot, WAF, or access-control vendor — and it is overwhelmingly Cloudflare (45% of reachable sites, 84% of every protected site).
The web's busiest sites run the LEAST anti-bot. By Tranco rank, protection rises as rank falls (top-1,000 44% → long-tail 54%) and Cloudflare rises with it (23% → 46%) — its dominance is a long-tail story — while enterprise bot-management (DataDome, PerimeterX, Akamai) drops the other way (7.6% → 0.6%). The head and the tail are guarded by different vendors.
Independently corroborated: W3Techs puts Cloudflare on ~48.7% of the top 1M and 23.2% of all websites; our scan finds its active anti-bot posture on 45% of the reachable top 1M. Imperva reports bots are now 51% of all web traffic — the demand behind the walls.
Difficulty is lower than the fear suggests. Among the sites people actually scrape, 85% need no browser at all — a plain HTTP client or a matched TLS fingerprint reaches them — once you escalate properly; only ~15% need a real browser, and four sign every request with a closed JS VM.
Protection tracks money: crypto (86%), marketplaces (81%) and AI tools (78%) are the most defended; search (18%), social (25%) and sports (33%) homepages are the most open — but the homepage is a lower bound for the deep pages you actually scrape.

Most "I'll just scrape it" projects do not die on parsing the HTML. They die when the page turns out to sit behind Cloudflare or DataDome and needs a completely different approach than the one you built. So we measured how common that actually is — twice. We scanned the whole Tranco top 1,000,000 to get the web-scale picture, and then hand-picked 1,005 real, public sites across 28 categories — the kinds of sites people actually scrape — and ran the full transport fleet against them. The short version: just over half of the reachable web runs managed bot defense, it is concentrated in one vendor, the busiest sites run the least of it, and the difficulty is wildly uneven.

The full, searchable dataset — every site, filterable by vendor, difficulty and category — lives in the Anti-Bot Adoption Index.

How much of the web runs an anti-bot or WAF?

Across the full top 1,000,000, we reached 818,614 sites (the rest were unresolvable, timed out, or blocked our datacenter vantage outright — about 18%, excluded from the percentages). Of the ones we reached, 53.5% expose a managed WAF, bot-management, or access-control vendor — not counting a CAPTCHA widget or rate-limiting alone:

Protection type	Share of reachable
Managed WAF	47.6%
CAPTCHA widget (reCAPTCHA, hCaptcha, Turnstile…)	12.7%
Access control	2.1%
Dedicated bot management	0.8%
Any managed anti-bot / WAF	53.5%

This is not a fragmented market. Cloudflare alone covers 45% of every reachable site — 84% of all protected sites — and the next-largest named layer is consumer CAPTCHA widgets. The pure-play bot-management vendors barely register at web scale: Akamai Bot Manager 0.6%, DataDome 0.16%, PerimeterX/HUMAN 0.09%.

These numbers line up with independent measurements. W3Techs (June 2026) puts Cloudflare on 23.2% of all websites and roughly 48.7% of the top 1M — we find its active anti-bot posture on 45% of the reachable top 1M, the same ballpark from a different method. And the Imperva 2025 Bad Bot Report finds automated traffic is now 51% of all web traffic (bad bots 37%) — the demand pressure that explains why so many sites have a wall at all. (For a peer methodology on web-scale measurement, see the HTTP Archive Web Almanac.)

The busiest sites run the least anti-bot

The web-scale number hides the most useful finding. Split the million by traffic rank and three curves all move together — protection, Cloudflare and the specialist vendors:

Tranco rank band	Protected	Cloudflare	Enterprise bot-mgmt¹
Top 1,000	44.2%	23.4%	7.6%
1k–10k	50.7%	34.0%	5.3%
10k–100k	52.3%	40.2%	2.6%
100k–1M	53.6%	45.6%	0.6%

¹ Akamai Bot Manager + DataDome + PerimeterX, as a share of reachable sites in the band.

Two things flip as you go down-rank. Cloudflare climbs (23% of the top-1,000 to 46% of the long tail): its web-scale dominance is a long-tail story — the giants run their own or enterprise infrastructure and keep homepages open for SEO, while the millions of smaller sites reach for turnkey Cloudflare. Meanwhile enterprise bot-management falls (7.6% to 0.6%): DataDome, PerimeterX and Akamai Bot Manager are a head phenomenon, bought by the high-value sites where automated data has a direct dollar cost. So the head and the tail aren't just protected at different rates — they're guarded by different vendors. (W3Techs sees the same direction: Cloudflare is lower in the very top-1,000 than across the broader top-1M.)

Set Cloudflare aside (it climbs across the board) and the specialist gatekeepers swap places by rank — Akamai guards the very top, while the long tail reaches for turnkey CAPTCHA:

Zoom into the ~1,005 high-value sites people actually scrape and the head's profile sharpens further: there, enterprise bot-management sits on 15.5% of sites, and the tail's cheap CAPTCHA-widget layer nearly vanishes. Adoption and sophistication are different axes: more of the web is "protected" than you'd think, but far less of it is hard.

Most of those walls are asleep

Here's the part that surprised us most. "53.5% protected" counts every site that exposes a managed vendor — but exposing one isn't the same as using it. Of the 818,614 reachable sites, only 79,835 (9.8%) actually challenged our homepage request; the other 358,022 walled sites ran the vendor passively, returning a clean 200 to a matched request. Follow the million all the way down to the few that actually fight back:

Which wall is "awake" depends heavily on the vendor. Cloudflare sits in front of 45% of the reachable web but actively challenged only ~16% of those homepages; Google reCAPTCHA almost never fires on a homepage (3%); the small unidentified-WAF group is the opposite — present rarely, but challenging 76% of the time.

The practical takeaway: a vendor fingerprint tells you what you'd face if the wall fires — but most homepages won't fire at all, which is exactly why reaching for a full browser by default wastes time and budget. Check the page, not the logo.

Zoom in: the ~1,005 sites people actually scrape

For the sites that matter to a scraper, we go beyond headers: any site that blocks a datacenter GET is re-probed through the full transport fleet (browser-impersonation HTTP → headless browser → stealth browser + residential IP), so the tier reflects what actually reaches the page, not a header guess. By that bar, 575 of the 1,005 (57.2%) run managed bot defense, and two vendors dominate it:

Vendor	Type	Sites	Share
Cloudflare	WAF + challenge	333	33.1%
Akamai Bot Manager	Bot management	111	11.0%
Akamai (edge)	CDN/WAF	53	5.3%
DataDome	Bot management	30	3.0%
Imperva (Incapsula)	WAF	15	1.5%
PerimeterX (HUMAN)	Bot management	15	1.5%
Cloudflare Turnstile	CAPTCHA	12	1.2%

Cloudflare and Akamai together account for 77% of every protected site in this set. What separates the vendors is what they inspect: Akamai and open Cloudflare paths lean hardest on TLS/JA3-JA4 fingerprinting — a matched HTTP client often reaches them — while DataDome and PerimeterX add real-time behavioral ML, so a clean fingerprint alone isn't enough.

Difficulty is wildly uneven — most of it is not a browser job

The more useful question isn't "is it protected" — it's "what does it take to get the public page reliably?" Running the real fleet against the curated set:

Tier	What it takes	Sites	Share
T1	Plain HTTP client	608	60.5%
T2	Browser-impersonation HTTP (matched TLS)	246	24.5%
T3	Headless browser that runs JavaScript	148	14.7%
T4	Stealth browser + residential IP + behavior	3	0.3%

85% of these sites need no browser at all — a plain HTTP GET or a matched TLS fingerprint reaches them. Only ~15% genuinely need a headless browser or more. Reaching for a browser by default is the most common (and most expensive) scraping mistake; the data says escalate only when a site forces you to.

This is also why "57% protected" and "85% need no browser" are both true: detecting a vendor is binary, but a vendor being present isn't the same as it actively challenging. Of the 575 protected sites, only 74 actively challenged our request — the rest run their vendor in passive CDN/WAF mode (a matched fingerprint returns 200).

The hard end: a few sites sign every request with a closed VM

The toughest class isn't a CAPTCHA — it's a proprietary in-browser bytecode VM that signs every request, so generic transport tooling can't mint a valid token. Four sites in the curated set ship one: TikTok's webmssdk VM (the X-Bogus / X-Gnarly signatures, on top of Akamai) and Kasada's proof-of-work VM (the x-kpsdk-ct token, on real-estate marketplaces). For these, "send a cleverer request" doesn't apply; you need a genuine browser execution context. They're rare, but they're the sites people most often ask "why can't I scrape this?" about. (We deliberately don't count an F5 BIG-IP load-balancer cookie as a VM — that's server load balancing, not bot defense.)

A block isn't a block — read why you were stopped

When a request doesn't pass, the reason tells you the fix — and the fix is completely different each time. After escalating through the fleet, only 74 of the 1,005 (7.4%) still didn't pass cleanly, and they break down as:

Why it stopped	Sites	What it means	The fix
Bot challenge (JS interstitial)	55	Cloudflare "Just a moment", a JS challenge	Run it in a real browser
CAPTCHA	18	An interactive puzzle was served	A browser (+ CAPTCHA service)
Geo-blocked	1	Gated by country/region	An IP in an allowed region

The headline is what the failures are not. Once you bring the right transport, genuine rate limits and outright IP bans are vanishingly rare at the front door — the wall is almost always a fingerprint/JS problem, not a proxy one. We were deliberately conservative on labels: a 401 only counts as a login wall if it carries WWW-Authenticate (WSJ and Reuters return 401 as a DataDome bot block, not authentication), and a generic 403 is "blocked," not "IP banned," unless the page says so.

The homepage is a poor proxy — protection moves per page

The biggest caveat in any study like this: a domain protects different page types completely differently. On LinkedIn, the homepage is light, a /company/ page is largely open, but a /in/ profile and search are login-walled. On Amazon, the homepage is open while search readily serves a "Robot Check." So every site page in the index ships an advisory deep-page test plan — which page types to check and what to expect on each — so you test the page you actually want, not the front door.

Difficulty is per-URL, not per-site. A site's homepage is frequently left wide open for SEO and link previews, while the profile, listing, or search page right next to it is bot-managed. Because this study probed homepages, the real adoption rate on the pages people actually scrape is higher than these numbers — so always check the exact URL you intend to scrape, not the domain. You can do that on any URL with the free anti-bot checker.

The defense gradient: crypto and commerce are fortresses, search is open

Split the curated set by category and a clear money gradient appears — the closer a page is to a transaction, the harder it's defended:

Category	% managed anti-bot / WAF
Crypto	86%
Marketplaces	81%
AI tools	78%
Finance & markets	75%
Forums & community	75%
E-commerce	72%
Real estate	72%
Travel & hospitality	72%
…
News & media	39%
Sports	33%
Social media	25%
Search engines	18%

Crypto and marketplaces fight hardest — prices, inventory and order books are the most-scraped data on the web. The low numbers for social, news and search are partly a method artifact: those homepages are deliberately open, but the deep pages (a profile, an article archive, a result page) flip to protected the moment you navigate in.

What this means if you're building a scraper

Check before you build. Knowing a page is behind Cloudflare versus DataDome versus a closed VM versus nothing changes the entire approach — and it changes per URL. A 30-second check saves an afternoon of wondering why your requests 403. See the companion guide on scraping sites that block bots for what each vendor actually does.
Escalate only as far as the site demands, and pay only when it works. 85% of the sites people scrape don't need a browser at all. The cost-efficient pattern is to try the lightest transport first and escalate, which is exactly how Crawlora's web scraping for AI and data pipelines is billed — pay-on-success, not per attempt.

None of this is about defeating a CAPTCHA or getting past a login. It is about knowing, before you commit engineering time, which public pages need which approach.

Try it yourself

Check any URL right now with the free anti-bot checker — paste a profile or listing page, not just the homepage, and see the vendor and the lightest transport that works.
Explore the full dataset — search every site, filter by difficulty, vendor or category — in the Anti-Bot Adoption Index.
Read exactly how we measured it, including the signatures and the difficulty heuristic.

How the check actually works (full transparency)

No magic — this is a deliberately simple, passive, reproducible check, run at two scales.

The samples. The web-scale figures come from the full Tranco top 1,000,000 (an aggregated, citable top-sites list with a permanent ID), of which 818,614 were reachable. The deep-dive figures come from 1,005 domains walked from the top of the same ranking into real, public, content-bearing sites across 28 categories, with pure infrastructure/CDN, ad/tracking and adult domains skipped.

The request. Each site's homepage is fetched with a real Chrome User-Agent, following redirects, from a datacenter IP — the honest "what a basic cloud scraper sees" vantage. We never submit a form, run a CAPTCHA, log in, or fetch anything behind a wall. The top-1M scan is this single datacenter GET. For the curated set we go further: any site that blocks the datacenter GET is re-probed through the full transport fleet (browser-impersonation → headless → stealth + residential), so its difficulty tier reflects what actually reaches the page rather than a header guess.

What we capture (and what we don't). From the response we keep the status code, the response headers, the names of any Set-Cookie cookies — names only, never values, so no session tokens or PII are stored — and a capped slice of the body. Nothing else is retained.

Naming the vendor. The vendor is identified by matching that evidence against a database of public, documented fingerprints — header names (cf-ray, x-datadome, x-iinfo, akamai-grn), Set-Cookie name prefixes (__cf_bm/cf_clearance, _abck/bm_sz, datadome, _px*, incap_ses_), and body markers that are only trusted on a challenge-shaped response. Header and cookie matches are high-confidence; body markers medium.

Limits. Every result is a lower bound — homepages are more open than the deep pages people actually scrape, and a datacenter IP sees more challenges than a residential one. The top-1M scan is headers-only and does not grade difficulty (a site that blocks a datacenter GET there may well open for a browser); the graded tiers and the category gradient come from the full-fleet curated run. Anti-bot deployments change continuously, so treat every result as a directional signal, not a guarantee. Snapshot: June 2026.

Open & reproducible. The classifier and the methodology are published in full — read the full methodology, or explore the searchable index. Cite it as "Crawlora Anti-Bot Adoption Index" with a link.

Sources

Frequently asked questions

How much of the web is protected by anti-bot?

We measured it twice. Across the full Tranco top 1,000,000 (818,614 reachable), 53.5% expose a managed anti-bot, WAF or access-control vendor — overwhelmingly Cloudflare (45% of reachable sites). On 1,005 hand-picked high-value sites run through the full transport fleet, 57.2% are protected. Both are lower bounds: deep pages (profiles, listings, search) are more defended than the homepages we probed, and protection is heaviest on crypto (86%) and marketplaces (81%), lightest on search, social and news.

Do you actually need a browser to scrape most sites?

No — and that's the most expensive misconception. Of the curated sites, 85% need no browser at all: about 60% answer a plain HTTP GET and another 25% only want a matched TLS fingerprint. Only ~15% genuinely need a headless browser or more. The cost-efficient pattern is to escalate only as far as a site forces you to, not to reach for a headless browser by default.

Which anti-bot vendor is the most common?

Cloudflare, by a wide margin — across the top 1,000,000 it covers 45% of reachable sites and 84% of every protected site. In the curated set it leads at 33%, followed by Akamai (Bot Manager 11% plus its edge CDN 5%). The specialist bot-management vendors DataDome (3%) and PerimeterX/HUMAN (1.5%) cluster on high-value verticals like marketplaces, travel and real estate, where automated data has a direct cost.

What is the hardest type of anti-bot to deal with?

A proprietary, closed in-browser JavaScript VM that signs every request — like TikTok's webmssdk (the X-Bogus/X-Gnarly signatures) or Kasada (the x-kpsdk-ct token); four sites in the curated set ship one. Unlike a CAPTCHA, generic transport tooling can't mint a valid token, so these need a genuine browser execution context rather than a cleverer request. They're rare but they're the sites people most often ask why they can't scrape.

How did you measure each site's anti-bot?

Two ways. The top-1,000,000 scan is a single passive homepage GET from a datacenter IP — matching response headers (cf-ray, x-datadome), Set-Cookie names (_abck, datadome, _px) and challenge markers against documented vendor fingerprints, the same signatures as Crawlora's anti-bot checker. For the curated set we go further: any site that blocks the datacenter GET is re-probed through the full transport fleet (browser-impersonation → headless → stealth + residential), so its difficulty tier reflects what actually reaches the page, not a header guess. Every result is a lower bound.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

India, Not the US, Has the Most AI Developers on GitHub

Tony Wang — Tue, 11 Aug 2026 08:51:28 +0000

Key takeaways

India has more geo-resolved GitHub developers tagged ml-ai than the United States — 4,810 versus 3,536 — a reversal of the overall GitHub population, where the US leads India 43,996 to 32,915.
28,603 of the 972,576 profiles in Crawlora's GitHub Users census (2.9%) carry the ml-ai interest-domain tag, inferred from repo topics and bio text.
ML/AI-tagged developers are dramatically more reachable than the general GitHub population: 56.5% expose a public contact channel versus 21.2% overall, and 21.1% mark themselves open to hire versus 7.5% overall.
Academic institutions crack the top employer/affiliation mentions for this group in a way they don't for GitHub overall — Northeastern University, Stanford, CMU, Tsinghua and Peking University all appear in the top ranks, alongside Microsoft, NVIDIA, Red Hat and Google.
Follower counts skew higher too: 90.5% of ML/AI-tagged developers sit in the lowest (nano, <100) follower tier, versus 95.3% for GitHub overall — the whole distribution shifts up a notch, though the single most-followed ml-ai account (38,746 followers) still falls short of GitHub's overall top accounts.

Our census of 972,576 GitHub developer profiles found the US narrowly ahead of India in raw developer count. Filter that same dataset down to the 28,603 profiles tagged ml-ai — Crawlora's inferred interest-domain label for AI/ML-focused developers — and the ranking flips: India has more geo-resolved AI/ML developers than the United States.

Method at a glance

Same census, one filter: the same 972,576-profile GitHub Users dataset behind our earlier study, filtered to domain=ml-ai — an interest-domain tag inferred from a developer's repo topics and bio text, not a self-declared label.

28,603 profiles carry the tag (2.9% of the full census). Geography figures below are over the subset of those that geo-resolve to a country/city, same caveat as the parent study: most developers don't publish a location at all.

Total vs. queryable: 972,576 total indexed profiles; a default search returns 970,617 — the 1,959 accounts flagged as suspected automation (mass-repo commit farms) are excluded by default. Baseline percentages compared against "GitHub overall" below use that 970,617 queryable base, the same denominator as the parent study.

Reachability and hireability are computed the same way as the parent study — a public contact channel (email, Twitter/X, blog, or social account) and GitHub's own "available for hire" flag, respectively.

Snapshot dated July 2026, pulled live from Crawlora's GitHub Users API.

The geography flips: India leads AI/ML, not the US

Across the whole GitHub census, the US leads India in raw developer count. Filtered to ml-ai, India pulls ahead:

By city, Bengaluru tops the list at 518 — consistent with it also leading the overall GitHub census — but the rest of the top cities show the same India-heavy pattern the country totals do: Hyderabad (289), New Delhi (254), Pune (235) and Chennai (218) all place in the global top 10, alongside San Francisco (337), New York (305) and London (304).

Show the top 25 cities by ml-ai-tagged developer count

Top cities by geo-resolved GitHub developers tagged ml-ai

City	Developers
Bengaluru	518
San Francisco	337
New York	305
London	304
Hyderabad	289
New Delhi	254
Pune	235
Chennai	218
Paris	192
Seoul	178
Dhaka	164
Mumbai	161
Toronto	155
Berlin	148
Beijing	143
Kolkata	138
Seattle	128
Lahore	127
Karachi	104
Boston	101
Noida	99
Coimbatore	97
Los Angeles	95
Nairobi	91
Islamabad	87

Top 25 cities by geo-resolved ml-ai-tagged GitHub developers, July 2026. Source: Crawlora GitHub Users dataset, domain=ml-ai filter.

Academia shows up here in a way it doesn't for GitHub overall

Our earlier study found Red Hat and Microsoft topping the employer mentions for GitHub overall, with zero universities in the top 10. Filtered to ml-ai, universities crack the list outright:

Microsoft, NVIDIA and Red Hat lead — the same pattern of open-source-forward companies from our earlier study — but Northeastern University (16 mentions) alone outranks SAP, Deloitte, ByteDance and Huawei among named affiliations, and Stanford, CMU, Tsinghua, Peking and Zhejiang all appear further down the list. AI/ML research still runs substantially through universities, and their researchers' GitHub bios reflect it in a way the general developer population's don't.

This population is dramatically easier to reach and recruit from

The starkest difference between ml-ai-tagged developers and the general GitHub population isn't geography — it's reachability:

56.5% of ml-ai-tagged developers expose some public contact channel, versus 21.2% of the general population — and 21.1% mark themselves open to hire, versus 7.5% overall. Both figures run roughly 2.7-2.8x higher than the GitHub-wide baseline. Activity levels don't explain the gap (93.9% of ml-ai-tagged developers pushed, opened a PR, or reviewed one in the last 90 days, essentially matching the overall population's 93.6%) — this looks like a hiring-market effect: developers building visibly in a hot field have more reason to make themselves findable.

Follower distribution shifts too, though more modestly. 90.5% of ml-ai-tagged developers sit in the nano tier (under 100 followers), versus 95.3% of GitHub overall; the micro tier (100-999) holds 8.3% versus 4.2% overall. The single most-followed developer carrying the ml-ai tag in this dataset — Sebastian Raschka (@rasbt, author of several ML textbooks) at 38,746 followers — doesn't reach GitHub's overall macro-tier leaders, a reminder that broad platform fame and deep specialist reputation are correlated but distinct.

What this means if you're hiring, sponsoring, or building AI tooling

For technical recruiting in AI/ML specifically, India isn't a secondary sourcing market — by this measure it's the largest single geography, ahead of the US. For OSS sponsorship and DevRel aimed at the AI/ML community, the 2.7x reachability gap means outreach lists built from this population will convert on contactability far better than a generic GitHub-wide list would. And for academic partnership or research-collaboration tooling, university affiliations are a real, measurable signal in this specific population in a way they simply aren't for GitHub developers generally.

For a look at how a different specialist population compares, see GitHub's security researchers — the US leads there instead of India, and the follower-tier skew is even steeper than what we found here.

Sources

Frequently asked questions

Does India or the US have more AI/ML developers on GitHub?

India, by this measure. Filtering Crawlora's 972,576-profile GitHub Users census to the 28,603 profiles tagged ml-ai (an interest-domain tag inferred from repo topics and bio text), India has 4,810 geo-resolved developers versus 3,536 for the United States - a reversal of the overall GitHub census, where the US leads India 43,996 to 32,915.

How many GitHub developers work in AI/ML?

28,603 of the 972,576 profiles in Crawlora's GitHub Users dataset (2.9%) carry the ml-ai interest-domain tag. This is inferred from repo topics and bio text, not a self-declared label, so it should be read as a lower-bound signal rather than an exact headcount.

Are AI/ML developers easier to reach and recruit than other GitHub developers?

Yes, substantially. 56.5% of ml-ai-tagged developers expose a public contact channel, versus 21.2% of GitHub overall - and 21.1% mark themselves open to hire, versus 7.5% overall. Both figures run roughly 2.7-2.8x higher than the GitHub-wide baseline, while activity levels (93.9% active in the last 90 days) barely differ from the overall population's 93.6%.

Do universities show up in AI/ML developers' GitHub profiles?

Yes, in a way they don't for GitHub overall. Northeastern University, Stanford, Carnegie Mellon, Tsinghua and Peking University all appear among the top named affiliations for ml-ai-tagged developers, alongside companies like Microsoft, NVIDIA, Red Hat and Google. The overall GitHub census's top 10 employer mentions include zero universities.

Originally published on crawlora.net. Crawlora is a structured web-data, search, and anti-bot API — dozens of platforms as normalized JSON, plus a hosted MCP server, with a free tier (no card).

The Official FIFA World Cup App Scores 2.65 Stars. Its Ticket App Scores 1.79.

Tony Wang — Sun, 09 Aug 2026 01:00:00 +0000

Key takeaways

FIFA runs three separate apps for this World Cup, and its own users rate them nothing alike: the tournament companion app averages 2.65★ on the App Store (712 ratings) and 2.97★ on Google Play; the dedicated ticket-delivery app averages 1.79★ (390 ratings) and 2.21★ on Google Play; FIFA's separate, year-round football app holds 4.62★ (2,163 ratings) and 4.52★ on Google Play.
It got worse as the stakes went up, not better. In the 100 most recent Google Play reviews on each app, the tournament app's 1-star share rose to 58% (from 43.7% lifetime) and the ticket app's rose to 84% (from 76.9% lifetime) — covering exactly the knockout-round and final window.
The ticket-app complaints aren't cosmetic: 41% of the negative reviews we sampled describe being unable to access or transfer a ticket already paid for, and 27% describe being locked out by login or verification failures — on the one app that determines whether you get into the stadium.
Real, dated reviews: one reviewer describes losing $2,100 and being unable to enter a Brazil match; another paid $2,600 and spent 90 minutes and four password resets the night before a flight to the game, still with no tickets. A third's own account of a SeatGeek ticket getting pulled back by the seller five days before delivery matches, almost beat for beat, the 'ghost ticket' pattern NPR reported on this same tournament.
Pulled live via Crawlora's App Store and Google Play endpoints on July 21, 2026, two days after the final. The star histograms are each app's lifetime cumulative distribution; the 'getting worse' read and the complaint themes come from the 100 most recent reviews on each app — a sample, not a full census.

FIFA runs three separate apps for this World Cup, and its own users rate them nothing alike. The tournament companion app averages 2.65 stars. Its dedicated ticket-delivery app — the one that determines whether you actually get into the stadium — averages 1.79, with more than three out of every four ratings at the lowest possible score. FIFA's separate, year-round football app, unrelated to this specific tournament, holds 4.62. Same organization, the same six weeks, a 2.8-star spread. We pulled the App Store and Google Play histograms directly, plus the most recent reviews on each app, to see exactly what's driving it.

What this post draws on

All three star histograms and every review quoted below came from a live pull through Crawlora's App Store (appstore-ratings, appstore-reviews) and Google Play (googleplay-reviews) endpoints on July 21, 2026 — two days after the July 19 final. Histograms are each app's full lifetime rating distribution; the "100 most recent reviews" samples used for the trend and complaint-theme reads are dated snapshots, not a full census. See the methodology section below for exactly how the theme percentages were counted.

Three apps, one World Cup, one 2.8-star spread

FIFA didn't build one World Cup app — it built (or commissioned) three, and split entirely different jobs across them:

App	Role	App Store rating	App Store ratings (n)	Google Play rating
FWC2026 Mobile Tickets	Ticket delivery & stadium entry — the only way in with a purchased ticket	1.79★	390	2.21★
FIFA World Cup 2026	Tournament companion — scores, fixtures, fantasy football	2.65★	712	2.97★
FIFA Official App	Year-round football app — clubs, transfers, unrelated to this tournament	4.62★	2,163	4.52★

The tickets app carries a bundle id of io.tixngo.app.fifatickets — it's a white-labeled product built by a third-party ticketing vendor, TixNgo, not FIFA's in-house app team. That's context, not an excuse: it's still the one app every ticket-holder is forced to use to get through the gate.

The full 1–5 star breakdown makes the gap even clearer than the averages:

Show the full star-by-star breakdown (App Store)

App	1★	2★	3★	4★	5★	Total ratings
FWC2026 Mobile Tickets	300 (76.9%)	10	8	5	67 (17.2%)	390
FIFA World Cup 2026	311 (43.7%)	100	39	50	212 (29.8%)	712
FIFA Official App	130 (6.0%)	16	33	184	1,800 (83.2%)	2,163

It got worse as the stakes went up, not better

If the low ratings were mostly early-tournament teething problems, the most recent reviews should look better than the lifetime average. They look worse:

App	Lifetime 1★ share	1★ share, 100 most recent reviews	Window covered
FWC2026 Mobile Tickets	76.9%	84%	May 20 – Jul 17, 2026
FIFA World Cup 2026	43.7%	58%	Jul 12 – Jul 20, 2026

The tournament app's most recent 100 Google Play reviews land almost entirely inside the knockout rounds and the final itself — exactly the window when usage, and the cost of the app failing, is highest. The tickets app's recent window runs from late May through five days before the final, covering the exact stretch NPR and other outlets were reporting ticket-delivery problems across the tournament.

What people are actually angry about

For the ticket app, the complaints aren't cosmetic — they're about whether the ticket you paid for actually gets you into the building:

What the negative review describes	Share of sampled 1–2★ reviews
Can't access or transfer a ticket already paid for	41.2%
Locked out by login or verification failures	27.1%
Confusing navigation or design	18.8%
Slow to load or freezes	12.9%

The reviews are specific and dated. One App Store review from June 19 reads simply: "Lost 2100 Dollars worth of tickets, CANNOT GET INTO BRAZIL GAME." Another, from July 1, describes paying $2,600 for tickets, then spending 90 minutes and four password resets trying to retrieve them the night before a flight to the match — "still no tickets so I will try again tomorrow. #Fail." A June 28 review: "total failure doesn't work on Samsung Galaxy S24 Ultra. Can not even verify login. Timeout error." A July 6 Google Play review: "day of match, app won't work — can't access the tickets I purchased."

One July 2 App Store review is essentially a first-person case study of the "ghost ticket" pattern NPR reported on this same tournament: the reviewer bought Senegal-vs-Belgium tickets through SeatGeek on June 12 as a birthday present for their son, was told the tickets would be available June 30, and found nothing on either the FIFA app or website that day or the next. At the stadium box office on match day, they learned the reseller had taken the seats back on June 25 — five days before FIFA had told them delivery would happen.

The tournament app's complaints run a different pattern — less about access, more about the app simply working:

What the negative review describes	Share of sampled 1–2★ reviews
Confusing navigation or design	23.7%
Slow to load	11.9%
Crashes or freezes	8.5%

"The app functions like it's been rushed through development," reads one July 3 review. "I've had to update the app twice in the past two days. For an app that only needs to be used for three weeks. It's inexcusable." A July 19 review, the day after the final: "here all is messed up nothing can be done by this app. no live match video no score post nothing." Others describe the app draining 60% of their phone's battery running in the background during a match, and one reviewer flags that ticket access is buried inside a separate app entirely — "it's also mysteriously buried so far down into the app without proper navigation."

It isn't unanimous complaint — both apps have five-star reviews too, including a couple of workaround tips from users who found the login loop by turning off their VPN, or by transferring tickets to a second account when the app claimed they were "already logged in elsewhere." But those are the exceptions in the sample, not the rule.

This isn't the first sign of ticketing trouble this tournament

Crawlora already covered the pricing side of this story: final tickets that reached $2.3 million on FIFA's own resale marketplace, a first-time dynamic-pricing system, and the "slow ticketing" pattern of metering out inventory in batches. That post used anti-bot scanning and contemporaneous reporting — Al Jazeera, Newsweek, NPR — as its evidence. This one uses a completely different data source, structured app-review mining, and lands on the same underlying story from the other side: real people, with real money on the line, unable to get a purchased ticket to actually work.

How we measured this

Pulled live via Crawlora's App Store (appstore-ratings for the full histogram, appstore-reviews for individual review text) and Google Play (googleplay-reviews) endpoints on July 21, 2026, US storefront. The three apps: FIFA World Cup 2026 (App Store id 6476561442 / Google Play com.fifa.tournament), FWC2026 Mobile Tickets (App Store id 6532603739 / Google Play io.tixngo.app.fifatickets), and FIFA Official App (App Store id 756904853 / Google Play com.fifa.fifaapp.android).

Star histograms are each app's full lifetime rating distribution as Apple reports it — a complete population, not a sample. The "100 most recent reviews" figures are exactly that: the newest 100 Google Play reviews returned per app at pull time, sorted newest-first, which is why the two apps' windows cover different date ranges (the tickets app's most recent 100 reviews stretch back to late May; the tournament app's only back to July 12 — a sign the tournament app is simply getting reviewed far more often right now). Complaint-theme percentages are keyword-matched counts over the 1–2 star reviews with review text in that same 100-review sample, not the full lifetime population — treat them as directional, not exhaustive, and expect some reviews to touch more than one theme.

Sources

Frequently asked questions

How bad are the ratings for FIFA's official World Cup 2026 app?

Bad, and getting worse. The tournament companion app ("FIFA World Cup 2026," with scores, fixtures and fantasy football) averages 2.65 stars on the App Store across 712 ratings and 2.97 on Google Play. In the 100 most recent Google Play reviews (July 12–20, 2026), 58% were 1-star — worse than its 43.7% lifetime rate, meaning the app got worse, not better, as the stakes rose through the knockout rounds.

What's FIFA's worst-rated World Cup app?

FWC2026 Mobile Tickets — the separate app that's the only way to access and enter with a purchased ticket. It averages 1.79 stars on the App Store (390 ratings, 76.9% of them 1-star) and 2.21 on Google Play. In its 100 most recent Google Play reviews, 84% were 1-star.

Why is FIFA's ticket app rated so much worse than its tournament app?

The complaints are functional, not cosmetic. Of the negative reviews sampled, 41.2% specifically describe being unable to access or transfer tickets already paid for, and 27.1% describe being locked out by login or verification failures — both on the one app that determines whether you get into the stadium. Real examples: one reviewer describes losing $2,100 and being unable to enter a Brazil match; another paid $2,600 and spent 90 minutes and four password resets still without tickets the night before a flight to the game.

Does FIFA have any well-rated apps?

Yes — its separate, year-round "FIFA Official App" (club football, transfers, fantasy, unrelated to this specific tournament) holds 4.62 stars on the App Store across 2,163 ratings and 4.52 on Google Play, with only 6.0% of ratings at 1-star. The contrast shows the problem is specific to the two tournament-built apps, not FIFA's app-development capability generally.

What are people actually complaining about in the World Cup tournament app?

Mostly navigation and performance, not access. In the sample of negative reviews, 23.7% describe confusing navigation or UI, 11.9% describe slow loading, and 8.5% describe crashes or freezes. Several reviewers also flag heavy battery drain from background activity and delayed live-score notifications.

How was this measured?

Pulled live via Crawlora's App Store ratings/reviews and Google Play reviews endpoints on July 21, 2026, two days after the July 19 final. The star histograms are each app's lifetime cumulative rating distribution. The "getting worse over time" reads come from the 100 most recent reviews on each app at pull time — a recent sample, not a full census — and the complaint-theme percentages are keyword-tagged counts over the negative (1–2 star) reviews with review text in that same sample.

Originally published on crawlora.net. App Review Analyzer is Crawlora's free tool for pulling and summarizing App Store reviews for any app.

YouTube's Top Comment on the World Cup Final Cites a Stat. TikTok's Top Comment Is About Trump.

Tony Wang — Sat, 08 Aug 2026 01:00:00 +0000

Key takeaways

We sampled the top comments on the World Cup final's two highest-view YouTube highlight videos (FIFA's own, 12.6M views, and Fox Sports', 2.9M) and three of the highest-view TikTok clips about the match. The two platforms argue about completely different things.
On YouTube, 16.7% of the 60 comments sampled independently cite the same fact — that Argentina had zero shots on target — and 26.7% use flat dominance/deserved-it framing. The single most-liked comment (14,000 likes), on FIFA's own highlights video: "The most deserved win at the whole tournament. Absolute domination."
On TikTok, it depends entirely on which clip you land on. On Fox Sports' trophy-ceremony clip (996K plays, 1,701 comments), the two most-liked comments by a wide margin — 7,430 and 4,247 likes, versus 625 for the next-highest — are both about Donald Trump's presence at the ceremony, not the football. A third of the top comments we sampled there mention him by name.
The stat that dominates YouTube shows up on TikTok too — contested. One long-form comment argues Argentina "let the match go," citing the same zero-shots-on-target fact as evidence of a conspiracy rather than dominance.
A different TikTok video — an account posting under the name "Christiano Ronaldo" (the misspelling suggests it isn't the real player) congratulating Spain — pulled 1.9M plays, and its comments are dominated by a Messi-vs-Ronaldo-vs-Yamal generational debate, plus at least one commenter asking outright whether the account is "AI or it's real."
Pulled live via Crawlora's YouTube and TikTok comment endpoints on July 21, 2026, two days after the final. Samples are each platform's top-ranked comments (by relevance/likes), not a full census — see the methodology section for exact counts and a pagination-overlap dedup we had to apply.

Ask what people actually said about the World Cup final, and the answer depends entirely on which app you open. We pulled the real comment threads under the final's biggest videos — not hashtag totals, not post counts, the actual top comments people are reading — on YouTube and TikTok. YouTube argues the result with a stat. TikTok, on one of its biggest clips, barely mentions the result at all.

What this post draws on

All comments and view/like counts came from a live pull through Crawlora's YouTube (youtube_search, youtube_comments) and TikTok (tiktok_search, tiktok_comments) endpoints on July 21, 2026, two days after the July 19 final. Every theme percentage below is counted over the specific sample described in that section — see "How we measured this" for exact video IDs, sample sizes, and a real gotcha (paginated comment calls returning duplicate rows) we had to filter out before counting anything.

YouTube: the same stat, independently, over and over

Search YouTube for the final and the highest-view results are exactly what you'd expect: FIFA's own highlights reel (12.6M views) and Fox Sports' (2.9M). What's less expected is how uniform the top comments are. Commenters on both videos, independently, keep reaching for the same piece of evidence:

What the comment argues	Share of 60 sampled comments
General dominance / "deserved it"	26.7%
Cites a shots-on-target stat	16.7%
Credits or blames the goalkeeper	13.3%
Argentina framed as dirty/violent	11.7%
"Saved football" / moralizing	10.0%

The single fact commenters keep independently reaching for: Argentina didn't register a shot on target. It shows up phrased at least four different ways — "Argentina made history by failing to register a single shot on target in a World Cup final" (4,200 likes), "Argentina having 0 shots on target until the last 5 minutes is insane" (9,200 likes), "Reminder that Argentina had 0 shots on target in 120 min" (2,600 likes), and as a joke — "Spain's goalie could have worked remotely" (3,900 likes). Nobody is quoting each other; these are independent top-level comments on two different videos, converging on the same number.

Layered on top of the stat is a moral framing that goes well past "Spain played better": "Argentina did not deserve to play in the World Cup final because they relied on excessive physicality, dirty play, and gamesmanship rather than fair competition" pulled 7,100 likes and 249 replies — one of the most-discussed comments in the sample. And the single most-liked comment on FIFA's own video isn't about a player at all: "I've never been so satisfied in a long time, thank you Spain for saving football" (12,000 likes). A near-identical line — "thank you Spain for saving football" — shows up independently on the Fox Sports video too, at lower volume. It isn't one viral phrase copy-pasted; it's the same sentiment arrived at separately, twice.

TikTok: it depends which clip you land on

TikTok doesn't converge the way YouTube does — the dominant topic changes entirely depending on which specific video you open, even for the same match:

Video (plays)	What the top comments are actually about
Fox Sports trophy-ceremony clip (996K)	Trump's presence at the ceremony
"Christiano Ronaldo"-named account, congratulating Spain (1.9M)	Messi vs. Ronaldo vs. Yamal — and whether the account itself is real
Yamal-focused edit (2.3M)	Generational awe that Yamal is 19

Fox Sports' own trophy-lift clip is the sharpest case. It has 1,701 comments total; we sampled the top 21. The two most-liked by a wide margin — "bro i'm crying, yall saw Trump tryna be involved in the trophy lift" (7,430 likes) and "why is trump there" (4,247 likes) — are both about the former US president standing near the trophy, not the match. The next-highest comment on any topic has 625 likes. A third of the sample (7 of 21) mentions Trump by name at all: "Who invited Donald trump," "The way they zoom in so you can't see Trump standing there," "trump WANTED TO BE Invited into their Celebration.... But NO." This is the same moment Crawlora's social-listening study already traced spreading across Reddit and dedicated TikTok reaction videos within four minutes of the live broadcast — what this comment sample adds is that the reaction wasn't confined to purpose-made reaction content. It's the top comment under a neutral, official trophy-lift highlight clip that isn't about Trump at all.

The other two videos show TikTok can converge too, just on something else entirely. The account posting as "Christiano Ronaldo" — a misspelling that suggests it isn't the real Cristiano Ronaldo — racked up 1.9M plays on a simple congratulations post, and its comments are almost entirely a Messi-vs-Ronaldo-vs-Yamal generational argument: "messi❌ c.Ronaldo✅ yamal✅," "yamal is better than ronaldo," "I don't like Argentina and Messi." One commenter cuts through it directly: "AI or it's real" — a real, live question about whether a viral World Cup account is even who it claims to be.

The contested stat

The "zero shots on target" fact that YouTube treats as settled evidence of dominance gets argued with on TikTok. One reply on the Fox Sports clip, unprompted and at length:

"I genuinely believe Argentina let that match go... What I find hard to believe is that a team with Argentina's quality finished with 0 shots on target and 0 real attacking threat. That just doesn't add up to me. [...] maybe Messi wanted Lamine Yamal to lift the trophy, maybe something else happened I don't know."

Another, shorter one on the same video: "it was rigged because the referee did not foul the Spain players." Neither comment disputes the underlying number — both treat it as suspicious rather than as proof, which is the opposite read from every high-vote YouTube comment citing the same fact. Same statistic, opposite conclusion, different platform.

How we measured this

Pulled live via Crawlora's youtube_search (sorted by view count, query "world cup 2026 final Spain Argentina highlights") and youtube_comments, plus tiktok_search (keyword "world cup 2026 final") and tiktok_comments, on July 21, 2026, US/English locale.

YouTube sample (n=60): the full top-comment set returned for two videos — FIFA's "Highlights | Spain 1-0 Argentina | FIFA World Cup 2026™ FINAL" (12,598,306 views, video id 6HaHNYjnghE) and Fox Sports' "Spain vs Argentina Highlights" (2,899,950 views, video id x-cpRHf4xd4). Both videos' comment streams were exhausted (the API returned an empty continuation token), so this is each video's complete top-ranked comment set at pull time, not an arbitrary cutoff. Theme percentages are keyword/pattern-matched over the full text of all 60; categories overlap by design (a single comment can cite the shots-on-target stat and call Argentina dirty in the same sentence), so the five shares don't sum to 100%.

TikTok sample: three videos identified from a view-sorted search — Fox Sports' trophy-ceremony clip (996,100 plays, aweme id 7664376797336751391, 1,701 total comments, top 21 unique comments sampled after dedup), the "Christiano Ronaldo"-named account's congratulations post (1.9M plays, aweme id 7664437295822540046), and a Yamal-focused edit from account "Conditinho" (2.3M plays, aweme id 7664394784974376224).

A real gotcha, worth flagging: paginating TikTok's comments endpoint two pages deep on the same video returned two comments that were exact duplicates of ones already seen on page one — same author, same text, same like count. This is the same overlapping-pagination behavior Crawlora's social-listening study already documented for TikTok's search and hashtag endpoints; it applies to the comments endpoint too. We deduped by comment text before counting anything, and the 21-comment TikTok sample and the Trump percentage above are both post-dedup.

Sources

Frequently asked questions

What do YouTube commenters say about the World Cup final?

They build a stats-driven moral case for Spain. Of 60 top comments sampled across FIFA's official highlights video (12.6M views) and Fox Sports' (2.9M), 26.7% use general dominance/"deserved it" framing, 16.7% independently cite that Argentina had zero shots on target, and 11.7% frame Argentina as dirty or violent. The single most-liked comment (14,000 likes): "I've never been so satisfied in a long time, thank you Spain for saving football."

What's the most common fact cited in the World Cup final's YouTube comments?

That Argentina registered zero shots on target. It shows up phrased at least four different ways across two different videos — as a stat ("Argentina made history by failing to register a single shot on target," 4,200 likes), as a joke ("Spain's goalie could have worked remotely," 3,900 likes), and combined with other stats ("more red cards than attempts at goal," 963 likes) — commenters reaching the same number independently, not copying one viral phrase.

What are TikTok commenters saying about the World Cup final?

It depends entirely on which clip. On Fox Sports' trophy-ceremony highlight (996K plays), the two most-liked comments by a wide margin (7,430 and 4,247 likes, versus 625 for the next-highest) are both about Donald Trump's presence at the ceremony, not the match. On a different clip — an account posting as "Christiano Ronaldo" congratulating Spain (1.9M plays) — the comments are a Messi-vs-Ronaldo-vs-Yamal generational debate instead.

Is the Trump reaction to the World Cup final trophy ceremony really that dominant on TikTok?

On the specific clip we sampled, yes — a third of the top 21 comments (post-dedup) mention him by name, and critically, the top two comments by likes are both about him. This is the same trophy-ceremony reaction Crawlora's social-listening study traced spreading across Reddit and dedicated TikTok reaction videos within four minutes of the live broadcast; this sample shows it also dominates the comment section of a neutral, official highlight clip that isn't about Trump at all.

Who is the "Christiano Ronaldo" TikTok account posting about the World Cup?

Unconfirmed — the misspelling (the real player's name is "Cristiano") suggests it is not Cristiano Ronaldo's real account, though we did not independently verify its identity. Its congratulations-to-Spain post pulled 1.9M plays, and at least one commenter directly asked "AI or it's real," questioning the account's authenticity in the comments themselves.

How was this measured?

Pulled live via Crawlora's YouTube (search + comments) and TikTok (search + comments) endpoints on July 21, 2026, two days after the July 19 final. The YouTube sample (n=60) is each of two videos' complete top-ranked comment set at pull time (the API's continuation token was exhausted, not an arbitrary cutoff). The TikTok sample is the top comments on three view-sorted videos, deduped after paginating turned up exact-duplicate comments on a repeat page — the same overlapping-pagination behavior already documented for TikTok's search endpoint, confirmed here to also affect comments.

Originally published on crawlora.net. YouTube API returns video metadata, comments, and transcripts as structured JSON — no scraping required.

Every World Cup Since 2014, Visa's SEC Filings Call It Out By Name — Marketing Spend Swings From -9% to +44%

Tony Wang — Fri, 07 Aug 2026 01:00:00 +0000

Key takeaways

Visa's SEC filings have named the FIFA World Cup as a driver of marketing-expense growth in every tournament cycle since at least 2014 — four World Cups running, always inside the "Marketing expenses" discussion of a 10-Q or 10-K.
The size of that swing is not a clean escalation. Marketing expense fell 9% year-over-year in the quarter tied to Brazil 2014 (a tough comp against the prior year's Sochi Olympics spend), then rose 8% for Russia 2018, 18% for Qatar 2022, and 44% for the 2026 tournament — its biggest jump of the four, and the only quarter that isn't final yet.
Every single cycle credits the World Cup jointly with that year's Winter Olympics, never alone. Visa sponsors both, and since 1994 the Winter Games and World Cup have always shared a calendar year — no filing isolates a World-Cup-only dollar figure.
Qatar 2022 is the only cycle where the reporting quarter fully contains the tournament dates (Nov 20 – Dec 18, 2022 sits entirely inside Visa's fiscal Q1 2023). The other three cycles' figures come from quarters that only partly overlap, or precede, kickoff.
The 44% figure for 2026 is a leading indicator, not a final result — it's from the quarter before the tournament (January–March 2026). The quarter that actually contains the June 11–July 19 tournament isn't due until early August 2026.

Visa's SEC filings have named the FIFA World Cup as a reason its marketing spend moved — every tournament cycle since at least 2014. We pulled the exact sentence from Visa's 10-Q or 10-K for Brazil 2014, Russia 2018, Qatar 2022, and the ongoing 2026 tournament, along with the dollar figures behind each one. The naming pattern holds for over a decade. The dollar swing behind it does not: marketing expense fell 9% year-over-year the quarter tied to 2014, then rose 8%, 18%, and 44% in the three cycles since.

What this post draws on

All filing text below is quoted directly from Visa Inc.'s (SEC CIK 1403161) own EDGAR filings — one 10-Q or 10-K per World Cup cycle since 2014 — pulled live via Crawlora's SEC filing-sections endpoint. Two caveats up front: the comparison windows aren't perfectly uniform, since each row uses whichever filing actually contains Visa's World Cup sentence, and that sentence lands in a different position relative to kickoff every cycle (exact dates in the table below); and every quote credits the World Cup jointly with that year's Winter Olympics, never in isolation — see "The World Cup never travels alone" below for why.

The pattern, cycle by cycle

Cycle	Host	Tournament dates	Filing	Period covered	Marketing expense	Prior year	Change
2014	Brazil	Jun 12 – Jul 13, 2014	10-Q, filed Jul 24, 2014	Q3 FY2014 (3 mo. to Jun 30)	$228M	$252M	−9%
2018	Russia	Jun 14 – Jul 15, 2018	10-Q, filed Jul 27, 2018	Q3 FY2018 (3 mo. to Jun 30)	$240M	$221M	+8%
2022	Qatar	Nov 20 – Dec 18, 2022	10-Q, filed Jan 27, 2023	Q1 FY2023 (3 mo. to Dec 31)	$332M	$280M	+18%
2026	USA / Mexico / Canada	Jun 11 – Jul 19, 2026	10-Q, filed Apr 29, 2026	Q2 FY2026 (3 mo. to Mar 31)*	$545M	$381M	+44%

*Pre-tournament quarter — the quarter that actually contains the 2026 tournament hasn't been filed yet.

What each filing actually says

2014 (Brazil) — 10-Q for the quarter ended June 30, 2014, filed July 24, 2014:

"Marketing decreased during the three month comparable period and increased over the nine month comparable period primarily due to elevated spend supporting the 2014 Sochi Winter Olympics campaign beginning in the third quarter of fiscal 2013 through the first half of fiscal 2014. The nine month comparable period also increased due to spend supporting the 2014 FIFA World Cup campaign."

2018 (Russia) — 10-Q for the quarter ended June 30, 2018, filed July 27, 2018:

"Marketing expenses increased primarily due to the higher level of spending to support a number of campaigns, including the PyeongChang 2018 Olympic Winter Games and 2018 FIFA World Cup."

2022 (Qatar) — 10-Q for the quarter ended December 31, 2022, filed January 27, 2023:

"Marketing expenses increased primarily due to increased spending in various campaigns, including the FIFA World Cup 2022™ and client marketing."

2026 (USA/Mexico/Canada) — 10-Q for the quarter ended March 31, 2026, filed April 29, 2026:

"Marketing expenses increased over the three and six-month prior-year comparable periods primarily due to higher spending for client marketing and various campaigns, both driven in part by the Olympic and Paralympic Winter Games Milano Cortina 2026 and the FIFA World Cup 2026™."

Four filings, four different fiscal quarters, one recurring sentence.

The World Cup never travels alone

Notice what's in every quote above: the World Cup is never the only event named. It's always paired with that year's Winter Olympics.

World Cup cycle	Same-fiscal-year Winter Olympics
2014 (Brazil)	Sochi 2014 (February)
2018 (Russia)	PyeongChang 2018 (February)
2022 (Qatar)	Beijing 2022 (February)
2026 (USA/Mexico/Canada)	Milano Cortina 2026 (February)

That's not a coincidence specific to Visa — since 1994, the Winter Olympics and the World Cup have shared the same calendar year every time, both running on their own four-year cycles offset from the Summer Olympics. Visa sponsors both events, and its October–September fiscal year always contains both in the same reporting window. The practical result: no Visa filing anywhere in this dataset isolates a World-Cup-only marketing figure. Every dollar change above is a joint attribution, and there's no clean way to split it from the filing text alone.

2026 is still a leading indicator

The 44% figure is the biggest of the four cycles, but it's also the least finished. It comes from Visa's fiscal Q2 2026 (January–March) — the quarter before the tournament's June 11 opening match. By contrast, 2014's and 2018's quarters ran through most of their tournaments, and 2022's quarter (October–December) fully contained Qatar's entire November 20 – December 18 window.

Visa's fiscal Q3 2026 10-Q — the filing that will actually cover the June 11–July 19 tournament window — is due in late July or early August 2026, roughly following the same ~30-day filing lag as its Q1 FY2023 report after Qatar. Until then, 44% is what Visa's pre-tournament ramp-up looked like, not what the World Cup itself did to its books.

How we measured this

Pulled directly from Visa Inc.'s (CIK 1403161) 10-Q and 10-K filings via Crawlora's SEC filing-sections endpoint, one filing per World Cup cycle — specifically, the fiscal quarter (or year, checked for 2022) in which Visa's own Management's Discussion and Analysis names the FIFA World Cup as a driver of marketing-expense growth. Accession numbers for the two oldest filings (2014, 2018) were located via SEC's company-submissions filing index rather than EDGAR's full-text search, which is tuned to surface recent filings first. Every figure is the as-reported GAAP marketing-expense line item — not adjusted or non-GAAP — read directly from each filing's Operating Expenses table.

Sources

Frequently asked questions

Has Visa mentioned the World Cup in its SEC filings before 2026?

Yes. Visa's own 10-Q and 10-K filings have named the FIFA World Cup as a driver of marketing-expense growth in at least four consecutive cycles: 2014 (Brazil), 2018 (Russia), 2022 (Qatar), and 2026 — always inside the "Marketing expenses" discussion of its Management's Discussion and Analysis.

How much did Visa's marketing spend change in each World Cup cycle?

Comparing the same fiscal quarter year-over-year: -9% in 2014 ($228M vs $252M), +8% in 2018 ($240M vs $221M), +18% in 2022 ($332M vs $280M), and +44% in 2026 ($545M vs $381M, though the 2026 figure is from a pre-tournament quarter).

Why did Visa's marketing spend fall during the 2014 World Cup quarter?

Because the year-over-year comparison quarter (fiscal Q3 2013) had unusually high marketing spend of its own, tied to the build-up for the 2014 Sochi Winter Olympics. Visa's own filing says marketing spend fell for the quarter but rose over the nine-month period, "due to spend supporting the 2014 FIFA World Cup campaign."

Does Visa break out how much it spends on the World Cup specifically?

No. Every filing credits the World Cup jointly with that year's Winter Olympics — Visa sponsors both, and because the Winter Olympics and World Cup have shared the same calendar year since 1994, they always land in the same fiscal-year reporting window. No filing isolates a World-Cup-only dollar figure.

Is the 2026 44% marketing jump the final number for this World Cup?

No. It's from the quarter ended March 31, 2026 — before the tournament started on June 11. Visa's next 10-Q, covering the quarter that actually contains the tournament, is due around late July or early August 2026.

Where does this data come from?

Directly from Visa Inc.'s (SEC CIK 1403161) 10-Q and 10-K filings, pulled via Crawlora's SEC filing-sections API — the same "Marketing expenses" line item and MD&A narrative Visa reports to the SEC every quarter. Accession numbers for the 2014 and 2018 filings were located via SEC's company-submissions filing index rather than full-text search.

Originally published on crawlora.net. Google Finance API returns structured company financials, market data, and news as JSON for sponsorship and market research.

The Job Market Runs on Workday: 63% of Open Reqs

Tony Wang — Thu, 06 Aug 2026 01:00:00 +0000

Key takeaways

1,128,909 open job postings, crawled from 11,703 company career boards across the 12 hosted applicant-tracking systems we cover. One vendor — Workday — carries 713,606 of them: 63.2% of every posting, by volume.
That is not the same as market share. The State of ATS 2026 report puts Workday at ~39% of large employers by company count. It runs 63% of the reqs because the companies on it are the giant hirers — CVS Health alone posts ~17,400 open roles from a single Workday board.
The startup-ATS world developers picture as 'the job market' is a rounding error by volume. All 1,063 Ashby boards — OpenAI, Notion, Ramp and every other high-growth startup on it — hold 24,844 postings between them. Domino's Pizza's single SmartRecruiters board holds 24,427. One pizza franchise ≈ the entire Ashby ecosystem.
The ATS a company runs is a size signal. Enterprise HCM boards (Workday, Oracle, SmartRecruiters) average 150–260 open reqs each; startup and SMB boards (Ashby, Personio, Rippling, Recruitee) average 11–23 — a ~10× gap.
It is not a tech job market. The largest boards are healthcare, retail and field services — CVS Health, BAYADA Home Health, Domino's, EōS Fitness, Carvana. Engineering is the single largest named department (13,626 roles) but a thin slice of 1.13M.
ATS-sourced and Common-Crawl-discovered: this is a large sample of companies that post through these 12 boards, not all hiring. Giants on internal systems (Amazon, Google, Meta) and vendors we don't crawl (iCIMS, SuccessFactors, Taleo) are absent. Aggregate/structural only.

Most writing about "the ATS market" counts companies: which applicant-tracking system each employer runs. That view is genuinely fragmented — the State of ATS 2026 report puts Workday at 38.6% of large employers, Greenhouse at 13.3%, and a long tail behind them.

Count open requisitions instead — the jobs themselves — and the market collapses into one vendor. We crawled 1,128,909 open postings from 11,703 company career boards, and 63.2% of them run through Workday. The gap between those two numbers is the whole story.

Method at a glance

Source: Crawlora's Jobs dataset — live postings crawled directly from companies' own applicant-tracking boards across 12 hosted ATS providers (Workday, SmartRecruiters, Greenhouse, Ashby, Oracle, UKG, Teamtailor, Personio, Rippling, Recruitee, Workable, Lever). Query it yourself via the free job search tool or the Jobs API. Snapshot observed mid-July 2026.

Coverage caveat, up front. Boards are discovered from Common Crawl, so this census is companies that run a public, hosted ATS board on one of these 12 providers — a real, large sample, but not all hiring. It structurally under-counts (a) the tech giants on internal systems — Amazon, Google, Meta, Apple all run proprietary boards; (b) ATS vendors we don't crawl — iCIMS, SAP SuccessFactors, Taleo, Avature, Eightfold, which the State of ATS report shows are common among large employers; and (c) postings that live only on LinkedIn, Indeed or a company's hand-rolled page. Every provider share below is share within these 12 hosted ATSs, not share of all hiring.

Boards, not unique companies. Some employers appear as more than one board — the same tenant gets crawled under case-variant slugs (CVS Health's Workday board shows up twice, as cvs_health_careers and CVS_Health_Careers). So the 11,703 board count is an upper bound on distinct companies. Posting-level counts (the provider volumes) are unaffected by this; the top-board table below is de-duplicated by hand.

What we don't measure: salary (rarely populated in the source postings), resolved legal-entity identity, or application outcomes. We report structure — how postings distribute across providers, boards and functions — not individual applicants.

Third-party context (the ATS-by-company-count figures) comes from the State of ATS 2026 report by ResumeAI, which verified 713 large employers against their live careers portals. Everything else here is first-party from the Crawlora dataset.

The job market, by volume, is a Workday market

Workday carries 713,606 of the 1,128,909 postings — 63.2%. Add SmartRecruiters (202,448) and the two enterprise HR-suite vendors alone hold 81% of everything we crawl. The three "startup" ATSs that dominate developer mindshare — Greenhouse, Ashby and Lever — hold 12.5% combined, and most of that is Greenhouse serving mid-to-large tech.

This is a supply-side census, so read it the way you'd read any supply map: it tells you where the open reqs are, not where the good jobs are. And the open reqs are overwhelmingly at large enterprises running Workday.

Volume is not the same as vendor share

Here is the trap. If you rank ATSs by how many companies use them, you get a fragmented, competitive market. If you rank them by how many jobs are actually posted, you get near-monopoly. Both are true at once, and the difference is company size:

ATS share by company count vs. by posting volume

ATS vendor	Share by company count	Share by posting volume
Workday	38.6%	63.2%
Greenhouse	13.3%	10.3%
SmartRecruiters	3.3%	17.9%
Oracle Cloud HCM	6.3%	1.8%
Ashby	2.1%	2.2%
Lever	2.1%	0.01%

Company-count share = State of ATS 2026 (ResumeAI), 713 portal-verified large employers. Posting-volume share = Crawlora Jobs dataset, 1,128,909 postings on the 12 hosted ATSs we crawl. The two universes differ — ResumeAI includes iCIMS, SuccessFactors and internal systems we don't, and our board pool skews smaller and more numerous — so this is a directional contrast, not a like-for-like restatement.

The comparison isn't apples-to-apples — the two datasets cover different universes, and I've flagged that in the caption rather than pretend they line up. But the direction is unmistakable and it holds within the overlapping vendors: Workday's grip on the req market (63%) is far tighter than its grip on the employer market (~39%), and SmartRecruiters is the mirror image — a small slice of companies (3.3%) but a huge slice of postings (17.9%), because the companies it serves are massive, high-churn hourly-hiring operations (Domino's, Accor, Bosch, Greene King).

Why the skew? Because a Workday customer is, on average, an enterprise with tens of thousands of employees and constant backfill. CVS Health posts ~17,400 open roles from one board. PwC's boards run to ~4,600 apiece. Northrop Grumman, Thermo Fisher, DaVita, Micron — each carries thousands. A typical Ashby customer is a startup with 15 open roles. Counting companies, they're one-to-one. Counting jobs, one CVS erases hundreds of startups.

The ATS you run reveals your size

That size effect shows up cleanly as average open reqs per board. It splits the 12 providers into two tiers with almost no overlap:

Hosted ATS providers — boards, open postings, share, and average reqs per board

Provider	Boards	Open postings	Share	Avg reqs / board	Typical customer
Workday	3,931	713,606	63.2%	182	Enterprise HCM
SmartRecruiters	777	202,448	17.9%	261	Enterprise / high-volume
Greenhouse	2,567	116,001	10.3%	45	Scale-up / tech
Ashby	1,063	24,844	2.2%	23	High-growth startup
Oracle Cloud HCM	136	20,497	1.8%	151	Enterprise HCM
UKG	371	15,205	1.3%	41	Hourly / workforce mgmt
Teamtailor	872	13,500	1.2%	15	SMB (Europe)
Personio	883	9,931	0.9%	11	SMB (Europe)
Rippling	732	8,034	0.7%	11	SMB / startup
Recruitee	296	3,380	0.3%	11	SMB
Workable	73	1,352	0.1%	19	SMB
Lever	2	111	0.01%	56	Startup (under-crawled)

Crawlora Jobs dataset, July 2026. 'Boards' counts discovered career boards, an upper bound on distinct companies (case-variant duplicates inflate it slightly). Lever's near-absence is a discovery artifact — only two Lever boards surfaced in this crawl, so read its row as coverage, not a market verdict.

The enterprise HR suites — Workday (182), SmartRecruiters (261), Oracle (151) — carry 150–260 open reqs per board. The startup and SMB tools — Ashby (23), Personio (11), Rippling (11), Recruitee (11), Teamtailor (15) — carry a tenth of that. If you can see the ATS behind a careers page (the apply-URL host gives it away: myworkdayjobs.com, boards.greenhouse.io, jobs.ashbyhq.com), you can guess the company's stage before you read a word of the posting.

One honest caveat on this table: Lever shows just two boards and 111 postings, which is not evidence Lever is dead — it's a Common Crawl discovery gap for Lever's board structure in this pass. Lever is a real, actively-used ATS; we simply didn't enumerate its boards here, and I'd rather show that hole than paper over it.

Domino's Pizza ≈ every Ashby startup, combined

Concentration inside the census is extreme. A handful of enormous single-employer boards dwarf whole ecosystems:

Largest single career boards by open reqs

Company (board)	ATS	Open reqs	Sector
Domino's	SmartRecruiters	24,427	Food / franchise
CVS Health	Workday	~17,400	Healthcare / retail
Groupement Mousquetaires	SmartRecruiters	9,416	Retail (France)
Accor	SmartRecruiters	6,002	Hospitality
PwC	Workday	~4,600	Professional services
AECOM	SmartRecruiters	4,861	Engineering / infra
Bosch	SmartRecruiters	4,741	Industrial
SGS	SmartRecruiters	4,203	Testing / inspection
Northrop Grumman	Workday	~3,450	Aerospace / defense
Thermo Fisher	Workday	~3,240	Life sciences

Crawlora Jobs dataset, July 2026, de-duplicated across case-variant boards (CVS, PwC, Northrop and Thermo Fisher each surfaced as two near-identical boards; shown once, ~-prefixed). These are careers-board totals, not resolved corporate headcounts.

The number that reframes the whole dataset: Domino's Pizza's one board holds 24,427 open reqs. Every Ashby-powered company on the internet — all 1,063 boards, OpenAI and Notion and Ramp included — holds 24,844. A single pizza franchise's hourly hiring is the size of the entire high-growth-startup ATS ecosystem. The "job market" that gets written about (hot startups, competitive tech roles) and the job market that actually exists by volume (enterprise backfill, hourly, field services) are barely the same market.

It isn't a tech job market

Because Engineering is the largest single named department (13,626 roles), it's tempting to read this as a tech dataset. It isn't. Look at what the biggest boards actually are:

The department field is messy — most postings don't carry a clean one, and the raw facet is full of company-specific labels — so the honest read is the top employers, not the top departments. The biggest boards in this census are CVS Health (pharmacy, retail, care), BAYADA Home Health (2,599), Pulse Healthcare (2,591), EōS Fitness (2,622), Domino's, Carvana (1,849) and Centria Autism (1,185). Healthcare, retail, fitness, field services and hourly operations — the roles a labor economist would call the actual US job market, and the ones tech coverage never mentions. Explicitly-remote postings are a thin 2.6% of the census (29,330 flagged remote), and even that is a floor, since the remote field is unevenly populated.

What this means, depending on who you are

If you're a job seeker or engineer: the boards you browse (Ashby, Greenhouse, Lever) are a curated 12% corner of the posted-req market. If you're optimizing a résumé, optimize it for the parser you'll actually hit — and by volume that's overwhelmingly Workday. The counterintuitive move for reach is to search by ATS provider, not just by title: the free job search tool lets you filter to a provider and see the enterprise reqs that never trend on tech Twitter.

If you're a recruiter or founder: the ATS you pick is read by everyone downstream as a size tell. Standing up an Ashby board signals a 15-role startup; standing up Workday signals an enterprise. That's not a reason to over-buy — it's a reason to know what your careers-page host communicates before you choose it.

If you're a researcher or journalist: the transferable warning is count the right unit. "ATS market share" by company count and by posting volume are different by nearly 25 points for Workday, and reporting one as the other is how "Workday runs the job market" becomes either obviously true or obviously false depending on which you meant. And whatever you're measuring off an ATS census, name what it excludes — internal-system giants and un-crawled vendors are not a rounding error.

Everything here is queryable — by provider, company, department, location or remote flag — over one REST API, with no modelling in between. If you think the coverage gaps change the conclusion, the query that would show it is one call away.

Sources

Frequently asked questions

What is the most common ATS by job posting volume?

Workday, by a wide margin. In Crawlora's July 2026 census of 1,128,909 open postings across the 12 hosted applicant-tracking systems it crawls, Workday carries 713,606 — 63.2% of every posting. SmartRecruiters is second at 17.9% and Greenhouse third at 10.3%. This is share of postings on those 12 hosted providers, not share of all hiring.

Does Workday really have 63% of the ATS market?

Only by posting volume, not by company count. The State of ATS 2026 report (ResumeAI) puts Workday at ~38.6% of large employers by company count. Workday runs a far larger share of the actual open requisitions (63%) because its customers are the giant hirers — CVS Health alone posts around 17,400 open roles from a single Workday board, whereas a typical startup ATS customer has a dozen. Counting companies, they look one-to-one; counting jobs, one enterprise erases hundreds of startups.

Which ATS do startups use, and how big is that segment?

High-growth startups cluster on Ashby and Greenhouse (Lever historically too). But by volume the startup-ATS segment is tiny: all 1,063 Ashby boards combined — OpenAI, Notion, Ramp and every other startup on it — hold 24,844 open postings. Domino's Pizza's single SmartRecruiters board holds 24,427. One franchise's hourly hiring is the size of the entire Ashby ecosystem.

Is the online job market mostly tech jobs?

No. Engineering is the single largest named department (13,626 roles) but that's roughly 1% of 1.13M postings. The largest career boards are healthcare, retail, fitness and field services — CVS Health, BAYADA Home Health, Pulse Healthcare, EōS Fitness, Domino's and Carvana — not tech companies. Explicitly-remote roles are just 2.6% of the census.

What does the ATS a company uses tell you about it?

Its size and stage. Enterprise HR suites (Workday, Oracle, SmartRecruiters) average 150–260 open reqs per career board; startup and SMB tools (Ashby, Personio, Rippling, Recruitee, Teamtailor) average 11–23 — about a 10× gap. The apply-URL host gives the ATS away (myworkdayjobs.com, boards.greenhouse.io, jobs.ashbyhq.com), so you can often infer a company's stage before reading a single posting.

What does this jobs dataset not cover?

It is ATS-sourced and Common-Crawl-discovered: a large sample of companies that run a public, hosted ATS board on one of 12 providers — not all hiring. It under-counts tech giants on internal systems (Amazon, Google, Meta, Apple), ATS vendors Crawlora doesn't crawl (iCIMS, SAP SuccessFactors, Taleo, Avature, Eightfold), and jobs posted only to LinkedIn, Indeed or hand-rolled pages. Board counts are an upper bound on distinct companies because some employers surface as multiple case-variant boards. It is aggregate and structural — no salaries or applicant-level data.

Originally published on crawlora.net. Jobs API aggregates listings across Workday, Greenhouse, Lever, and other ATS platforms as structured JSON.

How to Scrape Zillow in 2026 (API & Python)

Tony Wang — Wed, 05 Aug 2026 01:00:00 +0000

Key takeaways

The fastest way to scrape Zillow is a structured API that returns search results and listing details (price, beds, baths, address) as normalized JSON — no headless browser.
Zillow defends aggressively (CAPTCHAs, IP bans, shifting embedded JSON), so DIY needs proxies, fingerprinting, and a browser cluster at scale.
Public listing facts (price, address, beds/baths) are lower-risk to collect; listing photos and descriptions can be copyrighted — avoid republishing them.
Resolve locations with the autocomplete endpoint, then search by location or fetch a property by ZPID.

The fastest way to scrape Zillow in 2026 is to call a structured Zillow API that returns normalized JSON — search results, listing details, price, beds, baths, and address — instead of parsing Zillow's JavaScript-heavy pages and fighting its anti-bot defenses. You can build a DIY scraper, but Zillow actively blocks automation. This guide covers all three approaches, what each returns, where each breaks, and the legal basics.

Is it legal to scrape Zillow?

Scraping public Zillow listing data (price, address, beds/baths, status) is generally lower-risk public-web scraping — in the US, hiQ v. LinkedIn held that accessing public data isn't a CFAA violation, and facts like prices aren't copyrightable. But Zillow's Terms of Service prohibit automated access, and listing photos and descriptions can be copyrighted, so avoid republishing them. Use public, factual data; respect rate limits; review Zillow's terms. See is web scraping legal. Not legal advice.

Option 1: DIY in Python (and why it breaks)

Zillow renders with heavy JavaScript and defends aggressively, so you reach for a headless browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto("https://www.zillow.com/austin-tx/")
    # then parse the embedded __NEXT_DATA__ JSON, page by map region, and clear the press-and-hold CAPTCHA...

It demos and then breaks — Zillow is rated one of the hardest targets on the web:

PerimeterX (HUMAN) bot detection — behavioral analysis (mouse, keystrokes), browser fingerprinting, HTTP/2 and header-order checks, and a press-and-hold CAPTCHA; you need residential proxies and fingerprint spoofing.
Low safe rate — practitioners cap around 20–50 property-detail pages per day per IP, so scale means many rotating residential IPs.
Shifting embedded JSON — Zillow's client data (a __NEXT_DATA__-style blob) changes shape and breaks parsers.
Bypass rot — open-source PerimeterX bypasses typically last only a couple of months before they're patched.

Option 2: No-code tools

Visual extractors export CSV/JSON and suit one-off pulls, but are less convenient for in-product pipelines with predictable fields.

Option 3: A structured Zillow API

For repeatable workflows, a Zillow scraping API returns normalized JSON with no browser to run. Search by location:

curl "https://api.crawlora.net/api/v1/zillow/search?location=Austin,%20TX" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Fetch a single listing by ZPID in Python:

import requests

prop = requests.get(
    "https://api.crawlora.net/api/v1/zillow/property/12345678",
    headers={"x-api-key": "YOUR_API_KEY"},
).json()["data"]
print(prop.get("address"), prop.get("price"), prop.get("bedrooms"))

A response is normalized JSON you can store directly (fields are illustrative — check the docs):

{
  "code": 200,
  "msg": "OK",
  "data": [
    {
      "zpid": "12345678",
      "address": "123 Example St, Austin, TX",
      "price": 625000,
      "bedrooms": 3,
      "bathrooms": 2,
      "living_area": 1850,
      "status": "FOR_SALE"
    }
  ]
}

Resolve a location with autocomplete first, then search — or fetch one listing by ZPID:

h = {"x-api-key": "YOUR_API_KEY"}
base = "https://api.crawlora.net/api/v1/zillow"

place = requests.get(f"{base}/autocomplete", headers=h, params={"query": "Austin, TX"}).json()["data"]
results = requests.get(f"{base}/search", headers=h, params={"location": "Austin, TX"}).json()["data"]

Autocomplete returns a region_id and map bounds (west/east/south/north); passing those to search is the most stable request shape, so you don't have to guess Zillow's URL slugs.

What you can collect

Where the public listing exposes them: ZPID, address, price, beds, baths, living area, lot size, home type, status, broker, description, media, and estimates — plus the search or ZPID context you requested.

Limitations and common challenges

No open public API. Zillow's official API access is restricted to partners (Bridge / MLS programs), so collecting arbitrary public listings means scraping.
PerimeterX, Very-Hard rating. Zillow runs PerimeterX (HUMAN) with behavioral analysis and fingerprinting; DIY needs residential proxies, fingerprint spoofing, and low per-IP rates, and open-source bypasses rot in months — a structured API absorbs this behind one key.
Listing media is copyrighted. Price, address, beds/baths, and status are facts you can collect; listing photos and agent descriptions carry copyright — don't republish them.
Status changes fast and estimates are models. For-sale/sold/pending flips quickly, so re-scrape on a schedule, and treat Zestimate-style values as Zillow's model output, not ground truth.

Where this gets used

Property market intelligence — track listings, prices, and inventory by market. See the property market intelligence use case.
Comparables & valuation research — pull comparable listings for an area.
Lead and territory mapping — combine with local data for real-estate workflows.

Sources

Sources

Start collecting

Try it first, free: run any public URL through the Free Web Scraper, or check whether a site blocks bots with the Anti-Bot Checker — no signup.

Test the search endpoint in the Playground, check the schema in the API docs, and review pricing. For a full comparison against other Zillow data APIs, see best Zillow scraper APIs in 2026. To cover Redfin, Realtor.com, and Trulia alongside Zillow, see how to scrape real estate listings, or go straight to how to scrape Redfin to cross-check Zestimate against Redfin's own estimate; for the short-term-rental side, how to scrape Airbnb; also how to choose a web scraping API and is web scraping legal.

Part of our how-to-scrape guide series — every platform we cover, in one index.

Frequently asked questions

Can I scrape Zillow without getting blocked?

Zillow is one of the hardest targets on the web — it runs PerimeterX (HUMAN) with behavioral analysis, browser fingerprinting, and a press-and-hold CAPTCHA, so DIY needs residential proxies, fingerprint spoofing, and low per-IP rates (practitioners cap around 20–50 detail pages/day/IP). A structured API handles proxies and browser execution behind the endpoint.

Does Zillow have an official API?

Not an open one. Zillow's API access is restricted to partners (Bridge / MLS programs), so collecting arbitrary public listings means scraping. Crawlora's Zillow endpoints return public search and property data as normalized JSON from one API key.

What Zillow data can I get?

Public listing fields: ZPID, address, price, beds/baths, living area, lot size, home type, status, broker, description, media, and estimates, where available. Facts like price and address carry the least risk; photos and descriptions can be copyrighted.

How do I search a location?

Resolve it with the autocomplete endpoint (query, with status for_sale/for_rent/sold), then pass the returned region_id or map bounds to /zillow/search — the most stable shape — or fetch one listing with /zillow/property/{zpid}.

Can I store listing photos and descriptions?

Those can be copyrighted — treat them carefully and avoid republishing. Factual fields like price and address carry the least risk.

Originally published on crawlora.net. Zillow API returns structured property, price, and estimate data as JSON — no scraping infrastructure required.

Google vs Bing vs Brave: Do Results Match?

Tony Wang — Tue, 04 Aug 2026 01:00:00 +0000

Key takeaways

Three engines, three internets: across the searches where all three answered, Google, Bing, and Brave agreed on the #1 result only 29% of the time, and Google and Bing shared just 3 of the top 10 on average.
Each engine has a personality. Bing rewards traditional publishers (Forbes appeared in 9 of 15 top-10s, PCMag in 8). Google leans on its own properties and forums (Reddit and YouTube each showed up in 57% of Google SERPs, Wikipedia in 43%). Brave is a blend of both.
The platforms Google loves, Bing ignores: Reddit, YouTube, and Wikipedia appeared in 0% of the Bing top-10s we checked.
If you track rankings on one engine, you are blind on the others. Cross-engine divergence is the case for multi-engine SERP monitoring — not a single Google rank check.

Everyone talks about "ranking on Google." But Google is not the only place your customers search, and the other engines do not agree with it — or with each other. We ran the same 15 searches through Google, Bing, and Brave using Crawlora's search APIs and compared the top 10 results. The short version: the three engines return strikingly different pages, reward different kinds of sites, and rarely even agree on what belongs at #1.

How much do the engines overlap?

For each search we took the top 10 result domains from each engine and counted how many they shared. No pair shares even half its results on average, and all three engines agree on fewer than 3 of 10:

Engine pair	Avg shared (of 10)	Overlap
Google ∩ Bing	3.0	30%
Google ∩ Brave	5.1	51%
Bing ∩ Brave	4.0	40%
All three	2.7	27%

A page ranking #3 on Bing might be nowhere on Google. If your rank tracker only watches one engine, most of this picture is invisible to you.

They rarely agree on #1

The single most valuable position — the #1 organic result — matched across all three engines for only 2 of the 7 searches where every engine answered (29%). Here is the real head-to-head:

Search	Google #1	Bing #1	Brave #1	Agree?
best running shoes	runrepeat.com	wired.com	runnersworld.com	none
crm software	google.com	forbes.com	salesforce.com	none
electric cars	toyota.com	edmunds.com	caranddriver.com	none
web hosting	namecheap.com	pcmag.com	godaddy.com	none
email marketing software	emailvendorselection.com	pcmag.com	emailvendorselection.com	Google + Brave
best credit cards	nerdwallet.com	nerdwallet.com	nerdwallet.com	✅ all three
password manager	google.com	google.com	google.com	✅ all three

When the engines do agree (credit cards → NerdWallet, password manager → Google's own page), they agree completely. The rest of the time they each pick a different winner.

Each engine is a different internet

Aggregate the top 10s and each engine's taste is obvious. Bing leans on big editorial publishers. Google fills the page with its own properties (YouTube, its answer boxes) plus community content. Brave sits in between:

Engine	Most common top-10 domains
Google (7 searches)	google.com ×6 · youtube.com ×5 · reddit.com ×4 · wikipedia.org ×3
Bing (15 searches)	forbes.com ×9 · pcmag.com ×8 · wired.com ×4 · nytimes.com ×3
Brave (14 searches)	pcmag.com ×9 · nytimes.com ×6 · reddit.com ×6 · cnet.com ×5

The Reddit, YouTube & Wikipedia split

The starkest difference is how the engines treat the big community and reference platforms. We measured the share of each engine's top-10 SERPs that contained Reddit, YouTube, or Wikipedia at all:

Platform	Google	Bing	Brave
Reddit	57%	0%	36%
YouTube	57%	0%	0%
Wikipedia	43%	0%	29%

Google has clearly tilted toward forums, video, and reference pages — the "Reddit on every result" experience people complain about is real in our data. Bing, on the same queries, surfaced none of them and stuck with publisher round-ups. If your strategy assumes "search results look like Google," Bing users are seeing something completely different.

This is why rank tracking and SERP monitoring have to be multi-engine. A single Google rank check tells you nothing about where you stand on Bing (which feeds Microsoft Copilot and, indirectly, ChatGPT) or Brave. The gap between engines is the opportunity.

Why this matters for SEO and AI search

One engine is not the market. Bing powers Microsoft Copilot and a large share of ChatGPT's web citations; Brave runs its own independent index. Pages that rank there earn traffic and AI citations Google-only tracking never sees.
A win on one engine is not a win everywhere. Because the top 10s barely overlap, you have to measure each engine separately to know your real visibility.
The cheapest wins are off-Google. Bing and Brave have far less competition for the same commercial terms, and our data shows they reward different content — so a page that struggles on Google may already rank on Bing.

The practical takeaway: track keywords on Google, Bing, and Brave, on a schedule, and watch the deltas. That is exactly what a SERP tracker / rank checker API is for.

Try it yourself

Check any keyword across engines right now with the free SERP checker — enter your domain and see where you rank on each.
Build ongoing, multi-engine rank tracking with the SERP monitoring API on Google, Bing, and Brave — normalized JSON, one API key.
Before you commit tracking budget to a keyword, check whether anyone is actually searching it — how to scrape Google Trends shows how to pull that query-demand data alongside the rankings.

Methodology

We searched 15 popular commercial and informational queries — best running shoes, crm software, project management software, best vpn, electric cars, web hosting, standing desk, best credit cards, password manager, email marketing software, noise cancelling headphones, accounting software, best laptops, web scraping api, ai writing tools — on 2026-06-10, US region / English, and recorded the top 10 organic result domains per engine via Crawlora's Google, Bing, and Brave search endpoints. Domains were reduced to their registrable form (so www.reddit.com and old.reddit.com both count as reddit.com). Bing returned for all 15 searches and Brave for 14; Google's browser-rendered results returned within our timeout for 7 of 15 — its SERP is by far the slowest and hardest to fetch, which is its own argument for a managed API. Cross-engine and #1 statistics are computed only over the searches where the relevant engines both answered (Google pairs n=7; Bing∩Brave n=14). It is a snapshot, not a longitudinal study — search results shift constantly, which is the whole reason to monitor them.

Sources

Frequently asked questions

Do Google, Bing, and Brave return the same search results?

No. In our 15-search test, the engines agreed on the #1 result only 29% of the time, and Google and Bing shared just 3 of the top 10 on average. No pair of engines shared even half their top 10 results.

Why does Bing show different results than Google?

The engines weight sources differently. In our data Bing leaned on traditional publishers (Forbes appeared in 9 of 15 top-10s, PCMag in 8), while Google surfaced its own properties and community sites — Reddit and YouTube each appeared in 57% of Google SERPs and Wikipedia in 43%, versus 0% for all three on Bing.

Should I track rankings on more than one search engine?

Yes. Because the top 10 results barely overlap between engines, a single Google rank check misses where you stand on Bing (which feeds Microsoft Copilot and many ChatGPT citations) and Brave. Multi-engine SERP monitoring is the only way to see your real visibility.

Originally published on crawlora.net. SERP Checker is Crawlora's free tool for comparing live Google, Bing, and Brave rankings side by side.

Best Bing Search API Alternatives in 2026 (After the Retirement)

Tony Wang — Mon, 03 Aug 2026 01:00:00 +0000

Key takeaways

Microsoft retired the Bing Search APIs on August 11, 2025 — existing keys return HTTP 410 Gone, new signups are closed, and there is no first-party 'query in, JSON out' Bing Search API anymore.
Microsoft's official path, Grounding with Bing Search in Azure AI Foundry, returns LLM-grounded chunks (not raw results), needs an Azure AI Agent project, and lists at $35 per 1,000 calls — a 40–483% jump over the old tiers.
For a drop-in JSON replacement with one API key and no Azure project, use Crawlora's Bing Search API; for an independent index, Brave; for multi-engine SERPs, SerpApi or DataForSEO; for RAG/agents, Tavily or Exa.
Migrating is mostly mechanical: swap the endpoint, map mkt/count/offset to country, lang, count, and page, change the auth header, and re-point your parser at the normalized JSON.

If you landed here because your Bing Search API key suddenly returns 410 Gone, you are not alone. Microsoft retired the Bing Search APIs on August 11, 2025, and there is no longer a first-party "send a query, get ranked results as JSON" endpoint to sign up for. Whether Bing fed a rank tracker, a competitive-intelligence pipeline, an internal search box, or an AI agent, you need a replacement — and the replacement that fits depends on whether you want raw structured results or an AI-grounded answer.

This guide covers what happened, what Microsoft offers instead, what to look for in a replacement, and the best Bing Search API alternatives in 2026 — what each one does well, where it falls short, and when to choose it — plus a step-by-step migration.

What happened to the Bing Search API

Microsoft announced the retirement in May 2025, roughly 90 days before the shutdown, and decommissioned the v7 "Bing Web Search" and "Bing Custom Search" families on August 11, 2025. In Microsoft's words, the APIs "will be decommissioned completely, and the product will no longer be available for usage or new customer signup." (Microsoft Learn)

The retirement was total, not a tier trim:

It applied to every paid and free tier — the F1 and S1–S9 resources of Bing Search, and the F0 and S1–S4 resources of Bing Custom Search.
Existing keys stopped returning data; the endpoints now respond with HTTP 410 Gone.
New deployments were disabled ahead of the date, so you cannot spin up a fresh instance.

This is why search is full of queries like "bing search api deprecated", "bing search api key free", and "bing search api alternative" — they all trace back to the same event. The move reflects Microsoft's strategic shift from exposing search as a raw data endpoint to embedding it inside AI agents.

What people used the Bing Search API for

Before picking a replacement, it helps to name what you actually relied on. The v7 family was really a bundle of endpoints, and a good replacement only needs to cover the ones you used:

Bing v7 capability	Typical job
Web Search	Organic results for rank tracking, monitoring, search boxes
News Search	Headlines, sources, timestamps for media monitoring
Image Search	Image results and thumbnails
Video Search	Video results and metadata
Autosuggest	Query completions / typeahead
Spell Check	Query correction
Entity Search	Knowledge-panel style entities
Visual Search	Reverse-image and visual lookups
Custom Search	Scoped search over a curated set of sites
Local Business Search	Place and business results

Most teams used Web Search (and often News/Image/Video) as a data source. That "structured results, on demand" job is the one with the cleanest third-party replacements. Knowledge-panel and visual-search use cases are nicher and may need a different tool.

Microsoft's official replacement: Grounding with Bing Search

Microsoft points customers to Grounding with Bing Search, part of Azure AI Agents inside Azure AI Foundry. It lets an Azure-hosted agent pull real-time public web results into an LLM's answer. That is genuinely useful for agent grounding — but it is not a drop-in search API, and the differences matter:

You get answers, not results. "Developers and end users don't have access to raw content returned from Grounding with Bing Search." The model response includes citations and a link to the Bing query, but not a clean ranked list you can parse. (PPC Land)
It is a platform commitment. You stand up an Azure AI Foundry project, an Agent Service, model deployments, and role assignments — not a single key you call from a cron job.
Data leaves the compliance boundary. Each grounding query "is transferred, along with the resource key, outside of the Azure compliance boundary to the Grounding with Bing Search service."
There are display and model rules. You must display both the website URLs and the Bing search-query URL in your interface, and the tool works with the supported Azure OpenAI models (with documented exceptions).

And it is more expensive. Grounding with Bing Search lists at $35 per 1,000 transactions, on top of the Azure AI Agent Service usage it requires. Against the old Bing tiers — roughly $6 (S3), $15 (S2), and $25 (S1) per 1,000 — that is a 40–483% increase depending on which tier you were on. (PPC Land)

When Grounding is still the right call

If your goal is an agent that reasons over fresh web context inside Azure — and you do not need the raw ranked results — Grounding with Bing Search is the most "official" path and keeps you in the Microsoft ecosystem. If you wanted "query in, ranked JSON out," read on.

What to look for in a Bing Search API alternative

If you used Bing as a data source, a good replacement should:

Return structured JSON — organic results with positions, titles, URLs, and snippets — not an LLM summary you have to re-parse.
Accept the basics you already used: query, country, language, and paging.
Authenticate with a single API key you can call directly, with no Azure project or resource-group setup.
Offer predictable, usage-based pricing (ideally billed on successful responses), with a free tier to test.
Cover the modules you need — web plus news, video, and image where relevant.
Add engine breadth (Google and Brave alongside Bing) so one disruption doesn't stall your product.
Handle the unglamorous parts — proxies, anti-bot challenges, retries — and return documented errors instead of broken HTML.

The best Bing Search API alternatives in 2026

There is no single winner — the right pick depends on whether you need raw results, an AI-grounded answer, an independent index, or multi-engine SERPs. Here is the landscape at a glance, then a closer look at each.

Alternative	Type	Output	Setup	Free tier	Best for
Crawlora Bing API	Structured search API	Normalized JSON (web + news/video/image)	One API key	2,000 credits/mo	Drop-in JSON replacement, rank tracking
Azure Grounding	Agent grounding	LLM answer + citations	Azure AI Foundry project	No	Agents inside Azure
Brave Search API	Independent search API	JSON from a self-crawled index	One API key	Yes (limited)	Non-Google, non-Bing coverage, privacy
SerpApi	Multi-engine SERP scraper	Rich SERP JSON, 25+ engines	API key	100 searches/mo	SERP-feature depth across engines
Tavily	AI search API	LLM-optimized snippets	API key	1,000 credits/mo	RAG pipelines and agents
Exa	Neural/semantic search	Semantic results + contents	API key	Free credits	Meaning-based discovery, research
Google Custom Search	First-party JSON API	Google results JSON	API key + engine ID	100 queries/day	First-party Google data, small volume
DataForSEO	SERP + SEO datasets	SERP JSON + keyword/backlink data	API key	Pay-as-you-go	SEO platforms needing bundled datasets

1. Crawlora Bing Search API — the drop-in JSON replacement

If your only question is "how do I send a query and get Bing results back as JSON, today, without standing up Azure?", Crawlora's Bing Search API is the closest drop-in. It wraps the request handling, proxy routing, anti-bot work, parsing, and normalization behind one documented endpoint, authenticated with a single x-api-key. You send q, country, lang, page, and count, and you get normalized organic results back — plus context-panel data, related queries, people-also-ask, and news/video modules when Bing surfaces them.

curl -G "https://api.crawlora.net/api/v1/bing/search" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  --data-urlencode "q=web scraping api" \
  --data-urlencode "country=us" \
  --data-urlencode "lang=en-us" \
  --data-urlencode "page=1" \
  --data-urlencode "count=10"

import os
import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/bing/search",
    headers={"x-api-key": os.environ["CRAWLORA_API_KEY"]},
    params={"q": "web scraping api", "country": "us", "lang": "en-us", "page": 1, "count": 10},
)
for row in resp.json()["data"]["results"]:
    print(row["position"], row["title"], row["url"])

Or, with the official SDK, the call is one line:

from crawlora import CrawloraClient

crawlora = CrawloraClient(api_key=os.environ["CRAWLORA_API_KEY"])
result = crawlora.bing.search(q="web scraping api", count=10)

The response is normalized JSON you can store directly (check the API docs for the current schema):

{
  "code": 200,
  "msg": "OK",
  "data": {
    "results": [
      {
        "position": 1,
        "title": "Example result title",
        "url": "https://example.com/",
        "hostname": "example.com",
        "display_url": "https://example.com",
        "description": "Snippet text shown under the result."
      }
    ],
    "pagination": { "page": 1, "count": 10, "next_page": 2 }
  }
}

Capability	Retired Bing v7	Crawlora Bing API
Output	Structured JSON	Structured JSON
Auth	`Ocp-Apim-Subscription-Key`	`x-api-key`
Setup	Azure resource	One API key
Web results	Yes	Yes (with positions)
News / video / image	Separate endpoints	Separate endpoints (`/bing/news`, `/bing/videos`, `/bing/images`)
Paging	`count` + `offset`	`count` (1–50) + `page`
Other engines	Bing only	Google and Brave share the same shape
Availability	Retired	Active

When to choose it: you want Bing results as structured JSON with the least migration work, especially for rank tracking and SERP monitoring. The bonus is engine breadth — the Google Search and Brave Search endpoints return the same normalized shape, so the same code snapshots all three and you are no longer single-sourced on one engine.

2. Grounding with Bing Search (Azure AI Foundry) — the official path

Covered in depth above. As an alternative, it earns its place when you are building inside Azure and want an agent to reason over fresh web context with Microsoft-blessed grounding and citations. Pick it for agent answers; skip it if you need raw ranked results, predictable per-query pricing, or a setup lighter than an Azure AI Agent project. For an agent that needs structured web data (not summarized answers), a tool-based approach over MCP is often a better fit.

3. Brave Search API — an independent index

The Brave Search API is built on Brave's own independent index — not a Bing or Google reseller. That independence is the reason to choose it: if you specifically want non-Google, non-Bing coverage (for source diversity, rank tracking, or AI grounding), Brave adds a genuinely different result set rather than a re-skin of the same SERP. It returns web results plus news and video modules as JSON, runs privacy-first with no user tracking, and offers a free tier to evaluate.

When to choose it: you want a second or third engine with a truly independent crawl, or you care about the privacy posture. See how to scrape Brave Search for the request/response shape, and pair it with Bing and Google for cross-engine comparison.

4. SerpApi — multi-engine SERP depth

SerpApi scrapes and parses SERPs across 25+ engines — Google, Bing, DuckDuckGo, Yahoo, Baidu, and more — and returns rich structured JSON (organic results, ads, featured snippets, knowledge panels, "people also ask"). If your migration is really about replacing Bing and deepening SERP-feature coverage across engines, it is a mature specialist.

Two caveats. First, it is SERP-only — if you also need non-search platform data, you will run a second vendor. Second, in December 2025 Google filed a lawsuit against SerpApi over scraping Google results; the case is unresolved, and the practical takeaway for buyers is to avoid single-sourcing any one Google-SERP reseller. (Google) See SerpApi alternatives and Crawlora vs SerpApi for the full comparison.

When to choose it: you want broad SERP-feature breadth across many engines and are comfortable managing SERP-supply-chain risk.

5. Tavily — search built for RAG and agents

Tavily is a search API designed for AI agents and LLMs, optimized for retrieval-augmented generation. It returns LLM-ready snippets with fast response times, has native LangChain support, and ships search, extract, crawl, and map endpoints, with a free tier (1,000 credits/month) and transparent per-credit pricing.

When to choose it: you are prototyping or running a RAG pipeline and want AI-optimized search output rather than a raw SERP. It is less suited to precise rank tracking, where you want exact organic positions. For a side-by-side on the two output models — LLM-ready snippets and synthesized answers vs normalized, position-accurate SERP JSON — see Crawlora vs Tavily.

6. Exa — neural, meaning-based search

Exa is an embeddings-based search engine "built for machines": instead of keyword matching, it ranks by semantic similarity, and its standout Find Similar endpoint takes a URL and returns semantically related pages. It also offers contents, answer, and research endpoints, with free credits to start.

When to choose it: discovery and research where meaning beats keyword precision — finding related content, building recommendation or research tools. It is the wrong tool when you need the literal, position-accurate SERP a user would see. For a side-by-side on neural search vs normalized SERP JSON, see Crawlora vs Exa.

7. Google Custom Search JSON API — the other first-party option

If you want first-party Google results as JSON, the Google Programmable Search Engine / Custom Search JSON API is the sanctioned route. It is free for 100 queries/day, then $5 per 1,000 queries up to a daily cap. The catch is volume and scope: the daily limit makes it impractical for serious rank tracking, and it is built around a configured search engine rather than open-web SERP parity.

When to choose it: low-volume, first-party Google data where staying inside Google's terms matters more than throughput.

8. DataForSEO — SERP plus SEO datasets

DataForSEO pairs a SERP API with bundled SEO datasets — keyword search volume, difficulty, and backlinks — on pay-as-you-go pricing. If you are building an SEO platform and want search results and the supporting datasets from one vendor, it consolidates that stack. Compare it with focused options like Serpstack and Serpdog if you only need SERPs.

When to choose it: you are building SEO tooling and want SERP data alongside keyword and backlink datasets.

Bing Search API vs. Crawlora: feature by feature

For the most common migration — "I used Bing Web Search as a data source" — here is the head-to-head against the closest drop-in:

	Retired Bing Search API	Crawlora Bing Search API
Status	Retired Aug 11, 2025 (`410 Gone`)	Active
Output format	Structured JSON	Structured JSON (normalized)
Setup	Azure resource + key	Single API key
Auth header	`Ocp-Apim-Subscription-Key`	`x-api-key`
Query params	`q`, `mkt`, `count`, `offset`	`q`, `country`, `lang`, `count`, `page`
Result modules	Web, news, image, video, etc.	Web + `news` / `videos` / `images` endpoints
Positions in results	Yes	Yes
Cross-engine	Bing only	Google, Bing, Brave (same shape)
Pricing model	Per-transaction tiers	Credit-based, billed on success
Free tier	(was tiered)	2,000 credits/month, no card

How to migrate from the Bing Search API

The mental model maps over cleanly. Most of the work is renaming parameters and re-pointing your parser.

1. Swap the endpoint. Point requests at https://api.crawlora.net/api/v1/bing/search instead of the retired Bing endpoint. Use the dedicated /bing/news, /bing/videos, and /bing/images routes for those modules.

2. Map the parameters. Bing's market and paging become Crawlora's locale and page:

Retired Bing v7	Crawlora Bing API	Notes
`q` / `query`	`q`	The search query (required)
`mkt` (e.g. `en-US`)	`country` + `lang`	e.g. `country=us`, `lang=en-us`
`count`	`count`	Results per page, 1–50
`offset`	`page`	Page-based, 1-indexed
`responseFilter=news`	call `/bing/news`	Per-module endpoints
`Ocp-Apim-Subscription-Key`	`x-api-key`	Auth header

3. Replace the key. Swap the Azure/Bing subscription key for your Crawlora API key, sent as the x-api-key header.

4. Re-point the parser. Read the normalized JSON fields (data.results[].position, title, url, hostname, description) and handle documented errors and retries — for example, 503 when Bing serves a challenge page.

A before/after for a typical web-search call:

- GET https://api.bing.microsoft.com/v7.0/search?q=web+scraping+api&mkt=en-US&count=10
- Header: Ocp-Apim-Subscription-Key: <azure-key>
+ GET https://api.crawlora.net/api/v1/bing/search?q=web+scraping+api&country=us&lang=en-us&count=10
+ Header: x-api-key: <crawlora-key>

Scrape public results responsibly

A replacement returns public search results, but "public" is not blanket permission. Collect only public result pages, respect rate limits, and handle anything that could be personal data lawfully. See Is web scraping legal in 2026? for the longer version.

How to choose

You want Bing results as JSON with the least work → Crawlora's Bing Search API — one key, normalized output, news/video/image modules.
You are building an agent inside Azure → Grounding with Bing Search, for AI-grounded answers with citations.
You want an independent (non-Bing, non-Google) index → the Brave Search API.
You want deep SERP features across many engines → SerpApi (mind the SERP-supply-chain risk).
You are feeding a RAG pipeline → Tavily for AI-optimized snippets, or Exa for semantic discovery.
You want first-party Google data at low volume → Google Custom Search JSON API.
You are building SEO tooling → DataForSEO for SERPs plus keyword and backlink datasets.
Bing was one engine in a rank tracker → run Bing, Google, and Brave through one SERP monitoring workflow so no single engine is a single point of failure.

Next steps

Try it first, free: turn any URL into clean Markdown with the Free Web Scraper — no signup, no API key. To check where a domain currently ranks on Bing (or Google and Brave) without writing any code, try the free SERP checker.

Test /bing/search in the Playground, check the request and response schema in the API docs, and review credit costs on the pricing page. Start from the Bing Search API page for the full workflow, or wire it into an agent with the hosted MCP server.

Sources

Sources

Frequently asked questions

Why is my Bing Search API key returning errors?

Microsoft retired the Bing Search APIs on August 11, 2025. Keys issued before then stopped returning data and the endpoints now respond with HTTP 410 Gone; new signups are closed and the product was decommissioned completely.

When was the Bing Search API retired?

Microsoft announced the retirement in May 2025 and shut the v7 Bing Search and Bing Custom Search APIs down on August 11, 2025, across every free and paid tier (F1/S1-S9 and F0/S1-S4).

What replaced the Bing Search API?

Microsoft points customers to Grounding with Bing Search in Azure AI Agents, which returns LLM-grounded answers with citations rather than raw results. For a data-source replacement, third-party JSON search APIs (Crawlora's Bing API, Brave, SerpApi, Tavily, Exa, and others) stepped in.

Is there a drop-in Bing Search API alternative?

Crawlora's Bing Search API is the closest drop-in for the data-source use case: send q, country, lang, page, and count and get normalized organic results plus news, video, and image modules — with one API key and no Azure project.

Is there a free Bing Search API now?

There is no free first-party Bing Search API anymore. Crawlora includes 2,000 free credits per month (no card) covering Bing, Google, and Brave search endpoints, and Google's Custom Search JSON API allows 100 free queries per day.

How much does Grounding with Bing Search cost?

Grounding with Bing Search lists at about $35 per 1,000 transactions, on top of the Azure AI Agent Service it requires. Against the old Bing S3/S2/S1 tiers (roughly $6/$15/$25 per 1,000), that is a 40-483% increase depending on your prior tier.

What is the best Bing Search API alternative?

It depends on the job: Crawlora's Bing API for a drop-in JSON replacement and rank tracking, Azure Grounding for agents inside Azure, Brave for an independent index, SerpApi or DataForSEO for multi-engine SERPs and SEO datasets, and Tavily or Exa for RAG and semantic search.

How do I migrate off the Bing Search API?

Swap the endpoint to /bing/search, map your old q/mkt/count/offset parameters to q, country, lang, count, and page, replace the key with your Crawlora x-api-key, and re-point your parser at the normalized JSON.

Originally published on crawlora.net. Bing Search API returns Bing search, news, image, and video results as structured JSON — a drop-in replacement since Microsoft retired its own API.

Web Scraping for AI Training Data: A Compliant Guide

Tony Wang — Sun, 02 Aug 2026 01:00:00 +0000

Key takeaways

Most AI teams don't need to crawl the whole web — they need clean, well-sourced records from specific platforms, with provenance kept for every row.
The legal risk isn't one thing: reading public facts is the most defensible, but training on copyrighted works, circumventing anti-bot controls (DMCA §1201), and collecting personal data each add distinct exposure.
Terms of Service alone are weak against scraping public data (X Corp. v. Bright Data), and robots.txt is a request, not a lock — but ignoring machine-readable opt-outs erodes any good-faith story.
Governance is the real work: provenance on every row (source, timestamp, license, opt-out), a datasheet, and retention limits — treat the dataset as something you may have to defend.
Structured APIs make all of this easier: documented JSON, source context built in, a public-data focus, and no per-site parser.

Web scraping for AI training data and RAG is less about brute-force crawling and more about sourcing clean, well-governed records you can defend. The fetching is the easy part; licensing, provenance, copyright, and personal data are where projects get into trouble. This guide covers what the courts actually say in 2026, a practical compliance checklist, how to govern the dataset, and how a structured API makes all of it easier. (None of this is legal advice — get counsel for your use.)

What "training data" usually means here

Two adjacent needs get lumped together:

RAG / retrieval — current, structured records you index so an assistant can ground its answers. Freshness and provenance matter most. See the web data for RAG use case.
Training / fine-tuning / evaluation — curated datasets used to train or measure a model. Licensing, dedupe, and documentation matter most.

For both, the win is the same: clean records with source context, not a pile of raw HTML.

What the courts actually say (2026)

This is the most actively litigated corner of tech law right now, and courts are splitting — but a few threads are clearer than the headlines suggest.

Terms of Service alone are weak against scraping public data. In X Corp. v. Bright Data Ltd. (N.D. Cal., 2024), the court dismissed X’s breach-of-contract claims, reasoning that X’s users — not the platform — own their posts, so a platform can’t use its ToS to build a "private copyright system" over public content. How far public data may be copied, the court said, is governed by the Copyright Act, not the ToS.
But bypassing anti-bot controls is a different, riskier act. Newer suits target circumvention rather than reading public pages: Reddit v. Perplexity AI (filed late 2025, pending in 2026) alleges DMCA §1201 violations for evading rate limits and anti-bot systems — the same enforcement push behind Reddit shutting down unauthenticated .json access in 2026. Quietly defeating CAPTCHAs and rate limits is legally distinct from reading a public page.
robots.txt is a request, not a lock. In litigation involving OpenAI, a court found robots.txt does not "effectively control" access for DMCA purposes — it signals a preference, it doesn’t create a technical barrier. Ignoring it isn’t automatically "circumvention," but it can still breach terms and erode any good-faith story.
Training on copyrighted works still implicates copyright. The U.S. Copyright Office’s 2025 report on generative AI concluded that assembling a training set from copyrighted works "clearly implicates the right of reproduction" — so fair use is a defense to argue, not a guarantee, and the outcome is unsettled across courts.
Personal data is a separate track. The Bright Data court noted that privacy claims are not preempted by copyright. So even where copyright is defensible, GDPR and CCPA independently constrain scraping names, emails, and profiles.

The takeaway: reading public, factual data is the most defensible position; training on copyrighted works, circumventing anti-bot controls, and collecting personal data each add their own distinct risk.

The compliance checklist

Licensing & terms — check each source's terms and any dataset license before use, especially for training or redistribution.
Don't circumvent — bypassing rate limits, CAPTCHAs, or anti-bot systems is a distinct legal risk (DMCA §1201), separate from reading a public page.
Opt-out signals — respect robots.txt, ai.txt, and machine-readable TDM/‘noai’ reservations; treat a block as a no.
Copyright — facts (prices, stats) aren't copyrightable, but articles, photos, and descriptions are; using copyrighted works for training implicates reproduction, and fair use is unsettled.
Personal data — names, emails, and profiles are PII under GDPR/CCPA and aren't shielded by the copyright rulings; avoid them without a lawful basis.
Provenance — record source URL, fetch time, license basis, and opt-out status for every record, in an auditable store.
Dedupe & document — remove duplicates and write a datasheet (sources, dates, fields, licensing, known gaps).

This is a practical summary, not legal advice — see is web scraping legal in 2026 and get counsel for training or redistribution.

Dataset governance: provenance, datasheets, and roles

Whether you’re building a RAG index or a training set, treat the dataset as something you may have to defend later. Best practice converges on a handful of habits:

Provenance on every row. Record the source URL or identifier, fetch timestamp, request parameters, the license/terms basis, and any opt-out status — ideally in an append-only (immutable) store with an audit trail, so you can prove where any row came from.
A datasheet for the dataset. Document sources, dates, fields, collection method, known gaps, and licensing notes — the "Datasheets for Datasets" pattern — before anyone trains on it.
Honor machine-readable opt-outs. Under the EU DSM Directive’s text-and-data-mining exception (Article 4), commercial TDM is permitted only where the rightsholder hasn’t reserved rights "by machine-readable means" — so a machine-readable opt-out (robots.txt, ai.txt, metadata) is legally meaningful, not just polite.
Clear roles. Even a small team benefits from naming who owns data sourcing, rights/privacy review, dataset stewardship, and audit.
Retention limits. Keep only what the workflow needs, for only as long as it needs it.

Why structured data beats raw HTML for datasets

A structured API returns documented JSON for supported platforms, which makes a dataset far easier to govern than scraped pages:

Provenance is built in. Each response carries the source and the request context, so you can keep a column for where every row came from.
Consistent schema. Fields are documented and stable, so cleaning and dedupe are deterministic instead of per-site guesswork.
No parser rot. You don’t maintain selectors that silently break and corrupt the dataset.
Public-data focus. Platform endpoints return public records, which keeps you on the more defensible side of the line above.

# Collect a structured record with source context, ready to store as a dataset row
curl -s "https://api.crawlora.net/api/v1/google-search/search?keyword=web%20scraping%20api&country=us" \
  -H "x-api-key: $CRAWLORA_API_KEY"

import requests

rows = []
r = requests.get(
    "https://api.crawlora.net/api/v1/google-search/search",
    headers={"x-api-key": "YOUR_API_KEY"},
    params={"keyword": "web scraping api", "country": "us"},
).json()
for item in r["data"]:
    rows.append({**item, "source": "google-search", "collected_at": "2026-06-06"})  # keep provenance

Store the source and collected_at fields alongside the data so the dataset stays auditable.

A compliant workflow in five steps

Scope to sources you can defend — specific platforms with terms you’ve reviewed, not an indiscriminate crawl, and skip anything with a machine-readable opt-out.
Collect structured records — call documented endpoints (or a hosted MCP server) and keep the JSON. Don’t build circumvention into the pipeline.
Attach provenance — source, URL/identifier, timestamp, license basis, and request parameters on every row.
Clean & dedupe — normalize, remove duplicates, and drop or mask PII you don’t need.
Document — write a short datasheet (sources, dates, fields, licensing notes, known gaps) before training.

Responsible use

Crawlora provides public data infrastructure, not permission to use any content for any AI purpose. Training and redistribution raise licensing and copyright questions beyond ordinary collection — keep public, factual data, honor source terms and machine-readable opt-outs, avoid unnecessary personal data, and consult counsel for your specific use.

Sources

Sources

Next steps

See AI vs traditional web scraping, compare the best AI web scraping tools, and try the AI Web Scraping API in the Playground.

Frequently asked questions

Can I use scraped web data to train AI models?

Sometimes, but it is unsettled and actively litigated. The US Copyright Office's 2025 report found that assembling a training set from copyrighted works implicates the reproduction right, so fair use is a defense to argue, not a guarantee. Keep public, factual data, check each source's terms and any dataset license, avoid copyrighted media and personal data without a basis, and consult counsel. Crawlora is data infrastructure, not legal advice.

Do Terms of Service or robots.txt make scraping for AI illegal?

Not by themselves. In X Corp. v. Bright Data (2024) a court held a platform can't use its ToS to override copyright on user content, and courts have found robots.txt is a request, not a technical access control. But ignoring opt-outs can still breach terms, and bypassing rate limits or anti-bot systems is a separate DMCA risk.

Is scraping public data for AI considered fair use?

It's the central open question. Reading public, factual data is the most defensible; training on copyrighted creative works is where fair use is fought and courts are split. There is no blanket answer yet — analyze the specific content and get legal advice.

How do I keep an AI dataset compliant?

Scope to sources you can defend, honor machine-readable opt-outs (robots.txt, ai.txt, TDM reservations), don't circumvent anti-bot controls, keep provenance (source, URL, timestamp, license, opt-out status) on every record, dedupe, strip or avoid PII, and document the dataset in a datasheet before training.

Why use a structured API for AI training data?

Documented JSON makes datasets easier to govern: provenance and source context come built in, the schema is stable so cleaning and dedupe are deterministic, the focus is public records, and there is no parser to silently break and corrupt rows.

Is scraping for RAG different from scraping for training?

RAG needs current, well-sourced records you index for grounding, where freshness and provenance matter most. Training needs curated, licensed, documented datasets. Both benefit from clean structured records over raw HTML.

Originally published on crawlora.net. Crawlora Datasets ships pre-collected, structured datasets across dozens of platforms — skip the scraping step for AI training data.

Scraping Sites That Block Bots: Cloudflare, DataDome & PerimeterX

Tony Wang — Sat, 01 Aug 2026 01:00:00 +0000

For most of the web, scraping is a solved problem: fetch the URL, parse the HTML, done. The interesting sites — the ones with prices, listings, reviews, and inventory worth collecting — are exactly the ones that don't let you. They sit behind Cloudflare, DataDome, PerimeterX (now HUMAN), Akamai, or Kasada, and the moment a script asks for a page it gets a CAPTCHA, a "checking your browser" interstitial, or a flat 403. The hard part of modern scraping isn't parsing the page. It's getting the page.

This guide explains how that wall actually works — the signals these systems check and why a normal HTTP client trips every one of them — and then how a scraper gets through reliably without pretending the problem is simpler than it is.

Key takeaways

Bot detection is layered: IP reputation, the TLS/HTTP fingerprint, a JavaScript sensor that probes the browser, and behaviour over time. You have to pass all of them, not one.
A datacenter IP and a non-browser fingerprint are what get most scrapers blocked — not a missing User-Agent. Spoofing headers alone does almost nothing.
The reliable pattern is escalation: a cheap Chrome-impersonated request first, a real stealth browser only when the site demands it, and a fresh IP when the current one is burned.
When a browser earns a clearance cookie, cheaper requests can reuse it — and a request should only count as success when it returns the real page, not a challenge.
None of this is 100%. It's probabilistic, the hardest sites need residential or mobile IPs, and it applies to public data only.

The four layers of bot detection

Anti-bot vendors don't rely on a single check. They score every request across several independent layers, and a request has to look human on all of them. Understanding the layers is the whole game, because each one rules out a different class of naive scraper.

Layer	What it checks	Why a normal scraper fails
IP reputation	Is the address residential/mobile (trusted) or datacenter (suspect)? Has it made too many requests?	Scrapers run on cloud servers and datacenter proxies — ranges these vendors flag on sight
TLS / HTTP fingerprint	Does the TLS handshake and HTTP/2 frame order match a real browser (JA3/JA4, Akamai fingerprint)?	`requests`, `curl`, and most HTTP libraries have a fingerprint nothing like Chrome's, no matter the headers
JavaScript sensor	A script probes for `navigator.webdriver`, headless tells, missing APIs, canvas/WebGL/font quirks	Headless automation leaks dozens of signals; an HTTP client runs no JavaScript at all, so the sensor never reports back
Behaviour	Request cadence, mouse/scroll, navigation patterns, cookie continuity	Scripts hit pages faster and more mechanically than a person, from a session with no history

The reason "just set a realistic User-Agent" stopped working years ago is that the User-Agent is a single string in the least important layer. You can claim to be Chrome 140 all you like; if your TLS handshake says Python and your IP says AWS, you've already failed two checks before the page even loads.

Why headless Chrome alone isn't enough

Running Puppeteer or Playwright with stock headless Chrome fixes the JavaScript layer better than an HTTP client — it actually executes the sensor — but headless mode leaks its own tells (navigator.webdriver, automation-controlled flags, subtle rendering differences), and it does nothing for the IP or TLS layers. That's why "use a real browser" and "use a good proxy" are both necessary and neither is sufficient on its own.

What the vendors actually do

The big three behave differently enough that it's worth knowing which wall you're looking at.

Cloudflare issues a managed challenge and, on success, a cf_clearance cookie that's bound to your IP and User-Agent. Pass the challenge and the cookie buys you a window of access — change IP and it's void.
DataDome scores the request in real time and sets a datadome cookie; it's aggressive about datacenter IPs and replays of a fingerprint across too many requests.
PerimeterX / HUMAN runs a JavaScript sensor that POSTs a signal payload and grants a _px3-family clearance cookie. It's strongly IP-bound: a cookie minted on one IP, presented from another, reads as theft and gets blocked harder than no cookie at all.

The common thread: a clearance — the cookie that says "this client passed" — is only earned by executing the challenge in a real browser, and it's tied to the IP that earned it. That single fact dictates the entire strategy.

Which wall, which fix

Each vendor leaves a fingerprint of its own — a tell-tale cookie or block message — and yields to a different lever. This is the rough field guide:

Wall	How you know it's there	What gets through	Difficulty
Cloudflare	`cf_clearance` / `__cf_bm` cookies, a "Checking your browser…" interstitial	A real browser that solves the managed challenge; then reuse `cf_clearance`	Low–medium
DataDome	`datadome` cookie, a 403 with a DataDome CAPTCHA page	Trusted IP + real fingerprint; rotate IPs — it punishes replay	Medium
PerimeterX / HUMAN	`_px*` cookies, "Access to this page has been denied"	A browser that runs the JS sensor; clearance is IP-bound, so race fresh IPs	Medium–high
Akamai Bot Manager	`_abck` / `bm_sz` / `ak_bmsc` cookies	Real TLS fingerprint + browser; the `_abck` cookie must validate or you're shadow-blocked	High
Kasada	`x-kpsdk-*` headers, a `kpsdk` sensor script	Full browser execution of the sensor; among the hardest to pass headlessly	High

The pattern repeats: every one of them ultimately wants to see a real browser, from an IP it trusts, behaving like a person. The walls differ mostly in how strict each of those three checks is.

How to get through reliably

The mistake most scrapers make is doing one thing — a stealth browser, or a residential proxy — for every request. That's slow and expensive on easy pages and still fragile on hard ones. The reliable pattern is escalation: start cheap, and climb only as far as a specific site forces you to.

Chrome-impersonated HTTP. A plain request, but with a TLS and HTTP/2 fingerprint that matches real Chrome. This alone clears the fingerprint layer and is enough for a surprising number of "protected" sites — at a fraction of the cost of a browser.
A real stealth browser. When the page needs JavaScript executed — a Cloudflare or PerimeterX challenge — hand it to a fleet of hardened browser engines with patched automation tells and genuine fingerprints. Different engines beat different vendors, so racing or rotating across a fleet matters more than betting on one.
A fresh IP. Anti-bot is IP-reputation-first, so when the current exit is flagged, the highest-yield move is simply a different address. Firing several requests through a rotating pool at once — and taking the first that comes back with the real page — turns a coin-flip into a near-certainty, because one fresh IP usually passes while others are blocked.

Not all addresses are equal, though, and this is where cost enters. IPs come in tiers of reputation, and the more trusted the tier, the more it costs:

Datacenter IPs are cheap and plentiful, but the most flagged — whole ranges are known to belong to clouds and hosting providers, so vendors distrust them by default.
Residential IPs are real home-broadband addresses sourced through proxy networks. They look like ordinary visitors, carry far more trust, and cost meaningfully more.
Mobile IPs are carrier-NAT'd 4G/5G addresses — the most trusted of all, because thousands of real phones share one address, so blocking it risks blocking real customers. They're also the most expensive, usually billed by the gigabyte of traffic rather than per address.

The reflex on a hard wall is to reach straight for the priciest IPs. The cheaper play is the one this whole section is about: lean on the stack — escalate to a real browser, race several datacenter IPs at once, and reuse a clearance once you earn it — so you get through on low-cost addresses as often as possible, and only fall back to residential or mobile for the targets that genuinely demand them. You pay for reputation exactly when the page forces you to, and not a request sooner.

Two refinements make this fast as well as reliable:

Reuse the clearance. When a browser earns a cf_clearance or DataDome cookie, cache it per-domain for its short lifetime and attach it to later requests from the same IP. A cheap engine can then ride a clearance an expensive one paid for — higher success, lower cost.
Only count a real page as success. A challenge page, a "checking your browser" shell, and a 403 all return a 200 with bytes in the body. A scraper that treats those as success hands you garbage and learns nothing. Detecting the difference — and escalating instead of returning the shell — is what separates a number that looks good from data you can use.

The honest part: it's probabilistic, and IPs are the ceiling

No one beats every wall every time, and anyone claiming 100% is selling something. The technique raises the floor; the ceiling is your egress. Datacenter IPs — even rotated — get you through soft and medium protection. The hardest, IP-strict targets (some PerimeterX and DataDome deployments) only stay reliable from residential or mobile addresses. Honest success-based pricing matters here too: you should pay for the request that actually returned the page, not for every blocked attempt along the way.

A worked example: a PerimeterX-protected page

Here's how the pieces compose on a real, stubborn target — a news site behind PerimeterX, where a datacenter exit IP had been used enough that the wall had already flagged it.

One engine, one flagged IP: asking a single stealth browser to fetch the page from that burned IP succeeded only about 1 in 6 times. The engine was capable; the IP was the bottleneck.
Race fresh IPs: firing four requests for the same page concurrently, each through a different exit IP, and taking the first that returned the real article, lifted success to 5 in 6 — and it was faster, because the winner usually came back in a few seconds while the blocked attempts were abandoned.
Reuse the clearance: once one browser earned PerimeterX's clearance cookie, even a lightweight engine that never beats the wall on its own rode that cached cookie straight to the full page. The expensive request paid for the clearance; the cheap ones cashed it in.

No single one of these is a silver bullet. The result comes from stacking them — escalate to a browser, race fresh IPs, reuse what works — and from refusing to count a challenge page as a win. That last point is easy to get wrong: the naive version returns a 200 full of nothing and reports a great success rate.

Build it yourself, or buy it?

You can assemble all of this yourself, and for a single target it's a reasonable weekend project: a stealth-patched browser, a proxy, a cookie cache. The cost shows up later, as a treadmill.

Building it means owning a fleet of patched browser engines (the patches go stale every Chrome release), a residential or mobile proxy budget (the line item that actually sets your ceiling), the escalation logic that picks the cheapest method per site, success detection that isn't fooled by challenge pages, and a standing commitment to re-fix all of it every time a vendor ships a detection update — which is their full-time job and, for you, a side-quest that keeps interrupting the real one.

Buying it — a managed scraping API — trades that treadmill for a per-request price. The honest comparison isn't "API credits versus free code"; it's "API credits versus a proxy bill plus the engineering weeks you'll spend maintaining detection bypasses instead of shipping your product." For one or two easy sites, DIY wins. For a moving list of protected targets you need to keep reliable, the maintenance is the product — and that's the part worth outsourcing.

Where Crawlora fits

This is exactly the model Crawlora's Web Scraping API is built on. A single /web/scrape call escalates on its own — Chrome-impersonated HTTP, then a fleet of stealth browser engines, then fresh IPs raced concurrently — captures and reuses clearance cookies per domain, and only returns a page once it's confirmed to be real content rather than a challenge. You get clean Markdown (or HTML, links, and metadata) back, and you're billed for what succeeds.

A request is one call — ask for the formats you want and let it escalate:

curl -X POST "https://api.crawlora.net/api/v1/web/scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/123",
    "formats": ["markdown", "links", "metadata"],
    "render": "auto"
  }'

render: "auto" is the escalation switch: it starts with Chrome-impersonated HTTP and climbs to the stealth-browser fleet (and fresh IPs) only if the page demands it, so you don't pay browser prices for pages that don't need a browser. The same call from Python:

import requests

resp = requests.post(
    "https://api.crawlora.net/api/v1/web/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://example.com/product/123",
        "formats": ["markdown", "links", "metadata"],
        "render": "auto",  # escalate HTTP -> stealth browser -> fresh IP as needed
    },
    timeout=60,
)
data = resp.json()["data"]
print(data["markdown"])            # clean article text, not raw HTML
print(data["metadata"]["title"])   # parsed page metadata

If you just want to know what you're up against before you write a line of code, run a URL through the free can-I-scrape-this-site checker: it reports which protection a site uses and how hard it'll be to collect.

A closing note on scope: everything here is for public pages. Bot detection is not a login, and getting past one isn't the same as breaking into private data — but the responsible line is still public data only, with each site's terms and robots directives respected, and personal or copyrighted content left alone. For the legal landscape, see is web scraping legal in 2026; for the related question of gated articles, see how paywalls actually work. Cloudflare's own AI-crawler policy is evolving on top of this same detection stack — see Cloudflare's new AI-crawler defaults for 2026 for what changed and what it means for legitimate scrapers.

Sources

Frequently asked questions

How do websites detect and block scrapers?

Modern anti-bot systems check four layers at once: the IP's reputation (datacenter ranges are flagged, residential and mobile are trusted), the TLS and HTTP/2 fingerprint (a real Chrome handshake looks different from a Python or curl one), a JavaScript sensor that probes the browser for automation tells (headless flags, missing APIs, canvas/WebGL quirks), and behaviour over time. Failing any layer earns a CAPTCHA or a block page, so a scraper that only spoofs the User-Agent gets stopped immediately.

Why does my scraper get blocked by Cloudflare or DataDome?

Almost always the IP and the fingerprint. Requests from a cloud server (AWS, GCP, a datacenter proxy) sit in ranges these vendors treat as suspicious, and a non-browser HTTP client has a TLS/JS fingerprint that doesn't match a real Chrome. Cloudflare, DataDome and PerimeterX combine those signals — so the fix isn't a better User-Agent string, it's a real browser fingerprint coming from a trusted IP.

Can you scrape a site protected by Cloudflare, DataDome or PerimeterX?

Often, yes, for public pages — but it's probabilistic, not guaranteed. The reliable approach escalates only as far as a site demands: a Chrome-impersonated HTTP request first, then a real stealth browser that executes the challenge, then a fresh IP if the current one is flagged. Once a browser earns a clearance cookie, cheaper requests can reuse it. The hardest, IP-strict sites need residential or mobile egress to stay reliable.

Is it legal to scrape sites that block bots?

Scraping publicly accessible pages is broadly defensible, and a bot-detection wall is not itself an access control on private data the way a login is. But the law turns on what you collect and how you use it — respect each site's terms and robots directives, avoid personal data and copyrighted content at scale, and never use this to get past a login or a paywall. This is not legal advice.

Originally published on crawlora.net. Can I Scrape This Site? is Crawlora's free anti-bot checker — see exactly which defenses a target site runs before you build around them.

Best Web Search APIs for AI Agents in 2026

Tony Wang — Fri, 31 Jul 2026 01:00:00 +0000

Key takeaways

There are two kinds of 'web search API', and conflating them is the #1 mistake: LLM-ready APIs (Tavily, Exa, Perplexity Sonar, Linkup, Jina) return cleaned content or a synthesized answer; raw SERP APIs (Serper, SerpApi, Brave, Crawlora) return parsed search results you process yourself.
Pick by output and by job: an answer engine for finished cited answers, an LLM-ready search API for RAG context, the cheapest raw SERP for high-volume Google lookups, and a multi-engine/structured API when you also need Bing, Brave, or non-search platform data.
2026 reshaped the market: Microsoft retired the Bing Search API (the Azure replacement lists at ~$35/1k) and Tavily was acquired by Nebius — so 'what do I migrate to' is a live question.
Benchmark on cost per successful call for YOUR queries, not the headline price: raw SERP runs ~$0.30–$1/1k (Serper), LLM-ready search ~$5–8/1k, and answer/deep tiers higher.

If you are wiring web search into an AI agent or a RAG pipeline, the hard part is not finding an API — it is picking the right kind. Some APIs hand your model a finished, cited answer. Some return cleaned, LLM-ready content. Others return the raw search results an engine actually shows, and leave the reading to you. They are priced differently, they fail differently, and the wrong choice shows up as either a bloated token bill or an agent reasoning over 200-character snippets. This guide splits the field into two categories, ranks the main options with real 2026 pricing, and shows how to choose by job.

"Web search API" is two categories, not one

LLM-ready APIs — they search, fetch, clean, and (optionally) synthesize. You get cleaned page content (often markdown) and/or a model-written answer with citations, ready to drop into a prompt. Best when the agent needs to read and reason. Examples: Tavily, Exa, Perplexity Sonar, Linkup, Jina, Brave Answers.
Raw SERP APIs — they return the parsed search results page (organic links, snippets, knowledge graph, verticals) as JSON. No extraction, no answer; you add those. Best when you need the actual ranking, multiple engines, or the cheapest high-volume lookups. Examples: Serper, SerpApi, Brave Search, and Crawlora.

Most production stacks use both: a raw SERP or structured API for the ranked results and known platforms they hit constantly, and an LLM-ready layer (or their own extractor) for the open-web pages an agent has to read.

What to evaluate

Output you actually need — a cited answer, LLM-ready content, or raw ranked results.
Engine coverage — Google only, or Google plus Bing and Brave for source diversity and resilience.
Freshness and filters — recency windows and domain include/exclude are table stakes for RAG.
MCP support — a first-class MCP server matters if you are wiring tools into Claude, Cursor, or an agent runtime.
Anti-bot reliability — extraction silently fails on protected sites; check it on YOUR targets.
Cost per successful call — raw SERP, LLM-ready search, and answer tiers differ by ~10–50x.
Free tier shape — one-time credits vs a recurring monthly allowance.

Why 2026 is a migration year

Two events pushed a wave of teams to re-pick their search layer. First, Microsoft retired the Bing Search API on August 11, 2025; the official replacement, Grounding with Bing Search, returns LLM-grounded chunks (not raw results), requires an Azure AI Agent project, and lists at about $35 per 1,000 calls — see Best Bing Search API Alternatives. Second, Tavily was acquired by Nebius, prompting "Tavily alternatives" searches across developer communities. If you are choosing now, you are choosing in an unusually open market.

The best web search APIs for AI agents in 2026

There is no single winner — the right pick depends on whether you want a cited answer, LLM-ready content, or raw ranked results. Here is the landscape at a glance, then a closer look at each. Prices are headline rates read in June 2026; always confirm on the vendor's pricing page.

API	Category	Returns	MCP	Free tier	Approx price /1k	Best for
Tavily	LLM-ready	Snippets + cleaned content + optional answer	Official	1,000 cr/mo	~$4 basic / $8 advanced	The default agent search layer
Exa	LLM-ready	Neural results + full content + answer + find-similar	Yes	1,000 req/mo	~$7 search (+$1 contents)	Semantic discovery & research
Perplexity Sonar	Answer engine	A cited, model-written answer	Community	—	~$5–12 + tokens	One-call cited answers
Linkup	LLM-ready	Results / sourced answer / structured	Yes	$20/mo credit	~$5 ($6 answer)	Cost-sensitive EU RAG
Jina	LLM-ready	URL→markdown + search with content	Yes	10M tokens	token-priced	Cheap URL→markdown grounding
Brave Search	Raw SERP (+Answers)	Independent-index results; optional answer	Community	$5 credit/mo	~$5 / $4 answer	A non-Google, non-Bing index
Serper	Raw SERP	Parsed Google SERP JSON (10 verticals)	Community	2,500 cr once	$0.30–$1.00	Cheapest high-volume Google SERP
SerpApi	Raw SERP	Multi-engine SERP JSON + AI-overview parsing	Listed	250/mo	~$9–25	Deep SERP features across engines
Crawlora	Raw SERP + structured platforms	Normalized Google/Bing/Brave SERP JSON + platform data	Hosted	2,000 cr/mo	Credit-based, on success	Multi-engine SERP + structured data + MCP

Tavily — the default agent search layer

Tavily is purpose-built for agents and RAG: it returns ranked results with cleaned content and an optional synthesized answer with citations, plus extract, crawl, and map endpoints, an official MCP server, and native LangChain support. Choose it when you want a no-fuss, LLM-ready search layer. It is less suited to exact rank tracking, where you need literal organic positions.

Exa — semantic discovery and research

Exa is an embeddings-based search engine: it ranks by meaning, and its find-similar endpoint takes a URL and returns related pages. It also returns full page contents and answers and offers bulk enrichment (Websets). Choose it for concept-heavy retrieval and discovery; it is the wrong tool when you need the literal SERP a user sees.

Perplexity Sonar — one-call cited answers

Perplexity Sonar is an answer engine, not a results API: it returns a model-written answer grounded in live web search, with citations, across sonar / sonar-pro / sonar-deep-research tiers. Choose it when you want the answer step handled for you and minimal retrieval plumbing; skip it when you need raw results to process yourself.

Linkup — production RAG on a budget

Linkup returns raw searchResults, a sourcedAnswer, or structured output, ships an MCP server, and is priced around $5/1k with a recurring monthly credit. It is a strong, cost-sensitive, EU-friendly option for RAG.

Jina — cheap URL→markdown grounding

Jina's Reader turns any URL into clean markdown and its Search returns results with full content; it ships an MCP server and a generous free token allowance. Pricing is token-based, so it does not compare cleanly on a per-1k-search basis, but it is among the cheapest ways to get LLM-ready content.

Brave Search API — an independent index

The Brave Search API is built on Brave's own crawl — genuinely independent of Google and Bing. It returns results optimized for LLM context and offers an Answers endpoint. Choose it for source diversity or a privacy-first posture; note the old free tier was replaced by a metered $5 monthly credit.

Serper — cheapest high-volume Google SERP

Serper is a thin, fast Google SERP wrapper that returns parsed JSON across web, news, images, maps, places, shopping, and scholar — at roughly $0.30–$1.00 per 1,000, the cheapest raw Google SERP in this list. There is no content extraction or answer; you add those. Choose it for high-volume, Google-only lookups where price per query is the priority.

SerpApi — deep SERP features across engines

SerpApi parses SERPs across 25+ engines with rich features (ads, knowledge panels, AI overviews) and a Legal Shield. It is the mature specialist when SERP-feature depth across engines is the requirement; it is pricier (~$9–25/1k) and, like all single-engine resellers, carries SERP-supply-chain risk — see SerpApi alternatives.

Crawlora — multi-engine SERP plus structured platform data

Crawlora returns Google, Bing, and Brave results in one normalized shape — position-accurate JSON for rank tracking — alongside structured endpoints for maps, social, video, marketplaces, app stores, reviews, and finance, all behind one API key with managed proxies, retries, and a hosted MCP server. To be clear about the category: Crawlora returns raw, normalized search and platform JSON, not LLM-ready snippets or a synthesized answer — pair it with your own extraction/LLM step (or one of the LLM-ready APIs above) when an agent needs to read cleaned content. Choose it when you need more than one engine, structured platform data beyond search, and MCP-native tools in one place.

How to choose, by job

You want a finished, cited answer → Perplexity Sonar (or Tavily/Linkup's answer mode).
You are feeding a RAG pipeline and want LLM-ready content → Tavily or Linkup; Jina for cheap URL→markdown.
You want semantic discovery / find-similar → Exa.
You need the cheapest high-volume Google SERP → Serper.
You need deep SERP features across many engines → SerpApi (mind the supply-chain risk).
You want a non-Google, non-Bing index → Brave Search.
You need multiple engines, structured platform data, and rank tracking under one API + MCP → Crawlora.
You are replacing the retired Bing Search API → start with the migration guide.

Scrape and search public data responsibly

A web search API returns public results, but "public" is not blanket permission. Respect rate limits, robots and terms, and handle anything that could be personal data lawfully. See Is web scraping legal in 2026? for the longer version.

Sources

Sources

Frequently asked questions

What is the best web search API for AI agents?

It depends on the output you need. For a finished cited answer, use an answer engine like Perplexity Sonar; for LLM-ready content in a RAG pipeline, Tavily, Exa, or Linkup; for the cheapest high-volume Google SERP, Serper; and for multiple engines plus structured platform data and MCP under one API, Crawlora.

What is the difference between a web search API and a raw SERP API?

An LLM-ready web search API returns cleaned content or a synthesized answer your model can read directly. A raw SERP API returns the parsed search results page (links, snippets, knowledge graph) as JSON, and you handle extraction and reasoning yourself. Many stacks use both.

What replaced the Bing Search API for AI agents?

Microsoft retired the Bing Search API on August 11, 2025. The official path, Grounding with Bing Search, returns LLM-grounded chunks inside Azure at about $35 per 1,000 calls. For a drop-in JSON replacement, teams move to Crawlora, Brave, SerpApi, or an LLM-ready API like Tavily or Exa.

Which web search API is cheapest?

For raw Google SERP, Serper is among the cheapest at roughly $0.30-$1.00 per 1,000 by volume. LLM-ready search (Tavily, Exa, Linkup, Brave) clusters around $5-8 per 1,000, and answer or deep-research tiers cost more. Compare on cost per successful call for your queries.

Which web search APIs ship an MCP server?

Tavily, Exa, Linkup, Jina, and Crawlora ship MCP servers; Serper, SerpApi, and Brave have community MCP wrappers. MCP support matters if you are wiring search into Claude, Cursor, or an agent runtime.

Which is best for a RAG pipeline?

For LLM-ready content, Tavily or Linkup; for semantic discovery, Exa; for a one-call cited answer, Perplexity Sonar; and for structured, multi-engine results you control, Crawlora paired with your own extraction or LLM step.

Originally published on crawlora.net. Google Search API returns structured JSON search results for AI agents and RAG pipelines — no scraping infrastructure to maintain.