robots.txt is the one file on a website written for robots. So it surprised me that when a plain script asked 172 big websites for theirs, 20 of them said no.
Redfin, The Home Depot, Lowe's, Sephora, DoorDash, Udemy, Apartments.com and 13 others answered GET /robots.txt with an error: 19 with a 403 Forbidden, one (Wildberries) with a 498. Asking for the terms of service page went worse: 52 refusals, mostly 403s, plus a few 429s.
This is one slice of a small study I put together for Scraper Finder, which maps the ways to get data out of popular websites. For each of the 172 websites it covers (Amazon, LinkedIn, Zillow, Indeed, Google Maps, Reddit and so on) it records four things, each with the date it was read:
- whether there is an official API, and who may use it
- what robots.txt says to crawlers that have no rules of their own (
User-agent: *) - which AI crawlers robots.txt names
- whether the website answered one plain HTTPS request for robots.txt and for its terms page
"Plain" means one request, an ordinary browser user-agent string, no browser, no cookies, no retries, sent from a server. That last part matters: many of these websites refuse anything coming from a data-centre address, whatever it asks for. So a refusal tells you the front door screens automated traffic. It tells you nothing about the search or product pages behind it.
Here is what the numbers say, as of 14 November 2026.
Only 39 of the 172 have an API anyone can use
39 websites (23%) offer a public official API: sign up, get a key, call it. 47 limit access to approved uses, a paid tier or a narrow scope. 36 keep it for partners under contract. 45 have no API at all, and 5 retired the one they had.
So for roughly three websites in four, "just use the API" is not an option for most developers.
The websites without an API are the ones guarding the door
This is the part I did not expect to be so clear. Split the plain-request results by API status:
Websites with no API refused 21 times out of 45 (47%). Websites with a public API refused 8 times out of 39 (21%). Limited and partner-only APIs sit in between, at about a third.
The groups are small and it is a single request per file, so I read this as a tendency rather than a law. But it matches what anyone who has collected data at work has felt: the websites that offer no front door are usually the ones guarding the side door too.
17 websites' robots.txt says "nothing here for you"
Of the 137 websites whose robots.txt could be read, 115 disallow some paths for unnamed crawlers, 5 disallow nothing, and 17 disallow everything: Baidu, Facebook, Google News, IMDb, LinkedIn, LinkedIn Ad Library, LinkedIn Jobs, Meta Ad Library, Naver, Pinterest, Quora, Reddit, TikTok Ads Library, Trustpilot, X, Yelp and ZipRecruiter. (Three of those share a file: robots.txt lives at the root of a host, so LinkedIn Jobs and the LinkedIn Ad Library answer to linkedin.com's file, and the Meta Ad Library to facebook.com's.)
Four of them, Baidu, Pinterest, X and Yelp, do have a public API. Read together, the message is fairly direct: crawling is not welcome, the API is the route.
The other thirteen offer neither a public API nor a single path open to an unnamed crawler: Facebook, Google News, IMDb, LinkedIn, LinkedIn Ad Library, LinkedIn Jobs, Meta Ad Library, Naver, Quora, Reddit, TikTok Ads Library, Trustpilot and ZipRecruiter. If your project depends on one of them, that is worth knowing before you write the first line.
Almost half name AI crawlers
For 66 of the 137 websites, robots.txt gives at least one AI crawler or AI-training token its own rules, usually to keep it out.
GPTBot is named for 47 websites and ClaudeBot for 44. Six name GPTBot but not ClaudeBot, three the other way round, so most websites that bothered with one listed both. Snapchat names the most: 25 different AI agents.
Naming is not always blocking. Some files name a crawler only to allow it or to slow it down, which is why the per-website pages quote the actual rules rather than a yes/no.
What I take from it
If you are about to build a scraper, an n8n flow or an agent tool around one of these websites, a few things are cheap to check first:
- Look for the official API before anything else. For a quarter of these websites it exists and anyone can use it, and it will outlive any scraper.
- Try the two-file probe yourself. Fetch robots.txt and the terms page with a plain client from where your code will run. If both come back 403, plan for that from day one instead of discovering it in production.
- Read robots.txt as a convention and the terms as the contract. They can disagree, and the terms are the ones that matter legally. (General information, not legal advice.)
What this does not measure
Whether a page needs a real browser to render, shows CAPTCHAs, rate-limits, sits behind a login or blocks IP ranges. One request per file answers none of that, and nothing beyond those two files was requested. The court cases and open-source scraper counts in the data are curated lists, not exhaustive ones.
The data
Everything above comes from one CSV, free to reuse under CC BY 4.0: one row per website with the API status, both plain-request answers and their HTTP status, the robots.txt rule and rule count, crawl delay, AI crawlers named, court cases, open-source repositories and every date.
- The study, with a sortable table and a page per website: scraperfinder.com/what-it-takes
- The data as CSV and JSON (CC BY 4.0), archived with a DOI: 10.5281/zenodo.24197676
The files are re-read every week, so if a website changes its mind about GPTBot, it will show up there. If you spot a fact that is wrong, tell me in the comments with the source and I'll fix it.
Drafted with AI assistance; every number was checked against the published data before posting.


Top comments (0)