DEV Community: Nadeem Akram

Spot the bug: one character breaks this JSON

Nadeem Akram — Thu, 05 Nov 2026 08:19:55 +0000

Quick puzzle 🧩

The JSON in the image looks fine, but JSON.parse() rejects it, and so will most APIs.

One character is the problem. Which line is it on?

Hint: it's a habit JavaScript allows and JSON doesn't.

Comment your line number. We'll post the answer this evening.

When you hit "Unexpected token" for real, paste the JSON into a validator before you hunt through a 500-line file: https://seopeck.com/tool/json-validator

Is the Website Down? How to Actually Check, Step by Step

Nadeem Akram — Tue, 03 Nov 2026 12:38:37 +0000

![Is the Website Down? How to Actually Check, Step by Step]

A step-by-step way to figure out whether a site is down for everyone or just for you, and what each result actually tells you.

A website not loading could mean a dozen different things: the server is down, DNS has not propagated, your own network has an issue, or an SSL certificate has expired. Guessing wastes time. Working through the checks in order tells you exactly where the problem sits.

Step 1: Confirm it is not just you

Before assuming a site is down, rule out your own connection or device. Try loading the site on a different network, such as switching from Wi-Fi to mobile data, or ask someone else to check. If it works for them and not you, the problem is local to your network, browser cache, or DNS resolver, not the website itself.

If you cannot easily ask someone else, SEOpeck's Website Status Checker checks a site's live status from an independent location, which settles the question in one click.

Step 2: Check basic connectivity

The Ping tool tests whether a server is responding at all, and how long it takes to reply. No response at all points toward the server being down or a firewall blocking the request. A slow but successful response points toward a performance problem rather than an outage.

Step 3: Check the HTTP status code

If the site loads but shows an error, the HTTP status code tells you what kind of error it is. A 5xx code (like 500 or 503) means the server itself hit a problem. A 4xx code (like 404 or 403) means the server responded fine but is refusing or cannot find the specific request. The HTTP Status Code Checker shows this directly, and the Redirect Checker is useful when a URL seems to be looping or landing somewhere unexpected. Our guide to what each HTTP status code means covers the codes you are likely to see, and 301 vs 302 redirects explains how redirect loops and chains happen.

Step 4: Check DNS

If the domain will not resolve at all, DNS is a likely suspect, especially after a recent domain or hosting change. If every record has disappeared at once, check that the registration has not lapsed: the Domain Age Checker shows when a domain expires. The DNS Lookup tool shows the current A record and whether it points where you expect; the Hosting Checker tells you which host that address belongs to. If the record types are unfamiliar, DNS records explained covers A, AAAA, MX, TXT and CNAME. If DNS was changed recently, it can take time to fully propagate, so a record that looks correct but is not working everywhere yet is not necessarily broken.

Step 5: Check the SSL certificate

Browsers block access outright to sites with an expired or misconfigured SSL certificate, which can look identical to the site being "down" at first glance. The SSL Checker confirms whether the certificate is valid and how long it has left before it expires, which is worth checking on a recurring basis, since a forgotten renewal is a surprisingly common cause of sudden outages. If the browser shows a certificate warning, SSL certificate errors explained covers each fix.

Step 6: Check the mail and network layer, if relevant

If the issue is specifically email not arriving rather than the website itself, the MX Lookup tool confirms mail is routed correctly. For deeper server-side troubleshooting, the Open Port Checker tests whether a specific port is reachable, which is useful when a service on a server is unreachable but the server itself responds to a ping.

Putting it together

Confirm it is not local, check connectivity, read the status code, check DNS, then check SSL. Working through those five checks in order takes a couple of minutes and almost always identifies exactly where the problem is, instead of guessing and waiting for it to resolve itself.

301 vs 302 Redirects: Which One to Use, and How to Check Them

Nadeem Akram — Fri, 30 Oct 2026 09:34:47 +0000

Originally published at seopeck.com.

A redirect sends visitors and search engines from one URL to another. You need one whenever a page moves, a site switches to HTTPS, two domains merge, or you tidy up old URLs. The two redirects people use most are 301 and 302, and choosing the wrong one is one of the most common technical SEO mistakes.

The short answer

301 (permanent): the old URL is gone for good. Use it for site moves, HTTP to HTTPS, www to non-www, renamed pages and merged content.
302 (temporary): the old URL will come back. Use it for short-term promotions, A/B tests, or sending visitors to a page while the original is being fixed.

If you are not sure, ask yourself: "Will I ever want the old URL to show up in search results again?" If the answer is no, use a 301.

How search engines treat each one

With a 301, search engines gradually replace the old URL with the new one in their index and pass the old page's ranking signals, such as links pointing to it, to the new address.

With a 302, search engines usually keep the original URL indexed because you have said the move is temporary. Google has said that a 302 left in place for a long time may eventually be treated like a 301, but relying on that is slow and unpredictable. If a move is permanent, say so with a 301.

What about 307 and 308?

307 and 308 are the modern, stricter versions of 302 and 301. The difference is that they guarantee the request method stays the same, so a form sent with POST is resent with POST. For ordinary page moves, 301 and 302 are fine; 307 and 308 matter mostly for APIs and forms. Browsers also generate an internal 307 when a site uses HSTS to force HTTPS.

Common redirect mistakes

Redirect chains

A chain is when URL A redirects to B, which redirects to C. Each hop adds delay, and crawlers may stop following after several hops. After a migration, update old redirects so each one points straight to the final URL.

Redirect loops

A loop is when A redirects to B and B redirects back to A. Browsers give up with a "too many redirects" error. Loops often come from conflicting rules, for example one rule forcing www and another removing it.

Redirecting everything to the homepage

When pages are removed, sending all of them to the homepage is tempting but unhelpful. Search engines often treat it as a soft 404. Redirect each old page to the most relevant new page, and let pages with no equivalent return 404 or 410.

Mixing up HTTP and HTTPS, www and non-www

Pick one canonical version of your domain, such as https://example.com, and 301 every other variation to it in a single hop.

How to create a redirect

On an Apache or LiteSpeed server, redirects usually live in the .htaccess file:

Redirect 301 /old-page https://example.com/new-page

The .htaccess Redirect Generator writes these rules for you, including whole-domain moves. On Nginx, redirects go in the server configuration, and most content management systems have a redirect plugin or setting.

How to check a redirect

Paste the old URL into the Redirect Checker. It shows every hop, the status code of each one and where the chain ends. Check that there is exactly one hop, that it is a 301 for permanent moves, and that the final page returns 200. The HTTP Status Code Checker is a quick way to confirm the destination works, and the guide to HTTP status codes explains the rest of the codes you may see along the way.

How to Check MX Records (and Fix Email Not Arriving)

Nadeem Akram — Tue, 27 Oct 2026 09:14:25 +0000

See any domain’s MX records in seconds, online or with nslookup and dig, learn to read the priority numbers, and work through the checks that find out why email is not arriving.

To check a domain’s MX records, type the domain into an MX Lookup tool, or run nslookup -q=mx example.com on Windows or dig mx example.com +short on Mac and Linux. You get back a list of mail servers, each with a priority number. The lowest number is the server other mail systems try first.

If email to your domain bounces or never shows up, the rest of this guide shows how to read the result, what correct records look like for Google Workspace, Microsoft 365 and Zoho, and the checks to run, in order.

What an MX record does

When someone sends a message to you@yourdomain.com, their mail server asks DNS one question: "Which servers accept mail for yourdomain.com?" The answer is the domain's MX (mail exchanger) records. Each one names a mail server and gives it a preference number. The sender tries the lowest number first and moves to higher numbers only if that server cannot be reached.

Here are the two MX records for seopeck.com, as a DNS zone file would show them:

seopeck.com. 14400 IN MX 5 mx1.hostinger.com. seopeck.com. 14400 IN MX 10 mx2.hostinger.com.

Mail goes to mx1.hostinger.com (priority 5) first, and to mx2.hostinger.com (priority 10) as a backup. The 14400 is the TTL: resolvers may cache this answer for four hours. MX records only decide where mail is delivered, not how it is sent.

The quickest way: check MX records online

An online lookup is the fastest option and works from any device, including a phone.

Open the MX Lookup tool.
Type the domain, for example seopeck.com. You can also paste a full page address or an email address, such as https://www.seopeck.com/blog or you@seopeck.com; the tool reduces it to the domain.
Click Lookup.

For seopeck.com the result is two lines, sorted from the lowest priority number to the highest:

Priority: 5, MX Host: mx1.hostinger.com, IP Address: 172.65.182.103 Priority: 10, MX Host: mx2.hostinger.com, IP Address: 172.65.182.103

Each line has a Copy link for the hostname, and the share menu below lets you copy a link to the result or save it as a report for your host or IT person. If the tool says No MX records found for this domain, jump to the troubleshooting list below.

To see the MX records next to the domain’s A, TXT and CNAME records in one go, use the DNS Lookup tool instead; our walkthrough on checking all of a domain’s DNS records shows how to read that output.

Check MX records from the command line

Windows: nslookup

Open Command Prompt or PowerShell and run:

nslookup -q=mx seopeck.com

Real output:

Non-authoritative answer: seopeck.com MX preference = 10, mail exchanger = mx2.hostinger.com seopeck.com MX preference = 5, mail exchanger = mx1.hostinger.com

-type=mx does the same thing as -q=mx. Notice the order: nslookup lists the records in whatever order the DNS server sent them, not by priority. Always read the preference number, not the line order. "Non-authoritative answer" only means the reply came from a cache rather than from the domain's own name servers. It is normal.

In PowerShell you can also use Resolve-DnsName seopeck.com -Type MX, which prints a table with NameExchange and Preference columns.

Mac and Linux: dig and hos

dig mx seopeck.com +shor

10 mx2.hostinger.com. 5 mx1.hostinger.com.

Without +short, dig prints the full answer section, including the TTL. The host command gives a more readable line per record:

host -t mx seopeck.com

It prints lines in the form seopeck.com mail is handled by 5 mx1.hostinger.com. The trailing dot on the hostnames is normal DNS notation for a fully qualified name.

To ask one specific DNS server, add it to the command: nslookup -q=mx seopeck.com 8.8.8.8 or dig mx seopeck.com @8.8.8.8. That is useful right after a change, when your own network may still have the old answer cached.

How to read the result

Priority (preference)

Order in which senders try the servers. Lower is tried first.

Your provider’s main server has the lowest number. Servers with the same number share the load at random.

Host (mail exchanger)

The hostname of a server that accepts mail for the domain.

It belongs to your current email provider, not a previous one.

The IPv4 address that hostname resolves to.

A real address. In MX Lookup, if the IP column repeats the hostname instead of showing numbers, that hostname did not resolve to an IPv4 address and mail sent to it cannot be delivered.

The actual priority numbers do not matter, only their order. 5 and 10 behave exactly like 1 and 2.

What correct MX records look like for Google Workspace, Microsoft 365 and Zoho

These values come from each provider’s own setup pages (checked October 2026). Your provider’s admin console always has the final word for your domain.

A few notes:

Google: Google says the older aspmx records still work, so there is no need to change them if mail is arriving. Do not mix the old set with smtp.google.com, and remove every MX record that is not Google's.
Microsoft 365: copy the exact value from the admin center (Settings, Domains, DNS records). Microsoft is moving new domains to hosts under mx.microsoft, so do not assume the older mail.protection.outlook.com pattern.
Zoho: the domain ending depends on the data center your account uses (for example a .eu or .in account). Use the values shown in your Zoho admin console.

Email isn’t arriving? Check these in order

No MX records at all. If MX Lookup finds nothing, check the spelling of the domain first. If it is correct, the records were never added, or were lost when the domain’s name servers changed (for example after moving DNS to a new host or to Cloudflare). Add the records your email provider gives you at whichever company runs your DNS today.
A null MX. A single record with priority 0 and a host of just “.” (nslookup shows it as (root), dig as 0 .) is a "null MX", defined in RFC 7505. It means "this domain accepts no email", and senders give up immediately. example.com has one on purpose. If yours does and you want email, replace it with your provider's records.
A typo in the hostname. One wrong letter and the mail server name does not exist. Compare the host character by character with your provider’s setup page, and watch for a domain added twice, such as smtp.google.com.yourdomain.com. Some DNS panels append your domain unless the value ends with a dot.
The MX points to an IP address or a CNAME. An MX record must name a host, and that host must have its own A or AAAA record (RFC 5321, section 5.1). An IP address typed into the host field is read as a hostname and will not work. RFC 2181 also says an MX must not point to an alias, so do not aim it at a hostname that is a CNAME. Point it at a real mail server name and check that name resolves with Hostname to IP.
The old provider is still listed. After a move, leftover MX records from the previous host mean some mail still goes there. Google and Microsoft both tell you to remove the old records. If the old server has a lower number than the new one, almost all mail goes to the old host.
The change has not reached everyone yet. Resolvers keep the previous answer until its TTL runs out. See the next section.
Mail arrives, but in spam. Then MX records are fine; the problem is trust. Check that the domain publishes SPF, DKIM and DMARC records with the SPF, DKIM & DMARC Checker, that the sending server’s IP is not on a blocklist with the Email Blacklist Checker, and that its IP has reverse DNS with IP to Hostname. Pasting a received message’s headers into the Email Header Analyzer shows which servers handled it and whether SPF, DKIM and DMARC passed.

After you change MX records: how long it takes

New MX records work as soon as your DNS host publishes them, but resolvers that cached the old answer keep using it until the old TTL expires. With a TTL of 3600 (one hour) most of the world sees the change within an hour or two. With 14400 it can take four hours or more. Providers quote a safe upper limit: Google says up to 72 hours.

Two tips make the switch smoother:

A day before a planned move, lower the TTL on the existing MX records to 300 or 3600 seconds, so the change spreads faster when you make it.
Create the mailboxes at the new provider before you change the MX records, so mail that arrives there has somewhere to go.

To watch the change spread, enter the domain in the DNS Propagation Checker and choose MX. It shows which DNS servers around the world still return the old host. Our guide to DNS records covers TTLs and the other record types in more detail.

Frequently asked questions

How do I check MX records for my domain?

Enter the domain in the MX Lookup tool, or run nslookup -q=mx yourdomain.com (Windows) or dig mx yourdomain.com +short (Mac, Linux).

Why does MX Lookup say no MX records were found?

The domain is misspelled, does not exist, or has no MX records published at the DNS host that is currently in charge of it. Without MX records, senders may fall back to the domain’s A record, but that rarely points to a working mail server.

Can an MX record point to an IP address or a CNAME?

No to an IP address: the value must be a hostname that has its own A or AAAA record. A CNAME is not allowed either (RFC 2181). Some mail servers will still follow one, but many will not, so do not rely on it.

How long do MX record changes take to work?

Usually within the old record’s TTL, often one to four hours. Providers quote up to 48 or 72 hours as a worst case.

Do I need more than one MX record?

Not always. Google Workspace and Microsoft 365 each use a single record, and senders retry for days if a server is briefly down. Only add backups your provider gives you.

My MX records are correct, but email still goes to spam. Why?

MX records only route incoming mail. Spam placement depends on the sender’s SPF, DKIM, DMARC, reverse DNS and reputation. Start with the SPF, DKIM & DMARC Checker.

Next step

Run your domain through MX Lookup now and compare the hosts with the provider table above. If a whole website is unreachable rather than just its email, our website down troubleshooting guide walks through the checks for that.

About the author

Nadeem Akram Founder and developer of SEOpeck

Nadeem builds and maintains the SEOpeck tools and writes the guides on this blog. He is a web developer who works on SEO, performance and the everyday problems of running a website.

Originally published at https://seopeck.com on October 27, 2026.

Which AI Crawlers Do the Top 1,000 Websites Block? (robots.txt Study, October 2026)

Nadeem Akram — Sat, 24 Oct 2026 14:03:00 +0000

We read the robots.txt files of the 1,000 most popular websites: 20.9% block at least one of 16 AI bots. Per-bot shares, site strategies, common mistakes and open data.

On 12 October 2026 we fetched the robots.txt file of each of the 1,000 most popular websites and checked 16 AI crawlers against it. 209 of the 1,000 sites (20.9%) block at least one AI bot from the whole site; 789 (78.9%) let every one of them in. Training crawlers are blocked most often (CCBot on 15.5% of sites, GPTBot on 13.8%), bots that fetch pages for AI search less often (OAI-SearchBot on 7.0%).

Below are the five main findings, the bot-by-bot table, the strategies sites use, the mistakes we found and what it means for your own robots.txt. The per-site data and the method are open, so you can check any number.

Five key findings

About one site in five blocks AI bots. 20.9% block at least one of the 16 AI crawlers from their home page. Only 2.0% (20 sites) block all 16.
Training is blocked more than search. 20.6% block at least one training crawler, 12.4% at least one AI search bot and 11.8% at least one bot that fetches a page when a person asks for it.
Half of the sites that block GPTBot still let OAI-SearchBot in: 70 of 138. For Anthropic, 58 of the 131 sites that block ClaudeBot still allow Claude-SearchBot.
The biggest sites block most. 27% of the top 100 block at least one AI bot, against 23.8% of sites ranked 101–500 and 17.4% of sites ranked 501–1,000.
Files drift. 109 sites still list the retired anthropic-ai name and 94 list Claude-Web; 49 sites serve a web page instead of a robots.txt file, so none of their rules exist for a crawler.

Who we checked, in short

We walked down the Tranco research ranking (list 94GG2) and kept the first 1,000 domains that serve a real website, which took 2,364 domains: the raw list is full of infrastructure domains with no site behind them. For each site we read one robots.txt file with our own research bot, SEOpeckBot, and parsed it under RFC 9309, the robots.txt standard, using the same code as our AI Crawler Checker.

"Blocks a bot" in this study means the bot may not fetch the home page (/) while Googlebot still may. A site that only keeps a bot out of some folders is not counted as blocking. 168 sites refused our bot outright and are left out; the full method and its limits are at the end.

Three kinds of AI bot, not two

Most AI crawler studies split bots into "training" and "search". There is a third kind, and site owners treat each one differently:

Training crawlers collect pages to build AI models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta) and Amazonbot (Amazon). Google-Extended and Applebot-Extended belong here too, although they are control tokens rather than separate crawlers: Google and Apple read them to decide whether pages fetched by their normal crawlers may be used for AI models.
AI search bots index pages so an assistant can find and cite them: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Amzn-SearchBot.
User-triggered fetchers visit a page only when a person asks the assistant about it or pastes the link: ChatGPT-User, Claude-User, Perplexity-User and Amzn-User. Some operators say these fetchers may not apply robots.txt because a person asked for the page; Perplexity says so about Perplexity-User. Check each operator's documentation.

Websites blocking at least one AI bot of each kind

Training20.6%
AI search12.4%
User-triggered11.8%
Any of the 16 AI bots20.9%

0%25%

Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Kind of bot	Bots in the study	Sites blocking at least one
Training	8	206 (20.6%)
AI search	4	124 (12.4%)
User-triggered	4	118 (11.8%)
Any of the 16	16	209 (20.9%)

Almost every site that blocks AI bots blocks training: only 3 of the 209 block a search or user-triggered bot without blocking any training crawler. 48 sites block all eight training crawlers.

Bot by bot: who gets blocked

The table shows, for each of the 16 AI bots, the share of the 1,000 websites that block it from the whole site, and the share whose robots.txt names it at all.

Share of the 1,000 websites that block each AI bot site-wide

Training
AI search
User-triggered

CCBot15.5%
Bytespider15.2%
GPTBot13.8%
ClaudeBot13.1%
Meta-ExternalAgent11.8%
Google-Extended11.2%
PerplexityBot10.5%
Amazonbot9.8%
Applebot-Extended9.7%
ChatGPT-User9.0%
Perplexity-User8.4%
Claude-User7.9%
Claude-SearchBot7.5%
OAI-SearchBot7.0%
Amzn-SearchBot5.1%
Amzn-User4.8%

0%20%

Google-Extended and Applebot-Extended are control tokens, not separate crawlers. Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Bot	Operator	Kind	Blocked site-wide	Named in robots.txt
CCBot	Common Crawl	Training	155 (15.5%)	17.0%
Bytespider	ByteDance	Training	152 (15.2%)	14.8%
GPTBot	OpenAI	Training	138 (13.8%)	19.5%
ClaudeBot	Anthropic	Training	131 (13.1%)	17.5%
Meta-ExternalAgent	Meta	Training	118 (11.8%)	11.6%
Google-Extended	Google	Training (control token)	112 (11.2%)	16.2%
PerplexityBot	Perplexity	AI search	105 (10.5%)	16.4%
Amazonbot	Amazon	Training	98 (9.8%)	10.4%
Applebot-Extended	Apple	Training (control token)	97 (9.7%)	10.4%
ChatGPT-User	OpenAI	User-triggered	90 (9.0%)	14.5%
Perplexity-User	Perplexity	User-triggered	84 (8.4%)	9.1%
Claude-User	Anthropic	User-triggered	79 (7.9%)	8.6%
Claude-SearchBot	Anthropic	AI search	75 (7.5%)	8.9%
OAI-SearchBot	OpenAI	AI search	70 (7.0%)	13.4%
Amzn-SearchBot	Amazon	AI search	51 (5.1%)	2.3%
Amzn-User	Amazon	User-triggered	48 (4.8%)	2.0%

Two things stand out. First, naming a bot is not the same as blocking it: 195 sites name GPTBot, but 74 of them still let it fetch the home page, usually because they keep it out of some folders only. Second, Bytespider, Meta-ExternalAgent and the two Amazon AI bots are blocked by more sites than name them. That comes from "allowlist" files, explained in the next section.

For comparison, the search engines we used as controls are almost never blocked: Googlebot on 2 sites (0.2%) and Bingbot on 7 (0.7%).

Site strategies: where does your site fit?

Counting bots one by one hides how sites actually decide. So we put each site into one strategy, based on which kinds of AI bot it blocks from the home page.

How the 1,000 websites treat AI bots (one strategy per site)

Open to all AI bots78.9%
Blocks at least one bot of each kind11.0%
Blocks training only7.8%
Blocks training and AI search1.1%
Other mix1.0%
Blocks all crawlers, Googlebot included0.2%

0%80%

Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Strategy	Sites	Share
Open to all AI bots (no site-wide AI block)	789	78.9%
Blocks at least one bot of each kind (training, AI search and user-triggered)	110	11.0%
Blocks training only	78	7.8%
Blocks training and AI search, not user-triggered	11	1.1%
Other mix (for example search but not training)	10	1.0%
Blocks all crawlers, Googlebot included	2	0.2%

Read the second row carefully: blocking at least one bot of each kind is not the same as blocking every AI bot. Of those 110 sites, only 20 block all 16.

About 3% of sites, 34 in all, run an allowlist: a User-agent: * group with Disallow: /, plus groups that let named search engines in. Every crawler the file does not name, AI bots included, is shut out. We count these sites as blocking, because that is what the file tells an AI bot. 33 of the 110 sites in the second row work this way, and so do 16 of the 20 sites that block all 16 bots. pinterest.com is one example: none of the 16 AI bots is named, and all of them are blocked through the * group. Other sites list the bots one by one: bbc.com blocks 14 of the 16 by name and cnn.com 15.

Block training, allow AI search: the middle path

The 78 "training only" sites are the clearest group, but the pattern runs wider. Of the 138 sites that block GPTBot, 70 (50.7%) still let OAI-SearchBot fetch the home page. Of the 131 that block ClaudeBot, 58 (44.3%) still allow Claude-SearchBot. Perplexity has no separate training crawler, and 105 sites (10.5%) block PerplexityBot.

forbes.com is one example of the approach: its file blocks GPTBot and ClaudeBot and has groups that allow OAI-SearchBot, Claude-SearchBot and Claude-User. The logic is simple: a training crawler takes content without sending visitors, while an AI search bot can cite the page and link to it. Here is a minimal version of that setup, checked with our parser:

# Block AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

# Everyone else, including AI search and user-triggered fetchers
User-agent: *
Disallow: /admin/

Bigger sites block more

The share of sites blocking AI bots falls as you go down the ranking. "Rank" here is a site's place among the 1,000 websites in the study, not its raw Tranco rank.

Bigger sites block more

Block at least one AI bot
Block GPTBot

Rank 1–10027.0%20.0%
Rank 101–50023.8%15.5%
Rank 501–1,00017.4%11.2%

0%30%

Rank is the place among the 1,000 websites in the study. Source: SEOpeck robots.txt study, October 2026 (1,000 websites, Tranco list 94GG2).

Rank	Sites	Block at least one AI bot	Block GPTBot
1–100	100	27 (27.0%)	20 (20.0%)
101–500	400	95 (23.8%)	62 (15.5%)
501–1,000	500	87 (17.4%)	56 (11.2%)

Drift and mistakes

robots.txt files collect old lines. These are the patterns we counted, described as neutrally as we can: some are deliberate choices we cannot see the reason for.

Legacy names

109 sites still list anthropic-ai and 94 list Claude-Web, names Anthropic no longer documents; its current bots are ClaudeBot, Claude-SearchBot and Claude-User. Other agent names outside our 16 are common too: cohere-ai appears on 115 sites, Diffbot on 106 and YouBot on 94.

Old names do little harm next to current ones, and Anthropic has said ClaudeBot honours rules written for its retired names. Only 7 sites rely on the legacy names alone, with no ClaudeBot line. We call these legacy-only rules rather than broken ones, but a parser that follows RFC 9309 to the letter would not apply them to ClaudeBot. If you want to block Anthropic's crawler, name ClaudeBot.

One operator, mixed rules

Blocking GPTBot while allowing OAI-SearchBot is a clear choice. The reverse pairing for user-triggered bots is harder to explain: 22 sites block ChatGPT-User but allow OAI-SearchBot, 7 do the same with Claude-User and Claude-SearchBot, and 7 with Perplexity-User and PerplexityBot. The search bot may index the page, but the assistant cannot open it when a person asks about it. Worth a second look if that is your file.

Lines a strict parser ignores

9 sites have lines that RFC 9309 does not recognise: 5 use a versioned name such as User-agent: GPTBot/1.0, and 4 have Allow or Disallow rules before any User-agent line, so the rules belong to no group. Some crawlers are forgiving about these, but a strict parser, ours included, skips them. Write the bare product token: User-agent: GPTBot.

A web page instead of a robots.txt file

49 sites (4.9%) answer /robots.txt with an HTML page, usually their home page or an error page sent with a 200 status. Under RFC 9309 that is not a valid robots.txt file, so a crawler finds no rules in it and treats the whole site as open. If you think you have AI rules, check that the URL returns plain text.

New signals: llms.txt and Content-Signal

llms.txt: 192 of the 1,000 websites serve a file at /llms.txt. We fetched it only where robots.txt allowed our bot to, which left 42 sites unchecked, so the share is 20.0% of the 958 we could check. llms.txt is a proposed format, not a standard; a file being there says nothing about whether any AI system reads it. We counted any non-empty plain-text answer and did not validate the format.
Content-Signal: 30 sites (3.0%) have a Content-Signal: line, the format Cloudflare introduced to state preferences such as ai-train=no. 15 of them say ai-train=no, but only 5 of those 15 also block GPTBot in robots.txt. A Content-Signal line states a preference; it does not block anything.
Cloudflare's policy text: 7 sites include the licence-style text of Cloudflare's Content Signals Policy in their robots.txt.

What this means for your site

Decide per kind of bot, not per company. Blocking training says "don't learn from my pages". Blocking AI search and user-triggered fetchers also takes you out of the answers and links those assistants show. Our guide Should you block AI crawlers? walks through the trade-off.
Use current names. Name ClaudeBot, Claude-SearchBot and Claude-User, not only anthropic-ai. Write each name bare, without a version number.
Check what your file really does. Paste your domain into the AI Crawler Checker: it uses the same parser as this study and shows each bot as allowed, partly blocked or blocked. The Robots.txt Tester shows whether a URL is open for any user agent, and the Robots.txt Generator writes a clean file if yours has drifted.
Remember robots.txt is a request. The large AI companies say their crawlers follow it, but it cannot stop a bot that ignores it. For a hard block, use your firewall or CDN as well.

How this compares with earlier studies

The comparison is directional only: samples and methods differ. In 2023, Originality.ai counted 242 of the top 1,000 sites blocking GPTBot within weeks of its launch; we count 138 of 1,000 websites. Cloudflare's 2025 analysis of about 3,900 robots.txt files from the top 10,000 domains found GPTBot was the most disallowed bot; in our data it ranks third, after CCBot and Bytespider. Chris Humphrey's June 2026 study of the Majestic top 10,000 found 22.6% blocking at least one of 10 bots and 4% blocking all ten; we find 20.9% blocking at least one of 16 and 2.0% blocking all 16.

Full method

Sample

The list is Tranco list 94GG2, generated on 2 October 2026 from 30 days of data (3 September to 2 October 2026). We checked domains in rank order and kept a domain as a website if its home page answered with a 2xx HTML page on the same registrable domain, stopping at 1,000 websites. That took 2,364 domains. The 1,364 skipped domains break down as: no connection 694, robots.txt or home page redirecting to another domain 221, home page not answering 2xx 189, robots.txt refused (401/403) 168, home page disallowed for our bot and not open to Googlebot 37, home page not connecting 23, robots.txt server error 12, home page not HTML 9, timeout 9, rate-limited (429) 2.

Near-duplicates are kept as Tranco lists them (for example bbc.com and bbc.co.uk, or x.com and twitter.com), so the 1,000 websites are not 1,000 separate organisations.

Fetching

The crawl ran on 12 October 2026 from our own machine, not a server. SEOpeckBot fetched /robots.txt first (up to 5 redirects; if there was no valid file, once more on the www. or bare version of the domain, which found the file for 72 sites), then the home page only if robots.txt allowed it, then /llms.txt only if allowed. 35 sites disallow our bot's home page through their * group but let Googlebot in; we counted them as websites without fetching the page. The bot sent at most 12 requests at once, waited at least 2 seconds between requests to the same host, gave up after 10 seconds without a connection or 20 seconds in total, and read at most 500 KiB of a robots.txt file. Its user agent was:

SEOpeckBot/1.0 (robots.txt study; +https://seopeck.com/page/robots-txt-study)

The linked page explains the study and how to opt out. At the time of the crawl, no site had opted out or asked to be removed.

Counting

Of the 1,000 websites, 890 had a valid robots.txt, 59 had none (404 or 410), 49 served an HTML page and 2 gave another 4xx answer. Under RFC 9309 a missing or invalid file means no restrictions, so all 1,000 are in the denominator. The 168 sites that refused our bot (401 or 403) are not: a refusal says nothing about their rules. Counted as open sites instead, the headline figure would be 17.9% (209 of 1,168) rather than 20.9%.

Each file was parsed with the same code as the AI Crawler Checker, following RFC 9309: a bot obeys the group or groups that name it (case-insensitive), otherwise the * group; within a group the longest matching rule wins, and Allow wins a tie. A bot counts as blocked when / is disallowed for it. AI blocks are counted only where Googlebot may still fetch /; the 2 sites that block Googlebot as well form their own strategy. We classify Amazonbot as a training crawler, as our checker does. Legacy names such as anthropic-ai are counted when a file names them, not evaluated as live crawlers. After the crawl we re-fetched 25 random sites and re-parsed them; every result matched.

Limits

robots.txt is a stated policy. We did not test whether bots obey it, and many sites block AI bots at the firewall or CDN without any robots.txt rule, so the real share of sites keeping AI bots out can be higher.
It is one snapshot. robots.txt files change often; the per-site data shows each file as it was on 12 October 2026.
We only test the home page. A site that blocks an AI bot from its articles but not from / counts as open.
Tranco combines five popularity rankings; it is not a count of visits. The 168 sites that refused our bot are missing from the sample.

Rankings: Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. 2019. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." In Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS 2019). https://doi.org/10.14722/ndss.2019.23386. List used: Tranco list 94GG2, generated on 2 October 2026, available at https://tranco-list.eu/list/94GG2.

Download the data

Per-site CSV: one row per website with its Tranco rank, robots.txt URL and HTTP status, strategy, the result for all 16 AI bots and the two search engines, legacy names, ignored lines, Content-Signal and llms.txt.
Summary JSON: every figure in this post.

You are welcome to use the data and charts in your own work; please credit "SEOpeck robots.txt study, October 2026" with a link to this page. We plan to repeat the study every quarter and publish what changed.

Cite this study

APA: Akram, N. (2026, October 20). Which AI crawlers do the top 1,000 websites block? (robots.txt study, October 2026). SEOpeck. https://seopeck.com/blog/ai-crawlers-robots-txt-top-1000-websites

Link: SEOpeck robots.txt study, October 2026

What Is GZIP Compression, and Is It Working on Your Site?

Nadeem Akram — Fri, 23 Oct 2026 13:03:01 +0000

How GZIP and Brotli compression make pages load faster, what should and should not be compressed, how to switch it on, and how to check it is working.

Most of what a browser downloads to show a web page is text: HTML, CSS, JavaScript, JSON and SVG. Text compresses extremely well, often shrinking by 70 percent or more. HTTP compression uses that fact to send smaller files over the network, and it is one of the easiest speed improvements a site can make.

How it works

When a browser requests a page, it tells the server which compression methods it understands with a header such as Accept-Encoding: gzip, deflate, br. If the server supports one of them, it compresses the response and says which method it used with Content-Encoding: gzip (or br). The browser decompresses the file before using it. Visitors never notice anything except that the page arrives sooner.

GZIP vs Brotli

GZIP has been the standard for decades and is supported by essentially every browser and server. Brotli is a newer algorithm that usually produces smaller files for text, especially at higher compression levels. All modern browsers support Brotli over HTTPS. The best setup serves Brotli to browsers that ask for it and GZIP to everyone else; many hosts and CDNs do this automatically.

What to compress, and what not to

Compress: HTML, CSS, JavaScript, JSON, XML, SVG, plain text and web fonts in older formats such as TTF.
Do not bother: JPG, PNG, WebP, GIF, MP4, WOFF2 fonts and ZIP files. They are already compressed, so compressing them again wastes server time and can even make them slightly larger.
Images need a different approach: resizing, choosing the right format and lossy compression. Our guide on compressing images for the web covers that.

How to switch it on

Managed hosting and CDNs: compression is often already on, or available as a setting in the control panel.
Apache and LiteSpeed: enable mod_deflate (or the server's built-in compression) and list the text content types to compress, usually in the server config or .htaccess.
Nginx: set gzip on; and a gzip_types list in the configuration, and add the Brotli module if you want Brotli too.
After making changes, clear any cache or CDN layer so you are testing the new behaviour.

How to check it is working
Enter your page in the GZIP Compression Test. It requests the page with compression allowed and reports whether the server compressed it and how large the page is. You can also see the Content-Encoding header directly with the HTTP Headers Parser, and the On-Page SEO Checker flags compression alongside its other checks.

Remember to test your CSS and JavaScript files as well as the HTML page, since they are often the largest text files a page loads.

Beyond compression

Compression makes each file smaller; it does not reduce how many files you send. Once compression is on, look at removing unused CSS and JavaScript, minifying code with the CSS Minifier and JavaScript Minifier, and caching static files so returning visitors do not download them again.

Free PDF Tools That Never Upload Your Files

Nadeem Akram — Mon, 12 Oct 2026 15:08:32 +0000

A note for dev.to readers: these tools use pdf-lib to edit files, pdf.js to render pages, and qpdf compiled to WebAssembly for AES-256 encryption, all running client-side. Nothing is uploaded.

Contracts, bank statements, payslips, medical letters, ID scans: the PDFs we most often need to merge, shrink or lock are also the ones we least want to hand to a stranger. Yet the usual way to do it online is to upload the file to a website, let its server process it, and download the result. This guide explains what that actually involves, and how to do the same jobs without your file ever leaving your computer.

What happens when you "upload" a PDF

A typical online PDF tool copies your document to its server, works on it there, and keeps the result long enough for you to download it. Reputable services delete files after an hour or so, but you are relying on their word, their security, and every company they use for storage and logging. For a restaurant menu that is fine. For a signed contract or a passport scan, it is a real risk that is easy to avoid.

The alternative: tools that run in your browser

Modern browsers are powerful enough to read and rewrite PDF files themselves. SEOpeck's PDF Tools work this way: the page loads the program, and your file is opened, processed and saved by your own browser. Nothing is sent to our server, so there is nothing for us to store, leak or delete. You can check this yourself: open the browser's developer tools, switch to the Network tab and run a tool. You will not see your file go anywhere.

There is a side benefit, too. Because there is no upload or download round trip, large files are often faster to process locally than online, and the tools keep working on a slow connection once the page has loaded.

What you can do

Combine and rearrange

Merge PDF joins several PDFs into one, in the order you choose.
Split PDF breaks a document into separate files by page range.
Organize PDF lets you drag pages into a new order, and Rotate PDF fixes sideways scans.
Delete PDF Pages and Extract PDF Pages remove the pages you do not need, or pull out the ones you do.

Make files smaller

Compress PDF reduces file size so a document fits under an email attachment limit or an upload form's cap. Scanned documents usually shrink the most, because they are really a stack of photos.

Convert

PDF to JPG and PDF to PNG turn each page into an image.
JPG to PDF and PNG to PDF turn photos and screenshots into a single PDF.
PDF to Text pulls out the text so you can copy, search or reuse it.

Finish and sign

Add Page Numbers and Watermark PDF for reports and drafts.
Sign PDF to draw or type a signature and place it on the page.
PDF Metadata Editor to change or clear the title, author and other hidden details before you share a file.

Password-protect a PDF

The newest additions are Protect PDF and Unlock PDF. Protect PDF encrypts a document with AES-256, the same standard used by banks and by Adobe Acrobat, so it cannot be opened without the password. You can also switch off printing, copying or editing for people who do open it.

A few tips for passwords on documents:

Use a long passphrase, such as four unrelated words, rather than a short word with a number on the end. The encryption is only as strong as the password. Our Password Generator can make one for you.
Send the password separately. If the PDF goes by email, send the password by text message or say it on a call. A password in the same email protects nothing.
Keep a copy. There is no "forgot password" for an encrypted PDF. If the password is lost, the file is effectively lost too.
Restrictions are a polite request. Blocking printing or copying stops casual reuse in normal PDF readers, but it is not a strong lock. The open password is the real protection.

Remove a password you know

Password-protected files are a nuisance when you have to open them every day, such as your own bank statements. Unlock PDF takes the file and its password and saves a copy without encryption. It will not guess or crack a password you do not have; that is deliberate. Only remove protection from documents you own or are entitled to open.

When a server-based tool is still the right choice

Browser tools are limited by your device. A very large PDF of several hundred megabytes may be slow on an older phone, and some jobs, such as turning a scanned image into editable text with OCR or converting to Word with perfect layout, still need heavier software. For the everyday jobs above, though, there is no need to upload anything.

Summary

If a document contains anything personal, prefer a PDF tool that works inside your browser. All of SEOpeck's PDF Tools are free, need no account, add no watermark, and never send your file to our server.

Originally published on SEOpeck.

A 5-minute SEO checklist for your homepage

Nadeem Akram — Sat, 26 Sep 2026 10:16:46 +0000

A 5-minute SEO checklist for your homepage

Your homepage is usually the page with the most links pointing at it, so it is the one search engines and social networks look at first. You don't need an audit tool with a monthly fee to catch the most common problems. Five checks cover most of them, and each takes about a minute.

Open your homepage, right-click, choose View page source, and search (Ctrl+F) for the tags below.

1. One H1 that says what the page is about

Search for <h1. You want exactly one, and it should describe the page in plain words.

<h1>Free SEO and developer tools, no sign-up</h1>

Common problems: the H1 is missing because the logo is the only heading, there are several H1s because a theme reused the tag for styling, or the H1 just says "Welcome" or "Home". Headings help search engines and screen-reader users understand the page, so they should be about the content, not the layout.

2. A meta description you wrote yourself

Search for name="description".

<meta name="description" content="130+ free SEO and developer tools: check a page's SEO, test robots.txt, compress images and more. No sign-up.">

Google doesn't use the description for ranking, and it sometimes rewrites the snippet. A good description still helps, because it is often what people read in the search results before deciding to click. If the tag is missing, Google picks some text from the page itself. Keep it to about 150 to 160 characters and describe what the visitor gets.

3. A canonical URL, and no stray "noindex"

Search for rel="canonical" and for noindex.

<link rel="canonical" href="https://example.com/">

The canonical tag tells search engines which version of the URL is the main one (https or http, with or without www, with or without tracking parameters). It should point to the page's own preferred address.

A noindex in a robots meta tag tells search engines to leave the page out of their results. It often gets left behind after a site moves from staging to production, and it will quietly keep a homepage out of Google. It can also come as an X-Robots-Tag HTTP header, which doesn't appear in the page source, so check the response headers too.

4. Alt text on every image that carries meaning

Search for <img and look at the alt attributes.

<img src="dashboard.png" alt="SEO Checker report with a score of 92 out of 100">
<img src="divider.svg" alt="">

Alt text is read by screen readers and helps image search understand the picture. Describe what the image shows. Purely decorative images should have an empty alt="" so screen readers skip them. Leaving out the attribute entirely is the one option to avoid.

5. Open Graph tags, so shared links look right

Search for og:.

<meta property="og:title" content="SEOpeck: free SEO and developer tools">
<meta property="og:description" content="130+ free tools, no sign-up.">
<meta property="og:image" content="https://example.com/og-image.png">

Facebook, LinkedIn, Slack, WhatsApp and many other apps use these tags to build the preview card when someone shares your link. Without them you get a bare URL or a random image from the page. An image of 1200 × 630 px works well almost everywhere.

Check all five at once

If you'd rather not read page source, I built a free SEO Checker that tests all five of these on any URL. It also checks HTTPS, redirects, response time, subheadings, word count, the mobile viewport and structured data, and gives a score out of 100 with a short fix for each problem. No sign-up needed.

Run it on your homepage, fix anything marked as a problem, and then run it on your two or three most important pages. That's usually the fastest SEO win a small site can get.

Should You Block AI Crawlers? A Plain Guide to robots.txt and AI Bots

Nadeem Akram — Thu, 24 Sep 2026 21:07:43 +0000

Originally published at seopeck.com.

Alongside Googlebot and Bingbot, most websites are now visited by crawlers from AI companies. Some collect text to train language models; others fetch pages so an AI assistant can answer a question and cite its source. Site owners are increasingly asking the same question: should I let them in?

There is no single right answer. It depends on what you publish and what you want in return. This guide explains the main crawlers, what they are used for, and how to control them.

Two kinds of AI crawler

Training crawlers collect content to build and improve AI models. Examples are OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent and Common Crawl's CCBot, whose open dataset is used by many AI projects. Blocking these keeps your future content out of new training runs. It does not remove anything that was collected before.

Search and user crawlers fetch pages in response to what people ask right now. Examples are OAI-SearchBot and ChatGPT-User from OpenAI, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot. These are the bots that can show your page as a source and send visitors to it.

There are also control tokens that are not separate crawlers at all. Google-Extended lets you opt out of your content being used for Google's Gemini models without affecting Google Search. Applebot-Extended does the same for Apple's AI training.

How robots.txt rules work

robots.txt is a plain text file at the root of your site, for example https://example.com/robots.txt. It is made of groups. Each group names one or more crawlers with User-agent lines, followed by Allow and Disallow rules:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Two details trip people up:

A crawler follows only the most specific group that names it. In the example above, GPTBot obeys its own group and ignores the * group completely. A crawler that is not named anywhere falls back to *.
Within a group, the longest matching rule wins. Disallow: / with Allow: /blog/ means the blog is allowed and everything else is blocked.

Common setups

Block AI training, allow AI search. This is a popular middle ground for publishers who want citations and visits but do not want their work used to train models:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Allow everything. If you want the widest possible reach, including being part of the data AI tools learn from, you do not need any AI-specific rules at all.

Block everything AI. Add the search and user crawlers to the list above as well. Be aware that you will no longer appear as a source in those assistants' answers.

What robots.txt cannot do

robots.txt is a request, not a wall. The large AI companies say their crawlers respect it, but a badly behaved bot can ignore it. If you need to stop a crawler for certain, block it at the server or firewall level as well. Also remember that robots.txt is public: anyone can read it, so never use it to hide private areas. Protect those with a login.

Check what your site allows today

It is easy to get a rule slightly wrong, for example by adding a crawler to a group you did not intend, or leaving a typo in its name. A few free resources:

AI Crawler Checker: reads your live robots.txt and shows, crawler by crawler, whether it is allowed, blocked or partly blocked.
ai-crawlers-robots-txt on GitHub: an open list of 14 AI crawlers with their operator, purpose and official docs link, plus copy-paste templates (CC0).
Robots.txt Generator: builds a full robots.txt with per-crawler rules.

Disclosure: I build SEOpeck and maintain the GitHub list. Corrections and new crawlers are welcome as PRs.