DEV Community

seotto.com
seotto.com

Posted on

How to check if robots.txt is blocking Google (and fix it without breaking anything)

#webdev #tutorial #beginners #seo

robots.txt is a plain text file with a lot of power. One line in it can stop Google from crawling your entire site, and because every page still opens normally in a browser, nobody notices until traffic falls. Here is how to check whether yours is blocking Google, and how to read the rules correctly.

Step 1: Find the file

robots.txt always lives at the root of the host:

https://example.com/robots.txt
Enter fullscreen mode Exit fullscreen mode

Note that each host has its own: www.example.com, example.com, blog.example.com and the http:// versions can all serve different files. Check the one your canonical URLs use.

From a terminal you can also see the status code:

curl -sI https://example.com/robots.txt | head -1
curl -s https://example.com/robots.txt
Enter fullscreen mode Exit fullscreen mode

The status code matters more than people expect:

  • 200: Google follows the rules in the file.
  • 404 / 410: Google treats it as "no rules", so everything may be crawled.
  • 5xx or timeout: Google may stop crawling the site for a while, because it can't tell what is allowed. A broken robots.txt is worse than a missing one.

Step 2: Understand the groups

Rules are grouped by User-agent. A crawler uses the most specific group that matches it and ignores the others:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

Here, Bing and others can crawl everything except /admin/, but Googlebot is blocked from the whole site, because the Googlebot group wins and the * group no longer applies to it. This catches people out when an old rule for one bot was left in the file.

Step 3: Read Allow and Disallow correctly

For Google, when several rules match a URL, the longest matching rule wins. If an Allow and a Disallow are the same length, Allow wins.

User-agent: *
Disallow: /shop/
Allow: /shop/products/
Enter fullscreen mode Exit fullscreen mode

/shop/cart is blocked, /shop/products/red-shoes is allowed, because the Allow rule is longer.

Two wildcards are supported:

  • * matches any sequence of characters: Disallow: /*?sort= blocks sorted listing URLs.
  • $ anchors the end: Disallow: /*.pdf$ blocks URLs ending in .pdf.

And the most important line of all:

Disallow: /
Enter fullscreen mode Exit fullscreen mode

means "nothing on this host may be crawled". An empty Disallow: (nothing after the colon) means the opposite: everything is allowed. One character separates "fully open" from "fully closed".

Step 4: Test a specific URL

Search Console has a robots.txt report (under Settings) that shows which robots.txt files Google found for your property, when it last fetched them and any parsing problems. For a single page, the URL Inspection tool shows "Crawl allowed?" and says "No: blocked by robots.txt" when a rule matches.

If you prefer to test locally, Python ships a basic parser:

from urllib import robotparser

rp = robotparser.RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("Googlebot", "https://example.com/shop/products/red-shoes"))
Enter fullscreen mode Exit fullscreen mode

Be aware that Python's parser uses first-match order rather than Google's longest-match rule, so for tricky Allow/Disallow combinations, trust Google's own report or the open-source Google robots.txt parser.

Step 5: Know what robots.txt does not do

Blocking a URL in robots.txt stops crawling, not indexing. If other pages link to a blocked URL, Google can still list it, usually without a description. If you want a page out of search results, use a noindex meta tag or X-Robots-Tag header and leave the page crawlable so Google can see it.

Also, robots.txt is public and advisory. It is not a security tool. Anything private needs authentication.

The usual ways a site gets blocked by accident

  • Staging file shipped to production. The staging server had Disallow: / to keep it out of Google, and the deploy copied it over.
  • CMS "discourage search engines" switches. Some platforms change the virtual robots.txt when you tick that box.
  • A blanket rule written for one section. Disallow: /p meant for /print/ also blocks /products/ and /pricing, because rules are prefix matches.
  • Blocking CSS and JavaScript. Google renders pages; if /assets/ is blocked, it may see a broken layout.
  • The file starts returning 5xx after a server or CDN change.

Fixing it safely

  1. Edit the rule, and keep the change as small as possible.
  2. Check the file returns 200 and the new content.
  3. In Search Console, open the robots.txt report and request a recrawl of the file.
  4. Use URL Inspection on a few key pages to confirm "Crawl allowed? Yes".

Keep an eye on it

robots.txt rarely breaks on its own; it breaks when someone deploys. That's why it is worth checking after every release, or letting a tool do it for you. A monitor such as SEOtto reads robots.txt at the start of every crawl, keeps each version line by line, and alerts you when it starts blocking the site or Googlebot. Either way, five minutes reading the file today can save you weeks of lost traffic later.

Top comments (0)