DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Adding a Googlebot Block to robots.txt Can Open Your Whole Site to Googlebot

#seo #webdev #javascript #programming

robots.txt looks like the simplest configuration format in existence, which is exactly why the three rules that matter get skipped.

A working RFC 9309 evaluator — paste a file and a URL, and it tells you which line decided: https://dev48.infy.uk/solve/day66-robots-evaluator.html

1. One group applies. Never a merge. 

User-agent: *
Disallow: /admin/
Disallow: /private/

User-agent: Googlebot
Disallow: /private/
Enter fullscreen mode Exit fullscreen mode

Googlebot may now crawl /admin/.

A crawler picks the most specific group whose user-agent matches, and obeys only that group. Rules from * are not inherited. Somebody added a Googlebot block to tighten one path and silently removed every other restriction for that crawler.

It falls out of the code rather than needing a special case, if you score the wildcard as zero:

for (const agent of g.agents){
  if (agent === "*"){ if (!wildcard) wildcard = g; continue; }   // fallback only
  if (name.indexOf(agent) === 0 && (!best || agent.length > best.score))
    best = { group: g, score: agent.length };
}
return best ? best.group : wildcard;
Enter fullscreen mode Exit fullscreen mode

Note indexOf(agent) === 0 — prefix, not equality. Googlebot-Image matches the Googlebot group.

2. Longest match wins, not first 

Disallow: /search
Allow: /search/public
Enter fullscreen mode Exit fullscreen mode

Is /search/public/page allowed? Under RFC 9309 yes — both match, and the Allow is longer. Under the original 1994 convention it was first match, so the Disallow takes it. Plenty of parsers still do that, and on an overlapping file the two conventions give different answers on 4 of 7 paths.

On a tie, Allow wins. That asymmetry makes the file fail open.

3. The default is allow, in four different ways

An empty file, an empty Disallow:, no matching rule, and no matching group all mean allowed. A parser that returns "denied" for an unmatched path will quietly stop crawling an entire site.

return best ? best.type === "allow" : true;      // no rule matched -> ALLOWED
Enter fullscreen mode Exit fullscreen mode

The wildcards are real, and $ is not a regex

Escape every metacharacter first, then re-enable the one wildcard robots.txt actually has:

const body = p.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*");
Enter fullscreen mode Exit fullscreen mode

Do it the other way round and Disallow: /a.pdf also blocks /axpdf.

Two more, going opposite ways: paths are case-sensitive, user-agents and field names are not.

What it does not do

It is not access control — a disallowed URL is still served to anyone who asks. It does not remove pages from an index, and blocking a page prevents the crawler from ever seeing a noindex tag, which is the opposite of what people intend. And it is scoped to one origin: scheme, host and port.

48 reference cases, 0 failures — each one a case where a plausible implementation gives the opposite answer.

Part of a from-scratch series — one tool a day, all client-side: https://dev48.infy.uk/solvefromzero.php

Top comments (0)