This trips up a lot of people the first time they see it happen: you add Disallow: /private-page/ to robots.txt, expecting the page to disappear from search results, and it shows up anyway, sometimes with a "no information is available for this page" snippet instead of a real description. That's not a bug. It's exactly how robots.txt is supposed to work, and the confusion comes from a mismatch between what people assume the file controls and what it actually controls.
What robots.txt actually does
The Robots Exclusion Protocol tells well-behaved crawlers which URLs they're allowed to fetch. That's it. It's a crawling instruction, not an indexing instruction. If a search engine already knows a URL exists, for example because another page links to it, it can still index that URL using the link text and any other signals it has, even if it's never allowed to actually crawl the page's content directly.
Why this distinction matters
Crawling and indexing are two separate steps. Crawling is the search engine fetching the page's content. Indexing is the search engine deciding to store and potentially rank that URL. Blocking crawling doesn't reliably block indexing, because indexing can happen from external signals alone, without the crawler ever reading a single word of the page itself. Google's own documentation on robots.txt is direct about this: robots.txt is not a reliable mechanism for keeping a page out of search results.
What actually prevents indexing
If the goal is genuinely to keep a page out of search results, the correct tool is a noindex meta tag or X-Robots-Tag HTTP header on the page itself, not a robots.txt disallow rule. Critically, the page has to be crawlable for the search engine to see the noindex directive in the first place. Blocking it in robots.txt while also adding noindex is a common mistake that backfires, because the crawler never gets far enough to read the noindex instruction, and the page can still end up indexed off external signals.
When robots.txt is actually the right tool
Robots.txt earns its keep for a narrower set of problems: keeping crawlers out of infinite parameter combinations that waste crawl budget, blocking internal search result pages from being crawled, or steering crawlers away from staging environments and admin paths that shouldn't be fetched at all, indexed or not. A robots.txt generator handles the syntax for you, since a single misplaced wildcard or missing trailing slash can accidentally block far more than intended, sometimes an entire section of a site by mistake.
A common failure pattern worth checking for
Sites migrating platforms sometimes carry over a robots.txt file written for the old CMS's URL structure, which can end up blocking the wrong paths entirely on the new site. It's worth testing your current robots.txt against your actual sitemap after any platform migration, not just when you first set it up, since the two files tend to drift out of sync as a site evolves.
Testing before you ship changes
Before deploying a new robots.txt file, test it against the specific URLs you're trying to allow and block. A rule that looks correct in isolation can interact unexpectedly with a broader wildcard rule earlier in the file, since crawlers generally apply the most specific matching rule, not just the first one they encounter, which means order and specificity both matter more than most people assume.
Crawl budget is the underrated reason to care
Even on pages you don't mind being indexed, an unnecessarily permissive robots.txt can waste crawl budget on low-value URLs, like faceted navigation combinations or session-specific parameters, leaving less budget for the crawler to revisit your genuinely important pages. Google's own explainer on managing crawl budget is aimed at large sites specifically, since that's where the constraint is real; a small site a crawler can fully revisit in a single pass doesn't need to worry about it nearly as much.
Where this fits with structured data and indexing
None of this replaces the other signals search engines use to decide how to present an indexed page. Getting crawling and indexing right with robots.txt and noindex is table stakes; layering schema markup on top of pages you do want indexed is what makes them eligible for the richer result types once they're actually in the index.
robots.txt and subdomains are separate files entirely
A robots.txt file only applies to the host it's served from. A rule set on example.com/robots.txt says nothing about blog.example.com or shop.example.com, each of which needs its own robots.txt file if you want crawling rules applied there too. This trips people up on sites that grew organically across subdomains over time, where someone assumes a single robots.txt file at the root domain covers everything, only to find a subdomain has been crawled with no restrictions at all because nobody ever added a file there.
Wildcard and pattern matching gotchas
The * wildcard and $ end-of-URL anchor in robots.txt rules are supported by major crawlers but not part of the original protocol specification, which means behavior can vary slightly between different bots if you're trying to write rules that apply broadly. For anything beyond a simple flat disallow list, testing the specific pattern against real example URLs before deploying is worth the extra few minutes, since a pattern that looks right can match more, or less, than intended.
Why staging environments need extra care
A robots.txt disallow rule on a staging or development environment is a reasonable first layer, but it shouldn't be the only protection if that environment isn't meant to be public at all. Since robots.txt is a voluntary convention that well-behaved crawlers respect, and not an access control mechanism, anything genuinely sensitive on a staging site needs actual authentication, not just a polite request for crawlers to stay away.
The order rules are declared in the file rarely matters, but grouping does
Within a single User-agent block, rules are generally evaluated by specificity rather than by the order they're written, but mixing multiple User-agent blocks incorrectly, for example writing rules for a specific bot after a catch-all * block without clearly separating them, is a common source of confusion when debugging why a particular crawler seems to be ignoring a rule that looks correct on its own. Keeping each bot's rule set in its own clearly separated block, even when the rules overlap with the catch-all block, makes the file easier to reason about later.
Keep a changelog for a file this consequential
Because a single bad line in robots.txt can accidentally deindex an entire site if a wildcard rule is too broad, it's worth treating changes to this file with the same care as a production configuration change, including a quick note of what changed and why, even if it's just a comment in the file itself or a line in a deployment log. The blast radius of a robots.txt mistake is large enough that a thirty-second note now can save hours of confused debugging later.
A quick mental model to keep straight
If you remember nothing else: robots.txt is a request about crawling, noindex is an instruction about indexing, and the two only work together correctly if the page stays crawlable enough for the noindex instruction to actually be read. Keeping that one distinction straight resolves most of the confusion that sends people down the wrong path when a page they thought was hidden turns up in search anyway.
The short version
Robots.txt controls crawling, not indexing. If you want a page genuinely out of search results, use noindex and make sure the page stays crawlable enough for that directive to be seen. Use the free robots.txt generator by EvvyTools to get the syntax right, and check EvvyTools' homepage for the rest of the technical SEO toolkit that pairs with it.
Top comments (0)