WordPress robots.txt mistakes before launch

#wordpress #webdev #seo #developers

Part of the series: WordPress Pre-Launch Technical Checks

A WordPress site can look finished and still carry one of the most annoying launch problems: a robots.txt file that sends the wrong signals.

This is one of those technical details that often survives staging, migration, or last-minute changes. Everything looks fine on the surface, the homepage loads, the design is approved, the client is happy, and then... crawlability is not what you thought it was.

Before launching a WordPress site, robots.txt deserves a quick review.

What robots.txt actually does

robots.txt is a plain text file placed at the root of a site. Its purpose is to give crawl instructions to bots, especially around which paths should or should not be crawled.

It is not a magic privacy wall, and it does not guarantee deindexing by itself. But it does influence crawl behavior, which makes it part of any solid pre-launch technical review.

Why this matters before launch

During development, it is common to protect staging environments, block crawlers temporarily, or leave behind rules that made sense at one point but should not survive into production.

The problem is simple: if nobody checks robots.txt before delivery, those leftovers can go live with the site.

That can create issues such as:

important public content being harder to crawl than expected,
mixed indexing signals between WordPress settings and server-level files,
confusion after launch when a site does not behave as expected in search.

Common robots.txt mistakes I would check before launch

1. Temporary staging rules still in place

This is the classic one. A site was blocked during development, then migrated, cleaned, and published, but the robots.txt still contains directives that no longer belong there.

If the file still reflects a staging mindset, it needs review before the site goes live.

2. Rules added without checking the final indexing plan

Sometimes a file exists, but nobody has checked whether it actually matches the intended launch setup.

A production site should have a clear crawl and indexing strategy. If the WordPress settings say one thing and the robots.txt points in a different direction, that is messy technical delivery.

3. Assuming robots.txt controls everything

A lot of people treat robots.txt as if it completely controls whether a page appears in search. It does not.

If you want a clean launch, you need consistency between:

WordPress indexing settings,
robots.txt,
meta robots,
and the actual public URLs being served.

Looking at only one of those is how weird launch issues sneak through.

4. Never checking whether the file exists at all

Sometimes the issue is not a bad file. It is no file, an unexpected generated version, or a file that behaves differently than assumed.

Before launch, I would always confirm the final version that the public site is actually serving.

A simple pre-launch robots.txt review

If I were checking a WordPress site before delivery, I would review these points:

Does robots.txt exist and load correctly?
Does it reflect the intended production setup?
Are there any leftover temporary rules from staging or migration?
Do the crawl instructions align with the site's indexing settings?
Is the final public version of the site sending clear, non-conflicting signals?

This is not a long task, but it is exactly the kind of thing that gets skipped when a project is rushing toward delivery.

Why this check belongs in a repeatable process

The point is not to obsess over one file. The point is to avoid relying on memory.

Technical launch quality improves a lot when small checks become repeatable. Not glamorous, not sexy, just reliable.

That is especially true if you work with multiple client sites, handovers, migrations, or agency workflows where the final delivery stage can get rushed.