Part of the series: WordPress Pre-Launch Technical Checks
A WordPress site can look finished and still carry one of the most annoying launch problems: a robots.txt file sending the wrong signals.
Everything looks fine on the surface. The homepage loads, the design is approved, the client is happy… and then crawlability is not what you expected.
This is one of those details that easily survives staging, migrations, or last-minute changes. That’s why it’s worth a quick check before launch.
What robots.txt actually does
robots.txt is a simple file placed at the root of a site. It gives crawl instructions to bots, mainly about which paths should or should not be crawled.
It does not control indexing on its own, but it does influence how search engines explore the site. That makes it part of any solid pre-launch review.
Why this matters before launch
During development, it’s common to block crawlers, protect staging environments, or add temporary rules.
The problem is when those rules quietly make it into production because nobody checked the final file.
When that happens, you can end up with:
- important pages harder to crawl than expected,
- conflicting signals between different layers,
- confusion after launch when the site doesn’t behave as expected in search.
What usually goes wrong
Staging rules still in place
The classic case. The site was blocked during development, then published, but the robots.txt still reflects that staging setup.
Rules that don’t match the final setup
Sometimes the file exists, but nobody checked whether it actually aligns with the intended launch configuration.
If WordPress settings say one thing and robots.txt says another, things get messy.
Assuming robots.txt controls everything
robots.txt is often misunderstood. It does not fully control whether a page appears in search.
What matters is consistency between:
- WordPress indexing settings,
-
robots.txt, - meta robots,
- and the actual URLs being served.
Looking at only one of these is how small issues slip through.
Not checking the real file
Sometimes the issue is not a wrong file, but not knowing which version is actually being served.
Before launch, it’s worth checking the public URL directly.
A quick check before launch
At this stage, I usually just open the file and compare it with the intended setup.
If everything lines up, you’re good. If not, it’s usually an easy fix.
Why this belongs in a repeatable process
This is not about one file. It’s about not relying on memory.
When these small checks become part of a repeatable process, launches get cleaner and more predictable.
Especially in projects with migrations, multiple environments, or tight delivery timelines.
Where PreFlight fits in
PreFlight focuses on reviewing these kinds of details before a WordPress site is delivered or published.
If you want to check crawlability signals before launch, you can take a look here: https://preflightstandard.com/checks/robots-txt/
And if you want to run a broader launch-readiness review: https://preflightstandard.com/
Final thought
A site can look finished and still not be technically clean.
robots.txt is one of those small details that feels minor until it causes confusion. That’s exactly why it’s worth checking before launch.
Top comments (0)