robots.txt and noindex mistakes that hide small WordPress sites

On small WordPress sites, "Google won't index my pages" often traces back to two files of intent that contradict each other: robots.txt and meta/X-Robots-Tag noindex. This is not a soft-404 triage and not another sitemap hygiene rewrite. It is about access rules and index signals — myths about blocking CSS/JS, real accidental Disallow lines, noindex left in the sitemap, and staging leftovers that survived launch.

What each control actually does

robots.txt tells compliant crawlers which URL paths they may fetch. A blocked URL can still appear in search (usually without a snippet) if Google found it another way. Blocking does not reliably remove a URL from the index.

noindex (meta robots or X-Robots-Tag) tells Google not to keep the URL in results — but Google must be able to fetch the HTML (or headers) to see that tag. If robots.txt blocks the URL, Google may never see your noindex.

Sitemap is a discovery wishlist. Putting a noindex URL in the sitemap is mixed messaging: "please crawl this" + "please don't index this."

Get those three straight before you rewrite plugins.

Myth: "You must Allow all CSS and JS or Google can't render"

Google has asked for years not to block CSS/JS needed for rendering. That advice is still sound. The myth is the panic over-correction: people paste giant Allow: lists, open /wp-includes/ wholesale when nothing was blocked, or blame every indexation lag on a missing Allow rule.

Reality on modern small WordPress sites:

• Default WordPress robots.txt usually only disallows /wp-admin/ (with Allow for admin-ajax.php) plus whatever you or a security plugin added.
• If Google can load your public CSS/JS from /wp-content/ and /wp-includes/ already, you do not need a novel Allow liturgy.
• Rendering problems that matter look like: blocked theme assets under a custom path, a WAF/Disallow: /*.css$ "security" rule, or a CDN path you accidentally disallowed.

Do this instead: Fetch https://yoursite.com/robots.txt. Search for Disallow lines that hit /wp-content/, /wp-includes/, theme folders, or *.css / *.js. Remove those. Confirm in GSC URL Inspection → "Page resources" / screenshot that CSS/JS load. Skip cargo-cult Allow blocks if nothing was blocked.

Real accidental Disallow (the ones that actually hurt)

I see these more often than CSS myths:

Pattern	What happens
Disallow: /	Whole site fetch-blocked for that user-agent
Disallow: /* or over-broad Disallow: /*?	Query URLs, sometimes more than you meant
Staging copy-paste: Disallow: / left after go-live	Production never gets crawled cleanly
Security plugin: block /author/, /feed/, search, and random asset paths	Mixed — some OK, some break render
Disallow: /wp-content/uploads/	PDFs/images Google needed stop being fetched
Manual "hide /blog/ while redesigning" forgotten	Money content stays blocked for months

WordPress Settings → Reading → "Discourage search engines from indexing this site" writes a sitewide discouragement (effectively treating the site as non-indexable via robots output / meta depending on setup). Freelancers flip it on for staging, migrate the database, and forget. Check it on every launch.

Also check hosting panel robots, CDN edge robots, and a second robots.txt served from a reverse proxy. What you edit in Rank Math is not always what curl -I https://yoursite.com/robots.txt returns.

noindex mistakes on small WP installs

Common, boring, expensive:

• Category / tag / author archives set to noindex while you still want a hub category indexed
• Search (/?s=), thank-you, cart, account — correctly noindex, but also left in the XML sitemap
• "Noindex this post" used as a soft delete, then the URL stays published and linked
• Sitewide noindex from a maintenance / coming-soon plugin that was "turned off" in the UI but still injects headers
• X-Robots-Tag: noindex on the whole host from staging server config (Apache/Nginx) after DNS cutover

Rule: noindex utility and thin archives on purpose. Do not noindex the five pages that pay rent. Do not leave noindex on URLs you still list in the sitemap and promote on the blog.

Conflicting noindex + sitemap

If a URL is noindex, remove it from the sitemap (or fix the plugin so those post types/archives are excluded). If you want it indexed, remove noindex and make sure it is fetchable and linked.

GSC often shows Excluded by 'noindex' tag next to URLs you are still "submitting." That is not a Google bug. Your stack is arguing with itself. Same class of confusion as duplicate URL variants — settle the intended URL with clear canonical tags once fetch + index signals agree.

Staging leftovers (the migration tax)

Staging is where robots/noindex debt is born:

1. Discourage search engines on
2. Disallow: / in robots
3. Basic auth (password) — crawlers get 401, not your carefully written noindex
4. noindex,nofollow in the theme header "just in case"
5. Sitemap still submitted for staging.example.com in a leftover GSC property

After go-live checklist (15 minutes):

• [ ] Reading setting: discouragement off
• [ ] Live robots.txt: no sitewide Disallow; keep /wp-admin/ pattern; Sitemap line points at production HTTPS
• [ ] View-source on Home + one money URL: no noindex
• [ ] Response headers: no sitewide X-Robots-Tag: noindex
• [ ] GSC: production property verified; staging sitemap submissions removed or property ignored
• [ ] Internal links and shares use final production URLs (campaign links can use a URL shortener externally — do not leave staging hosts in on-site hrefs)

Password protection blocks fetch; robots Disallow blocks fetch; noindex requires fetch. Pick one intentional strategy for staging — preferably auth or IP allowlist — and strip all of it on production.

A sane default for ~50–200 URL WordPress sites

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

Add Disallow lines only when you can name the path and the reason (e.g. a private folder). Prefer noindex for thank-you pages and thin archives you still need humans to open. Prefer removal + 404/410 or login for content that should not exist publicly.

Do not use robots.txt as a security boundary. Do not use noindex as a substitute for authentication.

25-minute audit habit

1. Open live robots.txt; list every Disallow; delete accidents.
2. Confirm CSS/JS for the theme are not disallowed.
3. View-source Home + top 5 money URLs for robots meta.
4. Spot-check headers for X-Robots-Tag.
5. In the SEO plugin: which archives are noindex? Intentional?
6. Diff: noindex URLs vs sitemap children — remove conflicts.
7. Confirm Reading → discourage is off.
8. In GSC Coverage/Pages: sample "Blocked by robots.txt" and "Excluded by noindex" — fix causes, skip mass re-request as the only move.

Closing

Small sites rarely need clever robots gymnastics. They need fetchable HTML, honest noindex on utility URLs only, no staging leftovers, and no sitemap entries that contradict noindex. Fix the accidental Disallow before you rewrite content. Fix conflicting signals before you blame crawl budget.

More practical technical notes: blog.

— Riichi Mirza