DEV Community: Justice Manyika

336 mirrored pages appeared on my store overnight - diagnosis and rollback

Justice Manyika — Tue, 10 Nov 2026 22:05:48 +0000

I opened my WooCommerce store one morning and it was in Spanish. All of it. Three hundred and thirty-six translated product pages that I had never written, indexed under my domain, with my content and someone else's links.

This is what that looked like under the hood and how I rolled it back.

The symptom

Search Console started reporting a spike in indexed URLs for the domain. Not a few. Hundreds of new paths like:

/es/producto/some-product-name/

The pages were real. They returned 200. They carried my product titles machine-translated, my images hotlinked, and at the bottom, links to a domain I had never heard of.

How a site gets cloned

Three mechanisms cover almost every case, and they look identical from the outside:

An affiliate or content plugin with a translation feature. It registers a rewrite that serves translated copies of your own pages, usually under a language prefix, and appends its own tracking.
A scraper with a fast index. A third party mirrors your pages on their server. Your HTML is not on your site at all; what you see in Search Console is their domain, not yours. Check the URL prefix before panicking.
A compromised theme or plugin file. Rare, but it produces the same rewrite rules.

Mine was the first: an installed plugin quietly added a multilingual rewrite and my permalinks were being regenerated with a prefix.

The 60-second diagnosis

Ask the site itself which rules are answering those requests:

wp rewrite list --format=csv --path=/var/www/html

And ask WordPress whether the prefix corresponds to a real post type, taxonomy or language:

wp post list --post_type=page --fields=ID,post_name --path=/var/www/html | grep -i "es$"

If the URLs return 200 but there is no corresponding content, the route is being generated, not stored. That tells you to look at plugins and rewrite rules rather than at your database.

A faster check that requires no shell access: request one of the mirrored URLs with a cache-busting query, and look at the response headers.

curl -sI "https://example.com/es/producto/some-product/" | grep -i -E "x-|server|location"

A plugin-generated route usually still emits your caching headers, because the response is generated by your own stack. A scraper's copy does not.

Rolling it back

The order matters, because each step changes what the next one sees:

Disable the plugin generating the rewrite. Do this first. Removing URLs while the route still exists means they reappear on the next request.
Flush rewrite rules. In WP-CLI, wp rewrite flush --hard regenerates permalinks without the prefix.
Purge the page cache. Otherwise your cache keeps serving the translated HTML from stale entries and you will think the rollback failed.
Verify with curl, not a browser. Browsers hold their own cache. Request ten of the mirrored URLs and confirm 404 or 410.
Ask Google to remove them. In Search Console, the Removals tool takes a prefix and drops those URLs from results temporarily while the 404s propagate.

for u in $(cat mirrored_urls.txt); do
  printf "%s " "$(curl -s -o /dev/null -w '%{http_code}' -A 'Mozilla/5.0' "$u")"
  echo "$u"
done

You want a clean column of 404s or 410s with no 200s left.

What I would check weekly

A clone does not announce itself. What surfaces it is a change in indexed URL count for a prefix you do not recognise, so watch that number instead of your total. In my case the count went from my real catalogue to roughly double it within a week.

Also check the referrer and outbound link report. Mirrors almost always carry a footer link to the operator's domain, and that is the fastest fingerprint available.

The uncomfortable part

Nothing was hacked in the sense of a stolen password. A feature I installed did exactly what it was configured to do, and the configuration was mine, months earlier, forgotten. The rollback took twenty minutes. Finding it took a week, mostly because I kept checking whether the site was broken for visitors, and it never was.

Originally published on my diary: The Night the Site Spoke Spanish: Waking Up to 336 Mirrors I Never Built

The dead sitemap: how one July click kept a site invisible for three months

Justice Manyika — Tue, 10 Nov 2026 22:00:21 +0000

Search Console accepted my sitemap, listed it as Success, and told me nothing for three months. That combination is possible, and it cost me every new post I published from July to October.

What "Success" actually means

A submitted sitemap shows Success when Google has fetched the file. It says nothing about whether the URLs inside it were queued, crawled or indexed. Mine read Success with 0 discovered pages for weeks. I looked at the word Success and moved on.

The number that mattered was the discovered count next to it. Zero meant the file was fetched and then produced nothing usable.

The original mistake

In July I did something that seemed harmless: I edited the sitemap through a plugin's URL field and saved a value with a stray whitespace character at the end. The sitemap endpoint answered with a 200 and a body that was not valid XML.

Diagnosis in one command:

curl -s https://example.com/sitemap.xml | head -3

Valid output begins with <?xml. Mine began with a blank line before the declaration, which is fatal: XML declarations must be the very first bytes in the document. Browsers do not care. Parsers do.

Why the UI cannot tell you this

Search Console has no concept of "your sitemap is well-formed but semantically wrong". It fetched something, got a 200, and marked the row Success. The pages inside were never read because the document never parsed.

The read that finally surfaced it was the per-URL inspection: every new post showed "URL is unknown to Google", and the discovery column said sitemap. Unknown, from a sitemap listed as Success.

The fix, and the verification that matters

Remove whatever is generating the file and serve a real one, then verify the bytes rather than the dashboard:

curl -s https://example.com/sitemap.xml \
  | python3 -c "import sys,xml.dom.minidom as m; m.parseString(sys.stdin.read()); print('valid xml')"

Then count the URLs and compare against your post count:

curl -s https://example.com/sitemap.xml | grep -o '<loc>' | wc -l

If those two numbers disagree, the sitemap is not doing its job regardless of what the dashboard says.

What changed afterwards

After serving a valid file, the sitemap report moved from 0 discovered to 62 discovered pages within a day, and new posts started appearing in the per-URL inspection as "Discovered" rather than unknown.

It did not fix indexing on its own. Crawl stats still showed ~1 request per day, and pages sat in the queue. But it moved the problem from "Google cannot read my list" to "Google has my list and is choosing slowly", which is a solvable problem. Before that, nothing downstream could work.

The lesson I keep relearning

Dashboards report their own success criteria, not yours. Success on a sitemap row means "fetched". Success on a page row means "indexed". They are different claims and they are one click apart.

Verify the artefact, not the status. Especially the boring ones.

Originally published on my diary: The Dead Sitemap: How One July Click Kept My Site Invisible for Months

Twelve of Sixty-One - the three measurements that showed why my posts were invisible

Justice Manyika — Tue, 10 Nov 2026 21:47:13 +0000

I published 61 posts before I checked how many of them Google actually showed to anyone. The answer was twelve. Not because the writing was bad, but because the site had one inbound link and no path between its own pages.

This is how I found that out, and the three measurements that made the problem obvious.

1. Count what search shows, not what you published

The only number that matters is the one a stranger can see. Search your own domain in Google with the site: operator and count the results, including page two:

site:example.com

Then compare it to your post count. Mine was 12 of 61. Search Console's Pages report said 7 indexed, which was even worse, and it was three weeks stale. Two different numbers, both far below 61, and neither was in my dashboard where I could trip over it.

Measure it the same way every week. Same query, same browser, count the posts only, not the homepage or the About page. A number you take by hand once a week beats a report you read once a month.

2. Read crawl stats before writing another post

Search Console, then Settings, then Crawl stats. Look at two lines: total requests, and the purpose breakdown.

Mine read 87 requests over 90 days, about one per day, and 100% Refresh with zero Discovery. That is the whole diagnosis in one screen. Google was coming back to check pages it already knew about, and never once coming to look for new ones.

If your crawl purpose is all refresh, publishing more does not help. The crawler is not reading.

3. Check whether anything links to the page

For any URL that is not indexed, inspect it and look at "Referring page". Mine said "None detected" for the page I cared about. Nothing on the site linked to it and nothing outside did either.

A page with no inbound link is a page with no path to it. Internal links count. If you have 60 posts and no links between them, you have 60 orphan pages sharing one homepage.

What I changed, in order

Internal links from indexed pages to unindexed ones. I added a block to every post Google already had, linking the three newest entries. This costs nothing and it is the only lever entirely within your control.

One specific detail per page that cannot be copied. For me it is numbers: the crawl counts, the redirect counts, the dates. Anyone can write advice. Almost nobody writes down that their validation sat at zero processed URLs for ten days while the interface said Started.

Fewer index requests. I had requested around 60 URLs over three weeks with no movement. Requesting is not a strategy; it is a queue position on a crawl budget you have not earned.

External links, which I had been avoiding. One inbound domain is not a distribution problem you can write your way out of. This is the uncomfortable part and it stays uncomfortable.

The honest summary

Your site can be technically perfect and still invisible. Mine was: 200s everywhere, clean canonicals, a fresh sitemap, sub-400ms responses. None of that makes Google curious. Curiosity comes from other sites pointing at you, and from pages that are worth the crawl they cost.

Fix the mechanics first, because they are cheap. Then accept that the rest is reputation, and reputation is not a settings page.

Originally published on my diary: Twelve of Sixty-One: The Count I Kept Avoiding

Fixing "Redirect error" in Search Console: Blogger mobile 302s behind Cloudflare

Justice Manyika — Tue, 10 Nov 2026 21:41:02 +0000

My site had 18 URLs flagged as "Redirect error" in Search Console, and the validation sat on "Started" for weeks without moving. The cause was not a broken redirect I had created on purpose. It was Blogger's automatic mobile redirect meeting Google's smartphone crawler.

What the label means

"Redirect error" does not mean "this URL redirects". It means Google's last visit to that URL ended somewhere unexpected, or followed a chain it did not want to follow. The report shows the date of that visit, which matters: if you fixed the issue afterwards, the row is a historical record, not a current fault.

That distinction cost me a week. I was validating a fix that had never been crawled, and the validation stayed pending because there was nothing new to read.

How to reproduce it in a terminal

Send a mobile user agent to the URL and look at the status line and Location header:

curl -sI -A "Mozilla/5.0 (Linux; Android 13) Chrome/125 Mobile" \
  https://example.com/some-post.html

On my Blogger site this returned:

HTTP/2 301
location: https://www.example.com/some-post.html

Follow that and the second hop was:

HTTP/2 302
location: /some-post.html?m=1

Two things were wrong at once. The apex to www redirect happened on every request because the origin was reached over the naked domain, and Blogger then answered with a 302 to its ?m=1 mobile variant. A crawler sees a redirect chain ending on a URL it did not ask for, and it records friction.

Why this matters more than it looks

The redirect itself is not fatal. What matters is the cost. Every crawler visit spent on a three-hop chain is a visit not spent discovering new content. On a small site with roughly one crawl request per day, friction is not an annoyance, it is the whole budget.

The fix without touching the origin

Blogger's current UI no longer exposes a mobile template setting, and the origin will keep answering with ?m=1 for mobile user agents. If you run Cloudflare in front, you can normalise the request before it reaches origin:

Redirect http:// and naked-domain requests to https://www. in a single hop.
Rewrite crawler user agents to the desktop agent before origin fetch, so origin answers 200 with no mobile redirect.
Catch any residual 3xx whose Location contains m=1 and turn it into a 301 to the canonical.

const CRAWLERS = /Googlebot|AdsBot-Google|Mediapartners-Google|Google-InspectionTool|bingbot|PetalBot|Applebot/i;
const DESKTOP = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125 Safari/537.36";

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (url.protocol === "http:" || url.hostname === "example.com") {
      url.protocol = "https:";
      url.hostname = "www.example.com";
      return Response.redirect(url.toString(), 301);
    }
    const ua = request.headers.get("user-agent") || "";
    const headers = new Headers(request.headers);
    if (CRAWLERS.test(ua)) headers.set("user-agent", DESKTOP);
    const response = await fetch(new Request(request, { headers }));
    const location = response.headers.get("location");
    if (location && location.includes("m=1")) {
      return new Response(null, { status: 301, headers: { location: url.toString() } });
    }
    return response;
  }
};

Verify after deploying

Run the same curl against four user agents: smartphone Googlebot, AdsBot-Google, Mediapartners-Google and bingbot. All four should return 200 with no Location header. Then confirm a real mobile browser still gets the normal behaviour, because you do not want to break the site for humans to please a crawler.

The numbers moved for me: redirect errors fell from 18 to 15 in the next Search Console refresh, and indexed pages went from 7 to 14 over the following weeks. The redirect was not the only reason, but it was the one defect entirely in my control.

Originally published on my diary: Google Search Console Says Redirect Error. Here Is What It Actually Means.

What "Crawled, currently not indexed" actually means, and what I changed

Justice Manyika — Tue, 10 Nov 2026 21:39:29 +0000

I spent three weeks submitting URLs to Google Search Console and watching my indexed count go from twelve posts to eleven. That was the moment I stopped treating submission as a strategy and started reading the numbers properly.

This is what the labels actually mean, and what I did about the one page that mattered.

The four "not indexed" reasons are not the same thing

Search Console lists rejection reasons side by side, which makes them look interchangeable. They are not.

On my site, 56 pages were outside the index, broken down as:

Discovered, currently not indexed: 36. Google knows the URL exists and has not decided to spend a crawl on it.
Redirect error: 18. Every one of these came from mobile Googlebot hitting a 302 to ?m=1 before my Cloudflare fix.
Crawled, currently not indexed: 1. Google fetched this page, read it, and declined to include it.
Alternative page with proper canonical tag: 1. Correct behaviour, nothing to fix.

Three of those are access questions. Exactly one is a judgment about the content.

"Crawled, currently not indexed" is a quality verdict

The label means the crawler fetched the page successfully and then decided it did not add enough to what the index already holds. It is not a penalty. The word "currently" is doing real work: the decision is revisited.

My one page with this label was /p/start-here.html. The URL inspection panel showed:

Last crawl: 28 October, 06:24
Referring page: None detected
Discovery: sitemap (with a "temporary processing error" line that turned out to be a per-URL artefact, not a sitemap failure)

Nothing on my site linked to that page and nothing outside it did either. A short "start here" page with no inbound links, no outbound links and generic advice was competing against a million pages saying the same thing more thoroughly. It lost.

The number that explains the whole stall

Open Search Console, then Settings, then Crawl stats. Two lines matter:

Requests per day. Mine: roughly 87 over 90 days, about one a day.
Purpose. Mine: 100% Refresh. Zero Discovery.

Refresh means Google came back to check something it already knew about. Discovery means it came looking for something new. In three months it never once came looking.

That single fact explains why 36 pages sat in "Discovered" and why index requests changed nothing. A crawler that only refreshes does not read new URLs, no matter what you submit.

What I actually changed

1. Internal links from pages Google already has. I added a block linking the three newest posts to all eleven posts that were already indexed. If nothing links to a page, the crawler has no path to it.

2. Made the dismissed page specific. It now names what the site is, when it started, how many entries exist, what has failed so far, and the numbers. It links out to the entries that explain each failure. Specificity is the part a stranger cannot copy.

3. Stopped the submission grind. Around sixty URLs requested over three weeks produced no measurable movement. I paused the daily batch rather than keep spending twenty-five minutes a day on something inert.

4. Fixed the actual redirect bug. The mobile redirect was turning crawler hits into 302s. That was a real defect and it is worth checking before blaming content.

The honest conclusion

A page can be perfectly reachable and still be rejected. When that happens the fix is in the page, not in the plumbing. And if crawl stats show zero discovery requests, the problem is bigger than any single page: the domain has not earned the crawler's curiosity yet, and that comes from other sites pointing at it.

Originally published on my diary: Crawled, Currently Not Indexed: What It Means and What I Actually Did