I expected 23 of my 119 sites to be underperformers. I did not expect Google to have zero of their real pages in the index.
I sampled 15 URLs per property through the Search Console URL Inspection API and grouped the results. On 91 properties the sample came back with apex URLs, and 713 of 1,346 sampled URLs were Submitted and indexed: a 53.0% hit rate. On 23 properties the sample returned no apex URLs at all. Every URL Google knew about for those domains, 2,104 of them, sat on a subdomain.
Those subdomains were mine. I just never meant to publish them.
The measurement
The check is small enough to run in a loop. For each property, ask the inspection API for a sample of URLs and bucket each result by whether its host matches the property:
import json, urllib.request
def inspect_sample(domain, token, limit=15):
url = ("https://searchconsole.googleapis.com/v1/urlInspection/index:inspect")
body = json.dumps({"inspectionUrl": f"https://{domain}/",
"siteUrl": f"sc-domain:{domain}"}).encode()
req = urllib.request.Request(url, data=body, headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=30) as r:
return json.load(r)
def bucket(domain, rows):
same = [r for r in rows if r["url"].split("/")[2] == domain]
foreign = [r for r in rows if r["url"].split("/")[2] != domain]
return {"same_host": len(same), "foreign_host": len(foreign),
"indexed": sum(1 for r in same if r["verdict"] == "PASS")}
The split is blunt: 91 properties had a mix leaning toward apex URLs. Twenty-three had same_host: 0 and a foreign_host count in the dozens or low hundreds. Two more were lopsided rather than empty, with 62 apex URLs against 250 subdomain URLs each.
That last group is the tell. A property with 62 apex URLs and 250 subdomain URLs is not a property with a crawling problem. It is a property where something else is competing for the same index space, and winning.
The pages that lost that competition are the ones I actually maintain. These are not thin domains with nothing to index: they carry the same kind of work as the properties that sampled cleanly. On a healthy property that work does show up in the sample. predictionguidecn.com returned 183 apex URLs with 11 of its 15 sampled pages Submitted and indexed; eventmarketcn.com returned 145 apex URLs with 10 of 15 indexed. Content of that shape includes things like the encyclopedia entry on how liquidity is priced in prediction markets, the comparison of how Polymarket and Kalshi differ on fees and settlement, and a year of crypto market trend archives.
On the 23 properties, the same sampling loop came back empty. Same content pipeline, same templates, no apex URLs in the index at all.
The subdomains were alive, and they claimed themselves
Before assuming a DNS wildcard was pointing at a parking page, I fetched a few. 21 of the 23 responded 200. All 21 declared index, follow in their robots meta. And all 21 pointed their canonical at themselves:
https://a7a5.altcoinfundingdesk.com/
status : 200
canonical : https://a7a5.altcoinfundingdesk.com/
robots : index,follow,max-image-preview:large
title : A7A5 资金费率事件怎么复盘？
https://altcoinfundingdesk.com/
status : 200
canonical : https://altcoinfundingdesk.com/
robots : index
Both are correct by their own rules. Neither knows the other exists. That is the whole failure: two URLs, same publishing system, two independent canonical claims, no cross-reference. Google has to pick one, and on 23 properties it picked the subdomain.
Twenty-one out of twenty-one is not a random sample. When every single host self-canonicalizes, the canonical tag is not expressing an editorial decision. It is expressing whatever request.headers.host happened to be.
Why Google went looking there
A canonical tag does not create discovery. Something has to link to those subdomains first, or submit them.
The apex sitemap.xml was doing it.
I fetched the sitemap from eight of the affected domains and counted <loc> entries of each shape. 1,926 entries in total. Every one pointed at a subdomain sitemap. Zero pointed at a page on the domain that served the file.
cryptochartshub.com loc=256 subdomain=256 apex=0
altcoinfundingdesk.com loc=118 subdomain=118 apex=0
dexguidehub.com loc=278 subdomain=278 apex=0
tokenmarginlab.com loc=256 subdomain=256 apex=0
perpfundingwatch.com loc=256 subdomain=256 apex=0
coinwikicenter.com loc=256 subdomain=256 apex=0
web3academyhub.com loc=256 subdomain=256 apex=0
exchangeguidehub.com loc=250 subdomain=250 apex=0
--------------------------------------------------------------
total loc=1926 subdomain=1926 apex=0
100.0% subdomain, 0.0% apex. The file named sitemap.xml on cryptochartshub.com was a sitemap index listing 256 per-token microsites, and not one entry for cryptochartshub.com itself.
So the sequence was not mysterious. The sitemap told Google "here are 256 sites", Google crawled them, each one said index, follow and claimed itself as canonical, and the apex pages were never submitted at all. No contradiction existed for Google to resolve, because nothing ever told it the apex mattered.
One config value doing two jobs
The generator reads a content registry: one record per token, one per site. The registry is the source of truth for what to build. Somewhere along the way it also became the source of truth for what to submit.
The sitemap builder walked the registry and emitted every host it found:
def build_sitemap_index(registry):
# one loc per host the content registry mentions
locs = [f"https://{entry['host']}/sitemap.xml" for entry in registry]
return render_index(locs)
And the canonical helper did the same thing one layer down, reading the host off the request instead of off the config:
canonical = f"https://{request.headers['host']}{request.path}"
Two lines, same assumption: that "the host this code is running on" and "the host this content belongs to" are the same value. They are the same value in production with one deployment and one domain. They stop being the same value the moment a registry has more than one host in it, which is exactly the situation the registry exists to describe.
The fix is not a canonical tag on the subdomains. It is making the canonical host an explicit configuration value and deriving both the sitemap and the canonical tag from that value alone:
PRIMARY_HOST = config["primary_host"] # e.g. "cryptochartshub.com"
def canonical_for(path: str) -> str:
return f"https://{PRIMARY_HOST}{path}"
def build_sitemap_index(site):
# only the site's own sitemaps, never the registry's other hosts
return render_index([f"https://{PRIMARY_HOST}/sitemap-{n}.xml"
for n in site.sitemap_shards])
def guard_host(request):
if request.headers["host"] != PRIMARY_HOST:
return redirect(f"https://{PRIMARY_HOST}{request.path}", status=301)
Three changes, and they have to ship together. Fixing the sitemap alone leaves the subdomains reachable and self-canonical. Fixing the canonical alone leaves the sitemap advertising 256 hosts. The redirect is what actually retires the duplicates, because a 301 from the subdomain to the apex is the only signal that says "these were never separate things".
How to check your own site in two minutes
You do not need Search Console to catch this. The sitemap check is a single request per domain, and it finds the problem before the index does:
import re, urllib.request, urllib.parse
def audit_sitemap(domain):
req = urllib.request.Request(
f"https://{domain}/sitemap.xml",
headers={"User-Agent": "Mozilla/5.0 (compatible; sitemap-audit/1.0)"})
with urllib.request.urlopen(req, timeout=20) as r:
xml = r.read().decode("utf-8", "ignore")
locs = re.findall(r"<loc>([^<]+)</loc>", xml)
hosts = [urllib.parse.urlsplit(u).netloc for u in locs]
apex = sum(1 for h in hosts if h == domain)
sub = sum(1 for h in hosts if h.endswith("." + domain))
other = len(hosts) - apex - sub
return {"domain": domain, "locs": len(locs),
"apex": apex, "subdomain": sub, "foreign": other}
for d in ["cryptochartshub.com", "dexguidehub.com", "web3academyhub.com"]:
print(audit_sitemap(d))
Run it against your own domain. If apex is 0 and locs is not, you have the same bug, and it has been running for as long as the registry has had more than one host in it. If foreign is non-zero, someone else's host is in your sitemap, which is a different and more urgent problem.
What I would do differently
The index damage is not repaired by fixing the code, and I am not going to pretend otherwise. I have not finished the rollout, so I have no recovery number to report. Consolidating duplicate hosts takes weeks, not days. The honest measurement window is four to six weeks, and the metric to watch is coverageState in the URL Inspection response, not raw impressions, which bounce around for unrelated reasons.
Two things I would keep from this:
Treat the host as a config value, not a runtime fact. Any string built from request.headers.host that ends up in a canonical, an og:url, a sitemap, or a redirect target is a bug waiting for a second deployment target. There is no version of this that stays correct when the same code serves two hostnames.
Audit the shape of your sitemap, not just its size. Every SEO check I had counted URLs and checked status codes. All of them passed, on a sitemap where 100% of the entries belonged to somebody else. A sitemap index that is 100% subdomains looks perfectly healthy to any tool that only counts <loc> elements and fetches them.
The 2,104 subdomain URLs were not spam, not a hijack, and not an attack. They were my own build output, deployed exactly where the config told it to go, competing with the pages I wanted indexed. That is the part that took longest to accept: nothing was broken in the sense of returning an error. It was all working, and pointing at the wrong host.
Top comments (0)