Originally published on the WatchNext blog.
Google had crawled WatchNext and indexed none of it. The coverage report listed 11 pages as crawled, currently not indexed and 200 as discovered, currently not indexed — found, but never fetched.
The crawl stats report had what looked like the reason. Of 453 crawl requests over the month, 17% had ended in a DNS error. If Google couldn't reliably resolve the site, it would hardly be surprising that it wasn't indexing it.
None of those errors were WatchNext's.
The report was about four sites
WatchNext lives at watchnext.leyu.studio. Search Console had been set up as a domain property for leyu.studio, which is the convenient choice: one verification covers every subdomain. It also means every number in the report is the sum of every subdomain, with nothing on the overview telling you so.
The hosts table, further down the same report, splits it out:
| Host | Crawl requests |
|---|---|
shop.leyu.studio |
234 |
watchnext.leyu.studio |
134 |
leyu.studio |
77 |
checkout.leyu.studio |
8 |
An online shop, its checkout, this app, and the bare domain. The "WatchNext crawl report" was a report on four sites, of which WatchNext was a little under a third.
The site that didn't exist
Look at the third row. leyu.studio itself had nothing on it — no page, no server, and critically no DNS address record. The subdomains all pointed somewhere; the bare domain pointed nowhere, and its www variant didn't exist at all.
Google had still made 77 requests to it. And the DNS error rate, 17.00% of 453, is 77 requests. Almost every failure in the report — 77 of the 81 failed requests, with the other four being ordinary 404s — was Google repeatedly trying to reach a site that had never been built.
That match is worth trusting, and the reason is not that the numbers are equal. It's that there is a mechanism which forces them to be. A hostname with no address record cannot get past DNS, so every one of its requests has to fail at exactly that stage and no other. The count is the consequence of the cause.
The daily chart is consistent with it, too. On each of the last five days of the export, every request recorded no bytes downloaded and a response time of zero — which is what a request looks like when there was never a server to talk to.
One thing we can't explain: the same hosts table lists all four hosts with a status of "No problems", including the one that couldn't resolve. Whatever that column checks, it evidently isn't the fate of individual requests, so we didn't lean on it.
The match that meant nothing
Here is the part worth writing down, because it very nearly went into the diagnosis as a finding.
The crawl report also breaks requests down by file type, and 8.17% of them were HTML. WatchNext received 134 requests. Multiply one by the other and you get almost exactly 11 page fetches — and the coverage report said exactly 11 pages had been crawled and not indexed.
That is a beautiful result. Google had fetched precisely the 11 pages it then declined to index, and never got round to the other 200. It explains the coverage report completely.
It is also not a result at all. The 8.17% is the HTML share across all four hosts combined. Applying it to WatchNext alone assumes a Next.js app and a Shopify storefront request the same mix of JavaScript, images, stylesheets and pages — and there is no reason on earth they would. The multiplication borrows a ratio from a population it doesn't describe, and the tidy answer it produces is a coincidence.
So two numbers matched perfectly in the same investigation, one after the other, and only one of the matches meant anything. The difference between them is the difference between a count that a cause forces and a count that arithmetic happens to land on.
What was actually going on with WatchNext
With the apex's errors set aside, the rest of the report is quieter and more useful. 99.56% of crawl requests were refreshes of URLs Google already knew about; 0.44% were discovery. Google was barely looking for anything new.
It is tempting to add that the busiest day of crawling — 113 requests on 14 August — came three days before we committed the fix that gave these pages server-rendered content, so Google's opinion of the site would have been formed on the broken version. But that chart is for the whole property too, and it doesn't say which host those 113 requests went to. It's the same trap as the HTML share, one section later.
The evidence that does hold up is narrower. Inspecting a single post showed Google had found it through a link from another site, crawled it on 20 August — after the fix — fetched it successfully, and read the canonical URL we had declared. And it still hadn't indexed it. Nothing in that is a technical fault. It points at a young domain that Google has no particular reason to trust yet, which no DNS record will change.
What we did about it
We built something at leyu.studio, so the bare domain now resolves to a real page instead of nothing, and www redirects to it. We also added a separate URL-prefix property for each site, so WatchNext's report is only about WatchNext.
It is worth being plain about what that fixes. It removes the noise: the property should stop reporting a failure rate that was never about the app. It does not, by itself, get a single WatchNext page indexed. Those were never the same problem — they only looked like one because they were summed into the same number.
The general version
When two numbers from different places match exactly, it feels like confirmation, and it is very hard to argue with. Two questions separate the matches that mean something from the ones that don't.
Is there a mechanism that forces them to be equal? A hostname with no address can only fail at DNS, so its request count and its DNS failure count must agree. Nothing forces an 8% share to produce exactly the right number of page fetches; it just did.
Does the ratio describe the thing you applied it to? A percentage is a statement about a population. Apply it to a subset with a different make-up and it will still return a number — precise, plausible and wrong.
The new site went up on 12 September. Search Console's reports take weeks to reflect a change like this, so we don't yet know what the numbers look like afterwards — and given the rest of this post, we're not going to guess. We'll add them here when they exist.
WatchNext is a deliberately simple TV tracker: it tells you when the next episode of your favourite shows airs, for any show, in any country — and nothing else.
Top comments (0)