DEV Community

Ogi
Ogi

Posted on Originally published at brokeitwithai.blogspot.com

I let AI run two blogs for a month. Google indexed 1 post out of 81.

I built two fully automated blog pipelines. Pick a topic, research it, write the body, generate the images, publish — no human in the loop. One was WordPress on its own domain, the other was Blogspot.

WordPress ran from July 23 to August 23 and shipped 44 posts. Blogspot started August 8 and stopped on the same day the other one did, so 37 posts in 15 days. Eighty-one posts total.

Both pipelines ran on a schedule with no approval step. A run would ground itself against official sources, draft the body, generate three to five images, run a mechanical validation pass (word count, live-link checks, duplicate detection against previous posts, required-field checks), retry on failure, and publish. If validation failed too many times, the run gave up and logged it. That was the entire loop. It worked, in the sense that posts appeared on a schedule and none of them were broken.

Today I measured what Google actually did with them.

One post is indexed. Out of 81.

The one that made it is a guide to searching Illinois unclaimed property. I have no idea why that one and not the other 80. It is not the longest, it is not the most linked, and it is not the oldest.

The actual numbers

I pulled these from the Search Console URL Inspection API, one URL at a time. This is not an estimate off a dashboard summary — it is Google telling me, per URL, what state it thinks that page is in.

State WordPress Blogspot
Posts published 44 37
Crawled — currently not indexed 27
Discovered — currently not indexed 25
Redirect error 9
URL is unknown to Google 16 3
Indexed 1 0

Search performance over the same window: WordPress got 217 impressions and 0 clicks. Blogspot got 0 impressions.

Two hundred and seventeen times a human saw one of my pages in a result list and decided not to click it.

Stacked bar chart: 81 posts published across two sites, only 1 indexed by Google

How I measured this, in case you want to check your own site

The Search Console web UI will show you an aggregate coverage chart, which is where I had been getting my (wrong) sense that things were merely slow. The per-URL truth lives in the URL Inspection API, and it takes about fifteen lines to pull.

You need a service account with access to the property, and you need to match the property type exactly. A domain property is addressed as sc-domain:example.com; a URL-prefix property, which is what a blogspot subdomain verifies as, is addressed with the full https://example.com/ string. Getting this wrong returns a 403 that looks like a permissions problem but is not — I lost a round to that today.

Then for each published URL you call urlInspection().index().inspect() with the URL and the site, and read indexStatusResult.coverageState. The states you care about are:

  • Submitted and indexed — the only one that means anything good
  • Crawled - currently not indexed — Google read it and passed
  • Discovered - currently not indexed — Google never fetched it
  • URL is unknown to Google — Google has no record of it at all
  • Redirect error — something is broken in your redirect chain

Counting those five states across every post you have published takes a couple of minutes and tells you more than any dashboard summary. I would run it before drawing any conclusion about whether a site is working, and specifically before applying to any ad network.

The two sites failed in different ways, and that turned out to matter

My first read was just "both dead." But the two columns are not the same failure.

WordPress: 27 posts are "Crawled — currently not indexed." Googlebot fetched those pages. It read them. Then it decided they were not worth putting in the index.

Blogspot: 25 posts are "Discovered — currently not indexed." Google knows the URLs exist and never bothered to fetch them.

That distinction is the whole story. The first is a content problem. The second is a crawl budget problem, and no amount of writing better posts fixes it, because nothing is coming to read them.

I had been treating both sites as one failure with one cause. They were not.

Three causes, separated

1. I picked niches without looking at who already owns the results

Site one was unclaimed property and government benefits. Site two was product recalls and consumer rights. I checked search volume. The demand was real.

I never checked who currently sits in the top ten.

When I finally looked: unclaimed property results are dominated by state government sites. Every state runs its own official lookup, and there is no reason for a reader to prefer my page over the actual agency holding their money. Recalls were worse. Recall news breaks and major outlets take the top slots instantly on domain authority alone. For lemon law queries, six of the top nine results were law firms. In that market a single converted reader is worth thousands of dollars in fees, so law firms spend accordingly on SEO.

There was no seat left at either table. I built anyway.

Top search results for both niches, colored by site type. Independent blogs hold 0 of 19 spots.

2. I had AI write about things neither of us had done

Ask a model to write "how to find unclaimed property in Illinois" and it will. Fluently. With structure and headers and a FAQ.

The problem is that the output is a recombination of things already published elsewhere. I have never searched for unclaimed property. Neither has the model. There is no first-hand anything in those 81 posts.

I learned how that reads from the outside when AdSense rejected the site.

I did not think that was unfair. It was accurate. I had published 81 pages of information that already existed in better form, at more authoritative addresses. Google has no reason to index a worse copy.

3. I published into a vacuum

This is the part I would most want back.

I believed that consistent quality publishing gets you found eventually. So I published and waited. I never shared a post anywhere. Nobody linked to either site. Nobody mentioned them.

A new domain with zero external signal does not get crawl budget. Those 25 "Discovered — currently not indexed" URLs on Blogspot are what that looks like in the data. I could have published 100 more posts into that state and the number would not have moved.

Nine posts died to an unchecked box

Nine of the Blogspot URLs came back as "Redirect error." That one is not about content at all.

Blogspot has a setting called HTTPS redirect — it sends http requests to the https version. It was off. Google crawled those URLs while it was off, recorded them as redirect errors, and that is the nine.

The setting was turned on later. I fetched all nine of those URLs today, one by one: every single one returns 200 now. The problem was already fixed. But the Search Console record still reflects the state from when it was broken, because nothing has come back to recrawl them.

So those nine are not "currently broken." They are the scar from something that was broken, and the scar is not healing because the site never gets recrawled. Which folds right back into the crawl budget problem above.

I was confidently wrong about the cause

Worth writing down plainly: I had diagnosed those nine as a mobile redirect loop.

Blogspot appends ?m=1 for mobile requests, and the canonical tag on the resulting page points back at the original URL. Each end points at the other. I looked at that structure, decided it was a loop Google would choke on, and stopped there. It is a satisfying explanation. It is internally consistent. It is the kind of thing that sounds right when you say it out loud.

Then I spun up a fresh blog and measured it. The new blog has the exact same structure. Different theme, same behavior — because that is just how Blogspot serves mobile, on every blog on the platform, all of which index fine.

The real cause was a settings toggle. Much less interesting.

That is the most useful thing I learned today. A plausible mechanism is not evidence. I saw a structure, reasoned my way to a conclusion, and never built the one comparison that would have taken five minutes to falsify it.

Building that comparison turned up something else

While setting up the new blog I went through the settings and found two switches under crawlers and indexing: custom robots.txt and custom robots header tags. Both were on. So I opened the actual file.

/robots.txt was completely empty. Not one line.

Turning that switch on means "I'll write this myself." Nothing had been written. So the sensible default the platform generates for you got replaced with a blank file.

I turned the switch off and fetched it again. This is what should have been there the whole time:

User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /

Sitemap: https://example.blogspot.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Disallow: /search is the line that matters. Without it, every label archive and every search results page is fair game for crawling. On a site with healthy crawl budget that is a rounding error. On a site that gets almost no crawl budget, those listing pages eat the few fetches you were going to get.

Then I checked the old blog. Its robots.txt is still empty. It ran that way for the full fifteen days.

I'm not going to claim that's the cause of the 25 uncrawled URLs — zero external signal produces the same symptom, and I can't separate their contributions with the data I have. What I can say is that both conditions were bad at the same time.

Same lesson as above, in a different shape. A toggle you didn't need to touch will silently replace a working default with nothing, and no screen anywhere warns you. You only see it by fetching the file.

My blog served an empty robots.txt for 15 days, next to the default Blogger serves with Disallow: /search and a sitemap line.

What I am changing

Check occupancy, not volume. Before committing to a topic, search three to five of its head terms and look at what kind of site holds the top ten. Government agencies, major news outlets, law firms, or a funded company's content marketing — any one of those in control means a new site does not get in. Search volume tells you the room is crowded. It does not tell you the seats are taken.

Write only what actually happened to you. This is a strategy argument, not an ethics one. Recombined information loses to the source it was recombined from, permanently. But a failure I personally had is something no other site can publish. This post is the first thing I have written that meets that bar.

Publishing is the start of distribution, not the end of it. New domains need some external signal before Google spends crawl budget on them. That means putting the work somewhere humans already are, before expecting search to do anything.

And one open question. Blogspot published 37 posts in 15 days, roughly two and a half a day from a domain that was days old. I do not know whether those 25 uncrawled URLs are the result of zero external signal, publishing velocity, or both. The current data cannot separate them. Next time I am slowing the cadence down specifically so that variable is off the table.

One month, plus hosting, a domain, and API bills. The tuition was not cheap for the runtime, but at least I got numbers out of it instead of a hunch.

I am rebuilding now with all three of those inverted, and I will be publishing the measurements either way, including if it fails again.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.