DEV Community: Ogi

I thought my pipeline retried 6 times. It was making up to 48 billed calls.

Ogi — Fri, 28 Aug 2026 09:16:19 +0000

My publishing pipeline had a retry limit. MAX_TOPIC_ATTEMPTS = 6. If a draft failed validation — too short, dead link, too similar to an existing post — it would throw the topic away and try a different one, up to six times, then give up.

Six attempts. That was the mental model. Six is a small number, so I never thought about it again.

That last part is the actual mistake. I set the constant myself, I picked the value deliberately, and having picked it I stopped treating it as a question. A number you chose feels known in a way a number you inherited does not — which is backwards, because the one you chose is the one nobody else has checked.

What made me look was a day where the schedule ran and nothing appeared on the blog. No post, no error I'd been alerted about, just an empty run. I went into the logs expecting a crash and found the opposite: the pipeline had worked perfectly, generated draft after draft, rejected every one of them on validation, exhausted its attempts, and shut down exactly as designed.

Then I opened the billing page for that day.

Six was the outer loop

The retry limit I set was the outermost loop. Inside each of those six attempts, other things were also retrying, and they were doing it against the API.

MAX_TOPIC_ATTEMPTS = 6            ← the number I was thinking of

  per attempt:
    _run_content_prompt(max_retries=3)   ← JSON came back malformed, regenerate
    + expansion call                     ← draft came in under the word count
    + image generation                   ← separate model, separate charge

A comment I later wrote in that file puts the real figure at 4 to 8 API calls per attempt. Six attempts at four to eight calls each is 24 to 48 billed requests for what I had filed in my head as "six tries."

Nothing here was a bug in the ordinary sense. Every loop was doing exactly what it was written to do. The failure was that I had a number in my head — six — and the code had a different number, and I never multiplied them.

It is worth noticing why this shape is easy to build by accident. Each of those retries was added on a different day, for a different reason, by someone reasonable. The JSON retry went in because the model occasionally returns malformed output and regenerating fixes it. The expansion call went in because drafts sometimes land under the word count and asking for more is cheaper than starting over. The topic loop went in because some topics are simply duplicates of existing posts and the fix is to pick another. Every one of those is a good local decision. Nobody ever sat down and decided the system should be allowed to make 48 calls; the 48 is an emergent property of three sensible choices stacked on top of each other.

And the calls I thought were free were not

The second half of this was a pricing assumption I had never checked.

I had it in my head that image generation was the expensive part and text was more or less free. Images are visibly a "generation," they take seconds, they produce a file. Text felt like it was barely a request.

So when I looked for the cost, I looked at image count.

The billing breakdown said something else:

Text generation: 42% of the spend
Image generation: 58%

Not a rounding error. Nearly half the bill was the part I had been treating as free. And text generation is exactly the part that multiplies — every one of those 24 to 48 calls is a text call. Images generate once per post; text regenerates every time validation rejects a draft.

Which means the expensive failure mode is invisible. A run that publishes nothing costs more text tokens than a run that succeeds, because failure is what triggers regeneration.

The fix, and the trap inside the fix

I added a hard ceiling on the raw call count:

MAX_GEMINI_CALLS_PER_RUN = 20

def _check_gemini_call_budget():
    global _gemini_call_count
    _gemini_call_count += 1
    if _gemini_call_count > MAX_GEMINI_CALLS_PER_RUN:
        raise GeminiCallBudgetExceeded(...)

The important detail is where it is checked: immediately before every actual API request, not at the top of any loop. Loop-level limits are what got me into this — each loop respected its own limit perfectly and the product was still 48. A counter on the raw calls doesn't care how many loops are nested above it.

Then I nearly broke it in a way that would have been much harder to notice.

My first version had GeminiCallBudgetExceeded subclass RuntimeError. That felt right — it's a runtime problem, and the codebase already used RuntimeError for generation failures.

That would have made the circuit breaker do nothing.

The retry logic catches RuntimeError and treats it as "this attempt failed, try another topic." So the budget exception would have been swallowed by the exact machinery it was built to stop, and the run would have kept going — now with a ceiling that reported being hit while changing nothing.

The class now carries a comment explaining why it inherits from plain Exception:

Deliberately does NOT subclass RuntimeError — the retry blocks must NOT catch this and try yet another topic, which would defeat the whole point of a hard ceiling.

A circuit breaker that raises an exception your retry loop already catches is not a circuit breaker. It's a log line.

This one generalizes past API budgets. Any time you add a stop condition to a system that already has broad error handling, the question is not "does it raise" — it's "who catches it first."

How to check this on your own pipeline

Three things, none of which take long:

Count the worst case by hand. Find every retry limit in the generation path and multiply them, then add whatever fires once per attempt. If the answer surprises you, that's the finding. Mine was six in my head and 24-48 on paper.

Pull the actual billing breakdown by model, not the total. The total tells you what you spent; the split tells you which assumption was wrong. I would never have looked at text calls if I hadn't seen 42% sitting next to them.

Grep your error handling for the exception type your limiter raises. If anything upstream catches that type or a parent of it, your limit is decorative.

The debugging loop was billed too

There is a second-order version of this problem that took me longer to see.

When a run fails, you debug it. Debugging means running it again to watch what happens. Every one of those diagnostic runs went through the same generation path and cost the same money as a real one — and diagnostic runs fail by definition, which is the expensive branch.

So I added a preview mode. It generates the draft and runs the full validation pass — word count, live-link checks, duplicate detection, required fields — then stops before image generation and before publishing.

I want to be precise about what that saves, because I initially overstated it to myself. Preview mode is not free. It still makes the text calls; those are the whole point, since text generation is what I'm usually debugging. What it removes is image generation and the publish step. Given the 42/58 split, that is real money on a run I'm going to throw away anyway, but it is not zero.

The rule I use now: if the question is "why did validation reject this," preview mode answers it. If the question is "does the published output look right," that requires a real run, and I should expect to pay for it.

Testing it without paying to test it

There is an obvious problem with verifying a spend limiter: the straightforward test is to let it spend.

I mocked the API call to fail every time and ran the pipeline against the mock. No requests left the machine, every loop behaved exactly as it does in production, and the counter incremented on the same line it increments on for real.

It stopped on call 20. Not 19, not 21.

That took a couple of minutes and cost nothing, and it is the only reason I know the ceiling actually holds rather than merely existing in the source.

What I'd tell past me

Count calls, not loops. Every retry limit in a nested system is a factor in a multiplication, and the product is the number that gets billed. If you can't say what the worst-case call count of one run is, you don't have a limit — you have several limits that happen to be small individually.

Check what things actually cost instead of assuming. I assumed text was free because it felt lightweight. It was 42% of the bill, and it was the half that multiplies under failure.

Make the failure path visible. A run that publishes nothing looks like "nothing happened." In this system it was the most expensive outcome available, and there was nothing on any dashboard that said so.

Ask what catches your exception. This is the one I'd have missed for months. A safety mechanism that throws a type your existing error handling already absorbs is worse than no mechanism, because it reads as protection in the source and does nothing at runtime.

I let AI run two blogs for a month. Google indexed 1 post out of 81.

Ogi — Thu, 27 Aug 2026 13:24:18 +0000

I built two fully automated blog pipelines. Pick a topic, research it, write the body, generate the images, publish — no human in the loop. One was WordPress on its own domain, the other was Blogspot.

WordPress ran from July 23 to August 23 and shipped 44 posts. Blogspot started August 8 and stopped on the same day the other one did, so 37 posts in 15 days. Eighty-one posts total.

Both pipelines ran on a schedule with no approval step. A run would ground itself against official sources, draft the body, generate three to five images, run a mechanical validation pass (word count, live-link checks, duplicate detection against previous posts, required-field checks), retry on failure, and publish. If validation failed too many times, the run gave up and logged it. That was the entire loop. It worked, in the sense that posts appeared on a schedule and none of them were broken.

Today I measured what Google actually did with them.

One post is indexed. Out of 81.

The one that made it is a guide to searching Illinois unclaimed property. I have no idea why that one and not the other 80. It is not the longest, it is not the most linked, and it is not the oldest.

The actual numbers

I pulled these from the Search Console URL Inspection API, one URL at a time. This is not an estimate off a dashboard summary — it is Google telling me, per URL, what state it thinks that page is in.

State	WordPress	Blogspot
Posts published	44	37
Crawled — currently not indexed	27	—
Discovered — currently not indexed	—	25
Redirect error	—	9
URL is unknown to Google	16	3
Indexed	1	0

Search performance over the same window: WordPress got 217 impressions and 0 clicks. Blogspot got 0 impressions.

Two hundred and seventeen times a human saw one of my pages in a result list and decided not to click it.

How I measured this, in case you want to check your own site

The Search Console web UI will show you an aggregate coverage chart, which is where I had been getting my (wrong) sense that things were merely slow. The per-URL truth lives in the URL Inspection API, and it takes about fifteen lines to pull.

You need a service account with access to the property, and you need to match the property type exactly. A domain property is addressed as sc-domain:example.com; a URL-prefix property, which is what a blogspot subdomain verifies as, is addressed with the full https://example.com/ string. Getting this wrong returns a 403 that looks like a permissions problem but is not — I lost a round to that today.

Then for each published URL you call urlInspection().index().inspect() with the URL and the site, and read indexStatusResult.coverageState. The states you care about are:

Submitted and indexed — the only one that means anything good
Crawled - currently not indexed — Google read it and passed
Discovered - currently not indexed — Google never fetched it
URL is unknown to Google — Google has no record of it at all
Redirect error — something is broken in your redirect chain

Counting those five states across every post you have published takes a couple of minutes and tells you more than any dashboard summary. I would run it before drawing any conclusion about whether a site is working, and specifically before applying to any ad network.

The two sites failed in different ways, and that turned out to matter

My first read was just "both dead." But the two columns are not the same failure.

WordPress: 27 posts are "Crawled — currently not indexed." Googlebot fetched those pages. It read them. Then it decided they were not worth putting in the index.

Blogspot: 25 posts are "Discovered — currently not indexed." Google knows the URLs exist and never bothered to fetch them.

That distinction is the whole story. The first is a content problem. The second is a crawl budget problem, and no amount of writing better posts fixes it, because nothing is coming to read them.

I had been treating both sites as one failure with one cause. They were not.

Three causes, separated

1. I picked niches without looking at who already owns the results

Site one was unclaimed property and government benefits. Site two was product recalls and consumer rights. I checked search volume. The demand was real.

I never checked who currently sits in the top ten.

When I finally looked: unclaimed property results are dominated by state government sites. Every state runs its own official lookup, and there is no reason for a reader to prefer my page over the actual agency holding their money. Recalls were worse. Recall news breaks and major outlets take the top slots instantly on domain authority alone. For lemon law queries, six of the top nine results were law firms. In that market a single converted reader is worth thousands of dollars in fees, so law firms spend accordingly on SEO.

There was no seat left at either table. I built anyway.

2. I had AI write about things neither of us had done

Ask a model to write "how to find unclaimed property in Illinois" and it will. Fluently. With structure and headers and a FAQ.

The problem is that the output is a recombination of things already published elsewhere. I have never searched for unclaimed property. Neither has the model. There is no first-hand anything in those 81 posts.

I learned how that reads from the outside when AdSense rejected the site.

I did not think that was unfair. It was accurate. I had published 81 pages of information that already existed in better form, at more authoritative addresses. Google has no reason to index a worse copy.

3. I published into a vacuum

This is the part I would most want back.

I believed that consistent quality publishing gets you found eventually. So I published and waited. I never shared a post anywhere. Nobody linked to either site. Nobody mentioned them.

A new domain with zero external signal does not get crawl budget. Those 25 "Discovered — currently not indexed" URLs on Blogspot are what that looks like in the data. I could have published 100 more posts into that state and the number would not have moved.

Nine posts died to an unchecked box

Nine of the Blogspot URLs came back as "Redirect error." That one is not about content at all.

Blogspot has a setting called HTTPS redirect — it sends http requests to the https version. It was off. Google crawled those URLs while it was off, recorded them as redirect errors, and that is the nine.

The setting was turned on later. I fetched all nine of those URLs today, one by one: every single one returns 200 now. The problem was already fixed. But the Search Console record still reflects the state from when it was broken, because nothing has come back to recrawl them.

So those nine are not "currently broken." They are the scar from something that was broken, and the scar is not healing because the site never gets recrawled. Which folds right back into the crawl budget problem above.

I was confidently wrong about the cause

Worth writing down plainly: I had diagnosed those nine as a mobile redirect loop.

Blogspot appends ?m=1 for mobile requests, and the canonical tag on the resulting page points back at the original URL. Each end points at the other. I looked at that structure, decided it was a loop Google would choke on, and stopped there. It is a satisfying explanation. It is internally consistent. It is the kind of thing that sounds right when you say it out loud.

Then I spun up a fresh blog and measured it. The new blog has the exact same structure. Different theme, same behavior — because that is just how Blogspot serves mobile, on every blog on the platform, all of which index fine.

The real cause was a settings toggle. Much less interesting.

That is the most useful thing I learned today. A plausible mechanism is not evidence. I saw a structure, reasoned my way to a conclusion, and never built the one comparison that would have taken five minutes to falsify it.

Building that comparison turned up something else

While setting up the new blog I went through the settings and found two switches under crawlers and indexing: custom robots.txt and custom robots header tags. Both were on. So I opened the actual file.

/robots.txt was completely empty. Not one line.

Turning that switch on means "I'll write this myself." Nothing had been written. So the sensible default the platform generates for you got replaced with a blank file.

I turned the switch off and fetched it again. This is what should have been there the whole time:

User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /

Sitemap: https://example.blogspot.com/sitemap.xml

Disallow: /search is the line that matters. Without it, every label archive and every search results page is fair game for crawling. On a site with healthy crawl budget that is a rounding error. On a site that gets almost no crawl budget, those listing pages eat the few fetches you were going to get.

Then I checked the old blog. Its robots.txt is still empty. It ran that way for the full fifteen days.

I'm not going to claim that's the cause of the 25 uncrawled URLs — zero external signal produces the same symptom, and I can't separate their contributions with the data I have. What I can say is that both conditions were bad at the same time.

Same lesson as above, in a different shape. A toggle you didn't need to touch will silently replace a working default with nothing, and no screen anywhere warns you. You only see it by fetching the file.

What I am changing

Check occupancy, not volume. Before committing to a topic, search three to five of its head terms and look at what kind of site holds the top ten. Government agencies, major news outlets, law firms, or a funded company's content marketing — any one of those in control means a new site does not get in. Search volume tells you the room is crowded. It does not tell you the seats are taken.

Write only what actually happened to you. This is a strategy argument, not an ethics one. Recombined information loses to the source it was recombined from, permanently. But a failure I personally had is something no other site can publish. This post is the first thing I have written that meets that bar.

Publishing is the start of distribution, not the end of it. New domains need some external signal before Google spends crawl budget on them. That means putting the work somewhere humans already are, before expecting search to do anything.

And one open question. Blogspot published 37 posts in 15 days, roughly two and a half a day from a domain that was days old. I do not know whether those 25 uncrawled URLs are the result of zero external signal, publishing velocity, or both. The current data cannot separate them. Next time I am slowing the cadence down specifically so that variable is off the table.

One month, plus hosting, a domain, and API bills. The tuition was not cheap for the runtime, but at least I got numbers out of it instead of a hunch.

I am rebuilding now with all three of those inverted, and I will be publishing the measurements either way, including if it fails again.