I thought my pipeline retried 6 times. It was making up to 48 billed calls.

My publishing pipeline had a retry limit. MAX_TOPIC_ATTEMPTS = 6. If a draft failed validation — too short, dead link, too similar to an existing post — it would throw the topic away and try a different one, up to six times, then give up.

Six attempts. That was the mental model. Six is a small number, so I never thought about it again.

That last part is the actual mistake. I set the constant myself, I picked the value deliberately, and having picked it I stopped treating it as a question. A number you chose feels known in a way a number you inherited does not — which is backwards, because the one you chose is the one nobody else has checked.

What made me look was a day where the schedule ran and nothing appeared on the blog. No post, no error I'd been alerted about, just an empty run. I went into the logs expecting a crash and found the opposite: the pipeline had worked perfectly, generated draft after draft, rejected every one of them on validation, exhausted its attempts, and shut down exactly as designed.

Then I opened the billing page for that day.

Six was the outer loop

The retry limit I set was the outermost loop. Inside each of those six attempts, other things were also retrying, and they were doing it against the API.

MAX_TOPIC_ATTEMPTS = 6            ← the number I was thinking of

  per attempt:
    _run_content_prompt(max_retries=3)   ← JSON came back malformed, regenerate
    + expansion call                     ← draft came in under the word count
    + image generation                   ← separate model, separate charge

A comment I later wrote in that file puts the real figure at 4 to 8 API calls per attempt. Six attempts at four to eight calls each is 24 to 48 billed requests for what I had filed in my head as "six tries."

Nothing here was a bug in the ordinary sense. Every loop was doing exactly what it was written to do. The failure was that I had a number in my head — six — and the code had a different number, and I never multiplied them.

It is worth noticing why this shape is easy to build by accident. Each of those retries was added on a different day, for a different reason, by someone reasonable. The JSON retry went in because the model occasionally returns malformed output and regenerating fixes it. The expansion call went in because drafts sometimes land under the word count and asking for more is cheaper than starting over. The topic loop went in because some topics are simply duplicates of existing posts and the fix is to pick another. Every one of those is a good local decision. Nobody ever sat down and decided the system should be allowed to make 48 calls; the 48 is an emergent property of three sensible choices stacked on top of each other.

And the calls I thought were free were not

The second half of this was a pricing assumption I had never checked.

I had it in my head that image generation was the expensive part and text was more or less free. Images are visibly a "generation," they take seconds, they produce a file. Text felt like it was barely a request.

So when I looked for the cost, I looked at image count.

The billing breakdown said something else:

• Text generation: 42% of the spend
• Image generation: 58%

Not a rounding error. Nearly half the bill was the part I had been treating as free. And text generation is exactly the part that multiplies — every one of those 24 to 48 calls is a text call. Images generate once per post; text regenerates every time validation rejects a draft.

Which means the expensive failure mode is invisible. A run that publishes nothing costs more text tokens than a run that succeeds, because failure is what triggers regeneration.

The fix, and the trap inside the fix

I added a hard ceiling on the raw call count:

MAX_GEMINI_CALLS_PER_RUN = 20

def _check_gemini_call_budget():
    global _gemini_call_count
    _gemini_call_count += 1
    if _gemini_call_count > MAX_GEMINI_CALLS_PER_RUN:
        raise GeminiCallBudgetExceeded(...)

The important detail is where it is checked: immediately before every actual API request, not at the top of any loop. Loop-level limits are what got me into this — each loop respected its own limit perfectly and the product was still 48. A counter on the raw calls doesn't care how many loops are nested above it.

Then I nearly broke it in a way that would have been much harder to notice.

My first version had GeminiCallBudgetExceeded subclass RuntimeError. That felt right — it's a runtime problem, and the codebase already used RuntimeError for generation failures.

That would have made the circuit breaker do nothing.

The retry logic catches RuntimeError and treats it as "this attempt failed, try another topic." So the budget exception would have been swallowed by the exact machinery it was built to stop, and the run would have kept going — now with a ceiling that reported being hit while changing nothing.

The class now carries a comment explaining why it inherits from plain Exception:

Deliberately does NOT subclass RuntimeError — the retry blocks must NOT catch this and try yet another topic, which would defeat the whole point of a hard ceiling.

A circuit breaker that raises an exception your retry loop already catches is not a circuit breaker. It's a log line.

This one generalizes past API budgets. Any time you add a stop condition to a system that already has broad error handling, the question is not "does it raise" — it's "who catches it first."

How to check this on your own pipeline

Three things, none of which take long:

Count the worst case by hand. Find every retry limit in the generation path and multiply them, then add whatever fires once per attempt. If the answer surprises you, that's the finding. Mine was six in my head and 24-48 on paper.

Pull the actual billing breakdown by model, not the total. The total tells you what you spent; the split tells you which assumption was wrong. I would never have looked at text calls if I hadn't seen 42% sitting next to them.

Grep your error handling for the exception type your limiter raises. If anything upstream catches that type or a parent of it, your limit is decorative.

The debugging loop was billed too

There is a second-order version of this problem that took me longer to see.

When a run fails, you debug it. Debugging means running it again to watch what happens. Every one of those diagnostic runs went through the same generation path and cost the same money as a real one — and diagnostic runs fail by definition, which is the expensive branch.

So I added a preview mode. It generates the draft and runs the full validation pass — word count, live-link checks, duplicate detection, required fields — then stops before image generation and before publishing.

I want to be precise about what that saves, because I initially overstated it to myself. Preview mode is not free. It still makes the text calls; those are the whole point, since text generation is what I'm usually debugging. What it removes is image generation and the publish step. Given the 42/58 split, that is real money on a run I'm going to throw away anyway, but it is not zero.

The rule I use now: if the question is "why did validation reject this," preview mode answers it. If the question is "does the published output look right," that requires a real run, and I should expect to pay for it.

Testing it without paying to test it

There is an obvious problem with verifying a spend limiter: the straightforward test is to let it spend.

I mocked the API call to fail every time and ran the pipeline against the mock. No requests left the machine, every loop behaved exactly as it does in production, and the counter incremented on the same line it increments on for real.

It stopped on call 20. Not 19, not 21.

That took a couple of minutes and cost nothing, and it is the only reason I know the ceiling actually holds rather than merely existing in the source.

What I'd tell past me

Count calls, not loops. Every retry limit in a nested system is a factor in a multiplication, and the product is the number that gets billed. If you can't say what the worst-case call count of one run is, you don't have a limit — you have several limits that happen to be small individually.

Check what things actually cost instead of assuming. I assumed text was free because it felt lightweight. It was 42% of the bill, and it was the half that multiplies under failure.

Make the failure path visible. A run that publishes nothing looks like "nothing happened." In this system it was the most expensive outcome available, and there was nothing on any dashboard that said so.

Ask what catches your exception. This is the one I'd have missed for months. A safety mechanism that throws a type your existing error handling already absorbs is worse than no mechanism, because it reads as protection in the source and does nothing at runtime.