This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
The numbers first
- Documented quota: 5,000 RPM
- Actual load at peak (from Cloud Console): ~115 RPM (2.3% of quota)
- Budget: $6/month
- Cost after 4 days: ~$90
- Root cause: Gemini applying quota limits from the wrong service — bug confirmed in forum threads going back to September 2025
- Fix: batch embeddings + API rate caps
- Cost after fix: back to $6/month
What TLBrain is
TLBrain is a RAG memory system I built for my wife's ghostwriting business. She conducts 10–15 client calls per week. The information was getting lost between sessions — Claude couldn't remember clients across conversations.
The solution: record calls via TL;DV → auto-transcribe → sync to a vector database → query via a remote MCP server connected to Claude Cowork.
The sync pipeline looks like this:
Cloud Tasks handles the queue. Each transcript goes through a Firestore state machine: queued → downloading → imported → syncing → synced. If a task fails, Cloud Tasks retries it automatically.
That retry behavior is important. It's what turned a quota bug into a ~$90 incident.
The pipeline ran fine for a month
200+ conversations indexed. ~$44 total (one-time backfill). Ongoing cost: $6/month.
Then gemini-embedding-2 started returning 429s.
The failure
429 RESOURCE_EXHAUSTED: Quota exceeded for quota metric
'aiplatform.googleapis.com/global_embed_content_requests_per_minute_per_base_model'
The pipeline kept running — Cloud Tasks kept enqueuing jobs, Sync Service kept picking them up. But embeddings were silently failing. New transcripts weren't being indexed. Search degraded. My wife noticed that Claude stopped remembering recent calls.
The first sign something was wrong: search results were getting stale.
First diagnosis: wrong
My first assumption — I'm hitting the rate limit. I reduced parallel sync containers from 2 to 1. Still 429.
I checked Gemini's documented quota: 5,000 RPM for gemini-embedding-2. Cloud Console showed peak usage at 2.3% of quota — ~115 RPM. Even with one container, nowhere near the limit.
The investigation
I opened Google Cloud Console to check my actual usage metrics.
Near zero.
429 quota errors. Near-zero usage shown in the dashboard. That contradiction ruled out my code. Something else was happening.
I searched for the exact error. Found forum threads going back to September 2025 — multiple developers, multiple Gemini models, same pattern: 429 at a fraction of the documented quota, dashboard showing minimal usage.
That's when the error message clicked. Look at the quota metric name again:
aiplatform.googleapis.com/global_embed_content_requests_per_minute_per_base_model
I'm calling the Gemini API — generativelanguage.googleapis.com. But the quota being enforced belongs to aiplatform.googleapis.com — Vertex AI. A completely different service with different (lower) limits.
Gemini was routing my embedding requests through Vertex AI internally and applying Vertex AI quota limits against them. That's why my dashboard showed near-zero usage: the generativelanguage quota was barely touched. The Vertex AI quota was what was exhausted — and I had no visibility into that counter at all.
The diagnosis: Gemini applying quota limits from the wrong service. No official acknowledgment at the time of writing (9 months later).
Forum threads reporting the same pattern, going back to September 2025 — no official fix as of the time of writing (9 months later):
Why it cost ~$90: anatomy of the loop
Here's where it gets expensive.
Each Sync Service task processes a full transcript document window by window:
for each window:
1. generate summary via Gemini LLM ← billed ✓
2. generate facts via Gemini LLM ← billed ✓
3. embed [summary + facts] via Gemini Embeddings ← 429 ✗ → task fails
→ write to Qdrant + Firestore
Step 3 fails on the first window. Cloud Tasks sees the failure. Cloud Tasks retries the entire task from the beginning.
Which means every retry restarted the loop from window 1 — regenerating summaries and facts for every window using Gemini LLM before even reaching embeddings again.
[task starts]
window 1: summary ← billed | facts ← billed | embed ← 429, task fails ✗
[Cloud Tasks retries]
window 1: summary ← billed again | facts ← billed again | embed ← 429, task fails ✗
[repeat for 4 days]
Some tasks had retried 50+ times before I noticed.
I noticed after 4 days and stopped the sync service manually.
The retry backoff at the time: seconds. Not minutes.
The fix
Three changes. No exotic patterns — just things I should have had from the start.
1. Batch embeddings
Instead of N separate embed_content calls in a loop, one call with a list of all texts from the window.
A transcript can have up to 400 utterances. Processing happens window by window — keeping everything in memory to batch at the transcript level would require 500MB+ containers. Not practical for Cloud Run.
So the unit of batching is the window, not the transcript. For each window: generate summary + facts, then embed everything from that window in one request, then move on.
Before the fix, each text was embedded separately:
# Before: separate embed call per item
for window in windows:
summary, facts = generate_summary_and_facts(window)
summary_embedding = gemini.embed(summary) # 1 call
fact_embeddings = [gemini.embed(f) for f in facts] # N calls
# total: ~10 calls per window
After: all texts from one window go in a single batch call, then the next window:
# After: one batch call per window
for window in windows:
summary, facts = generate_summary_and_facts(window)
texts = [summary] + facts
embeddings = gemini.embed_batch(texts) # 1 call per window
# memory released after each window
The actual API call:
response = client.models.embed_content(
model="gemini-embedding-2",
contents=texts,
config=types.EmbedContentConfig(output_dimensionality=768),
)
return [e.values for e in response.embeddings]
Same data, ~10x fewer API calls. Memory stays flat across the entire transcript.
2. API rate cap on the Sync Service
Added a hard limit on Gemini API calls per time window. If the sync service hits the cap, it stops processing new tasks — but the MCP server and search keep running.
This is the architectural separation doing its job: sync service going down doesn't affect the main service. My wife can still query Claude. New transcripts just aren't indexed until the cap resets.
Set a daily request cap directly in Google AI Studio quota settings — no code changes needed. If the sync service hits the cap, it stops processing, but the MCP server and search keep running.
3. Increased backoff for 429 errors
Changed retry behavior from seconds to minutes.
If a quota error occurs, the next retry should be far enough away to matter — not a tight loop that amplifies the problem.
Before: [1, 2, 4] seconds — applied to all errors
After: [30, 60, 120] seconds — applied specifically to 429 errors
Quota reset cycles are per-minute. A 4-second backoff doesn't help — it just hammers the same exhausted counter.
After the fix
4 transcripts had been stuck in the failed state during the incident. Restarted them manually. Zero 429s.
Cost went back to $6/month.
What I learned
1. Third-party services can silently break their own contracts.
I planned for my code failing. I didn't plan for Gemini's quota system misfiring. I do now.
2. Dashboard showing zero doesn't mean zero — it means the bug is external.
This was the key diagnostic step. When your usage metrics don't match the errors you're getting, the problem is outside your code.
3. Retry loops amplify cost asymmetrically.
The part that failed (embeddings) wasn't the expensive part. The part that ran successfully on every retry (LLM summarization) was. Always check what runs before the failure point.
4. Architectural isolation saved the main service.
Sync stopping didn't break search. The two services are separate Cloud Run instances. That decision — made before this incident — is the reason my wife could still use the tool while I fixed it.
5. Backoff in seconds for quota errors is wrong.
Quota reset cycles are measured in minutes. Retrying every 10 seconds doesn't help — it just burns through whatever quota you have left.
The refund
I submitted a support request to Google about the erroneous charges. They refunded ~$90 — the full amount of the erroneous charges.
I didn't expect it. I asked anyway.
There's a lucky coincidence worth naming: I used Gemini for both summarization and embeddings. The same provider that had the quota bug was also the one billing me for the runaway loop. That made the refund case coherent — Google caused the problem and Google charged me for it.
If I had used Claude or GPT for summarization, those charges would have been legitimate — those providers did the work correctly. I couldn't have disputed them.
Though there's a flip side: Anthropic and OpenAI work on prepaid credits. The loop would have stopped when the balance ran out, not after 4 days. A natural spending cap I didn't have with GCP's postpaid billing.
Postpaid: can spiral, but you can dispute after the fact. Prepaid: naturally capped, but no recourse. Neither is obviously safer — they just fail differently.
TLBrain is open source: github.com/Inozem/tlbrain
Top comments (0)