I run a small automation: GitHub Actions starts a Python script every day, Gemini writes a blog post, and the Blogger API publishes it. It worked for weeks. Then it started failing, and I found three separate problems.
1. HTTP 503: "This model is currently experiencing high demand"
My log showed the same error again and again:
HTTP 503: This model is currently experiencing high demand.
Spikes in demand are usually temporary. Please try again later.
A 503 is a problem on the server's side, and it usually goes away. It is fine to retry, but only a couple of times, with a pause between tries.
2. HTTP 429: "You exceeded your current quota"
After the 503s, the last attempt returned a 429. I opened my Google AI Studio Rate Limit page, and it showed the free-tier limits for the model I was using at that time:
- 5 requests per minute
- 20 requests per day
These numbers can change, so check your own dashboard.
The cause was in my own script. It had an outer retry loop and an inner retry loop, so one run could send up to 9 requests. Add a few manual test runs, and the daily quota was gone quickly.
What I changed:
- Cap one run at 3 requests in total
- Keep only one place where retries happen
- Wait 15 seconds between requests
- On a 429, do not retry. Stop with a clear message, because retrying only burns more quota
class GeminiQuotaExhausted(RuntimeError):
pass
def call_gemini(...):
response = requests.post(url, params={"key": api_key}, json=payload, timeout=180)
if response.status_code == 429:
raise GeminiQuotaExhausted("Quota exhausted, will retry in the backup run")
...
3. A backup run and "skip if already done"
GitHub's scheduler can run late, and a model can be busy at one time and fine two hours later. So I added a second schedule, and I made the script safe to run twice:
on:
schedule:
- cron: "30 4 * * *" # main run
- cron: "0 9 * * *" # backup run
workflow_dispatch:
- name: Check if already posted today
id: skip
shell: bash
run: |
if ls posts/published/$(date +%F)-*.md 1>/dev/null 2>&1; then
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- name: Generate post
if: steps.skip.outputs.skip != 'true'
run: python scripts/generate_post.py
If today's post is already published, the backup run does nothing and stays green.
Cron runs in UTC. Convert the time for your own timezone.
4. The YAML error my local check missed
After my changes, GitHub refused to run the workflow:
Invalid workflow file: 'id' is already defined, 'shell' is already defined, 'run' is already defined
One step had the same keys twice. I had checked the file with Python before pushing:
import yaml
yaml.safe_load(open(".github/workflows/post.yml")) # printed no error
It passed, because that loader quietly accepts duplicate keys. GitHub does not. A linter can catch this before you push:
pip install yamllint
yamllint -d "{rules: {key-duplicates: enable}}" .github/workflows/post.yml
Lesson: a file that "parses fine" is not the same as a valid workflow.
Quick checklist
- [ ] Count the requests one run can send, including nested retries
- [ ] Don't retry a 429. Retry a 503 only a few times
- [ ] Add a pause between requests
- [ ] Add a backup schedule, and skip if the job is already done
- [ ] Run a YAML linter with duplicate-key checks before you push
- [ ] Test manually once, not ten times, because every run uses quota
Wrapping up
I thought my bug was a "busy server". It was partly that, but my own retry logic was the bigger problem. Looking at the real numbers on the dashboard made the cause obvious.
You can see the project on GitHub, and my portfolio here: ritika-rajput-portfolio.vercel.app.
Have you run into API quota limits in your own automation? What did you do about it?
Top comments (0)