Last week, I lost more time to AI infrastructure than to the actual creative work.
The task sounded simple: generate images through Vertex AI for a content pipeline.
What actually happened:
- one key worked yesterday and failed today
- one project returned
invalid_grant - another returned
permission denied - free tier worked sometimes, then hit quota walls
- the code looked fine, but the system still didn’t produce images
This is the part nobody tells you about AI workflows:
most failures are not model failures. They’re credential, project, and policy failures.
Here’s how I finally debugged the whole thing and got the image route working again.
The symptoms
At first, the failures looked unrelated.
I saw three different classes of errors:
invalid_grant: account not found
403 Permission denied
429 RESOURCE_EXHAUSTED
That usually means one of two things:
- your system is broken in multiple places
- your system is pointing at multiple environments and you don’t know which one is real
In my case, it was the second one.
Step 1: Stop guessing which path is canonical
The first useful move was brutally simple:
find the one script that the team actually trusts.
For us, that was:
~/clawd/ops/production/scripts/generate_panels.py
That became the source of truth.
Not old snippets.
Not half-working notebooks.
Not memory.
Once I checked the actual script, I immediately found one hidden problem:
PROJECT = "old-project-id"
The pipeline was still hardcoded to an old project.
So even when I updated the credentials, the requests were still going to the wrong place.
That alone explained a lot.
Step 2: Separate free-tier failure from paid-route failure
We had two different routes mixed together:
- Gemini API / AI Studio free tier
- Vertex AI paid route
That sounds harmless, but it creates terrible debugging conditions.
Because the failure modes are different:
- free tier dies with quota errors
- Vertex dies with IAM / service account / project errors
If you mix them, you start solving the wrong problem.
For example, this looked like a model problem at first:
429 RESOURCE_EXHAUSTED
But it turned out to be just a burned free-tier key.
Meanwhile the paid route was failing for a completely different reason.
Lesson: treat free and paid as separate systems, even if they use the same model.
Step 3: Verify the service account before touching the code
Once I had the new Vertex JSON, I didn’t start by generating an image.
I started by checking whether the credential could even mint a token.
That kind of test saves time because it tells you whether the problem is:
- auth
- project permissions
- or model invocation
In Python, the logic is basically:
from google.oauth2 import service_account
from google.auth.transport.requests import Request
creds = service_account.Credentials.from_service_account_file(
"vertex_ai_key.json",
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
creds.refresh(Request())
print(creds.token[:40])
If this step fails, don’t touch your prompts.
Don’t touch the model.
Don’t touch the rendering code.
You don’t have an image problem yet.
You have an auth problem.
Step 4: Watch for organization policies
This one burned the most time.
I created a new service account, everything looked correct, and then Google Cloud refused to create a JSON key.
The error turned out to be caused by this policy:
iam.disableServiceAccountKeyCreation
That wasn’t obvious from the first screen.
The UI showed one policy as “not enforced,” while a legacy constraint was still active somewhere above it.
That kind of mismatch is why cloud debugging feels cursed.
The practical fix was not to keep fighting the same project.
The practical fix was to create a clean personal project without inherited org-policy baggage.
That ended up being faster than trying to untangle admin policy state.
Step 5: Create one clean project and move on
The final working setup looked like this:
- a new clean Vertex project
- a new service account
- a fresh JSON key
- the canonical script updated to the new project id
- one successful test generation to prove the whole path worked
Only after that did I consider the route fixed.
Not when the key existed.
Not when the policy screen looked green.
Not when the script stopped crashing.
Only when this produced an actual file:
outputs/nanobanana_vertex_test.png
That was the only result that mattered.
The working mental model
When an AI image pipeline breaks, I now check things in this order:
- Which script is canonical?
- Which project is the request actually hitting?
- Can the credential mint a token?
- Is this free-tier quota or Vertex IAM?
- Are org policies blocking service-account keys?
- Did I generate one real image successfully?
That order is much faster than randomly changing keys and re-running prompts.
What actually fixed it
For us, the final fix was not “better prompting.”
It was:
- stopping reliance on stale project ids
- replacing broken credentials
- separating free-tier and paid routes
- avoiding inherited org-policy traps
- testing the pipeline end-to-end with a real output file
That’s not glamorous.
But it’s the difference between a pipeline you trust and a pipeline that only works when you’re lucky.
Final thought
A lot of AI tooling discourse is still obsessed with models.
But once you work with these systems in production, the real bottleneck is often much more boring:
identity, permissions, quotas, and project hygiene.
The model can be state of the art.
If your project graph is a mess, you still won’t ship.
Where Terminal Skills fits
This is also exactly the kind of workflow I want to turn into a Terminal Skill.
Not because a skill should hide the cloud setup behind magic, but because the debug order should not live only in someone’s memory.
A useful vertex-ai-image-pipeline skill would give an agent a repeatable checklist:
- identify the canonical script
- verify the configured project id
- test whether the service account can mint a token
- separate free-tier quota failures from Vertex IAM failures
- check for organization-policy blockers
- run one minimal generation test before touching prompts
- report the exact failing layer instead of guessing
That is the broader idea behind Terminal Skills: turn messy, real operational workflows into reusable agent skills.
I will probably translate this article into a proper Terminal Skills use case next, because this is the kind of boring production workflow agents need more than another prompt template.
If you’ve had to debug a broken AI pipeline recently, I’d genuinely love to hear what failed first for you.
Top comments (0)