DEV Community

Cover image for Gemini Worked Locally but Failed in Production — The IP Address Was the Difference
Aldin Kozica
Aldin Kozica

Posted on

Gemini Worked Locally but Failed in Production — The IP Address Was the Difference

A production debugging case study where Gemini image generation worked locally but returned IMAGE_OTHER from a Hetzner server, and routing requests through Google Cloud solved it.

I spent a full day debugging an AI image generation pipeline that worked perfectly on my Mac but silently failed in production.

Same code.

Same API key.

Same prompt.

Local: image generated

Production: finishReason: IMAGE_OTHER

I'm building ThumbAPI, a REST API for automated thumbnail and image generation from just a title using Google's Gemini image API.

Then I noticed a strange pattern.

A title like:

"The Man Who Refused to Quit After 5,126 Failures"

—a reference to James Dyson and his 5,126 failed prototypes—would generate successfully on my Mac but fail in production.

So I started eliminating variables.

1. Was it the prompt?

My first assumption was that Gemini's image safety system was blocking the request.

I changed the wording and removed anything that could potentially trigger a celebrity/person-related safety filter.

No change.

2. Were the reference images causing it?

ThumbAPI sends a grid of 6 thumbnails to Gemini as style references.

Some of those images contained faces.

So I built a face detection + blur pipeline using @vladmandic/face-api and blurred the faces before sending them to Gemini.

Same result.

3. Was it the reference grid itself?

I removed the entire reference grid.

Just a text-only prompt.

The result:

Local → image generated

Production → IMAGE_OTHER

At this point, the prompt and input images were no longer good explanations.

So I started looking at the environments.

The remaining difference: network origin

My backend runs on Hetzner.

My Mac uses a residential ISP.

When the backend calls Gemini, the request originates from the server's IP — not the user's IP.

So I ran the same script from different networks.

I tested both Hetzner IPv4 and IPv6.

The pattern was consistent:

Residential IP → works

Hetzner IPv4 → fails

Hetzner IPv6 → fails

That was the first strong indication that the network origin might be involved.

The test that confirmed it

I wanted to isolate the server completely.

So I created a small proxy using Google Cloud and routed the Gemini request through it.

Same prompt.

Same API key.

Same code.

The request went through the Google Cloud proxy instead of directly from Hetzner.

It worked immediately.

That was the breakthrough.

I still don't know exactly what Google is evaluating here.

I couldn't find public Google documentation explicitly stating that Gemini image generation treats datacenter IP ranges differently from residential IPs.

So I'm not claiming this is an official policy.

It could be IP reputation, abuse prevention, network-level safety infrastructure, or something specific to the Gemini image generation pipeline.

But the behavior was reproducible.

What I learned

When debugging an AI API, it's easy to focus entirely on the obvious variables:

  • prompt
  • model
  • API key
  • input images
  • generation parameters

But there can be another variable hiding underneath all of them:

Where is the request coming from?

I'm now treating these as part of the AI pipeline as well:

  • VPS / IP reputation
  • IPv4 vs IPv6
  • datacenter vs residential origin
  • provider-specific routing
  • network-level safety and abuse systems

The interesting part is that this initially looked exactly like a prompt safety problem.

It wasn't.

At least in my case, changing the prompt and removing the images didn't solve anything.

Changing the network origin did.

And yes, this probably means I'll be moving the AI portion of ThumbAPI to Google Cloud.

If you're building with Gemini or another AI API and something works locally but mysteriously fails in production, check the network origin before spending another six hours rewriting your prompt.

Top comments (0)