Olawale Afuye

Posted on Jun 1

Your Deleted Google API Key Is Still Working — Here's Why That's a Security Crisis

#security #webdev #googlecloud #devops

You just discovered your Google API key was leaked. Maybe it showed up in a GitHub search. Maybe a secret scanner flagged it. You panic, open the Google Cloud Console, and delete it.

Done. Crisis averted.

Except it isn't.

That key is still working. For the next 23 minutes, an attacker can keep using it — making requests, racking up your cloud bill, or accessing data you thought was already cut off.

This is not a theoretical risk. It's a documented vulnerability that Google initially dismissed as "expected behavior" — before later upgrading it to a P0/S0 critical bug.

Let's break down exactly what happened, why it matters, and what you actually need to do to protect yourself.

The Old Mental Model: "Google API Keys Are Low-Risk"

For years, Google told developers something that made sense at the time: it's fine to include certain API keys in your frontend code. These were project identifiers — used for services like Google Maps — not secrets. They were designed to be public.

So developers did exactly what they were told. They hardcoded these keys into JavaScript, committed them to GitHub, and shipped them in client bundles. It became standard practice.

Secret scanners now crawl the internet and find thousands of these keys in public repositories every day.

Here's where things changed.

Gemini Changed Everything

When Google launched Gemini, they plugged it into the same API key infrastructure that Maps had always used. Same key type. Same issuance flow. Same developer habits.

But the blast radius of a leaked Gemini key is completely different from a leaked Maps key.

A leaked Maps key might let someone make geocoding requests on your bill. Annoying, costly, but bounded.

A leaked Gemini key could:

Let attackers send unlimited LLM requests billed to your account
Potentially expose cached or uploaded data associated with your project
Serve as a foothold for broader GCP exploration depending on project permissions

The keys hadn't changed. The services they unlocked had.

The 23-Minute Window: What "Eventual Consistency" Actually Means in Practice

Security researcher Joe Leon discovered something that breaks the most basic assumption in incident response: deleting a key does not immediately stop it from working.

Here's the technical reason.

Google Cloud Platform runs on a globally distributed architecture. When you delete an API key, that deletion is written to a primary data store. But the authentication layer — the servers that actually validate incoming requests — relies on cached copies of credential data distributed across many nodes.

These caches are updated asynchronously. The system prioritizes availability and performance. The trade-off is that credential state changes don't propagate instantly.

The result: deleted keys remain valid for up to 23 minutes while those caches catch up.

This is called eventual consistency — a well-understood distributed systems pattern. The problem is that security teams don't think about it, because their mental model is:

Delete key → key is dead → attacker is locked out.

That model is wrong.

Google's Initial Response (And Why Disclosure Still Mattered)

When Joe Leon initially reported this, Google's response was to classify it as expected behavior — not a bug. The reasoning: eventual consistency is a known property of distributed systems, and this is documented.

This is technically true. It is also deeply insufficient.

The issue isn't whether eventual consistency is expected internally at Google. The issue is that it breaks the mental model of every developer and incident responder who presses the delete button and assumes the key is dead.

After Leon went public with the research, something shifted. The right people inside Google saw it. It got reclassified as a critical P0/S0 bug and escalated for remediation.

This is why responsible public disclosure matters — not to embarrass vendors, but because internal triage teams don't always have the context to understand the real-world security implications of architectural choices. Sometimes the right path through the bureaucracy is around it.

Does This Apply to AWS and Other Services?

Yes — though the specifics vary.

AWS (Amazon Web Services)
IAM access key deletions generally propagate quickly, but AWS also has edge cases:

There is a small propagation window (~4 seconds) for new requests
More critically: deleting an IAM key does not invalidate active STS sessions generated from that key. Those temporary credentials remain valid until they naturally expire — which could be hours

Postmark
Postmark is a positive outlier here. Their API tokens are designed for immediate invalidation — delete it and it's dead, no propagation delay.

The General Rule
Treat revocation as an asynchronous event, not a synchronous kill switch. The safer assumption during any security incident is: the key may still work for some window of time after deletion.

What This Means for Incident Response

Stop treating "I deleted the key" as the end of your response process.

Here's a more realistic incident response mindset:

1. Delete the key — but don't stop there.
Deletion starts the clock. It doesn't end the threat.

2. Rotate associated credentials immediately.
Any service accounts, tokens, or roles that interacted with the compromised key should be rotated. Don't assume one deletion closes the blast radius.

3. Invalidate active sessions explicitly.
On AWS, this means finding and revoking STS tokens derived from the key. On GCP, check for active OAuth sessions or service account tokens linked to the same project.

4. Monitor logs for the next 30 minutes.
After deletion, watch your Cloud Logging or CloudTrail for continued usage of the revoked key. If you're still seeing hits, your window is still open.

5. Apply emergency access restrictions.
If the situation is critical, use Service Control Policies (AWS) or VPC Service Controls (GCP) to add a hard constraint while the deletion propagates.

How to Properly Manage API Keys in a Frontend App

Since frontend code is inherently visible to users, the goal isn't to perfectly hide keys — it's to minimize what a leaked key can do.

1. Apply Strict Restrictions in the Cloud Console

Never deploy an unrestricted key. In GCP under APIs & Services > Credentials, configure:

Application Restrictions: Lock the key to specific HTTP referrers (e.g., https://yourdomain.com/*). A key stolen from your bundle becomes nearly useless when called from any other origin.
API Restrictions: Limit the key to only the APIs your app actually uses. A Maps key should not be able to touch Gemini. Full stop.

2. Never Hardcode Keys in Source Code

Use environment variables and keep them out of version control:

# .env.local
NEXT_PUBLIC_MAPS_API_KEY=your_key_here

# .gitignore
.env
.env.local
.env*.local

This doesn't prevent exposure — bundlers often inline these values — but it keeps secrets out of your git history and prevents accidental repository exposure.

3. Use a Backend Proxy for Sensitive APIs

If you need Gemini, billing-sensitive APIs, or anything with significant data access: don't call it from the frontend.

Route requests through your own backend:

Browser → Your API Server (holds the key securely) → Google Cloud API

Your backend can authenticate the user, apply rate limits, and call Google — without ever exposing the key to the client. This is the only approach that genuinely protects high-sensitivity credentials.

4. Set Up Billing Alerts

A leaked key often shows up as an unexpected spike in usage before anything else. Configure budget alerts in Google Cloud Billing so you get a notification the moment something looks abnormal. This is your earliest warning system.

5. Rotate Keys on a Schedule

Don't wait for a breach to rotate. Build key rotation into your regular ops cycle — monthly for sensitive services, quarterly at minimum for everything else. The shorter the valid lifetime of any given key, the smaller the window of any eventual compromise.

The Broader Lesson

The real vulnerability here isn't a code bug. It's a broken assumption baked into how developers and security teams think about credentials.

Google told developers keys were safe to expose. Developers believed them. Google's distributed architecture made deletion asynchronous. Nobody documented that clearly for incident responders. A researcher had to find it, get rejected, go public, and escalate through media coverage before the right people inside a trillion-dollar company treated it as critical.

That's a systems failure — technical, organizational, and communicative.

For you as a developer, the takeaway is straightforward:

Restrictions > secrecy for frontend keys
Backend proxies > frontend exposure for sensitive APIs
Revocation is asynchronous — respond accordingly
Monitoring doesn't stop when you press delete

The delete button is not a kill switch. Build your security posture around that fact.

DEV Community