DEV Community

Cover image for GCP Has No Automatic Kill Switch for Leaked API Keys. Here's What I Built.
Cloudsentinel.dev
Cloudsentinel.dev

Posted on

GCP Has No Automatic Kill Switch for Leaked API Keys. Here's What I Built.

#gcp #googlecloud #security #ai

And what you can do right now to protect yourself — whether you use my tool or not.

I kept seeing posts like this on Reddit:

"Woke up to a $128,000 Google Cloud bill. Key was compromised overnight. Google denied the adjustment request."

"3-person startup. Gemini API key silently reauthorized. Normal monthly spend was $180. Bill: $82,314 in 48 hours."

"Student. Pushed API key to a private GitHub repo that was accidentally public. Was on summer break. Never saw the alerts. $55,444."

I'm a developer building on GCP myself. After reading enough of these, I realized I had zero automatic protection. If one of my keys got leaked tonight, the only thing standing between me and a five-figure bill was:

  1. Hoping I'd see a budget alert email in time
  2. Being awake
  3. Logging in fast enough
  4. Finding the right key
  5. Deleting it manually

That's a terrible safety net.

So I built CloudSentinel — an automatic kill switch for GCP API keys. But this article isn't a pitch. It's an honest walkthrough of the problem, why GCP's existing tools fall short, and what you can do about it — including building your own solution if you prefer.

Why GCP's Built-In Tools Aren't Enough

Budget alerts lag 4-12 hours

GCP billing data is not real-time. It's aggregated and delivered with a delay. By the time a budget alert fires, the damage is often already done.

Here's the timeline of a typical incident:

00:00 — API key leaks (committed to GitHub, exposed in frontend, etc.)
00:03 — Automated scanners find the key
00:05 — Attacker starts making requests
04:00 — GCP billing data updates
04:30 — Budget alert email arrives
04:31 — You're asleep
08:00 — You wake up and see the email
08:05 — You log in and delete the key
08:05 — Damage: 8 hours of uncontrolled API usage
Enter fullscreen mode Exit fullscreen mode

Budget alerts are better than nothing. But they depend on someone being awake, seeing the email, and acting fast enough.

Spend Caps are a blunt instrument

Google recently announced Spend Caps for some services (Gemini, Cloud Run, Maps). Good first step. But:

  • They pause all traffic in your project — not just the abused key
  • Your other keys and services stop working too
  • They're spend-based, not request-based — they trigger after money is already spent
  • They don't cover every GCP API

If you have 5 API keys in a project and one gets compromised, Spend Caps kill all 5. Your app goes down. CloudSentinel revokes only the specific key that crossed the threshold — everything else keeps running.

API key restrictions help but don't stop abuse

You should restrict your API keys to specific APIs and referrers. This is good practice. But:

  • IP restrictions don't help if the attacker uses the same IP range as your legitimate traffic
  • Referrer restrictions can be spoofed
  • Restrictions don't stop someone who already has the key and knows what APIs it's authorized for

Restrictions reduce the blast radius. They don't eliminate the risk.

What Actually Works: Request Volume Monitoring

The GCP billing system lags. But the GCP Cloud Monitoring API doesn't.

Specifically, the serviceruntime.googleapis.com/api/request_count metric gives you near-real-time request counts per API key. This is what GCP uses internally. And it's accessible via the timeSeries.list API with just a monitoring.timeSeries.list permission.

Here's what a simple poll looks like:

// Simplified — real implementation handles auth via service account JWT
const response = await fetch(
  `https://monitoring.googleapis.com/v3/projects/${gcpProjectId}/timeSeries?` +
  `filter=metric.type="serviceruntime.googleapis.com/api/request_count"` +
  `&interval.startTime=${startTime}&interval.endTime=${endTime}`,
  {
    headers: { Authorization: `Bearer ${accessToken}` }
  }
);

const data = await response.json();
// data.timeSeries contains request counts per credential_id (API key UID)
Enter fullscreen mode Exit fullscreen mode

The response includes metric.labels.credential_id — the unique identifier of the API key. You can map this back to specific keys and compare against thresholds.

Key insight: GCP returns cumulative counts over the requested time window. If you poll every minute with a 24-hour window, you get the total requests for that key today. When that number crosses your threshold, you revoke the key.

How to Build Your Own (DIY Version)

If you want to build this yourself, here's the minimal architecture:

Step 1: Create a minimal IAM role 

gcloud iam roles create my_api_monitor \
  --project=YOUR_PROJECT \
  --title="API Key Monitor" \
  --permissions=apikeys.keys.delete,apikeys.keys.get,apikeys.keys.list,monitoring.timeSeries.list
Enter fullscreen mode Exit fullscreen mode

This is all you need. No owner role. No billing access. No editor permissions.

Step 2: Poll the monitoring API 

from google.oauth2 import service_account
from googleapiclient.discovery import build
from datetime import datetime, timedelta, timezone

def get_request_counts(project_id, credentials):
    service = build('monitoring', 'v3', credentials=credentials)

    end_time = datetime.now(timezone.utc)
    start_time = end_time - timedelta(hours=24)

    result = service.projects().timeSeries().list(
        name=f'projects/{project_id}',
        filter='metric.type="serviceruntime.googleapis.com/api/request_count"',
        **{
            'interval.startTime': start_time.isoformat(),
            'interval.endTime': end_time.isoformat(),
            'aggregation.alignmentPeriod': '86400s',
            'aggregation.perSeriesAligner': 'ALIGN_SUM'
        }
    ).execute()

    counts = {}
    for series in result.get('timeSeries', []):
        credential_id = series['metric']['labels'].get('credential_id')
        if credential_id and series.get('points'):
            counts[credential_id] = int(
                series['points'][0]['value'].get('int64Value', 0)
            )

    return counts
Enter fullscreen mode Exit fullscreen mode

Step 3: Evaluate thresholds 

def check_and_revoke(project_id, monitored_keys, request_counts, credentials):
    apikeys_service = build('apikeys', 'v2', credentials=credentials)

    for key in monitored_keys:
        credential_id = key['credential_id']
        threshold = key['threshold']
        current_count = request_counts.get(credential_id, 0)

        if current_count > threshold:
            # Revoke the key
            apikeys_service.projects().locations().keys().delete(
                name=key['resource_name']
            ).execute()

            print(f"Revoked {key['display_name']}: "
                  f"{current_count} requests > threshold {threshold}")

            # Send alert (email, Slack, etc.)
            send_alert(key, current_count, threshold)
Enter fullscreen mode Exit fullscreen mode

Step 4: Run it on a schedule

On a Cloud Run job, a cron on a cheap VM, or any scheduler:

# Cloud Scheduler → Cloud Run job every minute
# Or a simple cron:
* * * * * python3 /opt/api-monitor/check_keys.py
Enter fullscreen mode Exit fullscreen mode

That's the core of it. The full implementation adds:

  • Deduplication (don't revoke the same key twice)
  • Retry logic for GCP API failures
  • Proper JWT auth for service accounts
  • Storage for monitored keys and thresholds
  • Email notifications

The Gotchas I Hit Building This

1. Permission casing matters

monitoring.timeSeries.list — capital S in timeSeries. I spent an embarrassing amount of time getting 403 errors because I wrote monitoring.timeseries.list. GCP is case-sensitive here.

2. GCP doesn't return zero-count keys

If a key has zero requests in the time window, it doesn't appear in the timeSeries response at all. Your polling logic needs to handle this — keys with no data have zero requests, not missing data.

3. Billing vs monitoring delay

Billing data lags 4-12 hours. Monitoring data lags 3-5 minutes. If you're using Cloud Monitoring (not billing), your detection is much faster. Always use the monitoring API, not billing data, for real-time detection.

4. Cumulative vs delta counts

The API returns cumulative counts over the requested time window. Don't subtract consecutive polls to get deltas — just compare the total against your daily/hourly threshold directly.

5. GCP alerting policies add too much latency

I initially tried using GCP alerting policies + pub/sub + Cloud Functions. The additional latency from alerting policy evaluation added 15-25 minutes on top of the 3-5 minute metric delay. Direct polling is faster and simpler.

What I Actually Built

After going through all of the above myself, I packaged it into CloudSentinel — a hosted version with a dashboard, per-key thresholds, and automatic revocation.

The setup is one gcloud command:

gcloud config set project YOUR_PROJECT && \
gcloud services enable apikeys.googleapis.com monitoring.googleapis.com \
  --project=YOUR_PROJECT --quiet && \
gcloud iam roles create cloudsentinel_XXXXXXXX \
  --project=YOUR_PROJECT \
  --title="CloudSentinel Monitor" \
  --permissions=apikeys.keys.delete,apikeys.keys.get,apikeys.keys.list,monitoring.timeSeries.list \
  --quiet && \
gcloud projects add-iam-policy-binding YOUR_PROJECT \
  --member="serviceAccount:cloudsentinel@cloudsentinel-dev.iam.gserviceaccount.com" \
  --role="projects/YOUR_PROJECT/roles/cloudsentinel_XXXXXXXX" \
  --condition=None --quiet
Enter fullscreen mode Exit fullscreen mode

It's early — I launched last week. 14-day free trial, no credit card. If you'd rather build your own using the code above, that's completely fine too — that's exactly what this article is for.

What You Should Do Right Now

Whether you use CloudSentinel or build your own, here's the minimum you should have in place for any GCP project with API keys:

1. Restrict your API keys
In GCP Console → APIs & Services → Credentials → edit each key:

  • Restrict to specific APIs only
  • Add referrer/IP restrictions where possible

2. Set a budget alert
Not a replacement for monitoring, but a backstop. Set it at 50% of your expected monthly spend so you get early warning.

3. Never commit API keys to code
Use Secret Manager or environment variables. Add .env to .gitignore. Use git-secrets or gitleaks as a pre-commit hook.

4. Audit who has access to your GCP project
Go to IAM → check every member and role. Remove anything you don't recognize.

5. Monitor request volume
Either build the script above, use CloudSentinel, or set up a Cloud Monitoring alert on serviceruntime.googleapis.com/api/request_count. Any of these is better than nothing.

Final Thought

The developers who lost $55K, $82K, $128K weren't careless. They were building, shipping, and learning. API key leaks happen to good engineers. The gap is that GCP gives you alerts but no automatic response.

A budget alert tells you your house is on fire. CloudSentinel (or your own version of it) calls the fire department automatically.

Set up some form of automatic protection before you need it. The setup takes 5 minutes. The regret takes longer.

I'm David, one of the founders of CloudSentinel. If you have questions about the GCP monitoring API, the IAM setup, or anything in this article — drop a comment. Happy to help whether or not you use our tool.

CloudSentinel — automatic GCP API key revocation. 14-day free trial, no credit card required.

Top comments (0)