DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

The C$71K Bill Arrived Overnight. Your Budget Alert Just Watched.

#googlecloud #finops #sre #devops

This morning a post landed on r/googlecloud: "Got a C$71K Google Cloud bill overnight. Any advice?"

Seventy-one thousand Canadian dollars. Overnight.

A night-time cloud billing dashboard. A flat spending line spikes straight up to a red C$71K while three budget alert emails at 50%, 90% and 100% arrive, each stamped read. Caption: The alert just watched.

I won't guess that case's cause; the thread doesn't settle it. But the machinery that lets an AI key or agent bill all night is public.

The usual suspects

Cloud spend has no natural brake. An AI key or job burns at API speed while nobody watches.

  • A leaked key: a stranger points it at paid AI APIs. It just looks like spend.
  • A runaway query: one query scanning one big table, again and again.
  • A forgotten accelerator: a GPU left on after its job died.
  • A retry loop: an agent or job calling a paid model with no exit.

None of these page anybody. They're money events, not uptime events.

Why the alert could only watch

A Google Cloud budget doesn't cap spending. It sends alerts. That's the whole feature.

The default alerts fire at 50%, 90% and 100% of the budget, on actual spend. The first email lands when half the money is gone.

Billing data lags, sometimes by hours. A 3 AM email describes the past, not the fire you have now.

An alert is a witness. A kill switch is a guard.

Two progress bars. The top one is real-time usage racing ahead. The bottom one is the billing report trailing hours behind, with a widening red gap between them labelled the delay is where the money goes. An alert bell only rings once the delayed report crosses the budget line.

The kill switch

Google documents a real one, in three pieces:

  1. The budget publishes to Pub/Sub (Google's message bus). Each message carries costAmount and budgetAmount.
  2. A Cloud Run function (small code that runs when triggered) reads it.
  3. If cost passes budget, it calls the Cloud Billing API with an empty billingAccountName. Paid services stop.

Condensed from Google's own sample:

import base64, json
from googleapiclient import discovery

PROJECT_ID = "my-side-project"  # the project this switch protects

def stop_billing(data, context):
    pubsub_data = base64.b64decode(data["data"]).decode("utf-8")
    notification = json.loads(pubsub_data)
    cost = notification["costAmount"]
    budget = notification["budgetAmount"]

    if cost <= budget:
        return "No action: still under budget."

    billing = discovery.build("cloudbilling", "v1")
    name = f"projects/{PROJECT_ID}"
    # Empty billingAccountName = detach billing from the project.
    billing.projects().updateBillingInfo(
        name=name, body={"billingAccountName": ""}
    ).execute()
    return "Billing disabled. The project is dark; the bill is stopped."
Enter fullscreen mode Exit fullscreen mode

Before and after. Top: with alerts only, a spend curve climbs past the 50%, 90% and 100% markers and off the chart while the project keeps burning, labelled watched it happen. Bottom: with a kill switch wired, the same curve hits the budget, a chain from Budget to Pub/Sub to Function to Billing Detached lights up, the project goes dark and the curve flatlines, labelled stopped it.

The honest trade-offs

The switch rides the same delayed data. It shortens the night; it doesn't abolish it.

Google also warns that disabling billing can remove some resources for good. It's a fire axe behind glass. Aim it at projects you can afford to lose: sandboxes, experiments, forgotten test projects.

The dial

The final action is a dial. The same budget can page a human or trigger the kill switch. Pick per project, in daylight.

The first 24 hours

  • Stop the bleeding: detach billing or kill runaway resources; disable unknown keys.
  • Find the source: which project, which service, which key.
  • Open the billing case: state the timeline plainly. Any adjustment is Google's call, never a promise.
  • Document everything: screenshots, timeline, case number, saved outside the project.
  • Confirm the bleeding stopped: billing lags, so late charges can still land.
  • Build the guard before re-lighting: switch wired, keys rotated, one service at a time.

A 24-hour clock arc above six numbered steps that light up in order: stop the bleeding, find the source, open the case, document everything, confirm it stopped, and build the guard. The last step glows orange.

The prevention stack

  • A daily cost look: a weird bar on a cost-by-project chart beats a threshold email.
  • A byte cap on queries: BigQuery on-demand bills US$6.25 per TiB scanned. Set maximum_bytes_billed: a query past the limit fails and charges nothing.
  • Key hygiene, AI keys first: prefer short-lived credentials; restrict and rotate any key that must exist.
  • Budgets per project: thresholds you chose on purpose. 50/90/100 is a default, not a decision.
  • The kill switch on expendable projects where agents or keys run unattended: the only layer that acts while you sleep.

A 15-second animation. A cloud bill spikes to C$71K overnight. Three budget alert emails arrive and are read, but nothing stops. A replay with a kill switch wired shows the budget triggering Pub/Sub, a function and a billing detach, and the cost curve flatlines. It ends on a dark, safe project at dawn with the line Wire the guard before the night.

Set the guard before your night comes. It costs an afternoon. The other thing costs somewhat more.

A dark checklist card titled Before the night bill: the kill-switch checklist. Six items: budget per project, not per account; wire budget to Pub/Sub to a function; test the switch in daylight; set maximum_bytes_billed on every query; inventory your keys; print the 24-hour playbook. Footer: An alert is a witness. A kill switch is a guard.

Your agent loops tonight. What does your budget alert actually do: email someone, or stop something? Tell me in the comments.

New here? 3AM Files is one short, useful read a day on AI doing real DevOps work. Follow along so the next one finds you.

Honey Kiran · DevOps & Platform Engineering

I write about Kubernetes, cloud cost, and the automation that keeps both honest — the main series, the 3AM Files, and OpsBites.

Finding this useful? Follow along on Medium or Substack — the next post in the rotation is already being typed, and sharing this one is how it finds the people who need it before their 3 AM, not after.

© 2026 Honey Kiran. All rights reserved.

Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
  • Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
  • Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples are illustrative — validate in your own environment before running in production.

Top comments (0)