Setting a Budget Alert for Vertex AI Spend

#ai #machinelearning #tutorial #llm

A Cloud Billing budget scoped to Vertex AI is fifteen minutes of work and it is worth doing on day one. It is also not a spending limit, and the gap between what people assume it does and what it does is where the expensive surprises live.

A budget is not a cap

Google’s budgets documentation is unambiguous: setting an alerts-only budget does not automatically cap Google Cloud usage or spending. The budget observes and notifies. Nothing stops.

Two further properties compound that. Usage reporting lags — Google documents a delay between using a resource and the cost being reported to Cloud Billing — so an alert at 100% of budget can arrive when true spend is already well past it. And the alert is evaluated against a monthly period, so a runaway job on the first of the month has the whole month’s budget to burn through before any threshold trips.

The correct mental model is a smoke detector, not a circuit breaker. It is genuinely valuable, it will tell you about the class of problem nobody would otherwise notice for weeks, and it is not the control that stops the fire. The control that stops the fire is a quota, a maximum instance count, and a per-key limit in your own application — see budget controls that actually stop spend.

One practical obstacle before you start: budgets are resources on the Cloud Billing account, not on the project. The permission you need is billing.budgets.create on the billing account — carried by roles/billing.admin or roles/billing.costsManager — and Project Owner does not include it. In most organisations the billing account is held by a different team from the one running the workload, so the first step is usually a request rather than a command. The costs-manager role is the right one to ask for: it can create and manage budgets without carrying the ability to link or unlink projects.

Finding the right service ID

Budget service filters take Cloud Billing service IDs of the form services/XXXXXX-XXXXXX-XXXXXX, not API names like aiplatform.googleapis.com. These IDs circulate in blog posts and Terraform gists and they are frequently wrong, so look yours up. The Cloud Billing Catalog API lists them with display names:

# needs cloudbilling.googleapis.com enabled and an API key or
# an OAuth token with the cloud-platform scope
curl -sS \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  'https://cloudbilling.googleapis.com/v1/services?pageSize=500' \
  | python3 -c '
import json,sys
for s in json.load(sys.stdin).get("services", []):
    if "vertex" in s["displayName"].lower() or "agent platform" in s["displayName"].lower():
        print(s["serviceId"], "--", s["displayName"])
'

Two things to watch when you read the output. There may be more than one relevant service — generative model usage and the wider platform have historically been reported under different display names — and the display name may be the current product name rather than the one you searched for. The --filter-services flag accepts a list, so include every ID that matches rather than picking one and hoping.

Cloud Billing service IDs are stable once assigned, but which services exist, and what they are called, is not. Re-run the listing when a budget stops matching what you see in the billing report rather than assuming the budget is wrong.

Creating the budget

Google’s gcloud reference for budget creation documents the flags. --threshold-rule is repeatable, its percent is a fraction between 0 and 1, and its basis is either current-spend or forecasted-spend.

gcloud billing budgets create \
  --billing-account=0X0X0X-0X0X0X-0X0X0X \
  --display-name="Vertex AI monthly" \
  --budget-amount=2000USD \
  --filter-projects=projects/my-inference-project \
  --filter-services=services/XXXXXX-XXXXXX-XXXXXX \
  --credit-types-treatment=exclude-all-credits \
  --threshold-rule=percent=0.5 \
  --threshold-rule=percent=0.9 \
  --threshold-rule=percent=1.0 \
  --threshold-rule=percent=0.9,basis=forecasted-spend \
  --notifications-rule-pubsub-topic=projects/my-inference-project/topics/budget-alerts

--credit-types-treatment=exclude-all-credits deserves a moment. With credits included, a project running on trial or committed-use credits shows near-zero spend against the budget until the credits run out, at which point spend appears to jump from nothing to full rate overnight. Excluding credits means the budget tracks what the workload actually costs, which is the number you want to know about while the credits still exist.

Choosing thresholds that mean something

The default 50/90/100 pattern is fine and it is also entirely retrospective: by the time 90% of a monthly budget is spent, whatever caused it has been running for a while.

The threshold that earns its place is the forecasted-spend one. A rule at percent=0.9,basis=forecasted-spend fires when Google’s projection for the month reaches 90% of budget, which for a step change in usage can be days into the month rather than three weeks in. For a workload that is meant to be flat, a forecast alert on day four is the difference between catching a loop and paying for it.

One caution about forecasts: they are unreliable in the first days of a period and after any genuine step change, so a forecast rule will produce false positives when you legitimately scale up. That is an acceptable trade for the notice it buys, but it means a forecast alert should page a human to look rather than trigger anything automatic.

Making the alert do something

An email to a billing administrator is a weak signal — the recipients are often not the people who can act, and the message arrives with no context about which workload moved. The Pub/Sub notification is the version you can build on:

Create the topic before the budget references it: gcloud pubsub topics create budget-alerts.
Grant the budget service the ability to publish to it. The billing budgets service publishes as a Google-managed identity, and without the binding the budget is created and never notifies — the same silent-failure shape as the storage-trigger grant in triggering a function from a Cloud Storage upload.
Subscribe something that adds context. The message carries the budget name, the cost so far, the budget amount and the threshold that fired — but not which service or job caused the change. A handler that queries your billing export for the top movers since yesterday and posts that alongside the alert turns a number into something actionable.
Send it where the owning team already is, not to a billing inbox. An alert that arrives in the channel of the team who deployed the model gets acted on the same day.

A budget scoped to the Vertex AI service tells you that Vertex AI spend moved. It does not tell you which feature, customer or team moved it, because Cloud Billing sees API calls and not the reasons for them, and the moment a second provider is in the picture there is a second billing surface with different granularity again. Multigrid is an LLM gateway, so cost is recorded per request against the key that made it, which makes the attribution question answerable before the invoice arrives. The provider-neutral version of this problem is in allocating AI infrastructure cost.