Postmortem: The Tuesday Our AI Bill Had No Owner

I've read a lot of postmortems. The best ones aren't about blame. They're about the gap between what happened and what the system let you see.

This one is about that gap. Specifically, the gap between "our AI bill went up" and "here's exactly why."

This is a composite of patterns I've seen across platform teams, written in the standard blameless format. The names are made up. The lesson isn't.

---

Postmortem: The Tuesday Our AI Bill Had No Owner

Status	Resolved
Severity	SEV-3 (cost impact, no customer impact)
Time to detect	~8 hours after the spike started
Time to root cause	2 hours 14 minutes
Time it should have taken	About 5 minutes
Format	Blameless

---

1. Summary

At 07:40 on a Tuesday, our daily cost alert fired. Vertex AI spend in the shared platform project had roughly tripled overnight.

Nine services call Gemini from that project. All of them use the same service account. All of them call the same model.

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

The root cause turned out to be a retry loop in the review step of a nightly ticket summarization workflow. Finding it took over two hours of Slack threads, log digging, and educated guessing.

The deeper root cause: none of our model calls carried request labels. We had cost data, but no way to slice it.

---

2. Impact

• Customers: None. The job's output was eventually correct.
• Cost: About 3x a normal day's AI spend for that project.
• People: One on-call SRE, two developers, and a FinOps analyst pulled into a two-hour investigation.
• Trust: Finance asked, fairly, how we'd know if this happened again. We didn't have a good answer.

---

3. Timeline

Monday, 16:30 A developer ships prompt_version=v4 for the ticket summarization workflow. The new prompt asks the review step to be stricter about tone.

Monday, 23:00 Nightly ticket summarization job starts. The stricter reviewer starts rejecting drafts. Each rejection triggers a rewrite with the full ticket history plus the previous draft plus review notes. Context grows on every loop.

Tuesday, 07:40 Daily cost alert fires.

07:52 On-call SRE opens Cloud Billing. Filters by project. Sees one project, one service (Vertex AI), one SKU group. No breakdown below that.

08:15 Slack thread in #platform: "Did anyone ship anything AI-related yesterday?"

08:40 Three teams reply "no." One replies "maybe? we updated a prompt."

09:05 SRE starts grepping application logs across services for model call volume. Logs are inconsistent. Some services log token counts, most don't.

09:30 A developer notices the summarization job ran for six hours instead of the usual forty minutes.

09:54 Root cause confirmed: review step looping on rejections, with context growing each time.

10:10 Prompt rolled back to v3. Retry cap lowered. Job rerun successfully.

---

4. Root cause analysis (5 whys)

Why did spend triple?
The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

Why didn't we see it during the night?
Our only cost signal was the daily billing alert. Billing data isn't real-time, so it can't catch a loop while it's happening.

Why did root cause take two hours?
Billing showed one number for the whole project. Nothing told us which service, workflow, or step was responsible.

Why couldn't billing break it down?
None of our Vertex AI calls carried request labels.

Why weren't we labeling?
We label infrastructure (VMs, buckets, clusters) through Terraform. Model API calls have no resource to label, so they fell through the cracks. Nobody owned the gap.

---

5. What went well

• The daily cost alert existed and fired.
• The team responded fast and without blame.
• Rollback was clean once we knew where to look.

6. What went wrong

• No per-call attribution for AI spend.
• No real-time token metrics, only delayed billing data.
• No retry ceiling on the review loop.
• Prompt changes weren't tracked anywhere cost-related.

---

7. Background: how Vertex AI request labels work

Before the action items, here's the feature we were missing. If you already know it, skip to section 8.

Naming note: Vertex AI docs are moving under the Gemini Enterprise Agent Platform name. If a docs link redirects there, it's the same feature.

In Google Cloud, labels live at three layers:

Layer	Attached to	Good for	Blind spot
Project labels	The project	Org-level showback	Can't split shared projects
Resource labels	VMs, buckets, endpoints	Infra cost	Model APIs have no resource
Request labels	Each API call	Per feature, per agent, per step	Only what your code sends

Request labels are key-value pairs you attach to each model call. Vertex AI forwards them to Cloud Billing, where you can filter and group costs in billing reports and query them in the BigQuery billing export.

The rules, from the docs:

• Up to 64 labels per call for Google models, up to 32 for partner models
• Keys: 1 to 63 characters, starting with a lowercase letter
• Keys and values: lowercase letters, numbers, underscores, and dashes only
• Google models: labels go in the request body on generateContent and streamGenerateContent
• Partner models: labels go in an X-Vertex-AI-Labels header (base64-encoded JSON) on rawPredict and streamRawPredict
• Labels only reach Cloud Billing on the PayGo consumption option
• The direct Gemini API (AI Studio keys) does not support this field

Here's the simplest possible Gemini call with labels:

from google import genai
from google.genai import types

client = genai.Client(vertexai=True, project="your-project-id", location="global")

response = client.models.generate_content(
    model="YOUR_GEMINI_MODEL",
    contents="Summarize this support ticket in two lines.",
    config=types.GenerateContentConfig(
        labels={
            "team": "support",
            "env": "prod",
            "use_case": "ticket_summary",
        }
    ),
)

That's it. Three labels would have turned our two-hour investigation into one query.

---

8. Action items

ID	Action	Owner	Priority
AI-1	Define and publish a label taxonomy	Platform + FinOps	P1
AI-2	Ship a labeled client wrapper	Platform	P1
AI-3	Propagate labels through multi-agent workflows	App teams	P1
AI-4	Cover partner models via header	Platform	P2
AI-5	Enforce labels in CI	Platform	P1
AI-6	Build label coverage and cost queries	FinOps	P1
AI-7	Add real-time token metrics and alerts	SRE	P1
AI-8	Cap retries in agent review loops	App teams	P1

Each one is below, with the actual implementation.

---

AI-1: Define and publish a label taxonomy

The first temptation is to let every team pick their own labels. Don't. Six months later you'll have team=Support, team=support, team=support-team, and team=cx all meaning the same thing.

We settled on four groups:

Ownership (who pays)

• team: the owning team
• cost_center: the finance code

Lifecycle (where it runs)

• env: dev, staging, prod

Product (what it's for)

• app: the service
• use_case: the specific feature, like ticket_summary

Technical (how it ran)

• agent: which agent made the call
• step: which workflow step, like plan, retrieve, draft, review
• prompt_version: like v3
• route: which routing rule picked the model, like default or fallback

Notice that step and prompt_version alone would have pointed straight at our root cause: step=review, prompt_version=v4.

What never goes in a label:

• User IDs, emails, session IDs, request IDs
• Anything sensitive

Two reasons. Cardinality: billing data is aggregated, and a label with a million unique values makes every report useless and your queries expensive. Visibility: label values land in billing data, and anyone with billing viewer access can read them.

Our rule: if a label can have more than about 100 values, it belongs in logs, not billing.

And we treat the taxonomy like an API contract. It's written down, versioned, and changes go through review. Renaming a key later breaks every historical query.

---

AI-2: Ship a labeled client wrapper

Application code no longer calls the model directly. Every call goes through one wrapper that merges default labels, validates required keys, sanitizes values, and logs token usage.

# llm_client.py
import json
import re
import contextvars
from contextlib import contextmanager

from google import genai
from google.genai import types

REQUIRED_KEYS = ("team", "env", "use_case")
_bad_chars = re.compile(r"[^a-z0-9_-]")
_ctx_labels = contextvars.ContextVar("llm_labels", default={})


def _clean(value: str) -> str:
    return _bad_chars.sub("_", str(value).strip().lower())[:63]


def _clean_key(key: str) -> str:
    key = _clean(key)
    return key if key[:1].isalpha() else f"k_{key}"[:63]


@contextmanager
def llm_labels(**labels):
    """Attach labels to every model call made inside this block."""
    token = _ctx_labels.set({**_ctx_labels.get(), **labels})
    try:
        yield
    finally:
        _ctx_labels.reset(token)


class LabeledClient:
    def __init__(self, project: str, location: str, defaults: dict):
        self._client = genai.Client(vertexai=True, project=project, location=location)
        self._defaults = defaults

    def generate(self, model: str, contents, labels: dict | None = None,
                 config: types.GenerateContentConfig | None = None):
        merged = {**self._defaults, **_ctx_labels.get(), **(labels or {})}

        missing = [k for k in REQUIRED_KEYS if k not in merged]
        if missing:
            raise ValueError(f"Missing required LLM labels: {missing}")

        merged = {_clean_key(k): _clean(v) for k, v in merged.items()}

        config = config or types.GenerateContentConfig()
        config.labels = {**(config.labels or {}), **merged}

        response = self._client.models.generate_content(
            model=model, contents=contents, config=config
        )
        _log_usage(model, merged, response)
        return response


def _log_usage(model: str, labels: dict, response):
    usage = response.usage_metadata
    print(json.dumps({
        "message": "llm_usage",
        "model": model,
        "labels": labels,
        "prompt_tokens": getattr(usage, "prompt_token_count", None) or 0,
        "output_tokens": getattr(usage, "candidates_token_count", None) or 0,
        "cached_tokens": getattr(usage, "cached_content_token_count", None) or 0,
        "total_tokens": getattr(usage, "total_token_count", None) or 0,
    }))

Three design decisions worth explaining:

• Fail loud on missing labels. A ValueError in dev is annoying for exactly one afternoon. After that, everyone labels.
• Sanitize instead of reject. Team: "Customer Support" quietly becomes customer_support. A capital letter should never cause a prod outage.
• Log usage with the same labels. This is what makes AI-7 possible. Same dimensions in billing and in monitoring.

App code stays clean:

llm = LabeledClient(
    project="your-project-id",
    location="global",
    defaults={"team": "support", "env": "prod", "app": "helpdesk"},
)

response = llm.generate(
    model="YOUR_GEMINI_MODEL",
    contents="Summarize this support ticket in two lines.",
    labels={"use_case": "ticket_summary", "prompt_version": "v3"},
)

---

AI-3: Propagate labels through multi-agent workflows

Our summarization workflow has four steps across three agents. Passing a labels dict through every function signature would never survive code review.

The llm_labels context manager in the wrapper solves this. Set labels once at the top, and every call inside inherits them:

with llm_labels(use_case="ticket_summary", prompt_version="v4", agent="orchestrator"):
    plan = llm.generate(model="YOUR_GEMINI_MODEL", contents=plan_prompt,
                        labels={"step": "plan"})

    with llm_labels(agent="writer_agent"):
        draft = llm.generate(model="YOUR_GEMINI_MODEL", contents=draft_prompt,
                             labels={"step": "draft"})

    with llm_labels(agent="reviewer_agent"):
        review = llm.generate(model="YOUR_GEMINI_MODEL", contents=review_prompt,
                              labels={"step": "review"})

contextvars works correctly with asyncio, so this holds up when agents run concurrently.

If this had been in place on Monday night, the billing breakdown would have shown one line towering over the rest: agent=reviewer_agent, step=review, prompt_version=v4. Investigation over.

---

AI-4: Cover partner models via header

Some of our teams also call partner models through Vertex AI. For those, labels travel as a base64-encoded JSON header. Here's a Claude call as the example:

import base64
import json

import google.auth
import google.auth.transport.requests
import requests

PROJECT = "your-project-id"
MODEL = "YOUR_CLAUDE_MODEL"

creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
creds.refresh(google.auth.transport.requests.Request())

labels = {"team": "support", "env": "prod", "use_case": "ticket_summary"}
labels_header = base64.b64encode(json.dumps(labels).encode("utf-8")).decode("utf-8")

url = (
    f"https://aiplatform.googleapis.com/v1/projects/{PROJECT}"
    f"/locations/global/publishers/anthropic/models/{MODEL}:rawPredict"
)

resp = requests.post(
    url,
    headers={
        "Authorization": f"Bearer {creds.token}",
        "Content-Type": "application/json",
        "X-Vertex-AI-Labels": labels_header,
    },
    json={
        "anthropic_version": "vertex-2023-10-16",
        "max_tokens": 512,
        "messages": [{"role": "user", "content": "Summarize this support ticket in two lines."}],
    },
)

Two things to watch:

• The 32-label limit for partner models. If your wrapper adds a lot of defaults, trim them for partner calls.
• Check the supported model list in the docs before assuming labels work for a newly added partner model.

We folded this into the same wrapper so app code never needs to know which path it's on.

---

AI-5: Enforce labels in CI

A wrapper only helps if people use it. Five lines of bash close the gap:

#!/usr/bin/env bash
# Fail the build if anyone calls the model directly instead of through LabeledClient.
if grep -rn "models.generate_content(" --include="*.py" src/ | grep -v "llm_client.py"; then
  echo "Direct model calls found. Use LabeledClient from llm_client.py so labels are attached."
  exit 1
fi

It's crude. It works. If you want something smarter later, swap it for an AST-based lint rule. Ship the crude one first.

---

AI-6: Build label coverage and cost queries

The billing console is fine for a quick look. For real analysis, we use the BigQuery billing export. The standard export table is named gcp_billing_export_v1_<BILLING_ACCOUNT_ID>, and labels live in an array of key-value structs.

Query 1: Label coverage. This is the first number we put on a dashboard. What percentage of AI spend actually carries labels?

SELECT
  DATE(usage_start_time) AS day,
  ROUND(SUM(cost), 2) AS total_cost,
  ROUND(SUM(IF(EXISTS(SELECT 1 FROM UNNEST(labels) WHERE key = 'team'), cost, 0)), 2) AS labeled_cost,
  ROUND(100 * SAFE_DIVIDE(
    SUM(IF(EXISTS(SELECT 1 FROM UNNEST(labels) WHERE key = 'team'), cost, 0)),
    SUM(cost)
  ), 1) AS coverage_pct
FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX`
WHERE usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND service.description LIKE '%Vertex AI%'
GROUP BY day
ORDER BY day DESC;

On the day of the incident, ours was effectively 0%. Our target is 95% in prod.

Query 2: Net cost by team, use case, and step. The query we wish we'd had at 07:52.

WITH ai AS (
  SELECT
    DATE(usage_start_time) AS day,
    (SELECT value FROM UNNEST(labels) WHERE key = 'team')     AS team,
    (SELECT value FROM UNNEST(labels) WHERE key = 'use_case') AS use_case,
    (SELECT value FROM UNNEST(labels) WHERE key = 'step')     AS step,
    (SELECT value FROM UNNEST(labels) WHERE key = 'env')      AS env,
    cost + IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0) AS net_cost
  FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX`
  WHERE usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
    -- Check how AI usage appears in your export first:
    -- SELECT DISTINCT service.description FROM ... and adjust this filter.
    AND service.description LIKE '%Vertex AI%'
)
SELECT day, team, use_case, step, env, ROUND(SUM(net_cost), 2) AS net_cost
FROM ai
GROUP BY day, team, use_case, step, env
ORDER BY day DESC, net_cost DESC;

Note the credits line. The cost column doesn't include credits, so skip that and your numbers won't match the invoice.

Query 3: Unit economics. Join labeled cost with your own business metrics:

SELECT
  a.day,
  ROUND(SUM(a.net_cost), 2) AS ai_cost,
  t.tickets_closed,
  ROUND(SAFE_DIVIDE(SUM(a.net_cost), t.tickets_closed), 4) AS cost_per_ticket
FROM ai AS a  -- the CTE from Query 2
JOIN `your_project.app_metrics.daily_tickets` AS t USING (day)
WHERE a.use_case = 'ticket_summary' AND a.env = 'prod'
GROUP BY a.day, t.tickets_closed
ORDER BY a.day DESC;

daily_tickets is your own table. This is the query that turns an AI bill into a business conversation. "AI is expensive" gets projects cancelled. "This feature costs us a fraction of a cent per ticket" gets budgets approved.

---

AI-7: Add real-time token metrics and alerts

This is the action item that would have caught the loop at 23:30 instead of 07:40.

Billing data is not real-time. Usage reaches Cloud Billing at varying intervals, and the BigQuery export updates throughout the day. That's fine for chargeback. It's useless for catching a runaway loop in the middle of the night.

The fix: emit your own usage metrics with the same labels, in real time, and reconcile against billing daily.

The wrapper already logs usage_metadata as structured JSON. On Cloud Run or GKE, JSON on stdout becomes a structured log entry automatically. From there, we created a log-based metric:

# llm_tokens_metric.yaml
filter: 'jsonPayload.message="llm_usage"'
metricDescriptor:
  metricKind: DELTA
  valueType: DISTRIBUTION
  unit: "1"
  labels:
    - key: use_case
    - key: team
    - key: env
    - key: step
valueExtractor: 'EXTRACT(jsonPayload.total_tokens)'
labelExtractors:
  use_case: 'EXTRACT(jsonPayload.labels.use_case)'
  team: 'EXTRACT(jsonPayload.labels.team)'
  env: 'EXTRACT(jsonPayload.labels.env)'
  step: 'EXTRACT(jsonPayload.labels.step)'
bucketOptions:
  exponentialBuckets:
    numFiniteBuckets: 32
    growthFactor: 2
    scale: 1

gcloud logging metrics create llm_total_tokens --config-from-file=llm_tokens_metric.yaml

Now we have a token metric we can chart and alert on, sliced by the exact same dimensions as the bill.

Alerts we added:

• Tokens per minute for any use_case above its normal ceiling. This catches loops within minutes.
• Average tokens per call jumping for any step. This catches prompt bloat right after a deploy. It would have flagged prompt_version=v4 on Monday evening.
• Prod calls from an unexpected team. This catches misconfiguration and shadow usage.

If you already run OpenTelemetry, do the same with a counter and put the labels on as attributes. Same idea, different pipe.

Daily reconciliation. Compare logged token totals against the billing export for the same labels. If they drift more than a few percent, something is calling the model outside the wrapper. That drift check is our second line of defense after CI.

---

AI-8: Cap retries in agent review loops

Not a labels fix, but it would have limited the blast radius. Every review loop now has a hard ceiling on rewrites, and the reviewer gets a trimmed context on retries instead of the full growing history. When the cap is hit, the job flags the ticket for a human instead of trying forever.

Labels tell you where the fire is. Retry caps keep the fire small.

---

9. Rollout plan

We didn't flip a switch. Here's how we rolled it out without breaking prod:

1. Week 1: Audit. Turn on the billing export if it isn't on. Run the coverage query. Get the baseline. (Ours was near zero.)
2. Week 2: Wrapper with defaults. Ship LabeledClient with project-level defaults for team and env, and warnings, not errors, for missing use_case.
3. Week 3: Migrate callers. Move services over one at a time. Watch coverage climb.
4. Week 4: Enforce. Flip warnings to errors in non-prod. Turn on the CI check.
5. Week 5: Monitoring. Log-based metrics, dashboards, alerts.
6. After that: Chargeback. Only once coverage is stable above target. Charging teams on partial data destroys trust fast.

---

10. Residual risks and known limitations

Labels aren't magic. Here's what's still true after all of the above:

• PayGo only. Provisioned Throughput traffic won't carry your labels into billing. Your labeled spend won't add up to total AI spend if you use PT. Plan dashboards for that.
• Labels are not a security control. Anyone with model access can send team=someone_else. If attribution must be trustworthy, pair labels with separate service accounts per team.
• Values are visible. Anyone with billing data access can read them. No customer data, ever.
• Partner limits differ. 32 labels, header-based, and not every partner model is on the supported list.
• Taxonomy changes break history. Renaming a key is effectively a data migration.
• New models may lag. When a new model family lands in Model Garden, verify label support before relying on it for chargeback.
• The direct Gemini API doesn't support this. Teams using AI Studio keys won't show up in labeled billing.

---

11. Lessons learned

We were labeling the wrong layer. Our Terraform tagged every VM and bucket perfectly. But our fastest-growing cost line had no resource to tag. When the shape of your spend changes, your attribution has to change with it.

Cost is a reliability signal. A loop that burns tokens is a bug, even when the output is eventually correct. It deserves real-time alerting, not a next-morning email.

Attribution is a developer job. Only code can attach labels. FinOps can define the taxonomy and SRE can build the alerts, but if developers don't own the labels, nothing else works.

The best postmortem action items are boring. One wrapper. Three required labels. One coverage query. None of it is exciting. All of it would have turned a two-hour investigation into a five-minute one.

---

Your turn: if your AI bill spiked tomorrow morning, how long would it take your team to explain it? Five minutes? Two hours? Never? Tell me in the comments. 👇

---

References

• Google Cloud docs: Custom metadata labels (Vertex AI)
• Google Cloud docs: Monitor cost using custom metadata labels (Gemini Enterprise Agent Platform)
• Google Cloud docs: Consumption options overview
• Google Cloud docs: Provisioned Throughput overview
• Google Cloud docs: Request-response logging
• Cloud Billing docs: Structure of Standard data export
• Cloud Billing docs: Create, edit, or delete budgets and budget alerts