Originally published on hexisteme notes.
I run a small fleet of AI agents on a single machine, and one month the cloud bill for it started climbing. I had a suspect ready before I had any evidence: a CLI tool I'd been using daily to burn a Gemini subscription. It was the newest thing in the loop, I was using it a lot, and the timing lined up. So I opened an audit trail expecting to confirm what I already believed.
I was wrong, and the way I was wrong is the actual lesson here — not "check your assumptions," which everyone already nods along to, but a structural fact about metered credentials that most single-user setups violate without noticing: ownership of a key and being billed for a key are two different relationships, and a shell profile silently collapses them into one.
The suspect that had an alibi
The tool I blamed talks to Google's backend, and the maintainers of its GitHub project had already stated, in a public issue, that API-key authentication wasn't supported — the tool only does keychain-backed OAuth. That's a claim I could check against my own logs instead of trusting.
Ninety-two session logs, all of them tagged with the OAuth consumer auth method and an empty billing-project field. Every one of the 5,409 model calls in that window routed through the free-tier consumer endpoint, not a billed one. The tool's own internal state had zero project IDs recorded anywhere. Three independent signals, all pointing the same direction: this tool was structurally incapable of hitting my billed project.
Then I ran the number that actually settled it. I lined up the tool's daily call volume against the billed project's daily request volume and computed the correlation. Pearson r = 0.089, Spearman ρ = 0.133. Functionally zero. The day I'd used the tool the most was the day the billed project saw the least traffic. If the tool were the spender, that correlation would be strongly positive, not indistinguishable from noise. My prime suspect had a perfect alibi, and I'd been about to convict it on vibes and a shared calendar.
Finding the real spender
The actual answer was hiding in a shell profile most people (including me, most days) never think about as a piece of billing infrastructure. A Gemini API key was exported globally near the top of the file — the kind of line you add once so "everything just works" and then forget exists.
Two tools were reading it: an image-generation server and a cross-vendor model-verification step, whose default model turned out to be a paid-only preview tier, not the free one I'd assumed I was calling. Neither of these had anything to do with the CLI subscription tool. But both of them tended to run in the same working sessions as the tool I suspected, because that's when I was doing agent-heavy work in general. Same sessions, same days, rising together — a textbook common-cause correlation that looks exactly like causation until you separate it by authentication method and endpoint instead of by "what was I doing that day."
This is the part that generalizes past AI tooling entirely. Swap "Gemini key in .zshrc" for "payment-provider API key exported in a CI runner's shared environment," or "a third-party analytics token baked into a base Docker image every service inherits," and the shape is identical: a credential with metered billing, injected somewhere broad enough that you can no longer enumerate who's using it. The bill arrives itemized by project, service, or SKU. It never arrives itemized by process. Once a key crosses that inheritance boundary — shell export, shared base image, org-wide environment variable — those two facts combine into something worse than "hard to debug." They combine into a question that has no answer even in principle, because there's no record of which process read the variable, only that the value was available to be read.
Why the credit made it worse, not better
I had a monthly credit against this account, and for a while that made the corrected diagnosis feel less urgent than it should have. The credit offsets the net bill. The budget guard that was actually watching my spend was configured to exclude all credits and evaluate the gross number. So while I was mentally filing this as "effectively free because of the credit," the gross figure was climbing toward a threshold that had nothing to do with what I'd actually end up owing. Reassurance computed on the wrong side of a limit isn't reassurance, it's a blind spot with better PR.
Knowing the culprit didn't stop it
Here's the part I still think about. This wasn't the first time I'd looked at this. In an earlier investigation, I'd already correctly written down — in my own notes — which two tools were the real consumers and that they were inheriting the key via shell export. I had the right answer on record before the bill actually blew past the threshold.
I just didn't act on it. The export stayed in the profile. Four days after I'd correctly identified the consumers, the budget kill switch fired for real and auto-revoked billing on the account.
That gap is the actual failure, and it's a different failure from "I didn't investigate carefully enough." I had investigated carefully enough. Diagnosis and containment are separate pieces of work, and finishing the first one produces a satisfying sense of closure that has nothing to do with whether the second one happened. Naming the consumer doesn't shrink the consumer set. Only removing the credential from the place it was broadly reachable does that. A root-cause writeup that ends at "found it" is not a fix — it's a well-documented exposure.
There was collateral damage from waiting, too. The kill switch is a blunt instrument by design — it doesn't distinguish the unwanted paid-tier calls from the legitimate ones. When it tripped, it took down a paid-only image model I actually wanted access to, for close to three weeks, alongside the leak it was supposed to stop. A backstop that fires late fires indiscriminately. (Scoping the key correctly turned out to have its own timing gap between "fixed" and "in effect" — a separate failure worth its own writeup rather than a detour here.)
The fix is a framing change, not a config change
The natural first reaction to "a key I own is being spent by processes I didn't intend" is to reach for access control: who can read this file, is it in .gitignore, is the permission bit right. On a single-user machine, that instinct runs out of road fast — there's only one account, so "who can access it" is trivially "me," and it feels like there's nothing left to secure.
That's the wrong axis entirely for a metered credential. The question was never who can access this key, it was which process is the billing account holding responsible for it right now. A single person running three tools that all inherit one exported key is exactly as unattributable, from the vendor's billing perspective, as three separate people sharing one key would be. The number of humans involved doesn't change the shape of the problem; the fact that the charge is metered does. I keep a personal library of rules for exactly this kind of recurring failure, and the entry for this one collapses to a sentence: a metered key's consumer set has to stay closed, and the only thing that keeps it closed is injecting it at the process that needs it, not at the shell that starts every process.
Concretely, that means matching the injection point to how the consumer actually runs: a server process gets the value in its own launch config, a scheduled job gets it from a permissioned file that only that job's wrapper reads, a one-off interactive command gets it prefixed inline for that single invocation. None of those let the value outlive the process it was meant for. A shell-wide export, by construction, does — and it does it silently, which is the whole problem.
The universal version
Take the AI-specific nouns out and this is a shape every backend engineer has stepped in at least once: a payment gateway's secret key gets exported in a shared .env sourced by every local script; a staging job and someone's personal debugging session both pick it up without either of them declaring it; three weeks later a finance ops person asks why the sandbox account has real transaction volume, and there's no log anywhere that says which script did it, only that the key was live everywhere. The postmortem always converges on the same two mistakes: treating a billing-identity problem as an access-control problem, and treating "we found who did it" as equivalent to "we made sure it can't happen again." Those are different jobs, and the second one is the one that actually protects the next billing cycle.
If a key is metered, it doesn't go in the profile everything sources. It goes to exactly the thing spending it, and nowhere it can outlive that thing.
(A different fleet-health metric turned out to be its own false alarm — a separate story.)
More notes at hexisteme.github.io/notes.
Top comments (0)