A prepaid balance is not an invoice you can argue about next month. When it reaches zero the platform starts refusing calls, and it refuses them for every credential on the account at once — the payment worker, the nightly export, and the internal tool that drained it. That single fact decides this question. Use a separate, least-privilege API key for the admin console, give that key its own spend ceiling, and keep the production credential out of the console entirely.
The ceiling is the part people skip.
Most teams get as far as "the internal tool should have its own API key," ship it, and stop. Scopes get trimmed, the key gets a name, everyone feels responsible. Then the console spends the balance anyway, because a scope says what a credential may call, not how much of the shared wallet it may burn while calling it.
What a shared console key actually costs
Our admin console is an ordinary Node.js app: a few screens over the same backend the production workers use, sitting behind SSO, used by support to inspect a customer and re-run a failed job. For its first year it authenticated with the production key, because that key was already in the deploy environment and nobody wanted to file a ticket to get another one.
A support engineer opened the replay screen with a date filter that was wider than they meant it to be, got no feedback for a few seconds, and clicked again. Two replays, both accepted, both fanning out into per-customer work. The console did exactly what it was told.
What paged me hours later was not the console. It was the production consumer, refusing every call with 402 because the prepaid balance was gone, which in practice looked like a queue that had stopped draining and a pile of jobs whose deadlines had quietly passed. Duplicate deliveries went out. Then nothing went out at all. The recovery was not interesting either — top up the balance, wait for the consumers to catch up, apologize to the customers who got the same notification twice — and the postmortem action item wrote itself, with nothing in it about the replay screen's UX.
Here is the invariant that incident produced, and it's the one worth carrying into your own design: a credential that can spend the entire balance can refuse traffic for every other credential on the account. Blast radius is denominated in money, not in permissions. Least privilege that only covers verbs is half a control.
Should an internal admin console get its own API key, separate from production credentials?
Yes, once more than one person can open it — and the reasons are boring and operational rather than theoretical.
A named console key makes growth visible. Internal tools accrete capabilities; someone adds an export, someone adds a bulk retry, and each addition shows up as a scope you have to approve rather than a capability the tool silently already had. It also makes attribution possible: in a usage report, spend attributed to the console key is humans clicking, and spend attributed to the worker key is the product running. Those two numbers move for completely different reasons, and once they're separated you can alert on each.
Rotation is the third reason. A production credential lives in a deploy pipeline and rotates on whatever cadence your platform team enforces; an internal tool credential tends to live in an env var someone set years ago and forgot. Internal tools are not exempt from the rotation schedule. Giving the console its own key is what makes it auditable enough to rotate on the same clock as everything else.
Scope it to what the screens actually call. Read endpoints for the inspection views, one narrow write scope for the replay action, nothing else — the same discipline OWASP's secrets guidance and the OAuth scope model both push you toward, and the same discipline that makes a broken function-level authorization check in an internal tool a contained problem instead of an account-wide one.
Three ways to cap console spend, and what each one refuses
The decision axis here is uncomfortable: any ceiling you set will eventually refuse traffic someone wanted. The question is whose traffic, and whether the refusal is legible when it happens.
| Mechanism | What it refuses | Failure mode | Use when |
|---|---|---|---|
| Per-key window cap | Console calls past the cap, production untouched | A legitimate bulk action stops halfway through | The console has bounded, well-understood workloads |
| Reserve floor on the balance | Any console call once the balance nears the floor | Console goes read-only during a top-up delay | Production refusals are far more expensive than console refusals |
| Alert-only threshold | Nothing | Nobody is awake at 03:00 | You genuinely cannot predict console spend yet |
Alert-only is where most teams start, and it's a reasonable first week. It stops being reasonable the moment your alert has fired twice and nothing changed, because an alert that routinely fires without an action is a control that has been deleted without anyone deciding to delete it.
I run both of the first two. The window cap keeps a single bad click bounded; the reserve floor keeps a slow, unattended drain from crossing into production.
They fail differently, which is the point.
The guard loop, and why it has to be idempotent
The admission check itself is small. What matters is that it is atomic and idempotent, because the exact situation you're defending against — a double-clicked button, a retried request, a proxy that replayed something on a timeout — is also the situation that makes a naive read-then-write counter lie to you.
// Package spendguard admits console-originated calls against a per-key
// window cap and a reserve floor. Production keys do not pass through it.
package spendguard
import (
"context"
"errors"
"time"
)
var (
ErrCeiling = errors.New("console window cap reached")
ErrReserve = errors.New("balance at production reserve floor")
)
type Ceiling struct {
KeyID string // the console's own key, never the production one
Window time.Duration // 24 * time.Hour
WindowCap int64 // minor units the console may spend per window
ReserveFloor int64 // balance the console is not allowed to touch
}
type Decision struct {
Admitted bool
WindowSpent int64
}
type Store interface {
Balance(ctx context.Context) (int64, error)
// Reserve is atomic and idempotent on requestID: it admits the call only
// if the window total stays at or below cap, and replaying the same
// requestID returns the original decision instead of charging twice.
Reserve(ctx context.Context, requestID, keyID string, cost, cap int64, until time.Time) (Decision, error)
}
func (c Ceiling) Admit(ctx context.Context, s Store, requestID string, cost int64, now time.Time) error {
bal, err := s.Balance(ctx)
if err != nil {
return err // fail closed: an unknown balance is not permission to spend
}
if bal-cost < c.ReserveFloor {
return ErrReserve
}
d, err := s.Reserve(ctx, requestID, c.KeyID, cost, c.WindowCap, now.Add(c.Window))
if err != nil {
return err
}
if !d.Admitted {
return ErrCeiling
}
return nil
}
The requestID is not decoration. Derive it from something stable about the user's action — the screen, the target resource, the filter hash — so that the second click carries the same id as the first and reserves nothing new. That one line is the difference between a guard and a counter.
Fail closed on the balance read. I argued myself out of that at first, on the theory that a billing lookup hiccup shouldn't take the console down, and I was wrong: a guard that opens when it can't see the balance is exactly the guard that isn't there during the incident you built it for. The console degrades to read-only. That's an acceptable Tuesday.
Two operational details that make this survivable in production. Surface the refusal properly — return the upstream status and a Retry-After where you have one, so the console can tell a support engineer "this action is capped until 09:00" instead of showing a spinner forever. And test it with a fake clock: replay the same requestID twice, assert one reservation; walk the clock past the window boundary, assert the cap resets. Both tests take ten minutes to write and they're the only proof you have that the guard behaves at the boundary.
For the audit side, keep it dumb and scheduled — one nightly command that fails the build when any non-production key is older than the rotation window:
go run ./cmd/keyaudit -role console -max-age 90d -fail-on-stale
Where this stops being worth it
For a one-person project, all of this is overhead. One key, one balance, one person who notices — adopt the split when a second person can open the console, or when the console gains a write action that fans out.
The catch is that not every platform exposes per-key budgets or sub-account balances. If yours doesn't, you end up implementing the ceiling in a small proxy like the one above, and the trade-off is honest: you now own a component in the path of every console call, with its own failure modes and its own on-call story. Worth it when console spend is unpredictable. Hard to justify when the console does three read calls a day.
If you need a guarantee the provider enforces rather than one you enforce, stick with a separate prepaid sub-account per credential where that's offered, and accept the top-up overhead of funding two wallets instead of one. And if your admin console has no write path at all — read-only dashboards over a replica — a separate read-scoped key is still worth the ten minutes, but the spend ceiling probably isn't.
Top comments (0)