Spend Ceilings for Automated API Account Provisioning — Auto-Recharge as Admission Control

#api #billing #architecture #observability

The rule I use is simple: an automated provisioning path may attach a default payment method to a new API account only when the same transaction writes a hard spend ceiling for that account, and only when something in the request path can refuse traffic once the ceiling is reached. A stored card is the mechanical prerequisite for auto-recharge. It is not a substitute for a limit. Skip the limit and the setup quietly deletes the one backstop the system already had — a declined charge — and replaces it with unlimited willingness to pay.

Relying on a decline is a terrible control. It is also the control most teams are unknowingly running on.

Take a concrete system. An essay-feedback service sells to school districts; submissions arrive in a four-hour window after the school day ends, a single batch workload performs every model call, and the invoice for those calls shows up about three weeks later. The finance requirement is narrow: cap what that one workload may spend before the invoice arrives. The engineering requirement underneath it is narrower still — when the cap is reached, which requests get refused, and what does that refusal look like to the caller? Auto-recharge is the mechanism that makes this urgent, because a top-up turns a hard stop into a silent purchase.

Money is a resource you can run out of on purpose

Capacity planning already has the vocabulary for this. The standard answer to "more load than we can serve" is to shed the excess deliberately rather than queue it until everything degrades, and the Google SRE material on handling overload is still the clearest write-up of why graceful refusal beats indiscriminate slowdown. Spend is the same kind of resource with a slower feedback loop. The loop is so slow — weeks, not milliseconds — that you cannot discover the overshoot from the signal itself.

So the ceiling has to be enforced by a counter you own, on the critical path, before the chargeable call goes out. Four invariants make that enforcement auditable:

Every chargeable call maps to exactly one ceiling; a call that cannot be attributed is refused, not billed to a default bucket.
The ceiling is checked before the spend commits, not reconciled afterwards.
Refusal is a first-class response with a machine-readable reason, not a timeout.
An account without a ceiling cannot exist, because provisioning creates both or neither.

The counter that enforces a ceiling is not telemetry, and treating it as telemetry is where cost engineering usually goes wrong. I keep exactly three labels on it: account id, workload id, and billing period. Five hundred district accounts times six workloads times one open period is three thousand active series, which is nothing. Add model name, region, and request id "for debugging" and the same counter becomes a cardinality problem that happens to hold your spend limit; at that point the storage bill for the guardrail starts competing with the bill it guards. The retention math points the same way. Keep the ledger entries for 13 months, because invoice disputes span a fiscal year plus a month, and keep the raw per-request logs for 7 days, because after a week they only answer questions the ledger already answers.

Sampling is the sharpest version of this trade-off. Sample traces of these requests at 1% and nobody suffers. Sample the ledger at any rate and the ceiling becomes an estimate, which means the refusal threshold inherits the sampling error — a 1% sample of a $200 ceiling is a $200 ceiling plus or minus a lot. Full fidelity on the counter, aggressive sampling on everything derived from it, is the split that keeps both the guardrail honest and the observability bill flat.

Should a default payment method be a prerequisite for automated account provisioning?

Mechanically, yes, and there is no way around it: a prepaid-credit model can only top itself up if a reusable payment instrument is already on file, so auto-recharge and default payment method setup are the same feature viewed from two sides. As a control, no. The card answers "can this account pay?" and the ceiling answers "may this workload spend?" — different questions, different owners, and only the second one belongs in your code.

Two constraints shape how the automation touches any of this.

Card data must never enter the provisioning service. The account-creation call carries a token minted by a hosted form or a processor-side element, so your service holds a reference rather than a primary account number, which is what keeps the provisioning path out of the expensive parts of PCI DSS scope. The second constraint is that the credential able to attach payment methods is now the highest-value secret in the system, well above the runtime key that merely spends money inside a ceiling. The OWASP guidance on secrets management is the baseline here: separate credential, short lifetime, pulled from a secret store at start-up, never in an image layer or a CI log. Provisioning credentials and workload credentials should not even be the same kind of object.

Four ways to bound one workload's spend

The options differ mainly in who does the refusing, and how wide the damage spreads when the ceiling is wrong.

Option	What refuses	Blast radius when it triggers	Reasonable when
Prepaid credit, no default payment method	The provider, at zero balance	Every workload on the account stops together	The cap is legally fixed and overspend is unrecoverable
Default payment method plus auto-recharge and alerts	Nothing; alerts arrive after the money is spent	Unbounded until a human reads the alert	Spend is small relative to the cost of a false refusal
Default payment method plus a provider-side hard limit	The provider, at the limit	That account stops; refusal semantics are the provider's	You accept vendor-defined error shapes on the client path
Per-workload sub-account with a ceiling enforced in your own admission path	Your gateway, before the chargeable call	One workload degrades, siblings keep serving	Multiple workloads share one billing relationship

For the district service I would take the fourth row and keep the third as a backstop, which is ordinary defence in depth: your counter refuses first because it knows about workloads, and the provider limit exists to catch arithmetic errors in your counter. Products in the gateway family, Kong Gateway and Tyk among them, will refuse on quota, but their native counters measure requests and time windows rather than currency, so a currency ceiling still needs a mapping layer that you own and maintain. Metering platforms such as OpenMeter are organised around usage events, which fits the accounting side well and leaves the admission decision where it has to be anyway — in the request path.

The critical path: reserve, then spend

Provisioning first. The ceiling and the payment token travel in one request, because an account that exists without a ceiling is the failure mode the whole design is meant to prevent.

set -euo pipefail

BUDGET_API="https://budget.svc.internal/v1"
: "${PROVISIONER_TOKEN:?fetch from the secret store at start-up}"

curl --fail-with-body --silent --show-error \
  --request POST "$BUDGET_API/accounts" \
  --header "Authorization: Bearer ${PROVISIONER_TOKEN}" \
  --header "Idempotency-Key: provision-essay-feedback-2026-09" \
  --header "Content-Type: application/json" \
  --data '{
    "workload": "essay-feedback",
    "payment_method_token": "pmt_from_hosted_form",
    "ceiling_minor_units": 20000,
    "currency": "USD",
    "period": "monthly",
    "on_exceed": "refuse"
  }'

Then the per-request decision. The workload asks for a reservation before it spends, and the reservation response is the admission verdict; settlement with the real cost happens after the provider replies.

status=$(curl --silent --output /tmp/reservation.json --write-out '%{http_code}' \
  --request POST "$BUDGET_API/reservations" \
  --header "Authorization: Bearer ${WORKLOAD_TOKEN}" \
  --header "Idempotency-Key: ${SUBMISSION_ID}" \
  --header "Content-Type: application/json" \
  --data "{\"workload\":\"essay-feedback\",\"estimate_minor_units\":4}")

case "$status" in
  201) ;;
  409) echo "period ceiling reached; defer submission ${SUBMISSION_ID}" >&2; exit 75 ;;
  429) sleep "${RETRY_AFTER:-5}"; exit 75 ;;
  *)   echo "budget service answered ${status}; failing closed" >&2; exit 1 ;;
esac

Both calls carry an idempotency key, which matters more here than in most write paths: a retried provisioning call must not create a second account, and a retried reservation must not consume the ceiling twice. The header has an IETF Internet-Draft behind it rather than a finished RFC, so treat the name as a convention with wide industry adoption rather than a standard you can cite in an audit.

The refusal status code deserves a paragraph of its own, because the obvious choice is wrong. HTTP 402 reads as the natural fit, and RFC 9110 explicitly reserves it for future use, which means no client library, proxy, or retry policy agrees on what it does. I'm not convinced any single code is right for all callers. A queue-backed batch worker wants 429 with Retry-After, defined in RFC 6585 and RFC 9110 respectively, so its existing backoff logic does the work; an interactive caller is better served by 409 or 403 with a typed error body naming the ceiling and the period reset time. Pick one shape, document it, and keep the reason machine-readable — the caller has to distinguish "you are over budget" from "you are over rate" to react correctly, and a bare 5-something tells it neither.

The option I rejected, and when it is the right one

I rejected the prepaid-only design — no default payment method at all, hard stop at zero balance — even though it is the strictest possible spend ceiling and the easiest to audit. For an edtech workload the refusal lands on real student submissions in the middle of a four-hour window, support absorbs the fallout, and the team pays in trust rather than dollars. The trade-off is explicit: the fourth-row design accepts a bounded overshoot, roughly one in-flight batch of reservations, in exchange for never refusing traffic the organisation was happy to pay for.

Prepaid-only is still the correct answer in three situations I would not argue against. A grant-funded or contractually capped project where overspend cannot be recovered. A preview or ephemeral test environment, where a hard stop is a feature and a $5 balance is the whole policy. Any account whose credentials are exposed to a large number of people, because a stolen key with auto-recharge attached is a spend incident rather than an access incident.

There are two honest costs to the design I picked. It adds a dependency to the critical path, so you must decide in advance whether an unreachable budget service fails open or closed — I fail closed for batch work and open for interactive traffic, but that choice belongs to whoever owns the error budget. And per-call cost estimates drift, because token-priced calls are not knowable in advance, which means reservations over-reserve and settlement has to return the difference; if your estimate is systematically low, the ceiling leaks. Your mileage may vary with the workload's cost variance. Stick with a provider-side limit alone when one workload owns the entire billing relationship, when spend is flat month over month, and when nobody needs per-workload attribution — at that point your own reservation service is infrastructure that earns nothing.

DEV Community