Short answer: when API calls are suddenly refused, treat the event as a classification problem before changing a limit. For an edtech platform that issues one scoped key per tenant, the least complex reliable design is a Postgres credential ledger, a separate usage ledger, and a refusal record that preserves the provider's status, machine-readable error code, request identifier, and relevant headers. A budget cap follows accumulated spend over a billing window; a quota problem follows a count, rate, concurrency, or resource ceiling. The same user-visible symptom can represent either one, so the response body and local evidence tell you which path to take.
Start with the bill. For AI-assisted grading or tutoring, the dominant variable is usually metered work per successful request multiplied by request volume; key count and ledger rows are control-plane details, not the main consumption term. The useful change is therefore to cap and attribute work by tenant before it reaches the upstream service, while keeping enough refusal evidence to distinguish money from capacity. Do not start by rotating keys. Rotation may erase the correlation trail, will not replenish an exhausted account-level pool, and can widen the credential blast radius if an emergency key is shared.
How can I tell whether API calls were suddenly refused by a budget cap?
Read the response as evidence, not prose. Capture the HTTP status, a stable error code if one exists, the upstream request ID, retry guidance, limit headers, tenant ID, credential ID, and the usage-window snapshot that your own system consulted. Redact the secret itself. OWASP's Secrets Management Cheat Sheet recommends restricting who can access secrets, recording secret-management events, and designing rotation and revocation into the lifecycle; those controls also make refusal diagnosis possible without spraying credentials through logs.
A 429 is strong evidence of rate limiting, but it is not a universal proof of one particular quota dimension. A service can enforce requests per interval, concurrent work, tokens or bytes per interval, and longer-window allocation. A payment-related status or an explicit billing error points toward spend control. A 401 or 403 shifts the investigation toward invalid, expired, revoked, or insufficiently scoped credentials. The machine-readable code and published contract outrank assumptions based on status alone.
Status is not diagnosis.
Keep the classifier narrow:
| Evidence | Working classification | Immediate action | Why the blast radius matters |
|---|---|---|---|
| Explicit billing or spending-limit code, with the local budget exhausted | Budget stop | Pause chargeable work for that tenant and reconcile usage | A tenant key lets one tenant stop without freezing every classroom |
| Explicit rate, concurrency, or resource-limit code | Quota pressure | Honor retry guidance, reduce concurrency, or queue within the request deadline | A shared key can turn one noisy tenant into a platform-wide outage |
| Invalid, expired, or revoked credential code | Credential lifecycle | Confirm key state and deployment version; replace only the affected key | Scoped revocation contains exposure and avoids fleet-wide rotation |
| Ambiguous refusal with missing structured evidence | Unknown | Fail closed for chargeable work, preserve the response, and escalate against the contract | Guessing may retry a hard stop until every tenant is throttled |
Unknown is a real state. Keep it.
Make spend and quota separate data
A budget is a policy expressed in currency or another internal unit over a defined period. A quota is a capacity boundary expressed as requests, concurrent jobs, tokens, bytes, stored objects, or another resource. They may be correlated, but combining them into one limit_exceeded boolean destroys the information needed for a safe response.
The ledger should model four things independently: tenant budget policy; observed billable usage; quota observations from response metadata or configuration; and credential lifecycle. Store a provider request identifier alongside each chargeable operation so reconciliation can detect duplicates. Use an idempotency key for the local reservation path, because client retries and worker redelivery otherwise inflate your own accounting even when the upstream system charged once, or undercount when the reverse occurs.
Here is a deliberately small classifier. Its inputs are normalized at the integration boundary, so provider-specific strings do not leak into tenant policy:
from dataclasses import dataclass
from enum import Enum
class RefusalKind(str, Enum):
BUDGET = "budget"
QUOTA = "quota"
CREDENTIAL = "credential"
UNKNOWN = "unknown"
@dataclass(frozen=True)
class Refusal:
status: int
code: str | None
retry_after_seconds: int | None
BUDGET_CODES = {"billing_limit", "spend_limit"}
QUOTA_CODES = {"rate_limit", "concurrency_limit", "resource_limit"}
CREDENTIAL_CODES = {"invalid_key", "expired_key", "revoked_key", "insufficient_scope"}
def classify(refusal: Refusal) -> RefusalKind:
code = refusal.code or ""
if code in BUDGET_CODES:
return RefusalKind.BUDGET
if code in QUOTA_CODES:
return RefusalKind.QUOTA
if code in CREDENTIAL_CODES:
return RefusalKind.CREDENTIAL
return RefusalKind.UNKNOWN
Those example codes are an internal vocabulary, not claims about any external API. The adapter must map only documented upstream codes into it. Never infer BUDGET merely because the tenant is near its local cap: two independent limits can be reached at nearly the same time.
For an edtech workload, reserve estimated usage in a transaction before dispatch, then settle the reservation from the completed operation. The transaction locks the tenant's current budget window, rejects work that would exceed policy, and creates an immutable attempt row. A stale reservation needs an explicit expiry and reconciliation state; deleting it loses the reason an assignment was blocked.
One credential per tenant changes the failure boundary
The credential record should contain an opaque internal ID, tenant ID, encrypted-secret reference, scopes, state, creation time, rotation deadline, and revocation time. The application does not need the plaintext key in Postgres if a dedicated secret store supplies it at dispatch. Database uniqueness should prevent two active records from being mistaken for the same generation, while deployment logic permits an overlap window during rotation.
Scope is not decoration. A tutoring tenant that only submits inference jobs should not receive administration or billing privileges. If its credential leaks, the maximum reachable data and operations should be bounded by that tenant and those operations. A single account-wide credential makes operational attribution cheaper to build, but its blast radius is the whole platform and revocation interrupts unrelated schools. That is a poor exchange unless the upstream contract cannot support narrower credentials.
Revocation has two clocks: how quickly the control plane marks a credential unusable and how long cached copies can still authorize work. Test both. A database row changing to revoked is not proof that workers, queues, sidecars, or the upstream service have stopped accepting the material.
Measure both clocks.
Test the hard edges, not the happy path
Create deterministic contract fixtures for at least four refusals: budget, short-window rate, concurrency, and revoked credential. Verify classification from structured fields, then verify behavior. Budget stops must not enter automatic retry loops. Rate failures may retry only when the operation is idempotent, retry guidance permits it, and the remaining deadline can accommodate the delay. Credential failures should open an operational alert and stop that credential; blindly rotating on every authorization error can conceal a bad deployment or incorrect scope.
There is a nasty boundary at billing-window rollover. Two workers can both observe remaining budget and both reserve it unless reservation is serialized or enforced with an atomic constraint. Clock disagreement makes the edge worse. Choose one authoritative time source for the ledger window, record it, and test requests immediately before and after rollover.
Deployment deserves the same skepticism. Roll out the classifier in shadow mode first: record the proposed category while existing behavior remains unchanged, then compare it with manually resolved cases. No invented benchmark is needed. The acceptance condition is that every production category traces to documented evidence and every unknown remains visible rather than being coerced into the most convenient bucket.
Operationally, graph refusals by normalized kind, tenant, credential generation, endpoint class, and time window. Alert on changes in unknowns and credential errors, not merely on the total refusal count. Do not put raw keys, full authorization headers, student prompts, or response bodies containing student data into metrics labels or logs. High-cardinality request IDs belong in traces or searchable event storage with controlled retention.
Retention is part of the cost model
The useful storage equation is event volume multiplied by retained bytes per event multiplied by retention time, plus index and replica overhead. Measure each term in your environment. A verbose response body retained forever will dominate a compact credential ledger long before the row count itself becomes interesting, while indexes on tenant, request ID, and event time add write and storage cost that must be justified by an actual investigation path.
Keep immutable security events and financial reconciliation records according to the organization's legal and accounting requirements; no universal duration can be asserted for every school, jurisdiction, or contract. Keep bulky diagnostic payloads for a shorter, explicitly approved window, with sensitive fields removed at ingestion. Aggregate older rate-limit telemetry when per-request detail no longer changes an operational decision.
What should you deliberately stop keeping? Raw secrets should never enter the evidence store, and full upstream bodies should not survive merely because they were once useful during debugging. Dropping them reduces exposure and the dominant retained-byte term. The cost is real: an old, ambiguous refusal may no longer be reconstructable byte for byte, so retain the normalized code, status, request ID, timestamps, credential generation, usage reservation, and a schema version. That compact record is the minimum defensible trail.
The decision rule is plain: classify from explicit evidence, apply tenant-local policy, and keep unknowns out of automated retry. A scoped credential contains the incident; a separate spend ledger explains the money; quota telemetry explains capacity. Mixing the three makes the first refusal easy to handle and the hundredth impossible to audit.
This design has limitations. Per-tenant keys increase secret inventory, rotation work, and audit volume, while synchronous Postgres reservations add a database dependency to the dispatch path. It is not appropriate for a tiny, trusted, single-tenant service whose upstream contract exposes only one account-wide credential; there, a single key plus strict internal tenant accounting may be the honest boundary. The trade-off changes once independent revocation and tenant-level containment matter more than the extra control-plane work.
Top comments (0)