Treat refused launch traffic as an admission-control diagnosis: check the tenant's key state and your own spend ledger before asking whether an external balance caused the refusal. The deciding constraint is the trade-off between a hard spend ceiling and refused gameplay traffic. During a growth spike, opening the gate on ambiguous evidence can turn a bounded incident into unbounded usage.
Short answer: capture one refused request's tenant ID, key fingerprint, decision code, ledger timestamp, and upstream correlation ID. If the local ledger says the ceiling was reached, preserve the refusal. If it says capacity remained, trace the upstream result. If the evidence is stale or missing, classify the cause as unknown rather than guessing "balance."
This architecture decision record covers a gaming backend that issues and revokes one scoped key per tenant. It optimizes for a fast, defensible answer without putting raw credentials in logs.
What must remain true during the spike?
A scoped key is an authorization boundary, not a convenient tenant label. A request admitted for tenant guild-417 must be charged to that tenant, evaluated against that tenant's ceiling, and rejected after that tenant's key is revoked. No fallback key may silently broaden the scope.
The diagnostic path has four invariants:
- Every decision has a stable, non-secret key fingerprint and tenant ID.
- Revocation wins over remaining budget.
- A spend decision records the ledger version and observation time used to make it.
- An upstream refusal stays distinct from a local refusal, even when both surface as a failed game action.
That last distinction carries the investigation. A single client-facing status may be desirable to avoid leaking account state, but internal reason codes must remain specific. I use a small vocabulary: key_revoked, scope_denied, ceiling_reached, upstream_refused, and evidence_stale. These are operational categories, not messages sent to players.
Keep secrets out of the trail. The OWASP Secrets Management Cheat Sheet recommends least privilege, rotation, revocation, expiration, and auditing around secret use. A one-way fingerprint gives responders a join key without turning an observability system into another credential store.
How should I check refused traffic during a launch spend cap?
Start at the edge that made the decision. Do not begin with an account dashboard screenshot: it may describe a different scope, a later moment, or a different credential. The fastest trustworthy check follows one request across boundaries.
No guesswork.
| Evidence at refusal time | Classification | Immediate action | Why |
|---|---|---|---|
| Key revoked or requested scope absent | Local key policy | Keep traffic refused; verify the issuer event | Spend state cannot authorize a forbidden key |
| Fresh ledger at or above the tenant ceiling | Local spend ceiling | Keep the ceiling closed; use an approved override if policy permits | The bound is doing its job |
| Fresh ledger below the ceiling, upstream explicitly refuses | Upstream account state | Trace the correlation ID and verify state at that boundary | Local capacity existed |
| Timeout, transport failure, or no authoritative refusal | Neither is proven | Retry only under the operation's retry policy; inspect dependency health | No response is not evidence of balance |
| Ledger or key-state evidence is stale | Unknown | Apply the tenant's predeclared risk mode and refresh evidence | Guessing can violate availability or the ceiling |
Freshness is a policy input. Define it before launch in terms the system can enforce, then include the observed timestamp in the decision record. There is no universal safe duration: a tenant with a strict promotional budget and one running a low-stakes test server may choose different refusal behavior.
Be careful with retries. An OTP flow exposes a useful rule: a timeout does not prove that the remote side did no work. Retrying a chargeable, non-idempotent operation because no response arrived can duplicate work and consume the ceiling faster. The retry decision needs an idempotency key and a known operation contract; otherwise, investigate first.
Critical path in Python
The admission function should return a decision object, not throw every refusal into the same exception bucket. This example keeps adapters generic and makes evidence age visible. Its policy values are injected configuration, not universal constants.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Protocol
class Reason(str, Enum):
ALLOW = "allow"
KEY_REVOKED = "key_revoked"
SCOPE_DENIED = "scope_denied"
CEILING_REACHED = "ceiling_reached"
EVIDENCE_STALE = "evidence_stale"
@dataclass(frozen=True)
class KeyRecord:
tenant_id: str
fingerprint: str
scopes: frozenset[str]
revoked: bool
@dataclass(frozen=True)
class SpendSnapshot:
tenant_id: str
committed_units: int
reserved_units: int
ceiling_units: int
observed_at: datetime
version: str
@dataclass(frozen=True)
class Admission:
allowed: bool
reason: Reason
tenant_id: str
key_fingerprint: str
ledger_version: str | None
class Ledger(Protocol):
def read(self, tenant_id: str) -> SpendSnapshot: ...
def admit(
key: KeyRecord,
required_scope: str,
requested_units: int,
ledger: Ledger,
now: datetime,
max_evidence_age: timedelta,
) -> Admission:
if key.revoked:
return Admission(False, Reason.KEY_REVOKED, key.tenant_id, key.fingerprint, None)
if required_scope not in key.scopes:
return Admission(False, Reason.SCOPE_DENIED, key.tenant_id, key.fingerprint, None)
snapshot = ledger.read(key.tenant_id)
if now - snapshot.observed_at > max_evidence_age:
return Admission(
False, Reason.EVIDENCE_STALE, key.tenant_id, key.fingerprint, snapshot.version
)
projected = snapshot.committed_units + snapshot.reserved_units + requested_units
if projected > snapshot.ceiling_units:
return Admission(
False, Reason.CEILING_REACHED, key.tenant_id, key.fingerprint, snapshot.version
)
return Admission(True, Reason.ALLOW, key.tenant_id, key.fingerprint, snapshot.version)
now = datetime.now(timezone.utc)
The interesting line includes reservations. Without them, ten concurrent match-start requests can each read the same apparent headroom and all pass independently. Admission and reservation therefore need one consistency boundary: a transaction, compare-and-swap on the ledger version, or another atomic mechanism supplied by the datastore. The plain read above exposes policy order; the production adapter must make successful admission and reservation atomic.
Tiny detail, large consequence.
Log the reason, fingerprint, tenant, ledger version, evidence age, requested units, and correlation ID. Do not log the key. Metrics should split local policy refusals from upstream refusals and transport failures, because an aggregate "request failed" chart cannot answer the launch question.
Failure boundaries and the recovery rule
The issuer owns key creation, scope assignment, rotation, and revocation. The admission service owns key-policy evaluation and spend reservation. The upstream adapter owns translation of an accepted local request into the dependency's contract. Observability joins these boundaries but does not decide policy. This ownership map also prevents a tempting launch-day mistake: changing a key because the ledger is stale, or raising a ceiling because an upstream call timed out. Neither action repairs the boundary that actually failed. I choose refusal when evidence is stale for a strict-budget tenant because the spend promise is explicit; for an availability-first tenant, I would permit only the bounded uncertainty that its written policy already allows.
This makes the first five minutes mechanical. Pick a refused request, find its admission record, and stop if the reason is local. If admission allowed it, use the correlation ID to inspect the upstream classification. If the adapter has only a timeout or malformed response, the honest answer is neither: spend ceiling and balance are both unproven.
Stop there.
Recovery follows the evidence. A ceiling override should be explicit, scoped to one tenant, bounded in duration or units, and attributable to an approver. A revoked key is replaced through issuance and rotation, never re-enabled as a side effect of a budget change. An upstream account refusal belongs to the account boundary. A dependency outage belongs to resilience handling. Combining these actions into a generic "unblock launch" switch creates an audit problem and makes the next diagnosis slower.
Exercise revoked-key, missing-scope, exact-ceiling, over-ceiling, stale-ledger, concurrent-reservation, upstream-refusal, and timeout cases before the event. Roll out reason-code changes compatibly so older consumers do not collapse an unfamiliar value into allow. During launch, watch refusal counts by reason and tenant, reservation conflicts, evidence age, and reconciliation lag.
Rejected option and where it still fits
The rejected design is a single shared launch key plus one global spend switch. It looks quicker because responders have one credential and one number to inspect. It cannot satisfy this system's central requirement: issue and revoke a scoped key per tenant. A compromised or revoked tenant cannot be isolated cleanly, attribution becomes weaker, and one tenant can consume capacity intended for the rest.
A shared key can still fit a tightly bounded, single-tenant internal tool where every caller has the same authorization scope and cost owner. Even there, rotation, revocation, access control, and auditability remain necessary. It is the wrong shortcut for a multi-tenant gaming launch.
Decision: keep per-tenant scoped keys, perform local key and spend admission before the upstream call, reserve spend atomically, and record boundary-specific reasons. This design may refuse some traffic when evidence is stale. That is deliberate: each tenant chooses in advance whether uncertain evidence protects the ceiling or favors availability, and operators never reinterpret an unknown as a confirmed balance failure.
References
- OWASP, Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
Top comments (0)