A gaming service that lets a prepaid balance run out unattended has an access-audit problem before it has a billing problem. Short answer: keep plan entitlements in a versioned control plane, read them at runtime through a cache with an explicit freshness policy, and record the exact entitlement version used for every decision; hardcode only immutable safety bounds.
I model the incident with a player wallet that has 500 credits, a monthly plan quota of 20 matches, and three game regions. A deploy changes the limit in one region, while an old worker keeps enforcing the compiled constant. The player sees inconsistent access, support cannot explain which rule fired, and the audit trail contains a request ID but no policy version. The failure is not that a number was wrong. The failure is that the decision was not reproducible. During review, I trace the request across the API gateway, the match allocator, and the wallet ledger: the gateway reads plan pro, the allocator receives a stale cache entry for standard, and the ledger correctly rejects a duplicate debit but has no context for why the allocator tried twice. A dashboard showing only 2xx and 5xx rates calls this healthy, even though players in one region are being denied after paying. The fix is not a larger cache or a clever retry loop. It is a contract that carries policy version and evaluation time through every hop, plus a replayable event that lets support answer the same question the code answered.
Audit first.
What should SaaS teams choose for runtime entitlements and hardcoded plan limits?
Use runtime data for commercial policy: plan limits, feature flags, rollover rules, grace periods, and suspension thresholds. Keep code-level guards for invariants that protect the service itself, such as a maximum batch size or an absolute wallet debit ceiling. This split makes a useful buy-versus-build boundary: a managed entitlement store can reduce on-call work, while a small self-hosted table may be preferable when audit data must stay inside a regulated environment. Neither option removes the need for a clear contract.
The runtime path should return a decision plus metadata, not just allowed: true. A practical response contains subject, plan, effective time, expiry, policy version, and reason code. Persist those fields with the debit or match-creation event. If a player disputes a charge six weeks later, an engineer can replay the decision without guessing which deploy was live.
Hardcoded limits still have a job. They are deterministic during a control-plane outage and useful as a last-resort ceiling, but they are a poor place for values that product staff change daily. A constant also spreads silently: API handlers, background workers, and client hints can each carry a different copy.
The incident path: stale reads, retries, and balance safety
The dangerous sequence is predictable. A worker reads an entitlement, pauses, retries after a timeout, and then applies a debit using a newer wallet balance but an older quota decision. Solve this with an idempotency key and a single authoritative write. The entitlement read is a snapshot; the debit command must say which snapshot it consumed.
For prepaid balances, fail-closed is usually safer for debits, but it is not universal. A read-only match history page can serve a cached answer, while a purchase or wallet deduction should stop when the policy is older than the declared freshness window. Your SLO should name that window, for example “99.9% of entitlement decisions use data no older than 60 seconds,” and your alert should measure age, not only request errors.
Here is a compact Go shape for the decision boundary. It uses generic HTTP and leaves storage implementation to the platform team.
package entitlement
import (
"context"
"fmt"
"time"
)
type Snapshot struct {
Plan string
Quota int
Version string
Evaluated time.Time
}
type Store interface {
Read(ctx context.Context, accountID string) (Snapshot, error)
}
func CanStartMatch(ctx context.Context, store Store, accountID string, used int, now time.Time) (string, error) {
s, err := store.Read(ctx, accountID)
if err != nil {
return "", fmt.Errorf("entitlement unavailable: %w", err)
}
if now.Sub(s.Evaluated) > time.Minute {
return "", fmt.Errorf("entitlement snapshot is stale")
}
if used >= s.Quota {
return s.Version, fmt.Errorf("quota exceeded for plan %s", s.Plan)
}
return s.Version, nil
}
The caller should attach the returned version to an append-only event and make retries reuse the same idempotency key. Do not put a fallback constant in the error branch unless it is an explicit, documented safety ceiling; silently allowing a debit after a failed read defeats the audit goal.
Measuring the control plane like an SRE
Capacity planning starts with decision volume, not account count. Estimate peak entitlement reads per second, cache hit ratio, policy-change fan-out, and the number of regions that must converge. A 10-minute cache may cut load but can violate a one-minute freshness SLO. Conversely, forcing every match request to hit a primary database creates a dependency whose latency becomes player-facing.
Track four signals: snapshot age at decision time, policy-version mismatch rate, denied requests by reason code, and replay success for sampled events. Add a synthetic account whose plan changes from 20 to 21 matches every hour; it catches propagation gaps without touching real wallets.
I initially treat cache invalidation as a performance detail, then the audit review changes the priority: invalidation is policy distribution. That means schema migrations, clock skew, and rollback behavior belong in the runbook. Your mileage may vary if plans change only quarterly, but the evidence should come from change frequency and dispute cost, not intuition.
When runtime entitlements are the wrong tool
The catch is operational complexity. Runtime evaluation is not suitable when the team cannot operate a highly available control plane, define ownership for policy edits, or retain decision records. A small product with one plan and rare changes may stick with a reviewed constant plus a migration process. A latency-critical game loop may keep a signed, periodically refreshed snapshot at the edge and reserve online checks for wallet mutations.
Avoid turning the entitlement service into a second billing system. Billing remains the source of payment status; entitlements translate that status into permissions and quotas. Keep the interface narrow, document precedence when plans overlap, and make every policy change attributable to a human or automation identity.
The durable rule is simple: code should enforce safety, while data should express policy. Runtime reads earn their cost when you need frequent changes, consistent behavior across workers, and an audit trail that can reproduce a decision. Hardcoded limits win when the policy is truly invariant or when an explicit degraded mode is more valuable than freshness.
Top comments (0)