Short answer: evaluate a locally cached, versioned feature-flag snapshot on the Node.js request path, default the new pricing rule off when no valid snapshot exists, and emit one idempotent decision record for every price calculation. The kill switch then remains usable during a control-plane outage, while the record preserves enough context to reconstruct who received which rule and why.
This is an architecture decision, not a boolean-management trick. A gaming backend can charge the wrong amount without crashing, so availability alone cannot establish correctness; the design has to couple safe rollout, fallback defaults, and incident reconstruction without putting a remote flag service in the pricing critical path.
What must a Node.js feature flag kill switch preserve during an outage?
Adopt a data-plane/control-plane split. The control plane distributes signed or otherwise authenticated flag snapshots; each Node.js process evaluates its last accepted snapshot locally. A snapshot carries a monotonically advancing version, an expiry time, rollout configuration, and the kill-switch state. The request path never asks the control plane for a live answer.
The invariants are stricter than “the endpoint returned 200.” For a given player, game, pricing input, flag version, and ruleset version, retries must produce the same decision. A newer snapshot may replace an older one, but an older snapshot must never overwrite a newer one. If no snapshot is valid, the fallback selects the established pricing rule. Every calculation receives a stable decision ID, and the audit event records both the selected branch and the inputs required to explain it later.
The price can be wrong while every host is green.
Exactly-once delivery is usually the wrong promise across a request handler, queue, and analytical store. The useful mindset is exactly-once effect: derive the audit event's idempotency key from the pricing operation, allow transport retries, and enforce uniqueness at the durable write boundary. Keep the monetary ledger authoritative; observability records explain a decision but don't replace settlement or reconciliation records.
The failure boundaries should also be explicit. A stale-but-valid snapshot permits local evaluation. An expired or absent snapshot activates the conservative default. A malformed replacement is rejected before publication to request handlers. An audit sink delay must not silently change the price branch; instead, the service uses a bounded durable handoff whose saturation is visible and whose operational policy has been agreed in advance. If compliance requires synchronous retention before a charge is acknowledged, that requirement changes the latency and availability budget and belongs in the decision record.
Reconstruct one price before debugging the service
Treat the kill switch as cached configuration with safety semantics, not as a remote dependency. The process should load an initial snapshot before becoming ready, replace snapshots atomically, and retain the last accepted version until its explicit expiry. During a control-plane outage, a valid cached snapshot continues to govern requests; after expiry, the new pricing rule is off. That fallback is asymmetric because an unobserved failure should not expand exposure to a new monetary behavior.
There is a catch: “off” is safe only if the old rule remains executable and its dependencies are healthy. A migration that deletes the old code path while leaving the flag in place has no real fallback. Keep both paths deployable through the rollback window, test both against the same fixture corpus, and make the default a named policy such as established_pricing, rather than a bare false whose meaning will be forgotten six months later.
Rollout bucketing must be deterministic. Hash a stable subject key with the flag key and rollout salt, then map it into a fixed range; don't use process-local randomness, because retries routed to another instance could cross branches. Also decide what the subject is. Player ID may be appropriate for a player-level price, while purchase ID is better when every retry of one transaction must remain pinned even if account attributes change.
I'm not sure there is a universal snapshot expiry that works for every gaming backend. The correct value follows from the maximum tolerable exposure window, control-plane recovery objective, and how long the established path can remain valid; a team should document those inputs, rehearse expiry, and alert before the deadline rather than inherit an arbitrary timeout from a library default.
Work backward from an illustrative disputed purchase, purchase-7f3a, rather than forward from a dashboard. The ledger says the operation charged 1,200 cents at 14:03:21 UTC; the decision event says process pricing-12 evaluated snapshot 1842, ruleset pricing-v2, subject player-91, and chose inside_rollout; the configuration history says snapshot 1842 raised exposure from 500 to 1,000 basis points after review; and the audit writer shows one durable event for the operation ID despite two delivery attempts. That chain answers four separate questions: what happened, which immutable inputs caused it, which configuration authorized it, and whether a retry changed the effect. Now remove one link at a time. Without the ruleset version, responders cannot reproduce the arithmetic after code changes. Without the snapshot version, they cannot distinguish a rollout change from request nondeterminism. Without the operation ID, a duplicate audit record looks like a duplicate charge. Without a recorded reason, an established-rule result could mean kill switch, expired snapshot, or cohort exclusion. This reconstruction exercise determines the event schema before implementation and exposes a common mistake: collecting more application logs cannot recover decision inputs that were never named.
Three failure shapes, one architecture choice
The central comparison is where evaluation happens and what survives loss of the flag control plane. Cost and developer convenience matter, but they come after correctness, reconstruction, and bounded failure behavior.
| Option | Control-plane outage behavior | Reconstruction quality | Main limitation | Suitable use |
|---|---|---|---|---|
| Remote evaluation per request | Depends on network timeout and client fallback | Strong only if the returned version and reason are persisted | Adds a network failure boundary to pricing | Non-monetary flags where centralized, current targeting matters more than request-path isolation |
| Local evaluation from versioned snapshots | Continues on the last valid snapshot, then applies an explicit default | Strong when the snapshot version, subject, reason, and ruleset are recorded | Snapshot distribution and expiry policy become your responsibility | Monetary or entitlement decisions that need deterministic retries and outage tolerance |
| Deployment configuration only | Remains fixed until the next deployment or configuration reload | Simple if deployment revisions are retained | Kill-switch response is tied to deployment operations and may be too coarse | Rarely changed operational modes with no progressive targeting requirement |
The local-snapshot option fits this pricing rollout because it preserves a working kill switch without synchronously depending on its control plane. It is not suitable when targeting must incorporate rapidly changing centralized attributes that cannot be replicated safely. In that case, remote evaluation may be the honest choice, provided the timeout, fallback, returned evaluation metadata, and degraded-mode behavior are tested as part of the product contract. Stick with deployment configuration when progressive rollout adds no value and operational simplicity wins.
Measure the choice rather than trusting the diagram. Google's four golden signals give a useful outer frame: latency, traffic, errors, and saturation. For this path, add domain signals that can reveal a semantically wrong success: decisions by flag version and branch, fallback activations by reason, snapshot age, rejected snapshot count, audit handoff saturation, price-delta distribution between old and candidate rules, and reconciliation mismatches. A 200 response with an unexpected price belongs in the incident model.
Put the audit boundary in executable code
The following evaluator is intentionally small. A Node.js service can implement the same contract, while a Go reference makes state, inputs, and outputs unambiguous: the caller supplies an immutable snapshot and stable operation ID, and receives both the chosen amount and the audit event that must cross a durable boundary before the operation is considered reconstructable.
package pricing
import (
"crypto/sha256"
"encoding/binary"
"errors"
"time"
)
type Snapshot struct {
Version uint64
ExpiresAt time.Time
KillSwitch bool
RolloutBasisPts uint16
Salt string
}
type Input struct {
OperationID string
PlayerID string
GameID string
BaseCents int64
}
type Decision struct {
OperationID string
SnapshotVersion uint64
RuleVersion string
Branch string
Reason string
AmountCents int64
EvaluatedAt time.Time
}
func Evaluate(in Input, snapshot *Snapshot, now time.Time) (Decision, error) {
if in.OperationID == "" || in.PlayerID == "" || in.GameID == "" {
return Decision{}, errors.New("stable identifiers are required")
}
if in.BaseCents < 0 {
return Decision{}, errors.New("base amount must not be negative")
}
decision := Decision{
OperationID: in.OperationID,
RuleVersion: "pricing-v1",
Branch: "established",
Reason: "fallback_no_valid_snapshot",
AmountCents: in.BaseCents,
EvaluatedAt: now.UTC(),
}
if snapshot == nil || !now.Before(snapshot.ExpiresAt) {
return decision, nil
}
decision.SnapshotVersion = snapshot.Version
if snapshot.KillSwitch {
decision.Reason = "kill_switch_active"
return decision, nil
}
if bucket(in.PlayerID, in.GameID, snapshot.Salt) >= snapshot.RolloutBasisPts {
decision.Reason = "outside_rollout"
return decision, nil
}
decision.RuleVersion = "pricing-v2"
decision.Branch = "candidate"
decision.Reason = "inside_rollout"
decision.AmountCents = applyCandidateRule(in.BaseCents)
return decision, nil
}
func bucket(playerID, gameID, salt string) uint16 {
sum := sha256.Sum256([]byte(playerID + "\x00" + gameID + "\x00" + salt))
return uint16(binary.BigEndian.Uint32(sum[:4]) % 10_000)
}
func applyCandidateRule(baseCents int64) int64 {
return baseCents
}
applyCandidateRule deliberately preserves the base amount in this example; inventing a commercial pricing formula would distract from the safety mechanism. In production, the ruleset version and normalized inputs must identify the real calculation. The returned OperationID should become the idempotency key at the audit writer, with a uniqueness constraint that turns a retry into the same durable effect rather than a second event.
Record decision metadata, not an uncontrolled copy of the request. Player identifiers may be personal data, retention may be constrained, and pricing inputs may contain data that should never reach a general-purpose log. Tokenize or hash identifiers according to the threat model, restrict access, define deletion and retention behavior, and keep the schema narrow enough that an incident responder can use it without exposing unrelated account data. Compliance scope varies by jurisdiction and contract, so legal and security owners must set the actual limits.
For high-volume reconstruction, an analytical store can index append-only decision events by operation ID, snapshot version, and time; ClickHouse is one documented example of analytical storage, not a requirement. The authoritative ledger remains separate, and a reconciliation job joins stable IDs to detect missing audit effects, duplicate attempts, or disagreement between the charged amount and recorded decision. Test the join before rollout. Seriously.
Prove the rollback before raising exposure
Start with shadow calculation: run the candidate rule without charging its result, compare its output with the established rule, and emit a decision record marked as shadow. Then advance a deterministic cohort through small, reviewed steps. Each change to rollout percentage is a configuration event with actor, timestamp, previous version, new version, and reason; otherwise an incident timeline can explain request behavior but not who changed exposure.
A rollback drill should cover more than toggling the flag. Block access to the control plane, verify that local evaluation continues, advance time past snapshot expiry in a test environment, confirm that the established rule wins, retry the same pricing operation on another instance, and verify that one durable audit effect remains. Then activate the kill switch and confirm that the candidate-decision rate reaches zero while request latency and audit saturation stay within the team's declared objectives. This is where a specific fallback_no_valid_snapshot reason earns its keep: responders can distinguish distribution failure from an intentional stop without reading application prose or guessing from a graph.
The final go/no-go rule is concise: ship only when both pricing branches remain executable, retries are deterministic, the default is named and tested, snapshot freshness is observable, the audit write is idempotent, and an operator can reconstruct one operation from ledger entry to flag version and ruleset. If any link is absent, lower exposure or stop the rollout.
Stop there.
Why remote evaluation was rejected
Remote evaluation per request is the rejected option for this system. Its valid use case is a low-consequence feature whose targeting depends on centralized attributes that change too quickly to distribute, and where a conservative client fallback is acceptable. For a new gaming price, however, a timeout occurring between an initial attempt and its retry can select different branches unless every response is pinned and persisted; local deterministic evaluation removes that network race from the monetary path.
A kill switch without evidence is only a hopeful button.
Top comments (0)