The hard trade-off in a property-management experiment is freshness versus reconstructability: polling faster can shorten exposure to an unwanted flag state, but no polling interval can explain a past decision unless the application records which configuration generation, fallback path, and tenant cohort produced it. TL;DR: keep the last verified snapshot in process, define a conservative default for every flag, refresh with jitter, and emit a compact decision record beside the business event. Treat the remote flag service as a control plane, never as a synchronous dependency of a rent, maintenance, or tenant-notification request.
Evidence first.
Suppose a team is comparing a new maintenance-triage flow across tenant cohorts. A complaint arrives: tenants in one building stopped receiving the expected escalation path during an incident. The useful question is not merely "was the flag on?" It is "what did this process know, for this pseudonymous tenant, when it committed that action?" That distinction determines the cache, fallback, and telemetry design.
What must a tenant incident timeline prove?
A mutable dashboard shows present intent. Incident reconstruction needs historical evidence. Between evaluation and investigation, an operator may change the rule, a poll may replace the local snapshot, or a process may restart. Logging only enabled=true destroys those distinctions and makes a cohort experiment look deterministic when it was not.
The application should evaluate from one immutable in-memory snapshot and attach a decision envelope to the resulting business event. For a maintenance workflow, that event might be triage_assignment_created; its flag fields should identify the key, a non-secret variant, the configuration revision, the source (snapshot or default), snapshot age, and a correlation identifier. Record a pseudonymous cohort key rather than a tenant's name, address, email, lease text, or maintenance description. Data minimization and storage limitation are explicit principles in Article 5 of the GDPR, so an audit trail is not permission to copy personal data into logs.
This is an exactly-once mindset, not a promise of magical exactly-once delivery. Give the business command an idempotency key, persist the chosen outcome with the domain record, and make retries reuse that outcome. An observability event may be delivered more than once; consumers should deduplicate on the stable event identifier. The persisted business record remains authoritative.
How should Node.js feature flags combine caching and fallback defaults?
A fallback is a product and safety decision disguised as a Boolean. For new_triage_flow, defaulting to false might preserve the established assignment path when no verified snapshot exists. For a flag that disables outbound notices during a legal hold, the conservative value may be true. There is no universal fail-open or fail-closed rule; write the consequence of each state beside the flag definition and have the owning domain team approve it.
Use three states in the client rather than pretending the network returns only true or false:
- Verified snapshot: evaluate locally and record its revision and age.
- Stale snapshot: continue only until a flag-specific maximum age, while raising a freshness signal.
- No acceptable snapshot: use the code-owned default and record the fallback reason.
The maximum age should come from the harm window. As an illustrative policy, a cosmetic portal label could tolerate a 30-minute snapshot, while routing maintenance emergencies might require a five-minute limit. Those numbers are design inputs, not universal recommendations. Set them from rollback objectives, on-call response time, and the consequence of inconsistent treatment across tenants.
Polling then becomes straightforward. Start with a bounded interval, add per-process jitter to avoid synchronized refreshes, validate the complete response before swapping snapshots, and retain the previous verified snapshot after a timeout, authentication error, parse failure, or invalid configuration. Never partially update the map. A generation is accepted whole or rejected whole.
In Node.js, schedule refresh work outside request handlers and prevent overlapping polls. Timer callbacks can run later than their nominal delay when the event loop is busy, so measure observed snapshot age rather than assuming the configured interval equals freshness. Apply a request timeout shorter than the poll interval, use exponential backoff capped at a chosen ceiling after failures, and reset that backoff after a verified refresh. These production practices make caching behavior observable rather than assumed. Jitter matters more than a theoretically elegant cadence when hundreds of processes start together after a deployment.
Attach the audit record at the business commit
Per-evaluation debug logs are tempting and often noisy. A better boundary is the business effect: emit one structured decision record when the flag changes what the application commits, sends, or schedules. High-volume evaluations that produce no durable effect can be represented by counters, while the consequential event carries reconstruction fields.
| Field | Reconstruction question | Cardinality guardrail |
|---|---|---|
flag_key |
Which decision changed behavior? | Controlled registry |
variant |
Which non-secret branch ran? | Small fixed set |
config_revision |
Which immutable ruleset was used? | Do not use as a metric label |
evaluation_source |
Snapshot or code default? | Two-value enum |
snapshot_age_ms |
How stale was local knowledge? | Histogram, not an identifier |
cohort_key |
Which pseudonymous group was compared? | Log field, not a metric label |
reason |
Match, stale, unavailable, or invalid? | Controlled enum |
event_id |
Can duplicate telemetry be collapsed? | Log field |
Keep revision IDs, tenant-derived keys, and event IDs out of metric labels because their value sets grow without a useful bound. Metrics should answer fleet questions such as fallback rate, refresh failures, and snapshot-age distribution. Traces can connect an evaluation to the maintenance command, but baggage crosses service boundaries and may be propagated to downstream systems; the W3C specification warns that baggage can contain sensitive information and requires care around exposure. Put only deliberately safe, bounded context there.
Storage has an operational cost as well as a privacy cost. Per-GB log ingestion pricing makes indiscriminate evaluation logging an architecture concern, not a formatting detail. Sample routine diagnostics if necessary, but do not sample away the decision envelope attached to a durable business effect; instead, keep it compact and set retention from incident and compliance requirements.
Replay one tenant outcome before comparing cache strategies
The relevant comparison is not SDK convenience. It is what remains knowable after a disputed tenant outcome.
Retries happen.
| Strategy | Request dependency | Behavior during control-plane loss | Incident evidence |
|---|---|---|---|
| Synchronous remote evaluation | Remote call on the path | Latency or failure can reach the request | Remote history plus local correlation, if retained |
| In-process polling cache | Local read; remote refresh off-path | Verified snapshot, then explicit default after its age limit | Revision and source emitted locally |
| Push/streamed updates | Local read with an open update channel | Last verified state until policy expires it | Revision and disconnect interval |
| Deployment-time configuration | No runtime control plane | Stable until redeployment | Release artifact identifies state |
Polling is often the simplest fit for a small SaaS backend, but it pays for that simplicity with a bounded inconsistency window. Push narrows propagation delay but adds reconnect, ordering, and resynchronization cases. Synchronous evaluation centralizes policy at the price of coupling availability and latency. Deployment-time configuration is easy to audit but cannot support rapid cohort changes. Choose by the failure you are willing to own.
For cohort comparison, freeze the assignment inputs. A stable pseudonymous tenant identifier and an explicit experiment revision prevent a retry from drifting into another branch after percentages change. If a workflow spans several steps, persist the selected variant at the first durable boundary and carry that value forward; reevaluating midway can create a hybrid path that no cohort was intended to receive.
Migrate through a reconstruction drill
First, inventory flags that can alter durable tenant outcomes and give each one an owner, code default, acceptable snapshot age, and removal condition. Second, add the immutable snapshot and atomic replacement behavior, then test startup with no network, malformed refreshes, delayed timers, process restarts, and recovery after stale state.
Third, shadow the decision envelope without changing cohort behavior. Verify that an investigator can start from a maintenance event and recover the flag key, variant, revision, source, snapshot age, and stable event ID, while seeing no direct tenant identifiers. Fourth, enable a small cohort such as 5%, reconcile business outcomes against emitted decision records, and advance through predeclared stages only when fallback rate and snapshot age remain inside the team's limits.
Remove the flag after the experiment decision is complete. Long-lived flags accumulate branches that are difficult to test, and an obsolete fallback can quietly become policy. The durable outcome should be a simpler code path plus an audit record that still explains how the experiment was run.
Sources
- OpenFeature specification: https://openfeature.dev/specification/
- Node.js timers documentation: https://nodejs.org/api/timers.html
- W3C Baggage specification: https://www.w3.org/TR/baggage/
- GDPR, Article 5 principles: https://eur-lex.europa.eu/eli/reg/2016/679/oj
- Amazon CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/
Top comments (0)