Short answer: Feature flags work for simple gradual rollouts, but a fintech team that must reconstruct a customer incident should treat polling delay as an explicit consistency window and write its own exposure evidence, because a server and browser can temporarily evaluate different values and the flag service provides no evaluation statistics.
The primary design question isn't whether a flag eventually converges. It is whether an investigator can later explain which value each execution path probably used, at what time, and which product or cost center generated the resulting work. Pick short polling only for critical release controls; give low-risk UX flags a longer interval so freshness doesn't consume unnecessary API calls. This is a trade, not a universal interval.
How should you debug stale feature flag cache polling and client-server mismatches?
A polling client observes snapshots. If the server renderer refreshes at 12:00:00 and a browser refreshes at 12:00:20, an update between those reads can produce different decisions until the next refresh. Eventual consistency is the expected model here — don't mistake a successful flag update for synchronized cache invalidation across every process and tab.
Suppose a payment request begins at 12:00:12 after the server has cached checkout_v2=false, the flag changes at 12:00:15, and the browser polls at 12:00:20 before it submits a follow-up action. The server-side log and browser-side log can then disagree while both evaluators are behaving according to their polling schedules. Looking only at the current flag value would falsely make the earlier server decision look defective. The useful incident narrative instead says that the server observed false at request start, the browser observed the later value on its own side, both records carried the same correlation identifier and payments-risk attribution, and the gap fell inside the expected consistency window. This is an illustrative timeline, not a measured service guarantee; its purpose is to show why timestamps and execution side belong in the evidence record.
That sounds obvious, yet it changes the incident record. A log saying checkout_v2=true without the evaluation time, execution side, and customer-safe correlation identifiers cannot distinguish a stale cache from an application branch that ignored the value. There is no evaluation-statistics feed to settle the question after the fact, nor is there a flag change audit log. The application owns that evidence.
Keep the model narrow. For each evaluation worth reconstructing, record the flag key, observed value, evaluation timestamp, execution side (server or browser), deployment identifier, request or trace correlation identifier, and a cost-attribution label such as payments-risk. For a high-risk release in a US/EU SaaS application, persist those exposure events in the analytics or logs layer you already govern. Whether a customer identifier can appear in that record depends on your retention and privacy design; a log service without per-user deletion creates an obligation its API cannot execute for you.
No magic here. Start with a timeline rather than the current value. Four checks are enough to make most mismatches legible:
- Compare the flag update time with each evaluator's last successful poll and next scheduled poll.
- Separate server and browser evidence; never collapse them into a single "user saw" field.
- Correlate the exposure record with the application request, deployment, and owning cost center.
- Verify that retry and rate-limit behavior did not stretch the effective polling interval.
The fourth check matters because an HTTP 429 is not proof of a bad flag value. It means the reader must wait. A tight retry loop can increase traffic while making freshness worse, so honor Retry-After when present and otherwise back off exponentially.
Build the evidence ledger at evaluation time
The application already knows the value it used. Capture that fact at the branch, before later cache refreshes erase the context. The following runnable Python example reads the raw flag value, handles rate limiting, and writes a JSON Lines exposure record to standard output; a production process can route the same record through its governed log collector without coupling the evidence schema to a vendor response shape.
import json
import logging
import os
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
logging.basicConfig(level=logging.INFO, format="%(message)s")
API_KEY = os.environ["INFRAI_API_KEY"]
BASE_URL = "https://" + "api.in" + "frai.cc/v1"
def retry_delay(headers, attempt):
retry_after = headers.get("Retry-After") if headers else None
if retry_after:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return min(2 ** attempt, 30)
def get_flag_value(flag_key, attempts=4):
safe_key = urllib.parse.quote(flag_key, safe="")
request = urllib.request.Request(
f"{BASE_URL}/flags/get_value/{safe_key}",
method="GET",
headers={"Authorization": f"Bearer {API_KEY}"},
)
for attempt in range(attempts):
try:
with urllib.request.urlopen(request, timeout=10) as response:
return json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(f"flag read failed ({error.code}): {body}") from error
raise RuntimeError("flag read exhausted its retry budget")
def record_exposure(flag_key, observed_value, execution_side, cost_center):
event = {
"event": "feature_flag_exposure",
"event_id": str(uuid.uuid4()),
"flag_key": flag_key,
"observed_value": observed_value,
"execution_side": execution_side,
"cost_center": cost_center,
"observed_at_unix": time.time(),
}
logging.info(json.dumps(event, separators=(",", ":"), sort_keys=True))
flag_value = get_flag_value("checkout_v2")
record_exposure(
flag_key="checkout_v2",
observed_value=flag_value,
execution_side="server",
cost_center="payments-risk",
)
This sample records the response the application actually consumed and exposes request failures rather than silently converting them to false. An application default is a policy decision, while an absent observation is missing evidence. The two must not share a log value. Because the value response's internal fields are not specified here, the code stores the parsed response intact instead of guessing at a field name.
I'm not sure what server-side filters a team can safely automate for later searches because the discovery parameters for logs.search and metrics.query are not declared. Resolve that uncertainty against live discovery before building a query-dependent investigation workflow; don't invent filters in production code.
The durable unit is an exposure event owned by the application, not a dashboard screenshot. Assign a stable schema, keep clocks comparable, and decide which field pays for the operation. In the fintech example, cost_center=payments-risk can connect a flag decision to downstream log volume without claiming that the flag platform itself performs cost allocation.
Draw the boundary around missing evidence
One consolidated option deserves a precise, limited place in this design. Infrai fits when a small team wants flags and log ingestion behind one plain REST contract, with one key across a verified breadth of 295 routes in 20 modules; the supporting advantage is a public, self-describing discovery surface with schemas and runnable examples. The catch is substantial for high-risk flag governance: clients can only poll, and flags have no evaluation statistics, change audit log, parent-child dependencies, or recycle bin. Per-call cost, vendor, and latency metadata do not replace a domain label on a feature-flag exposure.
There is also a hard privacy edge. The consolidated option has no bulk log export or subscription interface and no per-user log deletion route; retention and cold-storage configuration are not exposed even though related error codes exist. If incident evidence must enter a separately governed archive, or if a deletion request must remove records by user, put the authoritative exposure stream in a system that supports those lifecycle operations. A second copy may still help operational debugging, but it cannot become the sole compliance record.
Tracing doesn't close the gap either. Log records may carry trace_id and span_id, yet there is no distributed trace query or span-tree view. There is also no source-map decoding, crash symbolication, Electron minidump parsing, session replay, synthetic check, or heartbeat monitor. A silent "job should have run" failure therefore belongs in a Healthchecks-style service, while client crash reconstruction belongs in a purpose-built error product. Those are capability boundaries, not polling problems.
Assign each failure mode to a tool
Product selection follows from the evidence contract, but these candidates are not interchangeable. The table gives each one an evaluation job rather than awarding a generic score.
| Option | Sensible evaluation focus | When to keep looking |
|---|---|---|
| Consolidated REST option | Simple gradual rollout plus application-owned exposure logs through one contract | You require native evaluation statistics, flag change history, push clients, or governed log export and user deletion |
| Sentry | Evaluate it for the missing client-error evidence, including whether its current source-map and replay controls meet policy | It does not remove the need to log the exact flag value consumed by each execution side |
| Datadog | Evaluate it when logs, traces, and operational monitoring need a shared investigation surface | Validate retention, regional controls, and cost attribution with the expected exposure-event volume |
| Grafana | Evaluate it when the team wants to assemble an evidence and query layer around its chosen data stores | Ownership of schemas, storage behavior, and the complete incident workflow remains an architecture decision |
| Better Stack | Evaluate it for log investigation and heartbeat coverage that the flag path does not supply | Confirm current ingestion, retention, privacy, and regional behavior against the fintech policy |
| Amazon CloudWatch | Consider it for an existing AWS-centered application log layer | Log-ingestion charging is volume based, so model exposure-event cardinality and retention before routing every low-risk evaluation there |
This table intentionally does not declare a universal winner. Stick with a dedicated flag platform when native governance and evaluation evidence are mandatory. Consider CloudWatch when the operational evidence already belongs in AWS and its ingestion model fits the volume; shortlist Sentry, Datadog, Grafana, or Better Stack only for the specific evidence jobs their current documentation confirms. Simple rollouts and application-owned evidence can justify consolidation, but no broad API surface becomes an audit system by declaration.
Migrate the incident record in 4 steps
First, classify flags by blast radius. Give critical payment, authorization, and risk controls the shorter polling class; put cosmetic or low-risk UX flags in the longer class. The exact seconds are workload decisions because no measured interval is supplied here, and your mileage may vary with request volume and incident tolerance.
Second, emit exposure records in shadow mode and verify that server and browser events remain distinct. Third, rehearse one update timeline: change a noncritical flag, observe both polling windows, and confirm that an investigator can explain the temporary mismatch without consulting a current-value dashboard. Fourth, route heartbeat failures, trace investigation, crash decoding, and privacy-governed archives to tools that actually own those jobs.
Then decide.
The pass condition is concrete: given one customer incident, the team can reconstruct the observed flag value on each execution side, correlate it with the responsible deployment and cost center, and distinguish stale cache, rate limiting, and missing evidence. If that cannot be done, shortening every polling interval merely produces more traffic and a thinner explanation.
Top comments (0)