Short answer: treat every feature-flag evaluation used by a pricing rule as versioned evidence, then page only when the server makes a consequential decision from a snapshot older than its declared freshness budget; a client-server mismatch by itself belongs in diagnostics, not in the pager.
That distinction matters during a fintech rollout. A browser may display the new offer while the server still charges under the old rule, or the reverse, because the two processes fetched configuration at different times. The frightening symptom is "the flag disagreed." The useful incident question is narrower: which evaluator decided the amount, which configuration version did it observe, how old was that snapshot, and was that state allowed by the rollout contract?
I've carried a pager that fired for differences that meant nothing and stayed quiet for the decision that mattered. I don't trust a green dashboard after that. I ask what page fired.
How should feature flags expose stale cache, polling interval, and client-server mismatch?
Expose a small decision record at the point where the pricing decision becomes authoritative. It needs the flag key, evaluated variant, configuration version, snapshot fetch time, evaluation time, subject identifier in a privacy-safe form, and the component that evaluated it. For a quote or charge, attach the decision record to the same trace or structured event as the pricing-rule result. Do not infer it later from an aggregate dashboard; by then, the causal join is guesswork.
The key metric is snapshot age at evaluation, not merely whether the latest poll succeeded. Define it as evaluation_time - snapshot_fetched_at. Compare that age with an explicit freshness budget derived from the business action. A page-view experiment can often tolerate convergence. A server-side price commitment may not. The polling interval contributes to the possible age, but scheduler delay, network delay, backoff, and process suspension can extend it, so "we poll every 30 seconds" is not evidence that every decision used configuration less than 30 seconds old.
Record versions on both sides, but don't turn every difference into an incident. If the browser renders version pricing-184 at 02:14:07 and the quote service evaluates pricing-183 at 02:14:08, the mismatch establishes divergence; it does not establish an incorrect charge. The server's decision contract decides that. An illustrative event should read like this:
{
"event": "pricing_rule_evaluated",
"component": "quote-service",
"flag_key": "new_pricing_rule",
"variant": "control",
"config_version": "pricing-183",
"snapshot_fetched_at": "2026-08-16T02:13:31Z",
"evaluated_at": "2026-08-16T02:14:08Z",
"snapshot_age_ms": 37000,
"freshness_budget_ms": 45000,
"decision": "quote-v1"
}
Those values are example data, not a universal service-level objective. Your mileage may vary, especially when mobile clients can sleep for hours. The invariant is that a responder can compare observed age with a declared budget without reconstructing polling behavior from log lines.
Reconstruct the incident as two clocks, not one timeline
The postmortem should separate the control-plane clock from the request clock. On the control-plane clock, an operator changes the pricing rule, the configuration system publishes a new version, and each evaluator obtains it. On the request clock, a client renders an offer, a server calculates a quote, and a payment path commits an amount. Eventual consistency connects those clocks, but it does not order them for you.
Start with the authoritative money event and work backward. Find its server evaluation record. Check the variant and version, calculate the snapshot age from source timestamps, and compare the result with the freshness budget. Then find the client evaluation by correlation ID and compare versions. Only after that should you inspect poll outcomes, reconnects, or process lifecycle. This ordering prevents a common failure mode: spending an hour proving that two caches differed while never proving that the difference affected a price.
I initially want a single convergence chart because it's quick. Then I remember what it hides: one low-volume service can make the only consequential stale decision while the fleet percentile remains healthy. A histogram of cache age is useful for capacity and trend work; the incident needs the exact decision record. Keep both.
No mystery remains once the clocks are visible.
The alert should also preserve the distinction between expected lag and violated safety. A diagnostic counter can track client_version != server_version. A ticket can track evaluators that repeatedly approach their freshness budget. The page should fire when the authoritative pricing path either evaluates beyond budget or cannot prove freshness and the policy says proof is required. This keeps signal quality ahead of noise: responders wake for exposure, not for the normal mechanics of convergence.
Make the authoritative path reject unprovable freshness
For a pricing rollout, the safest implementation is usually server authority plus a bounded cache. The client flag may choose presentation, but it must not establish the amount. The server evaluates from an immutable snapshot and checks freshness before applying the new rule. If freshness cannot be proved, the application follows a predeclared business policy, such as retaining the established pricing rule and emitting a high-signal decision event.
This Go example models that boundary. The evaluator doesn't promise immediate consistency. It promises that a new pricing rule is never selected from a snapshot older than the accepted budget, and it returns enough evidence for the caller to log beside the quote.
package pricing
import (
"errors"
"time"
)
var ErrSnapshotTooOld = errors.New("flag snapshot exceeds freshness budget")
type Snapshot struct {
Version string
FetchedAt time.Time
Flags map[string]bool
}
type Evidence struct {
FlagKey string
Variant string
ConfigVersion string
SnapshotAge time.Duration
FreshnessBudget time.Duration
EvaluatedAt time.Time
}
type Evaluator struct {
Snapshot func() Snapshot
FreshnessBudget time.Duration
Now func() time.Time
}
func (e Evaluator) NewPricingEnabled() (bool, Evidence, error) {
now := e.Now()
snapshot := e.Snapshot()
age := now.Sub(snapshot.FetchedAt)
enabled := snapshot.Flags["new_pricing_rule"]
variant := "control"
if enabled {
variant = "new-rule"
}
evidence := Evidence{
FlagKey: "new_pricing_rule",
Variant: variant,
ConfigVersion: snapshot.Version,
SnapshotAge: age,
FreshnessBudget: e.FreshnessBudget,
EvaluatedAt: now,
}
if age < 0 || age > e.FreshnessBudget {
return false, evidence, ErrSnapshotTooOld
}
return enabled, evidence, nil
}
The caller still owns the business response. Returning false with an error is deliberate: it prevents the new pricing rule from leaking through an untrusted snapshot, while forcing the caller to record that the established rule was selected because freshness was unprovable. Don't silently swallow the error, and don't let a browser retry determine a financial result. In tests, inject the clock and cover an age just below the budget, exactly at it, just above it, and negative age caused by clock assumptions. Also test a version change between consecutive snapshots and assert that each quote retains the evidence from the snapshot it actually used.
Choose the alert from the customer consequence
Here is the operational split I use when reviewing this design:
| Observation | Meaning | Response |
|---|---|---|
| Client and server versions differ, server is within budget | Expected convergence may be in progress | Record a diagnostic metric; do not page |
| Server snapshot approaches its freshness budget repeatedly | Margin is eroding | Create a ticket and inspect polling latency |
| Authoritative evaluation exceeds the budget | Pricing safety contract was violated | Apply the declared fallback and page |
| Version or fetch time is absent on a money decision | Freshness cannot be proved | Apply the declared policy and page if proof is required |
This is where logging cost enters the design. Per-GB ingestion pricing means high-cardinality decision evidence can become expensive if every evaluation is copied into several verbose streams. Keep the compact structured record with the authoritative transaction, use metrics for fleet-level cache-age distributions, and sample non-consequential client diagnostics according to a documented policy. Preserve all records required for financial audit or dispute handling; cost control is not permission to discard evidence with a retention obligation.
One page is enough.
A useful page names the affected service, freshness budget, observed age, configuration version, fallback action, and a query key for the authoritative transaction. It should not announce only "feature flags stale." That wording sends the responder to a dashboard without telling them whether any amount was exposed.
Where does this advice stop applying?
The catch is that bounded polling plus a conservative fallback is not suitable when every participant must switch atomically, when the old and new rules cannot coexist, or when regulation requires a centrally acknowledged activation before any decision proceeds. Use a coordinated release protocol or transactional configuration mechanism in those cases, and accept the availability and operational complexity that coordination introduces.
Stick with diagnostic-only mismatch tracking when flags affect presentation and the server remains authoritative. For offline or sleeping clients, don't promise a tight polling interval that the runtime cannot honor; carry the server-calculated price and configuration version across the boundary instead. If the product requires the client to calculate a binding amount while disconnected, this design cannot prove timely convergence. The architecture has to change.
I'm not sure a single freshness budget can serve quote creation, payment capture, and post-trade reporting in every fintech system. The evidence that would resolve it is the business contract for each decision and its permitted fallback. Set budgets per consequence, rehearse the stale-snapshot path before rollout, and write the postmortem around the first violated invariant rather than the loudest chart.
Top comments (0)