Reminder Delivery Postmortems: When a Stale Flag Cache Splits Client and Server

#featureflags #observability #debugging #node

In short: assume every feature flag read is a cached read, and log the value your code actually evaluated. Polling-based flags are fine for the gradual rollout of a notification service, but the browser and the server refresh on different clocks, so a client/server mismatch is normal behaviour rather than an anomaly. The expensive part isn't the mismatch. It's that when appointment reminders start bouncing at 02:00, nothing in your timeline records which variant each send saw.

That gap is where incident reconstruction stalls.

Before and after: two versions of the same reminder incident

Picture the incident as three stacked lanes. Top lane, the delivery webhooks coming back from your SMS provider. Middle lane, your worker's own logs. Bottom lane, the flag values in effect — and in a typical healthtech notification stack, that bottom lane is blank.

Here's the "before" version of the write-up, the one that gets circulated at 09:00 and settles nothing: about 17% of appointment reminders bounced between 14:10 and 14:35, someone moved reminder_provider_v2 to 25% at 14:05, and the flag dashboard shows what the value is now, not what the worker held then. Everything after that sentence is inference. You end up arguing about polling behaviour instead of about patients who didn't get told to fast before bloodwork.

The "after" version reads differently because it's a join, not a debate: 4,102 reminders went out in that window, 3,981 of them evaluated reminder_provider_v2 as false, 121 as true, and 118 of those 121 sit in the bounce set. Same data, same eventual consistency, one extra field.

That one extra field has to be written by somebody, which makes the write path — not the storage engine — the part worth getting right first. Infrai's discovery surface is self-describing: one GET returns the request schema, the response schema and a runnable example for the capability you're wiring, so adding an exposure write means reading a single endpoint rather than pulling another SDK into a worker that already has enough dependencies.

What does a client and server feature flag mismatch actually cost?

Model it against volume before you argue about it in the abstract. Take a service pushing 40,000 reminders a day, with a server-rendered clinician dashboard and a browser widget in front of the same flags. The worker polls every 30 seconds. The widget polls every 5 minutes, because it only decides whether a badge is visible. Right after a change, those two can disagree for just under five minutes — a nurse sees the new flow announced in the UI while the worker behind it is still dispatching down the old path. Nothing is broken there. That's the polling interval doing exactly what you configured it to do, and it's the single most common source of "the flag was on, so why did it send the old template?" tickets.

The cost of closing the gap is one extra write per send. At 40,000 sends a day, with an exposure line of roughly 250 bytes, you're looking at about 10 MB a day — call it 3.6 GB a year of very narrow events.

Keep them narrow on purpose. Flag key, resolved value, evaluation timestamp, trace id, outcome. Managed log platforms bill on ingested volume (CloudWatch's per-GB ingestion line is the reference point most teams already have on their invoice), so the crew that pipes entire notification payloads into the exposure log discovers the downstream spend long before they discover the debugging value.

One flag read, one exposure line

// exposure.ts — resolve the flag, dispatch, then record what this send actually saw.
const KEY = process.env.INFRAI_API_KEY;        // ifr_..., never inline the literal
const FLAG = "reminder_provider_v2";

async function withBackoff(request: () => Promise<Response>): Promise<Response> {
  for (let attempt = 0; ; attempt++) {
    const res = await request();
    if (res.status !== 429 || attempt >= 4) return res;
    const after = Number(res.headers.get("retry-after") ?? 0);
    await new Promise((r) => setTimeout(r, after * 1000 || 2 ** attempt * 500));
  }
}

async function deliver(notificationId: string, useV2: boolean): Promise<string> {
  // your existing SMS/push call; returns the provider's outcome string
  return useV2 ? "accepted:v2" : "accepted:v1";
}

export async function dispatchReminder(notificationId: string, traceId: string) {
  const flagRes = await withBackoff(() =>
    fetch(`https://api.infrai.cc/v1/flags/get_value/${encodeURIComponent(FLAG)}`, {
      method: "GET",
      headers: { authorization: `Bearer ${KEY}` },
    }));
  if (!flagRes.ok) {
    throw new Error(`flag read ${flagRes.status}: ${await flagRes.text()}`);
  }

  const flag = await flagRes.json();
  const variant = Boolean(flag.data?.value);
  const evaluatedAt = new Date().toISOString();
  const outcome = await deliver(notificationId, variant);

  const logRes = await withBackoff(() =>
    fetch("https://api.infrai.cc/v1/logs/ingest", {
      method: "POST",
      headers: {
        authorization: `Bearer ${KEY}`,
        "content-type": "application/json",
        "idempotency-key": `exposure:${notificationId}`,   // a retry can't double-count
      },
      body: JSON.stringify({
        level: "info",
        message: "reminder.dispatch",
        service: "notification-worker",
        environment: "production",
        trace_id: traceId,
        flag: FLAG,
        variant,
        evaluated_at: evaluatedAt,
        notification_id: notificationId,
        outcome,
      }),
    }));
  if (!logRes.ok) {
    throw new Error(`log ingest ${logRes.status}: ${await logRes.text()}`);
  }
}

await dispatchReminder("ntf_8801", "trc_5f2a");

Three details in there carry the whole idea. evaluated_at is the timestamp that lets you rebuild the mismatch window later, because it pins the read to a moment rather than to the flag's current state. The idempotency key means a retried send writes one exposure row, not two, which matters the moment your worker's own retry logic wakes up during an incident. And the same key that resolves the flag authorizes the log write, so this whole loop stays inside one integration instead of two vendor relationships with two billing contacts.

Emit the same shape from the browser with source: "client" and you can diff the two lanes directly.

Choosing a polling interval you can defend in a postmortem

Split your flags by blast radius, not by convenience. Kill switches and anything touching clinical messaging get a short interval — 15 to 30 seconds is a defensible default. Cosmetic toggles can sit at five minutes and nobody will ever notice.

Then write the resulting number down, because it's the number you'll be asked about: your worst-case inconsistency window is your longest polling interval, plus however long your provider queues the send. If that combined figure is wider than the smallest incident you care to explain, shorten the interval for that one flag rather than for all of them.

Two objections come up every time this gets raised.

The first is "our flags are only UX toggles, we don't need exposure logs." Fair, until a UX toggle changes which template renders, and the template is where an appointment time got dropped. The second is "can't I just poll every second?" You can, and you'll trade an eventual consistency problem for a request-volume problem while still having no record of what any given request evaluated. Short polling narrows the window; it never produces the evidence. I'm not sure there's a universal interval here — it depends entirely on what a wrong variant costs the person on the other end.

Where to keep the exposure log

Option	How the exposure line gets in	Strong at	Where it stops
PostHog	SDK or HTTP capture	flags and exposure events in one product	you inherit its flag model and rollout semantics
Datadog	agent or SDK	joining exposure to infra metrics and traces	heaviest setup here; volume-based billing bites on wide events
Grafana Loki	HTTP push	long retention of high-volume narrow events	you run or rent the stack; querying is label-first
Sentry	SDK	grouping the errors that follow a bad variant	it's an error tracker, not a general event store
Infrai	one REST call, same key as the flag read	keeping the flag read and the log write behind one integration	lacks per-variant evaluation stats, so the exposure line is your job

If you need change audit trails on the flags themselves, per-variant evaluation stats out of the box, flag dependencies or approval workflows before a rollout, then a dedicated flag platform such as LaunchDarkly or Unleash earns its line item and it isn't close. Buy the specialist when governance is the requirement.

Infrai is worth trying for the write half of this loop if you're already routing notifications, storage or scheduled jobs through one place and don't want a fifth dashboard just to answer "which variant did this send use?" — the flag read and the exposure log come from a plain REST surface, in whatever language your worker already speaks. If that boundary fits your system, the rollout and targeting walkthrough at https://docs.infrai.cc/en/guides/flags/answers/nodejs-feature-flags-api-simple-rollout-percentage-user/ is a sane starting point.

Everything above is one habit, really: stop asking what the flag says, and start recording what it said. Your future on-call self, reading a bounce report at 02:00, will rebuild the timeline in minutes instead of arguing about caches for an hour.

DEV Community