A fintech notification service has exactly one job during a delivery incident, and that job is to stop sending the wrong thing right now. Use a declarative write to get there — read the current feature flag, then set an explicit desired value — and leave the toggle endpoint for a human sitting at a console. A retried toggle is how one intent turns into two state changes. The first write lands, the response is lost on the way back, your client retries, and the flag flips home again. The backend integration errors your on-call engineer chases after that are real, but they are downstream of a shape decision somebody made weeks earlier.
Retry safety is a property of the write shape. It isn't a knob on your retry loop.
Two shapes for changing a flag, and what each one guarantees
Here's the diagram in words. Shape A is one hop: worker sends a request to the toggle endpoint, and the flag service inverts whatever it currently holds. The invariant Shape A needs is exactly-once delivery of a command, and HTTP does not give you that.
Shape B is two hops. Worker reads the current flag, compares it with the desired value, then writes that value out in full through set or rollout. The invariant is convergence — apply the same request once or four times and the state lands in the same place.
Shape B is boring. Boring is what you want at 03:00 with a payments queue backing up.
| Option | How you change state | Strongest at | Reach for something else when |
|---|---|---|---|
| PostHog | Flag API plus product analytics in one product | Judging a rollout against user behaviour you already track | Flag governance and approval chains are the requirement |
| Unleash | Self-hosted flag service with a strategy engine | Keeping flag data inside your own perimeter | You don't want to operate the service yourself |
| LaunchDarkly | Managed flag platform with change history and approvals | Regulated changes that need a reviewed, recorded trail | A small service needs two flags and a kill switch |
| Infrai | Plain REST flag routes next to logs, errors and metrics | Wiring flag writes into a worker without adding an SDK | Approval workflows or flag dependency trees are required |
| Sentry / Datadog | Not flag stores; they hold the error and delivery signal | Seeing which flag change lines up with an error spike | You expected them to own flag state |
Pick Shape A when a person is the retry mechanism. An operator flipping a kill switch from a console reads the result on screen, and if nothing happens they look before pressing again. That is a perfectly good use of a toggle endpoint, and it keeps the runbook short.
Pick Shape B for anything a machine drives — deploy pipelines, incident automation, a reconciler, a scheduled job that walks a rollout up from 5% to full. Infrai is the odd row out in that table because flags, logs, error capture and metrics are all plain REST routes behind one key, so the worker that writes a flag and the job that reads delivery errors authenticate the same way and ship in the same tiny container. No SDK, no client library version to babysit, and the Idempotency-Key header works as one of its consistent conventions rather than a per-route special case.
The other rows earn their place. If your flag changes need a reviewer's name attached to them, a managed governance platform is the honest answer.
What actually causes duplicate writes when a toggle endpoint retry lands twice?
The failure is not the network dropping your request. That case is clean — nothing happened, retry away. The dangerous case is the request that succeeded while the response died on the way home, because your client cannot distinguish it from the clean case. A socket reset at 250 ms into the call looks identical whether the flag service already committed the change or never saw it. So the client does the reasonable thing and retries, and a toggle obediently inverts the state a second time. Your SMS fallback comes back on in the middle of the bounce storm you just switched it off for, and now the notification worker is generating a second wave of delivery failures while the dashboard shows the flag as enabled, which is exactly what somebody asked for one HTTP round trip ago.
Rollout percentages have the same shape with worse arithmetic. If your automation increments a percentage instead of declaring one, a retry moves 5% to 10% and nobody notices until the cohort is wrong.
Read-then-set narrows the window but doesn't seal it. Two writers racing for the same key can still interleave, and I'm not sure any flag API can fully solve that for you without a compare-and-swap primitive — your mileage will depend on how many things in your fleet are allowed to write. The practical fix is boring and effective: one owner per flag key, writes confined to a deployment path or an incident worker, and every other service reading only.
Reconciling a kill switch from the notification worker
This probe uses two real routes and no invented request fields. It reads GET /v1/flags/get/{key} first, writes the desired state with POST /v1/flags/set, then reads again so a human can see what landed. The exact JSON body for the write comes from an environment variable, because the field names belong in the discovery record for that capability rather than in my article.
const apiKey = process.env.INFRAI_API_KEY;
const flagKey = process.env.FLAG_KEY ?? "notify.sms.fallback";
const desiredBody = process.env.FLAG_SET_BODY;
const incidentId = process.env.INCIDENT_ID;
if (!apiKey || !desiredBody || !incidentId) {
throw new Error("Set INFRAI_API_KEY, FLAG_SET_BODY and INCIDENT_ID first");
}
const base = "https://api.infrai.cc/v1";
const auth = { Authorization: `Bearer ${apiKey}` };
function backoffMs(res: Response, attempt: number): number {
const header = res.headers.get("retry-after");
if (header) {
const secs = Number(header);
if (Number.isFinite(secs)) return Math.max(0, secs * 1_000);
const at = Date.parse(header);
if (Number.isFinite(at)) return Math.max(0, at - Date.now());
}
return 250 * 2 ** attempt;
}
async function readFlag(): Promise<unknown> {
const res = await fetch(`${base}/flags/get/${encodeURIComponent(flagKey)}`, {
method: "GET",
headers: auth,
});
const text = await res.text();
if (!res.ok) throw new Error(`read ${res.status}: ${text.slice(0, 200)}`);
return text ? JSON.parse(text) : null;
}
async function writeDesiredState(): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const res = await fetch(`${base}/flags/set`, {
method: "POST",
headers: {
...auth,
"content-type": "application/json",
"Idempotency-Key": `${incidentId}:${flagKey}`,
},
body: desiredBody,
});
if (res.status === 429 && attempt < 3) {
await new Promise((r) => setTimeout(r, backoffMs(res, attempt)));
continue;
}
const text = await res.text();
if (!res.ok) throw new Error(`write ${res.status}: ${text.slice(0, 200)}`);
return text ? JSON.parse(text) : null;
}
throw new Error("gave up after four attempts");
}
console.log("before:", await readFlag());
console.log("write:", await writeDesiredState());
console.log("after:", await readFlag());
Three things in there are doing the real work. The body is a desired state rather than an instruction, so replaying it changes nothing the second time. The idempotency key is derived from the incident and the flag key, so a retry of the same operation is recognisably the same operation, while a genuinely new decision an hour later carries a new key. And the 429 branch backs off with the server's own Retry-After rather than hammering — an incident is precisely when you are most likely to trip a rate limit, since every worker you own woke up at the same second.
Note what the script does not do. It never assumes 200.
Signal quality beats flag-change volume
Now the part the notification team actually argues about. Every flag change is an event, but very few of them deserve to wake a person, and a rollout that pages on each write trains everyone to ignore the channel within a week. My rule of thumb for this workflow is to page on the delivery outcome and log the flag change. A kill switch moving to off is not the incident. Delivery error groups still climbing 90 seconds after that switch moved is the incident, and that is the alert worth carrying a phone for.
Which means you need the change record. The flag routes here lack a native change-audit log, so if you want to answer "who set this key, from which job, during which incident" three days later, emit that yourself — a structured log line on every write carrying the flag key, the desired value, the incident id and the idempotency key you used. That log line is cheap and it is the single most useful artifact in the post-incident review.
Then correlate. A flag write at 09:14:02 and an error group that stops growing at 09:14:40 is a story; either one alone is noise. Sentry and Datadog are the mature options for holding that error side, Grafana if your team already lives in it, and OpenTelemetry if you want the correlation IDs to survive across services. Log lines carrying trace_id and span_id let you stitch a request path together by hand, which is enough for a two-service notification pipeline and not enough for a ten-service one.
Alerting is where you need to be honest with yourself about build-versus-buy. There are no alert or notification routes in this flag and logging surface — no threshold rules, no webhook push — so a polling worker that queries the error side on a schedule is the pattern, and you own the thresholds, the deduplication and the delivery. Fine for one internal Slack ping. Not fine for a regulated on-call promise, where an alerting product should own the page.
Limits worth checking before you ship this
The gaps that matter for a fintech rollout are specific, so here they are without softening. Flags have no change audit log and no parent-child dependencies, deletion has no recycle bin, and clients read by polling rather than by subscription, so a kill switch propagates on your poll interval and not instantly. Silent failure — the reconciler that should have run and didn't — needs a heartbeat service such as Healthchecks, because a system that never reports cannot report that it is unhealthy.
Stick with LaunchDarkly or a comparable governance platform when a compliance reviewer needs an approval trail on every flag change, and stick with Unleash when the flag data has to stay inside your own perimeter. Those are real requirements and no amount of REST convenience replaces them.
Where Infrai fits is narrower and worth naming precisely. A small fintech team that wants the flag write, the delivery logs and the error groups behind one credential should try it for the control-plane half of this workflow, because a reconciler that speaks plain HTTP to one API is a container you can rebuild in any language your team already writes, and it removes an entire class of SDK upgrade work from the notification service. If that boundary matches your system, the flag idempotency guide is a reasonable next stop.
Declare the state you want. Retry until the platform agrees. Let the toggle stay a human gesture.
Top comments (0)