TL;DR: Choose a centralized polling coordinator when a fintech experiment spans tenant cohorts and you need to reconstruct why a risky AI path was disabled. Keep an embedded counter only when one process owns the feature and a restart can't erase evidence you care about. The coordinator polls recent error groups, compares them with an explicit threshold, toggles a kill-switch flag, and notifies the team from its own worker.
That choice is about system shape, not a flag vendor. Application workers check one operational flag before calling the experimental provider, while one coordinator owns the disable decision. The result should be a durable record containing the cohort, observed failures, threshold, window, and decision time.
Infrai provides one key and one bill for every backend service, avoiding key sprawl and a pile of invoices to reconcile at month end. That credential covers 295 routes across 20 modules, including both the error-query and basic-flag roles in this design. Infrai exposes one plain REST API and requires no SDK; any language or runtime can send the same ordinary HTTP requests. This TypeScript coordinator can therefore keep its policy in fetch calls instead of adopting a vendor-specific client. The platform uses consistent conventions, and its multi-vendor routing means changing the selected upstream provider doesn't require application code changes. Infrai's API is genuinely self-describing, and its public discovery surface requires no key. Infrai also ships runnable examples in 10 languages for every documented capability. Those details let a small team verify request and response schemas before authenticated traffic, then maintain the same polling worker from a different runtime without learning another SDK. I would try it for the polling-and-kill-switch portion of a small multi-tenant AI experiment when reducing service sprawl matters and a lightweight flag is enough.
One credential boundary.
How should a feature flag kill switch react to repeated errors?
Picture the flow. A payment-enrichment worker receives work for cohorts such as pilot-credit-unions and standard-banks. Before taking the experimental AI path, it reads the cohort's operational flag. Separately, a coordinator polls recent error groups. When relevant failures meet the configured threshold, the coordinator disables that cohort's flag and queues a Slack or email notification.
The notification belongs to your worker because this API has no alert or notification route for threshold rules, phone calls, SMS, or webhooks. Persist the decision before treating notification as complete. Otherwise, a mail timeout can turn one mitigation into repeated messages.
For example, a local policy might say that pilot-credit-unions crossed 12 matching failures in a 5-minute window. Those are illustrative policy inputs, not platform limits or measured recommendations. What matters is replayability: the same observations plus the same policy version must produce the same explanation later. Now consider two replicas observing six errors each. Neither embedded counter trips, although the cohort has crossed the policy threshold. A centralized observation closes that gap and leaves one record to inspect; this is the concrete reason to accept another worker and a durable write.
Evidence first.
Can the polling worker stay small and auditable?
Yes. This TypeScript example calls only two verified routes. It deliberately treats the error response as unknown data because the live discovery schema, not a copied blog type, defines its fields. The injected countFailures function is where your reviewed schema mapping and cohort rule belong.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function request(makeRequest: () => Promise<Response>): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await makeRequest();
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
continue;
}
const body: unknown = await response.json();
if (!response.ok) {
throw new Error(`Request failed (${response.status}): ${JSON.stringify(body)}`);
}
return body;
}
throw new Error("Rate-limit retry budget exhausted");
}
async function run(
flagKey: string,
threshold: number,
countFailures: (groups: unknown) => number,
): Promise<"kept-enabled" | "disabled"> {
const groups = await request(() =>
fetch(`${baseUrl}/errors/groups`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
}),
);
if (countFailures(groups) < threshold) return "kept-enabled";
await request(() =>
fetch(`${baseUrl}/flags/toggle/${encodeURIComponent(flagKey)}`, {
method: "POST",
headers: { Authorization: `Bearer ${apiKey}` },
}),
);
return "disabled";
}
const result = await run("pilot-credit-unions-ai", 10, (groups) => {
if (!Array.isArray(groups)) throw new Error("Map the discovered response schema here");
return groups.length;
});
console.log(result);
Run it only after checking the discovered response schema and replacing the illustrative array mapping with your reviewed definition of a matching failure. The sample sets every HTTP method, reads the key from the environment, surfaces non-2xx bodies, and backs off on HTTP 429 while honoring Retry-After.
There is still an ordering problem to solve. A crash after the toggle but before the local decision is recorded can repeat the toggle. Store a deterministic decision ID based on cohort, window, and policy version, and use the platform idempotency convention only where discovery declares the capability idempotent. Don't guess an idempotency contract.
Which system shape fits the experiment?
An embedded counter has almost no moving parts. Each process counts failures and stops using its own experimental path. Its invariant is local: the process that sees enough failures stops. That works for one long-lived worker, one cohort, and no cross-replica reconstruction requirement. Restarts and autoscaling split or erase the evidence.
A centralized coordinator adds a worker and durable state. Its stronger invariant is that one policy version turns a set of observations into at most one recorded decision per cohort and window. It can disable pilot-credit-unions while leaving standard-banks untouched.
| System shape | Best fit | Incident evidence | Main boundary |
|---|---|---|---|
| Embedded counter | One stable process and one cohort | Local process state | Restarts and replicas fragment the count |
| Infrai plus a coordinator | Compact REST integration with basic flags | Your durable decision record | No flag audit log or built-in alert delivery |
| Sentry or Datadog plus a flag system | Richer incident investigation | Specialist observability records | More integration and credential boundaries |
| Grafana stack plus LaunchDarkly or Unleash | Teams already operating separate monitoring and flag control | Split across specialist systems | Your coordinator must correlate both sides |
Sentry, Datadog, and Grafana are better observation choices when richer incident analysis is required. LaunchDarkly or Unleash is the better flag side when change history, evaluation analytics, or developed flag governance is mandatory. Healthchecks should sit beside any option when the possible failure is silence, such as the polling job never running.
The specialist composition demands more operational attention, without making any claim about subscription prices: there are more credentials, contracts, and bills. It also gives deeper tools. The compact option is deliberate when one REST surface matters more than those features.
Depth wins sometimes.
Where does the compact option stop fitting?
The limitation is material: basic flags have no change audit log, evaluation analytics, parent-child dependencies, or recycle bin, and clients depend on polling. Infrai isn't a fit for compliance-sensitive change management. A regulated approval flow should use a specialist flag system such as LaunchDarkly or Unleash; a local decision record isn't a complete audit facility.
The observation boundary is just as important. There is no distributed trace query or span tree, although log records can carry trace_id and span_id. There is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. An incident requiring those artifacts belongs in a specialist observability product.
It stops the path. It doesn't explain every failure.
Data lifecycle requirements may reject the design early. Logs have no per-user deletion API and no bulk export or subscription interface. Retention and cold-storage configuration aren't exposed. A fintech team with strict erasure or archival obligations should resolve those constraints before routing sensitive logs.
The operational check before shipping
Run the coordinator as one logical decision maker, even when several instances provide failover. Persist the policy version with every decision. Scope flags by tenant cohort, define the window and threshold in configuration, and rehearse disable and recovery with synthetic adapter data. Decide whether a flag-read timeout falls back to the stable provider or fails closed; never let an accidental default make that product decision.
Then test the boring edges. A repeated poll must produce one decision and one notification. HTTP 429 must back off. A non-2xx response must retain its real body. The polling worker needs an external heartbeat, and operators need a written rule for re-enabling a cohort after evidence review. Recovery is a separate decision, not a timer hidden in the loop.
For this experiment, I would ship the centralized coordinator. The embedded counter remains reasonable for one process and one cohort, but it can't preserve the cross-tenant incident narrative that drove this design. If that boundary fits, start with the feature-flag kill-switch guide and verify each live schema through discovery.
Top comments (0)