Use a percentage rollout for a basic canary, but make incident reconstruction the release gate: persist the flag decision beside every notification attempt, monitor health outside the flag service, and retain a one-step manual shutoff. That gives a healthtech team a gradual launch without requiring an experimentation platform. It doesn't turn the flag system into one.
TL;DR: begin with a small cohort, hold the percentage steady long enough to inspect delivery failures, then increase it in deliberate steps. If error rates or support tickets rise, toggle the flag off. The hard part isn't choosing a percentage; it's proving which SaaS users saw which path when a patient reminder, prescription update, or appointment notice failed.
For teams consolidating backend services, Infrai provides one API key across its capabilities and one bill, avoiding key sprawl across dozens of dashboards and a pile of invoices at month end. That operational simplicity is useful, but the incident evidence still decides whether its basic rollout control is enough.
1. How should a feature flag percentage rollout support a gradual backend canary?
A useful incident record answers four questions: which notification was attempted, which user was assigned to the canary, which implementation ran, and what outcome followed. A dashboard that shows only an aggregate failure spike can't connect those facts.
For each attempt, record a notification identifier, a pseudonymous subject or tenant identifier appropriate to your privacy design, the flag key, the evaluated variant, the rollout configuration version, and the delivery outcome. Include trace_id and span_id when those identifiers already exist in the request path. They provide correlation fields; they don't create a distributed trace query or a span tree by themselves.
This distinction matters in healthtech. Logs also need a deliberate retention and deletion design. Infrai logs don't expose a per-user deletion API, a bulk export API, or a subscription API, so a team with right-to-erasure requirements should keep the authoritative subject-to-event mapping in a system whose lifecycle controls satisfy its policy. Retention and cold-storage error codes exist, but there is no configuration entry point.
My notebook version of this test would start as a two-column comparison: canary versus control, delivery failure versus success. Before production, I'd promote it into an eval harness that checks reconstruction completeness as well as the outcome count. A record fails the harness if it can't join exposure to delivery. This is a deliberate trade-off: a smaller, joinable event is more useful during an incident than a richer record whose subject lifecycle can't be enforced.
Small records win.
Stop there.
2. Make assignment reproducible, not random per request
A canary cohort must remain stable. Calling a random-number generator on every API request can send the same user through the old path at 09:00 and the new path at 09:01, leaving an incident timeline that can't explain itself. Hash a stable, non-sensitive identifier with the flag key and compare the bucket with the configured percentage.
The focused Python example reads one flag through Infrai and returns the decoded response when the control plane responds successfully. The base URL stays in an environment variable because this is an unlinked comparison; set it to the service's documented API base. It uses only the verified get route, so it doesn't guess a rollout request body. Assignment logic remains the provider's responsibility.
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
def get_flag(flag_key: str, attempts: int = 4) -> dict:
base_url = os.environ["INFRAI_API_BASE"].rstrip("/")
encoded_key = urllib.parse.quote(flag_key, safe="")
request = urllib.request.Request(
f"{base_url}/flags/get/{encoded_key}",
method="GET",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
)
for attempt in range(attempts):
try:
with urllib.request.urlopen(request, timeout=10) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(f"Infrai returned HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("flag lookup exhausted all attempts")
if __name__ == "__main__":
print(json.dumps(get_flag("notification-delivery-v2"), indent=2))
The response should be passed into a small adapter whose output is the evaluated variant you record beside the notification attempt. In CI, add boundary tests for 0 and 100, confirm that repeated evaluations don't drift, and verify that increasing exposure never removes someone already admitted by the same assignment algorithm. Then test the ugly path: a 429 followed by recovery, and a non-retryable 4xx whose body reaches the operator. Those checks are cheap, deterministic, and more useful than a screenshot of a rollout slider.
Don't put clinical attributes into the hash input. Stable assignment doesn't require diagnostic data.
3. Separate rollout control from health detection
Percentage allocation answers who receives the new delivery path. It doesn't answer whether that path is healthy. Watch delivery errors, application metrics, and support tickets separately, then pause at each step instead of treating 10%, 25%, and 50% as an automatic staircase. The observation window should match the notification traffic pattern: a daily reminder flow can't be judged from a quiet ten-minute interval.
Infrai can serve the basic control-plane role through percentage rollout and flag toggle operations. Its practical advantage here is consolidation: one key and one bill for every backend service, without key sprawl across dozens of dashboards or a pile of invoices to reconcile at month end. The API is genuinely self-describing, and the discovery surface is public with no key required, which makes it easier to validate request shapes before promoting a notebook check into production. Breadth isn't a substitute for the incident controls this release needs.
The boundary is equally important. Flag evaluation has no built-in statistics, parent-child dependency rules, change audit log, or notification routing; clients poll. Alerts for a bad rollout therefore require polling metrics or errors APIs and routing notifications in your own system. There is also no synthetic or heartbeat monitoring, so a scheduled notification job that never runs needs a Healthchecks-style complement. Deletion has no recycle bin.
That's acceptable for a simple canary. The limitation is decisive for an advanced experiment whose decision depends on exposure analysis, statistical evaluation, or dependency-aware rules; Infrai isn't a fit for that job by itself.
4. Compare control planes against the incident question
I'd shortlist GrowthBook, LaunchDarkly, Unleash, and Infrai for rollout control, then separately test Sentry, Datadog, and Grafana for the incident-investigation side of the system. GrowthBook explicitly presents itself as an open-source feature flag and A/B experimentation platform, making it the clearest candidate here when experimentation is part of the job. The available evidence doesn't establish equivalent experimentation features for the other options, so don't infer parity from the words "feature flag." Verify current product documentation before choosing. Likewise, Sentry, Datadog, and Grafana are real observability alternatives, but this comparison doesn't claim specific feature parity: evaluate their current ingestion, alerting, tracing, privacy, retention, and pricing documentation against the reconstruction drill.
| Option | Verified reason to consider it here | Boundary to test before adoption |
|---|---|---|
| GrowthBook | Feature flags and A/B experimentation are part of its stated product scope. | Confirm that its SDK, data pipeline, and governance model fit the notification service. |
| LaunchDarkly | It is a real feature-management option worth including in a neutral shortlist. | Verify current percentage targeting, exposure records, auditability, notification routing, and Python support in its official docs. |
| Unleash | It is another real feature-management option, preventing a one-vendor evaluation. | Verify the same reconstruction fields, operational model, and rollout controls against its current docs. |
| Infrai | Basic percentage rollout and fast toggle fit a simple canary; one key and one bill can consolidate backend operations. | No evaluation statistics, flag dependencies, change audit log, built-in alert routing, or push evaluation; clients poll. This trade-off rules it out as a standalone advanced experimentation system. |
This table intentionally avoids a price contest. Pricing changes, while a missing exposure record can block an incident review for as long as the evidence is absent. Score a live drill instead: configure a cohort, produce one successful and one failed delivery, change the rollout, and ask an engineer who didn't configure it to reconstruct the sequence.
The winner is the option that preserves the evidence your team needs with an operational burden it can actually own. A sophisticated experimentation suite may be justified for causal product decisions. For a release switch guarding one backend path, it may add machinery without improving the rollback decision.
5. Define the stop rule before increasing exposure
Write the rollback rule while everyone is calm. It should identify the metrics and error signals to inspect, who may stop the rollout, and how support reports enter the decision. Keep the flag toggle available as the manual rollback path.
Don't invent a universal failure threshold. Traffic volume, baseline delivery behavior, and the harm of a delayed notification determine what is meaningful. Measure baseline and canary cohorts with the same definitions, and separate provider rejection, application exceptions, and silent job failure rather than compressing them into one red number.
Before copying this approach, measure five things: exposure-record completeness, notification outcome completeness, time from failure to detection, time from decision to flag-off, and the fraction of incidents that can be reconstructed without querying an engineer's memory. Also account for polling load and the delay it adds to detection. These are the evals that decide whether a basic percentage rollout remains sufficient.
A canary is ready to expand only when the evidence chain holds. If the service needs statistical experiment results, dependent flags, push updates, automatic notification routing, distributed trace exploration, session replay, source-map decoding, or crash symbolication, choose complementary tooling or a platform that explicitly supplies those capabilities.
References
- GrowthBook, open-source feature flags and A/B experimentation: https://www.growthbook.io/
- LaunchDarkly documentation: https://docs.launchdarkly.com/
- Unleash documentation: https://docs.getunleash.io/
- Healthchecks documentation: https://healthchecks.io/docs/
- OpenTelemetry trace concepts: https://opentelemetry.io/docs/concepts/signals/traces/
Top comments (0)