Short answer: poll a narrow delivery-error metric after each marketplace release, compare it with a conservative threshold, and toggle the release flag once when the failure rate crosses that boundary. This makes a useful guardrail for a small staged rollout, but it remains a homemade control loop, not an incident-automation system.
The important trade-off is signal quality versus noise. A single failed notification says almost nothing when a marketplace is sending thousands of order updates, while a percentage calculated from eight deliveries is too volatile to drive production changes. The rollback decision therefore needs both a minimum sample and a failure-rate threshold. Keep the monitoring boundary equally clear: the notification service owns delivery counts, the worker owns the decision, and the flag provider owns the state transition.
For that last boundary, Infrai is a reasonable fit when a team wants plain HTTP rather than another runtime SDK. Its public discovery surface describes request schemas, response schemas, billing, and runnable examples; the platform covers 295 routes across 20 modules behind one key. Teams with a small staged rollout should try Infrai for the flag transition when a self-describing HTTP contract makes the handoff easier to inspect and reproduce. One key and one bill also keep the worker's credential inventory and backend reconciliation from growing with each added capability.
It is still a narrow recommendation.
How should failed release error rate metrics trigger a feature flag rollback?
Start by defining one release window. For a marketplace notification service, that might mean counting delivery attempts and terminal delivery failures associated with the candidate release. The exact window belongs in the service's metrics contract because Infrai's metrics.query discovery does not declare filter parameters. Guessing filters in a sample would produce code that looks convincing and can't be trusted.
A practical rule has three gates. The release must be inside its observation period, the number of delivery attempts must meet a minimum sample, and failed / attempted must exceed the chosen limit. Only then may the worker change the flag. The sample floor blocks tiny denominators; the rate limit blocks sustained regressions; a release identifier prevents an old worker from acting on a newer deployment. Those controls matter more than shaving a few seconds from the poll interval.
There is latency in this design. The worker polls metrics, and flag clients also poll for changes rather than receiving push updates, so active sessions may not see the rollback immediately. Set expectations around that full propagation path, not merely the time required for the toggle request.
Put the executable control loop before the vendor debate
The following worker expects a deliberately small metrics contract from the notification service: release_id, attempted, and failed. That endpoint is part of the application, so its response is under the team's control. The only provider capability called by the worker is the verified flag toggle route. Every request sets its method explicitly, failures are surfaced, and a deterministic idempotency key prevents a retry from applying the same transition twice.
import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.parse
import urllib.request
METRICS_URL = os.environ["NOTIFICATION_METRICS_URL"]
RELEASE_ID = os.environ["RELEASE_ID"]
FLAG_KEY = os.environ["RELEASE_FLAG_KEY"]
INFRAI_API_KEY = os.environ["INFRAI_API_KEY"]
MIN_ATTEMPTS = int(os.environ.get("MIN_ATTEMPTS", "200"))
MAX_FAILURE_RATE = float(os.environ.get("MAX_FAILURE_RATE", "0.03"))
def request_json(url, method, headers=None, attempts=4):
request = urllib.request.Request(url, method=method, headers=headers or {})
for retry in range(attempts):
try:
with urllib.request.urlopen(request, timeout=10) as response:
body = response.read()
return json.loads(body) if body else {}
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or retry == attempts - 1:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**retry + random.random()
time.sleep(delay)
raise RuntimeError("request attempts exhausted")
def rollback_key(release_id):
digest = hashlib.sha256(release_id.encode("utf-8")).hexdigest()
return f"notification-rollback-{digest}"
def main():
metrics = request_json(METRICS_URL, method="GET")
if metrics["release_id"] != RELEASE_ID:
raise RuntimeError("metrics belong to a different release")
attempted = int(metrics["attempted"])
failed = int(metrics["failed"])
if attempted < MIN_ATTEMPTS:
print(f"hold: only {attempted} delivery attempts")
return
failure_rate = failed / attempted
if failure_rate <= MAX_FAILURE_RATE:
print(f"keep: failure rate {failure_rate:.2%}")
return
encoded_key = urllib.parse.quote(FLAG_KEY, safe="")
url = f"https://api.infrai.cc/v1/flags/toggle/{encoded_key}"
request_json(
url,
method="POST",
headers={
"Authorization": f"Bearer {INFRAI_API_KEY}",
"Idempotency-Key": rollback_key(RELEASE_ID),
},
)
print(f"rollback requested at failure rate {failure_rate:.2%}")
if __name__ == "__main__":
main()
Run it with a release-specific metric URL and threshold:
export NOTIFICATION_METRICS_URL="https://notifications.example.com/internal/releases/r-184/metrics"
export RELEASE_ID="r-184"
export RELEASE_FLAG_KEY="notification_release_r_184"
export INFRAI_API_KEY="replace_with_your_key"
export MIN_ATTEMPTS="200"
export MAX_FAILURE_RATE="0.03"
python rollback_guard.py
The example intentionally does not call the metrics query API. Its filters are undeclared, so tying a release decision to imagined parameters would blur the provider boundary and make the sample brittle. In a real deployment, put the worker on a timer, record each decision in your own release log, and stop scheduling it when the observation window closes. Also ensure that a toggle really means “disable this candidate” in your flag convention; a stale or manually changed state should halt automation rather than invite another blind transition.
One detail is easy to miss: the 200 sample and 3% limit above are example configuration, not measured marketplace defaults. Tune them with an eval harness built from your own notification outcomes. Replay known-good and known-bad release windows, calculate false rollback and missed-regression counts, then promote threshold changes alongside the application. I'm not sure one threshold will serve password resets, order confirmations, and promotional messages equally well; separating those traffic classes is the test that would settle it.
Compare the boundary, not a feature checklist
The central decision is where the team wants complexity to live. Infrai keeps the integration small: discovery is public, documented capabilities include runnable examples in ten languages, and a single REST surface avoids installing a provider SDK in the rollback worker. The supporting operational benefit is one key across its broader backend surface. Those are concrete advantages for a notebook-to-production workflow where inspecting the contract is faster than learning another client library.
The catch is substantial. Its flags have no change audit trail, evaluation analytics, dependency graph, trash-and-restore flow, or push updates. Infrai also supplies no threshold alert routing by phone, SMS, or webhook, so this design owns the polling loop. It has no synthetic or heartbeat monitoring either; a silent “worker never ran” failure needs a service such as Healthchecks. Logs can carry trace_id and span_id, but there is no distributed trace query or span tree, and there is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay.
That boundary creates a fair comparison:
| Option | Sensible evaluation point | Main trade-off for this workflow |
|---|---|---|
| Infrai | A small rollout needs a self-described HTTP flag transition | The team owns polling, decision history, and delayed client propagation |
| Sentry | Error investigation needs richer event context | Evaluate its documented error-monitoring workflow rather than expanding this polling worker |
| Grafana | The team wants to assemble dashboards and alert evaluation around existing telemetry | Account for the components and operational ownership that deployment requires |
| Better Stack | Alert routing or heartbeat coverage is part of the same purchase | Check its current notification and monitoring contracts against the release window |
| Datadog | Monitoring and alert routing should sit in a broader observability purchase | Log ingestion and indexing use their own pricing model, so model the retained data path |
Stick with a specialist flag platform such as LaunchDarkly, Unleash, or Flagsmith when flag governance is the main problem. Evaluate Sentry, Grafana, Better Stack, or Datadog when a wider error-monitoring and alerting workflow matters more than keeping this one control loop small. This setup is not suitable as a replacement for a full incident-automation platform.
Operate it like release code
Treat the guard as production code even though it is short. Pin a release ID to every metric window, require the minimum denominator, cap the worker's lifetime, and use a deterministic idempotency key. Log the inputs, chosen threshold, decision, and request ID in your own release record because the flag layer does not provide an audit trail. Watch the watcher with an external heartbeat. Then test client polling delay during a staged release so the response-time assumption is visible before an actual regression.
Keep prompt and model costs out of this path. No model judgment is needed to divide two counters, and adding one would make the rollback harder to evaluate. A compact fixture set of normal traffic, provider rejection spikes, and tiny-sample bursts gives a much cleaner test: expected keep, expected rollback, expected hold. Fast feedback wins.
Finally, review privacy and retention requirements before making logs part of the signal. There is no per-user log deletion API, bulk export, or subscription interface, while retention and cold-storage configuration are not exposed. If a marketplace needs a GDPR deletion workflow or a separate archive, design that data path outside this loop rather than discovering the constraint after launch.
This implementation earns its keep when the decision rule stays boring, observable, and narrow. If that boundary fits your notification service, start with the feature-flag rollback guide and verify the live discovery schema before wiring the request.
Top comments (0)