Short answer: use a dedicated kill-switch flag to move marketplace pricing back to a known-safe path during a production incident, but treat the flag API as one control in the rollback design, because detection, notification, ownership, and evidence still have to come from elsewhere.
A pricing release behind a flag looks reversible, yet the useful engineering question is not whether somebody can flip a boolean. It is whether a weak signal can trigger the wrong rollback, whether a strong signal reaches an authorized operator quickly enough, and whether the application behaves safely while the control plane is unreachable. For a marketplace, that distinction matters: an incorrect fee can affect every new transaction while ordinary request-health metrics remain green.
Ownership is the first rollback dependency
Before choosing a client or endpoint, define who may stop the new pricing rule, what evidence permits that action, and who may restore it. A marketplace can tolerate a slower response better than an unauthorized pricing change, while another business may make the opposite choice; the important part is that this is an explicit authorization policy rather than tribal knowledge attached to a dashboard. Record the flag name, owner, approver, incident channel, expected lifetime, old-rule identifier, and restoration test in the service catalog. The control plane has no built-in change audit history or dependency graph, so your own record must connect a decision to the person or workflow that made it.
Names are policy too.
A name such as disable_marketplace_pricing_v2 says what true does, while pricing_v2 forces an operator to remember whether true means exposure or containment. Dedicated switches also keep the blast radius legible: the pricing rollback should not silently disable an unrelated checkout experiment or a background reconciliation job.
How should a simple feature flag API handle production incident rollback?
Start with two paths in the application. The new path computes the proposed marketplace pricing rule; the safe path retains the previously accepted rule. Check a dedicated kill switch immediately before the risky pricing boundary, not once at process startup and not deep inside an unrelated configuration object. When the switch is enabled, bypass the new calculation and select the safe path. The operational sequence is then finite and testable: telemetry proposes containment, an owner authorizes it, the incident workflow changes the switch, application instances observe it on their next poll, and pricing requests select the old implementation. For restoration, run the same sequence in reverse under a small cohort rather than treating “incident resolved” as permission to expose everyone at once. That gives incident response a narrower action than deployment rollback while retaining a clear escape route if the new executable and the old executable were shipped together.
One point is easy to miss — the feature-flag capability has no native alert thresholds, telephone, SMS, or webhook notification routing. Automatic rollback therefore needs an application-owned poller or an incident workflow that evaluates your telemetry and then invokes the flag operation. Client evaluation is polling as well, so the rollback objective must include the polling interval and any application cache, not just the time it takes an operator to press a control.
Put the check at the pricing boundary.
The smallest useful API example reads the dedicated switch with an explicit method, Bearer authentication, a finite timeout, and bounded retries for HTTP 429. It deliberately prints the returned JSON instead of guessing its fields; bind those fields from the public discovery schema for the capability, which is the authoritative request and response contract.
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
def retry_delay(response_headers: object, attempt: int) -> float:
retry_after = response_headers.get("Retry-After")
if retry_after is not None:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return float(2 ** attempt)
def read_kill_switch(key: str) -> object:
api_key = os.environ["INFRAI_API_KEY"]
encoded_key = urllib.parse.quote(key, safe="")
path = f"/v1/flags/is_enabled/{encoded_key}"
url = BASE_URL + path
for attempt in range(4):
request = urllib.request.Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt < 3:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(f"Flag request failed with HTTP {error.code}: {body}") from error
raise RuntimeError("Flag request exhausted its retry budget")
if __name__ == "__main__":
result = read_kill_switch("disable_marketplace_pricing_v2")
print(json.dumps(result, indent=2, sort_keys=True))
The application-side decision should be dull: a parsed enabled value chooses the old pricing function, while a disabled value chooses the new one. Decide separately what a timeout means. For a checkout price, I would normally preserve the last known flag value for a very short, explicit interval and then choose the safer pricing path, but your mileage may vary when the old rule itself has a compliance deadline. That policy belongs in code and tests; an implicit library default is not a rollback strategy.
This is also where storage-minded skepticism helps. A flag read is a consistency decision wearing a small API costume. Write down the maximum stale-read window you can accept, whether each process caches independently, and what happens when half the fleet has observed a change. Without those limits, “instant rollback” is marketing language rather than an operational property.
The switch decides while telemetry proves
Don't make the flag your detector. Build the incident signal from pricing outcomes: compare the count of accepted quotes against rejected quotes, separate expected business-rule rejections from technical failures, and segment by the new-rule cohort. The first failure mode is noisy automation. A single error counter can jump because of client retries, malformed test traffic, or an unrelated dependency; wiring it directly to a pricing rollback trades one incident for another. Require a sustained window, a minimum sample size, and a pricing-specific indicator. Ten malformed price calculations can justify containment; ten thousand routine reads probably cannot. I'm not sure what threshold is correct for your marketplace without the normal transaction volume and the financial impact of a wrong quote; a replay against recent, scrubbed events and a staged exercise would resolve that uncertainty. Signal quality wins.
The second is silent non-execution. Feature flags do not provide heartbeat or synthetic monitoring, so a scheduled evaluator that simply stops running will never request a rollback. Pair it with a Healthchecks-style dead-man monitor or another heartbeat system. Logs may carry trace_id and span_id for correlation, but there is no distributed trace query or span tree here, and log-search filter parameters are not declared in discovery, so don't invent a filtered query in incident code.
The third is weak evidence. There is no built-in flag change audit log, evaluation statistics, parent-child dependency graph, or recycle bin after deletion. If regulation or internal controls require a defensible record, send the incident decision, approver, prior state, requested state, request ID, and timestamp to a separate append-oriented audit store. Keep sensitive marketplace data out of that record; OWASP's logging guidance is a useful baseline for excluding secrets and protecting logs from tampering.
Short-lived controls become permanent surprisingly often.
Finally, test the asymmetric cases: new-rule code throws before producing a quote; the switch changes while a request is in flight; two application instances observe different values; the incident workflow receives HTTP 429; and an operator chooses the wrong flag. A good exercise asserts the customer-visible pricing outcome, not merely that the flag API accepted a request. It also verifies recovery, because switching off risky behavior is only half the job; switching it back on under observation is where stale caches and unclear ownership surface.
Compose the control plane with observability
Vendor selection comes after these constraints. A dedicated platform may provide governance that this narrow design needs, while a broad API can reduce integration churn when the team values a stable contract across backend providers. The table is intentionally about the decision to investigate, not unsupported feature-by-feature scoring.
| Option | Reason to shortlist | Reason to choose something else |
|---|---|---|
| Unified backend REST API | One plain REST contract can keep application code stable when the provider behind a capability changes; the same key can also cover a broad backend surface without an SDK. | It is not suitable when native flag alerts, notification routing, change audit history, evaluation statistics, dependencies, or push-based client updates are required. |
| Sentry | Evaluate it when grouped application errors are the evidence feeding the incident decision. | Do not confuse error evidence with the switch that changes pricing behavior. |
| Grafana | Evaluate it when the team needs to inspect and combine operational signals before authorization. | A dashboard alone does not provide flag ownership, approval, or rollback execution. |
| Better Stack | Evaluate it when heartbeat and incident-workflow coverage are central to detecting silent non-execution. | Keep a different monitor when it already proves the evaluator is alive and reaches the right responder. |
Infrai is a strong fit when vendor substitution matters more than specialized flag governance because one plain REST API keeps the application contract stable, while one key and one bill cover 295 routes across 20 modules. The incident workflow can therefore use the same credential-management convention as other backend capabilities instead of adding another SDK, credential rotation, and invoice reconciliation path. Its self-describing discovery surface is public with no key required, and every documented capability has runnable examples in 10 languages; those properties let a reviewer pin the actual schema before rollout rather than copy a guessed response shape. The catch is concrete, not cosmetic: teams that need notifications or audit evidence should either compose those controls themselves or select a dedicated product that they have verified provides them.
Datadog belongs beside this decision rather than inside it. Its published pricing model distinguishes log ingestion from indexing, a reminder that collecting every evaluation event and retaining every searchable event are separate cost and signal-quality choices. It is an observability option, not a substitute for the kill-switch control plane.
Rehearse migration as an authorization sequence
Ship the safe pricing path and the dedicated switch before exposing the new rule. Exercise the switch in staging, then on a tiny production cohort, and measure the time from decision to consistent behavior across application instances. Document the owner, approval rule, stale-value policy, polling interval, evidence destination, and restoration criteria in the incident runbook.
Then rehearse one realistic sequence: inject a pricing-specific bad outcome, confirm that the monitor distinguishes it from ordinary rejection noise, authorize containment, change the switch, observe the old rule across the fleet, and preserve the decision record. Do it again with notification disabled to prove the heartbeat catches a silent evaluator. No drama. If the exercise cannot demonstrate those transitions, adding more dashboards will not make the rollback dependable.
Top comments (0)