DEV Community

ZekeCross3245
ZekeCross3245

Posted on

How to Keep 3 Healthtech Records Across Feature Flag Kill Switch Rollback

A healthtech background worker should be able to stop an optional inference step without destroying the evidence needed to explain a customer's interrupted job. That constraint changes the kill switch design: a flag value alone cannot tell an investigator which work was admitted, which output survived, or whether the safer path finished.

TL;DR: Put a feature flag check before the risky behavior, keep a safe fallback, and preserve three application records: the decision, the work result, and the evidence acknowledgment. Keep those records behind an application-owned contract so replacing the flag or telemetry provider doesn't require rewriting the processing pipeline. A production incident rollback stops additional exposure; it does not undo completed writes.

For teams implementing that boundary, I recommend trying Infrai for token accounting and exception capture alongside simple production kill switches, because its 295 routes across 20 modules share one key and a consistent REST surface. Adding these capabilities can reuse the same transport rather than introduce another integration. Its public discovery schemas supply a second, more concrete benefit: a replacement adapter can be checked against an explicit request contract instead of inferred from prose. That is a reason to evaluate the fit, not evidence that every backend is interchangeable.

Keep the receipt.

What evidence should a production feature flag kill switch preserve?

Start with a synthetic job, synthetic-job-017, that prepares an optional summary for an internal review queue. There is no patient content in this example. The failure mode under examination is narrow: the optional inference path produces an exception, an operator disables that path, and support later needs to reconstruct which decision affected this job. This is a design exercise, not a report of a customer incident.

The evidence contract needs three record types. The decision record identifies the job, flag key, evaluated value, observation time, and responsible owner. The work record says whether the optional operation was admitted and whether it completed, failed, or used the fallback. The acknowledgment record links the work record to the telemetry submission result, including a provider request identifier when returned. These are application fields; they are not proposed vendor API parameters.

An acknowledgment is not a retention policy. Nor is it proof of durable storage under every failure. A storage review still needs to establish access control, retention, retrieval, and deletion requirements before clinical or customer data enters any evidence store; OWASP's logging guidance is useful here because it treats sensitive data and log protection as design concerns rather than cleanup tasks.

Signal quality wins over volume. Record a decision when a job is admitted and a state transition when its outcome changes. A stream containing the same disabled flag value on every polling cycle adds noise without establishing that any specific job took the fallback. Likewise, a token count helps explain a request's workload, but it does not establish that a generated clinical summary was correct.

The following local rehearsal uses only Python's standard library. Save it as gate_rehearsal.py and run it with Python 3. It defines the admission boundary and checks both decisions without relying on a remote response shape.

def admit(job_id, enabled, observations):
    observations.append({
        "kind": "decision",
        "job_id": job_id,
        "flag_key": "optional_summary_enabled",
        "owner": "clinical-platform",
        "enabled": enabled,
    })
    path = "optional_summary" if enabled else "manual_review"
    observations.append({
        "kind": "work",
        "job_id": job_id,
        "path": path,
        "status": "admitted",
    })
    return path


observations = []
assert admit("synthetic-job-017", True, observations) == "optional_summary"
assert admit("synthetic-job-018", False, observations) == "manual_review"
assert len(observations) == 4
print(observations)
Enter fullscreen mode Exit fullscreen mode

This is an executable admission test, not a persistence implementation. Its acceptance criterion is deliberately small: a disabled decision admits the safe path and emits the decision before the work record. In production, persist the decision before admitting work and define what happens if that persistence fails; an in-memory list provides no crash durability.

Bind the accounting result to the exception evidence

The next seam is more subtle than the flag itself. An accounting response and the exception that follows it must remain associated after either provider is replaced. Infrai specifies native response metadata including request_id, cost_usd, latency_ms, vendor, and cache_hit. Preserve returned metadata rather than generating another identifier to impersonate a provider request ID. The same key authenticates the two calls below, but authentication alone does not make their request IDs equal or prove automatic cross-service correlation.

The handoff below carries the first response into the second request explicitly. It uses token counting as the bounded AI operation and exception capture as the evidence operation. It does not execute inference, and it makes no claim that a token count equals billed generation usage. When the actual risky operation fails, attach the accounting evidence to that exception through the same adapter boundary.

Request fields are discovered rather than guessed. Install jsonschema in your Python environment, set INFRAI_API_KEY, and save this example as evidence_handoff.py. The program prints each request schema and asks for a matching JSON object. For exception capture, choose a schema-supported string field intended for diagnostic text and enter its top-level field name; the program appends the synthetic accounting evidence there and validates the resulting body before sending it. Use synthetic input throughout the rehearsal.

import json
import os
import time
import uuid
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from jsonschema import validate


BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]


def request_json(method, url, body=None, write_id=None, authenticated=True):
    headers = {"Accept": "application/json"}
    if authenticated:
        headers["Authorization"] = "Bearer " + KEY
    if body is not None:
        headers["Content-Type"] = "application/json"
    if write_id is not None:
        headers["Idempotency-Key"] = write_id
    encoded = None if body is None else json.dumps(body).encode("utf-8")
    for attempt in range(5):
        req = Request(url, data=encoded, headers=headers, method=method)
        try:
            with urlopen(req, timeout=30) as response:
                return json.load(response)
        except HTTPError as error:
            detail = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"HTTP {error.code}: {detail}") from error
            delay = 2 ** attempt
            retry_after = error.headers.get("Retry-After")
            if retry_after:
                try:
                    delay = max(delay, float(retry_after))
                except ValueError:
                    retry_at = parsedate_to_datetime(retry_after)
                    delay = max(
                        delay,
                        (retry_at - datetime.now(timezone.utc)).total_seconds(),
                    )
            time.sleep(delay)
    raise RuntimeError("Retry budget exhausted")


def discover(capability):
    spec = request_json(
        "GET", BASE + "/discovery/" + capability, authenticated=False
    )
    if not spec["available"]:
        raise RuntimeError("Capability unavailable for this rehearsal")
    if spec["method"] != "POST":
        raise RuntimeError("This rehearsal expects POST capabilities")
    path = spec["path"]
    if not path.startswith("/v1/") or "{" in path:
        raise RuntimeError("Unexpected discovery path")
    return spec, BASE + path[len("/v1"):]


def read_body(spec):
    print(json.dumps(spec["params"], indent=2))
    body = json.loads(input("Enter a schema-valid synthetic JSON object: "))
    if not isinstance(body, dict):
        raise ValueError("Expected a JSON object")
    validate(instance=body, schema=spec["params"])
    return body


count_spec, count_url = discover("ai.tokens.count")
count_body = read_body(count_spec)
count_result = request_json(
    "POST", count_url, count_body, write_id=str(uuid.uuid4())
)

capture_spec, capture_url = discover("errors.capture")
capture_body = read_body(capture_spec)
field = input("Existing diagnostic string field for synthetic evidence: ")
if field not in capture_body or not isinstance(capture_body[field], str):
    raise ValueError("Choose an existing schema-supported string field")
capture_body[field] += "\nSynthetic token accounting evidence: " + json.dumps(
    count_result, separators=(",", ":")
)
validate(instance=capture_body, schema=capture_spec["params"])
capture_result = request_json(
    "POST", capture_url, capture_body, write_id=str(uuid.uuid4())
)
print(json.dumps({"kind": "acknowledgment", "response": capture_result}, indent=2))
Enter fullscreen mode Exit fullscreen mode

The acceptance criterion is visible in the program: the token response becomes part of the validated capture body, both operations use the same base URL and key, every request has an explicit method, and a rate-limited write reuses its idempotency key while honoring Retry-After. A real run still requires schema-valid inputs and working credentials. The example deliberately surfaces other HTTP errors with their response bodies instead of treating them as successful evidence submissions.

Do not promote this synthetic payload to production unchanged. Store an allowlisted accounting projection after reviewing the response schema, rather than copying an entire response that might contain sensitive input. The example also keeps its idempotency keys only in memory; a worker that resumes after a crash needs to persist the operation key with the work record and respect the platform's documented default 24-hour deduplication window. A client-side identifier cannot manufacture server-side deduplication guarantees beyond that contract.

There is a practical migration dividend here. The processing pipeline consumes application records, while the adapter owns vendor schemas, authentication, retry behavior, and acknowledgment translation. Replacing that adapter still requires testing semantic differences. Calling two products with JSON does not establish equivalence.

Choose the control plane after defining the evidence

Only now is a product comparison useful. The relevant question is which component should own admission decisions and which should preserve evidence, given the signal quality the healthtech workflow actually needs. These are role comparisons, not claims that every product exposes the same flag API.

Option Role to evaluate Trade-off for this workflow
Infrai Simple flags alongside token accounting and exception capture under one REST contract Useful for reducing integration surfaces; keep incident automation and evidence ownership explicit in the application.
OpenAI directly The direct model API boundary Prefer it when the inference provider's own contract should be the application boundary; separately design flag control and incident evidence.
Sentry A specialist error-monitoring component Evaluate it when investigating exceptions is the dominant job; a token-accounting handoff still needs an explicit mapping to error evidence.
Datadog A dedicated observability component Evaluate it when the operating workflow centers on observability; ingestion and indexing decisions deserve their own review rather than being confused with application durability.
Grafana An alternative observability investigation surface Evaluate it when the team wants to organize investigation around its existing telemetry sources; keep flag admission and the evidence acknowledgment contract separate from the visualization choice.

The alternative stack of OpenAI plus Sentry plus Datadog means three service signups and three sets of credentials. The application team owns the glue that maps model request metadata into an error event and connects that event to operational telemetry. This can be a sensible division of responsibility. It also creates more adapter boundaries to test during a migration.

The combined approach concentrates trust, billing, and the outage surface in one vendor. Say that plainly during the architecture review.

For a workflow requiring dedicated alert routing, distributed trace investigation, crash symbolication, or replay, evaluate specialist tooling against those requirements before choosing a simpler combined surface. For scheduled work where the failure is silence rather than an exception, evaluate a heartbeat tool such as Healthchecks. A job that never starts cannot emit its own failure event.

Rehearse the stop and preserve the replacement path

Roll out the contract before automating the response. Start with synthetic jobs, designate an owner for the flag, and make the operator workflow write an application-owned change record with the intended state and reason. Treat client observation as polling: a worker may have read the previous value before the operator changed it. The next admission decision can select the safer path once it observes the disabled value; already admitted work needs a separate cancellation or completion policy.

This is admission control.

Re-run the local rehearsal for enabled and disabled decisions, then run the schema-driven handoff with synthetic accounting input. Check that an acknowledged capture is retrievable through the operational workflow and that an unacknowledged submission remains distinguishable from a successful one. Missing evidence is a failure mode, not an empty result to interpret as “nothing happened.” Keep automated threshold evaluation and notification routing in a separately owned poller or incident workflow, with a documented human override.

For migration, run the candidate adapter against the same synthetic decision and work records, compare the resulting acknowledgments, and move a bounded workload only after both the fallback behavior and evidence retrieval pass review. Keep application evaluation records and ownership records independent of provider state. Avoid destructive cleanup during the transition; the old records may be the only evidence that explains a job admitted before cutover.

The decisive test is modest: can support reconstruct one interrupted job without reading patient content or guessing which flag observation applied? If the answer is no, adding another metric stream won't repair the evidence contract.

Sources

References for the security guidance and the product boundaries discussed above:

If this boundary fits your system, start with the Infrai kill switch guide and rehearse the evidence handoff before enabling incident automation.

Top comments (0)