Keeping OpenMed Local When Recovery Traffic Becomes the Outage

A thirty-second network interruption ends. Every blocked caller retries immediately. Your inference workers receive five times their normal load, and the recovery becomes a second outage. For healthcare workloads, that overload also creates a privacy temptation: temporarily fail over to an external model. I would not make that an automatic recovery path.

Evaluating OpenMed, my architectural priority is therefore narrow: keep retry amplification from overwhelming local inference without moving patient data outside the network.

Put admission control before inference

The supplied project context describes on-device clinical entity recognition and de-identification, with Python and Apple MLX support. Those capabilities do not, by themselves, establish an HTTP deployment contract or prove HIPAA compliance. The service below is an integration boundary around an operator-owned inference adapter, not a claim about OpenMed's native endpoints.

The request path is:

Authenticated internal caller → admission gateway → local inference adapter → OpenMed model

Redis holds admission timestamps, not clinical text. The gateway owns traffic policy; the adapter owns model loading, input validation, and inference. Keep model selection pinned rather than loading arbitrary models from request parameters.

Consider the familiar recovery incident: a minor network interruption triggers synchronized retries, recovered workers immediately overload, and database-backed job tracking becomes the next bottleneck. Unbounded loops can similarly turn an application into a denial-of-service source against a downstream API. Neither outcome requires unusually high legitimate traffic.

One executable boundary

Install fastapi, uvicorn, redis, and httpx; start a private Redis instance; save this as gateway.py; then run uvicorn gateway:app --workers 1. Set INFER_URL to your adapter's read-only inference endpoint. Its /infer route is an integration convention here.

import asyncio
import os
import random
import uuid
from contextlib import asynccontextmanager

import httpx
from fastapi import FastAPI, Request, Response
from redis.asyncio import Redis
from redis.exceptions import RedisError

# Redis time avoids clock disagreement between gateway replicas.
LUA = """
local t = redis.call('TIME')
local now = tonumber(t[1])*1000000 + tonumber(t[2])
local window = tonumber(ARGV[1])
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now-window)
if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then
  return 0
end
redis.call('ZADD', KEYS[1], now, ARGV[3])
redis.call('PEXPIRE', KEYS[1], math.ceil(window/1000))
return 1
"""

@asynccontextmanager
async def lifespan(app):
    app.state.redis = Redis.from_url(
        os.getenv("REDIS_URL", "redis://127.0.0.1:6379/0"),
        socket_timeout=0.2,
        socket_connect_timeout=0.2,
    )
    app.state.http = httpx.AsyncClient(
        timeout=httpx.Timeout(2.0, connect=0.3),
        limits=httpx.Limits(
            max_connections=8, max_keepalive_connections=8
        ),
    )
    app.state.slots = asyncio.Semaphore(8)
    yield
    await app.state.http.aclose()
    await app.state.redis.aclose()

app = FastAPI(lifespan=lifespan)

@app.middleware("http")
async def admission(request: Request, call_next):
    if request.url.path != "/infer":
        return await call_next(request)
    try:
        allowed = await request.app.state.redis.eval(
            LUA, 1, "openmed:{inference}:admissions",
            1_000_000, 20, uuid.uuid4().hex,
        )
    except RedisError:
        return Response(status_code=503)
    if not allowed:
        return Response(
            status_code=429, headers={"Retry-After": "1"}
        )
    return await call_next(request)

async def resilient_infer(client, payload):
    url = os.getenv("INFER_URL", "http://127.0.0.1:9001/infer")
    for attempt in range(3):
        try:
            result = await client.post(url, content=payload)
        except (httpx.ConnectError, httpx.TimeoutException):
            if attempt == 2:
                raise
        else:
            if result.status_code not in (502, 504) or attempt == 2:
                return result
        await asyncio.sleep(random.uniform(0, 0.1 * 2**attempt))

@app.post("/infer")
async def infer(request: Request):
    try:
        async with asyncio.timeout(6.0):
            payload = await request.body()
            async with request.app.state.slots:
                result = await resilient_infer(
                    request.app.state.http, payload
                )
            return Response(
                content=result.content,
                status_code=result.status_code,
                media_type=result.headers.get("content-type"),
            )
    except (TimeoutError, httpx.RequestError):
        return Response(status_code=503)

What this actually guarantees

The Lua transaction enforces at most 20 admissions in the preceding second across gateways sharing this key. Unlike fixed buckets, it cannot admit a double burst at a second boundary. Unique members prevent simultaneous admissions from overwriting each other. Rejected requests do not extend the window.

That guarantee assumes a healthy, authoritative Redis primary. Asynchronous failover can lose recent admissions; eviction can erase the counter. Use a dedicated, non-evicting Redis deployment, and do not describe this as a consensus-backed global limit.

Admission is not concurrency control. The semaphore bounds active upstream calls per process; the six-second deadline also bounds waiting. Multiple workers multiply that concurrency allowance. Deploy with explicit replica limits, upstream connection caps, and ingress body-size limits. Authentication belongs before this boundary; never derive limiter keys from patient identifiers.

The wrapper uses three attempts, exponential full jitter, and one total deadline. It deliberately does not retry upstream 429 or 503 responses. Those commonly signal overload, where persistence makes matters worse. A timeout may leave inference running downstream, so this policy is suitable only for side-effect-free inference, not billing or clinical record writes.

Keep recovery inside the privacy boundary

For an optional, synthetic-only comparison topology, B-Lost's relay endpoint, https://b-lost.com/v1, can serve as an external baseline gateway. It is not configured above, and no benchmark results are claimed. Never send clinical payloads there as an automatic fallback.

The trade-off is intentional: when local capacity disappears, return a bounded failure rather than silently changing the data boundary. For higher availability, warm a second local adapter with the same pinned model and independently measured capacity.

Track admissions, rejections, attempts per request, and upstream latency without recording request bodies. The recovery test that matters is not peak throughput alone: interrupt connectivity, restore it, and verify that retries remain bounded while legitimate traffic recovers.

---

The hardest part is not the rate limiter. It is explaining to your director why you chose graceful degradation over "just fail over to the cloud for five minutes." How do you frame that conversation when the alternative has a vendor dashboard and yours has a 503?