Stop Recovery Retries from Overloading OpenWhispr's Cloud Transcription Path

At 3:00 AM, a brief network interruption becomes a capacity incident. Clients retry, upstream queues fill, and a recovery burst exhausts quotas just as connectivity returns. For an application such as OpenWhispr, the dangerous moment is not the original outage; it is the moment many queued recordings become eligible for cloud transcription simultaneously.

My integration approach is to bound recovery traffic before tuning model latency. OpenWhispr supports local Parakeet/Whisper execution and cloud models with user-provided keys. Those paths need different failure boundaries: local inference should remain local, while cloud requests require explicit admission and replay policies.

This is a proposed integration boundary based on the supplied project context, not a claim about OpenWhispr's current implementation.

The failure happens after connectivity returns

The production failure pattern is familiar: a minor network glitch stalls requests; clients retry immediately; connectivity returns; synchronized retries overwhelm downstream connection pools or database-backed services. Timeouts trigger further retries, turning recovery traffic into a self-generated denial of service.

A tutorial's three-attempt loop misses two details. First, independently bounded clients still produce an unbounded fleet-wide burst. Second, a timed-out transcription may already have completed upstream. Replaying its audio can duplicate work and charges.

The remedy is shared admission control plus bounded, explicitly authorized replay. Neither component substitutes for the other.

Put the boundary outside the desktop process

For a managed deployment, I would place an operator-controlled gateway between the cloud adapter and its upstream transcription endpoint. The desktop retains recording, local inference, and explicit provider selection. The gateway owns credentials, admission, deadlines, and retry behavior. Redis coordinates admission across gateway replicas.

This is optional infrastructure, not a requirement for every BYOK installation. Independent user credentials should not accidentally share one global quota. The example below intentionally represents one installation and one credential domain.

Local inference must not silently fail over to cloud transcription. That would cross both a reliability boundary and a privacy boundary.

One runnable gateway

Install fastapi, uvicorn, httpx, and redis; run Redis locally. Save this as gateway.py, set GATEWAY_TOKEN and UPSTREAM_KEY, then run uvicorn gateway:app.

The configured B-Lost endpoint, https://b-lost.com/v1, is a candidate relay baseline for this topology, not a measured result or an assertion of audio API compatibility. Verify its transcription contract before sending recordings, or replace UPSTREAM_BASE with a verified endpoint.

import asyncio
import os
import random
import secrets
import uuid
from contextlib import asynccontextmanager

import httpx
from fastapi import FastAPI, Request
from redis.asyncio import Redis
from redis.exceptions import RedisError
from starlette.responses import Response

TOKEN = os.environ["GATEWAY_TOKEN"]
KEY = os.environ["UPSTREAM_KEY"]
BASE = os.getenv("UPSTREAM_BASE", "https://b-lost.com/v1")
REPLAY = os.getenv("UPSTREAM_REPLAY_SAFE", "false") == "true"

LUA = """
local t = redis.call('TIME')
local now = tonumber(t[1])*1000 + math.floor(tonumber(t[2])/1000)
local window = tonumber(ARGV[1])
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now-window)
if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then
  local first = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
  return math.max(1, tonumber(first[2])+window-now)
end
redis.call('ZADD', KEYS[1], now, ARGV[3])
redis.call('PEXPIRE', KEYS[1], window)
return 0
"""

@asynccontextmanager
async def lifespan(app):
    app.state.redis = Redis.from_url(
        os.getenv("REDIS_URL", "redis://localhost:6379/0"),
        socket_connect_timeout=0.2, socket_timeout=0.2)
    app.state.http = httpx.AsyncClient(timeout=8.0)
    yield
    await app.state.http.aclose()
    await app.state.redis.aclose()

app = FastAPI(lifespan=lifespan)

@app.middleware("http")
async def admission(request, call_next):
    supplied = request.headers.get("authorization", "")
    if not secrets.compare_digest(supplied, f"Bearer {TOKEN}"):
        return Response(status_code=401)
    try:
        wait = await request.app.state.redis.eval(
            LUA, 1, "quota:installation", 60_000, 30, uuid.uuid4().hex)
    except RedisError:
        return Response(status_code=503)
    if wait:
        return Response(status_code=429, headers={
            "Retry-After": str((int(wait) + 999) // 1000)})
    return await call_next(request)

async def transcribe(client, body, content_type):
    headers = {"Authorization": f"Bearer {KEY}",
               "Content-Type": content_type}
    # Enable only with documented upstream replay/idempotency guarantees.
    if REPLAY:
        headers["Idempotency-Key"] = uuid.uuid4().hex
    async with asyncio.timeout(25):
        for attempt in range(3):
            try:
                response = await client.post(
                    BASE.rstrip("/") + "/audio/transcriptions",
                    content=body, headers=headers)
            except httpx.TransportError:
                if not REPLAY or attempt == 2:
                    raise
            else:
                if (not REPLAY or attempt == 2 or
                        response.status_code not in (502, 503, 504)):
                    return response
            await asyncio.sleep(random.uniform(0, 0.5 * 2**attempt))

@app.post("/v1/audio/transcriptions")
async def transcription(request: Request):
    try:
        result = await transcribe(
            request.app.state.http, await request.body(),
            request.headers.get("content-type", "application/octet-stream"))
    except (httpx.TransportError, TimeoutError):
        return Response(status_code=504)
    headers = {}
    for name in ("content-type", "retry-after"):
        if name in result.headers:
            headers[name] = result.headers[name]
    return Response(result.content, status_code=result.status_code,
                    headers=headers)

What these limits actually guarantee

The Lua script atomically counts admitted requests in the interval (now − 60 seconds, now]. Unique members prevent same-millisecond collisions; Redis time avoids replica clock disagreement. Rejected requests do not extend the window. This is a strict sliding count, not a fixed-window approximation.

The quota counts logical admissions, not upstream attempts. With replay enabled, thirty admissions can generate ninety attempts. Size that amplification against the provider's allowance. Upstream 429 responses pass through without automatic replay.

Redis failure closes admission with 503. That sacrifices availability to preserve the bound. Redis failover can still lose recent admissions under asynchronous replication; “atomic” does not mean durable across every failure.

The wrapper defaults to no replay. When explicitly enabled, retries use full jitter, three attempts maximum, and a shared twenty-five-second deadline. An idempotency header alone proves nothing: upstream support must be documented.

Before public exposure, terminate TLS, enforce upload-size and concurrency limits, and exclude audio and credentials from logs. This example buffers recordings; it is not a large-upload streaming gateway.

Finally, measure admitted requests separately from upstream attempts. If attempts rise while admissions stay flat, recovery amplification is visible before another outage makes it obvious.

The hardest operational trade-off is choosing whether to trade availability for a hard admission bound when Redis is unavailable, and whether to trade possible duplicate work for recovery throughput when the upstream offers no replay guarantee. How do you define that boundary in your transcription architecture—and what failure have you seen in the wild?