DEV Community

缔造
缔造

Posted on

I built a gateway that retries LLM refusals until I get a real answer

#llm #python #fastapi #openai

If you work with LLM APIs you know the dance: you ask for working code, and you get a lecture. A disclaimer. Three paragraphs of "as an AI I cannot" followed by a suspiciously detailed explanation of why it cannot.

I got tired of it, so I built armor-gw — a self-hosted, OpenAI-compatible proxy that sits between your app and any upstream model and refuses to accept refusals.

How it works

Every response passing through the gateway gets inspected by 13 detector classes. Not just the obvious "I can't help with that" — also the sneaky ones:

  • The substitute answer ("instead, here are some security best practices...")
  • The lecture delivery (2000 words of hedging with zero code blocks)
  • The silent omission (a 4000-word prompt where the model answers everything except the one line you actually asked for)
  • The persona-wrapped refusal (a jailbreak shell gets the model to roleplay, and the refusal comes back dressed as character dialogue)

When a refusal is detected, the request gets reformulated by one of 62 mutation operators and retried — while a second variant from a different strategy family races it in parallel. First clean answer wins.

The strategy layer is a Thompson-sampling bandit that learns per-model which approach actually works: persona framing, encoding tricks, or inversion techniques. A diagnostic judge labels each refusal with its cause (policy / safety-speak / capability-doubt / evasion) so the next retry targets the actual failure mode.

What your app sees

A normal OpenAI endpoint. Drop-in compatible:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8787/v1", api_key="sk-your-key")
r = client.chat.completions.create(model="your-upstream-model",
                                   messages=[...])
Enter fullscreen mode Exit fullscreen mode

Multiple upstreams, per-model routing, failover. The console UI shows every retry round live — watching it classify a refusal, pick a strategy, and break through on round 3 is honestly a little addictive.

The interesting engineering problems

  1. Detecting refusals is harder than it looks. Keyword lists fail immediately — models rephrase. The most reliable signal turned out to be entity survival rate: if the technical nouns from your request never appear in the response, nobody answered your question no matter how confident the prose sounds.

  2. Output hygiene matters as much as breaking through. The gateway strips chain-of-thought leaks, disclaimers, filler questions, and "want me to elaborate?" endings. The deliverable is the deliverable.

  3. Strategies decay. Upstream models get quietly updated, and an operator that worked for weeks stops working. There's a regression bank (65 seeded test cases) that runs on demand and flags any topic whose success rate drops 20%.

Try it

Self-hosted, MIT, one-command deploy, runs on anything with Python 3.9+ (I develop on an aarch64 Alpine box):

github.com/Dixxxzao-01/armor-gw

Feedback welcome — especially which upstreams you'd want supported out of the box, and what refusal patterns you're hitting that the detectors miss.

Top comments (0)