Build a Refusal Gate in Five Stages, Then Break It

Picture a support queue on a Tuesday morning. An agent reads one ticket and drafts a refund promise. The ticket is missing the order state. Nobody notices until the customer replies.

That draft was a decision made with missing evidence. The decision owner was the support lead. The consequence landed on the customer. The point of reversibility was the send button.

So I stopped asking whether the model was trustworthy. I started asking where the refusal should live. A gate is cheaper than an apology. Here is the harness, in five stages, each with its own check.

Stage 1: Write the stop condition as data, not prose

Rules in a wiki get read once. Rules in a file get executed. Put yours in stop_conditions.json.

[
  {
    "id": "refund-promise",
    "trigger": "draft contains a refund commitment",
    "required_evidence": ["customer_id", "order_state", "refund_policy_id"],
    "owner": "support-lead",
    "allowed_action": "draft_only",
    "recovery": "ask_customer_for_order_id"
  }
]

Four fields matter most: the trigger, the evidence, the owner, the recovery. If you cannot name one owner, you have a policy, not a rule.

Verification: python -m json.tool stop_conditions.json > /dev/null && echo "schema ok" should print schema ok and exit zero. If this file is empty, your gate is a rubber stamp.

Stage 2: Stand up the harness on free model access

I keep a scratch server for gate work. The point is a machine you can break without breaking production.

MonkeyCode, an open-source project I use for this kind of experiment, advertises free model access and a free server option. Those are operator-supplied availability claims, and I treat them as availability, not as a promise. Check the current terms before you depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

ssh you@your-free-server
python3 -V                 # 3.9+ for the stdlib calls below
git clone <your-fork-url> stop-gate && cd stop-gate
python3 -m venv .venv && . .venv/bin/activate
export MONKEYCODE_API_KEY="<from your console>"
export MODEL_BASE_URL="<from your console>"
export MODEL_ID="<pick one your console lists>"

I am not naming models here, because the list changes and my naming it would age badly.

Verification: python -c "import os; print(bool(os.environ['MODEL_BASE_URL']))" prints True. One working transport call is enough for the whole harness.

Stage 3: Put the gate in front of the model

The gate checks evidence before any model call. A refusal therefore costs zero tokens.

# gate.py — reference implementation, stdlib only
import json, os, urllib.request

STOP = json.load(open("stop_conditions.json"))

def transport(prompt, base, key, model):
    """Fill in the request shape your console documents. This is a placeholder."""
    body = json.dumps({"model": model, "input": prompt}).encode()
    req = urllib.request.Request(
        base, data=body,
        headers={"Authorization": f"Bearer {key}",
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)

def coverage(case, rule):
    have = set(f for f in rule["required_evidence"] if case.get(f))
    return have, set(rule["required_evidence"]) - have

def decide(case, call_model=None):
    for rule in STOP:
        _, missing = coverage(case, rule)
        if missing:
            return {"decision": "REFUSE", "rule": rule["id"],
                    "missing": sorted(missing), "owner": rule["owner"],
                    "recovery": rule["recovery"], "model_called": False}
    text = call_model(case) if call_model else None
    return {"decision": "DRAFT", "rule": None, "missing": [],
            "output": text, "model_called": call_model is not None}

Now the part people skip. Prove the refusal never reaches the model.

# test_gate.py
from gate import decide

def boom(_):
    raise AssertionError("model must not be called on a refusal")

case = {"customer_id": "c-88", "draft": "we will refund you"}
out = decide(case, call_model=boom)
assert out["decision"] == "REFUSE", out
assert out["missing"] == ["order_state", "refund_policy_id"], out
print("refusal gate ok:", out["owner"], out["recovery"])

Verification: python test_gate.py prints refusal gate ok: support-lead ask_customer_for_order_id. When a refusal still calls the model, you have not built a gate. You have built a comment.

Stage 4: Keep the discarded information in the record

A refusal is data, not silence. Log what was missing, who owns it, and how a human recovers.

def record(path, case_id, rule, missing, decision, owner, recovery, ts):
    line = {"case_id": case_id, "rule": rule, "missing": sorted(missing),
            "decision": decision, "owner": owner,
            "recovery": recovery, "ts": ts}
    with open(path, "a") as fh:
        fh.write(json.dumps(line) + "\n")

Verification: jq -r 'select(.decision=="REFUSE") | .missing[]' decisions.jsonl | sort | uniq -c | sort -rn groups the blockers by count.

 14 order_state
  9 refund_policy_id

Fourteen refusals, one field. Fix the intake form instead of the prompt. That is the whole argument for logging last.

Stage 5: Show the refusal card to a human

The card has four lines, in this order: what the agent wanted to do, which evidence is missing, who can unblock it, what approving anyway means. Focus order must match that reading order. No color-only signals. Plain words, no system names.

My review protocol is small and repeatable. Two reviewers read one card aloud, one of them with a screen reader. I count hesitations and time how long it takes each to name the owner and the recovery path.

Success measure: every refusal card names exactly one owner and one recovery path within ten seconds. Stop measure: any approval that proceeds while a required field is empty. When that happens, halt the release and fix the rule. Never relax the fixture.

What this harness does not prove

It does not prove the model is safe, only that one class of action waits for evidence. Free access is availability, not a guarantee, and terms can change.

Skip this if a regulator requires a contractual model version pinned to a document. Skip it for sub-second interactive flows, because the gate writes a record. Skip it if your team cannot name a single decision owner. Anonymous refusals teach nobody anything.

So which missing piece of evidence should stop your approval this week? Write that one rule, run the fixture, and let the gate refuse for you.

If you want a scratch machine for the fixture, the free server option in MonkeyCode will run this harness end to end.