Confidence-gated auto-send: when your support agent should escalate to a human

Most support-bot demos optimise for the wrong number: how many tickets the bot answers. The number that actually hurts you is how many wrong answers reach a customer. A bot that answers 40% of tickets correctly and escalates the rest is useful. A bot that answers 95% and is wrong on 5% is a refund-policy lawsuit generator.

So the design question is not "how do I make the model answer?" but "when should the model be denied the send button?" Below is a small gate I use, in Python, with a RAG-backed answerer. The principle is: auto-send is the exception that must be earned; escalation is the default.

The decision object

Every ticket ends in one of two actions, and we keep the reason, because reasons are what you debug later:

@dataclass
class Decision:
    action: str            # "auto_send" | "escalate"
    answer: str
    citations: list[str]
    confidence: float
    triage: Triage
    reason: str = ""
    sources: list[str] = field(default_factory=list)

Gate 1: no evidence, no answer

Retrieve documents first. If nothing clears a minimum similarity score, escalate before the LLM is even called:

hits = [h for h in store.search(f"{subject} {body}", k) if h[1] >= min_retrieval]
if not hits:
    return Decision("escalate", "", [], 0.0, tri, "no relevant documents")

This is cheaper and safer than asking the model to be modest. Models are bad at saying "I do not know" when a plausible answer is available.

Gate 2: force cited, structured output

The prompt tells the model to answer only from the retrieved docs, to cite document ids, and to return an explicit empty answer if the docs do not help. The customer's text is wrapped as untrusted data:

def build_prompt(subject, body, hits) -> str:
    ctx = "\n".join(f"[DOC {d.id}] {d.text}" for d, _ in hits)
    return ("[ROLE:answerer]\nAnswer ONLY from the documents below. Cite doc ids. If they do not "
            'answer the question, reply {"answer": "", "citations": [], "confidence": 0}. '
            'Reply JSON: {"answer": str, "citations": [ids], "confidence": 0..1}\n'
            f"DOCUMENTS:\n{ctx}\nTICKET:\n<untrusted_data>{subject}\n{body}</untrusted_data>")

Unparseable output? Escalate. An empty answer? Escalate ("model declined" is a perfectly good outcome).

Gate 3: reject citations the retrieval never produced

This is the one people skip. A model can happily cite [refunds-v2] when you only retrieved refunds and shipping. Verify citations against the ids you actually put in the context:

valid = {doc.id for doc, _ in hits}
if not cites or not set(cites) <= valid:
    return Decision("escalate", answer, cites, conf, tri, "missing or invalid citations")

An uncited answer, or one citing a phantom document, never goes out. The draft answer is still attached to the decision, so a human can start from it.

Gate 4: hard rules, then the threshold

Some categories should not be automated no matter how confident the model sounds: urgent tickets, legal threats, chargebacks. Triage assigns a priority up front, and urgent always escalates. Only then do we apply the confidence threshold:

if tri.priority == "urgent":
    return Decision("escalate", answer, cites, conf, tri,
                    "urgent tickets always go to a human", sorted(set(cites)))
if conf < threshold:
    return Decision("escalate", answer, cites, conf, tri,
                    f"confidence {conf:.2f} below threshold {threshold:.2f}", sorted(set(cites)))
return Decision("auto_send", answer, cites, conf, tri, "", sorted(set(cites)))

A caveat worth being loud about: model-reported confidence is a signal, not calibrated probability. That is exactly why gates 1 to 3 are deterministic checks that do not rely on the model's self-assessment, and why gate 4's threshold needs measuring, not guessing.

Measuring it: wrong-answer rate among auto-sent replies

Plain accuracy hides the thing you care about. The metric that matters is: of the replies we sent, how many were wrong? Escalating a ticket the bot could have answered is a miss, and costs a human a few minutes. Auto-sending a bad answer costs far more. So track them separately:

@dataclass
class EvalReport:
    total: int = 0
    auto_sent: int = 0
    wrong: int = 0
    escalated: int = 0
    missed: int = 0

    @property
    def wrong_answer_rate(self):
        return self.wrong / self.auto_sent if self.auto_sent else 0.0

Golden cases declare what should happen:

cases:
  - {id: g1, subject: "How do I get a refund?", body: "Please refund my purchase from last week",
     expect: answer, must_contain: ["30 days"], must_cite: "refunds"}
  - {id: g5, subject: "Do you support dark mode on the moon base?",
     body: "Asking about lunar colony licensing", expect: escalate}
  - {id: g6, subject: "Refund URGENT chargeback", body: "ASAP refund or I file a chargeback", expect: escalate}

An auto-sent reply counts as wrong if the case expected escalation, if the answer lacks a required fact, or if it cites the wrong document. Now tuning is a real trade-off you can see: raise the threshold and the wrong-answer rate falls while misses rise; lower it and the reverse. Pick the point where your team is comfortable and re-measure when your docs or model change.

A note on the golden set: the one in the example above is synthetic and tiny. Yours should be built from real, anonymised tickets, and a bigger sample than seven. Do not read anything into a toy set's numbers.

Other things the gate needs around it

• Redact PII before it leaves your boundary. Strip emails, card numbers and similar from ticket text before calling a third-party LLM.
• Keep a human sampling auto-sent replies. The gate reduces wrong answers; it does not eliminate them.
• Log the reason string. "confidence 0.55 below threshold" and "missing or invalid citations" tell you which lever to pull.

The packaged version

This gate is the core of the Support Agent Kit in my Agent Forge Pro line: triage, a retrieval store (with an optional pgvector path), the cited-answer prompt, the gate, an offline mock LLM so the tests run without an API key, and the golden-set eval above. It is $197 (the core runtime is $297, SDR $197, doc-review $247, data analyst $147, bundles $497 and $597). MIT licensed for your own use, no resale, with a 30-day money-back guarantee. A free sample is available here: free sample.

Store: Agent Forge Pro

The pattern itself is free, though: retrieve, require citations, verify them, hard-escalate the risky categories, threshold the rest, and measure wrong answers among what you send.