DEV Community

T.M. Gunderson
T.M. Gunderson

Posted on

The Confidence Line: How to Let AI Sort Your Inbox Alone

Letting AI sort your inbox isn't a new idea. What's blocked it for years isn't the sorting — it's the trust problem. An AI label comes back looking exactly the same whether it's right or wrong, so someone still has to check everything, which defeats the point.

On November 11, Google shipped something that changes the shape of that problem: the Gemini Decisions API, a model that doesn't write anything at all. You hand it one input — an email, a photo, an audio clip, a PDF, or several together — and up to 1,000 fixed-answer questions in a single pass. It scores every allowed answer and returns probabilities plus a confidence score, in 150–250 ms end-to-end, at $0.10 per million input tokens with zero output tokens billed. There's a no-cost tier in Google AI Studio for prototyping, so you can test your own sorting job before spending anything.

The confidence score is not decoration. It's the whole product — because it's the piece that tells you when not to trust the answer.

The evidence: the score actually separates right from wrong

Skeptics should be skeptical here — "the vendor says the confidence score is calibrated" is exactly the kind of claim that needs independent checking. So it helps that a developer ran one: an independent 48-ticket benchmark published on dev.to on November 13, using hand-labeled support tickets (30 clear cases, 18 deliberately hard ones — mixed topics, sarcasm, German and Spanish, a chargeback threat, a ticket trying to instruct the classifier).

The test: split every answer at a confidence of 0.7 — the threshold Google's own documentation uses as an example — and check correctness on each side.

Question Confidence ≥ 0.7 Confidence < 0.7
Which team owns it 41 of 42 correct 5 of 6
Is a refund being requested 46 of 46 correct 1 of 2
How urgent is it 23 of 23 correct 8 of 25

Read the urgency row again. Every answer above the line was right. Every one of the model's 17 urgency mistakes landed below the line. The score found its own errors.

That's what makes automation possible: route everything at 0.7 and above automatically, and send the rest to a human. In that benchmark, the team question cleared the line on 42 of 48 tickets — meaning ~88% of routing happens with no human touch, and the pile a person reviews is small enough to actually review. (The urgency question cleared on only 23 of 48 — the line is per-question, not one number for everything. Set it per job.)

The catch that makes the story honest

Here's the part most coverage will skip. On raw accuracy — getting all three questions right per ticket — the Decisions API came in last of the five backends tested: 29 of 48, versus 40 of 48 for a generative model asked the same questions with a JSON schema. A naive read says: worse model, skip it.

The naive read is wrong, because it compares answers that aren't the same product. The generative models return one pick per question and nothing else — no probabilities, no confidence, no way to know which of their 40 correct answers to trust and which to check. The decision model returns the answer plus the number that tells you. Compare what's enforceable — accuracy above a line — and the generative models have no line at all.

The cost and speed side isn't close, either. The whole 48-ticket Decisions run cost $0.0018 (17,950 input tokens, zero output tokens billed) at a median 202 ms per ticket — 3.5x faster than Gemini 3.5 Flash-Lite, ~10x faster than the bigger models, which also billed thousands of output tokens writing their answers. One useful quirk from the error data: when the Decisions model missed urgency, it always rated one level too high, never too low — the cheaper direction to be wrong in for triage.

The drill: write your confidence line

This is the actionable part, and none of it requires a developer until the last step.

1. Pick one sorting job. Support inbox triage, lead routing, review tagging, ticket prioritization. One job, not five.

2. Write the questions with fixed answers. "Which team owns this: billing / scheduling / technical / other?" Add the catch-all, and route it to a human — a model forced into your categories will mislabel the weird stuff confidently.

3. Pull 50 real examples you already know the answers to. Last month's tickets, leads, or reviews — anything with a known correct label.

4. Run them and sort by confidence. Don't assume 0.7 is your number. Look for the clean break — the point below which errors cluster. That break is your line. If there's no clean break, this job isn't ready to automate, and you just found that out for pennies.

5. Wire the rule: above the line acts, below the line waits for a human. Then watch the below-the-line pile weekly. If it grows, the questions have drifted from reality and need rewriting.

Two boundaries worth respecting. First, keep this tier to sorting and routing — never let it carry a customer-facing reply. It can't write, and empathy, nuance, and tone are exactly the half of support that still costs money. Use it to decide which conversations need a human; use a human (or a generative model with review) for the reply. Second, one 48-ticket benchmark is evidence, not proof — it's the drill in step 3–4 that turns "calibrated, they say" into "calibrated, on my data." Re-run it when the vendor updates the model.

Where this fits

The pattern to take away isn't "Google's new API" — vendors and prices will churn. It's the discipline: never automate an AI decision without a number that tells you when to check it. If a tool can't produce that number, it isn't ready to sort anything unattended.


If you want that discipline pointed at the right jobs — which inbound streams you actually have, what's already touching AI in your business, and which one to wire first — that's exactly what the AI Clarity Sprint maps in two weeks, for a fixed fee of $1,495 CAD. The sprint ends with a prioritized plan you own: the confidence-line drill works best when it starts on the one job that pays for itself first.

Top comments (0)