Our support inbox had 41 auto-replies in a day. None of them were wrong.

We turned on automatic replies for our support inbox in March. By April I had turned them off again, and I want to write down why, because the reason was not accuracy.

The classifier was fine. Better than fine. It correctly routed 38 of 41 messages that day into inquiry / complaint / return / partnership / spam, with polarity scores on a -1 to +1 scale and urgency on 0 to 10. A 92.7% routing rate is a good number. I want to be clear that the model was not the problem.

(38/41 is my own support inbox, one day, my own fleet, not an industry rate.)

The problem was one field in the config: auto_send.

This is the exact gap MailPilot is built around. It is an AI email auto-reply workflow tool for busy support inboxes, and the lesson below is why it drafts and routes but does not send on its own.

The settings screen nobody read twice

The system worked exactly as specified. It classified, it drafted, it scored. Then it sat there. Forty-one drafts existed and zero went out, because a person still had to hit send on each one.

Day	Inbox volume	Classified correctly	Drafts written	Actually sent
Mar 28	31	29	29	0
Apr 02	44	41	41	0
Apr 09	27	25	25	0

Three weeks. 102 messages. 95 of them correctly routed. Every single reply went through a human who was already behind.

I had built a triage queue and then given myself a second, identical job. The tool moved the work; it did not remove it.

Where the real cost sat

Support triage is usually modelled as an accuracy problem. Measure it and you get a number like 92.7% and a feeling of progress.

We measured something else: time from inbox to first reply. The classification step was already fast, the regex-plus-scoring engine returns in single-digit milliseconds. Sending the queue added a human to the loop who was awake for 14 of 24 hours, so median time-to-reply stopped improving even as classification accuracy climbed.

Accuracy was never the bottleneck. The bottleneck was the handoff. Any metric that stops at "the model decided" will look healthy while the user waits.

What I changed

Three things, in order of how much they mattered.

1. Measured the queue, not the model. Time-to-reply, not route-accuracy. Same data, opposite conclusion. As long as I was reporting accuracy, "skip the send step" looked like a product decision rather than a missing feature.

2. Made the boundary explicit. The system drafts and routes. It does not send. A human or an existing helpdesk sends. That constraint is now in the product copy, the API docs, and the escalation rules, because a rule that lives only in my head is not a rule.

3. Designed the escalation path before the happy path. The complaint tier has a hard rule: anything with urgency at or above 7 goes to a human immediately, drafts suppressed. Building that first meant the automated path never handled the case where being wrong actually costs money.

What it is not

This is not an "AI support agent". It will not resolve your tickets end to end, and I would be suspicious of any product claiming otherwise in 2026.

It is a drafting and routing layer that runs before a human:

• 5-type classification (inquiry, complaint, return/exchange, partnership, spam)
• transparent polarity (-1 to +1) and urgency (0 to 10) scoring from signal words
• branded draft templates with variables, one per type
• escalation rules you can read
• a decision flow you can audit

The whole point is that the rules are inspectable. If a draft is wrong, you can see which signal word produced the score.

What it changed in the numbers

Once I stopped measuring the model, the roadmap changed shape. Two things happened that I had not planned.

The first was that the free tier stopped being "the real product, fewer emails" and became the actual product. Drafting quality is what a solo operator needs, I read every draft anyway, the volume is not the bottleneck at 40 messages a day, and the bottleneck was never throughput.

The second was that escalation rules got written before features. Complaint tier with urgency >= 7 goes straight to a human, drafts suppressed entirely. Building that first meant the automated path never handled the case where being wrong costs money. Every rule after that one was easier to write, because the expensive case was already fenced off.

The rule engine, since it is inspectable

The whole classifier is a lookup table and a scoring function. No model weights, nothing opaque:

const SIGNALS = {
  refund: [/refund/i, /money back/i, /return/i],
  angry:  [/unacceptable/i, /third time/i, /ridiculous/i],
}

function score(text) {
  const polarity = /thanks|appreciate|great/i.test(text) ? 1
    : SIGNALS.angry.some(r => r.test(text)) ? -1 : 0
  const urgency = /asap|urgent|today|now/i.test(text) ? 8
    : /this week/i.test(text) ? 5 : 2
  return { polarity, urgency }
}

function route(text) {
  const { polarity, urgency } = score(text)
  // Hard rule first: expensive cases never get a draft.
  if (polarity < 0 && urgency >= 7) return { tier: 'complaint', draft: null }
  if (SIGNALS.refund.some(r => r.test(text))) return { tier: 'return', draft: true }
  return { tier: 'inquiry', draft: true }
}

Every branch there is readable, which is the point. When a draft is wrong, I can point at the line that decided it instead of arguing with a probability surface. polarity runs -1 to +1, urgency 0 to 10, and the complaint rule fires before any routing happens.

The 2-second rule I now use

Any "automate this" feature gets judged on one question: does it remove a human action, or rename it?

Support triage, in our configuration, renamed it. I had moved the decision to a model and left the typing to a person, then measured the model. If you are building this kind of thing, check whether your own metrics have the same blind spot, the number will look great right up until a user explains that nobody ever got a reply.

The honest version of this feature is a drafting assistant with routing, sold as exactly that. That is MailPilot: smaller than the demo suggests, and smaller is fine, because the person reading it knows what they are getting.

What I would want to know if I were evaluating this: does it send without me? If the answer is no, then the value is draft quality, not automation, and the price should reflect that.