I Built an AI Pipeline That Reads Support Emails and Drafts Replies (Here's What Actually Broke, and the Math on Whether It's Worth It)

I run a one-person dev shop, and I keep hearing the same complaint from small business owners: answering customer support emails eats hours every day, especially when half the questions are variations of the same five things.

So I built a small pipeline: Gmail → Claude API classification → Slack notification with a drafted reply → human approves or edits before anything goes out. This post covers the architecture, three bugs that weren't obvious until I hit them, and — since "AI will save you time" is a claim everyone makes and almost nobody backs with numbers — an actual breakdown of what this costs to run versus what it replaces.

The pipeline, end to end

1. Pull unread messages from Gmail via the Gmail API (messages().list with q="is:unread in:inbox")
2. Extract the body - handling multipart/alternative, nested parts, and HTML fallback (more on why this is harder than it sounds below)
3. Classify the message using Claude with structured output: category, priority, a fine-grained intent label, and a confidence score (0-100)
4. Route to a human based on category and confidence - not just category (more below)
5. Generate a draft reply in the same API call, using the same structured response
6. Post a Slack card (Block Kit) with the classification, the draft, and who it's routed to
7. Mark the email as read and write the result to SQLite, so re-running the pipeline never double-processes a message
8. A human clicks Approve or edits the draft in Gmail directly - nothing sends automatically

Stack: Python 3.9, google-api-python-client, google-auth-oauthlib, requests (no Slack SDK - I hit the Slack Web API directly), SQLite for state, no web framework at all - the CRM view is a plain http.server.BaseHTTPRequestHandler serving a small HTML table. Three external dependencies total.

Classification: structured output, not "please return JSON"

The classification call uses Claude's structured output feature - you pass a JSON Schema directly in the request, and the API guarantees the response matches it. This removes an entire category of bugs that shows up in every "prompt the model for JSON" tutorial: truncated output, extra prose before or after the object, a field that's sometimes a string and sometimes an array. I'm not parsing free text and hoping - the shape is enforced.

The schema:

• category: one of four fixed values (Sales inquiry / Support-technical issue / Billing question / Complaint)
• priority: High / Medium / Low
• intent: a fine-grained snake_case label the model generates freely (sso_redirect_loop_outage, proration_charge_clarification) - useful for spotting patterns later even though it's not used for routing
• confidence: 0-100
• confidence_tier: derived from confidence, not generated by the model directly (see the routing section - this distinction mattered)
• suggested_reply: a drafted response, generated in the same call

One deliberate constraint: the model does not choose who handles the email. It's tempting to let the AI output an assignee name directly, but that means it can hallucinate a person who doesn't work there, or drift between "Sam" and "Samuel" and break your routing table. The model outputs category and confidence; a plain lookup table in code - not the model - decides the assignee. Keep anything with real-world consequences (assigning work to a specific person, sending money, deleting data) in deterministic code, and let the model only produce the judgment calls that a human reviews anyway.

Bug #1: the OAuth scope trap

I started with gmail.readonly + gmail.send. That's correct until you add a "mark as read" step, which needs write access — so I switched to gmail.modify (a superset of readonly).

The trap: don't request both readonly and modify in the same scope list. Google silently drops the redundant one from the actual grant, and the next time your token refreshes - not immediately, which is what made this confusing - google-auth throws a Scope has changed exception with no obvious connection to what you changed. I only tracked it down by reading the google-auth-oauthlib source after the second refresh failure. If you're touching Gmail scopes: request the narrowest single scope that covers everything you need, not the union of several.

Bug #2: text/plain losing to text/html for the wrong reason

Gmail messages are frequently multipart/alternative - the same content exists as both plain text and HTML, and the MIME structure can nest arbitrarily (a multipart wrapping another multipart wrapping the actual parts). My first body-extraction function recursed through the tree and, at each level, fell back to HTML if plain text wasn't found at that level.

The bug: if the HTML part happened to appear before the plain-text part in the tree - which depends on how the sending client built the message, not anything I control - the function returned HTML first, even though a plain text version existed one level deeper.

The fix is two full passes instead of one recursive pass with a fallback mixed in: first, walk the entire tree looking only for text/plain. Only if that complete search comes up empty do you do a second full pass for text/html. The general lesson: when you have a preferred format and a fallback format in a tree structure, don't let the fallback trigger based on local absence - it has to be based on absence across the whole structure.

Bug #3: routing wasn't a category → person table

My assumption going in: Sales → Jordan, Support → Priya, Billing → Sam, Complaints → Alex. Simple lookup. Then one test case broke it - a billing question routed to Alex (the complaints person), not Sam.

The actual rule, once I found it in the test data: anything below a confidence threshold escalates to one specific person, regardless of category. The category table only applies once the model is confident enough; below that, it doesn't matter what the AI thinks the category is - a human with broader context handles it. I'd been treating confidence as purely cosmetic (which icon to show in Slack), not as something that changes the actual routing decision. Once I separated "confidence as display" from "confidence as a routing input," the fix was a few lines - but it wouldn't have surfaced without a test case that specifically checked routing against confidence, not just against category.

This is also why confidence_tier (auto / approval_needed / human_review) is computed in code from the numeric confidence score, not asked of the model directly - the boundaries are a business decision (how much error are you willing to tolerate before requiring a human?), and business decisions belong in code you can read and change without touching the prompt.

Idempotency: making "run it twice" safe

Because this polls Gmail rather than using a webhook, and because a real deployment means "the script might run again after a crash," double processing has to be structurally impossible, not just unlikely:

• Marking a message as read removes it from the is:unread query, so a re-run never sees it again through the normal path
• Independently, the SQLite table has gmail_message_id TEXT NOT NULL UNIQUE, with ON CONFLICT(gmail_message_id) DO UPDATE - so even if something bypasses the first guard, inserting the same message twice updates the existing row instead of creating a duplicate

Two independent guards, not one - because "the email got marked read but the script crashed before writing to the database" is exactly the kind of half-completed state that a single guard doesn't protect against.

Now the part that actually matters: does this save money?

Here's the honest math, using numbers you can substitute your own values into.

What it costs to run:
The system prompt is about 3,100 characters (~840 tokens), sent on every call along with the email body. At Claude Sonnet's current pricing ($3/MTok input, $15/MTok output), a typical classification-plus-draft call - roughly 1,050 input tokens and 200 output tokens - costs approximately $0.0064 per email. Round up to $0.01 to be safe.

What it replaces:
Industry surveys on support ticket handling time put a straightforward inquiry (the kind this pipeline is built for - not novel or complex issues) at somewhere between 3 and 8 minutes of staff time: reading, understanding intent, checking context if needed, and writing a reply from scratch. Take the low end - 4 minutes - and a loaded hourly cost for support staff around $25/hr (US small-business range, fully loaded with overhead), and one manually-answered routine email costs roughly $1.67 in labor.

The comparison:
$0.01 per email in API cost versus ~$1.67 in labor for the routine cases this is built for - a difference of two orders of magnitude. The pipeline doesn't remove the human (every reply is still reviewed and approved before sending), but it collapses "read, understand, draft from scratch" down to "read a pre-classified summary and a drafted reply, then approve or edit" - which is a few seconds of review instead of minutes of composition, even before counting the API cost difference.

Where this breaks down: this math only holds for the routine, high-confidence cases - which is exactly why the confidence_tier routing exists. Complex or ambiguous emails fall through to human_review and get full human handling with no shortcut, because getting a complex case wrong costs far more than the minutes saved. The system is designed to be conservative about what it treats as "routine" rather than to maximize the percentage automated - across the 8 test scenarios I validated the confidence thresholds against, that split lands around half auto-approvable and half requiring a human look, which feels like the right ratio for a first deployment rather than something to be tuned aggressively for automation percentage.

What I'd change for a production deployment

• Prompt caching: the system prompt is fixed across every call and close to (but under) the ~1,024 token minimum cacheable length for Sonnet. Padding it slightly to clear that threshold would cut input cost further on any real volume.
• Webhook instead of polling: users().watch() with Cloud Pub/Sub instead of a batch run - relevant once this isn't a demo run on-demand but something meant to react within seconds of an email arriving.
• Slack rate limiting: chat.postMessage is limited to roughly one message per second per channel. At demo volume this doesn't matter; at real volume you need a delay between posts and a retry against Retry-After when you get a 429 - both already in place here, sized for a demo rather than tuned for a specific real load.

Try it

I put together a live version of this - the full flow, including the Slack card and the confidence-based routing. If your inbox has the "half of these are the same five questions" problem, this is the kind of automation work I do - happy to share the demo link and talk through it, just drop a comment.