DEV Community: Zero AI Developer

I Built an AI Pipeline That Reads Support Emails and Drafts Replies (Here's What Actually Broke, and the Math on Whether It's Worth It)

Zero AI Developer — Sat, 19 Sep 2026 06:01:39 +0000

I run a one-person dev shop, and I keep hearing the same complaint from small business owners: answering customer support emails eats hours every day, especially when half the questions are variations of the same five things.

So I built a small pipeline: Gmail → Claude API classification → Slack notification with a drafted reply → human approves or edits before anything goes out. This post covers the architecture, three bugs that weren't obvious until I hit them, and — since "AI will save you time" is a claim everyone makes and almost nobody backs with numbers — an actual breakdown of what this costs to run versus what it replaces.

The pipeline, end to end

Pull unread messages from Gmail via the Gmail API (messages().list with q="is:unread in:inbox")
Extract the body - handling multipart/alternative, nested parts, and HTML fallback (more on why this is harder than it sounds below)
Classify the message using Claude with structured output: category, priority, a fine-grained intent label, and a confidence score (0-100)
Route to a human based on category and confidence - not just category (more below)
Generate a draft reply in the same API call, using the same structured response
Post a Slack card (Block Kit) with the classification, the draft, and who it's routed to
Mark the email as read and write the result to SQLite, so re-running the pipeline never double-processes a message
A human clicks Approve or edits the draft in Gmail directly - nothing sends automatically

Stack: Python 3.9, google-api-python-client, google-auth-oauthlib, requests (no Slack SDK - I hit the Slack Web API directly), SQLite for state, no web framework at all - the CRM view is a plain http.server.BaseHTTPRequestHandler serving a small HTML table. Three external dependencies total.

Classification: structured output, not "please return JSON"

The classification call uses Claude's structured output feature - you pass a JSON Schema directly in the request, and the API guarantees the response matches it. This removes an entire category of bugs that shows up in every "prompt the model for JSON" tutorial: truncated output, extra prose before or after the object, a field that's sometimes a string and sometimes an array. I'm not parsing free text and hoping - the shape is enforced.

The schema:

category: one of four fixed values (Sales inquiry / Support-technical issue / Billing question / Complaint)
priority: High / Medium / Low
intent: a fine-grained snake_case label the model generates freely (sso_redirect_loop_outage, proration_charge_clarification) - useful for spotting patterns later even though it's not used for routing
confidence: 0-100
confidence_tier: derived from confidence, not generated by the model directly (see the routing section - this distinction mattered)
suggested_reply: a drafted response, generated in the same call

One deliberate constraint: the model does not choose who handles the email. It's tempting to let the AI output an assignee name directly, but that means it can hallucinate a person who doesn't work there, or drift between "Sam" and "Samuel" and break your routing table. The model outputs category and confidence; a plain lookup table in code - not the model - decides the assignee. Keep anything with real-world consequences (assigning work to a specific person, sending money, deleting data) in deterministic code, and let the model only produce the judgment calls that a human reviews anyway.

Bug #1: the OAuth scope trap

I started with gmail.readonly + gmail.send. That's correct until you add a "mark as read" step, which needs write access — so I switched to gmail.modify (a superset of readonly).

The trap: don't request both readonly and modify in the same scope list. Google silently drops the redundant one from the actual grant, and the next time your token refreshes - not immediately, which is what made this confusing - google-auth throws a Scope has changed exception with no obvious connection to what you changed. I only tracked it down by reading the google-auth-oauthlib source after the second refresh failure. If you're touching Gmail scopes: request the narrowest single scope that covers everything you need, not the union of several.

Bug #2: text/plain losing to text/html for the wrong reason

Gmail messages are frequently multipart/alternative - the same content exists as both plain text and HTML, and the MIME structure can nest arbitrarily (a multipart wrapping another multipart wrapping the actual parts). My first body-extraction function recursed through the tree and, at each level, fell back to HTML if plain text wasn't found at that level.

The bug: if the HTML part happened to appear before the plain-text part in the tree - which depends on how the sending client built the message, not anything I control - the function returned HTML first, even though a plain text version existed one level deeper.

The fix is two full passes instead of one recursive pass with a fallback mixed in: first, walk the entire tree looking only for text/plain. Only if that complete search comes up empty do you do a second full pass for text/html. The general lesson: when you have a preferred format and a fallback format in a tree structure, don't let the fallback trigger based on local absence - it has to be based on absence across the whole structure.

Bug #3: routing wasn't a category → person table

My assumption going in: Sales → Jordan, Support → Priya, Billing → Sam, Complaints → Alex. Simple lookup. Then one test case broke it - a billing question routed to Alex (the complaints person), not Sam.

The actual rule, once I found it in the test data: anything below a confidence threshold escalates to one specific person, regardless of category. The category table only applies once the model is confident enough; below that, it doesn't matter what the AI thinks the category is - a human with broader context handles it. I'd been treating confidence as purely cosmetic (which icon to show in Slack), not as something that changes the actual routing decision. Once I separated "confidence as display" from "confidence as a routing input," the fix was a few lines - but it wouldn't have surfaced without a test case that specifically checked routing against confidence, not just against category.

This is also why confidence_tier (auto / approval_needed / human_review) is computed in code from the numeric confidence score, not asked of the model directly - the boundaries are a business decision (how much error are you willing to tolerate before requiring a human?), and business decisions belong in code you can read and change without touching the prompt.

Idempotency: making "run it twice" safe

Because this polls Gmail rather than using a webhook, and because a real deployment means "the script might run again after a crash," double processing has to be structurally impossible, not just unlikely:

Marking a message as read removes it from the is:unread query, so a re-run never sees it again through the normal path
Independently, the SQLite table has gmail_message_id TEXT NOT NULL UNIQUE, with ON CONFLICT(gmail_message_id) DO UPDATE - so even if something bypasses the first guard, inserting the same message twice updates the existing row instead of creating a duplicate

Two independent guards, not one - because "the email got marked read but the script crashed before writing to the database" is exactly the kind of half-completed state that a single guard doesn't protect against.

Now the part that actually matters: does this save money?

Here's the honest math, using numbers you can substitute your own values into.

What it costs to run:
The system prompt is about 3,100 characters (~840 tokens), sent on every call along with the email body. At Claude Sonnet's current pricing ($3/MTok input, $15/MTok output), a typical classification-plus-draft call - roughly 1,050 input tokens and 200 output tokens - costs approximately $0.0064 per email. Round up to $0.01 to be safe.

What it replaces:
Industry surveys on support ticket handling time put a straightforward inquiry (the kind this pipeline is built for - not novel or complex issues) at somewhere between 3 and 8 minutes of staff time: reading, understanding intent, checking context if needed, and writing a reply from scratch. Take the low end - 4 minutes - and a loaded hourly cost for support staff around $25/hr (US small-business range, fully loaded with overhead), and one manually-answered routine email costs roughly $1.67 in labor.

The comparison:
$0.01 per email in API cost versus ~$1.67 in labor for the routine cases this is built for - a difference of two orders of magnitude. The pipeline doesn't remove the human (every reply is still reviewed and approved before sending), but it collapses "read, understand, draft from scratch" down to "read a pre-classified summary and a drafted reply, then approve or edit" - which is a few seconds of review instead of minutes of composition, even before counting the API cost difference.

Where this breaks down: this math only holds for the routine, high-confidence cases - which is exactly why the confidence_tier routing exists. Complex or ambiguous emails fall through to human_review and get full human handling with no shortcut, because getting a complex case wrong costs far more than the minutes saved. The system is designed to be conservative about what it treats as "routine" rather than to maximize the percentage automated - across the 8 test scenarios I validated the confidence thresholds against, that split lands around half auto-approvable and half requiring a human look, which feels like the right ratio for a first deployment rather than something to be tuned aggressively for automation percentage.

What I'd change for a production deployment

Prompt caching: the system prompt is fixed across every call and close to (but under) the ~1,024 token minimum cacheable length for Sonnet. Padding it slightly to clear that threshold would cut input cost further on any real volume.
Webhook instead of polling: users().watch() with Cloud Pub/Sub instead of a batch run - relevant once this isn't a demo run on-demand but something meant to react within seconds of an email arriving.
Slack rate limiting: chat.postMessage is limited to roughly one message per second per channel. At demo volume this doesn't matter; at real volume you need a delay between posts and a retry against Retry-After when you get a 429 - both already in place here, sized for a demo rather than tuned for a specific real load.

Try it

I put together a live version of this — the full flow, including the Slack card and the confidence-based routing: https://ai-ops-demo-site.vercel.app

If your inbox has the "half of these are the same five questions" problem, this is the kind of automation work I do — happy to talk through it.

Handwritten faxes, tilted phone-camera receipts, messy PDFs — I built an AI pipeline that turns them all into clean structured data

Zero AI Developer — Tue, 08 Sep 2026 08:06:38 +0000

Handwritten fax orders. Receipts photographed at an angle on a phone. PDFs that all use different layouts. I built a small pipeline using Claude Code that reads all of these and turns them into one clean, structured spreadsheet — and I want to walk through exactly how it performed, including where it broke.

This is meant to demonstrate something a lot of back-office teams deal with every day: manually copying information from paper/photos/scans into a spreadsheet. How far can AI actually take that job?

The Pipeline

The idea is simple:

Watch a folder
      ↓
Detect a new file (PDF/image)
      ↓
Have Claude read it
      ↓
Extract company name, date, line items, quantities, and totals as JSON
      ↓
Append to one CSV

The key point: the same pipeline handles all of these regardless of how messy the input looks. I deliberately generated three very different test files:

A clean, well-formatted PDF (fictional purchase order, 3 line items)
A receipt that looks like it was photographed on a phone at an angle (tilt, shadow, JPEG artifacts)
A fax-style image with handwriting (broken layout, faded text, noise)

Results

All three files produced results that matched the source content exactly. Total processing time for all three: about 25 seconds (roughly 5–10 seconds per file).

Here's a sample of what came out (all company names are fictional test data):

Source file	Vendor	Date	Item	Qty	Unit price	Total
fax_handwritten.png	(fictional)	8/20	Cardboard box, size 60	40	@98	approx. ¥62,000
fax_handwritten.png	(fictional)	8/20	Cloth tape 50m	12	@340	approx. ¥62,000
order_clean.pdf	(fictional)	Aug 21, 2026	A4 copy paper, 500 sheets	20	¥520	¥38,720
receipt_photo.jpg	(fictional)	Aug 19, 2026	USB drive 64GB	5	1,280	¥15,334

What I find most interesting is the handwritten fax row. Notice "@98" and "approx. ¥62,000" — it preserved the ambiguity of the handwritten notation exactly as written, instead of silently converting it into a clean number. That's by design: the extraction prompt explicitly instructs the model not to guess when something is genuinely ambiguous. Quietly "fixing" ambiguous handwriting into a confident-looking number would actually make this system more dangerous in a real back-office setting, not less.

All three files also correctly identified the issuing vendor rather than the addressee as the vendor field — a sign this isn't naive OCR, it's actually reasoning about document structure.

Implementation Notes

Running through a Claude Code subscription instead of metered API calls

I originally called the Anthropic API directly, but switched to invoking the Claude Code CLI as a subprocess instead. Simple reason: I didn't want to rack up API costs while still validating the idea.

claude -p "<extraction prompt>" --allowed-tools Read --output-format json

Each file is processed as a separate, non-interactive CLI call. Using --output-format json gives you a structured envelope (with fields like is_error, result) that's much more robust to parse than raw stdout.

This does come with a real limitation: spinning up a new process per file doesn't scale to bulk processing. If you need to run hundreds of documents at once, going back to the metered API is the more sensible choice. For a small-scale demo like this, that tradeoff is fine.

A bug I found the hard way: "failure" wasn't actually being treated as failure

While testing, I found a real gap in the design. When you feed it a corrupted file, Claude correctly returns all fields as null, per the instructions. The problem: that was being treated as a "success" — an empty row got appended to the CSV, and the source file got moved into the "processed" folder and effectively disappeared.

That meant a failure could silently destroy the only copy of that file's data, with no error surfaced. I fixed this by treating "all key fields are null" as an error case, so failed files stay in the input queue for retry instead of vanishing.

What this is actually useful for

The point of this demo isn't "it can read purchase orders." It's that work assumed to require manual handling — because every input looks different — can actually be automated with AI, with reasonable reliability.

The same pattern applies to things like:

Reconciling invoices that use a different format for every vendor
Digitizing orders that only ever arrive by fax
Turning paper forms/surveys into structured data

If you're trying to figure out how to actually wire this kind of thing into a real business process — not just "chat with an AI," but an actual pipeline — I take on contract work designing and implementing systems like this. Feel free to reach out.

Closing thoughts

This was a small-scale demo, not a production system. But it validated something real: turning messy, inconsistent input into meaningful structured data — a boring but very common back-office problem — is something AI can genuinely do today, not just in theory.

Hope this is useful if you're exploring something similar.

I Tried to Build a Company Where AI Autonomously Finds, Wins, and Delivers Work. Here's Why It Failed.

Zero AI Developer — Tue, 08 Sep 2026 06:12:03 +0000

"Just tell Claude Code what to do, and let AI handle everything — finding leads, qualifying them, doing the work, and delivering it."

That was the idea I spent about half a year trying to build, on my own. Spoiler: the system worked. All the tests passed. And I still ended up with zero contracts, zero dollars earned, and a net loss.

In this post I'll walk through the actual architecture I built, what happened across two experiment cycles, and why "being able to build it" turned out to be a completely different problem from "being able to sell it." I'm writing this in the hope that it saves someone else from repeating the same mistake.

The Architecture

I designed a 4-stage pipeline:

Research Agent (find opportunities)
      ↓
Qualification Agent (score them)
      ↓
Delivery Agent (execute & deliver)
      ↓
Ledger (log everything, track KPIs)

1. Research Agent

This agent searches for potential work based on a configurable domain, keywords, geography, and criteria. One important design decision: none of these were hardcoded. I wanted to be able to pivot the target domain without touching the code, since I fully expected the first domain to fail.

2. Qualification Agent

Every candidate opportunity gets scored on:

Willingness to pay — is there real evidence someone would actually pay this amount?
Existing alternatives — could this be solved with a free tool or in-house resources already?
Reachability — can I actually contact this person/company?
Risk factors — reputational or terms-of-service risk

I iterated on this scoring logic quite a bit. For example, I changed the risk-score weight from -3 to -5 partway through, so that an extremely risky candidate could single-handedly sink an otherwise strong score below the pass line.

3. Delivery Agent

Once a job was won, this agent used the Claude Code CLI as the execution engine to produce the deliverable. I deliberately avoided metered API billing and ran everything within an existing Claude Code subscription — I didn't want to rack up AI costs before earning a single dollar.

4. Ledger

Every opportunity, from discovery to outcome, gets logged. To avoid the trap of "this feels like it's working," I tracked exactly six KPIs:

KPI	Description
Opportunities discovered	Number of candidate jobs found
Jobs completed	Number of jobs actually finished
Revenue generated	Cash actually received (not invoices, not verbal agreements)
AI cost	Cost of AI usage
Human time	Hours of human involvement
Revenue per human hour	Revenue ÷ human hours

Deliberately excluding vanity metrics like signups or pageviews was, in hindsight, the right call. But as you'll see below, having the right metrics doesn't help much when the actual number of jobs is zero.

I also wrote 300+ tests. Engineering-wise, this system was solid.

Cycle 1: Let AI Discover Opportunities on Its Own → 10 Found, 0 Won

In the first cycle, I let the Research Agent autonomously search for opportunities from scratch. Results:

Opportunities discovered: 10
Passed qualification: 0
Won: 0
Revenue: $0

Digging into why all 10 failed, every single one fell into at least one of these buckets:

Willingness to pay was pure speculation. The pricing was based on "this seems like a reasonable amount," with zero evidence anyone had actually transacted at that price.
A free alternative always existed. Every candidate task could already be solved with a free tool or existing in-house resources.
No evidence of reachability. There was no verification that I could actually contact these targets, let alone pitch them.

In short: the AI found opportunities that looked plausible on paper, but none of them connected to a real person who was actually struggling with the problem and willing to pay to solve it.

Course Correction: Human Finds Jobs, AI Just Verifies Them

After Cycle 1, I flipped the division of labor:

Human: manually finds real job postings on freelance/gig platforms — real budgets, real clients
AI: only does supporting research on those human-sourced listings (e.g., checking if the asking price is reasonable)

This was a big shift — from "AI does discovery" to "discovery is human, AI only reinforces."

The Anonymity Wall I Didn't Expect

While comparing platforms at this stage, I hit a wall I hadn't anticipated: wanting to stay anonymous (no real name, no face photo) massively narrowed the field of usable platforms.

Requirement	Outcome
Real name + photo mandatory	Excluded
Pseudonym/business name OK, but requires ID + selfie verification	Skipped — too much friction
Anonymous-friendly, lightweight verification	Kept as a candidate

I also concluded that having AI autonomously crawl sites to find jobs was in a legal/ToS gray zone. In fact, one platform I was evaluating later discontinued its "auto-propose" feature entirely, switching to a "human must review before proposing" model. I took that as external confirmation that fully autonomous bot applications carry real platform risk.

What I ended up building instead: save search filters, receive email notifications for new listings, and have the AI read and analyze those notifications — never touching the site directly. It avoids ToS risk while still using AI for the analysis part.

Cycle 2 Results, and Why It Still Didn't Work

Even with this more careful setup, Cycle 2 also ended in zero contracts and zero revenue.

Technically, everything worked exactly as designed — the qualification logic, a "Policy Gate" that automatically filtered out sites with no-solicitation notices or bot bans, and the logging pipeline all functioned correctly.

The problem was never engineering. Winning work is not a problem software can solve.

When a person or company decides to hire someone, the deciding factor isn't "is this AI technically impressive?" It's "can I trust this person/company to actually deliver?" Trust comes from track record, referrals, and prior relationships — not something an automated system can manufacture on your behalf.

Looking back, I was essentially betting on:

Build it → put it out there → someone finds it

And the "someone finds it" part was where the whole plan broke down. Interestingly, I'd hit the exact same wall before, with a content site that depended on search traffic, and with several mobile apps I'd shipped — same structural problem every time. I could build things. I had never actually designed a distribution channel.

What I'm Doing With This Now

The original vision — "AI autonomously wins its own business" — is on hold indefinitely.

But the engineering work itself — designing and implementing a full discover → qualify → execute → log pipeline — wasn't wasted. I'm now repositioning that same technical capability: instead of using it to run my own autonomous business, I'm offering it as implementation work for companies that want to build AI-driven automation into their own operations.

Concretely, this includes:

Designing and building AI agent systems with Claude Code
Building automation pipelines that combine LLMs and APIs (scoring logic, risk evaluation, logging/dashboards — end to end)
Solo end-to-end iOS app development, from concept through App Store release (shipped 9 so far)

If you're trying to figure out how to actually wire AI into a real workflow — beyond just prompting a chatbot — feel free to reach out.

Closing Thoughts

Building something that automatically earns you money is a genuinely exciting idea for any indie developer. And with today's AI tooling, building a technically functional system is very achievable solo.

What I underestimated was the gap between "a system that correctly does what it's designed to do" and "a system that convinces someone to actually pay for it." That gap isn't closed with more code — it's closed with trust, built the slow way.

Hope this is useful to anyone chasing a similar idea.