DEV Community: Michael

The ROI Math Every Engineer Should Run Before Building an AI Agent

Michael — Thu, 10 Sep 2026 12:00:57 +0000

Most AI agent projects fail before a single line of code ships. Not because the tech is weak, but because nobody ran the numbers. Someone got excited, wired up an LLM to a Slack webhook, and six weeks later the demo impressed a VP and quietly died.

If you're the engineer being asked to build the thing, the ROI calculation is your best defense against building something nobody needed. Here's the framework I use.

Start With the Task, Not the Technology

An AI agent is only worth building if it replaces or accelerates a task that costs real money today. So the first number you need is the current cost of the work.

Break it down:

Volume: How many times per month does this task run?
Time per task: How long does a human spend on it?
Loaded hourly cost: Salary plus overhead, usually 1.3x base pay.

A support triage task that runs 4,000 times a month at 6 minutes each, handled by staff costing $45/hour loaded, burns roughly $18,000/month. That's your baseline. If the agent can't meaningfully dent that number, stop here.

Model the True Cost of the Agent

This is where engineers usually underestimate. The cost of an AI agent isn't just token spend. It's four buckets:

Build cost — engineering hours to design, integrate, and test.
Inference cost — LLM API calls, embeddings, vector DB queries.
Maintenance cost — prompt updates, model migrations, monitoring. Budget 15-25% of build cost annually.
Human-in-the-loop cost — the reviews and corrections the agent still needs.

Inference is the sneaky one. Multi-step agents call the model multiple times per task. Let's model it properly.

def monthly_inference_cost(
    tasks_per_month,
    llm_calls_per_task,
    avg_input_tokens,
    avg_output_tokens,
    input_price_per_1k=0.0025,   # $/1k input tokens
    output_price_per_1k=0.01,    # $/1k output tokens
):
    total_calls = tasks_per_month * llm_calls_per_task
    input_cost = (total_calls * avg_input_tokens / 1000) * input_price_per_1k
    output_cost = (total_calls * avg_output_tokens / 1000) * output_price_per_1k
    return round(input_cost + output_cost, 2)

# Support triage: 4000 tasks, 3 model calls each
cost = monthly_inference_cost(
    tasks_per_month=4000,
    llm_calls_per_task=3,
    avg_input_tokens=1200,
    avg_output_tokens=350,
)
print(f"Estimated inference: ${cost}/month")  # ~$78/month

Inference is often trivial next to labor. The real cost centers are build and maintenance, so weight your estimate there.

Calculate the Payback Period

Once you have monthly savings and total costs, payback period is simple:

def payback_period(build_cost, monthly_savings, monthly_run_cost):
    net_monthly_gain = monthly_savings - monthly_run_cost
    if net_monthly_gain <= 0:
        return None  # never pays back
    return round(build_cost / net_monthly_gain, 1)

months = payback_period(
    build_cost=40000,       # ~2 engineers for a month
    monthly_savings=13000,  # 70% of the $18k baseline
    monthly_run_cost=1500,  # inference + review + monitoring
)
print(f"Payback in {months} months")  # ~3.5 months

A payback under 6 months is easy to defend. Under 12 is reasonable. Beyond 18 months, you're gambling that requirements won't change — and they always do.

Discount for Reality

Here's the part that separates a defensible business case from a spreadsheet fantasy.

Automation rate is never 100%

Agents handle the happy path and escalate edge cases. If your agent fully resolves 70% of tasks and assists on another 20%, don't claim you eliminated the role. Model partial savings honestly.

Accuracy has a cost

A wrong answer in support isn't neutral — it can cost a customer. Factor in an error rate and the downstream cost of mistakes. An agent that's 95% accurate at scale still produces hundreds of errors a month.

Ramp time is real

No agent works well on day one. Assume 2-3 months of tuning before you hit projected performance. That pushes your effective payback out, so bake it in.

The Build vs. Buy Fork

Before you commit engineering hours, ask whether an existing tool already does 80% of this. Sometimes a $500/month SaaS agent beats a $40,000 custom build with ongoing maintenance. Custom only wins when the task is core to your business, deeply integrated with proprietary systems, or high enough volume that per-seat pricing gets absurd.

A One-Page Business Case

When you present, keep it to five numbers:

Current monthly cost of the task
Realistic automation rate
Total build cost
Monthly run cost
Payback period

That's the whole story. If those numbers work at conservative assumptions, build it. If they only work when you assume 100% automation and zero maintenance, you've found a demo, not a product.

The teams that win with AI agents aren't the ones with the fanciest prompts. They're the ones who picked the right task, ran the math first, and shipped something that paid for itself before anyone asked if it was worth it.

Originally published at getmichaelai.com

Stop Reaching for an AI Agent When a Cron Job Would Do

Michael — Wed, 09 Sep 2026 12:01:08 +0000

Everyone wants an AI agent now. It's the shiny thing. But half the workflows I get asked to "agentify" would run cheaper, faster, and more reliably as a deterministic pipeline you could have written in 2015.

The real skill in 2026 isn't building agents. It's knowing when not to.

Let's draw the line clearly.

The core difference: decision-making, not intelligence

Traditional automation follows a fixed path. Trigger → step → step → done. If the input matches the shape you designed for, it works every time. If it doesn't, it breaks predictably.

// Traditional automation: deterministic, cheap, boring, reliable
async function onNewInvoice(invoice) {
  if (invoice.amount > 10000) {
    await slack.notify('#finance', `Large invoice: ${invoice.id}`);
  }
  await sheets.append('invoices', invoice);
  await accounting.sync(invoice);
}

You know exactly what this does. You can unit test it. It costs nothing to run and it never hallucinates a vendor name.

An AI agent is different. It's given a goal, a set of tools, and the freedom to decide the steps. It reasons about what to do next based on context it wasn't explicitly programmed for.

# AI agent: reasons, chooses tools, handles ambiguity
agent = Agent(
    goal="Resolve the customer's billing dispute",
    tools=[lookup_account, issue_refund, escalate_to_human, send_email],
    guardrails={"max_refund": 200, "require_approval_above": 200},
)

result = agent.run(ticket="I was charged twice and I'm furious")

The agent might look up the account, spot the duplicate charge, issue a refund, and write an apology email — or decide it's over its refund limit and escalate. You didn't script that branching. It chose.

That flexibility is the feature. It's also the liability.

When traditional automation wins

Reach for a plain workflow when:

The path is knowable. If you can draw the flowchart, you don't need reasoning. You need a trigger and some steps.
The inputs are structured. Webhooks, form submissions, database rows, API payloads. Clean data going into clean logic.
Errors are expensive. Payments, provisioning, compliance. You want deterministic behaviour you can audit line by line.
Volume is high and margins are thin. An LLM call per row gets pricey fast. A function call does not.

Most "AI automation" projects I see in operations teams are actually this. Route the lead, enrich the record, sync the systems, fire the notification. No intelligence required — just plumbing done well.

When AI agents earn their keep

Bring in an agent when the problem has one of these traits:

Unstructured input

Inbound emails, support tickets, PDFs, meeting transcripts. When the shape of the input varies wildly, hard-coded parsing collapses. An LLM extracting intent and entities from a messy email is genuinely better than a hundred regex rules.

Branching you can't fully enumerate

A customer message could be a refund request, a feature question, a complaint, or all three. Writing every branch is a losing game. Let the model classify and route, then hand off to deterministic steps for the actual work.

Multi-step reasoning over tools

Research tasks are the sweet spot. "Find this company's recent funding, summarise their tech stack, draft a personalised outreach angle." That's search → read → synthesise → write, with the sequence depending on what's found along the way.

The pattern that actually works: agents and automation

The best systems aren't one or the other. They're deterministic pipelines with a thin layer of intelligence at the ambiguous points.

Use the agent for the fuzzy decision, then drop into rigid, testable code for the execution.

# Agent decides. Deterministic code executes.
category = classifier_agent.run(email.body)  # fuzzy → structured

if category == "refund":
    process_refund(email)      # plain function, fully tested
elif category == "bug_report":
    create_ticket(email)       # plain function
else:
    route_to_human(email)      # safe default

The agent does what only an agent can — turn mess into a clean signal. Everything downstream stays boring, auditable, and cheap.

A quick decision test

Before you build, ask three questions:

Can I write the flowchart? If yes → automation.
Is the input structured? If yes → automation.
Does the next step depend on interpreting meaning? If yes → agent, scoped tightly.

If you answered "agent" to number three, still wrap it in guardrails: spending limits, approval gates, human-in-the-loop for anything irreversible.

The strategy that saves money in 2026

Start with the cheapest tool that solves the problem. Deterministic first. Add intelligence only where determinism genuinely fails.

Teams that lead with agents burn budget on token costs, debug non-reproducible failures, and end up bolting on the exact validation logic they could have started with.

Teams that lead with clean automation, then surgically insert agents at the ambiguous joints, ship faster and sleep better.

The agent isn't the goal. The outcome is. Pick the dumbest tool that gets you there — and reserve the smart one for the problems that actually need a brain.

Originally published at getmichaelai.com

Stop Automating Randomly: A Scoring Model for Picking Your First AI Workflow

Michael — Tue, 08 Sep 2026 12:01:04 +0000

Most B2B automation projects fail for a boring reason: they start with the wrong workflow. Someone gets excited about AI, picks the flashiest process, and six weeks later the automation is more brittle than the manual steps it replaced.

The teams that win pick differently. They treat workflow selection like a portfolio decision — score everything, automate the boring high-frequency stuff first, and let early wins fund the ambitious builds.

Here's the model we use with clients, and the code to run it.

The payback equation nobody writes down

Every workflow has a real ROI you can estimate before writing a single line of code:

Annual payback = (minutes_saved_per_run × runs_per_year × loaded_hourly_rate / 60)
                 - build_cost - annual_maintenance

The mistake is optimizing for minutes_saved_per_run. A quarterly report that takes 4 hours feels painful, so people automate it. But 4 runs a year is nothing. A 3-minute task that runs 200 times a day is where the money hides.

Frequency beats duration. Almost always.

Score before you build

Don't trust gut feel. Give every candidate workflow four scores from 1–5:

Frequency — how often it runs
Determinism — how rule-based it is (high = safer, cheaper AI)
Data readiness — is the input structured and accessible?
Blast radius — how bad is a wrong output? (inverted: lower risk = higher score)

Here's a quick scorer you can drop into a script or a Jupyter cell:

from dataclasses import dataclass

@dataclass
class Workflow:
    name: str
    frequency: int      # 1-5
    determinism: int    # 1-5
    data_readiness: int # 1-5
    safety: int         # 1-5 (5 = low blast radius)
    minutes_saved: float
    runs_per_year: int

WEIGHTS = {"frequency": 0.35, "determinism": 0.25,
           "data_readiness": 0.25, "safety": 0.15}

def feasibility(w: Workflow) -> float:
    return round(
        w.frequency * WEIGHTS["frequency"] +
        w.determinism * WEIGHTS["determinism"] +
        w.data_readiness * WEIGHTS["data_readiness"] +
        w.safety * WEIGHTS["safety"], 2)

def annual_payback(w: Workflow, rate=60) -> int:
    return int(w.minutes_saved * w.runs_per_year * rate / 60)

candidates = [
    Workflow("Lead enrichment + routing", 5, 4, 4, 5, 6, 8000),
    Workflow("Quarterly board deck", 1, 2, 2, 3, 240, 4),
    Workflow("Support ticket triage", 5, 3, 4, 4, 4, 12000),
    Workflow("Contract review", 3, 2, 3, 1, 45, 300),
]

for w in sorted(candidates, key=feasibility, reverse=True):
    print(f"{w.name:30} feasibility={feasibility(w)}  "
          f"payback=${annual_payback(w):,}")

Run this and the ranking usually surprises people. The board deck — the thing everyone hates — scores near the bottom. Lead routing and ticket triage float to the top because they're frequent, structured, and low-risk.

The first-project shortlist

Across dozens of B2B builds, the same categories consistently deliver the fastest payback:

1. Inbound lead enrichment and routing

A form fills in your CRM. An agent enriches the company, scores fit, drafts a first-touch reply, and routes to the right rep. High frequency, structured input, low blast radius. This is almost always the best first project.

2. Support and email triage

Classify, tag, prioritize, and draft responses. You keep a human on send for anything risky, so errors are cheap. The volume makes the math obvious.

3. Data movement between tools

The unglamorous glue: sync deals to the data warehouse, push invoices to accounting, keep two SaaS tools in agreement. No AI needed for most of it — just reliable workflow automation in something like n8n.

4. Report and summary generation

Weekly pipeline summaries, meeting notes to action items, digest emails. High frequency, and LLMs are genuinely good at summarization.

Why n8n for the first build

Start with a visual orchestrator, not a from-scratch codebase. n8n gives you retries, error branches, and observability without you writing plumbing. A lead-routing flow looks like:

Webhook (form) → HTTP enrich → AI classify (fit score)
  → IF score > 70 → draft reply (LLM) → Slack notify rep
  → ELSE → tag "nurture" → add to sequence

Every node is a place to inspect payloads and add guardrails. When a step breaks — and it will — you see exactly where. That visibility is worth more than architectural purity for your first three projects.

The 20% you should not automate yet

Skip anything that scores low on determinism and low on safety at the same time. Contract review is the classic trap: high perceived value, low structure, catastrophic blast radius. Automate the intake around it — routing, deadline tracking, first-pass flagging — and leave the judgment call to a human.

Same rule for anything touching money movement, legal commitments, or irreversible customer communication. Draft, don't send. Recommend, don't decide.

Sequence for compounding wins

Ship one high-frequency, low-risk workflow in week one.
Measure actual minutes saved against your estimate.
Reinvest the saved time into the next-highest scorer.
Only after three shipped wins should you attempt a multi-step agent.

The goal isn't to automate everything. It's to build momentum and credibility with wins that pay for themselves fast — then let that compound. Score honestly, start boring, and the ambitious stuff gets a lot easier to fund.

Originally published at getmichaelai.com

Your AI Chatbot Has Freeloaders: A Developer's Guide to Killing Runaway Token Bills

Michael — Mon, 07 Sep 2026 12:01:05 +0000

You shipped a chatbot. Users love it. Then the invoice lands and finance asks why your "cheap AI feature" costs more than a junior engineer's salary.

Welcome to the token economy, where a single verbose system prompt multiplied across 40,000 daily requests turns into a five-figure surprise. The good news: most runaway LLM bills come from a handful of fixable patterns. Let's find them.

Where the money actually leaks

Before optimizing anything, understand what you're paying for. Every LLM call bills on two axes: input tokens (your prompt, history, retrieved context) and output tokens (the model's reply). Output usually costs 3-5x more than input.

The usual suspects:

Bloated system prompts. A 2,000-token instruction block sent on every single request. That's fixed overhead you pay forever.
Unbounded chat history. Naively appending the full conversation until context windows explode.
Overkill model selection. Routing "what are your hours?" to GPT-4-class models when a small model would nail it.
No output caps. Letting the model ramble for 800 tokens when 100 would do.
Retry storms. Failed calls that silently re-fire and double-bill.

You can't govern what you don't measure. So step one is instrumentation.

Instrument every call

Wrap your LLM client so no request escapes without logging tokens, cost, model, and the feature that triggered it. This one habit surfaces 80% of your waste.

import time
from dataclasses import dataclass

PRICING = {  # USD per 1K tokens
    "gpt-4o":      {"in": 0.005,  "out": 0.015},
    "gpt-4o-mini": {"in": 0.00015,"out": 0.0006},
}

@dataclass
class CallResult:
    text: str
    cost: float

def tracked_completion(client, model, messages, feature, user_id, max_tokens=300):
    start = time.time()
    resp = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=max_tokens,  # hard cap on output spend
    )
    usage = resp.usage
    p = PRICING[model]
    cost = (usage.prompt_tokens/1000)*p["in"] + (usage.completion_tokens/1000)*p["out"]

    log_event({
        "feature": feature,
        "user_id": user_id,
        "model": model,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "cost_usd": round(cost, 6),
        "latency_ms": int((time.time() - start) * 1000),
    })
    return CallResult(resp.choices[0].message.content, cost)

Now you can slice spend by feature and by user. That second dimension matters more than people expect.

Meet your token freeloaders

Every chatbot has power users, and a small slice of them generates a wildly disproportionate share of cost. Sometimes it's a genuine heavy user. Sometimes it's a script hammering your endpoint. Either way, you need per-user budgets.

A simple rolling budget stops one account from eating your margin:

def enforce_budget(user_id, redis, daily_cap_usd=2.0):
    key = f"spend:{user_id}:{today()}"
    spent = float(redis.get(key) or 0)
    if spent >= daily_cap_usd:
        raise BudgetExceeded(f"{user_id} hit ${daily_cap_usd} cap")
    return spent

def record_spend(user_id, redis, cost):
    key = f"spend:{user_id}:{today()}"
    redis.incrbyfloat(key, cost)
    redis.expire(key, 60 * 60 * 26)  # auto-clean

Call enforce_budget before the request, record_spend after. Freeloaders get a polite "you've hit today's limit" instead of wrecking your P&L.

Route by difficulty, not by default

The single biggest lever is model selection. Most chatbot traffic is trivial - greetings, FAQ hits, simple lookups. Sending all of it to a frontier model is like taking a helicopter to the corner store.

Use a cheap model as your default and escalate only when needed:

A pragmatic routing rule

Classify intent with a tiny, fast model (or even regex/embeddings for known FAQs).
Serve cached or template answers for high-frequency questions.
Escalate to the expensive model only for open-ended reasoning.

A solid RAG setup with cached answers can push 60-70% of traffic away from your priciest model. That alone often halves the bill.

Trim the prompt, cap the output

Two quick wins:

Compress history. Don't send 20 turns of raw chat. Summarize older turns into a short running memory and keep only the last few verbatim. You preserve context at a fraction of the tokens.

Cap output tokens. Set max_tokens aggressively per use case. A support answer rarely needs 500 tokens. If it does, that's a signal your prompt is unfocused.

Also lean on prompt caching where your provider supports it. Static system prompts and repeated context can be cached so you stop paying full price for the same 2,000 tokens on every call.

Set guardrails, then watch them

Cost control isn't a one-time cleanup - it's governance. Bake these into your stack:

Global daily budget with alerts at 50/80/100%.
Per-feature and per-user caps so no single path runs away.
Anomaly alerts on spend-per-request spikes (a sign of a prompt regression or abuse).
A weekly cost-per-conversation metric. Tie it to the value each conversation produces. If a resolved ticket costs $0.04 in tokens and saves $6 in support time, you're winning. If it costs $3, you're not.

The ROI math that actually matters

A chatbot's ROI isn't "we spent $X on tokens." It's cost-per-outcome versus the alternative. Track cost per resolved ticket, per qualified lead, per completed task - then optimize toward that number, not toward raw token minimization.

Because the goal was never to spend zero. It was to make every token pull its weight.

If your AI spend is climbing faster than the value it returns, that's usually an architecture problem, not a pricing problem - and it's fixable. That's exactly the kind of system design we build at Michael AI: automations and agents with cost governance baked in from day one, so ROI holds up when traffic scales.

Originally published at getmichaelai.com

The 40% Support Cost Cut That Doesn't Tank Your CSAT: An Engineering Playbook

Michael — Sun, 06 Sep 2026 12:01:00 +0000

Most AI support projects fail the same way. Someone bolts a chatbot onto the help center, it hallucinates a refund policy, CSAT drops eight points, and leadership quietly kills the initiative six weeks later.

The 40% number is real. We've hit it repeatedly. But it comes from architecture decisions, not from replacing humans with a magic model. Here's the actual playbook.

Start with deflection math, not model selection

Before you write a line of code, categorize your last 1,000 tickets. You're looking for three buckets:

Deflectable: password resets, order status, plan changes, doc lookups. High volume, low risk.
Assistable: the AI drafts, a human approves. Refunds, account changes, edge-case troubleshooting.
Human-only: churn saves, legal, angry escalations, anything touching money above a threshold.

In most B2B SaaS support queues, 55-70% of tickets are deflectable or assistable. That's where your 40% lives. Trying to automate the human-only bucket is where CSAT dies.

Run this on your export before anything else:

import pandas as pd

tickets = pd.read_csv("tickets_90d.csv")

# tag by intent using your existing categories or a quick classifier
deflectable = tickets[tickets.intent.isin([
    "order_status", "password_reset", "plan_change", "how_to"
])]

volume = len(deflectable) / len(tickets)
avg_handle_min = tickets.handle_time_min.mean()
hours_saved = (len(deflectable) * avg_handle_min) / 60

print(f"Deflectable share: {volume:.0%}")
print(f"Monthly human-hours reclaimable: {hours_saved:,.0f}")

If that number is small, an AI agent won't save you. Fix your product or docs first.

Build the agent around retrieval, not vibes

The fastest way to torch trust is letting the model answer from training data. Every response must be grounded in your content: docs, past resolved tickets, policy pages.

Use retrieval-augmented generation with a hard rule: no source, no answer.

def answer(question, kb):
    chunks = kb.search(question, top_k=5, min_score=0.75)

    if not chunks:
        return escalate(question, reason="no_confident_source")

    context = "\n\n".join(c.text for c in chunks)
    prompt = f"""Answer ONLY using the context below.
If the context does not fully answer, say you'll connect a human.
Never invent policy, prices, or steps.

Context:
{context}

Question: {question}"""

    reply = llm.generate(prompt, temperature=0.2)
    return reply, [c.source for c in chunks]  # always cite

Two details matter here. Low temperature kills creative fabrication. And the min_score threshold on retrieval is your first safety net: weak matches trigger escalation instead of a guess.

Escalation design is the whole game

CSAT rarely dies because the AI answered wrong. It dies because the AI trapped the customer. Loop them through the same three suggestions, refuse to hand off, force them to rephrase five times.

Design explicit escape hatches:

Confidence-based: retrieval score below threshold - escalate silently.
Sentiment-based: frustration detected in the message - escalate immediately, don't try to help.
Intent-based: refund, cancel, legal keywords - route to human queue.
Loop-based: same customer, same unresolved intent, second attempt - escalate.

def should_escalate(msg, history, retrieval_score):
    if retrieval_score < 0.75:
        return True, "low_confidence"
    if detect_frustration(msg) > 0.6:
        return True, "negative_sentiment"
    if any(k in msg.lower() for k in ["cancel", "refund", "lawyer", "gdpr"]):
        return True, "sensitive_intent"
    if repeated_unresolved(history):
        return True, "loop_detected"
    return False, None

A good AI agent should feel eager to hand off, not desperate to close the ticket itself. The metric to optimize is resolution, not containment.

Protect CSAT with measurement, not hope

Instrument from day one. Track these per conversation:

Deflection rate (resolved without human touch)
Escalation rate and reason
CSAT split: AI-only vs. AI-assisted vs. human-only
Reopen rate (did the "resolved" ticket come back?)

That last one is the honest metric. A high deflection rate with a high reopen rate means you're not resolving anything - you're delaying tickets and annoying people.

Set a CSAT floor. If AI-handled conversations drop below your human baseline minus two points, you route more aggressively to humans until you fix the gap. Cost savings that cost you retention aren't savings.

Roll it out like an engineer, not a gambler

Don't flip 100% of traffic. Stage it:

Shadow mode: AI drafts answers, humans send. Measure quality with zero customer risk.
Single intent live: turn on one deflectable category, e.g. order status.
Expand by confidence: add intents as each proves its CSAT and reopen numbers.

Within 8-12 weeks, a well-scoped agent handles 40-60% of volume end to end, your team focuses on the hard, high-value tickets, and CSAT holds or climbs because response times collapse from hours to seconds.

The 40% cost cut is a byproduct. The actual goal is a support system where the AI knows what it doesn't know - and gets out of the way fast when it doesn't.

Originally published at getmichaelai.com

RAG or Bust: Why Your Standard Chatbot Fails at Customer Support (And How to Fix It)

Michael — Sat, 05 Sep 2026 12:00:59 +0000

Most support chatbots fail for the same reason: they don't know anything about your business. They generate plausible-sounding text, but plausible isn't the same as correct. When a customer asks about your refund window or a specific error code, a base LLM will either hallucinate an answer or dodge the question.

This is the core decision teams face in 2026: build a standard chatbot on a prompt and system message, or build a Retrieval-Augmented Generation (RAG) agent that pulls from your actual knowledge. Let's break down when each makes sense, what it costs, and how to decide.

What a "Standard" Chatbot Actually Is

A standard chatbot is an LLM call wrapped in a system prompt. You give it instructions, maybe a few examples, and let it respond.

const response = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [
    { role: 'system', content: 'You are a support agent for Acme. Be friendly and concise.' },
    { role: 'user', content: userMessage }
  ]
});

This works fine for generic, low-stakes conversations: greeting customers, routing tickets, answering questions the model already knows from public training data. It's cheap and ships in an afternoon.

The problem: it has zero visibility into your private data. Your pricing tiers, your API docs, last week's policy change, the customer's order history. If it isn't in the prompt or the training set, the model guesses.

You can stuff more into the system prompt, but that hits a wall fast. Context windows are large, but not infinite, and cramming 200 pages of docs into every request is expensive and slow.

What RAG Changes

RAG separates knowledge from reasoning. Instead of baking answers into the prompt, you store your content in a vector database, retrieve only the relevant chunks per query, and inject those into the LLM call.

def answer_query(question: str):
    # 1. Embed the incoming question
    query_vec = embed(question)

    # 2. Retrieve top matching chunks from your knowledge base
    chunks = vector_db.search(query_vec, top_k=4)

    context = "\n\n".join(c.text for c in chunks)

    # 3. Ground the model in retrieved facts
    prompt = f"""Answer using ONLY the context below.
    If the answer isn't there, say you don't know.

    Context:
    {context}

    Question: {question}"""

    return llm.complete(prompt)

The payoff is threefold:

Accuracy. Answers are grounded in your real content, not the model's imagination.
Freshness. Update the knowledge base and the agent knows immediately. No retraining.
Traceability. You can show which document produced an answer, which matters for compliance and trust.

That "say you don't know" instruction is doing heavy lifting. It's the difference between an agent that admits uncertainty and one that invents a refund policy that doesn't exist.

The Cost Reality

RAG isn't free. Here's the honest breakdown.

Standard chatbot: one LLM call per turn. Pennies per conversation. Build time measured in hours.

RAG agent: you're now running an ingestion pipeline (chunking, embedding, storage), a vector database, and two model calls per turn (embedding + completion). Add retrieval tuning, evaluation, and re-indexing as content changes. Build time is weeks, and you'll maintain it.

For a mid-sized support operation, expect infrastructure costs in the low hundreds per month plus token usage. The bigger cost is engineering time to get retrieval quality right, because bad retrieval produces confidently wrong answers, which is worse than no answer.

A Decision Framework

Ask these four questions.

1. Does answering require private or frequently-changing data?

If yes, you need RAG. A chatbot cannot know your Q3 pricing or a customer's ticket history without retrieval.

2. What's the cost of a wrong answer?

For a marketing FAQ bot, a hallucination is embarrassing. For a fintech or healthcare support agent, it's a liability. High stakes push you toward grounded, traceable RAG.

3. How often does your knowledge change?

Static knowledge (a fixed set of product features) can sometimes live in a prompt. Content that shifts weekly demands a retrieval layer you can update without redeploying.

4. What volume are you handling?

At low volume, a well-crafted prompt-based bot with a human fallback may be enough. At scale, RAG's accuracy directly reduces escalations and support cost.

The Practical Answer for 2026

Most businesses building a serious AI support agent will land on RAG, often as a hybrid: retrieval for anything factual, plus tool calls for live data like order status, and a standard-prompt fallback for chit-chat and routing.

Start smaller than you think. Pick your top 20 support questions, build a focused RAG pipeline over the docs that answer them, and measure resolution rate against your current setup. Expand the knowledge base once retrieval quality is solid.

The teams winning at AI support aren't the ones with the biggest model. They're the ones who connected the model to the right knowledge and made it honest about what it doesn't know. RAG is how you get there.

Originally published at getmichaelai.com

Your Chatbot's Message Count Is a Vanity Metric: 7 KPIs That Actually Matter

Michael — Fri, 04 Sep 2026 12:00:59 +0000

Most chatbot dashboards are built to make you feel good, not to tell you the truth. Total messages, active users, average session length - these numbers go up whether your bot is useful or annoying. They measure activity, not value.

If you're running an AI chatbot for support or lead gen, you need metrics that tie back to money saved or money made. Here are the seven that do.

1. Resolution Rate

The percentage of conversations the bot fully closed without a human touching them. This is the single most important support KPI.

Be strict about the definition. A "resolved" conversation means the user got what they needed and didn't reopen the ticket or bounce to a human within, say, 24 hours. A bot that says "Was that helpful?" and gets ignored is not a resolution.

def resolution_rate(conversations):
    resolved = [
        c for c in conversations
        if c["closed_by"] == "bot"
        and not c["escalated"]
        and not c["reopened_within_24h"]
    ]
    return round(len(resolved) / len(conversations) * 100, 1)

Benchmark: a decent bot lands 40-60%. Anything above 70% either means great automation or a definition that's too generous. Audit it.

2. Deflection Rate

Deflection is close to resolution but framed around cost. It's the share of inbound volume that never reached a human agent - because the bot handled it, or the user self-served from a bot-surfaced article.

The difference matters: deflection includes cases where the user got their answer and left satisfied without a formal resolution event. Both metrics together tell you how much load you took off your support team.

Tie it to dollars:

const deflectedTickets = totalInbound - ticketsToHumans;
const costPerTicket = 6.50; // your fully-loaded agent cost
const monthlySavings = deflectedTickets * costPerTicket;

That one line converts a fuzzy "the bot is helping" into a number your CFO respects.

3. Escalation Rate (and Escalation Quality)

The inverse of resolution, but don't treat it as pure failure. A good escalation is a feature: the bot recognized it was out of depth and handed off cleanly with full context.

Track two things:

Raw escalation rate - what fraction goes to a human.
Escalation quality - did the agent get a summary, the user's intent, and prior steps? Or did the customer have to repeat everything?

A bot with a 45% escalation rate but seamless handoffs can beat one with a 30% rate that dumps confused users on agents.

4. Containment by Intent

Aggregate resolution hides the truth. Break it down by intent.

Your bot might crush "reset password" (95% resolved) and completely faceplant on "cancel my subscription" (12% resolved). The average looks fine. The reality is you have a broken cancellation flow bleeding trust.

from collections import defaultdict

def containment_by_intent(conversations):
    totals = defaultdict(int)
    resolved = defaultdict(int)
    for c in conversations:
        totals[c["intent"]] += 1
        if c["closed_by"] == "bot" and not c["escalated"]:
            resolved[c["intent"]] += 1
    return {
        intent: round(resolved[intent] / totals[intent] * 100, 1)
        for intent in totals
    }

This is where you find your highest-ROI fixes. Improve your three worst high-volume intents and your overall numbers jump.

5. Qualified Leads Captured

For a lead-gen chatbot, message count is meaningless. What matters is qualified leads - contacts that match your ICP, gave real intent signals, and got routed to sales.

Define qualification explicitly: budget mentioned, decision-maker role, timeline, or a demo booked. Then measure the conversion from conversation to qualified lead. A bot that talks to 5,000 people and produces 4 qualified leads is a very expensive toy.

6. Pipeline Influenced

This is the metric that gets budget approved. Connect chatbot-sourced leads to your CRM and track the pipeline value they generate.

Don't over-claim. Use influenced pipeline (deals the bot touched) alongside sourced pipeline (deals it originated). The honest split protects your credibility when leadership digs in.

When you can say "the bot booked 38 demos last quarter, contributing $210K in influenced pipeline," nobody asks about session length again.

7. Cost Per Resolution

The efficiency check. Take your total bot cost - platform, LLM tokens, engineering time, maintenance - and divide by resolved conversations.

const totalCost = platformFee + tokenSpend + engHours * hourlyRate;
const costPerResolution = totalCost / resolvedConversations;
// Compare against ~$6-$15 for a human-handled ticket

If your cost per resolution creeps toward human cost, something's wrong - usually token bloat from bad prompts or a bot escalating too much while still burning compute.

Put Them Together

No single number wins. Resolution and deflection prove support ROI. Escalation quality protects experience. Containment-by-intent shows you where to build next. Qualified leads and pipeline prove revenue impact. Cost per resolution keeps you honest.

Build a dashboard around these seven and kill the vanity charts. The goal isn't a bot that's busy - it's a bot that provably saves money and makes money. If you can't draw a straight line from your metrics to one of those two outcomes, you're measuring the wrong thing.

Originally published at getmichaelai.com

The AI Chatbot ROI Formula: How to Model Support Automation Spend Before You Ship

Michael — Thu, 03 Sep 2026 12:01:02 +0000

Every support automation pitch ends the same way: someone in finance asks "what's the return?" and the room goes quiet. People throw around "deflection rate" and "efficiency gains" without a number attached.

Let's fix that. Below is a concrete model you can run before writing a line of chatbot code, plus the code to calculate it. No hand-waving.

The core equation

Chatbot ROI is not mysterious. It's a comparison between what you save and what you spend:

ROI = (Annual Savings - Annual Cost) / Annual Cost

The hard part is not the ROI line. It's building an honest Annual Savings number. Most teams inflate it because they assume 100% resolution. Real bots resolve a fraction and escalate the rest.

The variables that actually matter

Ticket volume - tickets per month
Cost per human ticket - fully loaded agent cost / tickets handled
Automation rate - % of tickets the bot fully resolves without a human
Containment cost - what each bot-handled ticket costs you (LLM tokens, infra, platform)
Escalation drag - bot-handled-then-escalated tickets are more expensive than a direct human handoff

That last one is where naive models break. If your bot fumbles and hands off cold, you pay for the LLM call and the agent time. Model it.

The model in code

def chatbot_roi(
    monthly_tickets: int,
    cost_per_human_ticket: float,   # fully loaded, e.g. $6.50
    automation_rate: float,          # 0.0 - 1.0, fully resolved by bot
    escalation_rate: float,          # of remaining, how many bot touches then escalate
    cost_per_bot_ticket: float,      # tokens + infra, e.g. $0.12
    escalation_penalty: float,       # extra human minutes cost from cold handoff
    build_cost: float,               # one-time build/integration
    monthly_platform_cost: float,    # n8n, vector db, hosting, etc.
):
    annual_tickets = monthly_tickets * 12

    resolved_by_bot = annual_tickets * automation_rate
    escalated = annual_tickets * (1 - automation_rate) * escalation_rate
    pure_human = annual_tickets - resolved_by_bot - escalated

    # Cost of the old world: everything handled by humans
    baseline_cost = annual_tickets * cost_per_human_ticket

    # Cost of the new world
    bot_cost = (resolved_by_bot + escalated) * cost_per_bot_ticket
    human_cost = pure_human * cost_per_human_ticket
    escalation_cost = escalated * (cost_per_human_ticket + escalation_penalty)
    platform = monthly_platform_cost * 12

    new_cost = bot_cost + human_cost + escalation_cost + platform

    annual_savings = baseline_cost - new_cost
    total_cost = build_cost + platform
    roi = (annual_savings - build_cost) / total_cost

    payback_weeks = build_cost / (annual_savings / 52) if annual_savings > 0 else float("inf")

    return {
        "annual_savings": round(annual_savings),
        "roi": round(roi, 2),
        "payback_weeks": round(payback_weeks, 1),
    }


print(chatbot_roi(
    monthly_tickets=8000,
    cost_per_human_ticket=6.50,
    automation_rate=0.45,
    escalation_rate=0.20,
    cost_per_bot_ticket=0.12,
    escalation_penalty=2.00,
    build_cost=18000,
    monthly_platform_cost=900,
))

Run that and you get roughly:

{'annual_savings': 214000, 'roi': 7.4, 'payback_weeks': 4.4}

A 4.4-week payback. That's the number that ends the finance conversation - not "we deflect a lot of tickets."

Where these numbers come from

Don't invent the inputs. Pull them.

Ticket volume and cost per ticket live in your helpdesk (Zendesk, Intercom, Freshdesk). Export a quarter of data. Fully loaded cost = (agent salaries + tooling + overhead) / tickets resolved. Most B2B SaaS teams land between $4 and $12.

Automation rate is the one you can't guess. Before building, run a tagging pass over 500 real tickets. Bucket them: password resets, billing questions, "where is my order," edge cases. Sum the buckets a bot can realistically own. If 45% of tickets are five repeatable intents, that's your ceiling - and you won't hit the ceiling in month one.

Escalation rate starts high and falls. Assume 25-35% for the first quarter, dropping as you close knowledge gaps.

The mistakes that make ROI fiction

Counting deflection as resolution

A bot that answers and the user still opens a ticket didn't deflect anything. Only count tickets that close without human touch. Instrument this explicitly - tag every conversation with resolved_by_bot vs escalated.

Ignoring maintenance

That monthly_platform_cost should include the human hours to update prompts, retrain retrieval, and review transcripts. Budget 5-10 hours a month. A bot is not a fire-and-forget asset.

Modeling on peak, not average

Holiday spikes make automation look heroic. Use a trailing 90-day average so your CFO doesn't catch you cherry-picking.

The threshold that says "build it"

A quick rule from the projects we run: if payback lands under 12 weeks and automation rate on tagged tickets clears 30%, the build is defensible. Under 6 weeks, it's a no-brainer. Above 20 weeks, your ticket volume is probably too low or your intents too varied - route the budget elsewhere.

The formula's real value isn't the final ROI figure. It's that it forces you to measure automation rate before you build, kill projects that won't pay back, and set instrumentation that proves the number after launch.

Plug in your own numbers. If the payback is measured in weeks, you have your business case. If it's measured in years, you just saved yourself a build.

Originally published at getmichaelai.com

The Deflection Rate Trap: Building AI Support That Cuts Costs Without Burning Customers

Michael — Wed, 02 Sep 2026 12:00:41 +0000

Every support automation pitch leads with the same promise: deflect 60% of tickets, slash costs, done. Then six months later the CSAT graph looks like a ski slope and the churn team is asking questions.

The problem isn't AI. It's that most teams optimize for deflection rate as a vanity metric instead of engineering for resolution while protecting the customers who actually need a human.

Here's the playbook I use when building AI support systems for clients - the math, the architecture, and the guardrails that keep CX intact.

Start with the ticket taxonomy, not the bot

Before you automate anything, pull 90 days of tickets and cluster them. You're looking for three buckets:

Repetitive + deterministic - password resets, order status, refund policy, plan changes. These are pure automation wins.
Repetitive + judgment - billing disputes, cancellations, edge-case troubleshooting. AI can assist but shouldn't close alone.
Rare + high-stakes - outages, legal, angry enterprise accounts. Route to humans immediately. Never deflect these.

Most teams try to automate all three and wonder why customers revolt. The money is in bucket one. Bucket two is where AI augments your agents. Bucket three is where automation exists only to route faster.

The deflection math that actually matters

Deflection rate alone lies to you. A bot can "deflect" a ticket by frustrating someone into giving up - that's a hidden churn cost, not a win.

Track true resolution rate instead: the percentage of automated conversations that end with the problem solved and no follow-up human ticket within 72 hours.

def true_resolution_rate(conversations):
    resolved = 0
    for c in conversations:
        no_followup = not c.reopened_within_hours(72)
        no_escalation = not c.escalated_to_human
        positive_signal = c.csat is None or c.csat >= 4
        if no_followup and no_escalation and positive_signal:
            resolved += 1
    return resolved / len(conversations)

# Deflection can be 60% while true resolution is 35%.
# The gap is your hidden support debt.

If your deflection rate and true resolution rate diverge by more than ~15 points, your bot is pushing problems downstream, not solving them.

Build the escalation path first

Counterintuitive, but the fastest way to lose trust is a bot with no exit. Design the handoff before the happy path.

A good escalation trigger is multi-signal, not just "user typed agent":

function shouldEscalate(conversation) {
  const signals = {
    explicitRequest: /human|agent|representative|manager/i.test(conversation.lastMessage),
    frustration: conversation.sentimentScore < -0.4,
    loopDetected: conversation.repeatedIntentCount >= 3,
    lowConfidence: conversation.lastIntentConfidence < 0.6,
    highValueAccount: conversation.customer.tier === 'enterprise',
  };

  const triggered = Object.entries(signals)
    .filter(([, active]) => active)
    .map(([name]) => name);

  return {
    escalate: triggered.length > 0,
    reasons: triggered,
  };
}

When you escalate, pass full context to the human. Nothing enrages a customer more than re-explaining everything to a person after the bot already asked. Your handoff payload should include the transcript, detected intent, account tier, and any actions the bot attempted.

Ground the AI in your actual data

Hallucinated refund policies cost more than the tickets you deflect. Use retrieval over your real knowledge base and, critically, let the model say "I don't know."

Structure your system prompt so uncertainty routes to a human instead of inventing an answer:

SYSTEM_PROMPT = """
You are a support assistant. Answer ONLY from the provided context.
If the context does not contain the answer, respond exactly with:
{"action": "escalate", "reason": "insufficient_context"}
Never guess policies, prices, or account-specific details.
"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "system", "content": f"Context:\n{retrieved_docs}"},
        {"role": "user", "content": user_message},
    ],
    temperature=0.2,
)

Low temperature, strict grounding, explicit escape hatch. That combination is the difference between a helpful assistant and a liability.

The ROI model your CFO will actually believe

Conversational AI ROI is simple once you stop counting deflection as savings. The real formula:

Monthly savings = (tickets_resolved_by_ai × cost_per_human_ticket)
                  - (ai_platform_cost + build_amortization)
                  - (churn_cost_from_bad_automation)

That last term is the one everyone forgets. Model it explicitly. If your true resolution rate is solid, it's near zero. If you're chasing vanity deflection, it can wipe out the entire savings line.

A realistic mid-market example: 10,000 tickets/month, $6 fully-loaded cost per human ticket, 35% true resolution. That's ~$21,000/month in genuine savings before platform costs - and your agents now spend time on the hard tickets that retain accounts.

Roll it out in stages

Don't flip the switch on 100% of traffic. Sequence it:

Week 1-2: Shadow mode. AI drafts responses, humans send. Measure accuracy.
Week 3-4: Auto-resolve bucket one only. Everything else routes to humans.
Month 2: Expand to bucket two with human-in-the-loop approval.
Ongoing: Weekly review of escalation reasons to close knowledge gaps.

The teams that win treat support automation as a product, not a project. They watch the resolution-vs-deflection gap, feed failures back into the knowledge base, and protect the customers who need a human.

Cut the cost. Keep the customer. Those aren't in tension if you measure the right thing.

Originally published at getmichaelai.com

8 AI Automations Small Businesses Can Ship This Quarter (With Actual ROI)

Michael — Tue, 01 Sep 2026 12:00:41 +0000

Most "AI for small business" content is vaporware. It promises transformation and delivers a chatbot that hallucinates your refund policy.

This is the opposite. Below are eight automations you can build and deploy in a single quarter, each tied to a metric you already track: hours saved, revenue recovered, or response time cut. No moonshots. Just plumbing that pays for itself.

1. Lead follow-up that never sleeps

The average small business takes hours to respond to an inbound lead. The data on speed-to-lead is brutal: contact within 5 minutes and you're up to 100x more likely to connect than at 30 minutes.

Wire your form submissions to an agent that qualifies, enriches, and drafts a personalized first reply instantly.

// n8n Function node: score and route an inbound lead
const lead = $json;

const score =
  (lead.company_size > 50 ? 30 : 10) +
  (lead.budget === 'high' ? 40 : 15) +
  (/(ceo|founder|vp|director)/i.test(lead.title) ? 30 : 5);

return {
  ...lead,
  score,
  route: score >= 70 ? 'sales-hot' : 'nurture-sequence',
  sla_minutes: score >= 70 ? 5 : 60,
};

Hot leads ping a human in Slack. Everyone else drops into an automated nurture sequence. Zero leads rot in an inbox.

2. Invoice chasing without the awkward emails

Unpaid invoices are cash you already earned. Most owners hate chasing them, so they don't.

Build a workflow that watches your accounting API (QuickBooks, Xero, Stripe), detects overdue invoices, and sends escalating reminders on a schedule. Polite at day 3, firmer at day 14, "we're pausing service" at day 30. The AI adjusts tone per customer relationship.

One client recovered roughly 15% of their aged receivables in the first month. That's real money for a Tuesday afternoon build.

3. A support chatbot that actually knows your business

Generic chatbots frustrate people. A RAG-powered agent trained on your docs, past tickets, and policies is different.

# Retrieve relevant context, then answer grounded in it
from openai import OpenAI

client = OpenAI()

def answer(question, kb):
    context = kb.search(question, top_k=4)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content":
             "Answer ONLY from the context. If unsure, escalate to a human."},
            {"role": "user", "content": f"Context:\n{context}\n\nQ: {question}"},
        ],
    )
    return resp.choices[0].message.content

The non-negotiable rule: ground every answer in retrieved context and escalate when confidence is low. That single guardrail is the difference between deflecting 40% of tickets and getting a viral screenshot of your bot lying.

4. Meeting notes to CRM, automatically

Sales reps forget to log calls. So the CRM is garbage, and forecasting is guesswork.

Pipe your call recordings (Fireflies, Fathom, Otter) into an agent that extracts action items, deal stage, objections, and next steps, then writes them straight into the CRM record. No manual data entry, and a clean pipeline.

5. Content repurposing on autopilot

One long-form asset can become a newsletter, five LinkedIn posts, and a batch of tweets. Nobody has time to do it by hand.

Trigger on a new blog publish. An agent chunks the piece, generates platform-specific variants, and drops drafts into a review queue. A human approves in two minutes instead of writing for two hours.

6. Inbox triage that protects your focus

Executives lose hours to email. Build a classifier that tags incoming mail: urgent, needs-reply, FYI, spam. It drafts responses for routine requests and surfaces only what needs a human decision.

The measurable win is response time and reclaimed deep-work hours. Track "time to first reply" before and after and you'll have your ROI in a screenshot.

7. Onboarding sequences that adapt

New customer signs up. Instead of a static drip, an agent checks what they've actually done in your product and sends the next relevant step.

Haven't connected an integration? Send that guide. Already power-using? Skip the basics and pitch the upgrade. Behavior-driven onboarding lifts activation far more than time-based blasts.

8. Competitive and review monitoring

Set an agent to scan review sites, social mentions, and competitor pages daily. It summarizes sentiment, flags a scathing 1-star review the moment it lands, and alerts you to a competitor's price change.

You respond to problems in hours, not weeks. For a service business, one saved account pays for the whole system.

How to actually ship these

Don't try all eight. Pick the one where the pain is loudest and the metric is clearest.

Score by ROI, not novelty. Invoice chasing and lead follow-up touch cash directly. Start there.
Keep a human in the loop for anything customer-facing until you trust the outputs.
Instrument everything. If you can't measure hours saved or revenue moved, you can't defend the build.

The tooling is commodity now: n8n or Make for orchestration, an LLM API for reasoning, your existing SaaS for data. The moat isn't the model. It's connecting these pieces to a workflow that was quietly bleeding money.

Build one this month. Measure it. Then build the next.

If you'd rather have someone architect the whole stack for you, that's exactly what we do at Michael AI.

Originally published at getmichaelai.com

Off-the-Shelf Chatbot vs Custom AI Agent: The Decision Framework Nobody Gives You

Michael — Mon, 31 Aug 2026 12:00:53 +0000

Every team building with AI hits the same fork in the road. Do you plug in Intercom Fin, Voiceflow, or some "AI chatbot in 5 minutes" SaaS? Or do you build a custom agent on top of the OpenAI/Anthropic APIs, wired into your own stack?

The wrong answer costs you either six months of engineering time or a system that hits a wall the moment you ask it to do something real. Here's the framework I use with clients before writing a line of code.

Start with what the bot actually needs to do

Most "chatbot" projects are secretly two very different problems.

Problem A: Answer questions. "What's your refund policy?" "How do I reset my password?" This is retrieval over a knowledge base. Off-the-shelf tools are excellent here.

Problem B: Take actions. "Reschedule my appointment." "Check my order status, then issue a partial refund and log it in the CRM." This requires the bot to call your systems, chain steps, handle failures, and respect business logic.

If you're 90% Problem A, buy. If you're heavy on Problem B, buying often becomes a trap - you'll spend more fighting the platform's limits than you'd spend building.

The real cost comparison

SaaS chatbots look cheap until you scale. The honest math:

Off-the-shelf: $50-$1,500/month depending on seats and message volume. Fast setup (days). But you pay per resolution or per message, and costs climb non-linearly as usage grows. Customization is capped at whatever the vendor exposes.

Custom agent: Higher upfront (a few weeks of build time or an agency engagement), then you pay raw model tokens - often 5-20x cheaper per interaction at volume. You own the logic, the data, and the roadmap.

The crossover point usually lands around consistent daily volume plus any requirement the platform doesn't natively support. One integration the vendor charges $500/month for, or simply can't do, and the buy option stops being cheap.

Where off-the-shelf breaks

I've watched teams burn months trying to force a no-code platform to do things it was never built for:

Multi-step workflows that branch on live data
Writing back to systems (not just reading FAQs)
Custom auth and per-user context
Precise control over tone, escalation, and fallback behavior
Compliance requirements around where data lives

When you hit these, you're either stuck or paying enterprise pricing for a feature that's still someone else's roadmap.

Where custom is overkill

Going custom too early is the opposite mistake. If your use case is a support FAQ bot for a 10-person startup, spinning up your own agent, vector store, eval harness, and monitoring is a waste. Buy the SaaS, ship this week, revisit in six months.

Complexity you don't need is a liability, not a flex.

The middle path most people miss

The interesting option isn't build or buy - it's an orchestration layer that lets you buy the model and build the logic without a full engineering team. This is where a tool like n8n earns its keep.

You get an LLM's reasoning, your own tool integrations, and visual workflow control - a custom agent's flexibility at a fraction of the build cost.

Here's a stripped-down version of an agent node that decides whether to answer directly or call a tool:

// n8n Function node: route the user message
const message = $input.first().json.userMessage;

const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.openai.com/v1/chat/completions',
  headers: { Authorization: `Bearer ${$env.OPENAI_API_KEY}` },
  body: {
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: 'You are a support agent. If the user asks about an order, respond with JSON {"action":"lookup_order","orderId":"..."}. Otherwise answer directly.' },
      { role: 'user', content: message }
    ],
    response_format: { type: 'json_object' }
  },
  json: true
});

const result = JSON.parse(response.choices[0].message.content);

// Branch: hand off to the order-lookup workflow or reply directly
return [{ json: result }];

From there, a Switch node routes lookup_order to your database, formats the result, and feeds it back to the model for a natural reply. You've built Problem B behavior without a bespoke backend.

The decision checklist

Run your project through these questions:

Is it mostly answering or mostly doing? Answering leans buy. Doing leans build.
How many systems must it touch? Three or more integrations tilts toward custom or n8n.
What's your daily volume in 12 months? High volume rewards owning your token costs.
Do you have data or compliance constraints? Constraints usually kill off-the-shelf.
How unique is your logic? Generic = buy. Differentiated = build.
What can you maintain? Be honest about your team's bandwidth.

If you answered "buy" to most: grab a platform and ship. If you answered "build" to most but lack a full dev team, the n8n middle path is your friend. If you're a serious Problem B case at scale, invest in a proper custom agent.

The takeaway

Build vs buy isn't ideology - it's a fit question. Match the tool to the complexity, the volume, and the team you actually have. Start with the cheapest thing that clears your requirements, and only graduate to custom when the platform is the thing holding you back.

The expensive mistake is committing to either extreme before you've defined what the bot needs to do. Define that first. The answer usually writes itself.

Originally published at getmichaelai.com

Token Freeloading: Why Your Support Chatbot's Cloud Bill Doesn't Match Its ROI

Michael — Sun, 30 Aug 2026 12:00:40 +0000

You shipped an AI support bot. Deflection rates look great. Then finance forwards you the OpenAI invoice and asks a very reasonable question: why did it triple last month while ticket volume stayed flat?

Welcome to token freeloading. It's the slow leak that turns a profitable chatbot into a cost center nobody wants to own.

What token freeloading actually is

Token freeloading is any token spend that produces zero business value. Your bot burns compute, the meter runs, and the customer gets nothing useful in return - or worse, the traffic isn't even a customer.

The usual suspects:

Context bloat - stuffing the entire knowledge base into every prompt because retrieval was never tuned.
Retry storms - a flaky tool call fails, the agent retries three times, each retry re-sends the full context.
Bots talking to bots - scrapers, competitors, and curious engineers running your endpoint for fun.
Endless loops - a poorly bounded agent that keeps "thinking" until it hits the max token ceiling.
Whale users - the 2% of accounts generating 40% of token spend on questions your docs already answer.

None of these show up in a deflection dashboard. They only show up on the invoice.

Why it destroys ROI quietly

The math is deceptively simple. If a resolved ticket saves you $6 of human agent time and costs $0.40 in tokens, you're winning. But token cost isn't fixed - it scales with prompt size and conversation length, both of which drift upward over time.

Context creeps as you add more docs. Conversations get longer as you add "helpful" follow-up prompts. Six months later that $0.40 conversation costs $2.10, and nobody noticed because it happened one token at a time.

Step one: price every conversation

You can't control what you don't measure per unit. Aggregate monthly spend is useless. You need cost-per-conversation, tagged by user, intent, and outcome.

Wrap your LLM calls with a cost tracker:

PRICING = {
    "gpt-4o": {"input": 2.50 / 1_000_000, "output": 10.00 / 1_000_000},
    "gpt-4o-mini": {"input": 0.15 / 1_000_000, "output": 0.60 / 1_000_000},
}

def log_cost(model, usage, session_id, user_id, resolved):
    rate = PRICING[model]
    cost = (usage.prompt_tokens * rate["input"] +
            usage.completion_tokens * rate["output"])

    metrics.emit({
        "session_id": session_id,
        "user_id": user_id,
        "model": model,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "cost_usd": round(cost, 5),
        "resolved": resolved,
    })
    return cost

Now you can build the one report that matters: cost per resolved conversation. Unresolved conversations that still cost money are your freeloading signal.

Step two: cap the runaway cases

Most of your budget bleed comes from a small number of pathological sessions. Put hard limits in place before you optimize anything clever.

MAX_TOKENS_PER_SESSION = 25_000
MAX_TURNS = 12

class SessionBudget:
    def __init__(self, session_id):
        self.session_id = session_id
        self.tokens_used = 0
        self.turns = 0

    def check(self):
        if self.tokens_used > MAX_TOKENS_PER_SESSION:
            raise BudgetExceeded("handoff_to_human")
        if self.turns > MAX_TURNS:
            raise BudgetExceeded("handoff_to_human")

    def record(self, cost_tokens):
        self.tokens_used += cost_tokens
        self.turns += 1
        self.check()

When a session hits its cap, don't just cut it off - route to a human. That's a feature. If someone needs 25,000 tokens of AI thrashing, they needed a person 20,000 tokens ago.

Step three: fix the structural leaks

Once you have measurement and caps, go after the root causes.

Right-size the model per intent

A "where's my order" query does not need your flagship model. Route by classified intent. Cheap models handle 70% of support traffic at a fraction of the cost; reserve the expensive model for genuine reasoning.

Trim the context

Stop sending your whole knowledge base. A tuned retrieval step that returns the top 3 relevant chunks instead of top 20 can cut input tokens by 60%+ with no drop in answer quality.

Cache the obvious

A huge share of support questions are near-identical. Semantic caching - matching new queries against previously answered ones - lets you serve a response for zero LLM tokens.

def get_response(query):
    hit = semantic_cache.lookup(query, threshold=0.92)
    if hit:
        metrics.emit({"cache_hit": True, "cost_usd": 0})
        return hit.response
    response = llm_answer(query)
    semantic_cache.store(query, response)
    return response

Step four: govern the traffic

Rate-limit per authenticated user. Block unauthenticated bot traffic at the edge. Add per-account monthly token budgets for your API-tier customers so a single integration can't run up your bill.

Governance sounds bureaucratic, but it's just the same discipline you'd apply to any expensive downstream dependency. Your LLM is a metered API. Treat it like one.

The bottom line

Chatbot ROI isn't a launch-day number - it's a maintenance job. The deflection rate tells you the bot works. The cost-per-resolved-conversation tells you whether it's worth running.

Instrument first, cap the outliers, then optimize the structure. Do that and your bot stays in the black. Skip it, and token freeloading will quietly eat the savings you built the thing to capture.

If you're building AI support systems and want the cost governance baked in from the start rather than retrofitted after a scary invoice, that's exactly the kind of thing we build at Michael AI.

Originally published at getmichaelai.com