I built a production AI agent as a Honda service advisor. Then I read the textbook.

Anderson — Sat, 25 Jul 2026 16:22:26 +0000

I'm a Honda service advisor in South Florida. My day is a service drive: customers dropping off cars, phones ringing, and the same fifteen questions over and over. How much is a brake job? Is this covered? Can I come in Saturday? What does the wrench light with the "A1" mean?

So I automated the repetitive half of my own job and kept the human half. It's an SMS agent — customers text the dealership's number over iMessage, and an LLM system answers from the actual owner's manual, checks their service contract, books appointments conversationally, and escalates to my Telegram when it needs a human. It's been live with real customers for months. Six languages, because my customers speak six languages.

Here's a real exchange, after hours, in Spanish:

Customer: Hola, quiero hacer una cita
Agent: Claro — ¿qué tipo de servicio necesitas para tu Honda Ridgeline? Recomiendo en la mañana para que tengas tu vehículo de vuelta el mismo día.
Customer: El sábado en la mañana
Agent: Lo siento, no estamos abiertos los sábados. ¿Te vendría bien el viernes? Abrimos a las 7 — ¿te funciona a las 8:00 AM?

The agent knew my schedule, refused the closed day, offered a concrete alternative, and never left the customer's language.

I built all of it by hitting problems and fixing them. Then I read Chip Huyen's AI Engineering, and had the strange experience of finding names for things I'd invented in the dark — plus one glaring gap I hadn't noticed at all.

This is the mapping, and the gap.

The rule I learned by breaking production: the model classifies, code guarantees

If I could hand one sentence to someone shipping their first agent, it's that one. It cost me three separate production bugs to write it down.

Bug 1: the model cannot compare two dates

A customer with an expired service contract asked whether a repair was covered. The agent told them yes. It was reading the coverage data correctly — the expiration date was right there in the context — and it still got the comparison wrong.

I did what everyone does first: I fixed the prompt. Put today's date in the system message. Spelled out "compare the expiration date to today's date." Told it to be careful.

It failed 3 out of 3.

So I stopped asking. Now the date comparison happens in Python and the model only gets to relay the verdict:

def _annotate_expirations(in_md: str) -> str:
    """Stamp a computed past/active verdict next to every expiration date in the
    portal data. LLMs are unreliable at date comparison, so the comparison is
    done here and the model only relays the verdict."""
    ...
    if d < today:
        return line + "  ⚠️ [COMPUTED: this date has already PASSED — no longer valid]"
    return line + "  [COMPUTED: date is in the future — still active]"

Every dated line in the coverage data arrives at the model pre-judged. There is no date math left for it to get wrong. That function now has seven unit tests, and they cost zero API calls to run.

Bug 2: the model cannot add up a repair order

Customer asks for three services and then, "so what's the total out the door?" The agent quoted three correct prices and then produced a total that was simply wrong — shop supplies and Florida sales tax applied to a subtotal it had added up by itself.

Same fix. The fee formula lives in code exactly once:

def compute_total(prices: list[float]) -> dict:
    """Sum quoted service prices into an out-the-door breakdown, rounded to cents.

    Single source of truth for the fee formula so it never drifts between the
    prompt and the code:
      shop supplies = min(subtotal × 10%, $59.50)
      tax           = (subtotal + shop supplies) × 7%
    """

The model's job is to notice that the customer asked for a total and to say the resulting numbers like a human. Not to be a calculator.

Bug 3: the model saying "done" is not evidence that it's done

The booking agent returns structured output — a Pydantic schema, enforced by the API — with a complete flag. Early on, a booking saved with no service type in it. The model had marked itself complete, and I believed it.

Now the server independently re-checks that every required field is really filled before anything is written or emailed. The model's self-assessment is a suggestion. The check is the authority.

Three different bugs, one shape: any claim the model makes that code can verify, code should verify. Everything downstream of that rule got more reliable — and cheaper to test, because assertions on pure functions don't need an API key.

The stuff I'd built without knowing it had a name

Reading the book was less "here's something new" and more "oh, that's what I've been doing":

Retrieval-augmented generation. I'd indexed Honda owner's manuals into Pinecone because customers ask questions whose answers are buried in a PDF nobody opens. Turns out that's the canonical use case, not a hack.
Guardrails, input side. Incoming messages get scored and sanitized before they reach any agent — I wrote it after reading about prompt injection and getting nervous about a public phone number. It's a named layer with a literature behind it.
Structured outputs. I moved booking extraction from regex over prose to an enforced schema after data markers started leaking into customer texts. ("[BOOKING_COMPLETE]" is not a thing you want a customer to receive.)
Routing / orchestration. One classifier call per message decides intent, language, vehicle, and whether to escalate, then dispatches through a registry. Adding a capability is one registry entry plus one handler. I built it because a single mega-prompt got unmaintainable around 400 lines.
Context economics. Pricing and policy blocks get injected only when the customer's message actually needs them; vehicle context loads once per session and caches. I did this because my token bill got my attention.

None of that made the book a waste of time. The opposite: I paid for each of those lessons in production incidents, with real customers on the other end. The book costs thirty dollars. If you're about to build this, buy it first — you'll still hit your own bugs, but you'll hit better ones.

The gap: I had no evals

Here's what reading it actually changed.

My quality signal, for months, was "a customer complained, or a scripted terminal conversation happened to catch it." That's not a signal. That's luck with a feedback delay.

So I built the deterministic layer first — no LLM judges, no fancy framework, one file:

python evals.py                    # full suite
python evals.py --runs 3           # more runs → better flakiness signal
python evals.py --only orchestrator

Over 130 cases now. Every single one is either a real production bug or a behavior I can assert exactly. The sections: date annotation, orchestrator classification, tech/pricing answers, the outreach flow, appointment confirmation, price gating, total math, routing, and maintenance-code decoding. Most of them are pure functions — they run in seconds and cost nothing.

Three things about it worth stealing.

1. The hallucinated-price detector. My favorite eval in the suite, and it's nine lines. Regex every dollar amount out of the agent's reply, and assert each one exists in the pricing file:

def _known_amounts() -> set:
    with open(PRICING_PATH) as f:
        return set(_PRICE_RE.findall(f.read()))

def _hallucinated(reply: str) -> list:
    known = _known_amounts()
    return [a for a in _PRICE_RE.findall(reply) if a not in known]

Deterministic hallucination detection for the one category of hallucination that would actually cost my dealership money. No judge model, no rubric, no ambiguity. If you have a domain where the ground truth is a file, you can write this today.

2. It found a dormant bug in its first hour. The orchestrator can classify a short follow-up ("Why?", "How come?") as follow_up, which sticky-routes back to whichever agent answered last. Nice feature. Except follow_up was missing from the intent registry, so the validator silently coerced every one of those verdicts to tech — the entire sticky-routing path was dead code, and had been for weeks. Nothing crashed. No customer wrote in to say "your routing table has a missing enum member." It took writing down what I expected to happen for the gap to become visible.

3. Flakiness is a result, not a nuisance. Each LLM case runs N times and reports a pass rate. A case that passes 4/5 is telling me something a binary green check would hide. Cases where the model is unreliable but code catches it downstream are marked known_flaky — they report their rate and don't fail the suite. That's honest: the model is flaky there, and the guard is the thing I'm actually relying on.

The next layer is an LLM judge over full transcripts with binary rubric questions — is the reply native-sounding in the customer's language? is it grounded in the provided manual context? was escalating the right call? — calibrated against transcripts I label myself, since I'm both bilingual and the domain expert. That's the nice thing about building for a job you actually work: I am the ground truth.

If you're shipping your first agent

Write down what the model must never be trusted with. Dates, arithmetic, and its own claims of completion are on my list. Yours will be different but it will not be empty.
Make the model's job "notice and phrase," not "compute." Reliability went up and prompts got shorter.
Your first evals should cost zero API calls. Pure functions and regex over outputs caught more real bugs than anything clever.
Seed the suite with every bug you've already shipped. It's the cheapest dataset you'll ever have and it's already labeled by production.
Track pass rates, not pass/fail. Non-determinism is data.
Domain knowledge is the moat. The reason this agent knows not to book Saturday, that an expired pass gets quoted at regular price, and that "A1" means oil change plus tire rotation isn't the model. It's a decade on a service drive.

The code is on GitHub: Service-Advisor-AI-Agent. Architecture diagram, the eval suite, and a demo video of a real conversation are all in the README.

I'm a service advisor teaching myself to be an engineer in public, and I'm looking for applied-AI / conversational-AI work. If you build in this space — dealer tech especially — I'd like to hear from you.****

DEV Community: Anderson