Why our Messenger booking bot doesn't use an LLM to decide what to say (and went from 55% to 71% on real guest messages)

Jonald Penpillo — Wed, 04 Nov 2026 03:18:06 +0000

By Jonald Penpillo, AI developer and software engineer in General Santos City, Philippines.

Most guests of a sports venue in the Philippines don't open a booking page. They open Facebook Messenger and type something like:

magkano ang driving range

That's Tagalog for "how much is the driving range?". The next guest writes in English, the one after in Bisaya, and plenty write in Taglish, switching languages mid-sentence. They expect an instant answer, at any hour, and an exact price.

This is the story of Range AI, the assistant I built for Brigadahub, the booking and payments platform of Brigada Hub in General Santos City: a 30-bay golf driving range, pickleball courts with open play, and a kids playground. Range AI answers guests on the website chat and on Facebook Messenger, and it can take a guest from a first question all the way to a paid booking inside the chat.

On real guest messages, its accuracy went from 55% to 71%. Here's how it's built, and why the most important design decision was not letting a language model decide what to say.

The one rule: never invent a price

A venue chatbot that hallucinates is worse than no chatbot. If it quotes ₱200 when the rate is ₱250, the front desk has an argument and management loses trust in the whole system. A generic "LLM with a system prompt" is fluent, but fluency is exactly the problem: it will confidently produce a plausible wrong number.

So the system is split into two jobs:

Deciding what the answer is. This is a classification problem.
Saying it naturally. This is where a language model helps.

Only the second job uses an LLM, and only behind a guard.

Step 1: classify the intent

Every incoming message goes through an intent classifier: word-level and character-level TF-IDF features feeding a calibrated LinearSVC (scikit-learn), covering about 94 intents, from driving-range rates and club rental to pickleball court rental vs. open play, opening hours and the playground.

Why a classic model instead of an LLM?

Character n-grams handle Filipino chat. Guests misspell, abbreviate and mix languages. Character-level features catch "mgkano", "magkano" and "how much" landing on the same intent far more robustly than word features alone.
Calibration gives an honest confidence score. A calibrated classifier's probability means something, so "I'm sure this is a pricing question" and "I'm guessing" look different in the numbers instead of being the same raw score.
It's fast, cheap and testable. Every prediction can be checked against a labelled set of real messages, so a change either improves the score or it doesn't ship.

Step 2: answer from approved replies only

Each intent maps to a management-approved reply: the venue's own wording, with the real rates and links. For the driving-range question above, the reply is fixed text with the public rates (₱250 for 1.5 hours with unlimited balls, club rental ₱150, extension ₱150 per hour) and a follow-up question about the date and time of the visit.

The bot never writes a price. It retrieves one that management already signed off.

Step 3: let the LLM reword, behind a fact guard

Fixed replies can sound robotic, especially when the guest wrote in Bisaya and the reply bank is in English. So the approved reply is passed to Gemini Flash (via OpenRouter) to reword it naturally and match the guest's language: English, Tagalog, Taglish or Bisaya.

Then a fact guard checks the reworded text against the approved original and blocks any rewrite that changed a price, an amount or a link. The LLM improves the tone; it never gets the final say on facts.

Step 4: book and pay inside the chat

Answering questions is half the job. When a guest wants to book, Range AI turns the conversation into a booking flow: it asks whether it's a court rental or open play, offers live slot chips from real availability, shows the waiver and a quote, and ends with a pay-now button that completes the booking through PayMongo.

On the website, the same platform runs a five-step booking flow with live bay and court availability and PayMongo checkout, so guests can book whichever way they prefer.

Step 5: know when to hand off

When the bot shouldn't handle a conversation, Range AI hands it to staff and sends an email alert. A fast "let me get a teammate" beats a confident wrong answer.

Measuring on real messages: 55% → 71%

Accuracy was measured on real guest messages from an anonymized chat dataset, not on examples we wrote ourselves. That distinction matters: test phrases you write yourself are clean and predictable, while real guests misspell, abbreviate and switch languages mid-sentence.

The first version scored 55% on those real messages. Iterating on the classifier and its training data against that same real-world set brought it to 71%. Every change was judged by one question: does the score on real guest messages go up?

71% is not 100%, and that's fine, because the system is designed around the classifier sometimes being wrong. Staff handoff catches what the bot shouldn't answer, and the fact guard means a reworded reply can never come out with a changed price.

What I'd tell anyone building a business chatbot

Separate deciding from wording. Use the LLM where fluency matters and something deterministic where correctness matters.
Measure on real user messages. Your own test phrases are always easier than what customers actually type.
Calibrate your classifier. A confidence score is only useful if it's honest; calibration is what makes it mean something.
Guard the facts. Prices, amounts and links should be checked mechanically, never trusted to a model.
Finish the job. The win isn't answering "how much?". It's ending the conversation with a confirmed, paid booking.

Stack: React 19, TypeScript, Express 5, Supabase, PayMongo, Python, FastAPI, scikit-learn, Gemini via OpenRouter, Messenger API.

I'm Jonald Penpillo, an AI developer and software engineer in the Philippines. I build AI assistants and chatbots (RAG, LLM agents, Messenger bots) and the booking, payment and business systems they run in. If you need a developer to build yours, see penpillo-jonald.online or the full Range AI case study.

DEV Community: Jonald Penpillo