In 2024 a tribunal in Canada made Air Canada honor what its website chatbot had told a customer about refunds. The airline argued the bot was responsible for its own words. The tribunal said the company is responsible for everything on its site, chatbot included.
For a small business that is the real problem with site chatbots. A wrong opening time or an invented price costs more than the bot saves. So I built a demo where the model is only allowed to draft, and plain code decides what the customer sees: Front Desk.
The setup
The demo runs on Juniper Street Bakery, a bakery I made up. Its whole "site" is 13 short entries: hours, prices, cakes, allergens, delivery, payment. The assistant knows nothing else.
The model is Gemma 4 E2B, running in the browser through WebLLM at temperature 0. Every question goes through four steps:
- Find. Code picks up to three entries that match the question. If nothing matches, the model is not asked at all and the customer gets the phone number.
- Draft. The model reads only those entries and returns JSON: the answer, the id of the entry it used, and the sentence it took the answer from.
- Check. The quoted sentence has to be in that entry. Every number in the answer has to be in it too ("4 pm" counts as 16:00, "$4" does not count as $4.50). And every word that carries meaning has to appear somewhere in what the model was given.
- Show or step back. If the first sentence fails a check, the draft is thrown away and the customer sees the entry's exact text or the contact details. Later sentences that fail are cut.
What the first runs showed
I wrote 24 test questions with known answers, 6 of them about things the site never mentions (pizza, a loyalty card, Christmas hours).
Run 1: 17 of 24. No invented fact reached the customer, but most misses had the same cause. Asked "Do you deliver bread?", the model set answerable: false, because to it "no" felt like "not answerable". Rewording that rule fixed it.
Run 2: 21 of 24, and a real hole. Asked "Do you have a loyalty card?", it answered "No, we do not have a loyalty card." The site says nothing about loyalty cards. There were no numbers in that sentence, so my number check let it through.
That is where the word check came from: "loyalty" is not in anything the model was given, so the sentence is not backed by the site. The same check later cut "That is our capacity." from an otherwise correct answer about seats.
On questions I hadn't tuned on
After writing the checks I added 7 new questions and ran all 31 live. It got 28 right, 5 of the 7 new ones. One miss was my own bug: the model wrote "$8.00" and the site says "$8", and the number check treated them as different. With that fixed, the same recorded drafts pass 29 of 31.
The two misses left are worth showing, because the checks can't catch them:
- "Is the bakery open on Christmas Day?" got the regular opening hours. True, every word on the site, but not an answer.
- "Is the cinnamon roll vegan?" got a correct "no", but pointed at the pastry prices instead of the vegan entry.
A wrong yes or no where every word is on the site would pass the same way. That is why I think a bot like this needs a test set of the business's real questions before it goes live, not only a good prompt.
Do small businesses even have chatbots?
I took 1,020 business websites in Austin from OpenStreetMap (cafes, restaurants, dentists, clinics, salons, bakeries), dropped chains that share one domain, and fetched each homepage once. 592 loaded. 27 had any chat widget at all. 15 of those were Wix's built-in chat with the owner. The only ones I could identify as AI assistants were on chain sites.
So the chatbot market is crowded, but the local cafe mostly has no assistant at all.
A widget without a model
That led to a second mode. The widget is one script tag. Without a model endpoint it runs in quote mode: it finds the entry that matches and shows it word for word, or gives the phone number. It cannot make anything up and costs nothing per question. On the 31 test questions it shows the right entry or declines 29 times, though I also used those questions while choosing search words, so expect less on a new site.
With an endpoint added, it drafts answers and runs the same checks, falling back to quote mode when one fails.
Try it
- Demo with recorded answers, no download: https://phazww.github.io/front-desk/#recorded
- Live model and the test set: https://phazww.github.io/front-desk/#tests (needs WebGPU, downloads the model once)
- Code: https://github.com/phazww/front-desk
I set this up for small businesses on their own content, with a test set of their customers' questions. Cafes and bakeries first: https://phazww.github.io/front-desk/cafes/
What would you check that I'm not checking?
Top comments (0)