Allerbot - AI skincare assistant

I built an AI skincare assistant that can't hallucinate a safety verdict

Most of the time a wrong answer from an AI is just annoying. You shrug, rephrase, move on. But there's a category of product where a confident wrong answer is the risk, and skincare for people with diagnosed allergies is squarely in it.

"Yes, that moisturizer is fine for your fragrance allergy" is the one sentence I never want a language model to improvise. Someone with a real contact allergy will act on it.

I built AllerNote on my own. AllerBot is the assistant inside it. That single constraint — the model is not allowed to be wrong about safety — ended up shaping almost every decision I'm going to walk through here.

The rule everything else hangs off

The obvious version of this product is a chatbot with a nice system prompt: "You are a skincare expert. Here are some ingredients. Tell the user whether it's safe." It demos great. It's also quietly dangerous, because the verdict — the single load-bearing word in the whole interaction — is generated text. It can be wrong, fluently, with total confidence.

So I turned it inside out. In AllerBot, the model is never allowed to state a safety verdict from its own knowledge. A verdict isn't a sentence it writes; it's the return value of a tool that checks each ingredient against the user's saved allergens at a given severity. The model's job is smaller and a lot safer: figure out which tool to call, then explain the structured result in plain English. If the database didn't say "safe," the model can't say "safe" either.

That's the spine. Everything below exists to enforce it.

The model picks the tool and narrates the answer. The verdict itself comes back from the database as a return value — never as a sentence the model made up.

A tool-calling agent, not a prompt

AllerBot runs on Gemini Flash and Flash-Lite models with native function calling. Under the hood it's a pretty standard agent loop: call the model with the tool schema, run whatever tools it asks for, hand the results back, and repeat until the model stops asking for tools and just talks.

The loop ends when the model returns plain text. The 5-iteration cap is there for the times it doesn't.

There are about a dozen typed tools. The ones that carry the weight:

• get_user_allergen_profile — the signed-in user's allergens, with severity and notes.
• scan_ingredients / check_product_safety — run an ingredient list against that profile and return safe / warning / danger per ingredient. These are the only things allowed to produce a verdict.
• lookup_ingredients_batch — metadata for N ingredients in one call.
• recommend_safe_products — filter a catalogue by the user's allergens, plus anything they mentioned mid-chat but haven't saved yet.
• save_allergens_to_profile — the one write path, and it's gated behind explicit consent.

Within a single turn, every tool the model requests runs concurrently via asyncio.gather, and the whole batch goes back as one tool-response block. The 5-iteration cap is unglamorous but necessary: an agent that's allowed to keep calling tools will, every so often, decide to keep calling tools forever.

Getting it to shut up is mostly schema discipline

Wiring up function calling is the easy part. The SDK does it. The hard part is convincing a helpful model to stop being helpful at the exact wrong moment.

Left to its own devices, a good model will happily answer "is this safe?" directly, because it "knows" what fragrance is. You're fighting that instinct, and I ended up doing it in three places:

1. The system prompt is a contract, not a personality. Answer only from tool outputs. Don't invent statistics or verdicts. Refuse out-of-scope stuff, no diagnosis, and always call a tool before making any safety claim.
2. The tool descriptions are written as imperatives, because the model reads them as instructions and follows them weirdly literally. So scan_ingredients doesn't say "checks a product" — it says "always call this before stating whether a product is safe; never guess." The lookup_ingredients_batch description flat out tells it to pass every ingredient at once instead of looping one at a time.
3. The iteration cap so a confused loop fails closed instead of running off.

None of this is fancy. It's just the gap between a demo and something you'd hand to a person managing a real contact allergy.

Photos come in the same door

Nobody wants to type out an ingredient list off the back of a bottle. So you can paste a photo of the label straight into the chat. It goes to Gemini Vision for OCR — curved labels, INCI vs. common names, even traditional and ayurvedic ingredient terms — with a local doctr model as a fallback when that path struggles.

Here's the part I'm quietly proud of: the extracted ingredients flow into the exact same scan_ingredients tool as typed text. Two input modalities, one verdict path. The photo doesn't get its own slightly-looser code route where the grounding rule happens to not apply, because there is no second route to slip through.

Both inputs land on the same tool. There's no separate, looser path for the photo to take.

My favourite bug had nothing to do with AI

The least glamorous war story is the one I tell the most.

The first version held a database transaction open for the entire request, model round-trips included. On hosted Postgres behind PgBouncer, idle-connection timeouts would quietly kill the socket while the model was still thinking, and the request would then fall over on its next query with ConnectionDoesNotExistError. Intermittent, naturally, because whether it broke depended on how long Gemini took to answer.

The fix wasn't a retry or a longer timeout. It was changing how the agent relates to the database at all: hold no long-lived session. Each step — load history, save the user's message, run a tool, save the reply — opens its own short-lived session, commits, and closes. The slow model calls now happen with zero connections held open. As a side effect, a successful scan commits on its own, so it survives even if saving the final reply fails.

Before: one connection, held across the slow part, dying on a timeout. After: short-lived sessions, nothing held while the model thinks.

The takeaway generalises past this one bug: an LLM call is the slowest thing in your request by a wide margin. Look hard at what you're holding open — connections, locks, transactions — while it runs, and let go of it first.

Not everything should become a paragraph

Some tool results were never meant to be prose. When recommend_safe_products comes back with matches, I don't want the model writing five products up in markdown. I want product cards. When a user mentions an allergen they haven't saved, I don't want a sentence politely asking them to go save it. I want a "save this to your profile?" button.

So the loop pulls the structured payloads out of the tool results and returns them next to the text. The model writes the human sentence; the frontend renders the actual component. It keeps the model out of formatting UI, which it's bad at and which burns tokens for no reason.

What I'd tell you if you're building something where being wrong matters

A few things I keep coming back to:

Decide what the model is allowed to author. In most apps the answer is "everything," and that's fine. In ours it's "the explanation, never the verdict." Drawing that line up front changes the architecture, not just the prompt.

Put the source of truth in code, not in weights. RAG still hands the model retrieved text and asks it to compose the answer. If the answer has to be deterministic, make it the output of a function and let the model narrate what the function returned.

Your tool schemas are part of the prompt. The descriptions are instructions, and the model follows them more literally than you'd expect. Write them like you're briefing a contractor who takes everything at face value.

And the boring one that bit me hardest: watch what you hold open across the model call.

The stack

Nuxt 4 / Vue 3 on the front (SSR/ISR so crawlers and answer engines get real HTML), FastAPI with async SQLAlchemy and pgvector on the back, Firebase for auth, Gemini Flash for the agent and vision, and doctr as the OCR fallback. Built and maintained solo.

---

If you want to see the "never guess" thing in action, AllerBot is free and works without an account: allernote.com/allerbot. Sign in and it reads your saved patch-test allergens, so the verdicts get personal. Happy to get into the agent loop, the grounding contract, or the OCR path in the comments.