Keyword rules missed 36 of 45 emergencies: how we layer safety checks in a multilingual health chatbot

Pranav Ajay — Sun, 15 Nov 2026 06:42:44 +0000

If you build a chatbot that people ask about their health, some of those people will be having an emergency while they type. Someone with crushing chest pain will sometimes open a chat window before they call anyone. A parent will type "my son swallowed a button battery" and wait for an answer.

When we started building Saluvo, an AI health companion that answers in 80+ languages, we made one rule before anything else: the emergency warning must never depend on the model. This post is about how we tried to do that, where our first version failed, and what we ship now.

Rule one: check before the model, not after

The tempting design is to put "if this is an emergency, tell the user to call for help" in the system prompt and trust the model. It works most of the time. "Most of the time" is the problem:

Models drift with prompt changes. A tweak that improves tone can quietly weaken the emergency instruction.
Streaming answers start with whatever the model writes first. If it opens with three paragraphs of context, the important line arrives late, or after the person has stopped reading.
If the model call fails, times out or hits a rate limit, there is no answer at all, and no warning.

So the first check runs before any model call, in plain code. It returns a red flag (emergency or mental health) and a reason, and the UI shows a banner with the local emergency number. The model still answers underneath, but the banner doesn't wait for it.

type RedFlag = { kind: "emergency" | "mental-health"; reason: string };

function detectRedFlag(text: string): RedFlag | null {
  const cleaned = stripBenignPhrases(text);           // "food poisoning", "stroke risk"
  for (const rule of mentalHealthFirst(RULES)) {      // so the right helpline shows
    if (rule.patterns.some((p) => matchesUnnegated(p, cleaned))) {
      return { kind: rule.kind, reason: rule.reason };
    }
  }
  return null;
}

(Simplified; the real one also handles phrase lists for languages that don't suit regex.)

The boring parts that matter

A keyword check sounds trivial. Three details made it usable:

1. Benign phrases are blanked out first. "Food poisoning", "heat stroke", "how to lower my stroke risk", "swimming stroke". Without this, the banner fires on half of all health questions, and people learn to ignore it.

2. Simple negation. "No chest pain, just a cough" shouldn't trigger the chest-pain rule. We look at a short window before each match for words like no, not, without, denies. This is English-only and crude, and we accept that: a false banner is far cheaper than a missed one.

3. Normalise what phones type differently. Curly apostrophes, case, Arabic versus Persian forms of the same letter, the zero-width joiner in Persian. If you don't fold these, the same word typed on two keyboards matches once and misses once.

And for Hindi speakers, a lot of real input is Hinglish in Latin script: "seene mein dard", "saans lene mein dikkat". Those need their own patterns next to the Devanagari ones. If you serve India, test with how people actually type, not with textbook Hindi.

Where it failed: 36 of 45

We wrote a set of 45 hard test phrases: indirect descriptions, other languages, the way a worried relative would type rather than the way a textbook would. One of them was "my son swallowed a button battery".

The keyword check missed 36 of them.

That wasn't a bug; it's what keyword lists are. They know set phrases. People in a crisis don't use set phrases, and a list for one language doesn't transfer to 80.

Layer two: a small, fast model as a net, never a gate

So we added a second check that every channel shares (web chat, WhatsApp and voice): a small, fast model reads what the person said, in any language, and returns strict JSON:

{ "danger": "none | emergency | mental-health",
  "reason": "a few English words",
  "local": "the same words in the person's language",
  "lang": "ISO 639-1 code",
  "handled": true }

The design choices that matter more than the prompt:

It can only add a warning, never remove one. If the keyword check fired, it stands, whatever the model says.
A failed or slow check counts as "nothing found", with a short timeout. It must never be the reason someone gets no answer. The keyword layer is still there underneath.
Reassurance from someone else doesn't lower urgency. The prompt says so explicitly: "my husband says it's only gas" is not a reason to downgrade chest pain.
What the person typed is data, not instructions. It's a classifier reading untrusted text; treat it like one.
"handled" tells us whether the main answer already told them to get urgent help, so on text channels we can add the safety line when it didn't.

The check also returns the reason in the person's own language, so a Tamil speaker gets a Tamil warning, not an English one with a Tamil answer below it.

Mental health is not "another emergency"

Two things we treat separately:

Mental-health rules run first, so someone who mentions both pain and self-harm sees the crisis line, not only the ambulance number.
The tone is different. Warmth first, the crisis line second, never a lecture. We list a crisis line for the countries where we have one, and a "find a free helpline near you" link everywhere else.

Pick the number by country, and say when you don't know

Showing "call 911" to someone in India is worse than useless. We map the country (from the request headers) to its emergency and ambulance numbers. When we don't know the country, the copy says "or your local emergency number" rather than guessing.

What we'd tell another team

Put the emergency check outside the model and before it. Then add a model check as a net, not a gate.
Write a hard test set before you write rules, and include how people really type in each language.
Measure misses, not just false alarms. A banner that fires too often is annoying. A banner that doesn't fire is the failure that matters.
Your rules are a first draft until a clinician reviews them. Ours are a first version too: they still need that review, and more testing in both directions.

None of this makes a chatbot a doctor, and it shouldn't pretend to be one. But if people are going to ask an AI about their health at 2 a.m. anyway, the least we can do is make sure "call now" is the first thing they see when it matters.

I'm Pranav, co-founder of Saluvo. If you're working on safety in health or crisis-adjacent AI, I'd like to compare notes: we've written more about how to use AI symptom checkers safely.

DEV Community: Pranav Ajay