Teaching our on-device assistant to say "I don't know"

We're building FamiliaSync, an offline-first family organizer where all data stays encrypted on your own devices. That constraint extends to the assistant: every query has to run on the phone, no cloud calls allowed. Here's what happened when we replaced our hand-written intent router with a small on-device classifier.

A smarter first step

Every request to the FamiliaSync assistant now starts with a small intent classifier trained on-device. It replaces the hand-written pattern rules that used to decide what you meant. The pipeline behind it is unchanged in spirit: your text is routed to the right tool, or handed to Needle 2 — the on-device model bundled with the app — when a request needs open-ended reasoning.

The training corpus was researched, not generated in bulk: 105 intents backed by roughly 2,400 example utterances carrying the slots real commands need — dates, times, assignees, quantities, amounts, categories, and recurrence. Thirty-five of those intents map to real tools in the app.

Training the 'no' cases

Here's the part we didn't expect to matter most: 70 of the 105 intents are documented-only. They cover things the assistant can't do yet — notes, stores, loans, meter readings — and they exist so that an unsupported request falls through to the model for a plain-language answer instead of being forced into an unrelated tool. Training the 'no' cases turned out to be as important as training the 'yes' cases.

A confidence gate sits on top: if the classifier isn't sure, it doesn't route at all — the model handles the request instead. And everything happens on-device. Training is lazy and local, inference is local, and no query ever leaves the phone.

Real phrasing broke it within hours

Shipping the router was the easy part. Real queries surfaced three misroutes the same day, and each one taught us something:

• "what are todays plan" fell below the confidence threshold — and a stray keyword let the model invent a meal called "Sri, Ram" from family names in the prompt. Loose keywords were stripped, and 'today's plan' phrasings now map to listing events.
• "what is my meal plan for today" confidently matched the wrong tool — a read question nearly overwrote the shopping list, stopped only by the confirmation gate. That tool now requires an explicit generate signal, and the corpus grew question-form examples.
• The meal-plan answer that did work dumped raw record IDs and the entire week. It now renders meal names and accepts filters like 'today' or 'lunch'.

From commands to conversation

The bigger shift: the assistant can now ask back. Tell it 'create a reminder' and instead of guessing, it asks what to remind you about — then completes the action from your reply. Tools can request a missing argument mid-flow, and partially collected details carry over between turns.

Follow-ups exposed gaps we didn't know existed. When people answer in their own words, dates arrive as 'friday 6pm', 'tonight', or '19th spetember 2026' — so the date parser now handles casual formats, including typo-tolerant month names. And when someone answers 'anyone' or 'doesn't matter' to a member prompt, the event is created unassigned rather than silently picking the first family member.

An assistant that knows when it doesn't know is more trustworthy than one that always guesses.

Conclusion

Nine commits in one day: a trained intent router, honest misroutes fixed in the open, and an assistant that asks follow-up questions — all running entirely on-device, with settings copy updated in four languages. Every query stays on your phone. That's the point.

---

FamiliaSync is in pre-launch — if on-device AI and privacy-first architecture interest you, the waitlist is open and the technical write-ups live on our blog.

Comments

[Ahmet Özel]

Replacing pattern rules with a trained classifier usually improves the average case and quietly worsens the tail, so the abstention path is the interesting part rather than the accuracy number. A classifier over 105 classes with no "I don't know" will always pick one, and a confident wrong intent is a worse experience than the old rules simply failing to match.

Curious how you set that threshold, and specifically whether it was calibrated on out-of-scope examples rather than held-out in-scope ones. Those measure different things, and an on-device assistant with no cloud fallback has nowhere to escalate when it abstains.

2,400 utterances across 105 intents is roughly 23 per class, which is workable but makes the rare intents the ones to watch.

[Familia Sync]

The abstention path is where most of the iteration went, happy to go deeper. The router is a cascade of three layers plus a post-classification gate, not a single classifier behind a threshold:

• Exact utterance match: a verbatim corpus phrase is a deterministic contract.
• Nearest-utterance match: token-level similarity against the corpus, plus a typo-tolerant edit-distance pass. This layer exists because the classifier's scores saturate at this corpus size: tens of thousands of formulaic utterances blur intent boundaries, so raw confidence isn't a calibrated probability. The lexical layer stays anchored to real corpus phrasing, which is also what protects the ~23-utterance classes you flagged.
• The classifier itself, for paraphrases the lexical layers miss.
• Slot extraction as the second abstention gate: a confident supported intent whose extractor can't fill required slots doesn't execute. It becomes a clarification question ("what should I remind you about?") or defers. Abstention happens twice: at confidence and at slot-filling.
• On out-of-scope: the 70 documented-only intents are trained classes in the corpus itself, not a held-out eval set. Their whole job is capturing unsupported requests (notes, loans, meter readings) into their own intent instead of snapping onto the nearest supported one. Generic chit-chat trains an explicit out-of-domain class on top of that. On "nowhere to escalate": abstention hands off to a small general-purpose model bundled with the app, still fully on-device. Escalation is local by definition, no cloud to lean on. You're right about the tail though a read question nearly overwrote the shopping list on day one, and it was the confirmation gate that caught it, not the classifier. Rare intents are still the thing we watch. Growing the corpus from real phrasings is the fix path so far.