When people picture the hard part of building a personal AI assistant, they usually picture the model: which LLM, which prompt, which fine-tune. In our experience, the model is the part that's easiest to swap and the part users notice least. The hard problems are everywhere around it.
This is a practical list of where the real work tends to hide. It's opinion drawn from building in this space, not a benchmark.
1. Getting information in with no friction
An assistant is only as good as what it has captured. And people capture things in the middle of other things: while walking, in a chat thread, forwarding an email. If saving something takes more effort than remembering it, it won't happen.
That means supporting messy, multimodal input (text, voice notes, forwarded messages, web pages) and turning it into something usable without asking the user to fill in forms. We covered the user side of this in the capture inbox method.
2. Understanding time
"Next Friday", "in two weeks", "the day after the meeting". Relative dates are ambiguous, depend on the user's time zone, and depend on when the message was sent. Misparse one and you get a reminder at the wrong time, which destroys trust quickly. We wrote about the failure modes in why AI reminders misread relative dates.
3. Identity and entities
Is "Sam" the same Sam as last month? Is "the dentist" a person, a place, or a calendar entry? Resolving references across notes is a data-modelling problem that no prompt solves on its own. See how AI extracts people, places and dates from notes.
4. Retrieval that knows what "relevant" means
Semantic similarity is necessary but not sufficient: you also need recency, supersession and exact lookups. We discussed this in retrieval-augmented generation for personal notes.
5. Being present on many surfaces
Users live in chat apps, email, browsers and calendars. Supporting several surfaces means consistent behaviour, shared state and different constraints on each. A feature that works in a web UI may be impossible in a messaging thread.
6. Scheduling and follow-through
An assistant that only reacts is a chatbot. Following through means background jobs, retries, notification delivery and respecting quiet hours, all of which are classic distributed-systems problems wearing a friendly interface.
7. Trust, privacy and control
Personal data is sensitive. Export, deletion and clear permissions are product features, not legal footnotes. See reviewing connected apps and AI permissions.
What this means for builders
- Treat the LLM as a replaceable component behind clean interfaces.
- Invest early in capture, time parsing and entity resolution.
- Design retrieval as a pipeline, not a single vector query.
- Build for multiple surfaces from day one.
- Make scheduling reliable before making it clever.
None of this is glamorous, and none of it moves on a leaderboard. But it's where the difference between a demo and a daily-use product tends to come from.
This is the work we're doing at Brinn, a personal AI for reminders, lists and memory across WhatsApp, email, web and desktop. If you're building something similar, what did you find was the unexpectedly hard part?
Top comments (0)