I Built an AI Assistant for Android — The Hard Part Wasn't the LLM
Building an AI assistant sounds simple at first.
User sends a message → LLM processes it → assistant responds.
But the moment I started thinking beyond a chatbot, that architecture wasn't enough.
I wanted my Android assistant, FRIDAY, to eventually understand context, remember useful information, access tools, search for current information, interact with Android capabilities, and — most importantly — know when it should not take an action.
That changed the problem completely.
The Idea Behind FRIDAY
FRIDAY is my attempt to explore a simple question:
«What does it actually take to turn an LLM into a useful personal AI system?»
I don't want FRIDAY to simply be another interface around an LLM API.
The system I'm working toward looks more like:
User Request
↓
Context Processing
↓
Memory Retrieval
↓
Intent + Policy
↓
Tool Decision
↓
LLM / Tool Execution
↓
Permission Check
↓
Action
↓
Memory Update
The LLM is only one component.
The interesting engineering happens around it.
🧠 Memory Before Intelligence
One of the biggest limitations of a basic chatbot architecture is that every interaction can effectively become a new interaction.
For an actual assistant, that isn't enough.
FRIDAY needs different types of context.
There is information relevant only to the current conversation.
There is information that may be useful later.
And there is information that should never be stored unnecessarily.
So rather than blindly putting everything into a prompt, I'm exploring a memory layer where relevant information can be retrieved only when it is actually needed.
This introduces another interesting problem:
How does the system decide what is worth remembering?
Saving everything isn't memory.
It's storage.
Useful memory requires selection, retrieval and forgetting.
🛠️ Giving the AI Tools
Answering questions is one thing.
Taking actions is completely different.
Imagine asking:
«“Open Spotify.”»
The assistant probably doesn't need a powerful LLM call to execute that.
But consider:
«“Find something relaxing to play while I'm studying.”»
Now the system needs to understand intent before selecting an action.
This is why I'm designing FRIDAY around a tool layer.
Instead of allowing the model to directly control everything, capabilities can be exposed as specific tools.
Conceptually:
Intent → Tool Router → Tool → Result
The model shouldn't automatically receive unlimited access to the device.
And that leads to the part I'm particularly interested in.
🔐 Permission-Aware AI
Giving an AI agent tools is powerful.
It's also where things can go wrong.
Opening an application and sending a message should not necessarily have the same permission level.
Neither should:
- Reading information
- Searching the internet
- Opening applications
- Creating reminders
- Accessing files
- Sending messages
- Changing device settings
I'm therefore exploring a permission/policy layer between decision and execution.
Something like:
AI proposes action
↓
Policy evaluates action
↓
Low risk → Execute
Sensitive action → Ask user
Restricted action → Reject
The goal isn't maximum autonomy.
The goal is controlled autonomy.
🌐 When Should FRIDAY Search the Internet?
Another surprisingly interesting problem is deciding when not to rely on the model.
Ask:
«“Explain binary search.”»
Web search probably isn't necessary.
Ask:
«“What happened in AI today?”»
Now current information matters.
Ideally, the assistant should recognize that difference before generating the final answer.
That means web search becomes another tool rather than something that runs for every request.
⚡ Does Every Request Need an LLM?
Probably not.
This is where semantic caching becomes interesting.
Suppose similar requests have already been processed.
Instead of automatically doing:
Request → LLM → Response
the system could attempt:
Request
↓
Semantic Similarity Check
↓
Relevant cached result?
YES → Reuse / process cached information
NO → Continue to LLM
That potentially reduces unnecessary model calls while improving response latency.
But caching AI responses creates its own challenges.
When is a cached response too old?
How similar is “similar enough”?
Should responses involving current information ever be cached?
These are exactly the kinds of engineering decisions I want to explore through FRIDAY.
🎯 Context Engineering > Giant Prompts
Another thing I've learned while experimenting with AI systems is that giving the model more text doesn't automatically give it better context.
Instead of creating one enormous system prompt, I want FRIDAY's context to be assembled dynamically.
The model should receive the information relevant to the current request:
Relevant memories.
Available tools.
Permissions.
Current state.
Retrieved information.
Conversation context.
Not everything the system has ever seen.
For me, this is where context engineering becomes much more interesting than simply writing a better prompt.
🤖 The Architecture I'm Working Toward
At a high level:
User
↓
Intent / Context
↓
Memory Retrieval
↓
Policy Layer
↓
Decision Engine
↙ ↓ ↘
Memory Tool LLM
↓
Permission Layer
↓
Action
↓
Result / Response
↓
Memory Update
Each layer should have a specific responsibility.
That separation is important because I don't want the LLM to become the entire application.
It should be one reasoning component inside a larger engineered system.
The Questions I'm Exploring
FRIDAY is already a project I've worked on, but I now want to share more of the engineering behind it as I continue improving it.
Some of the questions I want to explore publicly are:
Memory
What should an assistant remember?
How should memories expire?
How do you retrieve the right memory without flooding the context window?
Tools
How should tools be registered?
How does the system select between similar tools?
What happens when a tool fails?
Security
Which actions can happen automatically?
Which actions always require confirmation?
How do you prevent the model from bypassing those rules?
Context
How much information should reach the model?
When should context be retrieved?
When should information be discarded?
Performance
Can semantic caching reduce unnecessary LLM calls?
What should run locally?
What actually needs a cloud model?
Why I'm Sharing the Process
There are already countless AI chatbot demos.
I'm more interested in what happens after the chatbot demo.
How do we turn these models into reliable software systems?
How do we manage memory?
How do we give them tools safely?
How do we control their actions?
How do we evaluate whether their decisions are correct?
And how much of the system should actually depend on an LLM?
That's what I want to use FRIDAY to explore.
I'll be sharing the architecture, experiments, failures and individual components as I continue engineering the assistant.
Because perhaps the interesting future of AI assistants isn't just about building a smarter model.
It's about building a better system around the model.
What would you consider the most important component of a personal AI assistant: memory, tools, context, security, or something else?
I'd be interested to hear how other developers would approach the architecture.
Top comments (0)