Most "AI agent on your phone" demos show the happy path: one clean voice command, one flawless run, cut before anything breaks. The interesting engineering is in everything the demo skips: permissions, app support, screen state that lies, and the confirmation boundary before an irreversible action.
This post walks through the 12 tasks phone agents are actually asked to do, but from an execution angle rather than a feature-list one: what each task needs to touch, where it gets hard, and why the interface an agent uses to act changes what's even possible. At the end I'll get into the screen-capture + USB HID approach we're building with Aiden, because it makes a different set of trade-offs than the app-only assistants most of these tasks assume.
The real question isn't "can AI control my phone"
It's "which tasks are safe, useful, and realistic today — and what does the agent have to touch to do them?"
Three execution models are competing right now, and they fail differently:
- App/API assistants — call documented APIs. Reliable where an API exists, blind where it doesn't.
- OS-level integrations (Apple Intelligence, Copilot+) — deep access, but locked to one vendor's hardware and app set.
- Screen-aware agents — interpret the UI visually and drive it. Work anywhere there's a screen, but inherit every ambiguity of reading pixels.
Keep those three in mind; each of the 12 tasks below lands differently depending on which one you're using.
The 12 tasks, ranked by how hard they actually are
Feasibility varies a lot. Reminders and research summaries are reliable today. Calendar, email, messages, meetings, travel, shopping, expenses and smart-home sit in a solid middle tier. Form filling and health support are the least mature — either because they touch fiddly interfaces or because they edge toward decisions that should stay with a human.
| # | Task | Hard part (the bit demos skip) |
|---|---|---|
| 1 | Reminders / task capture | Almost none. Parse natural language → structured task. The reliable baseline. |
| 2 | Calendar scheduling | Timezone math, conflict detection, write-access confirmation before it sends invites. |
| 3 | Email triage + draft | Drafting is easy; sending to the wrong recipient is the failure that kills trust. |
| 4 | Message summarization | Group-chat context windows and knowing which thread "that" refers to. |
| 5 | Research summarization | Source quality and not fabricating a citation. Reliable when read-only. |
| 6 | Meeting notes + follow-ups | Audio capture permissions and turning talk into action items without hallucinating owners. |
| 7 | Travel planning | Multi-app, multi-step state that goes stale mid-run (prices, availability). |
| 8 | Form filling / navigation | The API-less wall. No endpoint, no accessibility tree — this is where execution model matters most. |
| 9 | Shopping / price comparison | Cross-site state, and the confirmation boundary right before "buy". |
| 10 | Receipt / expense organization | OCR quality and structured extraction from messy inputs. |
| 11 | Smart-home automation | Device API fragmentation across ecosystems. |
| 12 | Health / fitness habits | Non-clinical only. The hard part is refusing to give medical advice. |
Two of these — 8 (form filling / navigation) and 9 (shopping) — are where the execution model stops being an implementation detail and becomes the whole game. Let's zoom in.
Task 8 is where most agents hit a wall
Form filling and in-app navigation is the task that separates the three models, because a huge number of real apps expose no API and no clean accessibility tree. An app/API assistant simply can't act there. An OS-level integration can, but only inside its vendor's boundaries.
This is the case for a screen-level approach: if the agent can see the rendered screen and send input like a human does, it doesn't need the app to cooperate. No SDK, no integration, no waiting for a vendor to ship an API.
The catch — and it's a real one — is observability. If you only send input blindly, you can't tell whether the tap landed, whether a spinner resolved to success or an error, or whether the screen even changed. Reading pixels to infer state is where screen agents get brittle.
How a screen-capture + USB HID approach handles it
This is the part I actually work on, so here's the concrete version rather than the marketing one.
Aiden is a physical mobile AI agent device: a small board that plugs into a phone or computer over USB. It captures the target's display over HDMI and sends actions back as a standard USB HID keyboard/mouse. To the phone, it looks like an external monitor plus a keyboard — no app installed on the target device, no root, no ADB.
That combination is a direct answer to the Task 8 wall and the observability problem underneath it:
- Acting without an API — HID input drives any app that accepts keyboard/pointer/touch, which is all of them. The app doesn't need to expose anything.
- Closing the observability gap — because Aiden also reads the screen over HDMI capture, the same loop that acts can see the result, instead of firing input blindly.
- Privacy posture — there's no Aiden backend; screenshots and audio go only to whatever model endpoint you configure, and the whole thing is self-hostable.
On the current development board, the agent runtime runs on-device (Go), with frame capture, a voice loop with VAD, the decision loop, and HID output as independent goroutines. It's a dev board, not a finished consumer product — the repo is the open-source firmware and agent runtime, not a shipping SKU:
github.com/AidenAI-IO/aiden-hardware-demo
The honest caveats stand: reading pixels to verify an action still breaks when loading states lie, and any screen-reading agent inherits the prompt-injection surface of whatever it looks at. Those are unsolved for everyone in this space, not just us — worth saying plainly on a dev audience.
The pattern across all 12 tasks
Whichever model you build on, the tasks that work today share a shape: bounded workflow, low blast radius, human confirmation before anything irreversible. Reminders and summaries are safe because the worst case is a bad note. Email, payments, account changes and health decisions are exactly where you want an explicit confirmation gate, no matter how good the model is.
The useful design question isn't "can the agent do more?" It's "can the user reliably stop or redirect it mid-run, and does the agent verify before it commits?"
FAQ
Which of these tasks are safe to automate without confirmation?
Reminders, research summaries, and read-only organization. Anything that sends, buys, changes an account, or touches health decisions should require an explicit user confirmation step.
Why can't a normal app-based assistant do form filling in every app?
Because many apps expose no API and no accessibility tree. Without one, an API-based assistant has nothing to call. Screen-level approaches get around this by reading the rendered UI and sending human-style input instead.
What's the catch with screen-reading agents?
Observability and injection. Inferring success from a re-read of the screen breaks when loading states lie, and an agent that reads screens will read whatever an attacker puts on them. Both are open problems.
Does Aiden run on the phone?
No — it's an external board that plugs in over USB, captures the screen over HDMI, and sends input over USB HID. Nothing is installed on the target phone. It's currently a development board, and the firmware/agent runtime is open source.
Can I self-host the model?
Yes. There's no vendor backend in the loop; model, STT and TTS endpoints are whatever you configure, so you can keep everything on your own infrastructure.
If the execution side of this is interesting, the firmware, HID gadget config and capture pipeline are all in the open-source repo: github.com/AidenAI-IO/aiden-firmware. More on the project at aidenai.io. Curious how others here are handling the observability problem on screen-driven agents — verifying an action actually landed is the part I keep coming back to.
Top comments (0)