DEV Community: Nat

Aiden: A Physical AI Agent Device for the Phone You Already Have

Nat — Wed, 16 Sep 2026 08:22:51 +0000

Hi, dev.to community. We're Aiden, and this is our first post here, so let's start with the obvious question: what are we actually building?

The problem we started with

AI agent phones are coming. OpenAI's won't ship until 2028. If you've been following the "agentic AI" space, you've probably noticed the same pattern we did: the interesting hardware is always a year or two out, locked to a specific new device you have to buy, and not something you can actually touch today.

We didn't want to wait for a new phone. So we built a small piece of hardware that plugs into the phone or computer you already own.

What Aiden actually is

Aiden is a physical mobile AI agent device. You connect it via USB to your phone or computer, no app install required. Once connected, it sees the screen, hears your voice, and operates the device for you: navigating apps, sending messages, checking things, completing tasks, the way a person would if they were sitting next to you holding your phone.

The goal isn't to replace your phone. It's to give any phone or computer you already own genuine agentic capability, without waiting for a manufacturer to build it in.

The hardware, honestly

We're currently at dev-board stage. Early Access, no public price or launch date yet. Here's what's actually inside:

MCU: RV1106G3, ARM Cortex-A7 at 1.2GHz
RAM: 256MB
NPU: 1 TOPS
Connectivity: WiFi 6, BLE 5.x
Battery: 1500mAh (~18 hours)
Ports: USB-C, HDMI 1080p capture
Storage: microSD up to 256GB

We're being upfront about the "dev-board stage" part because we'd rather you know exactly what you're looking at than oversell a roadmap. This is real hardware doing real work today, not a rendering of what's coming.

The firmware, and why it's open source

Aiden's firmware is open source, and it's where most of the actual engineering happens. A few things worth knowing if you're the kind of person who reads dev.to:

Language mix: primarily Go, with C++ for hardware-level work and Python for tooling and benchmarks
Package management: OPKG/Entware
Persistent memory: Aiden remembers context across sessions instead of starting fresh every time
Provider flexibility: one-click switching between LLM, STT, and TTS providers, so you're not locked into one model or vendor
Cross-platform: works across iPhone, Android, macOS, Windows, and Linux
Updates: free, and shipped regularly

We publish real PR numbers, real technical writeups, and real dev-board caveats, not polished announcement copy. If you check our blog, you'll find posts about things like the actual architecture behind coordinating real-time voice with background device-control tasks, or the discipline of distinguishing "acknowledged a request" from "verified it actually happened." That's the level of detail we think is actually useful to share.

Where Aiden sits next to everything else

We're not the only people building in this space, and we don't think we should pretend otherwise. A few honest comparisons:

Rabbit R1 is probably the closest hardware form factor to what we're doing.
Humane AI Pin is a cautionary tale worth knowing: discontinued, acquired by HP for $116M in 2025.
Plaud Note overlaps in the efficiency-focused audience but serves a genuinely different function (capture vs. execution).
UiPath / Automation Anywhere represent the enterprise RPA end of the spectrum. Aiden is closer to the accessible, consumer-facing alternative.
Android Accessibility + ADB scripts are the DIY route for technical users who'd rather build this themselves than buy hardware. Respect to anyone doing that.

If you're into embedded systems, agent architecture, voice interfaces, or on-device AI, we'd genuinely like you looking at what we're doing and telling us where it's wrong. That's not a courtesy line. Some of our best fixes have come directly from people testing dev-board hardware and reporting back exactly what broke.

Firmware: github.com/AidenAI-IO/aiden-firmware
Discord: discord.gg/bcJavjcnYz
Site: aidenai.io

We'll be posting real technical writeups here regularly, not just announcements. If you want the unfiltered version of what we're building, this is a good place to start.

Aiden Extends Phone Bridge to Remote Mobile Agent Benchmarks

Nat — Mon, 14 Sep 2026 12:58:49 +0000

PRs #636, #643, #655, and #656 extend Aiden's environment bridge mode so it can proxy Phone Bridge traffic between a bridge Agent and a remote Agent, giving benchmark daemons a direct relay route instead of requiring the benchmark layer to act as the final device-control endpoint.

The architecture

The bridge Agent stays the endpoint your Aiden App connects to. When an environment bridge endpoint is configured, the bridge Agent can proxy eligible traffic to a remote Agent through that route.

Foreground traffic uses bidirectional WebSocket forwarding, active communication flows both directions, and the proxy reports remote connection status. Background traffic takes a different path: commands go through the configured bridge's HTTP queue, where they're queued, polled, and routed toward the remote device environment, with results returning through the same route.

What changed in command handling

Empty command polls now return an empty array, a defined no-command response rather than an ambiguous one. Clipboard writes validate text before queueing, input validation added to the background relay path, though that doesn't confirm the write completed on the target device. Input routing (iOS keyboard actions, enter_text) skips local USB HID isolation and goes through the remote path, but only when an environment bridge endpoint is actually configured, not a general change to local HID behavior.

Benchmark daemons can now relay Phone Bridge commands through the device environment Agent, giving a benchmark a route through the device environment instead of requiring it to be the final control endpoint itself.

The distinction that matters most

Command handling states stay explicitly separate. A queued acknowledgement means the queue accepted the request. A routed command means transport directed it through the configured bridge. Neither proves the remote Agent executed it or the target device completed the action. Confirmed device-side outcome requires actual returned evidence.

Validation covered Go tests plus a manual clipboard_read round trip through benchmark relay mode, confirming one clipboard-read path was exercised and returned correctly, not universal coverage.

The boundary

Merged code with reviewed validation, not confirmation of deployment to every device, benchmark environment, or app installation, and not a claim of complete benchmark coverage or universal iOS/Android compatibility.

PRs: #636, #643, #655, #656

FAQ

What's routed through this path? App-related interactions, URL capabilities, clipboard operations, calendar commands, contact commands, and local-notification capabilities.

Does a routed command mean it executed? No — routing confirms transport, not execution. Confirmed outcome needs explicit returned evidence.

Is this a general USB HID behavior change? No — the remote-path input routing only applies when an environment bridge endpoint is configured.

Aiden Adds a Self-Knowledge Skill for Clearer Agent Boundaries

Nat — Mon, 14 Sep 2026 12:52:40 +0000

PR #624 merges a self-knowledge Skill into Aiden's firmware, giving the Agent a bundled, project-specific reference for describing its own hardware ownership, routing, setup, and recovery boundaries instead of answering those questions from broad assumptions.

The problem

Questions about an agent's own environment need answers grounded in the actual project, not inference. Ask Aiden what it owns, where work routes, which setup path applies, or how recovery should proceed, and the honest answer depends on distinguishing documented context from a guess.

Related but non-interchangeable subjects were the real issue: a request might involve routing, a device setting, or a recovery path, and the correct response depends on what's actually documented for that specific part of the system.

What the Skill actually does

It's bundled reference material covering hardware ownership, runtime routing, Phone Bridge behavior, board configuration, phone setup, observation, verification, and recovery boundaries. Where companion-app context is relevant, it stays limited to public behavior and settings — it doesn't expose internals.

The part worth calling out specifically: the Skill reinforces that an acknowledgement is not proof a task is complete. An acknowledgement shows a request was received or accepted. Verified completion requires actual evidence from the relevant observation or verification path. This is a narrower, more useful standard for agent runtime boundaries — it helps stop status language from getting ahead of what the agent can actually verify.

Recovery guidance is a good example of the same discipline: during frame-service recovery specifically, stale images must not be treated as evidence for deciding what happened or claiming recovery succeeded. That's a rule scoped to frame-service recovery, not a blanket statement about every image-related workflow.

Validation

Merged with skillopt lint reporting zero issues, YAML frontmatter validation, focused Agent skill-loader tests, and git diff checks. That confirms the content is formatted, loadable, and reviewed in the firmware repo — it doesn't guarantee every future answer, setup attempt, or recovery action succeeds.

The boundary

This makes the Agent's description of its own project clearer. It doesn't make the Agent infallible, guarantee every answer, or grant automatic discovery of every configuration. Merged open-source firmware content, not a deployment announcement for every device or environment.

PR: github.com/AidenAI-IO/aiden-firmware/pull/624

FAQ

Does this let Aiden discover its own configuration automatically? No — it's a bundled reference for what's already documented, not a general capability-discovery mechanism.

Does the acknowledgement-vs-completion distinction apply everywhere? The principle is general, but specific guidance (like the frame-service recovery rule) is scoped to the path it was written for.

Is this live on every Aiden device? It's merged into the open-source firmware repo — that's separate from deployment across every device or product configuration.

Aiden Can Now Install AI Agent Skills Directly from a URL

Nat — Mon, 07 Sep 2026 10:00:32 +0000

PR #586 gives Aiden a dedicated way to install AI agent skills from remote URLs. Give it a supported HTTP(S), GitHub tree, or GitHub blob Skill URL, and it routes to skill_manage install — staging, validating, and publishing the skill in one controlled flow.

The problem

Before this, a remote Skill URL could get interpreted as a general content-retrieval request instead of an installation request. The agent might reach for a web scraper, call shell or curl, create files manually, then patch them into place — an indirect path for something with a clear, single intent.

The extra steps also made installation status hard to reason about. Content could get retrieved before the agent had even established whether the expected skill files and structure were supported. A failed step meant digging through multiple tool calls to figure out what had actually been downloaded, written, or changed.

What changed

The reviewed real-agent reproduction for PR #586 tested this with a natural-language prompt containing a GitHub Skill URL. After the routing change, the full trace used exactly one skill_manage install call — no web scraper, no shell, no curl, no manual create-and-patch.

The new install action gives remote skill installation its own path: Aiden downloads SKILL.md and supported companion files into a staging area first. The staged content gets validated before it becomes an active skill. If validation passes, Aiden publishes the completed skill atomically instead of exposing files one at a time mid-installation.

Retrieval prepares a candidate. Validation checks it against supported requirements. Publication happens only after acceptance. The action also returns concise metadata — file writes, changes, integrity result — and when a patch expects content that no longer matches the current file state, you get more useful mismatch details instead of a silent failure.

What you can do with it

If you maintain a skill at a supported HTTP(S) location or a GitHub tree/blob, hand its URL to Aiden in a natural-language request. No more guiding the agent through content fetching and manual file assembly. Useful for testing a skill revision, sharing a skill location with another Aiden environment, or reproducing an extension setup from a known URL.

The boundary

This is merged into the open-source firmware repo — that doesn't mean it's deployed to every Aiden device or product yet. Direct installation is limited to supported URL formats and files that pass validation; Aiden doesn't treat arbitrary websites, repos, or software packages as installable through this path. Validation also doesn't replace your own responsibility to review third-party skill content before using it.

PR: github.com/AidenAI-IO/aiden-firmware/pull/586

FAQ

What URL formats are supported? HTTP(S), GitHub tree, and GitHub blob Skill URLs.

Does this work for any GitHub repo? No — only supported files that pass validation get installed. Arbitrary repos or packages aren't treated as installable through this path.

Is this live on all Aiden devices? It's merged into the open-source firmware repo. That's a different thing from being deployed across every device or product.

How Aiden Agents Survive Running Out of Context Mid-Task: A Technical Deep Dive

Nat — Tue, 25 Aug 2026 14:37:36 +0000

Every long-running agent hits the same wall: the context window can't hold everything the task needs. This is a structural certainty, not an edge case, for any agent running multi-step tasks over a bounded window. Here's how Aiden's firmware handles it, and why, shipped this week across #497, #498, #530.

Two overflow conditions, one bad outcome

Cumulative overflow: enough turns accumulate that history exceeds the limit, even with no single large message. Single-event overflow: one tool call returns more than fits, regardless of prior history.

Two naive responses exist, both unacceptable in production:

Hard failure — task stops, no resume path.
Blind restart — agent repeats work it already did, because it lost the record of what happened. On a physical agent that can tap buttons and submit forms, this isn't cosmetic. A context error says nothing about whether the external device state changed.

What shipped: three coordinated behaviors

Context compression — reduces active material. Explicitly lossy; never sufficient alone to justify continuing.

Session switching — moves to a fresh session. Still requires preserved task info; not a blank-slate restart.

Saved-result-file recovery — the core mechanism. Task state persists outside the active context window, read back deliberately at recovery time.

The recovery read: four pieces of evidence

Before deciding whether to continue, the system reads:

Field	Purpose
Continuation ID	Anchors recovery to the specific execution chain
Saved state	Progress: completed, pending, current phase
Saved errors	Distinguishes benign overflow from a different failure
Output tail	Recent activity clue — not a full execution history

The actual design contribution is the three-way branch: continue, verify-then-retry, or stop-and-surface-to-user. Not binary continue-or-fail. Treating "uncertain" as a first-class outcome, distinct from success and failure, is what makes this a decision rather than a default.

overflow detected
  → compress context / switch session
    → read saved result file
      → continuation ID, state, errors, output tail
        → enough evidence to continue safely?
            yes         → resume
            no/unclear  → stop, surface to user

Why external state, not in-context summarization

Task bookkeeping (what step, what a tool returned, what errored) isn't reasoning content — it doesn't need to survive in the same budget-constrained space the model uses to think. Conflating the two means an overflow event destroys both simultaneously. Externalizing bookkeeping means overflow only costs the reasoning state, not the facts about what happened.

Related prior art: LangGraph's durable execution checkpoints workflow state for resume-from-last-step. OpenAI's conversation-state guidance offers persistent conversation objects to avoid re-deriving context per request. Aiden's approach is closer to the checkpoint model — the artifact is read selectively (four fields), not replayed wholesale.

Ties to the broader reliability loop

Aiden's device-interaction loop follows observe → interpret → act → verify — a conceptual pattern, not a specific architecture claim. Context recovery adds a continuity check: inspect persisted evidence before the next action, and for interface-facing tasks, re-observe the current interface rather than trust a stale checkpoint. Saved record = what the agent believed happened. Fresh observation = what's currently true. Neither substitutes for the other.

Explicit limits

Compression is lossy by construction.
Saved state can be missing, stale, partial, or corrupted — recovery has to handle an untrustworthy artifact, not just the happy path.
Doesn't cover provider outages, network failures, unrelated tool failures, auth issues, or unexpected device states.
No formal exactly-once execution guarantee — this reduces duplicate/lost work, it doesn't eliminate it.

Test matrix

Test area	What to validate
Provider-limit recovery	All four evidence fields read before continuation
Oversized tool outputs	No silent discarding of critical info
Malformed saved files	Fails safely, not silently
Partial completion	Correct behavior mid-step interruption
Duplicate-action risk	Distinguishes completed / partial / unstarted work
Session handoff	Bounded relevant info, not a blind start
Changed device state	Forces fresh observation, doesn't trust stale checkpoint
Human controls	Pause/redirect/confirm function mid-recovery

The malformed-file and changed-device-state tests specifically expose implementations that look correct in a demo and fail under real conditions.

Honest summary

Turns one common failure — running out of context mid-task — into a decision grounded in persisted evidence instead of "restart and hope." No unlimited context, no guarantee every task finishes safely, no replacement for human oversight on consequential actions.

Firmware: github.com/AidenAI-IO/aiden-firmware
Discord: discord.com/invite/bcJavjcnYz

Browser Agents Can Use the Web. Aiden Is Built to Use the Device.

Nat — Tue, 18 Aug 2026 18:51:36 +0000

Perplexity Comet is capable inside a browser session. Aiden is being built as a device automation agent for work that moves across visible phone and computer interfaces — beyond what any browser session can reach.

Comet deserves the comparison

Perplexity positions Comet as an AI-first browser and personal assistant, available across Mac, Windows, iPhone, iPad, and Android, with a usable free tier. For research, page-level questions, tab-heavy synthesis, and supported web actions, that browser-native model is genuinely strong.

The distinction isn't that browser agents are less useful — it's that every agent works through a control surface. Comet's surface is the browser session: tabs, webpages, browser-accessible documents, web apps. Aiden's surface is the connected device itself.

This is a real, expanding category, not a single-product comparison:

Fellou — explicit "self-driving browser" positioning for multi-step web tasks
Dia (The Browser Company) — strong cross-tab context, though more suggestive than autonomous
Opera Neon — experimental, agentic, broader creation-workflow emphasis

Each reflects the same direction: AI moving from answering questions to participating in interface work. The practical difference is where that participation can continue.

Where the browser boundary actually shows up

Workflow moment	Browser-session strength	Where the boundary appears
Researching sources	Web search, reading, tab comparison	Task stays browser-contained
Filling a web form	Browser-rendered fields, on-page actions	External identity checks or system dialogs
Saving findings	Web notes, browser-accessible docs	Native notes/desktop app needs another path
Working with local files	Downloads, web uploads	File pickers, desktop software, permission prompts
Continuing on a phone	Mobile browser context	Native apps, device settings, cross-app handoffs

flowchart LR
    A[User instruction] --> B{Current control surface}
    B --> C[Browser session]
    C --> D["Tabs, webpages, web apps"]
    D --> E[Browser-supported action]
    E --> F[Review browser outcome]
    B --> G[Connected device interface]
    G --> H["Browser, native app, desktop app, or system dialog"]
    H --> I[Observe visible state]
    I --> J[Propose next action]
    J --> K[User confirms, redirects, or stops]
    K --> L[Execute supported interaction]
    L --> M[Review outcome]

How Aiden actually works

Aiden's firmware repo documents a development-board implementation: HDMI-based screen capture, USB HID input. It observes a connected screen and sends keyboard, pointer, or touch-style input — no app-specific integration required on the target device.

That's evidence of a documented dev-board implementation. It's not a claim of universal compatibility, task-success guarantees, or consumer availability. Aiden's being developed for Android and iPhone workflows specifically, and the team is upfront that real-device interaction has real constraints — setup conditions, OS versions, accessibility settings, interface variability.

Choosing between the two isn't a category war

Need	Browser-first (Comet)	Device-level (Aiden)
Research across webpages	Strong fit	Relevant only if research continues onto a device
Native mobile app handoff	Depends on integrations	Core workflow category Aiden targets
Desktop application interaction	Depends on external access paths	Relevant when visible desktop interfaces matter
Consequential decisions	User review required	User confirmation/interruption stays central

Aiden isn't presented as a Comet replacement. Use a browser agent for web research and browser-native tasks; use the device-level approach when a real interface handoff becomes necessary.

Repo: github.com/AidenAI-IO/aiden-firmware
Discord: discord.com/invite/bcJavjcnYz — engineers active, technical questions welcome.

Does Aiden Store Your Data? A Straight Answer on Privacy and Self-Hosting

Nat — Wed, 12 Aug 2026 14:08:52 +0000

Short answer: our current dev-board architecture has no Aiden-hosted backend. Longer answer: that doesn't mean no data is ever stored anywhere, and we'd rather explain the actual data path than let you assume either way.

The architecture

Aiden runs on a bring-your-own-provider model. The board captures a connected device's display via HDMI capture, sends screenshots to a multimodal model endpoint you configure, and controls the target device through USB HID (keyboard, pointer, touch). Voice works the same way — audio goes to the STT, LLM, and TTS endpoints you've selected.

What that means concretely:

No Aiden-hosted backend in the current architecture.
Screenshots, audio, and text go to your configured endpoints — not to us.
You can point it at a cloud provider, a self-hosted model, or something fully local.
Firmware is open source — inspect it yourself.

This is a deliberate split from the usual pattern where the product company, the storage layer, and the model provider are the same entity. We separated those roles on purpose.

Where it gets more nuanced

Aiden also supports persistent context, preferences, and skill optimization. So "no backend" doesn't mean "nothing is ever stored" — a deployment can retain local context. We haven't yet published the exact schema, retention period, or deletion process for that, and we're not going to pretend we have.

Same caveat applies to screenshots specifically: they may leave the board if you've configured a remote endpoint, and we haven't published a complete account of how caching/buffering works or what each provider retains on their end.

What we haven't documented yet (full list, no spin)

Topic	Status
Data retention periods	Not published yet
Local memory deletion workflow	Not published yet
Data export	Not published yet
Default logging/diagnostics	Not published yet
Telemetry/analytics use	No public statement yet
Storage jurisdictions	Not published yet
Encryption/key management	Not published yet

We're listing this because the absence of a published policy shouldn't be read either way — not as "they're hiding something" and not as "everything's fine." We just haven't written it up yet.

The part that actually matters for builders

Your configuration is doing as much work as the hardware. A cloud endpoint, a self-hosted model, and a fully local model produce three different data-flow outcomes from the exact same physical device. Before you point Aiden at sensitive workflows, check what your chosen model/STT/TTS provider actually does with inputs — that's now part of your privacy surface, not an implementation detail.

Repo: github.com/AidenAI-IO/aiden-firmware
Discord: discord.com/invite/bcJavjcnYz — our engineers answer technical questions directly.

Building Aiden: a physical AI agent device that plugs into any phone/computer over USB and operates it like a human would. Go + C++ + Python stack, no API needed. https://github.com/AidenAI-IO/aiden-firmware, AMA on the architecture if curious.

Nat — Thu, 06 Aug 2026 07:23:16 +0000

GitHub - AidenAI-IO/aiden-firmware: AI Agent hardware for mobile phone · GitHub

AI Agent hardware for mobile phone. Contribute to AidenAI-IO/aiden-firmware development by creating an account on GitHub.

github.com

AI Agents and Apps With No API: The Real Engineering Tradeoffs

Nat — Tue, 04 Aug 2026 09:20:48 +0000

"Just automate it through the UI" sounds simple until you're the one maintaining that integration. When an API is missing, incomplete, or unsuitable for a task, an agent can often operate through the visible interface instead, the same DOM, accessibility tree, or screen pixels a person would use. That's a real capability. It's also a different engineering problem than an API integration, not an easier one.

The interface as integration layer

An API is built for software-to-software communication: defined actions, structured data, predictable error responses. Strip that away and an agent has to work with whatever's actually on screen:

A web page's DOM and browser state
An accessibility tree exposing control names, roles, and values
A desktop app's window and UI controls
A mobile UI hierarchy
Screen pixels interpreted via OCR or computer vision
Standard keyboard, pointer, touch, and gesture input

The core loop changes shape: observe state, pick one bounded action, perform it, verify the result, then pause or recover if the evidence is insufficient. "Can click" isn't the bar. Whether the system can recognize uncertainty and stop safely is the actual bar.

Six methods, ranked by how much structure you get

Method	What it uses	Best fit	Main limitation
Browser automation	DOM, browser protocol, page state	Stable browser tools and web forms	Selectors and page states can change
Accessibility interaction	Roles, labels, values, control hierarchy	Accessible web/desktop/mobile UIs	Metadata may be missing or inaccurate
Screen and OCR interaction	Pixels, screenshots, visible text	Legacy apps, remote desktops	Visual interpretation is less deterministic
Keyboard/pointer/touch input	Standard user input	Cross-app and real-device tasks	Needs pairing with reliable observation
RPA	Rules, selectors, OCR, files	Narrow, repeatable legacy workflows	Exception handling grows over time
Hybrid API + UI	Approved APIs for some steps, UI for gaps	Partially integrated workflows	Requires careful state reconciliation

The pattern worth internalizing: pick the most structured permitted method for the specific task, not the one with the broadest reach. Broad reach (screen + OCR) is also the least deterministic option, reach for it because you need to, not because it's the default.

Browser automation (W3C WebDriver, Playwright) is usually the strongest option for stable web apps, locate by semantic role and label, wait for load state, verify the confirmation state actually appeared. Still breaks on dynamic rendering, nested frames, A/B tests, and expiring sessions, explicit post-transition checks matter more than assuming a click worked.

Accessibility-tree interaction (WAI-ARIA, Accessible Name and Description Computation, plus platform frameworks like Microsoft UI Automation, Android UI Automator, Apple XCTest) gives you a semantic layer instead of raw pixels, when the app actually implements accessibility correctly. When it doesn't, you're back to guessing.

Screen and OCR interaction is the fallback for legacy apps and remote desktops with no exposed structure at all, and it's exactly as fragile as it sounds. Least deterministic, most universally applicable, use it when nothing else works, not as a default.

Authorization comes before method selection, not after

This is the part that's easy to skip past: no-API access doesn't override terms of service, platform rules, or the account owner's actual permissions. Before picking a method, confirm the task is something the agent (and the person deploying it) is actually authorized to do. A capability existing isn't the same as it being sanctioned.

Two hard lines, regardless of which method you're using:

CAPTCHAs and MFA prompts are boundaries, not obstacles. Pause and hand control back to a person, or use a vendor-approved path. Never attempt to defeat them.
Consequential actions need a human checkpoint. Sending a message, submitting a form, changing account settings, moving money, anything hard to reverse, should require confirmation before it executes, no matter how confident the agent's reasoning looks in the logs.

What actually needs verification

Real production systems working this way need instrumentation that API-only systems mostly don't:

Screen-state capture at each step
Action traces (what was observed, what was decided, what was sent)
Replayable sessions for debugging failures after the fact
Explicit post-action verification, not an assumption that the action succeeded
Failure classification (expired session vs. layout change vs. genuinely blocked)

If you're building or evaluating a system like this, the useful question isn't "can it click the button." It's "when the button isn't where it expected, what happens next, and can a human see exactly what the agent tried."

We build Aiden around this exact problem for physical devices, HDMI-based screen capture plus USB HID input, no API dependency, verification and human confirmation built into the action loop rather than bolted on after. Repo: github.com/AidenAI-IO/aiden-firmware.

Curious what verification patterns others here have found actually catch failures early, screenshot diffing, structured re-query of the accessibility tree, something else?

Command+V Silently Fails on iOS When a Keyboard and Mouse Are Both Present (Real Bug, Not Ours)

Nat — Thu, 30 Jul 2026 14:38:46 +0000

We build Aiden, a physical mobile AI agent that connects to a phone over USB and presents itself as a keyboard and mouse to reproduce real-world iPhone interactions. During real-device testing, we hit a bug that took controlled experiments to actually isolate, and it turned out to live inside iOS itself, not our stack.

The symptom

Ordinary letters worked fine. Modifier-key combinations intermittently didn't.

Input	Expected result	Affected result
`x`	Inserts `x`	Inserts `x`
`Shift+X`	Inserts `X`	May insert lowercase `x`
`Command+A`	Selects all	No selection
`Command+V`	Pastes into the active field	No paste action

Our write to the Linux USB Gadget device at /dev/hidg0 completed successfully every time. Nothing happened on the phone's screen.

Two iOS settings matter here: AssistiveTouch has to be enabled for an external mouse to work on iPhone at all, and Show Onscreen Keyboard (inside AssistiveTouch) keeps the software keyboard available while a physical keyboard is connected.

First finding: it's not the HID layer

System logs distinguish plain keyboard events from modifier-based key commands, and log the KeyboardFocus target each command resolves to.

Working case:

cmd-v -> <keyboardFocus; pid: 9085; token: MobileNotes>

Broken case, identical keypress:

cmd-v -> <keyboardFocus; pid: 3738; token: com.apple.springboard>

iOS recognized Command+V correctly both times. In the broken case, it routed the resulting command to SpringBoard (the Home Screen process, hosting a lot of system-level interactions) instead of the actual foreground app. The keystroke arrived fine, the routing afterward is where it breaks. We call this "iOS keyboard focus loss" internally, not an Apple term, just our shorthand: the keyboard stays connected, plain keys still land correctly, but modifier commands stop reaching the foreground app.

Ruling things out

Checked and did not fix it:

Full Keyboard Access toggle (Settings > Accessibility > Keyboards & Typing), reproduced with it both on and off
Boot Keyboard protocol=1/subclass=1 declaration
LED output behavior
Apple keyboard descriptor and handshake sequence
Keystroke timing
Stale HID file descriptors after USB re-enumeration

Some of these fixed real, separate edge cases. None touched the core failure.

We also suspected toggling AssistiveTouch off/on after connecting broke the keyboard session, since that sequence reliably triggered the bug. A later round of tests removed AssistiveTouch toggling entirely and just added a mouse to an already-stable keyboard-only setup. Bug came back anyway. Demoted from root cause to one of several triggers.

The A/B test that actually isolated it

With AssistiveTouch left on throughout:

Test setup	Device topology	Result
Aiden emulated composite device	Keyboard + pointer/mouse	Intermittently routed to SpringBoard when switching between apps
Physical gaming keyboard	Keyboard + firmware-declared virtual mouse	Command/Shift/Option commands routed to SpringBoard
Aiden keyboard-only	Keyboard-only, pointer: 0	32 Command+V attempts, 0 SpringBoard-only events
Same keyboard + physical USB mouse	Keyboard-only + separate mouse	11 attempts, 4 SpringBoard-only events
Keychron K2 Max, Bluetooth mode	BLE keyboard + mouse collection	32 attempts, 26 SpringBoard-only events

The critical pair is rows 3 and 4: identical keyboard implementation, identical iOS settings, identical test procedure. Plugging in an ordinary USB mouse brought the failure back. Unplugging it, all 22 follow-up runs succeeded.

Conclusion: with AssistiveTouch enabled, if iOS detects pointer or mouse capability on the connected device at the same time as a keyboard, it can misroute modifier-based commands to SpringBoard instead of the foreground app.

This isn't specific to our HID implementation, it reproduces on a real gaming keyboard declaring a virtual mouse, and on a Bluetooth keyboard with a mouse collection in its report descriptor. That rules out an Aiden-specific compatibility explanation. The logs place the fault boundary inside iOS: the system receives the modifier key, generates the correct key command, then routes KeyboardFocus to SpringBoard anyway. Without iOS source access, we can't say why pointer/mouse capability affects that resolution, only that it reliably does.

We also tried substituting an HID touchscreen/digitizer for the mouse, keeping pointer: 0. Six Command+V attempts, all landed correctly. Further support for the pointer/mouse link, but any keyboard interface present still makes iOS hide the software keyboard, and a one-time Eject only restores it briefly, so this isn't viable for our actual product experience.

Our workaround

Aiden only needs modifier keys when executing a shortcut, and the trigger is tied specifically to pointer/mouse capability being present. So:

Two separate USB HID profiles: normal operation keeps keyboard + pointer; a modifier-key action first re-enumerates to keyboard-only
Different USB Product IDs and serial numbers between the two profiles, so iOS doesn't reuse stale state from the mouse-equipped profile
Once the shortcut completes, pointer capability is restored
Switching scope covers a whole agent action, not a single keypress, to avoid the software keyboard popping and retracting repeatedly mid-action
The normal profile is restored on action completion, error, cancellation, or panic cleanup

To be direct: this is a workaround, not a fix. For the brief window a modifier key needs to take effect, we simply don't let iOS see pointer/mouse capability at all.

Has anyone else hit this?

It reproduces on hardware that has nothing to do with us, so if you're driving iOS with an external keyboard plus pointer and have seen a shortcut silently fail, this might be why.

Repo: github.com/AidenAI-IO/aiden-firmware

AI Agent vs Automation App: Why 'It Taps Like You Do' Matters

Nat — Tue, 28 Jul 2026 14:09:19 +0000

"Just use an AI agent instead of automation" and "just use automation instead of an agent" are both wrong the same amount of the time. The actual question isn't which category is better, it's which control model your workflow needs.

It's a control-model difference, not a generation gap

A workflow automation app follows a predefined pattern: trigger happens, run a fixed set of actions. Predictable, testable, brittle when the input or integration changes shape.

An AI agent interprets a goal, chooses steps, uses tools, and adapts to context within boundaries (this is close to how IBM frames it too). More flexible, harder to evaluate with a simple pass/fail test.

	Automation app	AI agent
Follows	Predefined rules and triggers	Goals, context, tools, policies
Best fit	Stable, repeatable workflows	Fragmented or ambiguous tasks
Main strength	Predictability	Adaptability
Main risk	Brittleness when inputs change	Misreading context, wrong next step
Human role	Configure and monitor	Observe, interrupt, redirect, confirm

Simple rule: every step known in advance, reach for automation. Path depends on screen state or ambiguous intent, an agent is worth the added complexity.

Where automation apps actually break down

Automation apps are the right call more often than agent enthusiasm suggests. But real workflows expose gaps:

No API exists for the action you need
Connectors expose only part of an app's functionality
OAuth scopes, admin approvals, or platform rules block execution
UI scripts break when layouts or labels change
Branching logic gets ugly once the task needs judgment, not rules

That's not a knock on automation apps, they're optimized for known paths. They're excellent at "when X happens, do Y." They're bad at "look at what's on screen, decide what matters, and ask before submitting."

flowchart TD
    A[User task] --> B{Is every step known in advance?}
    B -- Yes --> C[Use a workflow automation app]
    B -- No --> D{Does the task depend on screen state or context?}
    D -- Yes --> E[Consider AI agent automation]
    D -- No --> F[Use a simple script or manual process]
    E --> G{Could the action have real-world consequences?}
    G -- Yes --> H[Pause for human confirmation]
    G -- No --> I[Proceed within defined boundaries]

"It taps like you do" is an observability claim, not a trust claim

UI interaction automation means the system operates the visible interface: clicking, typing, navigating, reading the screen. Anthropic's computer use does this for Claude; it's explicitly labeled beta with acknowledged risks, worth reading directly rather than assuming maturity.

The phrase gets misread constantly. What it actually buys you:

The agent acts through the visible UI, same as a human would
You can watch the sequence of actions as it happens
You can review what was visible at each step
The system can pause before something consequential
You can interrupt or redirect mid-task

What it does not mean: that the agent understands consequences the way a person would. Visible tapping improves inspectability. It does not improve correctness. Those are different properties and conflating them is where overconfidence creeps in.

This is genuinely different from hidden API automation, which can be faster when good APIs exist, but gives you less visible context about what actually happened during a run.

The control points that matter for any UI-level agent

Control point	Why it matters
Visible execution	You see what's happening instead of trusting a black box
Interruptibility	Stop the task when the path looks wrong
Redirection	Correct the goal without restarting from zero
Confirmation gates	Consequential actions pause for approval
Action traces	Developers can inspect what happened during a run
Boundaries	Defined tools, allowed actions, stop conditions
Recovery paths	Stop, retry, escalate, or ask, don't just fail silently

Consequential actions specifically deserve a hard confirmation gate: sending external messages, submitting forms, deleting or overwriting data, changing account settings, initiating payments. No amount of model capability substitutes for a pause here.

Evaluation changes too. Standard automation gets tested with logs and integration checks. UI-level agents need: screen-state capture, step-by-step action traces, replayable sessions, failure classification, before/after visual review, and confirmation-behavior tests specifically.

flowchart TD
    A[Agent observes current screen] --> B[Agent proposes next action]
    B --> C{Low-risk navigation?}
    C -- Yes --> D[Proceed visibly]
    C -- No --> E{Could this affect another person, account, data, or money?}
    E -- Yes --> F[Pause for human confirmation]
    E -- No --> G[Proceed within policy]
    D --> H{User interrupts?}
    H -- Yes --> I[Stop or redirect]
    H -- No --> J[Continue to next screen state]
    F --> K{User approves?}
    K -- Yes --> J
    K -- No --> L[Cancel or revise plan]

Where Aiden sits in this

Aiden is a physical mobile AI agent device built around real smartphone and computer interfaces, not a chatbot, not a conventional automation connector. The current development-board architecture uses HDMI-based screen capture and USB HID input, with a Go-based agent runtime that sends screenshots to a configured multimodal model and writes resulting input commands to device nodes. In plain terms: it sees the screen and acts through input, rather than depending on app-specific automation APIs existing at all.

That matters specifically for the "no API" failure mode above. If the app you need to operate has no exposed API and no clean accessibility hooks, a UI-level agent doesn't care, it drives the interface the same way a person would.

Real setup caveat, stated plainly rather than glossed over: iOS control currently requires AssistiveTouch to be enabled on the target device. That's a genuine setup step, not a one-tap connection. Android and iPhone workflows are both in active development, but the path isn't identical on both yet.

For anyone evaluating this kind of system, the useful questions aren't "can it tap":

What screen state did it act on?
What model made the next-action decision?
What input event got sent, and what happened after?
Where did it pause, and could a human actually stop it there?
Can the run be reproduced or debugged after the fact?

That's the actual engineering case for "it taps like you do", it turns agent behavior into something inspectable instead of a background process you have to trust blind.

The practical takeaway

Reach for automation when the task is stable, repetitive, trigger-based, and API-accessible. Reach for an agent when the task is fragmented across apps, dependent on visual screen state, or too ambiguous to express as fixed rules.

Agents don't make automation obsolete, they extend what's attemptable into the territory where workflows cross from structured data into messy real interfaces. That extension is exactly why the guardrails above matter more, not less.

Repo (HID config, capture pipeline, agent runtime): github.com/AidenAI-IO/aiden-firmware. Community and engineering discussion: Aiden Discord.

Curious how others here draw the line between "add a confirmation gate" and "just don't automate this step at all", where's that boundary sat for you in practice?

What Can an AI Agent Actually Do on Your Phone? 12 Tasks, and the Execution Problem Behind Them

Nat — Tue, 28 Jul 2026 08:40:25 +0000

Most "AI agent on your phone" demos show the happy path: one clean voice command, one flawless run, cut before anything breaks. The interesting engineering is in everything the demo skips: permissions, app support, screen state that lies, and the confirmation boundary before an irreversible action.

This post walks through the 12 tasks phone agents are actually asked to do, but from an execution angle rather than a feature-list one: what each task needs to touch, where it gets hard, and why the interface an agent uses to act changes what's even possible. At the end I'll get into the screen-capture + USB HID approach we're building with Aiden, because it makes a different set of trade-offs than the app-only assistants most of these tasks assume.

The real question isn't "can AI control my phone"

It's "which tasks are safe, useful, and realistic today — and what does the agent have to touch to do them?"

Three execution models are competing right now, and they fail differently:

App/API assistants — call documented APIs. Reliable where an API exists, blind where it doesn't.
OS-level integrations (Apple Intelligence, Copilot+) — deep access, but locked to one vendor's hardware and app set.
Screen-aware agents — interpret the UI visually and drive it. Work anywhere there's a screen, but inherit every ambiguity of reading pixels.

Keep those three in mind; each of the 12 tasks below lands differently depending on which one you're using.

The 12 tasks, ranked by how hard they actually are

Feasibility varies a lot. Reminders and research summaries are reliable today. Calendar, email, messages, meetings, travel, shopping, expenses and smart-home sit in a solid middle tier. Form filling and health support are the least mature — either because they touch fiddly interfaces or because they edge toward decisions that should stay with a human.

#	Task	Hard part (the bit demos skip)
1	Reminders / task capture	Almost none. Parse natural language → structured task. The reliable baseline.
2	Calendar scheduling	Timezone math, conflict detection, write-access confirmation before it sends invites.
3	Email triage + draft	Drafting is easy; sending to the wrong recipient is the failure that kills trust.
4	Message summarization	Group-chat context windows and knowing which thread "that" refers to.
5	Research summarization	Source quality and not fabricating a citation. Reliable when read-only.
6	Meeting notes + follow-ups	Audio capture permissions and turning talk into action items without hallucinating owners.
7	Travel planning	Multi-app, multi-step state that goes stale mid-run (prices, availability).
8	Form filling / navigation	The API-less wall. No endpoint, no accessibility tree — this is where execution model matters most.
9	Shopping / price comparison	Cross-site state, and the confirmation boundary right before "buy".
10	Receipt / expense organization	OCR quality and structured extraction from messy inputs.
11	Smart-home automation	Device API fragmentation across ecosystems.
12	Health / fitness habits	Non-clinical only. The hard part is refusing to give medical advice.

Two of these — 8 (form filling / navigation) and 9 (shopping) — are where the execution model stops being an implementation detail and becomes the whole game. Let's zoom in.

Task 8 is where most agents hit a wall

Form filling and in-app navigation is the task that separates the three models, because a huge number of real apps expose no API and no clean accessibility tree. An app/API assistant simply can't act there. An OS-level integration can, but only inside its vendor's boundaries.

This is the case for a screen-level approach: if the agent can see the rendered screen and send input like a human does, it doesn't need the app to cooperate. No SDK, no integration, no waiting for a vendor to ship an API.

The catch — and it's a real one — is observability. If you only send input blindly, you can't tell whether the tap landed, whether a spinner resolved to success or an error, or whether the screen even changed. Reading pixels to infer state is where screen agents get brittle.

How a screen-capture + USB HID approach handles it

This is the part I actually work on, so here's the concrete version rather than the marketing one.

Aiden is a physical mobile AI agent device: a small board that plugs into a phone or computer over USB. It captures the target's display over HDMI and sends actions back as a standard USB HID keyboard/mouse. To the phone, it looks like an external monitor plus a keyboard — no app installed on the target device, no root, no ADB.

That combination is a direct answer to the Task 8 wall and the observability problem underneath it:

Acting without an API — HID input drives any app that accepts keyboard/pointer/touch, which is all of them. The app doesn't need to expose anything.
Closing the observability gap — because Aiden also reads the screen over HDMI capture, the same loop that acts can see the result, instead of firing input blindly.
Privacy posture — there's no Aiden backend; screenshots and audio go only to whatever model endpoint you configure, and the whole thing is self-hostable.

On the current development board, the agent runtime runs on-device (Go), with frame capture, a voice loop with VAD, the decision loop, and HID output as independent goroutines. It's a dev board, not a finished consumer product — the repo is the open-source firmware and agent runtime, not a shipping SKU:

github.com/AidenAI-IO/aiden-hardware-demo

The honest caveats stand: reading pixels to verify an action still breaks when loading states lie, and any screen-reading agent inherits the prompt-injection surface of whatever it looks at. Those are unsolved for everyone in this space, not just us — worth saying plainly on a dev audience.

The pattern across all 12 tasks

Whichever model you build on, the tasks that work today share a shape: bounded workflow, low blast radius, human confirmation before anything irreversible. Reminders and summaries are safe because the worst case is a bad note. Email, payments, account changes and health decisions are exactly where you want an explicit confirmation gate, no matter how good the model is.

The useful design question isn't "can the agent do more?" It's "can the user reliably stop or redirect it mid-run, and does the agent verify before it commits?"

FAQ

Which of these tasks are safe to automate without confirmation?
Reminders, research summaries, and read-only organization. Anything that sends, buys, changes an account, or touches health decisions should require an explicit user confirmation step.

Why can't a normal app-based assistant do form filling in every app?
Because many apps expose no API and no accessibility tree. Without one, an API-based assistant has nothing to call. Screen-level approaches get around this by reading the rendered UI and sending human-style input instead.

What's the catch with screen-reading agents?
Observability and injection. Inferring success from a re-read of the screen breaks when loading states lie, and an agent that reads screens will read whatever an attacker puts on them. Both are open problems.

Does Aiden run on the phone?
No — it's an external board that plugs in over USB, captures the screen over HDMI, and sends input over USB HID. Nothing is installed on the target phone. It's currently a development board, and the firmware/agent runtime is open source.

Can I self-host the model?
Yes. There's no vendor backend in the loop; model, STT and TTS endpoints are whatever you configure, so you can keep everything on your own infrastructure.

If the execution side of this is interesting, the firmware, HID gadget config and capture pipeline are all in the open-source repo: github.com/AidenAI-IO/aiden-firmware. More on the project at aidenai.io. Curious how others here are handling the observability problem on screen-driven agents — verifying an action actually landed is the part I keep coming back to.