DEV Community

蒋亦韬
蒋亦韬

Posted on Fully Autonomous

An iPhone agent in Safari, with an ESP32 as its Bluetooth hands

#opensource #showdev #iot

Our open-source project (MIT): cccat6/iphone-agent-harness

Demo and browser installer · Setup guide

We built iPhone Agent Harness to explore a simple interaction: describe a phone task in Safari, let a vision model inspect the screen, and use a nearby ESP32 to supply the taps and keyboard input. Our current demo shows the agent setting an alarm in the iPhone Clock app.

The phone stays wireless. The ESP32 is separately powered and sits within Bluetooth range, while both devices use the same normal Wi-Fi network.

How the pieces fit together

The ESP32 is an input bridge. Safari runs the conversation, screen capture and model interaction.

iPhone screen
    -> Safari screen capture
    -> selected context sent to your chosen vision model
    -> action plan validated by the Harness
    -> local WebRTC commands to the ESP32
    -> Bluetooth mouse / keyboard input to the iPhone
    -> next screen observation
Enter fullscreen mode Exit fullscreen mode

AssistiveTouch lets the phone respond to pointer input. The board does not run the model or decode the video stream. Keeping those responsibilities on the phone lets the firmware concentrate on delivering input.

That division also makes setup different from a native app workflow: the agent interface is a website, and firmware installation is available through desktop Chrome or Edge. There is no native iOS agent app to build and sign for this route.

From a demo to repeatable tasks

A useful phone agent needs more than a single prompt. The Harness includes conversation management and skills that can be customized, shared and updated. Skills describe how to handle recurring tasks and improve the agent's taps, typing and gestures. MCP tools can also be configured in Settings.

You choose the model service. Conversations, screenshots and credentials are stored in the current browser. Selected messages and images go to your chosen AI provider, and configured MCP requests go to their respective servers. Browser storage does not mean that model inference happens locally on the phone.

Trying it

  1. Check the README for supported chips and flash requirements. Firmware is provided for C2, C3, C5, C6, S3 and classic ESP32.
  2. Flash the board with the browser installer.
  3. Join your Wi-Fi, pair the bridge over Bluetooth and enable AssistiveTouch.
  4. Open the Harness in Safari, configure your model access, and follow the screen-sharing instructions.
  5. Start with a short task you can observe, then inspect the result.

The current requirement is iOS / iPadOS 27 or later. The phone and bridge need to share a Wi-Fi network. The setup guide also covers the optional clipboard Shortcut and installing skills.

Scope and tradeoffs

The alarm recording demonstrates one real workflow. It is not a benchmark of arbitrary apps or long-running tasks. A model can misread the screen or choose an incorrect action, so task success needs to be checked on the phone. Compatibility and connection feedback from additional devices are especially useful.

This project was inspired by onehumanbeing/esp32-iphone-agent, which we credit in the README. We are exploring a Safari + Bluetooth HID approach; that project uses a native iOS app with an AirPlay/USB HID path.

If you build with ESP32, WebRTC or phone automation, we would welcome feedback on pairing, Wi-Fi/BLE coexistence and the tasks you would like to automate.

Source, firmware and issue tracker: https://github.com/cccat6/iphone-agent-harness

Top comments (0)