Most phone automation needs a computer on a cable, a developer account, or a jailbreak. ESP32 iPhone Agent takes a different route: a small ESP32-S3 board plugs into the iPhone and acts as the agent's eyes and hands, while an iPhone app runs the vision model loop. At run time it is just the phone and the board.
This guide walks through the setup from the project's README, start to finish, so you can type a goal like "Open Settings and go to Wi-Fi" and watch a vision model do it on a real iPhone.
How it works in one paragraph
The board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as ESP32 Agent. The phone mirrors its screen to the board over AirPlay, and the board forwards that stream to the app over Bluetooth LE. The app sends screenshots to any OpenAI-compatible vision model, which plans the next action. The board is also a standard USB mouse and keyboard, so with AssistiveTouch on, a click is a tap and a drag is a swipe. Loop until done.
What you need
- An ESP32-S3 board with native USB and PSRAM. The README tested a Seeed Studio XIAO ESP32S3 (plain version, 8 MB PSRAM, not Sense, not Plus), listed at about $7.
- An iPhone on iOS 18 or later with cellular data. USB-C iPhones plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board.
- A vision model behind any OpenAI-compatible chat API that accepts images: a cloud API or your own server (the README names vLLM, SGLang, llama.cpp and Ollama).
- A Mac with Xcode, once. Only to flash the board and build the app. It is not needed at run time.
Step 1: Flash the board
Plug the board into your Mac. Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:
arduino-cli lib install "NimBLE-Arduino@2.5.1"
hardware/flash.sh # optional: [serial port] [--debug]
Always build with flash.sh. It applies the BLE buffer sizes the firmware needs.
Step 2: Build and install the iPhone app
cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID
open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone
A free Apple ID can sign it, but you have to re-sign every 7 days. The one-tap hotspot join needs the Hotspot Configuration capability (a paid account). Without it, the app shows the hotspot password so you can join by hand.
Step 3: Run the four setup screens
Plug the board into the iPhone and open ESP32 Agent. Setup checks each step itself:
- Plug Me In: pair the board over Bluetooth.
- Let Me See the Screen: join the board's Wi-Fi, then Control Center, Screen Mirroring, ESP32 Agent.
- Let Me Use the Pointer: turn on AssistiveTouch. The app calibrates the pointer on its own.
- Give Me a Brain: enter your model endpoint and key (stored in the iOS Keychain), then pick a vision model from the endpoint's list or type a model ID. You can set separate planner and actor models.
Step 4: Give it a task
Type a goal such as "Open Settings and go to Wi-Fi" and press Start. Stop it any time from the Dynamic Island. Once you have tasks you repeat, Siri, Shortcuts, widgets and Control Center can run saved tasks.
Step 5: Review what it did
Every run is kept on the phone. Open Runs, pick a run and replay it step by step with the taps and swipes drawn on top. To debug with another agent, use Runs, open a run, ⋯, Export trace to get the screenshots, actions and annotations.
Tuning it for your workflow
The planner and actor prompts are plain text files in prompts/ and are bundled with the app. Edit them, rebuild the IPhoneAgent scheme, and compare the new run against the old one in Runs. If you suspect the board rather than the app, hardware/tools/board_test.py plays the app's role over BLE from a Mac (see hardware/tools/README.md and docs/ble-protocol.md).
Limits to know before you start
- Bluetooth LE carries about 300 to 500 kbps, so the mirror is small (480x1040, at most 5 fps) and each step takes a few seconds, mostly model time.
- iOS does not mirror some screens (passcode entry, Apple Pay, DRM video), so the agent cannot see them. Mirroring stops when the phone locks.
- While it runs, the phone's Wi-Fi is the board's hotspot, so the model is reached over cellular data.
- Keyboard input is ASCII only, and landscape apps are not supported yet.
- The README warns: only follow the install steps in the README, and never run "quick install" one-liners posted in issues or comments. They are not from the maintainers.
Only automate devices and accounts you own or are allowed to use. The agent acts with real hands on the phone, so watch it and keep the stop button within reach.
More ways to use it
We wrote up five concrete use cases for this repo, with steps and copy-paste agent prompts:
- Have an AI model complete a task on a real iPhone
- Replay and export phone agent runs for analysis
- Start saved phone tasks from Siri or Shortcuts
- Edit the planner and actor prompts for a custom workflow
- Test the ESP32 board over BLE from a Mac
Project credit: ESP32 iPhone Agent by Henry Yuan (https://github.com/onehumanbeing/esp32-iphone-agent), MIT licensed, with the AirPlay part under GPL-3.0.
Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com
Top comments (0)