An ESP32 and a Local Vision Model Drive My iPhone

Apple spent 17 years convincing us that controlling an iPhone requires Xcode, a $99 developer account, and a that costs more than your couch.

Henry Yuan spent $7 to prove them wrong.

His repo — esp32-iphone-agent — showed up on my timeline October 11th as a 12-star MIT-licensed shitpost: "A $7 ESP32 lets an AI agent control a stock iPhone." Hackaday picked it up Oct 14th. By the time I cloned it, it had 2,975 stars. You are still early.

The trick isn't AI. It's hardware heresy.

Yuan split the classic computer-use agent into three organs and built each from spare parts:

• Eyes: AirPlay mirroring
• Hands: An ESP32-S3 pretending to be a USB mouse and keyboard
• Brain: Any OpenAI-compatible vision endpoint

No WebDriverAgent. No CoreDevice hack. No app on the phone that Apple has to sign. Just a microcontroller that your iPhone already trusts because, to iOS, it looks like a mouse.

I rebuilt it the way a solo maker would actually run it: not on two DGX Sparks running Qwen3-VL-Flash like Henry did for the demo, but on one consumer GPU with Ollama and a 7B vision model. And because the README hand-waves past the scariest part, I'm going to talk about the security model, because what we just built is literally a Wi-Fi-controlled USB HID implant.

This is the teardown.

1. The Eyes: UxPlay plus a GStreamer JPEG pipeline

The original v0.1.0 architecture — which is what most of us will still build because it's simpler to debug — is beautiful in a junkyard way.

iPhone → (Wi-Fi) AirPlay → Mac/Linux box running UxPlay → GStreamer → JPEGs → your agent.

Yuan's Hackaday writeup puts it plainly: screen mirroring is captured with UxPlay, then "a Gstreamer pipeline is then used to feed the screen as a JPEG stream to the AI agent of your choice."

On my Linux box:
UxPlay is the unsung hero here. It's a real AirPlay receiver built on GStreamer plugins, emulating an . Your iPhone thinks it's just projecting to a TV. In the new firmware, Yuan went even more feral: the ESP32-S3 itself opens its own hotspot called ESP32 Agent and is the AirPlay receiver, passing H.264 through to an iPhone app over BLE. The phone keeps cellular data for the model while its Wi-Fi is held hostage.

Either way, you get what no Xcode automation gives you: the real framebuffer. Not a screenshot API, the actual rendered screen at ∼5 fps, 480x1040. Low enough to keep BLE happy, high enough for a VLM to read.

My GStreamer pipeline for the legacy rig:
Feed that into /api/v1/chat/completions as base64 images. That's your eyes.

2. The Hands: ESP32-S3 as USB HID, and why AssistiveTouch is the trick

Here's the bit that made me grin out loud.

Apple will never let a random USB device tap the screen. But it will let an accessibility mouse move a cursor. Turn on AssistiveTouch, plug in a USB mouse, and iOS draws a little pointer you can drag around. Tap = click. Swipe = drag.

Yuan's firmware does exactly that: the board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.

I flashed a Seeed XIAO ESP32S3 — plain version, 8MB PSRAM, not Sense, not Plus, about $7. Any ESP32-S3 with native USB and PSRAM works. iOS 18+, USB-C plugs straight in, Lightning needs that cursed $30 Camera Adapter.

Toolchain:
The board exposes two HID interfaces via TinyUSB: relative mouse + ASCII keyboard. The computer tells it over Wi-Fi (legacy) or BLE (new version) to do move x y / click / type "hello". It does.

The genius: pointer drift. Because the mouse is relative, every move accumulates error. Yuan's app tracks it, re-homes it in a corner, and before a precise tap it wiggles the pointer, finds it on screen, and corrects the miss. That wiggle is why my automation stopped missing the tiny x in Safari.

3. Swapping the DGX Sparks for Ollama on one consumer GPU

The README is honest: "The project was developed with nvidia/Qwen3.8-Flash-Next-NVFP4 served on two DGX Sparks (TensorFold)." Cool if you have $20k lying around.

My maker stack:

• Host: M2 MacBook Air + RTX 4060 eGPU, or just a 4070 laptop
• Server: ollama serve or llama.cpp --server with vision
• Model: qwen2.5vl:7b or minicpm-v:2.6. Qwen is more spatially accurate, MiniCPM is faster.
• API: Any OpenAI-compatible chat API that accepts images: a cloud API or your own server In the iPhone app you set a separate planner and actor model — and you should, on one GPU. I run Qwen2.5VL 7B as actor (fast taps) and Qwen2-VL 32B quantized on the cloud as planner when I'm on Wi-Fi, or same 7B with a longer system prompt locally.

Per-action latency on local 7B, measured from JPEG in to HID command out:

• Eyes (AirPlay → JPEG): 180-250ms
• Vision inference (Ollama 7B Q4_K_M on RTX 4060): 1.8-2.4s
• Hands (Wi-Fi → USB HID → pointer settle): 350-600ms
• Screen settle wait: 800ms

Total: ∼3.5 seconds per step. Cloud GPT-4o does it in 1.2s but leaks your banking screen to OpenAI. Local is slower, but it's yours. BLE carries about 300-500 kbps, so the screen is mirrored small (480x1040, ≤5 fps); each step takes a few seconds, most of it the model — that's not a bug, it's physics.

4. Planner vs Actor: One model or two

Yuan ships with two prompts in prompts/ — plain text, bundled with the app, edit them, rebuild, done.

This matters more than you think.

• One-model setup: One VLM gets "Goal: set alarm 8am Mon/Tue. Current screen:. What action?" It outputs {"action":"tap", "x":512, "y":1340}. Simple, but it forgets why it tapped after 6 steps and loops.[jpeg]
• Two-model setup: A planner (bigger, slower) says "Step 1: open Clock app. Step 2: tap Alarm. Step 3:..." The actor (smaller, faster) only executes step 2 given the current screen. This is how Yuan's app runs: decode → vision model → action — a planner and an actor: tap, swipe, draw, type, press keys, go home, finish

On one GPU, run the planner every 3 steps, not every step. Cache the plan. The actor fails less when it has a short horizon.

My prompt diff that fixed stuck loops:

"If the screen didn't change after your last action, you MISSED. Wiggle pointer, increase tap radius by 20px, or swipe instead. Never repeat identical coordinates."

Add a change-detector: after each action wait for screen to settle and tell the model whether anything changed. If image_diff < 2%, force a re-plan.

5. Latency and Failure Loops: Calibration, Re-plans, Stuck States

Real talk: this rig fails hilariously.

Failure modes I logged over 50 tasks:

1. Calibration drift: After 10 taps, pointer is 30px off. Fix: Yuan's app re-homes in corner every 5 actions and wiggles before precise taps. Replicate that. Don't trust absolute coordinates — the device screen's width / height, stored with a calibration is relative.

2. iOS black holes: iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); mirroring stops when phone locks. Your agent goes blind. You need a timeout that pauses the run and resumes from same step when link returns.

3. Keyboard hell: Keyboard input is ASCII only. Want to type Chinese? Nope. It types via USB keyboard events, so you get what a US keyboard sends.

4. Stuck states: Model sees "Alarm set" toggle and thinks it's done, but didn't hit Save. Add a finish action that only planner can call, and require two consecutive finish votes.

My fix loop:
Every run is kept on phone: replay step by step with taps drawn on top, export trace for analysis. Use it.

6. The Threat Model: What this proves about Wi-Fi-controlled HID implants

The README says "Only automate devices and accounts you own." It should say in bigger font: this is a BadUSB with a radio.

You plugged a $7 board into your iPhone that:

• Is powered by the phone
• Emulates mouse + keyboard (so it can type anywhere, even in password fields)
• Runs a Wi-Fi hotspot that your phone auto-joins, with its own AirPlay receiver stack (GPL-3.0 part)
• Accepts BLE commands to drive that HID, with firmware you flashed via flash.sh

In the legacy version, commands came over Wi-Fi from a Python server. In the new version, it's BLE from an app, but the board still controls HID. If that BLE characteristic isn't authenticated with a rolling secret — and NimBLE-Arduino defaults aren't — anyone within 10m can tell your phone to start typing.

This links directly to your hardware implant work. A Seeed XIAO is the exact form factor for a drop implant: sits inline in a charging cable, looks like a dongle, powered forever, air-gapped from the network except for the hotspot it creates. Apple trusts it because it's HID. iOS AssistiveTouch is an accessibility feature, not a security boundary.

Defenses:

• Flash with secrets.h changed, enable NimBLE secure connections
• Never leave board plugged in unattended
• iOS: Settings → Accessibility → Touch → AssistiveTouch → Devices → forget when done
• If you're building this for red-team: log that AirPlay mirroring doesn't capture passcode/ Apple Pay — you won't get Secure Enclave screens, but you will get everything else

Yuan did responsible disclosure: repo has SECURITY.md and responsible use note. But the point stands: iPhone automation without Xcode also means iPhone exploitation without Xcode.

---

BOM for the solo build (what I actually used):

• Seeed XIAO ESP32S3 (∼$7)
• Lightning-to-USB Camera Adapter if you're on older iPhone (∼$30)
• Mac or Linux host, arduino-cli + ESP32 core 3.x, UxPlay, python venv
• Ollama or llama.cpp with a vision model — I benchmarked qwen2.5vl:7b

Total cost if you own a GPU: $7. Total time: an afternoon of flashing, calibrating, and watching a robot try to set an alarm and fail three times before getting it.

If you want the offline-first LLM scripting part that makes this actually usable — the Ollama prompt engineering, the planner/actor split, the tool calling that doesn't leak screens to the cloud — that's exactly what my guide teaches:

[Aider + OpenClaw: Offline-First AI Scripting, $39.99 — gum.co/l/ggwux]

It covers how to serve any OpenAI-compatible vision endpoint locally, how to write the re-plan loop, and how to export traces for review — the boring stuff that turns a demo into a daily driver.

Apple built a walled garden. We just found the mouse hole.