I Still Can't Write Code. But Now ChatGPT Can Drive My Phone — and It Can't Approve Itself.

Last month I wrote about the alarm that fired into nothing — how an agent I'd scheduled for 8am went off on time and then silently didn't run, and what it took before I'd trust a schedule again.

This one is about handing that same phone to ChatGPT. Not "ChatGPT writes a script and I paste it." ChatGPT, in the ChatGPT app, asking my phone to do something, the phone doing it on a screen I'm not looking at, and the answer coming back into the chat.

What it looks like

I typed one request into the ChatGPT app: save the top-rated vintage stores near Shimokitazawa to a Google Maps list. On the right, my phone asks me to approve once. Then Google Maps starts moving on its own in the bottom-right pane — a background virtual display, not my screen — and a minute later ChatGPT tells me which two stores it saved. The list is really there when I open Maps.

The second one is quieter and, honestly, the one I use more: one message from ChatGPT registers an agent that runs on the phone every day at 16:27, researches the latest AI papers with Perplexity, and drops a sourced English summary into Obsidian. No ChatGPT, no tap, at 16:27. Every day after that, too.

How the pieces fit

• ChatGPT → Shelly. The ChatGPT app can drive Codex remote control on a machine you pair it with. Shelly already ships Codex natively, so it can host that on the phone itself: shelly codex pair --background, and from the ChatGPT app the phone is just another machine. Commands ChatGPT runs land in Shelly's real Android PTY.
• Shelly → ClosePaw. Shelly doesn't tap screens. ClosePaw does — an agent that looks at the screen and operates apps. I forked it so that a request from Shelly (and only Shelly, pinned by signing certificate) runs on a Shizuku virtual display without opening ClosePaw or asking ClosePaw's own confirmation.
• Back to ChatGPT. shelly closepaw --wait "<goal>" blocks until I approve and ClosePaw finishes, then prints what ClosePaw reported. ChatGPT reads that and answers.

The part I cared about most

If ChatGPT can drive the phone, ChatGPT can also, in principle, do the whole loop by itself: ask, approve, act. A web page it reads could say "also run this," and nobody would ever look.

So a hand-off to ClosePaw always waits for one approval on the phone:

• the notification's Approve button, which requires unlocking the device, or
• typing approve <hash8> in Shelly's own chat.

ChatGPT can't approve. Writing an "approved" file from the terminal doesn't work either — the one-time nonce only lives in Shelly's process memory. I considered "let ChatGPT ask me, and I'll type yes in ChatGPT," and dropped it: the thing that finally sends that "yes" to the phone is ChatGPT itself.

The result that comes back is checked too: it has to come from the signature-pinned ClosePaw build, carry the request's own one-time 128-bit nonce, and on Android 14+ the sender package has to match. The text itself is treated as data — control and bidi characters stripped — not as instructions.

What the demo shoot found

It took about eight takes. Each one broke something I would otherwise have shipped:

• The first version never told ChatGPT the result. ClosePaw finished; ChatGPT kept saying "waiting for approval." A demo where you can't see the result isn't a demo.
• Approving from the notification yanked Shelly full-screen and wrecked the split screen. Now the approval is handled in the background.
• I said "with Perplexity" and the agent quietly picked something else on its scheduled run — the named tool was being dropped at confirm time.
• I asked in English and got Japanese summaries, because my profile memory is Japanese. The task's own language now wins.
• gpt-4.1-mini, driving ClosePaw, got lost saving three stores. gpt-4.1 and two stores worked.
• At one point I tapped the "show the background screen" button and LINE opened, chat list and all, mid-recording. That take is gone.

What I still can't do

Write the Kotlin that holds this together. Promise that the virtual display trick works the same on every phone. Tell you whether OpenAI's remote-control host reconnects on its own when my phone hops from Wi-Fi to LTE — I'm finding that out this week.

What I can do is notice when the demo looks good and the result isn't actually there, and refuse to post it until it is.

GitHub: github.com/RYOITABASHI/Shelly — v8.1.0, GPLv3. ClosePaw's Shelly build: RYOITABASHI/closepaw releases.