DEV Community

Zihan Dai
Zihan Dai

Posted on Edited on

Talk to your coding agent by writing and drawing on an iPad

InkRoundtrip demo: a handwritten page on the iPad, and the agent's ticks, circle and note appearing on it

We talk to coding agents in three ways today: typing, voice, or pasting a screenshot. Some things are easier with a pen, though. A quick sketch of a system. A derivation. An arrow that means "this one, not that one". Getting a handwritten page to an agent usually means screenshot, upload, copy, and then reading a reply in a chat window that has nothing to do with your page.

So I built InkRoundtrip, an open-source bridge that makes the iPad page itself the conversation.

How it works

  • You write or draw on the iPad with Apple Pencil, in a native PencilKit app or a web board in Safari.
  • Each page is saved to a tiny Python server on your Mac (standard library only, protected by a pairing code).
  • When you ask, your agent (I use Claude Code; Codex works the same way) reads the page straight from your Mac and replies with circles, arrows, ticks and short notes at the exact spot, using annotate.py.
  • The reply shows up on the iPad a few seconds later, on its own layer above your ink.
  • Each reply remembers which version of the page it was about. Change the page and the app flags the old marks.

Where it came from

I found the idea while revising for exams: I wrote proofs by hand and wanted the agent to mark them like a tutor, on the page instead of in a chat. Once that loop worked, it was clear it wasn't really about marking. It's a pen-and-ink channel to your agent: ask a question in handwriting, sketch an architecture and get arrows back, circle the part of a plot you mean.

Try it

git clone https://github.com/PDGGK/InkRoundtrip.git
cd InkRoundtrip
python3 server.py --lan
Enter fullscreen mode Exit fullscreen mode

Open the printed link in Safari on your iPad, or on your Mac with ?mouse=1 to draw with the trackpad. Copy skills/inkroundtrip into your agent's skills folder and ask it to look at your page and answer there.

It's an early developer preview: Mac only, one writer per page, plain HTTP on your local network. The interface is in English and Chinese.

Repo: https://github.com/PDGGK/InkRoundtrip

I'd love to hear what you would use a pen-to-agent channel for.

Top comments (1)

Collapse
 
packetexchange_dev profile image
Packet Exchange •

The detail that got me is each reply remembering which version of the page it was about. Stale answers are what quietly kill trust in anything async, and you handled it on day one. Lovely touch for a preview release.

One small step on top: tying each mark to the nearby strokes instead of the whole page would let a tick survive when you only edit the far corner.

Does the flag currently fire on any change to the page, or only when the ink near the mark moves?