I've been experimenting with AI coding agents for a while now, and there's something that always felt off. When I'm working on architecture or planning features, I sketch things out on a whiteboard or Excalidraw. But my agent just sees text. I describe diagrams, paste screenshots, try to explain spatial relationships—it's awkward and error-prone.
Most solutions I tried either required the agent to output some diagram format that I'd then render, or involved clunky screenshot pipelines. Neither felt natural. The agent can't look at what I'm pointing at, reference it in conversation, or edit it incrementally based on feedback.
What I wanted was the agent to actually sit with me at the canvas. Write AGENT: next to something, have it see what's around that area, edit it, and mark the note done. That's closer to how I think.
I ended up building something that does exactly this—connects Claude Code, Codex, or opencode to a live Excalidraw canvas. It handles the vision, the scene state, and bidirectional communication between the chat and the canvas.
Curious if others have run into this same friction, or if most people just don't try to use visual tools with their agents at all.
Top comments (0)