Automating an app with no DOM: driving Flutter/canvas editors with coordinates only

In my last post I said that for normal HTML pages, element-based automation (find / read_page) beats coordinates every time. This post is about the apps where that advice is useless.

Flutter Web apps. Canvas-rendered editors. Every button and panel you can see on screen doesn't exist in the DOM — it's all pixels painted onto a single canvas. find returns nothing. read_page's accessibility tree is effectively empty.

I got Claude to drive the Rive editor (an animation tool built with Flutter) all the way through selecting assets and exporting them. Here's the procedure that survived contact with reality.

Step zero: confirm you're actually in this situation

Coordinate automation is fragile, so you should only accept it after ruling out the alternative. The test is quick: run read_page. If the visible UI has almost no corresponding nodes, you're looking at a canvas-rendered app, and coordinates are the only interface you have.

The four rules

1. Wait for the window size to settle before anything else

Same failure mode as my previous post: right after load, the viewport hasn't reached its final width (I measured 1664→1920 over 2–3 seconds), and clicks based on an early screenshot land to the right of the target. Read innerWidth via javascript_tool twice; only proceed when two consecutive reads match. But matching innerWidth alone isn't enough — also confirm devicePixelRatio hasn't changed since the screenshot you're about to act on (a follow-up to my previous post surfaced this: when DPI or scaling changes, the whole coordinate space rescales the same way, but the new values stabilize immediately, so an innerWidth-only check can't catch it). Canvas apps deserve extra paranoia here, because there is no element-based fallback when a click misses.

2. Read text by zooming, not by extracting

Text painted on canvas can't be pulled out of the DOM. To read a menu item or panel label, zoom into that region and read the enlarged screenshot as an image. Full-page screenshots make small text easy to misread — make "zoom before you read" a reflex.

3. Demand visual evidence after every click

A missed coordinate click produces no error. Nothing. And if you proceed without noticing, every subsequent step operates on a false premise.

So: click → screenshot → confirm the visual evidence of selection (in Rive, the blue highlight; in other apps, a selection outline or color change) → only then continue. No evidence, no progress — re-derive the coordinates and retry.

4. Build double-clicks from two rapid single clicks

The double_click action can exceed the app's double-click threshold and degrade into two single clicks. Canvas editors love double-click-to-enter-hierarchy interactions, so this will block you. Fire two left_clicks at the same coordinates inside one browser_batch — the gap tightens enough to register.

A real run (Rive editor)

Putting it together, "select an asset and export it" looks like this:

1. Open the editor → wait until innerWidth stabilizes
2. Full screenshot → locate the target panel
3. Rapid double-fire on the object → screenshot → confirm blue highlight
4. Click the menu coordinates → zoom the menu region → read the item labels
5. Click Export → screenshot → confirm the dialog appeared

Every step has a verification wedged after it, so the operation count is easily double what a human would need.

Slow, but it keeps moving.

Chasing speed with coordinate automation produces chains of unnoticed misses, and those cost far more time than the checks do. That was the actual lesson of running this in production.

Limits and caveats

This approach assumes a stable screen layout. Pages that reflow constantly (ads, notifications) degrade coordinate reliability even further.

Any specific app's UI layout changes with updates — Rive's included. What transfers is the procedure, not the pixels: settle-wait → zoom-read → evidence-check → rapid double-fire.

Verified on Windows / 1920×1080. Different DPI scaling may change coordinate behavior.

---

Verified: June 2026. Environment: Windows 11 / Chrome + Claude in Chrome extension / Rive editor (Flutter/canvas).