In my last post I said that for normal HTML pages, element-based automation (find / read_page) beats coordinates every time. This post is about the apps where that advice is useless.
Flutter Web apps. Canvas-rendered editors. Every button and panel you can see on screen doesn't exist in the DOM — it's all pixels painted onto a single canvas. find returns nothing. read_page's accessibility tree is effectively empty.
I got Claude to drive the Rive editor (an animation tool built with Flutter) all the way through selecting assets and exporting them. Here's the procedure that survived contact with reality.
Step zero: confirm you're actually in this situation
Coordinate automation is fragile, so you should only accept it after ruling out the alternative. The test is quick: run read_page. If the visible UI has almost no corresponding nodes, you're looking at a canvas-rendered app, and coordinates are the only interface you have.
The four rules
1. Wait for the window size to settle before anything else
Same failure mode as my previous post: right after load, the viewport hasn't reached its final width (I measured 1664→1920 over 2–3 seconds), and clicks based on an early screenshot land to the right of the target. Read innerWidth via javascript_tool twice; only proceed when two consecutive reads match. But matching innerWidth alone isn't enough — also confirm devicePixelRatio hasn't changed since the screenshot you're about to act on (a follow-up to my previous post surfaced this: when DPI or scaling changes, the whole coordinate space rescales the same way, but the new values stabilize immediately, so an innerWidth-only check can't catch it). Canvas apps deserve extra paranoia here, because there is no element-based fallback when a click misses.
2. Read text by zooming, not by extracting
Text painted on canvas can't be pulled out of the DOM. To read a menu item or panel label, zoom into that region and read the enlarged screenshot as an image. Full-page screenshots make small text easy to misread — make "zoom before you read" a reflex.
3. Demand visual evidence after every click
A missed coordinate click produces no error. Nothing. And if you proceed without noticing, every subsequent step operates on a false premise.
So: click → screenshot → confirm the visual evidence of selection (in Rive, the blue highlight; in other apps, a selection outline or color change) → only then continue. No evidence, no progress — re-derive the coordinates and retry.
4. Build double-clicks from two rapid single clicks
The double_click action can exceed the app's double-click threshold and degrade into two single clicks. Canvas editors love double-click-to-enter-hierarchy interactions, so this will block you. Fire two left_clicks at the same coordinates inside one browser_batch — the gap tightens enough to register.
A real run (Rive editor)
Putting it together, "select an asset and export it" looks like this:
- Open the editor → wait until
innerWidthstabilizes - Full screenshot → locate the target panel
- Rapid double-fire on the object → screenshot → confirm blue highlight
- Click the menu coordinates → zoom the menu region → read the item labels
- Click Export → screenshot → confirm the dialog appeared
Every step has a verification wedged after it, so the operation count is easily double what a human would need.
Slow, but it keeps moving.
Chasing speed with coordinate automation produces chains of unnoticed misses, and those cost far more time than the checks do. That was the actual lesson of running this in production.
Limits and caveats
This approach assumes a stable screen layout. Pages that reflow constantly (ads, notifications) degrade coordinate reliability even further.
Any specific app's UI layout changes with updates — Rive's included. What transfers is the procedure, not the pixels: settle-wait → zoom-read → evidence-check → rapid double-fire.
Verified on Windows / 1920×1080. Different DPI scaling may change coordinate behavior.
Verified: June 2026. Environment: Windows 11 / Chrome + Claude in Chrome extension / Rive editor (Flutter/canvas).
Top comments (2)
This sequence is the part most coordinate-automation posts skip. Step zero (rule out the alternative first) is the one that decides whether the rest is worth the cost. The settle-wait -> zoom-read -> evidence-check -> rapid-double-fire loop is the same shape we ended up at when we were capturing annotated screenshots from a Flutter canvas for AI routing, and the only place we ever saved real time was the evidence step. Without it, every 'missed click with no error' propagated into a bad prompt and the retry cost was 10x what the screenshot would have been. The thing I'd add: keep a coordinate-confidence score per click derived from the zoom-read step. If the second screenshot doesn't match the first within a tight pixel band, treat the click as unconfirmed even if the API call returned 200. That single rule cut our silent-miss rate from ~15% to under 2% in a similar Flutter editor pipeline, and it generalizes to any canvas app where the visual evidence check is the only ground truth you have. Did you hit a similar pattern in Rive, or did the asset-tree panel give you a fallback that canvas-only apps never do?
Same pattern — and no fallback. I re-checked it live to be sure.
Rive's editor is Flutter on the CanvasKit renderer: the entire UI — hierarchy panel, toolbar, every property field — is painted into a single
<canvas>. The DOM holds zero<button>,<input>, or<div>for any of it (~48 nodes total, almost all Flutter host tags + scripts). An accessibility-tree read of the whole editor returns one generic node; a semantic search for "hierarchy item / tool button / property field" comes back with nothing but "a button for enabling accessibility." So the asset/hierarchy tree is sitting right there on screen — and it's completely unqueryable. A visible tree in a canvas app is still just pixels; it never bought us an escape hatch.So yes, coordinate-only — and both failure modes carried straight over:
innerWidthstabilizes and you land off-target with no error.