Asking a vision model what and where in the same call makes it worse at both

Handrail takes a screenshot of whatever you are stuck in, answers your question about it, and then draws an arrow on the real control you need to touch. The obvious way to build that is one call: here is the screen, here is the question, give me the answer and the coordinates.

That is how I built it first, and it is worse at both halves.

Two passes, one job each

Handrail now makes two vision calls against the same screenshot.

1. The answering pass. The screenshot, the question, the last few turns, and any attached files go to the model. It replies with a short answer, or a checklist when the job genuinely takes several steps, and it names the one control you need to touch.
2. The pointing pass. The same screenshot goes back with a single question: where is that control? Answer as coordinates.

Separating them helped for a reason that is obvious in hindsight. The answering pass is a reading and reasoning task over the whole screen. The pointing pass is a localisation task over one named target. Asking for both in one response makes the model hold a spatial answer in working memory while it composes prose, and the coordinates are the part that degrades.

It also makes failure legible. If the answer is right and the arrow is wrong, you know exactly which pass to fix.

Coordinates that survive the real world

The second pass does not return pixels. It returns a position on a 0-1000 normalised grid, which Handrail then maps onto actual screen pixels.

This sounds like a detail and it removes an entire category of bug. Screens differ in resolution, in DPI scaling, and in how many of them are plugged in. If the model returns pixels, every one of those becomes arithmetic you have to get right on someone else's hardware. Normalised coordinates are resolution-independent by definition, so DPI and multi-monitor never enter the maths at all.

The two passes also do not get the same image. Answering uses a 1600px JPEG, which is enough to read a screen and cheap to send. Locating uses a native-resolution PNG, because the thing you are pinpointing may be a 12px chevron.

The model choice this forced

Splitting the work also changed which model I could use. The default is now the cheapest one that reliably reads a screen. I tried going cheaper still, and the lite tier failed in the most dangerous way available to a screen assistant: it invented menu paths for UI that was not in the screenshot.

That is the failure mode worth designing against. A model that says it cannot see the control is recoverable. A model that confidently names a menu item that does not exist sends someone hunting through Settings for something that was never there, and the whole product exists to stop exactly that.

What stays on the machine

Since the thing is looking at your screen, the trust boundary matters more than the features.

• The API key is encrypted at rest by the operating system's own keychain, not by me, and never crosses into the UI layer after setup.
• Conversations and attachments are JSON on disk in the app's data directory. No database, no account, no server.
• Web search is off by default, and the only thing that ever leaves is the screenshot and your question, to the model you chose.
• One production dependency. Everything else is Electron and Node built-ins.

A screen assistant that phones home is a different product, and a worse one.

Handrail is open source, Apache-2.0, with tagged Windows and macOS releases: https://github.com/M19K/handrail