Asking a vision model what and where in the same call makes it worse at both

Maaz Kazi — Fri, 25 Sep 2026 12:23:29 +0000

Handrail takes a screenshot of whatever you are stuck in, answers your question about it, and then draws an arrow on the real control you need to touch. The obvious way to build that is one call: here is the screen, here is the question, give me the answer and the coordinates.

That is how I built it first, and it is worse at both halves.

Two passes, one job each

Handrail now makes two vision calls against the same screenshot.

The answering pass. The screenshot, the question, the last few turns, and any attached files go to the model. It replies with a short answer, or a checklist when the job genuinely takes several steps, and it names the one control you need to touch.
The pointing pass. The same screenshot goes back with a single question: where is that control? Answer as coordinates.

Separating them helped for a reason that is obvious in hindsight. The answering pass is a reading and reasoning task over the whole screen. The pointing pass is a localisation task over one named target. Asking for both in one response makes the model hold a spatial answer in working memory while it composes prose, and the coordinates are the part that degrades.

It also makes failure legible. If the answer is right and the arrow is wrong, you know exactly which pass to fix.

Coordinates that survive the real world

The second pass does not return pixels. It returns a position on a 0-1000 normalised grid, which Handrail then maps onto actual screen pixels.

This sounds like a detail and it removes an entire category of bug. Screens differ in resolution, in DPI scaling, and in how many of them are plugged in. If the model returns pixels, every one of those becomes arithmetic you have to get right on someone else's hardware. Normalised coordinates are resolution-independent by definition, so DPI and multi-monitor never enter the maths at all.

The two passes also do not get the same image. Answering uses a 1600px JPEG, which is enough to read a screen and cheap to send. Locating uses a native-resolution PNG, because the thing you are pinpointing may be a 12px chevron.

The model choice this forced

Splitting the work also changed which model I could use. The default is now the cheapest one that reliably reads a screen. I tried going cheaper still, and the lite tier failed in the most dangerous way available to a screen assistant: it invented menu paths for UI that was not in the screenshot.

That is the failure mode worth designing against. A model that says it cannot see the control is recoverable. A model that confidently names a menu item that does not exist sends someone hunting through Settings for something that was never there, and the whole product exists to stop exactly that.

What stays on the machine

Since the thing is looking at your screen, the trust boundary matters more than the features.

The API key is encrypted at rest by the operating system's own keychain, not by me, and never crosses into the UI layer after setup.
Conversations and attachments are JSON on disk in the app's data directory. No database, no account, no server.
Web search is off by default, and the only thing that ever leaves is the screenshot and your question, to the model you chose.
One production dependency. Everything else is Electron and Node built-ins.

A screen assistant that phones home is a different product, and a worse one.

Handrail is open source, Apache-2.0, with tagged Windows and macOS releases: https://github.com/M19K/handrail

It agreed with the reference 100% of the time. It was right 75% of the time.

Maaz Kazi — Wed, 23 Sep 2026 20:41:20 +0000

Swap a cheap model in behind an expensive one and the obvious way to check it is shadow traffic: send the same request to both, compare the answers, count how often they agree. High agreement, ship it.

I measured that on 60 live calls while building SuperRouter. The routed model agreed with the reference 100% of the time. It was correct 75% of the time.

Both numbers are real. The gap between them is the whole problem.

Agreement is not correctness

Agreement asks whether two models produced the same answer. It never asks whether the answer was right. And two models from the same era, trained on overlapping data, fail in the same direction far more often than they fail independently, so agreement is highest exactly where it protects you least.

A shadow run reporting 100% is consistent with two different worlds: a cheaper model that is genuinely as good, and a cheaper model that is wrong in precisely the same places as the expensive one. Nothing in the agreement number separates them.

What I do instead

Score against ground truth, per failure mode, and never against the other model.

Ground truth is generated, not hand-labelled. Faults are planted deliberately, so the right answer is known before any model sees the case.
A planted defect has to move pixels. One of 18 fault classes returned success and changed nothing on screen. Every fixture is now gated against a healthy frame of the same screen, because a fault no model could possibly have caught was quietly inflating every score.
Both halves of the exam have to be hard. The faithful cases were verbatim copies of the source, so false-alarm rates sat at 0-3% across seven models and that axis measured nothing at all.
Runs from different versions of a set are never compared. Every run fingerprints the exact cases it sat. Without that, the table ranked a model measured on 90 easy cases above one measured on 592.

The part that surprised me

I assumed a published leaderboard could stand in for measuring your own product. Across two products, rank order mostly transfers when a model is judging (0.83), but every model dropped a median 22 points in absolute terms. When the task was pointing at the right control rather than judging, the order barely transferred at all (0.49).

So a leaderboard tells you roughly who is good in general. It does not tell you who is good at your thing, and the second question is the only one that decides your bill.

The takeaway

If you are routing to save money, agreement rate is the metric you will reach for first and the one that will mislead you hardest. Measure correctness against something you constructed, split by the ways your product can actually break. Otherwise you are measuring how similar two models are and calling it quality.

SuperRouter is open source, Apache-2.0, with no runtime dependencies: https://github.com/M19K/superrouter

DEV Community: Maaz Kazi