Why coordinate-based desktop automation breaks, and what to do instead

FlowBotCommander — Wed, 23 Sep 2026 10:09:44 +0000

Every desktop automation script starts the same way. You record a click, you get click(842, 517), and it works. It keeps working for exactly as long as nothing on the screen moves.

Then someone plugs in a second monitor. Or Windows scaling goes from 100% to 125%. Or the app ships an update that moves a toolbar down by twelve pixels. The script keeps running, keeps clicking 842/517, and now hits whatever happens to be there. It fails silently, which is the worst way for automation to fail.

I have been building a tool in this space for a while. Here are the three things I had to learn, none of which were obvious to me at the start.

1. Finding the button by its image is the right idea, done naively

The obvious fix: do not store a position, store a picture of the button and look for it.

AutoHotkey has ImageSearch, and most automation toolkits have something similar. The problem is that the classic implementations match exactly. Every pixel has to agree.

That sounds robust until you list what changes pixels without changing what a human sees:

font anti-aliasing differing between machines
a drop shadow from another window overlapping the edge of your button
the app switching between light and dark theme
display scaling at 125% or 150%
compression artifacts in remote sessions
a hover state, because the mouse happened to be nearby

Any one of those and an exact match returns nothing. You have swapped clicking the wrong thing for doing nothing at all. That is an improvement, because a silent wrong action is worse than a visible stall. It is not a solution.

2. What you want is a tolerance, and the user has to own it

The fix is template matching with a similarity score rather than a boolean: compute how well the template correlates with each region of the screen, take the best match, compare it against a threshold.

The interesting part is not the algorithm. OpenCV's matchTemplate has done this for years. The interesting part is that the threshold is a product decision, not an implementation detail.

Set it too high and you are back to exact matching. Set it too low and your Confirm button matches the Cancel button next to it, because at 70% similarity two grey rounded rectangles with a word on them are basically the same thing.

There is no value that is right for everyone. So the threshold has to be visible and adjustable per step, with a way to see what matched. A confidence number in a log file is not enough. People need the box drawn around what the tool found. Almost every support question I get traces back to a threshold that was fine on one screen and wrong on another.

A related trap: two visually identical buttons where one is disabled and greyed out. A tolerant match will happily click the disabled one. Brightness and saturation checks alongside the shape comparison turned out to matter more than I expected.

3. Clicking is the easy half. Reading is the hard half.

Automation that only clicks is a macro. Automation that reacts has to read the screen, and that means OCR.

OCR on screen content is a different problem from OCR on scanned documents. It is cleaner, because you have crisp rendered text instead of paper artifacts. It is also less forgiving, because you usually want one specific number rather than a paragraph, and you want it right every single time. A 5 read as a 6 in a document is a typo. In an automation loop it is a wrong decision, repeated.

Two things helped more than reaching for a better model: restricting recognition to a small region the user defines, and restricting the expected character set. If a field can only hold digits and a decimal separator, say so, and a whole class of errors disappears.

When not to do any of this

This is the part tools in this category tend to leave out, so let me be direct: screen automation is a workaround. It is the right tool when there is no better interface, and the wrong one when there is.

If the program has an API, an export or a command-line interface, use it. It will be faster and it will not break when a button moves.
If you only need hotkeys and text expansion, AutoHotkey is free, mature and built exactly for that.
If it is a one-off job, doing it by hand is probably quicker than building the flow.
Check what you are allowed to automate. Terms of service exist, and in games in particular, single-player only is a line worth respecting.

Screen automation earns its place in a narrow band: repetitive work, in software you do not control, where no interface exists. That band is wider than it should be, mostly filled with line-of-business software nobody maintains any more. But it is a band, not a universe.

Disclosure: I build FlowBotCommander, a Windows tool that does the above in a visual editor. It is free for 100 runs a day. This post was drafted with AI assistance and edited by me.

DEV Community: FlowBotCommander

Why coordinate-based desktop automation breaks, and what to do instead

1. Finding the button by its image is the right idea, done naively

2. What you want is a tolerance, and the user has to own it

3. Clicking is the easy half. Reading is the hard half.

When not to do any of this