Gemini Computer Use GA: Screenshot Loops, Coordinate Grids, and Why Accessibility Trees Still Win on Windows Latency

#ai #windows #tooling #agents

On October 30, 2026 Google flipped Computer Use to general availability across both the Interactions API and GenerateContent. Four models now support it (Gemini 4 Argon, 3.8 Flash, 3.5 Flash-Lite, 3 Flash Preview). The recommended default is gemini-3.8-flash. Your code sends a screenshot + prompt; the model returns a structured action (click, type, scroll, etc.) that you execute, then you send the next screenshot. Classic agentic loop.

The interesting part for anyone shipping a real-time Windows tool is the latency and reliability tradeoff against the accessibility tree.

The loop in practice

Minimal Python sketch (using the official google-genai client ≥ 2.7.0):

from google import genai
import base64

client = genai.Client()

def run_turn(screenshot_png: bytes, prompt: str):
    interaction = client.interactions.create(
        model="gemini-3.8-flash",
        input=[
            {"type": "text", "text": prompt},
            {"type": "image", "data": base64.b64encode(screenshot_png).decode(),
             "mime_type": "image/png"}
        ],
        tools=[{"type": "computer_use", "environment": "desktop"}]
    )
    # Parse function_call from interaction
    return interaction

# Then you execute the returned action (click(x,y), type(text), ...)
# and loop until done or safety filter fires.

Coordinates come back on a 0-999 normalized grid. You scale them to the real screen or window. Get the scaling wrong once and every click lands in the wrong place. The intent field on newer models is useful for logging, but it does not replace a deterministic check that the target control actually exists.

Screenshot tax vs accessibility tree

Every turn pays:

Screenshot capture (GDI BitBlt or DXGI duplication on Windows)
Base64 + network round-trip
Model inference
Action execution + wait for UI to settle
Repeat

On a local machine that can easily be 400-800 ms per step before the UI even reacts. For a coding agent that is fine. For an overlay that needs to stay in sync with a live call, it is not.

The alternative that Windows has had for years is UI Automation (UIA). You walk the accessibility tree, read control names, bounding rectangles, and patterns (Invoke, Value, Selection), and act directly on the element. No pixels, no coordinate scaling, no “is this the right button?” OCR step.

A previous post here already covered Vercel’s agent-desktop doing exactly this with the accessibility tree instead of screenshots. Computer Use is the screenshot path; agent-desktop is the tree path. They solve different constraints.

When the screenshot path still makes sense

Cross-platform or non-Windows targets where UIA does not exist
Custom-drawn or game-like UIs that do not expose a proper accessibility tree
One-shot “do this task” agents where 1-2 second steps are acceptable
When you want the model to reason about visual layout that the accessibility tree cannot express

Practical constraints on Windows

WASAPI loopback + cpal still gives you the cleanest dual-track audio path (mic + system) with sub-100 ms capture budgets if you stay in shared mode and do your own resampling. That same process can also hold a UIA walker if you need the agent to click something in the call UI without joining the meeting. The two techniques are complementary, not competing.

Safety filters and the optional enable_prompt_injection_detection flag are on by default in the docs examples. For a desktop environment that is running under the user’s own session you still want an explicit allow-list of actions and a hard stop on anything that looks like credential entry.

Architecture note

[Screenshot / UIA snapshot]
        |
        v
[Gemini Computer Use or local model]
        |
        v
[Action executor] --> [Win32 SendInput / UIA Invoke]
        |
        v
[Next state] -----> loop

The executor must be idempotent and must verify the post-condition (control disappeared, value changed, window closed) before declaring the step done. Models are happy to re-issue the same click forever if you do not check.

Tradeoffs that actually matter

Path	Latency per step	Reliability on standard Win32/WPF	Works on custom-drawn	Needs network
Screenshot + Computer Use	400-800 ms	Medium (coordinate + OCR)	Yes	Yes
UIA accessibility tree	10-50 ms	High	No	No
Hybrid	Variable	High for controls, fallback screenshot	Yes	Optional

Hybrid is what most production desktop agents will end up with: prefer the tree, fall back to pixels when the control is missing or the tree is incomplete.

For a live interview or call assistant the hard part is still the audio pipeline and turn-taking, not the UI automation. Computer Use being GA just means one more tool in the box when you need the agent to drive a settings dialog or a browser tab without writing a custom parser.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.

DEV Community