Qwen's new GUI agent report describes something rarer than a model: a laboratory of more than a hundred physical Android phones running over 150 real apps, with a scheduler that leases a working phone, app, account, network and display combination for each training run and blacklists unhealthy targets until a human repairs them. The agent is a multimodal model, but the substantive contribution is the operation around it.
Key facts
- More than 100 physical Android devices and 150-plus real apps, with virtual displays letting one phone host multiple isolated app sessions.
- The system includes a "User Agent" that seeks confirmation for sensitive or payment actions and hands control back for CAPTCHAs - bounded autonomy by design, not unattended control.
- Evaluation uses AutoJudge, a five-model majority vote over full trajectories, with a third
env_erroroutcome removed from the success-rate denominator. - Primary source: the Qwen-UI-Agent technical report, arXiv:2607.28227.
Anyone who has tried to run agents against real software knows the actual problem, and it is not clicking. It is that live apps fail constantly for reasons having nothing to do with the agent: sessions expire, a promo modal appears, the network drops, an account gets rate-limited, a permissions dialog fires. Train on that raw signal and the model learns from noise. Train on an emulator instead and you get clean signal for a world that does not exist.
Qwen's answer is a leasing system. A scheduler hands out viable combinations of device, app, account, network and display, reroutes around failures, and quarantines targets that keep breaking until someone physically fixes them. Virtual displays multiply capacity by letting one handset host several isolated sessions. It is closer to how a car manufacturer runs a durability track than to how a lab runs an experiment - and the sim-to-real framing the report uses is apt.
The design choice worth stealing is the User Agent. Rather than pursuing unattended end-to-end control, the system explicitly stops: it asks for confirmation before sensitive or payment actions, requests information it does not have, and returns the phone to a person for CAPTCHAs and other mandatory human steps. That is a bounded-autonomy architecture, and it is a more honest description of what a phone agent can safely do today than any benchmark score.
The action space is similarly pragmatic. It mixes ordinary touch gestures with shell commands, API calls, questions to the user, and batched sequences - so the agent clicks where visual state matters and drops to a command line where structured files or bulk transformations make clicking wasteful. The report's most revealing case study is not a shopping app but a browser game, where screenshots remain the state source while locally written code handles key presses and obstacle detection. That is the right instinct: the screen is an interface for humans, and an agent should use it only where it is the best available interface. This site covered the related argument that agents should edit the file instead of the screenshot.
The training loop is a flywheel: agents generate tasks, environments and some verifiers, inspect failed trajectories, classify the failures, and generate targeted tasks for the next round. The authors are careful to call this agent-driven rather than autonomous - humans still oversee and correct it.
Which brings us to the part that decides whether any of it holds. The live benchmark, MobileWorld-Real, runs several hundred human-written tasks across more than 100 live apps with real account state, changing content, pop-ups, expired logins, permissions and CAPTCHAs. It deliberately excludes scenarios whose starting state cannot be reliably restored - carts, orders, payment cards, purchase histories. So it is far more realistic than an emulator and still bounded for repeatability, which is the correct trade but worth knowing about.
Success is scored by AutoJudge: five vision-language models vote on the full screenshot and action trajectory, and runs classified as environment errors are dropped from the denominator. The authors report high agreement with expert annotation on a held-out set and concede residual measurement uncertainty.
That concession is where a second paper lands hard. OSReward built human-gold trajectory judgments across web, mobile and desktop specifically to test how well vision-language judges evaluate agents, and found they tend to accept an agent's account of its own success rather than verify what happened onscreen. The dominant error is approving incomplete work; false approvals substantially outnumber false rejections, and a failure-heavy challenge set exposes the weakness sharply. That does not refute Qwen's numbers, but it is a strong reason not to treat a model-as-judge majority vote as ground truth, especially on long trajectories where the agent produces a confident closing summary.
The alternative exists. Echoverse builds stateful applications whose success condition is checked against their own database, then uses every rollout twice - model failures become training signal, environment and verifier defects become repairs. Checking the database rather than the screenshot is the difference between marking an exam against an answer key and asking the student whether they got it right.
A third paper supplies a boundary condition. A study of local Qwen3-VL, UI-TARS and OpenCUA models on OSWorld found that inference-time scaling fails the intuitive test: a little screenshot history prevents loops, but piling on more history, more allowed steps, a planner layer or multiple candidate plans quickly costs far more than it returns - and longer runs commonly convert "stuck" failures into agents confidently stopping before the task is done.
The honest caveat: everything above is author-reported, none of it is independently replicated, and the three critical papers test different models than Qwen's. The defensible summary is that Qwen has moved GUI-agent work from "can a model click through a benchmark" to "can an organisation operate a messy phone lab, train on it, audit it and hand control back safely." The unresolved bottleneck is whether its evaluation can tell completion from an articulate near-miss.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)