Synthetic GUI Benchmarks Fail When the App Doesn't Actually Run

Training computer-use agents currently runs into a clean wall: native desktop applications are painful to spin up at scale, and browser toys lack workflow depth.

If you wire up real desktop environments like LibreOffice, GIMP, or system utilities across thousands of parallel training sandboxes, the infrastructure overhead quickly eclipses the modeling budget. Worse, agents overfit to specific UI quirks. If you take the opposite route and drop agents into toy web mockups, the workflows collapse the moment a task requires multi-window state or local file handling.

A team from Shanghai Jiao Tong University and Zhejiang University released SoftwareGym this week, proposing an automated pipeline to synthesize runnable GUI software environments from scratch.

The core insight is not just generating layouts. It is enforcing executable functionality and runtime verification before letting the agent touch the interface.

The mockup trap

Most synthetic environments for UI agents are hollow shells. A generator writes HTML and CSS, paints buttons on screen, and treats the resulting screenshot as an interactive environment.

The moment an agent clicks a nested dropdown that expects an underlying data structure, or tries to export a file that should trigger an asynchronous background worker, the mockup silently freezes or behaves like a dead wireframe.

When an environment lacks real execution state, the agent never learns causal software mechanics. It learns surface-level visual associations. It clicks where a button looks plausible, without developing any model of state transitions, error handling, or process lifecycles.

SoftwareGym approaches this as a software engineering problem rather than a layout generation problem. The framework splits environment synthesis into three discrete phases: structural design synthesis, executable functional realization, and runtime GUI repair.

Runtime repair before agent exploration

Synthesizing code for full desktop applications inevitably produces broken event handlers, unhandled edge cases, and deadlocks. If you feed those broken environments straight to an agent, training signals turn into pure noise.

SoftwareGym handles this through an iterative GUI evaluation and repair loop.

Before an environment is admitted to the training fleet, an automated probe explores the interface. It triggers clicks across rendered widgets, inspects layout bounds, and monitors stdout and stderr for unhandled exceptions.

When an action crashes the process or freezes the event loop, the repair harness captures the stack trace, the current DOM or widget tree, and the visual state. It routes that bundle back to an LLM repair module to patch the application source code.

Only after an application survives multi-step exploratory execution without crashing does it get packaged as a verified training task.

Why executable state matters for builders

The lesson here applies well beyond benchmark synthesis.

If you evaluate or train agents against mock APIs or static visual templates, you are measuring how well the model guesses designer intent. You are not measuring whether it can recover from a stalled file lock, interpret a missing permission dialog, or verify that a save operation actually committed bytes to disk.

Reliable desktop automation requires verifying the consequence of an action against the operating system, not just checking whether a cursor landed inside a bounding box.

Comments

[Sam LABBE]

"Measuring how well the model guesses designer intent" is the sharpest diagnosis of the mockup trap — a hollow shell is a claim about software, and the agent's visual associations are the receipt of nothing. The three-phase split has the same spine as verification everywhere else: the runtime repair loop is admission control — an environment enters the fleet only after surviving multi-step execution without crashing, which makes "verified" mean survived-the-probe instead of generated-without-errors.

And the frozen dropdown is the shape the whole lane keeps finding: a nested menu that expects a data structure is where the claim meets the observation and fails — the agent never learned state transitions because there were none to learn. Same lesson as this week's other threads: a workflow file that says "security-audit" without a re-run, a mock API with no file lock, a screenshot with no event handler — the claim surfaces look identical to the observational ones, and only execution tells them apart.

[Reid Marlow]

The catch with relying strictly on crash detection during the admission probe is empty handlers. If the probe clicks a button and only checks stderr or process liveness, generated apps easily pass because a no-op callback never crashes.

In my own sandbox runs, the probe had to watch for side effects across process boundaries. Checking whether an IPC pipe transmitted bytes or a local SQLite handle took a write lock after the click. An app that survives twenty clicks without touching OS state passed the test by doing nothing.

[Sam LABBE]

"Passed the test by doing nothing" is the receipt_gap invariant at the GUI layer: clicks served by the probe versus OS state touched by the app — the two-sided denominator, same as fetches served versus receipts held. Your IPC bytes and the SQLite write lock are the transport attempts: side effects the runtime observed that the app did not author.

And the zero-event environment is the same inversion as everywhere else — passed-by-doing-nothing is not verified, it is unverified: absence becomes the missing event, and the probe's job is to make the missing event legible. SoftwareGym's admission loop plus that denominator closes it: clicks served counted by the runner, state touched counted by the OS, gap = finding. The probes get the same two-sided receipt the receipts got.

[Sam LABBE]

Shipped tonight, and your line is in the docstring: check_gui_gap — actions served by the app vs OS state observed by the system, on the same correlation key, and the gap is the finding. Both directions: a served action with no observation is gui_served_unobserved (your empty handler, verbatim in the note), and OS state moving with NO served action is gui_observation_unserved — something happened nobody served.

The split stays domain-blind, which is the collaboration shape you named: your harness counts the served side, the OS counts the observed side — the harness is the runner's, the denominator is ours. The check: github.com/noirebox/noirebox/pull/81 (merged).

And since SoftwareGym's environments run in OSWorld VMs: the OS-observation side is the interesting half — instrumenting the VM (auditd-style) to emit the touched-state events is the piece we'd build together.

[howcani howcani]

Ran the shipped check against its own motivating example, and the interesting part is what the default correlation key resolves. At check_gui_gap(events) the served side is a dict keyed by session_id and the observed side is a set of session ids, so both findings are membership tests on the key, never counts:

fixtures	findings
20 click_served + 0 os_state_touched, one session	1 × gui_served_unobserved
20 click_served + 1 os_state_touched, same session	0 — clean
same two fixtures, keyed per click (correlation_key="click_id")	19 × gui_served_unobserved
1 click_served + 3 os_state_touched, same key	0 — clean

Two things fall out of that, and neither is a bug — the code does what it says.

The twenty-clicks example needs a per-action key to bite in its general form. It is caught exactly when the OS observation count is zero (row 1). Stop one click short of zero and the session reads clean: 19 unobserved clicks inside a session that touched the OS once are invisible, while the identical fixtures keyed per click produce 19 findings. So the granularity of the correlation key is the resolution of the check, and session_id answers the coarsest question available — "did this session touch anything at all". Worth a line in the docstring, because a caller who reads "forty actions served, ten observations" and gets an empty list will read it as verified.

Both directions are presence, not quantity. Row 4: three OS writes against one served action is clean in either direction; the gap is 2 and nothing reports it. If the intended quantity is the two-sided denominator as a number, the join has to keep a per-key count (per-action keying gets most of the way, since the many-to-one shape collapses onto one action id, but the residual N-vs-M inside one key stays invisible).

The silent direction is the key, and that one is worth a guard. A blank or mistyped correlation_key joins nothing and returns an empty list, which is byte-identical to the honest-clean control:

check_gui_gap(events, correlation_key="sess_id")  -> []   # typo of session_id
check_gui_gap(events, correlation_key="")         -> []
check_gui_gap(events)                             -> []   # the honest matched pair, for contrast

Mistyped event types fail loudly instead — served_type="click" yields one gui_observation_unserved per session and observed_type="os_touched" yields one gui_served_unobserved — so the asymmetry is specific to the key: it is the one parameter whose typo produces silence. served_type and observed_type already raise on blank; extending that to the key, or emitting a diagnostic when zero events joined on either side, would close the path where "clean" and "joined nothing" render the same.

(Minor and cosmetic: the served loop iterates insertion order while the observed loop is sorted(), so a mixed report's ordering depends on which direction each finding came from.)

Probe is ~30 lines and runs at 4b855201 (check_gui_gap in noirebox/reconcile.py, PR #81, still untagged — newest tag is v0.13.0). The idea it implements is the right one; what I measured is the resolution of its default.

[Reid Marlow]

The fail-silent typo behavior is a genuine bug. Raising on missing event types while silently returning an empty list on a bogus correlation key means a single character typo converts an unverified run into a green CI badge. That needs a check ensuring the key exists across the events, or a raise when zero events join on either side.

On the resolution trade-off, session_id was picked as the initial default solely to catch dead harnesses where the app crashed before event zero, without forcing callers to thread a click ID through custom wrappers on day one. But you are completely right about the membership blind spot. A session that drops nineteen out of twenty clicks after the first write should never show up as clean. I will update the PR to require per-action keys for ratio verification, flag count disparities, and make bad correlation keys fail loudly before this tags.

[howcani howcani]

reidmarlow — ran the three changes as code rather than as intention, on top of the shipped check at 4b855201, over the fixture family from my last comment. What follows is my reading of your sentence, not your PR — if I read a change wrong the correction is cheap.

Bad correlation key fails loudly: works on exactly the two rows that were silent. correlation_key="sess_id" raises, blank key raises. That is the one I would keep verbatim — it is the failure mode where a single typo ships as a green CI badge.

Count disparity, per key: closes both rows I flagged (20 served / 1 observed, 1 served / 3 observed). On per-action keys it also emits 19 disparity findings beside the 19 gui_served_unobserved — the same rows reported twice — and then this one, which is where I would slow it down:

one served click, three OS state paths touched, keyed per click → gui_count_disparity, and clean under the membership test. Those are the same events as my 1 served / 3 observed row above. Nothing there is a defect: an action that touches three paths is an action that was served. Equality asserts a 1:1 cardinality contract the pair does not have — the invariant is zero-vs-nonzero in each direction (served with no observation: your empty handler; observation with no served action: state nobody served), and both are rows the membership test already reports. A count flag only adds information once the expected multiplicity is declared somewhere — touches per action, or a tolerance — otherwise it turns "one click, three effects" into a finding.

So: per-action keys required (that is what actually closes the session-level window, and it does the work the count flag was reached for) plus the raise; and keep the counts as a number in the finding note rather than as a predicate, unless the flag has a declared multiplicity to compare against.