A measured study of multi-provider LLM consensus as a locator-healing signal — and why Automation Sandbox keeps the LLM out of the decision. Originally published on the project site.
TL;DR
When a UI refactor deletes an element, a self-healing engine must decline and ask a human — not latch onto a neighbouring button. I tested whether asking several independent LLMs and requiring them to agree can act as that "the element is gone" detector.
Across four live multi-provider runs (2026-08-16 to 2026-08-18, 133 usable scenarios):
- On elements that still existed (moved/relabelled), unanimous provider agreement was correct 52 / 52 times.
- On elements that had been deleted, unanimous agreement was wrong 34 / 34 times — every single time, a confident pick of the wrong neighbour.
- In 7 of those 34, three independently-sourced model families (Cloudflare/Qwen, Mistral, OpenRouter/gpt-oss) agreed on the same non-existent element at once.
- Widening the provider pool from 3 to 7 did not reduce the failure rate (25% → 45% → 58% → 40% agreement on deleted elements, no downward trend; 100% wrong throughout).
The consensus check is real protection — but it comes from providers disagreeing with each other, not from any model recognising that the element is gone. When they happen to agree on a deleted element, they are unanimously, confidently wrong. That is why in Automation Sandbox the LLM is an opt-in fallback gated by an independent-agreement quorum, and a heal is only ever committed after the retried action actually succeeds — the model is never the decision maker.
Full data, methodology, and the trade-off curves live in the Benchmark & Calibration guide (§3, §4, §6). This post is the standalone story; that page is the source of truth for every number.
1. The problem: a deleted element has no right answer
Locator healing handles the easy 90%: an AutomationId changes, a label is reworded, a control moves 100px in a layout pass. The engine re-resolves the element from structural evidence — control type, parent, sibling position, name similarity, geometry — and retries.
The dangerous case is the other kind of change: the element is deleted outright. A checkout button is removed; the test that clicked it should now fail loudly so a human looks at it. What it must not do is quietly heal onto the "Cancel" button next to where "Submit" used to be, pass green, and hide a real regression — a "false heal".
A pure-heuristic scorer cannot solve this case on its own, and I proved that rigorously: a surviving sibling in a deleted control's container is structurally indistinguishable from a control that genuinely moved next to that sibling. Their similarity-score distributions overlap ([0.665, 0.955] for deleted-element decoys vs [0.749, 0.874] for true compound drift), so no confidence threshold, runner-up margin, cluster-density, or control-type filter can draw a line between them.
That negative result is what motivated the LLM experiment. Semantic reasoning asks a different question than geometry does — maybe independent models, forced to agree, could tell "moved" from "gone".
2. Method: controlled multi-signal ablation
Natural locator drift across real releases is too sparse to benchmark. So I invert it: take a real, organically-built application UI tree and systematically mutate its authored locators.
- Source tree: HandBrake 1.8.2 (WPF), captured live — 149 nodes, 42 unique authored locators. A second application, ShareX v21.0.0 (WinForms), checks that findings are not HandBrake-specific.
- Five mutation tiers: pure rename (opaque ID), name/label drift, position shift, compound (text + layout), and element removal (whole subtree deleted).
- The two questions: for tiers 1–4 the engine should find the successor; for removal it should decline.
Leakage protection. Every candidate AutomationId in a mutated tree is rewritten to the same opaque format (ablation- + SHA-256 hex), not just the target. Otherwise a model could solve the scenario by spotting the "odd one out". IDs carry no semantic hint — which means these scores are a lower bound: in production, a descriptive btnSaveDocument gives the model more to work with than a synthetic hash does.
LLM evaluation protocol. Temperature 0. Every provider's raw vote recorded alongside the agreement telemetry. A scenario counts as usable only when ≥ 2 providers actually returned an answer — one opinion can neither agree nor disagree with anything. Token cost is managed by targeting the two informative subsets: 25 compound-drift + 42 removed-element scenarios.
3. The heuristic baseline (no LLM)
Running the deterministic scorer at the shipped default (MinimumConfidence = 0.50) over all 176 HandBrake scenarios:
| Mutation tier | n | Correct | False heals | Missed (review) | Score range |
|---|---|---|---|---|---|
| Renamed AutomationId | 42 | 40 | 0 | 2 | [1.000, 1.000] |
| Name drift | 25 | 23 | 0 | 2 | [0.877, 0.950] |
| Position shift | 42 | 34 | 0 | 8 | [0.832, 0.955] |
| Compound drift | 25 | 6 | 2 | 17 | [0.749, 0.874] |
| Removed element | 42 | 25 | 17 | 0 | [0.344, 0.955] |
40.5% of deleted elements are false-healed at the default threshold. On ShareX the same figure is 57.1%. Raising the threshold trades this away against auto-heal recall linearly — at 0.90 the false-heal rate on removed elements is still 5.5% while three-quarters of all locators are forced to manual review. There is no free setting.
This is the gap the LLM was asked to close.
4. The result: consensus separates the bands — then fails as a gate
Four runs, 2026-08-16 to 2026-08-18, as the provider pool grew from 3 to 7 and Groq's model was swapped twice:
| Run | Usable (n) | Compound-drift unanimous | Removed-element unanimous |
|---|---|---|---|
| 1 (08-16) | 19 | 6 / 7 | 3 / 12 |
| 2 (08-16) | 39 | 16 / 17 | 10 / 22 |
| 3 (08-16) | 33 | 14 / 14 | 11 / 19 |
| 4 (08-18) | 42 | 16 / 17 | 10 / 25 |
| Total | 133 | 52 / 55 | 34 / 78 |
Providers reached unanimous agreement on 94.5% of scenarios where a successor existed, and on 43.6% where the element was gone. That gap is real — it is the only signal in the entire project that separates the two populations at all.
But unanimity is not a safe acceptance gate:
Every one of the 52 unanimous verdicts on a surviving element was correct.
Every one of the 34 unanimous verdicts on a deleted element was a false heal.
Zero exceptions in either direction, across four runs with four different provider sets.
An earlier draft of this analysis reported "33%" wrong — that figure pooled both mutation types into one denominator and understated the real rate. Read per type, the deleted-element rate is 100%: agreement never once happened to be right about an absence.
5. Why: the mechanism is disagreement, not recognition
The hypothesis predicted two safe outcomes on a deleted element — providers decline, or providers scatter across different decoys. In practice:
- Declining essentially never happens. Across all 78 usable removed-element scenarios, "all providers declined" occurred exactly once. Every other provider that answered pointed at a specific — wrong — candidate.
- Every correct rejection came from providers contradicting each other, not from any model saying "that element is gone". The models do not know the control was deleted; each confidently names a different neighbour, and the engine rejects the heal only because the votes fail to match.
Widening the pool made this clearer, not better. One run recorded 10 unanimous false heals; 7 of those 10 had three independent model families agreeing on the same non-existent element — three vendors, three architectures, one wrong answer, unanimously. Going 3 → 7 providers moved the removed-element agreement rate 25% → 45% → 58% → 40% with no downward trend and 100% wrong throughout.
The protection is a byproduct of independence, and independence alone does not bound the failure rate: a genuinely capable, independent reasoner finds a deleted control's surviving neighbour a convincing answer often enough that adding more reasoners does not reliably break the tie. It also cannot be strengthened by asking for more confidence — the failing cases are already maximally confident.
6. What this means for the design
Automation Sandbox is built around this result rather than despite it:
| Design choice | Why |
|---|---|
| Heuristic-first, deterministic. A pure C# structural scorer decides on its own, ~23ms for 3,000 controls, zero tokens. | The LLM is never on the default path. Most healing never touches a model. |
| LLM is opt-in and quorum-gated (≥ 2 providers must independently name the same candidate). | A single model's confidence is worthless here. Agreement is permission to consider a pick — never evidence it is correct. |
| A heal commits only after the retried action succeeds; the shipped default mode changes no locators and only routes candidates to a report. | A wrong pick that cannot actually perform the test step is caught before it is persisted. |
| Every decision is written to an audit report (JSON + HTML) — which signal contributed what weight, which providers voted, what the outcome was. | "The AI healed it" is not an acceptable answer. You can see exactly why. |
The honest bottom line: unassisted absence detection is mathematically bounded by the structural score overlap. Automation Sandbox does not claim to have solved it. It makes the failure visible and declinable instead of silent and green.
7. Reproduce this yourself
From a clean clone (needs the .NET SDK; the LLM run needs provider API keys):
# 1. The deterministic heuristic baseline and threshold sweep - no keys, no tokens
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_RunsEndToEndAndReportsMetrics"
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_ThresholdSweep"
# 2. Calibrate against your own captured UI tree
dotnet run --project samples/CalibrationCli -- <your-tree.json> --app YourApp
# 3. The live multi-provider consensus evaluation (set provider keys as env vars first)
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_LlmConsensus_LiveEvaluation"
Every claim is guarded by a committed regression test — see the "Regression guards" lines throughout the Benchmark & Calibration guide.
8. What this study does not establish
- One dataset family. Two real applications, one mutation methodology. The direction is consistent across both; the magnitude is not portable (40.5% vs 57.1% false-heal on deleted elements at the same setting).
- Non-determinism band. A run over 42 removal scenarios carries roughly ±14% statistical uncertainty. The 34/34 result is striking precisely because it left no room for that band to matter, but a fifth run could see a unanimous verdict that is accidentally correct.
- Production ID fidelity. Ablation IDs are opaque hashes; real descriptive IDs would give models more signal. The LLM numbers here are a lower bound on production performance — which does not change the finding that agreement on a deleted element is unreliable.
- Prompting is not exhausted. This tests one prompt design (bounded top-N shortlist, temperature 0). It does not prove no prompt could do better — it proves that provider agreement, as a mechanism, does not carry the safety guarantee people assume it does.
Automation Sandbox is MIT, pure C#/.NET, seven packages on nuget.org. It does locator self-healing and intent-driven test generation for Windows desktop (FlaUI/UIA3) and web (Playwright). If you maintain UI tests that break on every refactor, I'd genuinely like your feedback — especially from anyone who has shipped locator self-healing and hit the deleted-element problem.
Top comments (0)