chinmayajha

Posted on Nov 9 Originally published at optics-framework.org

Self-heal broken locators with a 2B model on your own GPU

#ai #testing #python #llm

When a step cannot find its element, Optics Framework can ask an AI model to find it. This is AI self-heal. Optics Framework sends the failed locator, the rows of the screen and a screenshot to the model. The model selects the correct row. The model can run on any OpenAI-compatible server. You can use a small open model on your own GPU.

I used Qwen3.5-2B, a model with 2 billion parameters. I tried two ways to teach it this job. The first way was a direct edit of the weights, with no gradients. The second way was a small LoRA. Then I connected the LoRA model to Optics Framework 1.11.3 and ran a suite on a Pixel 4a.

The job of the model

This is part of a self-heal request from the phone run. I broke the locator of the Start button in the Google Clock app:

TARGET ELEMENT (not found by normal locators): //*[@resource-id="com.google.android.deskclock:id/primary_action"]
TARGET ELEMENT NAME (from the test suite): start_button
CURRENT SCREEN:
e3 android.widget.TextView "00:00.00" #com.google.android.deskclock:id/stopwatch_time_text
e4 android.widget.Button "Start" [tap] #com.google.android.deskclock:id/start_stop_button
e5 android.widget.FrameLayout "Alarms" [tap] #com.google.android.deskclock:id/tab_menu_alarm

The model replied:

{"reason": "The target is the 'Start' button. Row e4 matches the target name 'Start' and has the #id 'start_stop_button'.",
 "action": "keyword", "keyword": "press_element", "target": "e4", "params": [], "completed": true}

The locator primary_action has no word in common with "Start". The element name from the suite, start_button, gives the model the link. Optics Framework sends this name with each self-heal request. Optics Framework also adds 3 hint rows. These are the rows that have the most words in common with the locator.

I measured two types of broken locators. A text locator repeats the text of the button, for example id/see_1_tab for "See 1 tab". A purpose locator gives only the purpose of the button, for example id/skip_notification_button for "Not now". The locator primary_action above is a purpose locator. The model makes more errors on purpose locators.

Two ways to teach the skill

My first method came from weight-space programming (WSP). With WSP, an agent wrote facts into Qwen 0.8B in closed form. It changed 5 of 488 tensors and used no gradients. A fact is one link, for example "the capital of Japan" to "Tokyo". I wanted to know if the same method can add a skill.

My best edit used the difference between two runs of the model, with a hint and without the hint. The hint tells the model to select the row whose text matches the target name. I added this difference as a rank-1 update to the attention output projection in 5 of 24 layers. Each matrix changed by 0.6% to 1.7%.

The second method was a LoRA. The LoRA adds 0.7% more weights to the model. I trained it for 1 epoch on 2,628 examples.

Each score comes from recorded screens of 96 Android and iOS apps, with 4-fold cross-validation by app. In each fold, the model did not train on the apps that I scored.

Setup	Text locators	Purpose locators
Qwen3.5-2B with the Optics Framework self-heal prompt	95.5%	77.9%
The same, with the LoRA	98.6%	84.9%

The weight edit added 1.6 points on text locators, from 86.3% to 87.9%. This prompt had no hint rows. On a separate set of 377 steps, the edit got 58% of the gain of the hint. A stronger edit broke correct answers at the same rate as it repaired incorrect answers.

The LoRA added 3.1 points on text locators and 7.0 points on purpose locators. It got these scores on apps that it did not see when I trained it. I used only one model and one task.

On a phone

I served the LoRA model with vLLM and connected Optics Framework 1.11.3 to it. The suite operates Google Clock on a Pixel 4a. It has 2 test cases, with an assert after each press. I broke the locators in 7 ways and ran each break set 2 times. With strict_element_match: true, Optics Framework does not use its fuzzy match. Each broken locator goes to the model.

A heal from the purpose set. The broken locator is id/primary_action. The model selected Start, and the next assert passed.

Break sets	Correct heals	Test cases passed
Wrong ID (4 sets: old ID, camelCase, package, class)	64 of 64	16 of 16
Wrong words (3 sets: new label, purpose, purpose with correct tabs)	49 of 50	11 of 12

The model healed 109 of the 114 locators in 1 step and 5 in 2 steps. 1 of the 2-step heals was incorrect. The log of the purpose set shows each heal and the reason of the model:

AI self-heal: attempting recovery for 'press_element' on '//*[@content-desc="Chronometer"]'
AI self-heal step 1/5 for 'press_element': ... reason=The target is 'stopwatch_tab'. Row e13 "Stopwatch" matches the target name exactly.
AI self-heal: attempting recovery for 'press_element' on '//*[@resource-id="com.google.android.deskclock:id/halt"]'
AI self-heal step 1/5 for 'press_element': ... reason=The target is 'stop_button'. Row e10 "Stop" matches the target name exactly.

Use the model with Optics Framework

The adapter (60 MB) and the scripts are in the qwen-self-heal repository. You need a GPU with 16 GB of memory or more for the server.

Download Qwen/Qwen3.5-2B from Hugging Face.
Download the adapter from the release of the repository.
Merge the adapter into the base model with scripts/merge_adapter.py.
Start the server with vllm serve. Use the flags in the repository README.
Add the server to config.yaml:

ai_self_heal: true
llm_models:
  - openai:
      enabled: true
      url: http://<gpu-host>:8000/v1
      capabilities:
        model: qwen-heal
        response_format: json_object
        temperature: 0.0
        max_tokens: 400

Run your suite with optics execute.

The value of model must be the same as --served-model-name in the server command. vLLM asks for a key only when you start it with --api-key. For a server with a key, add api_key: ${QWEN_API_KEY} under capabilities. Optics Framework reads the variable when it loads the file. The logs show **** in place of the key.

Tips

Use the settings from the measurements: a screenshot of 1 megapixel maximum, enable_thinking: false, temperature 0 and json_object. The vLLM flags in the README set the first two.
Give your elements descriptive names in the suite. The model uses the name when the locator has no word in common with the screen.
Keep an assert after each press. The assert confirms each heal on the next screen.
After a run, read the self-heal lines in the log.
Update the locators that healed.

The full results, the edit methods that did not work and the phone report are on the results page. For the self-heal settings, read Recover failing steps with AI self-heal.

Source: github.com/mozarkai/optics-framework
Model and scripts: github.com/chinmayajha/qwen-self-heal

I work on Optics Framework. This post first appeared on the Optics Framework blog.

DEV Community