DEV Community: Sharmin Sirajudeen

From Intent Classification to Open-Ended Action Spaces: Why Mobile Testing Needed a New Paradigm

Sharmin Sirajudeen — Mon, 06 Apr 2026 01:50:48 +0000

From Intent Classification to Open-Ended Action Spaces: Why Mobile Testing Needed a New Paradigm

I'm the creator of Drengr, an MCP server that gives AI agents eyes and hands on mobile devices. I started this blog to share the engineering behind it. No pretending to be a neutral observer writing a think piece — I built this, and I'm here to talk about it.

Google recently shipped AI Edge Gallery — an on-device AI sandbox app with a feature called "Mobile Actions" that lets you control your phone with natural language. Say "turn on the flashlight," and a 270M parameter model called FunctionGemma figures out the intent, extracts the parameters, and dispatches the right function call. It runs entirely offline. It clocks 1,916 tokens/sec prefill on a Pixel 7 Pro. And it's impressive.

But it also reveals a ceiling.

The Closed-World Assumption

FunctionGemma is, at its core, a tiny NLU engine performing intent classification and slot filling. You speak. It classifies your sentence into one of a fixed set of intents — turnOnFlashlight, createCalendarEvent, showLocationOnMap — and extracts the relevant slots: a time, a location, a contact name. The native app code then dispatches the structured output to the corresponding platform API.

This is a closed-world system. Every possible action is known at compile time. Every function is pre-registered. Every slot is pre-defined. The model's job is pattern matching over a bounded action space — the same fundamental design that Dialogflow, Alexa Skills, and SiriKit Intents have used for years, now running on-device at remarkable speed. These platforms have evolved over time — Apple's App Intents, Alexa's generative AI features — but the underlying intent-schema architecture remains fundamentally closed-world by design.

It works beautifully for what it is. But it cannot do what it has never been told exists.

The Open-World Problem

Now consider a different scenario. You're a QA engineer. You need to verify that a flower delivery app correctly applies a promo code at checkout, that the cart total updates, and that the confirmation screen renders the right order summary. The app was built by your team. No one pre-registered its UI elements as callable functions. No one fine-tuned a model on its screen taxonomy.

This is an open-world problem. The action space is unbounded. The UI is arbitrary. The screens have never been seen by the testing agent before.

This is the problem Drengr solves.

Text-First Perception, Schema-Never

Drengr is an MCP (Model Context Protocol) server — the open protocol that connects AI models to external tools and data sources, in the same way LSP (Language Server Protocol) connects editors to language servers. Drengr is purpose-built for mobile UI interaction. It doesn't require your app to expose an API. It doesn't need accessibility labels (though it uses them when available). It doesn't ask you to define intents or register functions.

Instead, it operates through three primitives:

drengr_look — Captures the current screen state as a compact text description (~300 tokens per screen) or an annotated image with numbered elements. Text-first by default — vision only escalates when less than 60% of elements have labels. 100x cheaper than sending screenshots every step.
drengr_do — Performs 13 actions on the device: tap, type, swipe, long press, back, home, launch, wait, key press, install, clear and type, scroll to top, scroll to bottom. Each action returns a situation report — a structured diff of what changed on screen (new elements, disappeared elements, crash detection, stuck detection).
drengr_query — Structured queries about device and app state: list connected devices, check current activity, detect crashes, find elements by text, explore app navigation, read network calls, check keyboard state, dump the raw UI tree, and more.

The AI client — Claude Desktop, Cursor, Windsurf, VS Code, any MCP-compatible host — acts as the brain. Drengr provides the eyes and hands. The agent looks at a screen it has never seen, understands what's there, decides what to do, and does it. No pre-training on your app. No test script maintenance. No brittle XPath selectors that break every sprint.

Why This Distinction Matters

The difference between closed-world function dispatch and open-world UI interaction is not incremental. It is architectural.

	Closed-World (FunctionGemma)	Open-World (Drengr)
Action space	Fixed, pre-defined functions	Arbitrary, discovered at runtime
UI knowledge	Compiled into the model	Observed per-screen via text scenes + vision fallback
New app support	Requires fine-tuning or function registration	Works immediately against any app
Failure mode	"I don't have a function for that"	"I can see the screen — let me figure it out"
Architecture	NLU → function dispatch	Perception → reasoning → action

FunctionGemma is a classifier. Drengr is an agent.

The MCP Advantage

Drengr is built as an MCP server — the same architectural pattern that made LSP the backbone of every modern code editor. Anthropic themselves draw this parallel in the MCP specification: both protocols solve the M×N integration problem. LSP connects M editors to N language servers. MCP connects M AI clients to N tool servers. Both use JSON-RPC 2.0 transport.

This means Drengr isn't married to a single LLM. Today, a developer can wire up Claude Code, Cursor, or Windsurf as the reasoning layer, and Drengr handles the device interaction. Tomorrow, when a better model drops, you swap the brain without touching the tools.

This separation of concerns — the model thinks, the server acts — is what makes the architecture durable.

Who This Is For

QA engineers tired of maintaining Appium scripts that break every release cycle.
Mobile developers who want to validate user flows without writing test code.
Engineering leads exploring agentic testing as a force multiplier for small teams.
AI tooling teams evaluating MCP-compatible infrastructure for mobile automation.

The Testing Problem, Reframed

Traditional mobile test automation asks: "How do I script a robot to press the right buttons?"

Drengr asks: "What if the robot could just look at the screen and figure it out?"

That reframing — from scripted automation to perceptual agency — is the paradigm shift. It's the difference between giving someone a map with every turn pre-marked, and giving them eyes and the ability to navigate.

Google proved that on-device NLU can dispatch to a handful of OS functions at blazing speed. Drengr proves that an LLM with the right tools can operate across any app, any screen, any flow — without ever being told what to expect.

Drengr is free to use and available on npm. It supports Android (physical devices, emulators), iOS simulators (full gesture support), and cloud device farms (BrowserStack, SauceLabs, AWS Device Farm, LambdaTest, Perfecto, Kobiton). Built in Rust. Single binary. No runtime dependencies.

Connecting Claude to a Real Phone via MCP

Sharmin Sirajudeen — Sun, 05 Apr 2026 03:01:57 +0000

I Gave Claude My Phone and It Tested My App

The setup: 90 seconds

I plugged an Android phone into my MacBook. Opened Claude Desktop. Added one line to the MCP config:

{
  "mcpServers": {
    "drengr": {
      "command": "drengr",
      "args": ["mcp"]
    }
  }
}

That's the entire setup. No Appium. No Selenium grid. No environment variables pointing to Java homes and Android SDK paths. Just npm install -g drengr, plug in the phone, and tell Claude what to do.

"Open YouTube and find a video about MCP servers"

I typed that into Claude. Here's what happened over the next 40 seconds:

Claude called drengr_look — got a text description of the home screen
It saw YouTube in the app list and called drengr_do to launch it
YouTube opened. Claude called drengr_look again — got the YouTube home feed as a list of labeled elements
It tapped the search bar, typed "MCP servers," and hit search
Results appeared. Claude read the titles and tapped the most relevant video
The video started playing. Claude confirmed: "Found and playing 'MCP Server Explained' by IBM Technology"

Six actions. No scripts. No selectors. Claude read the screen, made decisions, and executed actions — exactly like a human would, except it took 40 seconds instead of 2 minutes.

The moment it got interesting

I then asked: "Now go to Shorts and swipe through a few."

Claude navigated to the Shorts tab, swiped up three times, read the titles of each short, and told me what it saw. It handled the vertical scroll, the full-screen video player, the overlay buttons — all without any special configuration.

This is the kind of interaction that breaks traditional test frameworks. Shorts uses a custom renderer, the UI tree is minimal, the scroll behavior is non-standard. A selector-based test would need a custom handler for every quirk. Claude just... used the app.

What's actually happening under the hood

Claude doesn't see the phone directly. Drengr sits in between:

Claude → calls MCP tools → Drengr → talks to the device → Phone

Drengr handles the messy parts:

Capturing the screen and parsing the UI tree into a format the AI can read
Translating "tap element 3" into the right platform command
Reporting back what changed after every action (the situation report)
Detecting if the app crashed or the UI got stuck

Claude handles the smart parts:

Looking at the screen description and deciding what to do
Adapting when something unexpected happens
Knowing when the task is complete

This separation is deliberate. The AI is the brain, Drengr is the hands. When better AI models come out, Drengr doesn't need to change — the hands stay the same, the brain gets smarter.

The things that surprised me

It recovers from mistakes. At one point Claude tapped the wrong video. It noticed the title didn't match what it expected, pressed back, and picked the right one. No retry logic, no error handling code — the AI just adapted.

It works across apps. I asked Claude to "check my notifications" after the YouTube test. It pressed home, pulled down the notification shade, read the notifications, and summarized them. No app-specific setup needed.

Text mode is almost always enough. Out of ~30 actions across the session, Claude only needed the annotated screenshot twice — both times on screens with custom-rendered content. The rest worked with the ~300 token text description. That's 10x cheaper than sending images every step.

What it can't do (yet)

I'm not going to pretend this replaces manual testing today. Some limits are real:

Speed — Each step takes 2-3 seconds (LLM round-trip). A human tester can tap faster. But the human can't run 50 test flows in parallel on a device farm.
Visual verification — Claude can tell if an element exists, but not if it "looks right." Color, alignment, spacing — these need human eyes or a visual regression tool.
Complex gestures — Standard taps, swipes, long presses, and pinch zooms work. But game-specific multi-touch patterns aren't there yet.

The sweet spot today is regression testing: "does the checkout flow still work after this deploy?" That's the 80% of QA time that's spent running the same flows every sprint. Let the AI handle that, and let humans focus on exploratory testing.

Try it yourself

npm install -g drengr
drengr doctor          # check your setup
drengr setup --client claude-desktop  # generate MCP config

Connect a device, open your AI client, and tell it what to test. The first time an AI agent navigates your app without a single line of test code, you'll understand why I built this.

Your Mobile QA Team Is Still Writing XPath. In 2026.

Sharmin Sirajudeen — Sun, 05 Apr 2026 02:54:42 +0000

Your Mobile QA Team Is Still Writing XPath. In 2026.

The test that breaks every sprint

You know the drill. Your QA engineer writes a beautiful test suite. Login, browse catalog, add to cart, checkout. Fifty selectors, careful waits, retry logic for flaky network calls. It passes on Monday.

Tuesday, the design team moves the checkout button. Three selectors break. The test fails. The QA engineer spends half a day updating locators. The test passes again.

Wednesday, a new feature adds a bottom sheet that overlaps the cart icon. The tap lands on the sheet instead of the cart. The test fails. Another half day.

This cycle repeats every sprint, in every mobile team, everywhere. The test suite doesn't test the app anymore — it tests whether the selectors still match the UI.

The root cause: selectors were never the right abstraction

XPath, resource IDs, accessibility identifiers — they're all addresses. "Tap the element at this path in the view hierarchy." The moment the hierarchy changes, the address is wrong.

Humans don't navigate apps by address. They look at the screen, see "Checkout," and tap it. They don't care that the button moved from //android.widget.Button[@resource-id='checkout_btn'] to //android.widget.FrameLayout/android.widget.Button[2]. They just see the button and tap it.

AI agents can do the same thing — if you give them the screen, not a selector tree.

What "giving them the screen" looks like

When an AI agent connects to Drengr, it asks: "What's on screen?" Drengr responds with either a compact text description:

[1] "Checkout" (Button)
[2] "Your Cart: 3 items" (TextView)
[3] "Remove" (Button)
[4] "Continue Shopping" (Button)

Or an annotated image with numbered elements. The AI reads this, decides "tap element 1," and calls drengr_do. After the action, it gets a situation report telling it what changed.

No selectors. No XPath. No element IDs to maintain. The AI sees the screen the way a human does — by what's visible, not by where it lives in the view tree.

"But what about reliability?"

Fair question. If the AI is interpreting the screen every time, doesn't that introduce non-determinism?

Yes. And that's the point. A deterministic test that breaks when the UI changes isn't reliable — it's rigid. An AI agent that adapts to UI changes is more reliable in practice because it handles the variations that break selector-based tests.

Drengr adds guardrails:

Stuck detection — if the screen doesn't change after an action, the agent knows to try something else
Crash detection — if the app dies, the agent knows immediately and can restart
Situation reports — after every action, the agent gets a structured diff of what changed, so it stays oriented

The AI doesn't just blindly tap. It observes, acts, and adapts. That's more robust than a fixed script that works exactly one way.

The cost argument

"AI calls are expensive." Sure, if you're sending screenshots to GPT-4o on every step.

Drengr's text-only mode compresses a screen to ~300 tokens. A 15-step test flow costs about $0.05 on GPT-4o pricing. The same flow with screenshots costs $0.45.

But here's the real cost comparison: how much does your QA team spend maintaining selectors? If one engineer spends 2 hours a week updating broken tests, that's $5,000/month in salary going to XPath maintenance. The AI API costs are rounding errors next to that.

A test suite that survives redesigns

app: com.example.shop
tasks:
  - name: browse
    task: "Search for wireless earbuds and open the first result"
    timeout: 60s
  - name: purchase
    task: "Add to cart and complete checkout with the test card"
    timeout: 90s
  - name: verify
    task: "Go to order history and verify the order appears"
    timeout: 45s

This YAML survived 3 redesigns of our test app. The checkout flow moved from a separate page to a bottom sheet to a full-screen modal. The YAML didn't change. The AI adapted every time because it reads the screen, not the selectors.

drengr test tests.yml runs it. JUnit XML output plugs into any CI pipeline. No Appium server to maintain, no Selenium grid, no element locator spreadsheet.

This isn't theoretical

Drengr runs on real Android phones, iOS simulators, and cloud device farms (BrowserStack, SauceLabs, AWS Device Farm, LambdaTest, Perfecto, Kobiton). It connects to any MCP-compatible AI client — Claude Desktop, Cursor, Windsurf, VS Code.

One binary. One install:

npm install -g drengr

Your QA team can stop writing XPath. The AI can read the screen.

AI Can Browse the Web. Why Can't It Tap a Phone?

Sharmin Sirajudeen — Sun, 05 Apr 2026 02:54:30 +0000

AI Can Browse the Web. Why Can't It Tap a Phone?

The gap nobody talks about

Every week there's a new "Show HN" for AI-powered browser testing. Playwright agents, Puppeteer bots, Chrome extensions that turn the DOM into JSON for LLMs. The web automation space is overflowing with AI-native tools.

Then someone asks: "How do I do this on a phone?"

Silence.

The best answer the industry has is Appium — a tool from 2013 that requires you to set up a Selenium grid, write XPath selectors, and maintain brittle element locators that break every time a designer moves a button. Or Espresso/XCTest, which require you to embed test code inside the app itself.

None of these are AI-native. They were built for humans to write scripts, not for LLMs to reason about screens.

Why mobile is harder than web

The web has one universal API: the DOM. Every browser exposes the same tree of elements with the same attributes. Playwright reads the DOM, the AI decides what to click, done.

Mobile doesn't have that. Android has uiautomator. iOS has its own accessibility framework. They return different structures, different attributes, different coordinate systems. Cloud device farms add another layer — now you're talking to a device over Appium WebDriver, which adds its own abstraction on top.

The result: every mobile testing tool is platform-specific, setup-heavy, and hostile to AI agents that just want to know "what's on screen?" and "tap that button."

What I built instead

Drengr is a single Rust binary that sits between the AI and the device. It exposes exactly 3 tools over the Model Context Protocol (MCP):

drengr_look — tells the AI what's on screen, either as an annotated image or a compact text description (~300 tokens instead of a 200KB screenshot)
drengr_do — executes an action (tap, type, swipe, long press, scroll, launch app, etc.) and reports back what changed
drengr_query — answers questions without touching the screen (is the app crashed? what HTTP calls happened? what's the current activity?)

The AI client — Claude Desktop, Cursor, VS Code, whatever — is the brain. It decides strategy. Drengr is the hands. It handles the platform mess so the AI never has to think about ADB vs simctl vs Appium.

The thing that makes it work: situation reports

After every action, Drengr doesn't just say "ok, done." It tells the AI what changed:

{
  "screen_changed": true,
  "new_elements": [12, 15],
  "disappeared_elements": [7],
  "activity_changed": true,
  "crash": false,
  "stuck": false
}

The AI reads this and immediately knows: the screen updated, two new elements appeared, one vanished, we navigated somewhere new, and the app is still alive. No need to take another screenshot and visually diff it.

This is what browser testing tools don't need — the DOM gives you change events for free. On mobile, you have to build this layer yourself. I spent months on it so you don't have to.

A real test looks like this

app: com.example.app
tasks:
  - name: login
    task: "Log in with user@test.com and password123"
    timeout: 60s
  - name: checkout
    task: "Add headphones to cart and complete purchase"
    timeout: 90s

That's it. No selectors. No XPath. No element IDs to maintain. The AI reads the screen, decides what to do, and Drengr executes it. When the UI changes, the YAML doesn't break — because there's nothing brittle in it.

Run with drengr test tests.yml and get human-readable output, JSON, or JUnit XML for your CI pipeline.

Why I think this gap exists

Browser testing got AI-native tools early because the DOM is an open, text-friendly format that LLMs can reason about directly. Mobile UIs are visual, proprietary, and locked behind platform-specific APIs that nobody unified.

The MCP protocol changes this. It gives AI agents a standard way to connect to tools — and Drengr is the tool that bridges MCP to mobile devices. Android, iOS, simulators, cloud farms — one interface, one binary, one install:

npm install -g drengr

The web got its AI testing moment. Mobile's turn is now.