How to Give an AI Agent Full Control of a Real iOS or Android Device (over MCP)

krish pavuluri — Wed, 05 Aug 2026 17:44:07 +0000

AI agents are getting good at writing code, browsing the web, and calling APIs. But there's one surface most of them still can't touch: a real phone. Not a simulator, not a screenshot fed back to the model — an actual iOS or Android device, driven with the fidelity of a human's fingers.

This guide shows you how to give an agent exactly that, over the Model Context Protocol (MCP). By the end, your MCP client — Claude Desktop, Cursor, Cline, or your own — will be able to open a real device, tap and type on it, see what the app is doing, capture and mock network traffic, spoof GPS, and even explore an app and generate test scripts. All of it as MCP tools the agent can call directly.

What you'll be able to do

Once connected, the agent has the same reach a human tester has on a real device:

Control — real taps, press-and-hold, swipes, drag, and typing (native, hardware-level input, not a laggy screen-share overlay)
See — live screenshots, on-device web inspection, the UI element tree / DOM
Inspect the network — capture every request the app makes, and mock any request or response to force error states and edge cases
Simulate — set GPS location anywhere on earth
Automate — explore an app, find issues, and generate runnable test scripts

Prerequisites

An MCP-capable client (Claude Desktop, Cursor, Cline, or any client that speaks MCP)
A free RobotActions account (that's what exposes the real devices as MCP tools)

Step 1 — Create a free account

That's all the setup you need: you'll authorize your MCP client in the next step and it fetches your access automatically, so there's no token to create, copy, or paste.

Step 2 — Connect your MCP client

You have two paths. Pick the one that matches your client.

Option A — Claude.ai / Claude Desktop (remote connector)

Add RobotActions as a remote MCP connector pointing at:

https://mcp.robotactions.com/mcp

Then click Connect and authorize. Signing in fetches your access automatically — no manual token entry.

Option B — Cursor, Cline, or a local MCP config

Use the connector package to wire it into your MCP config:

npx @robotactions/mcp init

Or point your client's MCP config at the same endpoint https://mcp.robotactions.com/mcp, then authorize the same way when prompted. (RobotActions is also listed on the official MCP registry, Smithery, and glama if your client installs from there.)

Reload your client and you should see the RobotActions tools appear.

Step 3 — Your first commands on a real device

Now just talk to your agent. Try:

"List the available devices, open an Android one, and take a screenshot."

The agent will call the device-list and screenshot tools and hand you back a real frame from a real phone. From there:

"Open the Settings app, go to Wi-Fi, and turn it off."

The agent taps and swipes on the actual device — real touches, not a simulated overlay.

Step 4 — The part agents couldn't do before

This is where it gets interesting. Because the agent has full access — visual and programmatic — you can ask it to do things a screen-share bot never could:

Inspect and mock the network:

"Capture the network calls this app makes on launch, then mock the /profile response to return an empty state, and show me how the app renders it."

Force a location:

"Set the device's GPS to Tokyo and check that the store locator updates."

Explore and test:

"Walk through the checkout flow, flag anything that looks broken, and generate a test script that reproduces it."

The agent sees the screen, reads the network and the UI elements, and drives the input — so "test this app" becomes a single instruction instead of a week of setup.

Why this matters

Mobile test infrastructure was built for humans staring at a remote screen. As agents start doing real work, they need to operate real devices with human-level fidelity — and reach everything programmatically, not just the pixels. Exposing real-device control as first-class MCP tools is what turns "an agent that can see a phone" into "an agent that can actually use one."

Try it

You can do everything above right now on a live demo device — no cables, no local setup, no simulators.

Sign in with Google or GitHub and try it free →

How We Built a Chat AI Agent Into Live Device Testing Sessions

krish pavuluri — Tue, 10 Mar 2026 20:40:36 +0000

We ship a cloud device farm — real Android and iOS devices you can control from a browser. Our users are mostly SDETs and QA engineers running Appium tests.

The problem we kept hearing: finding the right locator wastes too much time.

The typical workflow looks like this:

Open a device session
Notice an element on screen
Switch to Appium Inspector
Inspect the element tree
Copy the locator
Paste it into your test
Run the test, fail, go back to step 3

We wanted to collapse that loop. So we built a Chat AI Agent that lives inside the device session.

What It Does

The agent can see the live device screen. You can ask it in plain English:

"What's the XPath for the equals button?"
"Give me a UIAutomator2 selector for the digit 7"
"What's the Accessibility ID of the login button?"

And it responds instantly with working locators — in whatever language you're using (Java, Python, Swift, Kotlin, WebDriverIO).

No switching tools. No Appium Inspector. Just ask.

How We Built It

Screen visibility

Our sessions already stream device screens via WebRTC. We grab frames from the stream at the point of the user's question — a single screenshot at query time. This keeps latency low and avoids sending a continuous video feed to the model.

The model

We send the screenshot + user message to a vision-capable LLM. The prompt is structured to return locators in a specific format — we parse the response and render it with syntax highlighting in the UI.

Locator formats

We support:

XPath
CSS Selector
UIAutomator2 (Android)
XCUITest (iOS)
Accessibility ID

The model is instructed to return all applicable formats for the visible element, not just one.

Code output

Users pick their language from a dropdown (Java, Python, Swift, Kotlin, WebDriverIO). We wrap the locator in idiomatic framework code for each:

# Python / Appium
driver.find_element(AppiumBy.XPATH, "//android.widget.Button[@content-desc='equals']")

// Java / Appium
driver.findElement(By.xpath("//android.widget.Button[@content-desc='equals']"));

UI integration

The panel sits alongside the device stream — it doesn't overlay the screen. Users can keep testing while asking questions. The conversation history stays within the session.

What We Learned

The hardest part wasn't the AI integration — it was the prompt engineering. Getting the model to return clean, parseable locator output (not prose with embedded code) required iteration.

We also found that grounding the model on the visible screen state (not a DOM or accessibility tree) made responses feel more natural. Users think in terms of what they see, not what's in the XML hierarchy.

Try It

The Chat AI Agent is live now in the RobotActions portal. Free trial available.

We'd love feedback from anyone doing Appium or mobile automation — especially if you've built similar tooling. Drop a comment or reach out directly.

DEV Community: krish pavuluri