<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: 林乐扬</title>
    <description>The latest articles on DEV Community by 林乐扬 (@_7e6aad2e724239c05ce6f).</description>
    <link>https://dev.to/_7e6aad2e724239c05ce6f</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3460748%2Fcd954097-7938-450d-9a02-d906fc091a6d.jpg</url>
      <title>DEV Community: 林乐扬</title>
      <link>https://dev.to/_7e6aad2e724239c05ce6f</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/_7e6aad2e724239c05ce6f"/>
    <language>en</language>
    <item>
      <title>Running a screenshot-based automation agent on an Android phone</title>
      <dc:creator>林乐扬</dc:creator>
      <pubDate>Mon, 02 Nov 2026 13:18:17 +0000</pubDate>
      <link>https://dev.to/_7e6aad2e724239c05ce6f/running-a-screenshot-based-automation-agent-on-an-android-phone-51lb</link>
      <guid>https://dev.to/_7e6aad2e724239c05ce6f/running-a-screenshot-based-automation-agent-on-an-android-phone-51lb</guid>
      <description>&lt;p&gt;Tapilot is an open-source Android automation project with a specific constraint: the user should be able to describe and run a task on the phone, without a computer connected.&lt;/p&gt;

&lt;p&gt;Here is how the phone-side workflow fits together, and where its limits are.&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/TwuMuZ59yog" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h3&gt;
  
  
  A task starts in Android
&lt;/h3&gt;

&lt;p&gt;The native chat interface accepts a text task. Voice is another input path once a speech provider is configured. The embedded runtime runs the agent and keeps task context on the phone.&lt;/p&gt;

&lt;p&gt;Midscene interprets screenshots, locates visible targets and guides actions. Android’s AccessibilityService provides the device capabilities, including screenshots, gestures, navigation and text input. The next screenshot provides evidence of what changed.&lt;/p&gt;

&lt;p&gt;The distinction is between visual targeting and device execution. Screenshot-based targeting does not mean Android system interfaces disappear from the implementation.&lt;/p&gt;

&lt;h3&gt;
  
  
  The model is still remote
&lt;/h3&gt;

&lt;p&gt;The phone hosts the agent, but inference uses the configured model provider. Screenshots and task context are sent to that service. Voice recognition and playback use separately configured providers.&lt;/p&gt;

&lt;p&gt;This lets the phone run tasks without a connected computer. It does not make the workflow offline or remove the need to configure a model.&lt;/p&gt;

&lt;h3&gt;
  
  
  A report makes an attempt inspectable
&lt;/h3&gt;

&lt;p&gt;Midscene’s HTML report contains screenshots, plans, actions and errors. The demo shows a phone receiving a spoken instruction and opening WeChat, followed by a report view.&lt;/p&gt;

&lt;p&gt;The video is edited. It demonstrates the workflow, rather than establishing a success rate. Protected screens, custom-rendered inputs and vendor background restrictions can still interrupt execution.&lt;/p&gt;

&lt;h3&gt;
  
  
  A computer can be an optional controller
&lt;/h3&gt;

&lt;p&gt;Tapilot also includes a Skill and Codex plugin that connect to a paired, unlocked phone over Wi-Fi. In default steps mode, the host agent chooses actions, the CLI executes them, and MCP supplies the embedded preview panel.&lt;/p&gt;

&lt;p&gt;That is a separate entry point. Users do not need it to run the phone-side agent.&lt;/p&gt;

&lt;h3&gt;
  
  
  Trying it
&lt;/h3&gt;

&lt;p&gt;You need an ARM64 Android 11+ phone, accessibility permission and a supported vision model. Check the current installation instructions and the difference between the downloadable build and the latest source before starting.&lt;/p&gt;

&lt;p&gt;Demo and setup: &lt;a href="https://tapilot.lifeos.md" rel="noopener noreferrer"&gt;https://tapilot.lifeos.md&lt;/a&gt;&lt;br&gt;
Source: &lt;a href="https://github.com/tapilot-ai/tapilot" rel="noopener noreferrer"&gt;https://github.com/tapilot-ai/tapilot&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For a useful bug report, include the device, model and a small reproducible task. Check screenshots and logs for private information before sharing them.&lt;/p&gt;

</description>
      <category>android</category>
    </item>
  </channel>
</rss>
