Why Your Android AI Agent Works in Dev and Breaks in Production

There’s a gap between “it works on my device” and “it works on a user’s device” that Android accessibility-based agents fall into hard.

Here’s what it looks like, and what actually causes it.

The demo setup

In development, you’re running the agent on your own device. You’ve already granted BIND_ACCESSIBILITY_SERVICE. You know where the UI elements are. You’ve used the app enough that the accessibility tree is predictable.

Everything works.

What changes in production

1. Accessibility permission isn’t granted by default

The user has to go to Settings → Accessibility → Installed Apps → enable your service manually. There’s no programmatic shortcut. You can deep-link them close, but they have to flip the toggle themselves.

Most users won’t do this unless the value is immediately obvious. Your onboarding has to earn that permission before asking for it.

`fun isAccessibilityServiceEnabled(context: Context): Boolean {
    val service = "${context.packageName}/${YourAccessibilityService::class.java.canonicalName}"
    val enabledServices = Settings.Secure.getString(
        context.contentResolver,
        Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES
    ) ?: return false
    return enabledServices.contains(service)
}`

Check this on every resume. Users revoke it without telling you.

1. The accessibility tree varies by device and OEM

Samsung’s OneUI, Xiaomi’s MIUI, and stock Android expose the same apps differently in the accessibility tree. A node your agent finds reliably on a Pixel may not exist — or may have a different viewIdResourceName — on a Galaxy.

Don’t target by resource ID alone. Build a fallback chain:

`fun findSendButton(rootNode: AccessibilityNodeInfo): AccessibilityNodeInfo? {
    // Try resource ID first
    rootNode.findAccessibilityNodeInfosByViewId(
        "com.google.android.apps.messaging:id/send_message_button"
    ).firstOrNull()?.let { return it }

    // Fall back to content description
    rootNode.findAccessibilityNodeInfosByText("Send").firstOrNull()?.let { return it }

    // Fall back to class + position heuristic
    return findByClassAndBounds(rootNode, "android.widget.ImageButton")
}`

Test on at least three OEMs before calling it production-ready.

1. App updates break your agent silently

When WhatsApp or Gmail ships an update, resource IDs can change. Your agent doesn’t crash — it just can’t find the node and either fails silently or acts on the wrong element.

This is where the verification layer matters most. After every action:

`enum class ActionStatus {
    CONFIRMED,          // State verified — sent folder updated, etc.
    FAILED,             // State unchanged after action
    DENIED_UNRESOLVED   // State unverifiable — don't assume success
}`

DENIED_UNRESOLVED is your safety net for the cases you can’t see coming. An app update ships at 2am, a node ID changes, your agent thinks it sent a message. Without this, that’s a silent success. With it, it’s a flagged unresolved action the user can check.

The honest summary

iOS doesn’t expose this layer the same way — what works on Android either isn’t available or requires entitlements Apple doesn’t grant easily. Android gives you the power, but the production surface is messier than the demo.

Build the permission check, build the fallback chain, build the verification layer. In that order.

StareBrain is an on-device AI agent for Android built on exactly this stack. Pre-launch — waitlist on the product page.

Comments

[baumgaerben]

Thực tế gap lớn nhất thường không phải ở model mà ở runtime environment. Em thấy 3 điểm hay bị bỏ qua khi test trên dev device:

1. Battery optimization / App Standby Buckets — production user bật "Adaptive Battery", hệ thống ép process vào bucket restricted sau vài ngày không mở app. Inference queue bị delay 10-30s hoặc kill hẳn. Dev device thường whitelist app debug nên không thấy.

2. Thermal throttling — Pixel 8 Pro dev room 22°C chạy NPU full 15 phút không sao. User ngồi cafe 35°C, case dày, charge cùng lúc → CPU/GPU downclock sau 2-3 request, latency nhảy từ 800ms lên 4s. Cần hook PowerManager.addThermalStatusListener để fallback sang model nhỏ hơn hoặc offload server.

3. Model quantization mismatch — Build pipeline export TFLite int8 cho production nhưng dev test float32. Trên Snapdragon 8 Gen 3 diff <2%, nhưng trên MediaTek Dimensity 7000 (nhiều mid-range VN dùng) diff accuracy 8-12% do delegate NNAPI handle int8 khác. Phải test matrix: 5 chipset phổ biến × 3 Android version × 2 quantization.

Em thêm CI step: chạy benchmark tự động trên Firebase Test Lab (matrix device thật) mỗi PR, so sánh P95 latency + accuracy với baseline. Bắt regression ngay lúc merge, không đợi user report PS: the tool I meant is on labagent .tech