Image: AI-generated illustration
How could Shelly send a LINE message or publish an X post by operating another Android app? Its app.act implementation used Accessibility Service recipes, but cold-start timing, an ambiguous-match safety trip, and a focus-stealing capture UI led to complete removal on 2026-08-31. Which execution conditions mattered, what can developers check when similar symptoms appear, and how do API-based posting approaches change the operation and its requirements?
The failure was in operating the target screen
During on-device testing, three distinct bugs surfaced in a single day: cold-start timing, an ambiguous-match safety trip, and a focus-stealing capture UI. The removal decision also identified a compile-time two-app allowlist and breakage from target-app screen changes as constraints of the approach.
The native execution path had used resourceId-matched AccessibilityNodeInfo actions without gesture dispatch. Its scope had been LINE+X, with canRetrieveWindowContent only and the bundled recipes line.send-message and x.post.
Android distinguishes the capability to inspect a screen from the capability to dispatch gestures: canRetrieveWindowContent must be true to inspect the UI hierarchy, while canPerformGestures must be true to dispatch gestures programmatically. For an individual node, the documented action API is performAction(); system-wide actions, such as pressing Back or opening the notification shade, use performGlobalAction(). [1]
The 2026-08-31 decision was to remove the UI-automation approach rather than continue hardening it against changes in target-app screens. The resulting change covered the observation, recipe-execution, and dispatch parts of the same action path.
| Failure-chain stage | Before removal | After removal on 2026-08-31 |
|---|---|---|
| Observe the target app | ShellyAccessibilityService supplied the package-scoped accessibility layer. | The accessibility service was removed. |
| Load and execute a recipe | AppActRecipeStore.kt and AppActExecutor.kt supplied recipe loading and execution. | The recipe store, Kotlin executor, and bundled recipes were removed. |
| Dispatch the action | The app-act action type and associated fields connected the agent action to the generated script. | The action type, fields, and generated-script path were removed. |
A diagnostic checklist: startup, matching, capture
Check whether service initialization has completed
For a cold-start symptom, has service-dependent initialization reached onServiceConnected() before the action begins?
Android’s guide places initialization in onServiceConnected(). [1] The app.act-specific observation was a cold-start timing bug during on-device testing.
The guide also places runtime configuration updates through setServiceInfo() in that callback. XML configuration defines the events a service handles and the feedback it provides, while setServiceInfo() allows configuration changes at runtime. [1]
A related check is whether the event being handled matches the configured event types: onAccessibilityEvent() is called when the system detects a matching event, such as a focus change or button click. The service must also be declared in the manifest with BIND_ACCESSIBILITY_SERVICE enforced so that only the system can bind to it. [1]
Check the selected node on the target screen
For a matching symptom, does the recipe’s resourceId select the intended AccessibilityNodeInfo on the screen being operated?
The app.act matching failure had been an ambiguous-match safety trip in its resourceId-based execution path. Target-app screen changes had also been identified as a source of breakage in the removal decision.
Android represents UI elements such as buttons, lists, and text as AccessibilityNodeInfo objects in an accessibility tree. An AccessibilityEvent provides an entry point to that tree, and the guide’s traversal example checks whether the event’s source node is null before inspecting it. [1]
For a node action, another concrete check is whether the node supports the intended interaction: the guide’s click example checks isClickable before calling performAction(AccessibilityNodeInfo.ACTION_CLICK). [1]
Check the foreground package before capture
When a screen snapshot is requested, is the foreground package still inside the intended allowlist?
The 2026-08-29 captureScreenSnapshot() implementation had explicitly rechecked the foreground package against the allowlist, in addition to the operating system’s packageNames boundary. That authoring work had retained the LINE/X scope.
Before that flow, adding a recipe beyond the bundled pair had meant hand-writing recipe JSON after reading a logcat node dump. The new snapshot method had produced a read-only structured JSON dump with depth/node-count caps.
Those limits belonged to the observation path introduced for recipe authoring, alongside recipe drafts and user saving. AppActRecipeStore.load() had also gained lookup for user.-prefixed recipe IDs.
Check focus ownership and window availability
Does opening the capture interface change where keyboard input goes or what the accessibility service is inspecting?
The subsequent on-device failure had been a capture UI that stole focus. Android distinguishes input focus, where keyboard input goes, from accessibility focus, which identifies what the service is inspecting. [1]
For a capture symptom, check whether rootInActiveWindow is available before looking for the focused node: the official guide allows it to be null when the window is unavailable. Its focus example uses findFocus(AccessibilityNodeInfo.FOCUS_ACCESSIBILITY) to locate the node with accessibility focus. [1]
Android’s guidance is to respect the user’s focus state and avoid stealing focus unless explicitly triggered by a user action. [1]
Check the approval condition separately from the recipe
If the action is requested unattended, which execution gate applies?
At the Phase 4 checkpoint on 2026-07-14, app-act dispatch had refused unattended execution, matching intent/dm-reply’s interim shape. The later target design had instead treated a fixed recipe and target as consented once at registration, with ongoing Sidebar visibility.
That registration design had made app-act a deliberate exception to the cli/intent/dm-reply “never one-tap” pattern and had omitted runtime target resolution of the kind used by intent/dm-reply. Registration-time consent and the implemented unattended-execution gate had therefore represented different design stages.
For parameters shown during approval, check which values reach the preview: resolve_app_act_params() had resolved {{result}} against an already-redacted preview and then applied a second redact_secrets_text pass. Approval requests had carried appActRecipeId and appActParamsResolved.
The native path had also included LockPromptActivity.kt for a lock-screen wake-and-unlock challenge, using an implementation in which 2 real concurrency bugs had already been resolved.
Implementation milestones
- 2026-07-14 — Commit 86862ae3d added appActRecipeId, appActParams, and appActMethod as Phase 2 schema groundwork, with appActParams covered by both textForSecretScan copies. Dispatch had not yet read those fields, so an app-act agent had failed closed through toPlanAction’s default unsupported branch.
- 2026-07-14 — Commit bde82f4e6 introduced the native AccessibilityService layer as Phase 3.
- 2026-07-14 — Commit e0ed1217c connected shell dispatch, PlanSpec, and the RN bridge end to end, changing AGENT_SCRIPT_VERSION from 10->11 in lockstep with AgentRuntime.kt.
- 2026-08-29 — Commit 1e42490ae introduced the observe-only snapshot, recipe-draft, and user-save flow described above.
- 2026-08-31 — Commit 325e9d702 implemented the complete removal shown in the table.
Other approaches: posting operation and remaining requirements
For X, POST /2/tweets creates Posts on behalf of authenticated users, making the posting operation an API request rather than a recipe operating a target screen. The authentication requirement is a User Access Token obtained through OAuth 2.0 PKCE or 3-legged OAuth. [2]
For Bluesky, the posting operation uses an app.bsky.feed.post record with required text and createdAt fields. Image posting adds a separate com.atproto.repo.uploadBlob step before the uploaded files are referenced in the record. [3]
Sources
- [1] Create an accessibility service
- [2] Manage Posts on the X API v2
- [3] Posts
- The author's own notes
AI disclosure: this post was researched, written and reviewed by AI tools.
Publishing: a post that passes automated checks may be published automatically, before the author reads it.

Top comments (1)
For the LINE/X case specifically, there's a middle ground between UI automation and the posting APIs: an explicit ACTION_SEND intent with setPackage() set to the target app. You pass the text (and images as content:// URIs with FLAG_GRANT_READ_URI_PERMISSION), the app opens its own composer prefilled, and the user taps send. Nothing depends on resourceIds, so target-app redesigns don't break it, and since Play's Accessibility API policy is strict about services acting inside other apps, dropping that path probably saved a review headache as well. Did you look at share intents before moving to the APIs?