Building an Android AI Agent, Part 10: Handling Unexpected Screens Mid-Task

In Part 9 we built a flat task executor that walks a step list with a cursor, pauses for confirmation, and keeps state on the client.

The assumption baked into that executor: when the agent arrives at a step, the expected screen is there.

It often isn't.

A permission dialog appears. The app updated overnight and the UI changed. A loading spinner is still running. A previous step succeeded but triggered a side effect — an ad, a rate prompt, a "what's new" overlay.

The agent needs to handle this without asking the user every time, and without silently doing the wrong thing.

What "unexpected" actually means

There are three distinct cases:

1. Transient obstruction — a dialog, overlay, or spinner that wasn't part of the plan but can be dismissed or waited out. The underlying screen is still there.
2. Navigation drift — the app moved to a different screen than expected, either because a previous step had a side effect or because the app's flow changed.
3. Hard failure — the target element doesn't exist on this screen and there's no recoverable path forward without a new LLM decision.

Treating all three the same way is the mistake. A spinner should be waited out. A permission dialog should be handled inline. A hard failure needs a new plan.

Detecting the current screen state

data class ScreenSnapshot(
    val packageName: String?,
    val activityName: String?,
    val visibleTexts: List<String>,
    val interactableIds: List<String>
)

fun captureSnapshot(rootNode: AccessibilityNodeInfo?): ScreenSnapshot {
    val texts = mutableListOf<String>()
    val ids = mutableListOf<String>()
    fun traverse(node: AccessibilityNodeInfo?) {
        node ?: return
        node.text?.toString()?.let { texts.add(it) }
        node.viewIdResourceName?.let { ids.add(it) }
        for (i in 0 until node.childCount) traverse(node.getChild(i))
    }
    traverse(rootNode)
    return ScreenSnapshot(
        packageName = rootNode?.packageName?.toString(),
        activityName = null, // filled from AccessibilityEvent if available
        visibleTexts = texts,
        interactableIds = ids
    )
}

Before executing each step, the executor captures a snapshot and compares it against what the step expects.

The obstruction detector

object ObstructionDetector {
    private val DISMISS_TEXTS = setOf(
        "not now", "maybe later", "skip", "no thanks",
        "dismiss", "close", "cancel", "got it", "ok"
    )
    private val PERMISSION_TEXTS = setOf(
        "allow", "deny", "while using the app",
        "only this time", "don't allow"
    )
    private val LOADING_IDS = setOf(
        "android:id/progress", "progressBar", "loadingSpinner"
    )

    sealed class ObstructionType {
        object None : ObstructionType()
        object Loading : ObstructionType()
        data class Dismissible(val dismissText: String) : ObstructionType()
        data class Permission(val allowText: String) : ObstructionType()
        object Unknown : ObstructionType()
    }

    fun detect(snapshot: ScreenSnapshot): ObstructionType {
        if (snapshot.interactableIds.any { id -> LOADING_IDS.any { id.contains(it) } }) {
            return ObstructionType.Loading
        }
        val lower = snapshot.visibleTexts.map { it.lowercase() }
        val permission = PERMISSION_TEXTS.firstOrNull { p -> lower.any { it.contains(p) } }
        if (permission != null) {
            val allowText = snapshot.visibleTexts.firstOrNull { 
                it.lowercase().contains("allow") 
            } ?: "Allow"
            return ObstructionType.Permission(allowText)
        }
        val dismiss = DISMISS_TEXTS.firstOrNull { d -> lower.any { it.contains(d) } }
        if (dismiss != null) {
            val dismissText = snapshot.visibleTexts.firstOrNull { 
                it.lowercase().contains(dismiss) 
            } ?: dismiss
            return ObstructionType.Dismissible(dismissText)
        }
        return ObstructionType.None
    }
}

Handling obstructions inline

suspend fun handleObstruction(
    obstruction: ObstructionDetector.ObstructionType,
    service: StareBrainService
): Boolean {
    return when (obstruction) {
        is ObstructionDetector.ObstructionType.Loading -> {
            delay(1500)
            true // retry the step
        }
        is ObstructionDetector.ObstructionType.Dismissible -> {
            service.clickText(obstruction.dismissText)
            delay(300)
            true
        }
        is ObstructionDetector.ObstructionType.Permission -> {
            // Permissions are always user decisions — don't auto-allow
            false // escalate to confirmation flow
        }
        is ObstructionDetector.ObstructionType.Unknown -> {
            false // escalate to LLM
        }
        is ObstructionDetector.ObstructionType.None -> true
    }
}

Permissions are never auto-handled. The agent dismissing a "allow microphone access" dialog on behalf of the user is a trust violation, even if it's technically possible. It escalates to confirmation instead.

Plugging it into the executor

kotlin
suspend fun executeStep(step: AgentStep): StepResult {
    val snapshot = captureSnapshot(service.rootNode)
    val obstruction = ObstructionDetector.detect(snapshot)

    val canProceed = handleObstruction(obstruction, service)
    if (!canProceed) {
        return StepResult(
            status = StepStatus.FAILED,
            reason = "Unexpected screen: ${obstruction::class.simpleName}"
        )