Building an Android AI Agent, Part 9: Multi-Step Task Execution Without a State Machine

In Part 8 we added context-aware confirmation — the agent now knows whether it's operating inside a banking app or a notes app and adjusts its confirmation threshold accordingly.

Now the harder problem: what happens when a task requires more than one action across more than one app?

"Send the invoice to the client I just messaged, then mark the project as done in my task app."

That's three steps, two apps, one instruction. The agent needs to execute them in order, recover if one fails, and not lose track of where it is.

The temptation: a state machine

The obvious solution is a state machine. Define states, define transitions, track current state, handle errors per state.

We didn't do that. Here's why.

A state machine works well when the set of possible states is known at design time. An AI agent operating across arbitrary apps on an arbitrary device doesn't have that property. The apps change. The user's workflow changes. A rigid state machine becomes a maintenance problem the moment a new app enters the picture.

Instead, we use a flat task list with a cursor.

The data model

data class AgentTask(
    val id: String = UUID.randomUUID().toString(),
    val description: String,
    val steps: List<AgentStep>,
    val currentStepIndex: Int = 0,
    val status: TaskStatus = TaskStatus.PENDING
)

data class AgentStep(
    val id: String = UUID.randomUUID().toString(),
    val instruction: String,
    val targetPackage: String?,
    val status: StepStatus = StepStatus.PENDING,
    val result: StepResult? = null
)

enum class TaskStatus { PENDING, IN_PROGRESS, COMPLETED, FAILED, AWAITING_CONFIRMATION }
enum class StepStatus { PENDING, IN_PROGRESS, COMPLETED, FAILED, SKIPPED }

The AgentTask holds the full list of steps. currentStepIndex is the cursor. The LLM generates the step list upfront; the executor walks it.

The executor

class TaskExecutor(
    private val accessibilityService: StareBrainService,
    private val confirmationGuard: ConfirmationGuard,
    private val scope: CoroutineScope
) {
    suspend fun execute(task: AgentTask): AgentTask {
        var current = task.copy(status = TaskStatus.IN_PROGRESS)

        for (i in current.currentStepIndex until current.steps.size) {
            val step = current.steps[i]

            if (confirmationGuard.shouldConfirm(
                    step.instruction,
                    null,
                    null,
                    step.targetPackage
                )
            ) {
                return current.copy(
                    currentStepIndex = i,
                    status = TaskStatus.AWAITING_CONFIRMATION
                )
            }

            val result = executeStep(step)
            val updatedStep = step.copy(status = result.status, result = result)
            val updatedSteps = current.steps.toMutableList()
            updatedSteps[i] = updatedStep
            current = current.copy(
                steps = updatedSteps,
                currentStepIndex = i + 1
            )

            if (result.status == StepStatus.FAILED) {
                return current.copy(status = TaskStatus.FAILED)
            }
        }

        return current.copy(status = TaskStatus.COMPLETED)
    }

    private suspend fun executeStep(step: AgentStep): StepResult {
        // Dispatches to accessibility service actions
        return accessibilityService.performAction(step)
    }
}

Two things worth noting:

1. Confirmation pauses mid-task. If step 2 of 4 requires confirmation, the executor returns with AWAITING_CONFIRMATION and the current index. The UI shows the confirmation prompt. When the user approves, execution resumes from where it stopped — no replay from the beginning.

2. Failure stops the chain. If a step fails, the task halts and the UI surfaces which step failed and why. The user can retry that step or abandon the task. We don't silently skip failures.

Resuming after confirmation

suspend fun resume(task: AgentTask, approved: Boolean): AgentTask {
    if (!approved) {
        return task.copy(status = TaskStatus.FAILED)
    }
    // Skip the confirmation check for the step we just approved
    val step = task.steps[task.currentStepIndex]
    val result = executeStep(step)
    val updatedStep = step.copy(status = result.status, result = result)
    val updatedSteps = task.steps.toMutableList()
    updatedSteps[task.currentStepIndex] = updatedStep

    val resumed = task.copy(
        steps = updatedSteps,
        currentStepIndex = task.currentStepIndex + 1,
        status = TaskStatus.IN_PROGRESS
    )
    return execute(resumed)
}

The resume function re-enters the executor from the current index with the confirmation check bypassed for that one step. Everything after it still runs normally — including any further confirmation checks further down the list.

Why the client holds state

The backend generates the step list. The client owns execution state. This is intentional.

If the backend owned execution state, every step would require a round trip to persist progress. On a mobile device with unreliable connectivity, that's a reliability problem. The client already has the accessibility context, the current app state, and the user's confirmation response. Keeping the task state on the client means the agent can continue executing even if the backend is momentarily unreachable.

The backend only gets called again if a step requires a new LLM decision — for example, if an unexpected screen appears mid-task and the agent needs to figure out what to do next.

What's next

Part 10: handling unexpected screens mid-task — when the agent reaches a step and the expected UI isn't there.