Building an Android AI Agent, Part 11: Mid-Task Replanning Without Starting Over

In Part 10 we added an obstruction detector that handles spinners, dismissible overlays, and permission dialogs inline. When none of those apply and the expected screen simply isn't there, the executor returns a FAILED step with a reason.

That's where this part picks up. The task hasn't failed — two of four steps completed successfully. What failed is the plan for step three. The agent needs a new plan for the remaining steps, built on top of what already happened.

The problem with restarting from scratch

The naive recovery is to send the original instruction back to the LLM and regenerate the full step list. This has two problems.

First, it replays completed steps. If step one opened the right app and step two filled in a form, regenerating from scratch may produce a plan that tries to do both again — which either fails (the form is already filled) or causes a side effect (it submits twice).

Second, it throws away context. The LLM that generated the original plan didn't know what the screen looks like right now. The replanning call does — and that's the only reason to make it.

The right call: send the completed steps, the failed step, the failure reason, and the current screen snapshot. Ask for a revised plan for the remaining steps only.

The replan request

data class ReplanRequest(
    val originalInstruction: String,
    val completedSteps: List<AgentStep>,
    val failedStep: AgentStep,
    val failureReason: String,
    val currentSnapshot: ScreenSnapshot,
    val remainingStepCount: Int
)

data class ReplanResponse(
    val revisedSteps: List<AgentStep>,
    val explanation: String
)

The completedSteps tell the LLM what has already happened and must not be repeated. The failedStep and failureReason tell it what went wrong. The currentSnapshot tells it what the screen actually looks like right now. The remainingStepCount gives it a soft target — how many steps were originally planned for the rest of the task, so it doesn't over-engineer the recovery.

Serializing the snapshot for the LLM

The ScreenSnapshot needs to be readable by the LLM without overwhelming the context window. A flat summary works better than a full node dump.

fun ScreenSnapshot.toPromptSummary(): String {
    val sb = StringBuilder()
    sb.appendLine("Current screen: ${packageName ?: "unknown"}")
    if (visibleTexts.isNotEmpty()) {
        sb.appendLine("Visible text (first 10):")
        visibleTexts.take(10).forEach { sb.appendLine("  - $it") }
    }
    if (interactableIds.isNotEmpty()) {
        sb.appendLine("Interactable elements (first 10):")
        interactableIds.take(10).forEach { sb.appendLine("  - $it") }
    }
    return sb.toString()
}

Capping at 10 items each keeps the prompt tight. The LLM doesn't need the full accessibility tree — it needs enough to understand what's on screen and what's tappable.

The replan call

suspend fun requestReplan(
    request: ReplanRequest,
    apiClient: StareBrainApiClient
): ReplanResponse {
    val prompt = buildString {
        appendLine("Original instruction: ${request.originalInstruction}")
        appendLine()
        appendLine("Completed steps:")
        request.completedSteps.forEachIndexed { i, step ->
            appendLine("  ${i + 1}. ${step.instruction} [DONE]")
        }
        appendLine()
        appendLine("Failed step: ${request.failedStep.instruction}")
        appendLine("Failure reason: ${request.failureReason}")
        appendLine()
        appendLine("Current screen state:")
        appendLine(request.currentSnapshot.toPromptSummary())
        appendLine()
        appendLine("Generate revised steps for the remaining task only.")
        appendLine("Do not repeat completed steps.")
        appendLine("Target approximately ${request.remainingStepCount} steps.")
    }
    return apiClient.replan(prompt)
}

Merging the revised steps back into the task

fun AgentTask.withRevisedSteps(revised: List<AgentStep>): AgentTask {
    val kept = steps.take(currentStepIndex) // completed steps stay
    val merged = kept + revised
    return copy(
        steps = merged,
        status = TaskStatus.IN_PROGRESS
    )
}

The completed steps are preserved exactly. The revised steps replace everything from the failure point forward. The executor picks up from currentStepIndex as normal — it doesn't know a replan happened.

When not to replan

Not every failure deserves a replan. Two cases where replanning is the wrong call:

Repeated failures on the same step. If the revised plan hits the same failure on the same screen twice, the screen state is genuinely unexpected and the LLM isn't going to reason its way out of it. Surface the failure to the user with enough context to decide whether to continue or abandon.

Confirmed steps that failed. If the user confirmed an action and it still failed, that's a product bug, not a planning problem. Replanning would just generate another confirmation prompt for the same broken action.

fun shouldReplan(step: AgentStep, task: AgentTask): Boolean {
    if (step.result?.confirmedBy != null) return false // confirmed action failed — surface to user
    val previousAttempts = task.steps.count { 
        it.instruction == step.instruction && it.status == StepStatus.FAILED 
    }
    return previousAttempts < 2
}

What's next

Part 12: the AgentContext system — how StareBrain builds and maintains a model of what the user is doing across app switches, so the agent has context before the user says anything.